MongoDB indexes

Each MongoDB EventStore implementation creates a set of indexes on the events collection the first time it starts up. MongoEventStore, SpringMongoEventStore, and ReactorMongoEventStore all create the same indexes. Which ones get created depends on which capabilities the store has, STREAM or DCB, and whether the store writes a position. The groups below follow that split.

Always created

Name	Fields	Unique	Sparse	Purpose
`id` + `source`	ascending `id`, ascending `source`	yes	no	The CloudEvents spec requires `id` and `source` to be unique together. Without this index, two events with the same `id` and `source` could both get stored.

Stream capability

This index is created for a store with the STREAM capability, and also for a DCB-only store, because the DCB append path looks up the current stream version per partition too.

Name	Fields	Unique	Sparse	Purpose
`streamid` + `streamversion`	ascending `streamid`, ascending `streamversion`	yes	no	Enforces one version per stream slot and gives a fast lookup of the latest version on append. Without it, every append needs a collection scan to find the latest version, and duplicate versions become possible.

If this index already exists with different options than the ones above, the store fails to start rather than run without the uniqueness guarantee that stream and DCB writes depend on. Drop and recreate the index as unique out of band, then restart the store.

Global position

This index is created whenever the store writes a position. That is every DCB store, and every STREAM store unless position writing is turned off.

Name	Fields	Unique	Sparse	Purpose
`position`	ascending `position`	yes	yes, only events that carry a `position` field are indexed	Backs position-ordered reads and catch-up subscriptions. Without it, those reads fall back to a collection scan.

If you are upgrading an existing store to start writing position, see the position-backfill runbook for how to build this index up front instead of at startup.

DCB capability

These indexes are created for a store with the DCB capability, in addition to the streamid+streamversion and position indexes above. The dcbTags index is created only for a store that has both STREAM and DCB, the other two for every store with DCB.

Name	Fields	Unique	Sparse	Purpose
`dcbTags`	ascending `dcbTags`	no	yes	Created only for a store with both `STREAM` and `DCB`. Backs a DCB query that matches every DCB event (a read, `count` or `exists` with `DcbCriteria.all()`), and the append check of `DcbAppendCondition.wholeStoreLock()`. Only DCB events have a `dcbTags` field, so this index holds just those, and the query reads the DCB events instead of every stream event that has a `position` in the range. A test with 200,000 stream events and 200 DCB events examined 200 documents with this index and 200,200 without it. The `dcbTags` + `position` index can’t do this, because it also holds every stream event that has a `position`. A DCB-only store doesn’t get this index, because when its collection holds only DCB events, the index would hold the same events as the `position` index and add an entry per tag to every append.
`type` + `position`	ascending `type`, ascending `position`	no	yes	Backs a DCB query that has a type but no tags to match. Without it, the query walks the `position` index instead and checks the `type` of every DCB event in the range, one document at a time. A test on a 50k-event, 50-match skewed dataset examined all 50,050 documents to return 50 without this index.
`dcbTags` + `position`	ascending `dcbTags`, ascending `position`	no	yes	Backs a DCB tag-boundary query that also needs results in position order. Without it, the results are sorted in memory, or spilled to disk on MongoDB 6.0 and later, after every matching document is fetched. A test on a 305,000-event dataset with a 5,000-event popular tag used an in-memory sort stage instead of reading the index in order, without this index.

All indexes above are created automatically at startup. You do not need to create them yourself, except when following the position-backfill runbook for an existing store.

Adding a capability to an existing store

Declaring a new capability on a store that already holds events builds the new indexes when the application starts. Enabling DCB on a stream store adds dcbTags, type + position and dcbTags + position. The shared streamid + streamversion index is already there.

Enabling STREAM on a DCB-only store adds the dcbTags index, which a store with both capabilities needs. Without it, a DCB query with DcbCriteria.all() and the append check of DcbAppendCondition.wholeStoreLock() read every stream event that has a position as well as the DCB events. The results are the same, but the queries are slower, and they get slower as the number of stream events grows.

Unless the collection already has the index, startup builds dcbTags over the whole collection, and the store doesn’t start until MongoDB has finished building it.

On a large collection, do not let startup run those builds. Create the new indexes yourself before deploying the version that declares the capability, as a rolling build on MongoDB Atlas or any replica set, the same way step 1 of the position-backfill runbook pre-builds the position index. For dcbTags, run this before you enable STREAM on a DCB store:

db.events.createIndex({ dcbTags: 1 }, { sparse: true })

Startup then finds identical indexes and creates nothing. If one of your indexes has the same fields as an index in the tables above but different options, startup can fail. It always fails when your index also has the name the store gives that index, such as dcbTags_1.

Removing STREAM from a store that has both capabilities keeps the dcbTags index, because Occurrent never drops an index. A store that goes from STREAM alone to DCB alone doesn’t get dcbTags, though. If its collection still holds stream events that have a position, create dcbTags yourself with the command above.

A DCB-only store logs a warning with that command at startup when its collection holds a stream event with a numeric position and has no usable index on dcbTags alone. An index on dcbTags is usable when it:

is keyed on dcbTags and nothing else, ascending or descending
isn’t hidden
is sparse with no partialFilterExpression, or has { dcbTags: { $exists: true } } as its whole partialFilterExpression

If the collection has an index on dcbTags alone that isn’t usable, the warning names it and gives the commands that drop it and create the sparse one. When being hidden is the only thing wrong with it, the warning gives the collMod command that unhides it instead.

To allow for fast queries, for example when using EventStoreQueries, it’s recommended to create additional indexes tailored to the querying behavior of your application. See MongoDB indexes for more information on how to do this. If you have many adhoc queries it’s also worth checking out wildcard indexes which is a new feature in MongoDB 4.2. These allow you to create indexes that allow for arbitrary queries on e.g. the data attribute of a cloud event (if data is stored in json/bson format).