MongoDB change streams as a sync source

MongoDB is the second engine redfly supports, after SQL Server, which is available now. This article describes how MongoDB reports changes and how redfly uses that mechanism, so a team running MongoDB can see what the setup involves before support lands.

MongoDB stores documents rather than rows, and a document is the unit that changes. From the cache's point of view the mechanics are the same as for a relational table: something was inserted, updated or deleted, and Redis has to reflect it. We say documents below where MongoDB does, and rows where we mean the general idea.

Where change streams come from

A MongoDB replica set (a group of servers holding copies of the same data, one of which, the primary, accepts writes) keeps an operations log, called the oplog, that every write is appended to. The other members read that log to stay in step with the primary. It is the same idea as the write-ahead log in PostgreSQL: a journal that exists for the database's own replication and recovery.

Change streams are a supported way for an application to read that journal without touching it directly. A client opens a change stream against a collection, a database, or the whole deployment, and the server delivers each insert, update, replace and delete as an event, in order, once a majority of the replica set has saved it. The event carries the document's key, the kind of change, and either the full document or the fields that changed.

Because change streams require a replica set, a standalone MongoDB server cannot produce one. In practice that is rarely a constraint; production MongoDB runs as a replica set for durability anyway, and a single-node replica set is enough for development.

How redfly consumes the stream

The Sync Service opens a change stream on each collection it has been told to serve and applies every event to Redis. An insert, replace or update brings the cached copy up to date, a delete removes it, and reads through the redfly API then see the new state on the next call.

Nothing is polled. The stream stays open and the server pushes events as they happen, so the lag between a write and Redis reflecting it is the time to receive and apply the event. That is the same shape as PostgreSQL logical replication and different from SQL Server, where redfly reads change tracking on a short interval.

For remote sync, the same stream feeds a different destination. A site running MongoDB has the sync service subscribe to its change stream, queue the events on site, compress and encrypt them, and send them on to the central store. The central store is itself MongoDB, so the documents arrive as documents, stamped with the site they came from.

Resume tokens: picking up after an interruption

Every event in a change stream carries a resume token, a value the reader keeps but never needs to interpret, which marks that event's position in the oplog. A consumer that saves the token after applying each event can reopen the stream later and ask to resume from that token. The server then replays every event that happened after it, in order, and the consumer catches up without having missed anything.

This is what makes a change stream a safe sync source rather than a convenient one. Network faults, restarts, deployments and a change of which server accepts writes (an election) all interrupt the stream. With the token saved, each of those is a pause rather than a loss. Without it, a consumer would have to rescan the collection to be sure it had not missed a change, which on a large collection is the very load a cache exists to remove.

Save the token after the write, never before it.

The order of operations matters. The token is saved only once the event has been applied to Redis or handed to the site queue. Saving it first and then failing to apply means a change is skipped for good. Applying first and then failing to save means the event is replayed on resume, and applying the same insert, update or delete twice leaves Redis in the same state, which is why delivery is at-least-once and reconciled, and why it is safe to be so.

The cautions

Resume tokens point into the oplog, and the oplog is a fixed-size log that discards its oldest entries as new ones arrive. If a consumer is offline for longer than the oplog retains, its token no longer points at anything and the resume fails. The fix is a backfill of the affected collection from current data, which is the sync orchestrator's job; the prevention is an oplog sized for the longest outage you expect, which is a setting the administrator controls.

Two smaller points. An update event can carry only the changed fields rather than the whole document, so a consumer that needs the full document either asks the server to include it in the event, which MongoDB supports, or fetches it afterwards. And change streams respect the same access controls as any other read, so the account redfly uses needs read access to the collections it serves and to the change stream itself, and nothing more.

redfly fits here as the consumer that opens the streams, saves the tokens in the right order, and keeps Redis and the central store in step, once MongoDB support lands next after SQL Server.