Database change notifications as a first step before adopting an API

Many teams we talk to have the same hesitation. They like the idea of reads served from memory and a cache they never have to maintain, but pointing an application at a new API is a change to code that works, and code that works is not something anyone changes lightly. So we offer a smaller first step: database change notifications.

The idea is simple. The redfly Sync Service already watches the database for changes so it can keep Redis in step. Before anything reads from Redis, and before a single line of application code moves, a team can subscribe to those same changes for the tables it chooses and react to them in the systems it already has.

What a notification is

A change notification is a message that says a row in a given table was added, changed or deleted, and which row it was. It comes from the same change capture mechanism the full product uses: change tracking read on a short interval on SQL Server, logical replication streamed on PostgreSQL, change streams streamed on MongoDB.

Nothing is installed inside the database beyond what that mechanism needs. No triggers, no reader of the transaction log (the database's own journal of every write) on SQL Server, no extra table the application has to write to. The service connects with the credentials it is given and sees only the tables that connection permits and the team has chosen.

Why start here

The thing most teams are doing today, if they have any caching at all, is polling. A job wakes up every few seconds, asks the database what changed since last time, and refreshes whatever it finds. Polling is load on the database that produces nothing most of the time, and it is always a little late. Notifications replace the question "has anything changed?" with the answer "this row changed", delivered when it happens.

That swap costs very little to adopt. The application does not change. The database does not change. A small consumer (a program that receives the notifications) is written to act on them, and the existing caching, however hand-rolled it is, gets a correct signal instead of a guess.

What you can build with notifications alone

Quite a lot, as it turns out. Three common uses:

• Cache refresh. The team's existing cache, whatever it is, drops or reloads an entry when the row behind it changes, instead of expiring it on a timer and hoping.
• Search index updates. A product or document index is updated when the source row changes, so search stops lagging the database by however long the nightly rebuild takes. The full product can sync to Azure Search directly; notifications let a team do the same for whatever index it already runs.
• Alerts and workflows. A row moving into a certain state, an order marked shipped, a stock level crossing a threshold, sends a message or starts a process, without anyone polling for it.

None of these require the redfly API. They require a subscription, a consumer, and a decision about which tables matter. That is a small piece of work, not a project.

The cheapest way to trust a system is to let it do one small thing correctly for a while.

What notifications do not give you

Being honest about the limits matters more than the list above. Notifications tell you a row changed; they do not serve the row to your application from memory. Your reads still go to the database, and the cache is still yours to keep correct, even if the signal driving it is now better. The 70% saving on the database line of the cloud bill comes from moving reads off the database, and that is the second step, not this one.

Delivery from the sync service is at-least-once, and a consumer of notifications should assume the same. It may see the same change twice, particularly after a restart or a dropped link, so it should treat a notification as "this row is now worth re-reading" rather than as an event to be counted.

The path from notifications to the API

Once a team has run notifications for a while, the second step is smaller than it looked at the start. The sync service is already connected, already authorised for the right tables, and already trusted to see every change. The next step is letting those same changes also land in Redis, the memory cache the service keeps in step with the database.

From there, reads move one place in the code at a time. A permission check, a profile load, a lookup that runs on every page: each one is switched from a hand-written query to the matching redfly entry point, Get or GetRows, and the rest stay as they were until someone gets to them. Writes keep going straight to the database throughout. If Redis is ever unreachable, reads fall back to the database on their own, so a half-migrated application is never worse off than an unmigrated one.

Notifications are the bridge redfly offers between a database you are not ready to change and an API you are not yet ready to trust; the same sync service carries you across both.