redfly.ai insights

How to centralize warehouse data from every site without replacing what runs the warehouse

2026-09-25T00:00:00Z

The question head office asks is simple: how many orders shipped from every site today, and how many are still waiting? The reason it is hard to answer is not the arithmetic. It is that each warehouse runs its own warehouse management system (the software that tells people what to pick, pack and ship), on its own server, in its own building, and none of those servers know the others exist.

The instinct is to fix this by standardizing: one system, one database, every site. That is a multi-year program with real risk to the operation, and most of it produces no new revenue. There is a less disruptive shape, and it starts from a different assumption: the warehouse system is fine where it is.

Leave the warehouse alone

The system that runs the floor was chosen, tuned and trusted over years. The people on the floor know its screens. The scanners talk to it. When it stops, trucks wait, so nobody with operational responsibility wants to swap it out to get a better report.

So the first rule of centralizing is that nothing at the warehouse is replaced or rewritten. The warehouse management system stays. Its database stays. The version stays. All that is added is a small service, installed on a machine inside the site's own network, next to the database it already runs.

What the small service does

That service watches the tables you choose, the ones head office actually needs: orders, shipments, inventory movements, whatever matters for the reports. When a row in one of those tables is added, changed or deleted, the service notices, queues the change on site, compresses it, encrypts it with that site's own encryption, and sends it outward to a central store in your cloud account.

Outward is the important word. The service opens connections from inside the site to the outside and never the reverse. Nothing reaches back in, and the central store never writes anything to the warehouse database. The flow is one direction, site to center.

After an initial copy of the chosen tables, only what changed leaves the site. A busy warehouse sends a steady trickle of small changes, not a fresh export of the whole database every night.

What changes for the head office

Every site's changes land in one central store. Each row is stamped with the site it came from, and each site keeps its own space in that store, so a row from the Texas depot is never confused with a row from the Georgia one. The central copy is current to within a couple of minutes of what the warehouse database holds.

That is what your reports read. A dashboard, a scheduled export, a regional manager's morning view, a customer's tracking portal; all of them point at the same central copy, not at thirty separate warehouses. The numbers agree because they come from one place.

The practical difference is who does the waiting. Today, head office waits for somebody at each site to run an export and send a spreadsheet, then waits again to find out why two spreadsheets disagree. With a central copy, the question of how many orders are still waiting has one answer, and it is a few minutes old rather than a day old.

Your team builds those reports and any portal. We advise on the shape of the data and supply tools to build against it, but the reports are yours, in whatever reporting tool you already use.

What does not change for the floor

The floor should notice nothing. The warehouse system behaves exactly as before, because it has not been touched. The pickers use the same screens, and the scanners hit the same database.

Two details keep that promise honest. First, the service holds back when the site's machines are busy; it is a guest on that server and behaves like one. Second, reports run against the central copy rather than the warehouse database, so the reporting queries that used to compete with picking and packing are gone from the onsite server entirely. That leaves the warehouse server with more headroom than it had before, which is what keeps it steady through a peak.

If the internet link drops, the service keeps queuing changes on site and sends them when the link returns, resuming table by table from where it stopped. The floor does not notice that either. The onsite queue is a holding queue for outbound changes, nothing more; it is not a copy the warehouse can read from while cut off.

Rolling it out

Each site needs its own setup. A person from our side, with your one IT contact, installs the service, chooses the tables, and runs the first sync (the initial copy of the chosen tables). It is the same rollout every time rather than a fresh build, but it is still work done by a person, not a switch you flip. Adding a table later is a setting change, and again someone does it.

We recommend starting with one site and one set of tables, getting something useful into the hands of head office early, and then rolling out at your own pace.

The warehouse keeps running its own system; the head office finally gets to see all of them at once.

redfly builds and runs this service with logistics companies as design partners, starting with a single site on SQL Server, MongoDB or Postgres.

Why reports should never run on the warehouse database

2026-09-25T00:00:00Z

Ask a warehouse manager what the database is for and the answer is immediate: it tells the floor what to do next. Which order to pick, which tote to pack, which dock door the truck is at. Every second it spends on anything else is a second the floor is waiting.

Now ask who else runs queries against it. The finance team's month-end extract, the regional manager's morning dashboard, and the customer service tool that refreshes order status every thirty seconds. A consultant's one-off analysis that turned into a scheduled job two years ago and never got turned off.

None of these people are doing anything wrong. They are all sharing one server with the pickers, and the pickers did not agree to share.

Reports and the floor compete for the same machine

A database server has a fixed amount of processor, memory and disk throughput. A picking transaction needs a sliver of each, thousands of times an hour. A report that sums six months of shipments needs a large slice of all three, once. When they run at the same time, the report does not politely wait; the database splits its attention between the two, and the picking transactions get slower.

On a quiet Tuesday nobody notices. The server has room, the report finishes, the scanners respond in a fraction of a second. The problem shows up on the day the server has no room to spare.

Peak is when it matters

The busiest day of the year is also the day everyone wants a report. Volume is up, head office wants hourly numbers, customers are refreshing tracking pages, and the same reports that were harmless in March are now being run against a server already working hard. That is the moment the scanners start taking two seconds instead of a fraction, and a two-second scanner delay across a floor of pickers is a lot of lost throughput.

The reflex is to buy a bigger server. That buys headroom for one peak, at the price of a hardware step every time volume grows, and it still leaves the reports on the same machine as the floor. The reflex is treating the symptom.

Move the reports, keep the database

The alternative is to take the reporting load off the warehouse database entirely, without touching the warehouse system. A small service beside the database watches the tables that reports need and sends every added, changed or deleted row outward to a central copy in your cloud account. After the first copy only changes are sent, compressed and encrypted per site, and the central copy is current to within a couple of minutes.

Reports read from that central copy. All of them: the dashboards, the month-end extract, the customer portal, the consultant's forgotten job. The warehouse database goes back to doing one thing, which is running the floor.

This is not a claim about speed. The central copy is not faster than the warehouse database and we do not sell it as such. The claim is about headroom: a server that no longer serves reports has capacity it did not have before, and that spare capacity is what keeps it steady when the peak arrives.

What headroom buys you

Headroom is resilience. A server with room to spare absorbs a surge; a server running close to its limit falls over at the first one. Removing the reporting load is the cheapest way we know to move a warehouse server from the second condition to the first, because it removes work rather than adding hardware.

It also changes who owns the risk. Today, a badly written report can slow the floor and nobody finds out until the pickers complain. When reports run against the central copy, a badly written report is slow for the person who ran it, and the floor never knows.

The service itself is built to be a good guest. It holds back when the site's machines are busy, so even the work of sending changes outward gives way to the floor. If the internet link drops, changes queue on site and resume when the link returns; nothing is lost, and nothing on the floor stops.

The rule, stated plainly

The warehouse database exists to run the warehouse. Anything that reads from it for another purpose is borrowing capacity from the floor, and the loan comes due on the busiest day of the year. Reports should read from a copy, and the copy should be kept current by sending changes outward, not by running exports against the database at night.

There is one honest limit to state. The central copy lags the warehouse by a couple of minutes, so it is the wrong place for a floor supervisor who needs the next pick right now. That supervisor stays on the warehouse system, which is exactly where the warehouse system's capacity should go.

Every report on the warehouse database is a loan against the floor, repayable at peak.

redfly supplies the service that keeps the central copy current across every site; your team builds the reports against it, with our support.

What offline support really means when a warehouse loses its link

2026-09-25T00:00:00Z

Offline support is one of those phrases that means whatever the listener hopes it means. To a warehouse manager it may sound like the floor keeps working from a full local copy of everything. To a network engineer it may sound like a promise nobody can keep. It is neither, so this is a precise description of what the redfly sync service does when a site loses its link, and what it does not do.

The shape of the system

At each site a small service runs inside the network, next to the warehouse database. It watches the tables you have chosen and captures rows that are added, changed or deleted. On SQL Server it reads change tracking (the engine's built-in list of which rows changed) on a short interval; on PostgreSQL it uses logical replication and on MongoDB change streams, both of which stream each change to the service as it happens.

Captured changes go into a queue on site. Each batch is compressed and encrypted for that site, then sent outward over an ordinary internet connection to a central document store in your cloud account. The service opens outbound connections only; nothing connects inward.

What happens when the link drops

The capture side does not care about the link. It keeps reading changes from the database and appending them to the onsite queue for as long as the warehouse keeps working. The queue is compressed, so a site can be cut off for a long time before space becomes a concern.

The sending side notices the failure and stops. It discards nothing. When the link returns, sending resumes.

Resumption is per table. Each table remembers its own place and picks up from there. A restart of the service or the machine has the same effect; each table resumes from where it stopped rather than starting over. A long outage therefore costs you the backlog transfer time and nothing else.

Delivery is at-least-once, reconciled on arrival

Here is the part worth being precise about. When the link fails mid-batch, the service cannot always know whether the central store received that batch before the failure. The safe choice is to send it again. That means delivery is at-least-once: every change will arrive, and some changes may arrive twice.

Duplicates are reconciled on arrival. Every row is stamped with the site it came from, and the central store recognizes a change it has already seen, so a repeat does no harm. The central copy ends up correct; it gets there by tolerating repeats rather than by trying to prevent them, which over an unreliable link is the approach that actually works.

Because each table resumes on its own, one table may briefly be further along than another while a backlog catches up. A report that joins two tables during that window may see one ahead of the other until the backlog clears.

What is not promised

The onsite queue is a holding area for outbound changes. It is not a read copy. The warehouse management system cannot read from it, and no application at the site can use it as a local database while the site is cut off. The warehouse keeps working during an outage because its own database is untouched and local, not because of anything the sync service provides.

Nothing flows back in. The central store never updates the warehouse database, and the service is not built to do so. If you need changes to travel toward the site, this is not the tool.

Freshness is a couple of minutes end to end when the link is healthy, roughly thirty seconds to two minutes. During and after an outage it is however long the outage lasted plus the backlog transfer. There is no promise that any particular change is visible centrally by a particular moment.

The service holds back when the site's machines are busy. Under sustained heavy load at the site, the central copy lags further than usual until the load eases. That is by design; the warehouse always comes first.

Why it is built this way

Every one of these choices trades a stronger guarantee for a system that keeps working over links that cannot be trusted. A delivery guarantee with no repeats would require coordination that a dropped connection breaks; at-least-once with reconciliation does not. A readable local copy would put a second database at every site to install, patch and keep consistent; a queue does not.

The result has moved more than ten billion rows over ordinary public internet links in production. That is the evidence we have that the trade is the right one, and it is the reason we describe the guarantees narrowly rather than generously.

Offline support means nothing is lost and nothing has to be restarted; it does not mean the site can read a copy while cut off.

redfly runs this service at each site and keeps the central store current; the reports that read from it are yours to build.

One report across time zones: getting the clocks to agree

2026-09-25T00:00:00Z

A logistics company with a warehouse outside Atlanta, another outside Dallas and a third outside Los Angeles has three different ideas of what today means. When the Georgia site closes its day, California is still shipping for three more hours. When head office asks for the day's numbers, every site answers about a slightly different day.

This is usually handled by not handling it. Each site runs an export at its own midnight, someone at head office stitches the spreadsheets together the next morning, and the definition of yesterday is whatever the stitching produced. It works until someone asks a question the stitched sheet was not built to answer.

Where the clocks disagree

Three separate problems hide inside the phrase time zones, and they need separating.

Each warehouse database stamps its rows in local time, or in universal time (the single worldwide clock that ignores zones), or in whatever the system's vendor chose years ago; the three sites rarely match.
Exports run at different local moments, so the daily file from each site covers a different window of universal time.
Head office adds its own zone on top when it reads the files, so the same shipment can appear on two different days depending on who is looking.

None of these are solved by asking the sites to change their systems. The warehouse management system's clock behavior is baked in, and nobody wants to touch the system that runs the floor to fix a reporting problem.

One copy, every row stamped with its site

The approach we take leaves each site's system exactly as it is. A small service beside each warehouse database sends every added, changed or deleted row outward to one central store in your cloud account. Every row that lands centrally is stamped with the site it came from, and each site keeps its own space in the store.

That stamp is what makes the time zones tractable. Once you know a row came from the Texas site, you know its local zone, and you can convert its timestamps to a single reference consistently, every time, in the report rather than at the warehouse. Three inconsistent clocks become three known offsets from one clock.

The central copy is current to within a couple of minutes, end to end. There is no nightly export, and so there is no export window to reconcile. The question of what shipped in the last hour across all three sites has an answer at ten past the hour, not tomorrow morning.

A day, a shift or a peak

With every row stamped and current, head office can define its own windows and apply them uniformly.

A calendar day in one reference zone, so a day is the same twenty-four hours at every site.
A shift, defined in local time at each site, so the night shift in Georgia and the night shift in California are compared like for like.
A peak event, from a chosen start moment to a chosen end moment, measured against all three sites together as it unfolds.

The first is the finance view; the second is the operations view; the third is the one everybody wants at nine in the morning on Cyber Monday and could never get before. They are three reports on the same data, not three data pipelines.

What head office gives up, and what it does not

The central copy is not the warehouse database, and it lags it by a couple of minutes. For a report that is a rounding error. For a floor supervisor who needs to know what the next pick is, it is the wrong tool, and that supervisor should stay on the warehouse system, which is untouched.

What head office no longer does is wait. It does not wait for a site to run an export, does not wait for a spreadsheet to arrive, and does not spend the morning working out why two sheets disagree. Regional managers and a customer portal read the same central copy, so there is one set of numbers rather than one per person who asked.

Keeping it honest across bad links

Sites are not always connected. If the link at one site drops, its changes queue on site, compressed and encrypted, and resume from where they stopped when the link returns. Nothing is lost, but during the outage that site's rows stop arriving, and a cross-site report for that window will be missing them until they catch up.

A useful report shows this rather than hiding it: the time of the newest row received from each site, right on the dashboard. A regional manager who sees that the Texas feed is forty minutes behind knows to read the Texas column accordingly. That is one of the first things we recommend building, and it costs almost nothing once the site stamp is there.

Three warehouses, three clocks, one copy stamped by site; the arithmetic becomes a report, not a morning.

redfly keeps the central copy current from every site; your team builds the day, shift and peak views on top of it, with our support.

The true cost of a cloud copy of your warehouse data, and the cheaper shape

2026-09-25T00:00:00Z

When a logistics company decides to centralize its warehouse data, the moving of the data gets all the attention. It is the visible part: the export jobs, the network, the schedule. It is not where the money goes. Most of the cost of a central copy is what you pay for it to sit there, month after month, waiting to be read.

So it is worth being clear about the shape of the bill before choosing a design, because two designs that produce the same reports can differ in cost by a large multiple.

The expensive shape

The conventional design lifts everything into a cloud relational database (a database engine like the one already running the warehouse, only rented by the hour). It is familiar, the reporting tools connect to it, and it is the first thing a cloud vendor suggests.

Three things make it expensive.

A relational database engine is priced for transactions, not for holding data. You pay for processing power capable of running a warehouse, twenty-four hours a day, to answer reports that run a few times a day.
It is sized for peak. To be safe on the busiest day, the rented server is sized for the busiest day and billed for it every day.
Each site's data adds to the same engine, so the rented server grows with every warehouse you add, and licensing often grows with it.

The result is a central copy that pays for a database engine, per site, every month, forever, for data whose main job is to be summed once a week.

The cheaper shape

The alternative changes what the central copy is, not what it contains. Instead of a relational engine, the central store is low-cost document storage: a store that holds rows cheaply and charges mostly for the space they occupy. It is priced for holding data, which is what a reporting copy does nearly all the time.

The ratio matters more than any particular number. Storage of this kind costs a small fraction of a comparably sized relational database, and it carries no per-site engine license. Adding a site adds storage in proportion to that site's data, and nothing else.

Reports still run. Your reporting tools read from the document store, and for the reports a logistics company actually needs, daily throughput, open orders, inventory by site, exceptions, the store is entirely adequate. We do not claim it is faster than a relational engine; we claim it is cheap to keep, which is the dimension that dominates the bill.

Send changes, not copies

The second cost is moving the data, and here too the shape matters. The conventional design exports each site's tables every night and reloads them centrally. Every night, the whole table, over the wire, whether or not it changed.

The design we use sends only what changed. A small service at each site watches the chosen tables and sends added, changed or deleted rows outward, compressed and encrypted, after a single initial copy. A site whose tables hold years of history but change only a little each hour sends only that little. The volume over the wire tracks activity at the site, not the size of its history.

That volume is small enough to travel over the site's ordinary internet connection. No dedicated line, no private circuit into the cloud, no monthly cost per site for the link. Sites with high-latency or spotty connections are handled, because the service queues changes on site when the link drops and resumes when it returns.

Your account, your rates

The central store can run inside your own cloud account. That means whatever discounts you have already negotiated with your cloud vendor apply to it, and there is no hosting markup layered on top by us. It also means the data never leaves your control.

Putting the shapes side by side: the expensive design pays for a peak-sized relational engine that grows with every site, forever, plus nightly full exports over links sized for them. The cheaper design pays for storage in proportion to data held, transfer in proportion to activity, and an ordinary internet link the site already has. The reports come out the same.

What the cheap shape does not do

It does not replace the warehouse database, and it is not meant to. The warehouse system keeps running on its own server, untouched. It is not a live operational copy either; the central store is current to within a couple of minutes, which is right for reporting and wrong for running a floor.

And it is not free of effort. Each site needs its own setup, driven by a person, and adding tables later is a change someone makes. The cost that disappears is the recurring one, the engine that sits there running, which is the cost that was never worth paying for reports.

Most of the price of a central copy is not the moving; it is what you pay for the data to sit there.

redfly builds this cheaper shape with logistics companies as design partners, one site first, on a subscription that includes the architecture review.

What a design partnership looks like for a logistics company: one site first

2026-09-25T00:00:00Z

We do not sell this as a self-serve product, and we are not going to pretend it is one. Centralizing warehouse data across sites is work done with a company, not to it, and the way we do that work is a design partnership. This is what one involves, so you can judge whether it fits.

Who it is for

Four things make a logistics company a good fit.

More than one site. If you run a single warehouse, you can already see across it. The problem we solve starts at two.
SQL Server, MongoDB or Postgres at the sites. Any of those is a quick start; SQL Server is ready now, MongoDB next, Postgres after that. Other databases are possible, but tell us what you have before assuming.
One IT contact who can get us access to one site. Not a committee, not every site's administrator; one person at one site is enough to begin.
An appetite to start small. One site, one set of tables, something useful in your hands early, then the rest at your pace.

If you are missing the first or second, we are probably not the right partner yet. If you are missing the third, that is the thing to find first.

The first site

The first site is where the shape gets decided, so it gets the most attention. It begins with an architecture review against your actual schema (the tables and columns your database really has): which tables the head office needs, how rows are identified, how deletions show up, what the site's internet connection looks like, and what the reports should say. That review is part of the subscription rather than a separate engagement.

Then a person from our team, working with your IT contact, installs the small sync service inside the site's network, next to the warehouse database. It sends changes outward and nothing connects inward, and nothing at the warehouse is replaced or rewritten. The first sync copies the chosen tables to the central store in your cloud account, and from then on only changes travel.

Within a short time you have a central copy of one site, current to within a couple of minutes, and a first report reading from it. That is the moment to judge the thing: not a slide, but your rows, from your warehouse, on a screen at head office.

Who does what

A person drives the rollout. The service is not something that installs and configures itself, and we do not describe it that way. Our engineer runs the rollout tool, makes the choices, checks the result, and stays in contact as the first days of data arrive.

Your team builds the reports and any portal. We do not build them for you, and this is deliberate: you know what a regional manager needs to see and we do not. We advise on the shape of the data, supply tools for building against the central copy, and recommend what to build first. The reports, the dashboards and the portal are yours, in your reporting tool, owned and changed by you.

Your IT contact grants access and answers questions about the site. That is the extent of the demand on them for the first site.

Adding sites and tables

Each additional site needs its own setup. It is the same rollout every time, not a fresh build, but it is a planned piece of work with a date, a person and a checklist, and the site's own IT staff involved. We schedule sites at your pace.

Adding a table to a site that is already running is a setting change, and again a person makes it. It is quick, but it is work someone does, and we say so because a promise that you can just flip a switch is the kind that causes trouble later.

How the subscription works

Design partners pay a subscription rather than a project fee. Keeping a central copy current from many sites is continuous work; schemas drift, sites go offline, someone has to notice when a warehouse quietly stops sending. A one-off fee would leave you with something nobody is paid to look after.

The minimum term is twelve months, paid annually at the lower rate or monthly across the same term. The subscription includes the architecture review, deployment, support, monitoring and improvements for as long as you subscribe, with direct access to the engineers who built it. There is no free trial and no free review; the first site, on the subscription, is the proof.

Design-partner pricing is an early-customer rate held for the length of your contract, and the program is open to a small number of companies rather than indefinitely.

One site, one set of tables, your own rows on a screen at head office; then decide how far to go.

redfly works this way with a small number of logistics companies at a time, starting from a single site.

Outbound only, encrypted per site: how warehouse data leaves the building safely

2026-09-25T00:00:00Z

Any proposal to send warehouse data to the cloud arrives at an IT security reviewer's desk eventually, and the reviewer's job is to assume the worst. This is a description of how the redfly sync service moves data out of a site, written for that reviewer, with the questions they are likely to ask and the honest answers.

Nothing connects inward

The service runs on a machine inside the site's network, behind the site's existing firewall, next to the warehouse database. It opens connections outward, to the central store, and to nothing else. It accepts no inbound connections; nothing reaches in.

The consequence for the firewall is simple. No inbound rule is needed; the service only has to be allowed out to the central store. If the site's outbound policy is restrictive, that is the single exception to add.

The service reads from the warehouse database using login details you provide, and it touches only what those details permit. It needs to read the tables you choose; on SQL Server it reads change tracking (the engine's built-in list of which rows changed) for those tables. It never writes to the warehouse database, and the login you provide need not allow it.

What leaves the building

Only rows from the tables you have chosen, and only when they are added, changed or deleted after the initial copy. The service does not read tables outside its list.

Before anything leaves, each batch of changes is compressed and then encrypted with that site's own encryption, so that only you can decrypt it. Compression comes first, because encrypted data does not compress; encryption second, so the bytes on the wire are unreadable to anyone in between.

Encrypting per site matters for a specific reason. If one site's encryption is ever exposed, it opens that site's data and nothing else. A single shared key across thirty warehouses would make every site's data hostage to the least careful one.

Where it lands

The central store is low-cost document storage, and it can run in your own cloud account, under your identity and access rules rather than ours. Each site keeps its own space in the store, and every row is stamped with the site it came from. Access to reports can therefore be granted per site: a regional manager sees their region's spaces, a customer portal sees only the rows for that customer, head office sees all of it.

When the store is in your account, your existing logging, retention and key management apply to it, and your rows sit alongside nothing that is not yours.

The questions a reviewer asks

The same questions tend to come up in this review. Here are the ones that matter and the straight answers.

Can the central side reach into the site? No. There are no inbound connections and the service is not built to accept one. The central store cannot write to the warehouse database.
What if the service's machine is compromised? What is on that machine is the read-only login to the chosen tables, that site's queued changes and that site's encryption. Other sites are encrypted separately, and a read-only login cannot alter the warehouse database.
What if the central store is compromised? The store sits in your cloud account under your own access controls, so what can be read there is governed by the same policy that protects the rest of your account.
Does it use triggers or a transaction-log reader on SQL Server? No. It reads change tracking on a short interval, a built-in feature of the engine; there are no triggers and no log reader.
What happens on the wire during an outage? Nothing. Changes wait on site, compressed and encrypted in the queue, and resume when the link returns.
Is delivery guaranteed? At-least-once, reconciled on arrival. Nothing is lost; some changes may be sent twice and are recognized centrally.

What we do not claim

We do not claim the service is invisible to the warehouse server; it is a process that uses some processor time and disk, and it holds back when the machine is busy. We do not claim the central copy is a security boundary for the warehouse; the warehouse's own controls remain the controls that matter. And we do not claim the data is anonymous once it arrives; it is your operational data, in your account, and how it is governed there is your policy applied to your store.

What we do claim is narrow. Data leaves the site through one outbound channel, encrypted for that site, only for the tables you chose, and lands in a space of its own in a store you control.

One outbound connection, one key per site, one space per site, nothing reaching back in.

redfly installs and runs this service at each site as part of a design partnership, with your IT contact and your security reviewer in the loop from the first site.

Peak season for warehouses: why headroom matters more than horsepower

2026-09-25T00:00:00Z

Every logistics operation knows its peak. For most it is the stretch from Black Friday through Cyber Monday and the shipping days after, when volume is several times a normal week and every hour of throughput is money. It is also the week when the warehouse server, which nobody thinks about the other fifty-one weeks, decides whether the operation has a good peak or a bad one.

The conventional preparation is to make that server bigger. More processors, more memory, faster disks, ordered in September and installed in October. It is the horsepower answer, and it is the wrong one, for reasons that only become visible on the day.

What actually happens at peak

On a normal day the warehouse database spends most of its capacity on the floor: picks, packs, ship confirmations, inventory moves. It also serves reports; the head office dashboard, the customer service refresh, the month-end extract, a dozen scheduled jobs nobody remembers creating. On a normal day the two workloads fit comfortably on the same machine.

At peak the floor workload grows several times over. So does the reporting workload, because peak is precisely when everyone wants hourly numbers and customers refresh their tracking pages constantly. Both grow at the same time, on one machine with a fixed ceiling, and when they reach the ceiling the database does not prioritize the pickers. It slows everything, evenly.

Why horsepower is the wrong answer

A bigger server raises the ceiling. It does not change what is under it. The reports are still there, growing with the floor, and the two workloads still collide; they collide later, at a higher volume, on a more expensive machine.

Three other problems come with it.

The upgrade is sized for a peak that is guessed at in September. Guess low and the server is short on the day; guess high and you have paid for capacity that sits idle for fifty-one weeks.
It is a step, not a slope. Servers come in sizes, and the next size up is often a multiple of the price of the current one, for a fraction more capacity than you needed.
It is a change to the machine that runs the floor, made weeks before the busiest day of the year, which is exactly when nobody wants to be changing anything.

Horsepower treats the ceiling as the problem. The problem is what is competing for the room under it.

Headroom: removing work instead of adding capacity

The other answer is to take the reporting workload off the warehouse server entirely. A small service beside the warehouse database sends every added, changed or deleted row outward to a central copy in your cloud account, compressed and encrypted per site, current to within a couple of minutes. Reports read from that central copy, all of them, and the warehouse database stops serving reports at all.

What that leaves behind is headroom: capacity the server had all along but was spending on reports. At peak, that headroom goes to the floor. The server that was close to its limit on last year's Cyber Monday has the same hardware this year with one whole workload removed, because the reports are gone rather than accommodated.

This is not a speed claim. Reports do not get faster by running against a central copy, and the floor does not get faster either. The claim is resilience: a server with room to spare absorbs a surge, and a server with none falls over at the first one. Headroom is what keeps the floor steady when the volume arrives, and removing work is the cheapest way to get it.

The service knows when to stay out of the way

The sync service is itself a process on the site's machine, so it is fair to ask whether it becomes a third workload at peak. It is built not to. It holds back when the site's machines are busy, sending fewer changes while the floor is working hard and catching up when the pressure eases. The warehouse always comes first.

The cost of that restraint is that the central copy may lag a little further than usual during the busiest hours. For head office that is the right trade; a dashboard a few minutes behind on Cyber Monday is fine, and a slow scanner on the floor is not.

Peaks also stress the network. If a site's internet link drops under load, changes queue on site and resume from where they stopped when it returns, table by table. Nothing is lost, and the floor never notices, because its own database is untouched and local.

What to do in September

The better September project is moving the reports rather than ordering a server: the tables head office needs are chosen, the central copy is stood up from one site, the dashboards and the customer portal are pointed at it, and the reporting load leaves the warehouse server before the peak rather than during it. We have handled Black Friday and Cyber Monday peaks with the database running close to its limits, and headroom, not horsepower, is what got the floor through.

A bigger server raises the ceiling; taking the reports off it gives you the room.

redfly supplies the service that moves the reporting load off each warehouse server; the reports themselves are built by your team, with our support.

Transparent caching: why your codebase should not know a cache exists

2026-09-25T00:00:00Z

Open any mature backend and search for the word "cache". You will find it in the data access layer, in a handful of controllers, in a background job that warms keys on deploy, and in a comment that says "do not remove, see incident". The cache started as one helper class. Years later it is a second data model that nobody fully owns.

That spread is not carelessness. It is what happens when the application is responsible for both asking for data and deciding where the data lives. We think those two jobs should be separated, and that the application should only ever do the first one.

What transparent actually means

A transparent cache is one the application cannot see. The code asks for a row by its key, or asks for a list of rows matching some fields, and gets an answer. Whether that answer came from memory or from disk is not the application's concern, and there is no way for the code to tell.

In practice this means the codebase contains no cache client, no time-to-live settings (the timers that decide how long a cached value is trusted before it is thrown away), and no invalidation logic (the code that removes or refreshes a cached value after something changes). Those three things are where most cache bugs live, and removing them from the application removes the bugs with them.

One API, two paths

redfly does this by putting a single API between the application and the database. The API offers six operations: Get, Insert, Update, Delete, GetRows and GetTotalRowCount. They are available over REST (ordinary web calls) and gRPC (a compact, fast way for services to call each other). That is the entire surface the application sees.

Behind the API, reads and writes take different paths. Reads are served from Redis, an in-memory data store that answers in about a millisecond. Writes go straight to the database, which remains the source of truth. The application does not choose the path; it simply calls Get or Update and the API routes the call.

Who keeps the cache honest

The obvious question is how Redis stays correct if nobody in the application is updating it. The answer is a separate sync service that watches the database for changes and pushes them into Redis. On SQL Server it uses change tracking, read on a short interval. On PostgreSQL it uses logical replication, and on MongoDB it uses change streams; both of those are streamed as they happen.

Because the sync service watches the database rather than the application, it does not matter where a write came from. Rows changed by the API, by a nightly batch job or by an administrator at a console all reach Redis the same way. Hand-written caches usually miss the last two, which is why stale data so often appears after a "harmless" script runs.

Why every hand-written cache layer rots

We have watched this pattern for two decades, and the decay follows the same steps each time.

A developer adds a cache in front of one slow query. It works, and it is small.
Other developers copy the pattern for their own queries, each choosing their own key format and expiry timer.
A new write path appears, often a bulk import or a fix script, and it does not know about the cache. The first stale-data bug ships.
Someone adds invalidation calls after every write they can find. They miss some. The invalidation code itself now has bugs.
The team stops trusting the cache, shortens every timer, and the database load creeps back up. The cache is still in the codebase, but it is barely doing anything.

At every step the cache is a little more entangled with the product code, and a little harder to remove. The rot is not a sign of a weak team. It is the natural result of putting cache policy in the same place as business logic.

A cache the application can see is a cache the application will eventually break.

What the application gains

When the cache is invisible, a few things change for the engineers who work on the product. Code reviews no longer include the question "did you remember to invalidate". Test suites do not need a fake cache to be set up and torn down. A new engineer can read the data access code and understand it in an afternoon, because it looks like ordinary data access.

One engineering leader who reviewed a system built this way said they understood how the code worked but could not find where the caching was done. That is the intended result.

The honest limits

Transparent does not mean magic. The cache is only as fresh as the sync service keeps it, so there is a short window after a write in which a read may return the previous value. For most application reads, such as permission checks, lookups and profile loads, that window is fine. For the rare read that must see its own write instantly, the design conversation has to happen up front.

Fallback is part of the contract

A transparent cache also has to be transparent when it fails. If Redis is unreachable, the redfly API serves reads from the database instead, automatically. The application sees slower responses, not errors, and no code in the application has to switch modes.

redfly sits between an application and its SQL Server, MongoDB or PostgreSQL database, serving reads from a Redis cache it keeps in sync and passing writes straight through. The codebase never sees the cache, which is the point.

Joins on Redis: serving multi-field queries from memory

2026-09-25T00:00:00Z

Ask an engineer what Redis is for and the answer is usually "fast lookups by key". Give it a user ID, get the user back in about a millisecond. That is what Redis, an in-memory data store, is best at, and where most caching projects start and stop.

The trouble is that most screens in a real application are not a lookup by key. They are lists: orders for this customer in this date range, open tickets assigned to this team, products in this category that are in stock. Those reads need a filter across several fields, or a join across two or more tables, and a key-value store has no native way to answer them.

Where the database quietly takes the load back

So teams draw a line. Single-key reads go to Redis; anything with a filter (a WHERE clause, in database terms) or a join goes to the database. The cache hit rate (the share of reads answered from memory rather than disk) looks healthy on the dashboard, because the single-key reads are numerous. But the list screens are the ones users sit on all day, and every one of them is still a database query.

The database ends up carrying the reads that cost the most, while the cache absorbs the reads that were cheap anyway. That is the opposite of what a cache is for. It is also why teams that "already have Redis" often still see the database at high load during peaks.

Why joins are hard in a cache

The reason teams stop at single keys is not laziness. Serving a join from memory means the cache has to hold complete, current copies of every table involved, in a shape that can be filtered and combined without going back to disk. If any row in any of those tables changes, the cached answer to every query that touched it may now be wrong.

With a hand-written cache, that quickly becomes unmanageable. Either the team caches whole query results and accepts that they go stale, or they try to track which cached results depend on which rows, which is a small database engine written by accident. Most give up and send the query to the real database, which is the sensible call given the tools they have.

Keeping the data itself in step

redfly takes the other route. Its sync service keeps the data itself in Redis, rather than remembered answers to past queries, and keeps it in step with the database as rows are added, changed and deleted. On SQL Server that is done with change tracking, read on a short interval; on PostgreSQL with logical replication; on MongoDB with change streams.

Once the data is current in memory, a multi-field query or a join is a question asked of data that is already there. There is no stale query result to expire, because nothing is remembering results; the data itself is what is kept current. The application asks through GetRows with its filters, and the answer comes from Redis, not the database.

The cache stops being a list of remembered answers and becomes a current copy of the data itself.

What it changes for read-heavy screens

The practical effect is that the reads that used to be excluded from the cache are the ones that now benefit most. Consider what a typical list screen does.

It filters on two or three fields: a status, an owner, a date range.
It joins to one or two lookup tables for display names.
It pages through results, so it also needs a total count.
It is opened hundreds of times a day by every user in the department.

Every one of those steps can be served from memory. GetRows handles the filtered and joined read, and GetTotalRowCount handles the count for paging. The database sees none of that traffic, and is left with writes and the genuinely complex queries it is good at.

The shape of the data is our job, not yours

Serving joins from memory requires the data in Redis to be laid out so that it can be filtered and combined quickly, and updated so that a deleted row disappears from every join that touched it. None of that is trivial, and all of it has to survive schema changes.

That work is redfly's responsibility. The application does not describe cache keys, does not decide what to precompute, and does not maintain any of the structure in memory. It calls the API with the query it wants. When the schema changes, keeping the in-memory shape correct is part of the service, not a ticket in the customer's backlog.

The honest limits

Not every query belongs in memory. Heavy analytical work across millions of rows with complex grouping is still better suited to the database or a reporting store. The point is not that Redis replaces the query engine, but that the everyday reads that make up most of an application's traffic no longer need it.

There is also a short window after a write during which a read can return the previous state, because the sync service has to notice the change and apply it. For list screens and lookups that is almost always acceptable. For the rare read that must reflect its own write immediately, that is a design conversation to have early.

redfly sits between an application and its SQL Server, MongoDB or PostgreSQL database and serves reads, including joins and multi-field queries, from a Redis cache it keeps in step. Writes go straight to the database.

Cache invalidation with no invalidation code

2026-09-25T00:00:00Z

There is an old joke that there are only two hard problems in computer science: cache invalidation, naming things, and off-by-one errors. It survives because the first item is true. Deciding when a cached value is no longer trustworthy, and making sure something acts on that decision, is where a large share of production data bugs come from.

We think the joke points at the wrong culprit. Invalidation is hard when it is done by hand, from inside the application, at every place a write might happen. Move the job to something that watches the database instead, and most of the difficulty goes with it.

Where stale data bugs actually live

After an update, the code deletes or refreshes the cached copy of the thing it changed. That line is fine on its own; the problems are everything around it.

A second write path, such as a bulk import or a support script, does not include the line.
The update touches a row that appears in several cached lists, and only one of them is refreshed.
The invalidation call runs before the database has finished saving the change, so a read arriving in between refills the cache with the old value.
A deploy adds a new field to a cached object and the old cached copies are silently missing it.

None of these are exotic. They are what happens when the same rule has to be remembered in dozens of places by people who joined at different times. The cache is only as correct as the least careful write path.

The time-to-live compromise

The usual defense is a time-to-live, or TTL: a timer that throws a cached value away after a fixed period whether or not it changed. It limits how long a bug can persist. It also guarantees that the cache serves stale data for up to that period on every change, and that hot values are repeatedly thrown away and re-fetched from the database for no reason.

Teams tune TTLs downward when they get burned and upward when the database struggles. There is no right value, because the timer is standing in for information the cache does not have: whether the row actually changed.

Let the database say what changed

That information exists. Every database engine keeps track of its own writes, and each one exposes that in a way an outside service can read. redfly's sync service uses whichever mechanism the engine provides.

On SQL Server, it uses change tracking, a built-in feature that notes which rows changed, and reads it on a short interval. No triggers are added and nothing reads the transaction log.
On PostgreSQL, it uses logical replication, which streams each saved change out of the database as it happens.
On MongoDB, it uses change streams, which deliver each change as it is saved.

In each case the sync service applies the change to Redis, the in-memory store that serves reads. A changed row is updated, a deleted row is removed, a new row is added. There is no invalidation step, because there is nothing to invalidate; the cache is being kept current rather than being told when it is wrong.

What disappears from the codebase

Because the sync service watches the database, it sees every write regardless of where it came from. The bulk import, the support script and the application update all land in the same place and are all picked up the same way. The forgotten-write-path class of bug is not fixed; it is removed.

There are no invalidation calls after updates, no TTL settings to tune, no warming jobs on deploy. The application calls Get, Insert, Update, Delete, GetRows or GetTotalRowCount through the redfly API, and correctness of the cache is handled behind it.

The cache is correct because something keeps it correct, not because every engineer remembered to.

What a moment of staleness means

Between a write landing in the database and Redis reflecting it, there is a gap: short on the streamed engines, and bounded by the read interval on SQL Server. A read in that gap returns the previous value. So the question every team has to answer is whether their reads can tolerate being a moment behind.

For most reads the answer is yes. A user opening a list, a permission check on a request, a product page, a dashboard tile: none of these are harmed by seeing the state from a moment ago, and most of them were already seeing older data under a TTL scheme measured in minutes.

Where it is not acceptable

A read that must reflect a write made a moment earlier by the same user is the common exception. Showing a form the user just saved, confirming a balance immediately after a transfer, or enforcing a uniqueness rule are cases where a moment of staleness is visible or unsafe. Those reads need to be identified up front, and handling them is part of the design conversation before anything is deployed.

The trade, stated plainly

Hand-written invalidation offers the possibility of zero staleness for the writes you remembered, and unbounded staleness for the ones you forgot. A sync service offers a small, predictable staleness for every write, with no code to maintain. For most systems we have seen over two decades, the second is the better deal.

redfly sits between an application and its SQL Server, MongoDB or PostgreSQL database, serving reads from a Redis cache that its sync service keeps current by watching the database for changes. Writes go straight to the database, and the codebase carries no invalidation code.

Automatic database fallback: reads that survive a cache outage

2026-09-25T00:00:00Z

Most conversations about caching are about the good days. What is the hit rate, how fast is a read, how much load came off the database. Those numbers matter, but they describe the system when everything is working. The more revealing question is what the application does on the day Redis is not there.

Redis, the in-memory store that serves cached reads, is a separate piece of infrastructure with its own failure modes. It can be restarted, run out of memory, lose a network path, or be taken down for maintenance by someone who did not know what depended on it. If the application has been written to expect it, a missing cache becomes a missing application.

How a cache outage usually plays out

In a hand-written cache layer, the read path typically looks like this: check Redis, and on a miss, query the database and store the result. That handles a missing key. It does not necessarily handle a missing Redis.

When the cache client cannot connect, one of a few things happens. The call throws and the request fails outright. Or the client waits for a timeout on every read, pages that took milliseconds now take seconds, and soon every worker that handles requests is stuck waiting, so the whole service stalls. Or, in the well-prepared case, a developer wrote a fallback, but it was written two years ago, has never run in production, and nobody is sure it still works.

The pattern is familiar because caches are usually added for speed, and speed is what gets tested. Availability is assumed.

What automatic fallback means

With redfly, the application does not talk to Redis at all. It calls the redfly API, and the API decides where a read is served from. On a normal day that is Redis. If Redis is unreachable, the API serves the same read from the database instead.

Nothing in the application changes for this to happen. There is no degraded-mode switch to flip, no configuration setting to change, no code path the customer's engineers wrote and hope still works. All the application notices is that responses are slower, because a read from disk takes longer than a read from memory. Requests that would have succeeded still succeed.

Read availability should track the database, not the cache.

Why this matters more than hit rate

Cache hit rate, the share of reads answered from memory, is the number most teams optimize. It is a real measure of how much load the cache is absorbing. But a system with a very high hit rate and no fallback has traded a database problem for a Redis problem: it is fast until the cache is gone, and then it is down.

Fallback changes the shape of the risk. Instead of two components that must both be up, there is one component that must be up, the database, and a second that makes things faster when it is present. That is a much easier system to reason about and to operate at three in the morning.

It also makes the cache safer to touch. Upgrading Redis, resizing it, or moving it to another host becomes far less of an event, because the application keeps answering reads throughout.

Can the database carry the load?

The honest question that follows is whether the database can cope with all reads landing on it at once. The answer is not always yes.

The database was probably sized with the cache in mind. During fallback it sees the full read load. Whether it copes depends on a few things.

How long the outage lasts. A restart measured in seconds is very different from a lost host that takes an hour to replace.
When it happens. A cache outage during a quiet period is a non-event; the same outage during a peak is not.
How much headroom the database has. If the cache allowed the database to be sized smaller, some of that headroom has been spent.

Our view is that fallback should be treated as a short-lived condition, not a steady state. It exists so that a brief Redis interruption is a slowdown rather than an outage. Deliberately running on fallback for a long period, at peak, on a database that was shrunk to match the cache, is a decision to make with eyes open.

Thinking about it in practice

The useful exercise is to ask two questions before going live. First, what does the database look like under the full read load for a few minutes, and does it stay responsive. Second, how quickly can Redis be restored, and is that time shorter than the database can comfortably hold out. If both answers are reassuring, fallback does its job. If not, that is a sizing conversation, and it is better to have it before the outage than during it.

What it does not promise

Fallback is not a guarantee that nothing ever goes wrong. If the database itself is down, reads are down, because the database is the source of truth and the cache is not a substitute for it. Fallback does not make a slow database fast, and it does not make an under-sized database large.

What it does is remove one entire category of failure, the cache-down outage, from the list of things that can take an application offline.

redfly sits between an application and its SQL Server, MongoDB or PostgreSQL database, serving reads from a Redis cache it keeps in sync and passing writes straight through. If Redis is unreachable, reads go to the database automatically.

SQL Server change tracking: why we read changes on a short interval

2026-09-25T00:00:00Z

A cache is only useful if it tells the truth. redfly serves reads from Redis, a memory store, and keeps that copy in step with the customer's SQL Server database. To do that we need a reliable answer to one question, over and over: which rows changed since we last looked?

SQL Server gives two built-in answers. One is change tracking, a lightweight log of which rows changed and in what order. The other is a heavier feature that reads the transaction log (the database's own journal of every write) and reconstructs each change in full. We use change tracking, and we read it on a short interval.

What change tracking actually keeps

When change tracking is switched on for a table, SQL Server keeps a small side table of primary keys and a version number. Every saved (committed) insert, update or delete bumps a database-wide version counter and notes the key of the affected row against that version. It does not store the old values or the new values; it stores the fact that a row changed, and when, in version order.

That is deliberately minimal. The side table stays small because it holds one entry per changed row, not one entry per change, and SQL Server prunes it on a retention period the administrator sets. Because the bookkeeping happens inside the same transaction as the write, it is consistent with the data; there is no window where a row has changed but the tracking table does not yet know.

The reader's job is then simple. Remember the last version number you processed, ask SQL Server for every key that changed after it, fetch the current values of those rows, and save the new version. That is the whole loop.

Why we poll a version number instead of reading the log

The log-reading alternative is more powerful on paper. It gives before and after images of each row. It also needs more from the server: a log reader process that scans the transaction log, a scheduled job to run it, and extra tables holding the captured history. Those pieces have to be monitored, kept working after schema changes, and cleaned up on their own schedule.

For a cache, that power is wasted. Redis does not need to know what a row used to contain; it needs the row's current state, or to know that the row is gone. Change tracking answers precisely that question, and the current values come from an ordinary query against the table itself.

Reading a version number on an interval also has a pleasing operational shape:

No triggers on the customer's tables, so writes carry no extra code path.
No log reader and no agent job, so nothing extra runs on the database server between our reads.
A single number tells you where you are; recovery after a restart is "start from the last version you saved".
The load each read places on the server is bounded by how much changed, not by how much time passed.

That last point matters on a busy production server. The cost of one read is one lookup against a small side table plus a fetch of the changed rows. In a busy period it scales with the write rate, which is the fairest deal a database can offer.

What "a short interval" means for freshness

The honest consequence of reading on an interval is that Redis lags the database by a little. A write commits, the next read of change tracking picks it up, the changed rows are fetched and written to Redis, and only then does a read through the redfly API see the new value. On the streamed engines, PostgreSQL and MongoDB, the database pushes each change to us as it happens. On SQL Server we go and ask, so the lag is the interval plus the time to apply the change.

We describe that as a short lag, not instant. For the reads redfly is built to serve, permission checks, lookups, profile loads and the rest of the small constant traffic behind every click, that is well inside what applications already tolerate from read-only database copies (replicas) and caches. Writes still go straight to the database, so anything that must see its own write immediately can read it back from the source of truth.

A cache a moment behind, and kept correct, beats a cache that is instant and occasionally wrong.

Two things keep the lag from quietly growing. The Sync Service watches for drift (the copy quietly diverging from the source) between Redis and the database and repairs it, and it can backfill a table from scratch if the version it remembers has aged out of the retention window. Those are the parts that are easy to forget when a team builds this by hand.

What this asks of the database administrator

Change tracking has to be enabled on the database and on each table redfly will serve. It is a setting, not a schema change, and it does not alter the tables themselves. Retention should be set longer than the longest outage you expect the sync process to survive; a couple of days is a common choice.

Beyond that, the connection redfly uses needs permission to read change tracking and to read the tables it serves.

redfly fits here as the piece that runs this loop for you: reading change tracking on a short interval, keeping Redis in step, and handling backfill, drift and recovery so that your application code never contains a line of cache logic.

PostgreSQL logical replication as a cache feed

2026-09-25T00:00:00Z

PostgreSQL support in redfly comes after SQL Server, which is available now, and after MongoDB, which is next. We are saying that plainly up front so nobody reads this as a description of something they can switch on today. It describes how the engine reports changes and how we use that, because the approach is decided even though it is later in the queue.

Every write to PostgreSQL is first recorded in the write-ahead log, a sequential file the database appends to before it touches the tables. That log exists so the database can recover after a crash. Logical replication reuses it for a second purpose: it decodes the log into a stream of row-level changes, inserts, updates and deletes with the values of each row, and hands that stream to anyone who subscribes. redfly subscribes.

Publications: what the database agrees to share

A publication is a named list of tables, defined inside the database, whose changes are to be published. It can cover every table or a chosen few, and it can be limited to certain kinds of change, for example inserts and updates but not deletes. Creating one changes nothing about the tables; it is a declaration of what is available to subscribers.

For a cache feed the publication matches the set of tables redfly serves. Adding a table later means adding it to the publication and letting redfly backfill it, which is work someone does deliberately rather than a switch that flips itself. Tables outside the publication are simply never seen, which is a clean way to keep sensitive data out of the sync path entirely.

One detail matters for correctness. Each published table needs a way to identify a row, normally its primary key, so that an update or delete in the stream can be matched to the right row in Redis. A table without one needs extra setup before its updates and deletes can travel through the stream; we treat that as something to resolve before sync starts, not to work around while running.

Replication slots: the database's promise to keep the log

A replication slot is the other half. It is a named bookmark on the server that marks how far a particular subscriber has read in the write-ahead log. Its purpose is to guarantee that the database will not discard any log the subscriber has not yet consumed, so that a subscriber that disconnects, whether for a restart or a network fault, can reconnect and continue from where it left off without missing a change.

That guarantee is what a cache feed needs. When the sync process comes back after an interruption, it resumes from the slot's position and applies the changes it missed, in order. Nothing has to be rescanned and Redis does not have to be rebuilt. The database has, in effect, agreed to wait for its reader.

A slot is a promise, and promises cost disk.

The operational caution every administrator should know

The promise cuts both ways. A slot holds write-ahead log on disk until its subscriber has consumed it. If the subscriber stops reading and the slot stays in place, the log keeps accumulating. On a busy database that can fill the disk, and a PostgreSQL server that cannot write its log stops accepting writes. This is a common way logical replication hurts a production system, and it is entirely avoidable.

The rules of the road are straightforward:

Monitor the lag on every slot, measured as how much log it is holding back, and alert on growth.
Drop a slot the moment its subscriber is permanently retired; an orphaned slot never releases anything.
Set a ceiling on how much log a slot may retain, which recent PostgreSQL versions support, and accept that a subscriber which falls behind that ceiling will need a fresh backfill.
Run the subscriber as a supervised service, so a crash is restarted rather than discovered at the disk-full alarm.

Our design treats the slot as something we own and answer for. A slot that has been invalidated by the retention ceiling is a recovery case for the sync orchestrator, whose job is backfill, drift detection and recovery: the affected tables are rebuilt from the current data rather than by asking someone to intervene by hand. The failure mode still exists; the point is that it is caught early and recovered in a known way.

Streamed, not polled

Unlike SQL Server, where redfly reads change tracking on a short interval, PostgreSQL pushes each change to us as it is saved. There is no interval to tune and no question to ask; the stream arrives. The lag between a write and Redis reflecting it is the time to decode the change and apply it.

The stream carries the full row values for inserts and updates, and the key for deletes, so the stream itself holds what is needed to keep the copy in step. Schema changes are the case to plan for: they do not travel through the stream, so a column added to a published table is something the sync process must notice and handle, and it is one of the things the orchestrator's drift detection exists to catch.

redfly fits here as the subscriber that owns the publication, minds the slot, and keeps Redis in step from the stream, once PostgreSQL support arrives after SQL Server and MongoDB.

MongoDB change streams as a sync source

2026-09-25T00:00:00Z

MongoDB is the second engine redfly supports, after SQL Server, which is available now. This article describes how MongoDB reports changes and how redfly uses that mechanism, so a team running MongoDB can see what the setup involves before support lands.

MongoDB stores documents rather than rows, and a document is the unit that changes. From the cache's point of view the mechanics are the same as for a relational table: something was inserted, updated or deleted, and Redis has to reflect it. We say documents below where MongoDB does, and rows where we mean the general idea.

Where change streams come from

A MongoDB replica set (a group of servers holding copies of the same data, one of which, the primary, accepts writes) keeps an operations log, called the oplog, that every write is appended to. The other members read that log to stay in step with the primary. It is the same idea as the write-ahead log in PostgreSQL: a journal that exists for the database's own replication and recovery.

Change streams are a supported way for an application to read that journal without touching it directly. A client opens a change stream against a collection, a database, or the whole deployment, and the server delivers each insert, update, replace and delete as an event, in order, once a majority of the replica set has saved it. The event carries the document's key, the kind of change, and either the full document or the fields that changed.

Because change streams require a replica set, a standalone MongoDB server cannot produce one. In practice that is rarely a constraint; production MongoDB runs as a replica set for durability anyway, and a single-node replica set is enough for development.

How redfly consumes the stream

The Sync Service opens a change stream on each collection it has been told to serve and applies every event to Redis. An insert, replace or update brings the cached copy up to date, a delete removes it, and reads through the redfly API then see the new state on the next call.

Nothing is polled. The stream stays open and the server pushes events as they happen, so the lag between a write and Redis reflecting it is the time to receive and apply the event. That is the same shape as PostgreSQL logical replication and different from SQL Server, where redfly reads change tracking on a short interval.

For remote sync, the same stream feeds a different destination. A site running MongoDB has the sync service subscribe to its change stream, queue the events on site, compress and encrypt them, and send them on to the central store. The central store is itself MongoDB, so the documents arrive as documents, stamped with the site they came from.

Resume tokens: picking up after an interruption

Every event in a change stream carries a resume token, a value the reader keeps but never needs to interpret, which marks that event's position in the oplog. A consumer that saves the token after applying each event can reopen the stream later and ask to resume from that token. The server then replays every event that happened after it, in order, and the consumer catches up without having missed anything.

This is what makes a change stream a safe sync source rather than a convenient one. Network faults, restarts, deployments and a change of which server accepts writes (an election) all interrupt the stream. With the token saved, each of those is a pause rather than a loss. Without it, a consumer would have to rescan the collection to be sure it had not missed a change, which on a large collection is the very load a cache exists to remove.

Save the token after the write, never before it.

The order of operations matters. The token is saved only once the event has been applied to Redis or handed to the site queue. Saving it first and then failing to apply means a change is skipped for good. Applying first and then failing to save means the event is replayed on resume, and applying the same insert, update or delete twice leaves Redis in the same state, which is why delivery is at-least-once and reconciled, and why it is safe to be so.

The cautions

Resume tokens point into the oplog, and the oplog is a fixed-size log that discards its oldest entries as new ones arrive. If a consumer is offline for longer than the oplog retains, its token no longer points at anything and the resume fails. The fix is a backfill of the affected collection from current data, which is the sync orchestrator's job; the prevention is an oplog sized for the longest outage you expect, which is a setting the administrator controls.

Two smaller points. An update event can carry only the changed fields rather than the whole document, so a consumer that needs the full document either asks the server to include it in the event, which MongoDB supports, or fetches it afterwards. And change streams respect the same access controls as any other read, so the account redfly uses needs read access to the collections it serves and to the change stream itself, and nothing more.

redfly fits here as the consumer that opens the streams, saves the tokens in the right order, and keeps Redis and the central store in step, once MongoDB support lands next after SQL Server.

The database is the biggest line on your cloud bill: a breakdown

2026-09-25T00:00:00Z

Take a typical cloud bill for a read-heavy application (one that looks up data far more often than it changes it) and sort the lines by size. The application servers are usually near the top. Storage and network are further down. The managed database, the one the cloud vendor runs for you, is almost always the largest single line, and it is the one that has been growing fastest since the day the bill was first paid.

That is not an accident of any one vendor's pricing. It comes from how a managed database is bought, and from what a read-heavy application asks of it.

Where the money goes

Most cloud bills for this kind of application break into four rough buckets. The application servers, which run your code. The managed database, which holds your rows. Storage for files and backups, and the network, which is what you pay to move data out.

The application servers are elastic; you add them when traffic rises and drop them when it falls, and you pay by the hour. Storage is cheap per unit and grows slowly. Network charges track how much data leaves, which for most business applications is modest.

The database is different from all three. It is bought in fixed tiers, it runs all day and all night, and it is sized for the busiest hour of the month rather than the average one. Industry analysts have called databases the largest and fastest-growing slice of cloud spending, and that matches what we see on the bills we are shown.

Why the database line grows fastest

Three things push the database line up, and none of them slow down on their own.

Every new user adds reads, and reads are what the database spends most of its capacity on. Permission checks, profile loads, lookups, list pages; each click fires a handful of them.
The tier is a step, not a slope. When the current size runs hot, the next size costs roughly double, and there is nothing in between.
Nobody scales down. The upgrade that got you through one busy quarter becomes the new baseline, and the baseline only ever ratchets upward.

Application servers do not behave this way, because you can add and remove them one at a time. The database is one box, and its price moves in jumps.

What the 70% figure means

When we say redfly cuts the bill by about 70%, that figure refers to the database line of the bill, not the whole bill. It is worth being precise, because the two get confused. If the database is half of your cloud spend, a 70% cut to that line is roughly a 35% cut to the total. If it were three quarters, the total would fall by around half.

The mechanism is simple. Around 95% of what an application asks a database to do is reading. If those reads come from a memory cache instead, the database only needs to be sized for the remaining 5% (the writes) plus enough headroom to take reads back if the cache is ever unreachable. That is a much smaller box.

The database is not expensive because it is a database; it is expensive because it is sized for reads it should never have been serving.

What moving reads into memory changes

redfly sits between your application and your database. Reads are served from a memory cache (Redis, a store that keeps data in memory rather than on disk) that redfly keeps in step with the database as rows change. Writes go straight to the database as before. Your codebase carries no cache code and no rules about when to refresh it; that logic lives behind the redfly API.

On the bill, this does three things.

The database drops to a tier sized for writes plus fallback.
A memory cache line appears. It is smaller than the database line it replaces, because memory for the rows in active use costs less than a database engine sized to answer every read from disk.
The peak-day upgrade is no longer forced, because the peak is made of reads, and reads are no longer the database's problem.

What it does not touch

It is equally important to say what this leaves alone. Your application servers cost the same; they still run your code. Storage for files and backups is unchanged; the rows still live in the database. Network charges are unchanged, and if you pay for a database licence separately, that licence remains.

The change is confined to one line, the largest one. Everything else on the bill is what it was, which is also why the change is easy to verify: compare the database line before and after, and the rest of the bill is an unchanged baseline to compare against.

In our experience, the total bill falls in proportion to how much of it the database had been eating, and the peak-season upgrade stops being the reflex.

redfly is one way to make that change without rewriting the application: a cache kept in sync with your database, with the database still the source of truth and still taking every write.

Vertical scaling is a step-function bill

2026-09-25T00:00:00Z

Ask a cloud vendor for a slightly bigger database and you will find there is no such thing. A managed database is bought in tiers: a fixed ladder of sizes, each with its own amount of processing power, memory and connections, and each with its own price. When your current size runs out, the next size up is the next rung, and the next rung usually costs about twice as much.

That shape, a staircase rather than a slope, is the single most important thing to understand about database spending. It explains why the database line jumps rather than creeps, and why "just scale up" turns out to be such an expensive reflex.

The staircase and the curve

Draw your application's demand over a month as a line. It rises during the working day, falls overnight, spikes on a handful of busy days, and sits far below its peak for most of the hours in between. Now draw the database tier you are paying for as a flat line above it. The gap between the two lines is what you pay for and do not use.

With a staircase, that gap is never small. You cannot buy the tier that fits your average day, because the busy days would fall over. So you buy the tier that fits the busiest hour of the busiest day, and you pay for it every hour of the month.

One non-profit we spoke with had exactly one upgrade option on its database, and it doubled the monthly cost. There was no rung in between.

Headroom you use a few hours a month

Consider what the extra rung actually buys. If the previous tier coped with everything except a peak on a few afternoons, the new tier's additional capacity is in use for those afternoons and idle the rest of the time. Over a month of roughly 720 hours, headroom that matters for perhaps ten of them is being paid for in all 720.

Vendors are not hiding this; it is simply how a fixed-size engine has to be sold. A database cannot be grown one slice at a time the way application servers can; it is one machine, either the bigger one or the smaller one.

Application servers scale in units, so the cost tracks demand fairly closely.
The database scales in doublings, so the cost tracks the peak, not the demand.
The peak is made mostly of reads, which is the part of the load that does not have to hit the database at all.

Why "just scale up" is a habit, not a decision

Scaling up is the easiest button in cloud operations. It is one setting, it takes effect quickly, and it makes the alarms stop. When the database is hot at three in the afternoon and customers are complaining, nobody convenes a cost review; someone clicks the next tier.

The trouble is that the click is permanent in practice. Scaling down is a change nobody wants to own, because if it goes wrong it goes wrong in public. So every upgrade becomes the new floor, and the floor only ever rises. A company that has scaled up four times over three years is paying roughly sixteen times its original database cost, for a workload that has almost certainly not grown sixteen-fold.

The other route, horizontal scaling (spreading the load across copies of the database and a distributed cache), is a large engineering project with long development and testing cycles. Most teams look at it, estimate the quarters involved, and click the tier button instead.

A step-function bill punishes small growth with large jumps, and rewards nobody for scaling back down.

Serving reads from memory changes the shape

The way to stay on a smaller rung for longer is to take load off the database that never needed to be there. For a typical application the large majority of that load is reads: lookups, permission checks, profile loads, list pages. If those are served from a memory cache that is kept in step with the database, the database itself only has to handle writes and act as the fallback if the cache is unreachable.

That does two things to the staircase. First, the database drops one or more rungs, because the peak it must be sized for is now a peak of writes, which is far smaller. Second, growth in users no longer pushes it toward the next rung, because new users mostly add reads, and reads are absorbed in memory. We have handled peak shopping days with the database at 99 percent of its processing and memory capacity and kept serving.

The cache has a cost of its own, but memory for the rows in active use is a much smaller line than a database engine sized to answer everything from disk, and it scales in small units rather than doublings.

What to check on your own bill

Look at your database tier history for the last two years. Count the upgrades. For each one, ask what hour of what day forced it, and whether that hour was made of reads or writes. If the answer is reads, that rung was bought to serve traffic a cache could have carried.

redfly is built to make that move without a rewrite: a memory cache in front of the database, kept in sync by redfly, with writes going straight through and reads falling back to the database automatically if the cache is ever unavailable.

The 95/5 rule: what read-heavy really means for spend

2026-09-25T00:00:00Z

Every application that talks to a database does two kinds of work: it reads rows and it writes rows. Writes are what most people picture when they think of a database. A customer places an order, a user updates a profile, a sensor logs a reading. Reads are everything else, and everything else is most of it.

For a typical business application the split is somewhere around 95 reads to 5 writes. The exact ratio varies, but the shape does not. Even applications that consider themselves write-heavy tend to read each row many times for every time they change it.

Where the reads come from

Reads are not mostly the big reports. They are the small, constant requests behind every click: is this user allowed to see this page, what is their name and plan, what are the twenty items on this list, how many rows are there in total. A single screen can fire a dozen of them, and a single user can open hundreds of screens in a session.

Permission and session checks, on nearly every request.
Profile, settings and configuration loads.
List pages, search results and detail views.
Counts, totals and lookups that feed other requests.

None of these is expensive on its own; a fast database answers each one in a few milliseconds. The cost is in the volume. Millions of small reads a day add up to a database that is busy all the time, and a busy database is a database that needs the next tier.

What a database tier is actually sized for

When a team picks a database tier, it is choosing enough processing power, memory and connections to answer its busiest hour. That hour is dominated by reads, because reads are 95% of the traffic and because busy hours are busy with people looking at things, not changing them.

So the tier is, in effect, a read-serving budget. The writes ride along inside it and would fit comfortably in a much smaller box. This is the part that is easy to miss: the expensive engine is being bought to do the cheap kind of work, at volume.

A database tier is priced for the 95%, and most of the 95% never needed a database.

What changes when reads come from memory

Now suppose the reads are answered from a memory cache (Redis, which keeps data in memory rather than on disk) that is kept in step with the database as rows change. The 95% moves off the database. What remains for the database is the 5% of writes, plus one more thing that matters: the ability to take the reads back if the cache is ever unreachable.

That fallback is why the database does not shrink to nothing. It has to be sized for writes plus a realistic fallback share, not for writes alone. In practice that still lands well below where it was, which is where the figure of roughly 70% off the database line comes from. It is not 95% because the fallback headroom is real and we keep it.

The memory cache adds a line to the bill, but memory for the rows in active use costs less than a database engine sized for every read, and it grows in small steps rather than doublings.

What it means for peak days

The 95/5 split gets more extreme on the days that matter. On a peak shopping day, or the last day of a reporting period, the number of people looking at things climbs far faster than the number of things being changed. Browsing outpaces buying by a wide margin. The peak is a read peak.

That is why peak days are usually what force a database upgrade, and it is why moving reads into memory changes the peak-day conversation. When reads are absorbed by the cache, the database sees a write peak, which is a fraction of the read peak. We have handled peak shopping days with the database at 99% of its processing and memory capacity and kept serving.

What it means for the bill

Put the pieces together and the shape of the saving becomes clear.

The database tier drops to what writes plus fallback need.
Growth in users mostly adds reads, so it is the cache that has to grow, not the database tier.
The peak-season upgrade, the one that was bought for a read peak, is no longer forced by reads.
The cache line that appears in its place is smaller than the database line it replaced.

Nothing about the rows themselves changes. The database is still the source of truth, still takes every write, and still holds every row. It has simply stopped being the thing that answers every question.

redfly is one way to make that move without changing the application: reads served from a memory cache that redfly keeps in sync with your database, writes going straight through, and reads falling back to the database automatically if the cache is unavailable.

Running in your own cloud account: why there is no hosting markup

2026-09-25T00:00:00Z

Much of the software that runs alongside your database is sold with the hosting folded into the price. The vendor rents the machines, marks them up, and charges you a bundle. It is convenient, and it hides two costs: the markup itself, and the fact that your data is now sitting in somebody else's account.

redfly can be run three ways: inside your own cloud account, fully managed by redfly in redfly's cloud, or on your own servers onsite. We prefer the first, and this article is about why that choice matters to the person who signs the cloud bill.

Three ways to run it

In your own cloud account. redfly deploys into a subscription you already own. The memory cache, the sync service and the API run on your compute, next to your database. This is the preferred option.
Fully managed by redfly. We run it in our cloud and you point your application at it. Available for teams that would rather not operate any of it.
Onsite. The same software on your own hardware, for databases that are not in the cloud at all.

The subscription covers the software, support, improvements and the architecture review, with a minimum twelve-month term. It is not per seat and not per request. What changes between the three options is who pays for the machines, and to whom.

No resale of compute

When redfly runs in your account, we do not buy compute and sell it back to you. The cache and the services consume your cloud resources at your rates, and the charges show up on your own bill as ordinary line items. There is nothing in our price for hosting, because we are not hosting anything.

This is worth stating plainly because it changes how the saving is measured. During the architecture review we size the footprint redfly will need in your account, and the savings we quote are already net of it. The comparison is your database line before, against your database line plus the redfly footprint after, on the same bill, at the same rates.

There is no hosting markup in the price because there is no hosting in the price.

You keep your discounts

Cloud accounts of any size carry negotiated terms: committed-spend discounts, reserved capacity, enterprise agreements, credits. Every one of those was earned by your finance team, and none of them apply the moment a vendor hosts a workload for you, because the workload is now on the vendor's terms rather than yours.

Running in your own account keeps all of it. The memory cache is compute you already get at a discount. If you have committed spend to use up, redfly's footprint counts toward it. If your rates fall next year, the footprint gets cheaper with them.

Your data stays in your boundary

The second thing a hosted bundle hides is where the rows go. If a vendor runs the cache, a copy of your data lives in the vendor's account, behind the vendor's access controls, subject to the vendor's security review rather than yours.

In your own account, the rows never leave your boundary. The cache sits next to the database, inside the same network and the same identity controls your security team already manages. The sync service connects with the credentials you provide and touches only what those credentials permit. A security review of redfly becomes a review of software you run, not of a third party that holds your data.

No dependency on proprietary services

There is a quieter cost in cloud economics: the one you pay when you cannot leave. Software built on one vendor's proprietary services (the managed products that exist on that cloud and nowhere else) is cheap to adopt and expensive to move. The industry's standard advice to companies that lifted their systems into the cloud, to rebuild everything cloud-native, ends in exactly that position.

redfly does not depend on any cloud vendor's proprietary services. It runs on a database, a memory cache and ordinary compute, which every cloud provides. We prefer Azure, and SQL Server on Azure is where onboarding is fastest, but any cloud works, and so does a server room. The same deployment can move if your terms change, which is itself a negotiating position worth having.

What this means for the bill

Taken together, the shape is simple. One flat subscription for the software and the people behind it. A small footprint in your own account, at your own rates, sized in advance and already deducted from the savings we quote. No second bill from a hosting vendor, no rows outside your boundary, and no architecture that works on one cloud alone.

The database line, the largest on the bill, comes down because reads are served from memory. Everything around it stays where it was: your account, your discounts, your data, your choice of cloud.

redfly fits here as software you run rather than a service you rent hosting from: a cache kept in sync with your database, in your account, with the database still the source of truth.

Scaling a legacy .NET application without a rewrite

2026-09-25T00:00:00Z

Picture the application: ten years old, written in C#, talking to SQL Server through a data access layer that has grown a little every quarter. It works, and customers depend on it. It is also slow at nine in the morning, when everyone logs in at once and the database server sits at high CPU while the pages crawl.

The usual advice arrives in two flavours. Rewrite it, ideally as a set of services built around one cloud vendor; or buy a bigger database server and hope the next size holds for another year. The first is a multi-year project with no new revenue in the meantime. The second is a bill that jumps at every size step, with a hard ceiling.

Where the load actually comes from

Before choosing a fix, look at the traffic. In most business applications of this age, the vast majority of database calls are small reads: a permission check, a customer lookup, a dropdown list, the same profile row loaded on every screen. Each one takes a few milliseconds. Multiplied by every user and every click, they consume the database's capacity long before the reports and the heavy writes do.

Those reads rarely change the data they touch. A customer's address is read thousands of times between edits. If the reads come from memory instead of from disk, the database is left with the writes and the genuinely complex queries, a fraction of what it handles today.

What goes in front of the database

The approach is to put an API (a service the application calls over the network, instead of calling the database directly) between the application and SQL Server. Behind that API sits Redis, a memory store that answers in about a millisecond. The API serves reads from Redis and passes writes straight through to the database, which remains the source of truth.

The part that makes this workable is keeping Redis truthful. A sync service watches SQL Server using change tracking, a built-in feature that lists which rows changed, and reads that list on a short interval. Every insert, update and delete in the database is carried across to Redis.

The API exposes typed entry points: Get, Insert, Update, Delete, GetRows and GetTotalRowCount, over REST and gRPC (two standard ways for programs to call a service over a network). It also serves joins and multi-field queries from Redis, not only lookups of one row by its identifier, because a real screen rarely needs just that.

Moving screens over one at a time

Nothing about this requires one large cutover. The existing application keeps working while migration happens screen by screen, starting with the read-heavy pages that hurt most.

Pick one screen whose reads are frequent and whose data changes slowly, such as a customer summary or a product list.
Replace the data access calls on that screen with calls to the API. The rest of the application is untouched.
Ship it, watch the database load on that path drop, and confirm the numbers on the screen match.
Repeat with the next screen. Stop whenever the database is comfortable, or continue until every read path is served from memory.

Writes can move at the same pace or stay on the old path for a while; either way they land in SQL Server, and the sync carries them into Redis. Because the old and new paths read the same truth, a half-migrated application is not a risk. It is the normal state for months.

What stays and what changes

What stays: the database, its schema (the layout of its tables), its stored data, the C# language, the hosting, the deployment pipeline and the reports that already run. What changes: the screens you choose to move now call an API instead of composing SQL, and the read load on the database falls with each one.

What you do not add is cache code. There is no time-to-live setting in the codebase, no invalidation logic, no list of keys to clear after a write. A common reaction from engineers who inspect a finished migration is that they cannot find where the caching happens, because it is not in their code.

The database keeps its job as the source of truth; it simply stops answering questions it has already answered a thousand times.

How the fallback keeps risk low

The obvious question is what happens when Redis is down. The answer is that reads fall back to the database automatically. The API notices Redis is unreachable and answers from SQL Server, with nothing to switch over. Read availability tracks the database, exactly as it does today.

This is what makes the incremental path safe. The worst case for a migrated screen is that it behaves like the unmigrated version and reads from the database; the new path is a faster one that steps aside when it cannot help.

Two honest limits. On SQL Server the cache is updated on a short interval rather than streamed, so a row read a moment after it is written may briefly show the previous value; for the few screens where that matters, read those rows through the database. And the sync needs a working connection to both the database and Redis, so both deserve ordinary production monitoring.

Where redfly fits

redfly is this pattern packaged: the API, the sync service and the automatic fallback, deployed into your own cloud account, run by us as a managed service, or installed on your own servers, starting with SQL Server. We set it up with design partners against their existing tables, one screen at a time.

Multi-tenant freemium: keeping the free tier off the database

2026-09-25T00:00:00Z

Every freemium product ends up with the same shape of traffic. A small number of paying tenants generate the writes and the revenue. A much larger number of free users generate the reads: opening the app, loading a profile, scrolling a list, checking whether anything changed. Roughly ninety-five percent of the calls do not need the database at all, and most of those calls come from people who will never pay.

The shared database problem

Most multi-tenant applications (one codebase and one database serving many customer organisations, each called a tenant) keep every tenant in the same tables, with a tenant identifier on each row. It is the sensible design: one set of tables, one deployment, one backup. It also means every tenant shares the same processor, the same memory and the same queue for the disk on the database server.

That is where free users become expensive. A burst of free sign-ups after a marketing push does not stay in its lane. It queues in front of the paying tenants' queries, and a paying customer's page load slows down because someone on the free plan is scrolling. The database cannot tell which read matters, and the customer paying for a service level notices first.

The instinctive fixes are all costly. A bigger database server raises the ceiling and the bill in one step. Separate databases per tier double the operational work and still leave the free tier hitting a database. Capping how often free users can call the service protects paying tenants by making the free product worse.

Serve the free tier from memory

A different approach is to change where reads are answered. Put an API in front of the database, keep a copy of the frequently read data in Redis (a memory store that answers in about a millisecond), and serve reads from there. Writes still go straight to the database, which remains the source of truth. A sync service watches the database for changes and carries every insert, update and delete across to Redis, so the copy is kept in step without the application managing it.

The effect on the free tier is direct. A free user's session is almost entirely reads: their own profile, the content they are browsing, the shared reference data every tenant sees. All of it comes from memory. The database sees that user only when they write something, which for a free user is rare.

The paying tenant's experience changes for the opposite reason. Their reads also come from memory, and the database they share is now doing a fraction of its previous work, most of it writes and the complex queries it is actually good at. Headroom on the same hardware goes up, and the pressure to buy the next server size goes away.

The free tier is a marketing cost you can move from the database bill to the memory bill, where it is far cheaper.

Tenant isolation, honestly

A shared cache raises the same question as a shared database: can one tenant's data leak into another tenant's read? The tenant filter is the boundary. Every read carries the tenant it is for, and the application filters on it through the API's multi-field queries, exactly as its SQL did. Nothing about that boundary moved; only the store answering the query did.

Noisy neighbours are a separate matter. Redis is shared too, so a free tenant hammering reads still consumes memory throughput, but a memory read costs a tiny fraction of a disk read. The burst that once queued in front of paying tenants' database queries is now absorbed by a store built for exactly that.

The limits worth knowing

This pattern does not make writes cheaper. If free users generate heavy writes, such as uploading content or logging every action, the database still carries that load and needs to be sized for it. Memory-fronted reads help most when reads vastly outnumber writes, which describes the free tier of almost every marketplace, social app or travel product.

Freshness is bounded, not instant. On SQL Server the sync reads change tracking on a short interval; on PostgreSQL and MongoDB changes are streamed. A free user may see a value a moment older than the database holds, which is fine for a feed or a profile and not fine for a balance. Route the handful of reads that must be current through the database and leave the rest in memory.

And if Redis is unreachable, reads fall back to the database automatically. The application does not change, and the load goes back to where it was before, which is a state you have already survived.

What changes on the bill and in the codebase

The database line of the cloud bill falls because the database server no longer has to be sized for the free tier's read peaks. That line is usually the biggest one, and this is where a cut of around seventy percent on it comes from, instead of growth with every sign-up wave.

The codebase gets smaller, not larger. There is no cache code, no time-to-live setting and no invalidation logic to write, because the sync keeps Redis correct from the database's own change signals. Engineers call the API for the rows they need, and the tenant filter is the same one they always wrote.

Where redfly fits

redfly is that API and sync service, run against your existing SQL Server database in your own cloud account, with per-tenant read paths served from Redis and no cache code in your codebase. We put it in place with design partners against their existing tables.

Microservices without per-service data scaffolding

2026-09-25T00:00:00Z

Ask a team with a dozen services what it costs to add a thirteenth, and the answer is rarely about the business logic. It is about the scaffolding: a data access layer to write, a cache to stand up, invalidation rules to get right, a web interface to expose, page sizes and error formats to agree. Each is a solved problem, solved again, slightly differently, by every team, every time.

What a new service carries

Take a service that owns nothing new and simply reads customer and order data already in the shared database. To do that well, someone writes:

data access code: queries, mapping rows to objects, connection handling
a cache in front of those queries, because the read rate will not fit on the database
invalidation logic, so the cache stops serving stale rows after a write elsewhere
a web or gRPC interface with consistent paging, filtering and error formats
tests for all of the above, plus monitoring for the cache in particular

None of that is the feature. All of it is required before the feature can ship. And because each service does it separately, quality varies by who wrote it and when.

The invalidation piece is the one that bites. A write in the order service must somehow reach the cache in the reporting service, so teams add events, message queues and code that clears the cache after every write, which drift over time. When they drift, the symptom is a customer seeing yesterday's data, and the investigation crosses three teams.

One API in front of the data

The alternative is to stop building that layer per service and build it once, driven by the database schema (the layout of its tables). An API sits in front of the database and exposes typed entry points for every table: Get, Insert, Update, Delete, GetRows and GetTotalRowCount, over REST and gRPC (two standard ways for programs to call a service over a network), with the same paging, filtering and error formats everywhere. Every service reads through it.

Behind the API, reads are served from Redis (a memory store that answers in about a millisecond) and writes go straight to the database, which stays the source of truth. A sync service watches the database for changes using the mechanism suited to the engine, such as change tracking on SQL Server read on a short interval, and carries every change into Redis. Invalidation is no longer something any service implements; the cache is correct because the sync keeps it correct.

Joins and multi-field queries are served from Redis too, not only lookups of one row by its identifier. Most useful reads are "orders for this customer in this status", not "order by id"; without joins in the cache, teams fall back to the database for anything beyond a simple lookup and the cache stops earning its keep.

A new service should cost the effort of its own logic, not the effort of rebuilding the data layer around it.

What a team under twenty engineers gains

The gain is concentrated in small teams because they have no platform group to absorb the scaffolding. In a team of twelve, the engineer who writes the cache is also the engineer who ships the feature, and every quarter spent on plumbing is a quarter of product not shipped.

Adding a service becomes a matter of wiring its logic to entry points that already exist. No data access layer, no cache, no invalidation.
Every service behaves the same way at the data boundary, so an engineer moving between services does not relearn a custom data layer.
Cache correctness is one property of one system, not twelve properties of twelve systems written by twelve people.
The database does far less read work, so the shared data layer stops being the reason services are slow.

A cache kept correct by the sync rather than by careful code review does not need a specialist watching it, and a data layer serving reads from memory does not need a database administrator tuning it every month. Generalists ship the work.

Where the pattern does not fit

It does not fit a service whose data is genuinely its own and never shared; keep that store private to the service.

It does not fit workloads dominated by writes. The API passes writes straight through to the database, so a service that takes in a constant flood of device readings gains little on the read side and still needs a database sized for its writes.

It does not fit reads that must reflect the very latest write. On SQL Server the cache is updated on a short interval, so a read may briefly return the previous value; a service that cannot tolerate that should read those few rows through the database. It does not remove the need for service boundaries and ownership; it removes the plumbing inside each one, not the design work between them.

The reasonable objection is that one API for every service becomes one thing to break. If Redis is unreachable, reads fall back to the database automatically, so the cache is a faster path that steps aside rather than a new point of failure; and the database was already the shared dependency underneath twelve separate caches.

Where redfly fits

redfly is the API and the sync service described here, run against your SQL Server database in your own cloud account, so a team under twenty engineers gets one data layer instead of building one per service. We put it in place with design partners against their existing tables.

Ten billion rows over the public internet: what we learned

2026-09-25T00:00:00Z

The number people ask about is the ten billion. The part that matters more is the link. Every one of those rows travelled over the same public internet that carries video calls and email: shared, congested at odd hours, dropping for a minute here and an afternoon there. Nothing leased, nothing dedicated. This is what that taught us.

Send only what changed

The first lesson is the oldest one. A full copy of a table, sent every night, is the wrong shape for a shared link. It is large, it repeats work already done, and it fails halfway through more often than you would expect.

So after one initial sync, only the rows that were added, changed or deleted leave the site. The sync service asks the database what moved since it last looked. On SQL Server that means change tracking, a built-in feature that notes which rows changed, read on a short interval; on PostgreSQL it is logical replication, streamed; on MongoDB it is change streams, streamed. Traffic falls to a trickle that follows the real rate of work.

Compress, then encrypt, per site

Changes are queued on site, compressed, then encrypted before they go anywhere. The order matters: compression works by finding repeated patterns, and encrypted data has none, so compressing after encrypting achieves nothing.

Encryption is per site, with a separate key for each. That sounds like a small detail until one site has an incident and you need to be certain the others are untouched; a key for one site opens nothing from another, which is the property an auditor asks about first.

Assume the link is down

We stopped thinking of an outage as an exception early on. A link that is up most of the time is still down for hours across a month, and a warehouse does not pause because the internet did.

When the link drops, changes wait on site in a queue and go when it comes back. That queue is a holding area, not a read copy; applications at the site cannot read from it while cut off, and we do not pretend otherwise. What it gives you is the promise that nothing is lost.

An outage is not a failure of the design; it is one of the inputs.

Resume per table, not per site

When something restarts, the naive approach is to start the whole site over. On a site with hundreds of tables and one very large one, that is a disaster: you reload the small tables endlessly while waiting for the big one to finish.

Each table remembers its own position instead. After a restart, or after the link returns, every table picks up exactly where it stopped. A table that finished stays finished. The big table carries on from the last position the central store confirmed rather than from the beginning.

Reconcile on arrival

Delivery is at-least-once. Over a bad link, a change can be sent, arrive, and have its acknowledgement (the reply confirming it arrived) lost on the way back; the sender then sends it again. Promising that every change arrives one time and one time only is a promise you cannot keep over the public internet, and we do not make it.

Instead, the central store reconciles each change on arrival. Every row is stamped with the site it came from, and each site keeps its own space in the store, so a repeated change is recognised and applied once, and an edit or a deletion is carried through rather than dropped. The copy converges on the truth even when the transport was messy.

Hold back when the source is busy

The site database has a day job. It runs the warehouse, the depot, the plant. Sync is a guest, and a guest that hogs the machine during the afternoon peak will be switched off by whoever runs the site, and rightly so.

So the service holds back when the machines at the site are working hard, and catches up when they are not. Reports never run against the site database; they run against the central copy. That is the real benefit for the site: it sheds its reporting load and keeps its headroom for the peaks that matter.

What surprised us

Three things. First, the deletions. Everyone plans for new rows; edits and deletions are where copies drift apart quietly, and carrying them through correctly was more of the work than moving new rows ever was.

Second, silence. The hardest failure was never a loud one. It was a site that quietly stopped sending, and nobody noticed for a while. Detecting a silent site became as important as moving its data.

Third, how little the pipe mattered once the design was right. Sending only changes, compressed, meant the ordinary public internet was more than enough. The dedicated line we might once have asked for would have been paid for and barely used.

Where it landed

End to end, a change lands in the central store within roughly thirty seconds to two minutes. That is not instant, and we say so. For reporting across many sites, current to within a couple of minutes is what the business actually needed, and the store it lands in is low-cost document storage rather than a cloud relational database billed by the hour.

This is the machinery behind redfly Remote Sync: the same sync service that keeps Redis in step beside a database, pointed outward over the internet at a central store instead.

IoT and fleet telemetry: high writes, flaky links, millisecond reads

2026-09-25T00:00:00Z

A fleet is a flood of data on a bad connection. Every vehicle, sensor and robot reports position, temperature, battery, load and fault codes, over and over, all day; that stream of readings is telemetry. Between reports the link drops: a truck goes under a bridge, a depot loses its internet for an afternoon, a vessel spends a week out of range. Meanwhile somebody in an operations room is looking at a screen and expects it to show what is happening now.

Two jobs that pull in opposite directions

The write side wants durability. Every report has to land somewhere safe, in order, and stay there, because a missed fault code is a missed inspection. The read side wants speed. A dispatcher refreshing a map of four hundred vehicles does not care about the history; they care about the latest known state of each one, and they want it in milliseconds, not seconds.

Asking one database to do both is where most telemetry systems run into trouble. The writes are relentless, and every dashboard refresh lands on the same disk, competing with them. The usual answer is a bigger database, then a hand-built cache in front of it, then a team to keep that cache honest.

Writes go to the database

We keep the database as the place writes land, and only that. The ingestion code, whether it calls the redfly API or writes straight to the database, ends up putting rows in SQL Server, MongoDB or Postgres. That is the source of truth, and nothing else pretends to be.

This matters because telemetry is not just appends. The row for a vehicle gets updated with its latest position; a job row changes status from assigned to in progress to done; a decommissioned sensor is deleted. Edits and deletions have to carry through to wherever the reads come from, or the operations screen starts telling quiet lies.

Reads come from memory

Reads are served from Redis, a store that keeps data in memory and answers in about a millisecond, which the redfly sync service keeps in step with the database. The mechanism depends on the engine: on SQL Server the service reads change tracking on a short interval; on PostgreSQL it follows logical replication, streamed; on MongoDB it follows change streams, streamed. When a row changes in the database, the change reaches Redis without anyone writing invalidation code.

The application asks for data through one API, with entry points such as Get, GetRows and GetTotalRowCount, over REST or gRPC (two standard ways for programs to call a service over a network). The map screen asks for the latest state of every vehicle in a region and gets it back from memory in milliseconds. Joins and multi-field lookups are answered from Redis too, so a query like all vehicles in this depot with an open fault does not fall back to the database just because it touches two tables.

Writes: straight to the database, unchanged
Reads: from Redis, kept current by the sync service
If Redis is unreachable, reads fall back to the database automatically
No cache code, no time-to-live settings, no invalidation logic in the codebase

When the link is the problem

A vehicle that is offline does not write, so the last row it wrote is the truth until it reconnects. The operations screen should show that last known state with its timestamp, plainly, rather than hiding it or guessing. When the device comes back and its backlog lands in the database, the sync service carries those rows to Redis and the screen catches up.

Depots and remote sites are the harder case. A site often runs its own database on its own network, because the operation cannot stop when the internet does. redfly Remote Sync runs a small service inside that network, watches the tables you choose, and sends only the rows that were added, changed or deleted outward to a central store. Changes are queued on site, compressed and encrypted, and if the link drops they wait and resume from where each table left off when it returns. Nothing is lost during the offline period; it just arrives later.

The device that has been silent for an hour is not a bug in the dashboard; it is a fact the dashboard should show.

What we would not promise

We would not promise reads faster than a millisecond, and we are wary of anyone who does. A read from Redis is fast, in the low milliseconds for a typical lookup, and that is what a dispatcher notices as instant. Above that, network hops between the application and Redis matter more than the store does.

We would also not promise that the central copy is current to the second. Remote sync over the public internet lands changes in roughly thirty seconds to two minutes end to end. For fleet reporting across sites that is plenty; for a control loop that steers a machine, it is not, and that loop belongs on the device.

Where it has run

Software our team built has been used by more than one hundred enterprise customers, and the remote sync design has carried more than ten billion rows over ordinary public internet links in production, with offline periods and high latency as everyday conditions rather than surprises. Telemetry is the same shape at a smaller scale: constant writes, unreliable links, and a screen that has to be right.

redfly fits where the database is taking the writes and the dashboard is slow, or where sites and vehicles cannot stay connected and the central view has to be complete anyway.

Drift, backfill and recovery: the unglamorous parts of sync

2026-09-25T00:00:00Z

Moving a change from one place to another is the easy part of sync. Anyone can write the first version in a week, and it will look fine in a demo. The work that fills the next two decades is everything that happens when the copy and the source stop agreeing and nobody has noticed yet.

What drift is

Drift is two copies of the same data that quietly disagree. The source database says an order has forty lines; the copy that reports read from says thirty-eight. Nothing crashed, no alert fired, and the report has been wrong for three weeks.

Drift does not announce itself. It is discovered by a person who happens to compare two numbers, usually at the worst moment, and by then the question is not how to fix it but how long it has been wrong.

How drift happens

A change is missed: the sync service was down, or busy, when a row changed, and the signal that would have told it about that row has since expired.
A change is applied twice or out of order: a retry over a bad link replays an old update on top of a newer one.
A deletion is dropped: many systems only carry new rows and edits, so rows removed at the source live on forever in the copy.
A restart loses its place: the service comes back up and either starts from the beginning or, worse, from an approximate point it guessed.
The schema (the layout of the tables) changes: a column is added, renamed or given a new type at the source, and the copy keeps writing the old shape without complaint.

Every one of them is a normal Tuesday in a system with more than one site and more than one table.

Detecting it

The first job of a sync service is to notice. That starts with knowing where it is: each table keeps its own position, a marker for the last change it is certain the destination has. The mechanism that provides the marker depends on the engine. On SQL Server it is change tracking, a built-in feature that notes which rows changed, read on a short interval; on PostgreSQL it is logical replication, streamed; on MongoDB it is change streams, streamed.

Beyond position, the service has to check the copy against the source rather than trust that following changes was enough. A copy that was never compared with the truth is a copy you are hoping is right. Comparing is dull, and it is the part that turns a hope into a fact.

A sync that has never been checked against its source is a rumour with a schedule.

Repairing it

Three repairs cover almost everything, and the trick is picking the cheapest one that works.

Resume from a known position. After a restart or an outage, each table picks up exactly where it stopped. This is the everyday case and it should cost nothing beyond the changes that piled up while the service was away.
Reconcile on arrival. Every change is applied against what is already there, not appended blindly. A change that arrives twice is recognised and applied once; an edit replaces the row; a deletion removes it. Every row is stamped with the site it came from, so two sites cannot overwrite each other.
Backfill the table. When the position is lost or the schema has changed under the copy, the honest move is to reload that table in full and then resume following changes from the point the reload began. This is the expensive repair, so it is done per table rather than for the whole site.

That last point matters more than it sounds. A site with hundreds of small tables and one enormous one should never reload the small ones because the big one had a problem. Position, reconciliation and backfill are all per table, so a repair stays as small as the fault.

Why at-least-once is the honest design

Every sync service has to pick a delivery promise. Over the public internet, with links that drop and acknowledgements (the replies confirming arrival) that get lost on the way back, a change will sometimes be sent twice. You can design the sender to never repeat itself, but then a lost acknowledgement means a lost change, which is worse.

So we send at least once and reconcile on arrival. Duplicates are expected and harmless because the destination knows how to apply the same change twice and end up with one result. It is a less impressive promise, and it is the one that holds up on a bad day.

Schema changes

Schemas move. A column gets added for a new regulation; a field gets renamed because the old name was embarrassing. A sync service that silently keeps writing the old shape is a drift machine.

In practice this means schema changes are part of running the service, not an incident. The copy is updated to the new shape, the affected table is backfilled if the change cannot be followed forward, and then ordinary following resumes. It is work someone does, and a person drives it; it is not a switch that flips itself.

Where redfly fits

The redfly sync service keeps Redis in step with a database beside it, and keeps a central store in step with sites over the public internet, using the same position tracking, reconciliation and per-table backfill described here. Drift, backfill and recovery are not features we added later; they are the problems our team has spent two decades on.

Surviving Black Friday with the database at 99% CPU

2026-09-25T00:00:00Z

Every online retailer has a chart they look at once a year with their breath held. It is the CPU (processor) load line on the database server, and on the biggest shopping weekend of the year it climbs towards the top of the graph and stays there. We have been on the operations side of that weekend with the database running at 99% of its CPU and memory, and the site stayed up. This article is about why that was possible and what it takes.

What the database is actually doing at peak

A shopper who buys one item generates a single order, which is one write to the database. Before that shopper reaches checkout, they have loaded a dozen product pages, checked stock on three of them, looked at their basket twice and opened their account to confirm a delivery address. Every one of those is a read.

On a normal day the ratio is lopsided; on peak day it is extreme. The number of orders goes up several times over, but the number of reads goes up far more, because the shoppers who never buy still browse. The database spends most of its capacity answering the same questions over and over: what does this product cost, is it in stock, who is this customer.

What breaks first

The first thing to fail is not the order. It is the product page. When the database is saturated, every request waits its turn in a queue, and the pages that need the most reads to render are the ones that time out first. Product listings, stock checks and account pages go slow, then go blank.

The order path usually survives longest, because it is a small number of writes. But by then it does not matter. A shopper who cannot load the product page never reaches checkout, so the site is effectively down even though orders would technically still go through. The retailer loses revenue on the one day it planned for all year.

The database does not fall over from orders; it falls over from the browsing that happens before them.

Why buying a bigger database for one weekend is the expensive answer

The reflex is to move the database up a size for the peak. Cloud databases are sold in fixed sizes, and the next size up is roughly double the price, not ten percent more; the higher price applies to every hour it runs. Scaling down again afterwards is a second change, with its own risk, and many teams quietly never do it.

There are further problems. The larger size has to be set up and tested before the weekend, which means a rehearsal weeks ahead. The biggest size still has a ceiling, and a hot product launch can blow through it. A team that scales up every peak ends up paying for the largest size several weekends a year, then wondering why the database line on the cloud bill keeps growing.

Serving the reads from memory instead

The alternative is to stop asking the database the same questions millions of times. redfly sits between the application and the database. Reads are answered from a memory cache (Redis, a store that keeps data in memory rather than on disk, so it answers in about a millisecond). Writes, which for a retailer means orders, go straight to the database as before.

The part that makes this safe is keeping the memory copy correct. A sync service watches the database for changes; on SQL Server it reads the built-in change tracking on a short interval, and on PostgreSQL and MongoDB it streams changes as they happen. When a price changes or stock moves, the memory copy is updated shortly after. Nobody on the retailer's team writes cache code, sets expiry times or remembers to clear anything after a write.

What peak day looks like after the change

The product page, the stock check and the account page are now answered from memory. The database sees the orders and the updates that come from them, and very little else. Its CPU line still rises on peak day, but the rise is driven by the writes that actually earn revenue, and that is a far smaller number than the reads were.

If the memory cache ever becomes unreachable, reads fall back to the database automatically. The site does not switch into a degraded mode; it simply gets slower until the cache is back. That is the same position the retailer was in before, with nothing extra to lose.

The honest limits

This does not make the database infinitely large. Orders still have to be written, and a retailer whose writes alone saturate the database needs a different conversation. The memory copy lags the database by a little, so a stock count shown to a shopper can be a moment old; the order itself is still checked against the database when it is placed, which is where that check belongs.

It also takes preparation. The tables to serve from memory have to be chosen, the memory store has to be sized for them, and the application has to be pointed at the new read path. That is real work done before the first peak, not a switch flipped the night before.

Where redfly fits

redfly supplies the read path and the sync that keeps it correct, working with retailers as design partners against their own SQL Server, MongoDB or PostgreSQL database. The retailer keeps its store, its checkout and its database; the database just carries a lot less of the weekend.

Checkout-path caching: catalog, inventory deltas and customer state

2026-09-25T00:00:00Z

An online store's checkout is a sequence of reads that ends in one write. Along the way the application asks for the product catalog, the current stock position and the state of the customer's own session. Each of the three changes at a different rate, and treating them as one problem is how most caches end up either stale or useless.

Three kinds of data, three rates of change

The catalog is the slow one. Product names, descriptions, images and list prices change when a merchandiser edits them, which for most rows is rarely and for the whole catalog is a steady trickle. It is also the most read: every product page, search result and basket line pulls from it.

Inventory is the fast one, and it changes in small deltas (a delta is the difference between the old value and the new one, here a stock count going down by one or up by a pallet). Every order, every return, every warehouse receipt moves a number. The absolute count matters less than the direction; the question the page asks is whether the item can still be bought.

Customer state is per session. The basket, the chosen delivery address, the saved payment method and the loyalty balance belong to one person and change while that person is active. It is read many times in one session and then goes quiet for weeks.

How a synced cache treats each

redfly keeps a copy of the tables the application reads in Redis (a memory store that answers in about a millisecond) and serves reads from there. A sync service watches the database for changes: on SQL Server it reads change tracking on a short interval, and on PostgreSQL and MongoDB it streams changes through logical replication and change streams. Changed rows land in Redis shortly after. The application calls Get, GetRows or GetTotalRowCount and never sees which store answered.

For the catalog, that means the product page is a memory read, and a price edit shows up shortly after the merchandiser saves it. There is no expiry timer to tune, because the copy is refreshed by the change, not by the clock.

For inventory, each stock movement is a row change like any other. The sync service carries the new value to Redis, and the product page shows it on the next read. The cache never tries to apply deltas itself; the database is where the arithmetic happens, and the cache reflects the result.

For customer state, the basket row and the address row are synced the same way. A shopper who adds an item writes to the database; the change reaches Redis a moment later; the next page load reads it from memory.

The cache does not decide what is fresh; the database's own list of changes does.

Writes go straight to the database

Placing the order is the one write on the path, and it does not touch the cache on the way in. The application calls Insert, the row lands in the database, and the same change signal that carries every other row carries the order out to Redis afterwards. The order is committed with the database's usual guarantees, and the stock reduction that goes with it is a database transaction (a set of changes applied together or not at all), not a cache operation.

This is the part that makes the design safe for money. Nothing that decides whether a sale happened depends on memory. Memory only decides how quickly the shopper sees the pages around the sale.

The honest limits

The copy in Redis lags the database by a little, so a stock count on a product page can be a moment old. For a page that is fine. For the moment of purchase it is not, and the application should treat the final stock check as part of the order write, which is what the Insert already is. A store that shows three left and sells four in the same second has a problem no cache solves; the database transaction on the order is what refuses the fourth.

Session data that never reaches the database, such as a basket held only in browser storage, is outside the sync and stays outside it. If Redis is unreachable, reads fall back to the database automatically; the checkout keeps working, more slowly, until the cache is back.

The tables to cache also have to be chosen and the memory sized for them. A catalog of a million rows with images stored as text is a different sizing conversation from a catalog of ten thousand rows with image links.

What this removes from the codebase

No cache keys, no expiry settings, no invalidation calls after writes, no logic that decides which of three stores holds the truth. The application reads through one API and writes through the same API, over REST or gRPC (two standard ways for programs to call a service over a network). The freshness rules for catalog, inventory and customer state are all the same rule, because the sync treats every row alike.

That also removes a class of bug. The hand-built version of this design usually has three caches with three expiry policies, and the failure is a basket that shows an old price next to a stock count that is newer than the order. One sync, one lag, one rule.

Where redfly fits

redfly is the API and the sync service in this picture, deployed in the retailer's own cloud account or on their own servers, working with them as a design partner. The store, the checkout and the database stay theirs.

Making dashboards feel instant without a bigger database

2026-09-25T00:00:00Z

An operations dashboard is a screen that asks the same questions all day. How many orders are open, which shipments are late, what is the stock position at each site. Twenty people have it open, it refreshes every thirty seconds, and every refresh runs the same queries against the same database that is also trying to run the business.

Why dashboards are hard on a database

A dashboard is read-heavy and repetitive. It does not create data; it summarises data other systems created. But because the summaries are recomputed on every refresh, the database does the same expensive work over and over, for each viewer, whether or not anything changed since the last time.

That work competes with the operational load. The warehouse system inserting a shipment and the dashboard counting shipments are on the same server, and at peak both get slower. The usual outcome is a dashboard that takes eight seconds to load at exactly the moment a manager most wants to look at it.

The reflex fixes and their costs

The first reflex is a bigger database. That is a step up in monthly cost for every hour of the year, to serve a screen that needs it for a few hours a week. The second reflex is to refresh less often, which makes the dashboard less useful. The third is a separate reporting copy of the database, which doubles the licence and adds a job that someone has to keep running.

Each fix trades away something the business cares about, and none of them addresses the actual waste: the same read, repeated.

A dashboard does not need a faster database; it needs to stop asking the database.

Serving the reads from memory

redfly sits between the dashboard's data layer (the code that fetches its numbers) and the database. The rows the dashboard reads are kept in a memory store (Redis, which holds data in memory rather than on disk and answers in about a millisecond). A sync service watches the database for changes and carries them to memory shortly after they happen; on SQL Server it reads change tracking on a short interval, and on PostgreSQL and MongoDB it streams the changes as they happen. Pre-computed totals and counts can live there too, so the expensive part of the query is done once, not once per refresh.

The dashboard's refresh now reads from memory. The database sees the writes that changed the underlying data, and only those. Twenty viewers refreshing every thirty seconds cost the database nothing extra, because the memory copy is answering all twenty.

What the customer still builds

redfly does not build the dashboard. The customer builds it with the reporting tool or the web framework they already use, and we support that with advice and development tools. What redfly supplies is the data path: the memory copy, the sync that keeps it correct, and an API the dashboard reads through.

This matters because the dashboard is where the business logic lives. Which shipments count as late, how a region is defined, which manager sees which sites; those are the customer's decisions and they belong in the customer's code. The data path underneath is generic, and that is the part worth not building twice.

What instant honestly means

The screen is fast because its reads come from memory; page loads under two hundred milliseconds on the same database hardware is the outcome this is built for. The numbers on it sit a short lag behind the database, not exact to the instant. For an operations dashboard that is the right trade. A manager looking at late shipments does not need the count to be exact to the millisecond; they need it now and they need it not to freeze.

If the memory store is ever unreachable, reads fall back to the database automatically. The dashboard slows to its old speed until the cache is back; it does not go blank.

What it takes to set up

The tables the dashboard reads have to be chosen, and any totals worth pre-computing have to be identified. The memory store has to be sized for those tables. The dashboard's data layer is then pointed at the redfly API instead of at the database directly. That is a piece of work with an engineer on each side, done once, not a switch someone flips.

Where redfly fits

redfly supplies the read path and the sync for the customer's own SQL Server, MongoDB or PostgreSQL database, running in the customer's cloud account or on their own servers, as a design partnership. The dashboard stays the customer's; the database stops carrying it.

Pre-computed results: when to cache aggregates

2026-09-25T00:00:00Z

An aggregate is a number computed from many rows: the count of open orders, the total value of stock at a site, the average delivery time this month. The database has to read every contributing row to produce it, and the result is a single value that fits in a few bytes. That asymmetry, expensive to compute and cheap to store, is the whole argument for caching aggregates. The question is when the argument holds.

The cost shape of an aggregate

A lookup of one row by its identifier costs about the same whether the table has a thousand rows or a hundred million; the index (the database's own lookup table pointing straight to the row) does the work. An aggregate scales with the number of rows it covers. A count over a table of fifty million rows means the database reads every one of them, or at best every entry in an index, and it takes as long every time it runs.

Dashboards, list pages that show a page of twenty out of fifty thousand, and reports all run aggregates, and they run them repeatedly. The database recomputes the same total for each viewer and each refresh. Storing the result once and reading it from memory turns that full read into a single lookup.

When to keep an aggregate in memory

The test is the ratio of reads to changes, weighed against how much the reader cares about the last few seconds.

The aggregate is read far more often than the rows underneath it change. A total row count on a catalog table that changes a few times an hour is read on every list page; keep it.
The computation is heavy. A sum across millions of rows with a filter and a grouping is worth caching even if it is only read every minute.
The reader tolerates a value that is a moment old. Almost every dashboard, list page and report does.

redfly exposes GetTotalRowCount alongside Get and GetRows, and it is a good example of the first case. The count is kept in Redis (a memory store that answers in about a millisecond) and served from there, so a paged list does not run a count query per page.

When to compute on demand

The aggregate depends on the caller. A total filtered by an arbitrary date range, a free-text search or a per-user permission set has too many distinct answers to store; each combination would be its own entry, and most would be read once.
The rows underneath change more often than the aggregate is read. A running total that moves on every write and is read once an hour is cheaper to compute when asked.
The reader needs the value as of this transaction. A balance check before a debit, or a stock check at the moment of sale, belongs in the database transaction (the set of changes applied together or not at all), not in a cache.

Cache the answers that many people ask for; compute the answers only one person will ever ask.

How a short lag changes the trade-off

The traditional way to cache an aggregate is a timer: recompute every five minutes and serve the stored value in between. That forces a choice between a stale number and a busy database, and the number is oldest exactly when the reader looks.

A sync service watching the database changes that. On SQL Server it reads change tracking on a short interval; on PostgreSQL and MongoDB it streams changes through logical replication and change streams. Changed rows reach Redis shortly after, and the pre-computed results that depend on them are refreshed because the data changed, not because a clock ran out. The stored total is a short lag behind the database, with no timer to tune and no recompute on a schedule.

That moves the line. Aggregates that changed too often to cache on a five-minute timer become reasonable to cache when the lag is a moment rather than minutes. The cases left for on-demand computation are the per-caller ones and the transactional ones, and those were never good cache candidates anyway.

Pitfalls

The first is caching an aggregate whose inputs are not all in the synced tables. If a total depends on a table the sync service is not watching, it will drift and nobody will be told. Every input table has to be in the sync.

The second is treating a cached count as a promise the database enforces. A page that says fifty items remain is a page, not a promise; the order insert is where the database enforces the truth.

The third is the fallback path. If Redis is unreachable, reads fall back to the database automatically, and the aggregate that was a millisecond lookup becomes a full read of the table again. The database has to be able to survive that, which means the cache should reduce its load, not be the sole reason it is still standing.

The fourth is the shape of the aggregate. A pre-computed result that groups by every dimension a reader might want is a table in itself. Keep the stored aggregates to the ones actually rendered, and let the rare combinations run on demand.

Where redfly fits

redfly keeps the rows and the row counts the application reads in Redis, kept in step with the customer's own SQL Server, MongoDB or PostgreSQL database by a sync service matched to the engine, and serves them through one API with fallback to the database built in.

Millisecond authorization: caching permissions safely

2026-09-25T00:00:00Z

Every request an application handles begins with the same question: is this caller allowed to do this? The answer lives in a handful of tables: users, roles, the mapping between them, and the permissions (sometimes called scopes) each role carries. Checking those tables against the database on every call is the most repeated read in the system, and on a busy service it is the read that sets the floor for latency.

Why the database is the wrong place for the constant check

A permission check is small, but it is on the path of everything. A page that makes ten calls to the backend makes ten permission checks, and each one is a round trip to the database before any useful work starts. At a few milliseconds each that is the difference between a fast page and a sluggish one, and the p99 (the latency the slowest one percent of requests see) is the number that suffers most, because those are the requests that queue behind a busy database.

The load side is worse. Permission tables change rarely, so the database is answering the same question millions of times a day with the same answer. That is capacity spent on nothing.

Why the hand-built cache is also wrong

Most teams know this and build an in-memory permission cache inside the application. It works until an administrator revokes a role and the user keeps their access for as long as the cache entry lives. Expiry timers are the usual fix, and they force a choice: a short timer means the database is hit often anyway, and a long timer means a revoked permission lives on for minutes.

There is also the many-servers problem. A cache inside each application server is a separate copy per server, and revoking a permission means clearing all of them. Teams either broadcast clear-the-cache messages, which is a sizeable project of its own, or accept that some servers will be stale.

The permission cache is the one cache where being stale is not a performance problem but a security one.

Serving permission rows from a synced memory store

redfly keeps the user, role and permission tables in Redis (a memory store shared by every application server, answering in about a millisecond) and serves reads from there through its API. A sync service watches the database for changes to those tables. On SQL Server it reads change tracking on a short interval; on PostgreSQL and MongoDB it streams the changes through logical replication and change streams. A changed row reaches Redis shortly after.

The application calls Get or GetRows for the caller's roles and scopes and gets a memory read. There is one shared copy, so there is nothing to broadcast. Joins are served from Redis too, so the user-to-role-to-permission lookup is one call, not three. And there are no expiry timers in the application, because the copy is refreshed by the change itself.

The safety questions, answered honestly

How fast does a revoked permission disappear? Shortly after the row changes in the database; on SQL Server the interval at which change tracking is read sets the bound, and on the streamed engines it is the small delay in the stream. That is not instant. For a session that must end the moment an administrator clicks, the application should also check a short-lived token (a signed pass that expires quickly) or make a database read at the sensitive operation, and we say so rather than promise otherwise.

What if the cache is unreachable? Reads fall back to the database automatically. The permission check gets slower, not wrong; it reads the same tables it would have read before the cache existed. There is no mode where the check is skipped.

Can the cache grant a permission the database never did? Only by being stale in the other direction: a newly granted role takes a moment to appear. That is the safe direction, and it is the one to design for.

What about the credentials themselves? Password hashes (the scrambled form passwords are stored in) and other secrets should not be in any cache, and the tables synced to Redis are chosen by the customer; the sync touches only what its database connection is authorized to read. Keep the secret tables out of the sync and check them against the database.

Where the lines are

A short lag is right for authorization on ordinary requests: viewing a page, listing rows, calling a service. It is not right for the action that moves money or deletes data, where the application should confirm against the database inside the same transaction. The cache takes the millions of ordinary checks off the database so that the rare expensive check has a database with headroom to answer it.

The lag itself should be measured, not assumed. Measure the read interval on SQL Server, or the stream delay on the other engines, and write the number into the security review so the people who own the permission model know what they are accepting.

Where redfly fits

redfly supplies the API and the sync service for the customer's own SQL Server, MongoDB or PostgreSQL database, deployed in the customer's cloud account or on their own servers. The permission model, the tables and the decision about which checks go to the database stay with the customer's team.

Why we generate the data access layer

2026-09-25T00:00:00Z

Every application has a layer of code whose only job is to move rows between the database and the rest of the program. Load a row by its key. Save a changed one. Fetch a page of rows that match a filter. It is dull code, there is a great deal of it, and in most codebases it is written by hand, one table at a time.

Inside redfly we do not write that layer by hand. We generate it from the database schema (the list of tables, columns, keys and types that the database already knows about). The generated code covers both halves of the job: talking to the database, and talking to Redis, the memory cache that serves reads.

What the generator reads and what it writes

The input is the schema of the customer's database. From it we produce a typed client for each table (code that knows in advance what fields each row has): the entry points the redfly API exposes, which are Get, Insert, Update, Delete, GetRows and GetTotalRowCount. Each entry point knows the shape of its rows, the key that identifies them, and which fields can be filtered or joined on.

The same schema also drives the cache side. The generator decides how a row is stored in Redis, how a joined result or a multi-field lookup is served from Redis rather than from the database, and which cached entries must be refreshed when a given table changes. That last part is the one most teams get wrong by hand, and it is the reason we wanted a machine to do it.

Consistency is the first win

Hand-written data access drifts. One engineer caches a lookup, another does not. One remembers to refresh the cache after a write, another forgets, and the bug surfaces weeks later as a stale screen nobody can reproduce. The rules for the orders table end up subtly different from the rules for the shipments table, not because the data differs but because different people wrote them on different days.

Generated code cannot drift in that way. Every table gets the same treatment because the same generator produced it. When we fix a mistake in how a cache entry is refreshed, we fix it in the generator once, regenerate, and every table picks up the fix. There is no hunt through the codebase for the other twelve places the same pattern was copied.

The cache is correct because the generator makes it correct, not because someone remembered.

Testable once, not once per table

A hand-written data layer needs tests for every table, and in practice most tables get none. A generated layer inverts that. We test the generator against a small set of representative schemas: a table with a simple key, a table with a key made of several columns, tables joined through a shared column. If the generator produces correct code for those shapes, it produces correct code for the customer's three hundred tables that share those shapes.

We also test the generated output directly, running the same suite of reads and writes through the generated client against a real database and a real Redis, and checking that what comes back from the cache matches what the database holds. The open-source part of redfly includes the performance test program we use to compare cache hit rate and latency with conventional techniques, so the method is open to inspection.

Cheap to change when a table changes

Schemas move. A column is added, a type widens, a table is split in two. With hand-written code, each of those is a ticket: find every place the table is touched, change it, review it, test it, and hope nothing was missed. With a generator, the change is to re-read the schema and regenerate. The diff shows exactly what moved, and the tests run against the new output the same way they ran against the old.

This is why we can include schema changes in the subscription rather than treat them as billable project work. It is not because schema changes are trivial; it is because the cost of one is bounded by the generator, not by the size of the codebase.

About the 4,000x number

We say redfly is 4,000 times faster than the do-it-yourself alternative, and we want to be plain about what that compares. The figure is measured against hand-written data access and caching code produced by a team of developers, not against the raw database. A read served from Redis is faster than one served from disk, but that is not the comparison being made. The gap between a generated, cache-aware data layer and a typical hand-rolled one, with its extra round trips, its cache misses and its over-fetching (reading more than the screen needs), is where the number comes from.

We make this distinction because the number is easy to misread, and we would rather it be understood than merely impressive.

The trade we are making

Generating the data layer means giving up some hand-tuning. An engineer cannot write a bespoke query for one heavily used read and cache it in a special way, because the generator owns that path. In exchange, the whole layer behaves the same way, is tested the same way, and changes at the same cost regardless of how many tables the database has grown to. For the systems we build and run, that trade has been worth it every time.

A team using redfly gets that generated layer behind the API and never sees it; their code calls Get or GetRows, and keeping the rest correct is our job.

Testing a sync pipeline that must never lose a row

2026-09-25T00:00:00Z

The redfly Sync Service has one job: every row that is added, changed or deleted in the source database must show up, correctly, in the copy. The copy might be Redis sitting next to the database, or a central store on the far side of a slow public internet link. Either way, a missed row is not a minor bug. It is a wrong answer on a report that someone will act on.

Testing that promise is different from testing ordinary application code. The interesting failures do not happen when everything goes smoothly. They happen when the process dies halfway through a batch, when the network drops for an hour, when a row is edited twice in the time it takes to send it once. So our tests are built around those moments.

The design we are testing

The tests only make sense against the design, so here it is in brief. Changes are picked up from the source using the mechanism each engine offers: change tracking read on a short interval on SQL Server, logical replication streamed on PostgreSQL, change streams streamed on MongoDB. Each change is queued, and each table remembers its own position in that queue.

Delivery is at-least-once: a change can arrive at the copy more than once, but never zero times. On arrival, the copy reconciles: it applies the change if it is new, and recognises it if it has already been applied. We do not claim, and do not test for, a guarantee that each change arrives one time only. We test that nothing is lost and that duplicates do no harm.

Deterministic replay of change sets

The foundation of the suite is a library of recorded change sets: sequences of inserts, updates and deletes against known tables, with the expected end state of each table written down beside them. Some are small and pointed, such as insert a row, update it, delete it, then insert a row with the same key again. Others are long and noisy, generated from a seed (a starting number that makes a random sequence repeatable) so they can be reproduced exactly.

Because the same change set can be replayed any number of times, we can run it through the pipeline under different conditions and compare results. The end state should be identical whether the run was smooth or interrupted six times. If it is not, the interruption exposed something, and the seed lets us reproduce it on demand.

Killing the process mid-batch

The first hostile condition is a crash. A test program runs the sync service against a replayed change set and terminates the process at a chosen point: after a batch is read from the source but before it is sent, after it is sent but before the position is saved, in the middle of writing to the copy. The service is then restarted and allowed to finish.

The check afterwards is simple. Every table's saved position must be at or before the last change actually applied to the copy, never after it. If the position ran ahead of the data, a restart would skip rows, and that is exactly the loss we are guarding against. Positions that lag behind are fine; they just mean some changes are re-sent, and the reconciliation on arrival absorbs them.

Dropping the network

The second condition is a lost link. For the remote case, where changes leave a site and travel over the public internet, we run the pipeline with a network layer we can cut and restore on command. Tests cut it at awkward moments: mid-transfer, during the acknowledgement (the copy's confirmation that a batch arrived), repeatedly in quick succession.

While the link is down, changes must keep queueing on the source side, compressed and encrypted, and the source database must not be read any harder than usual. When the link returns, each table must resume from its own saved position, not from the beginning and not from where some other table happened to be. A site that was offline for a day should catch up on its own once the link is back, and the tests check that it does.

Comparing source and copy row by row

Every scenario ends the same way. Once the pipeline reports that it has caught up, a separate checker reads every row from the source and every row from the copy and compares them. Not counts; rows. A count can match while two different rows are wrong. The checker reports each row that is missing from the copy, each row present in the copy that no longer exists in the source, and each row whose fields differ.

This checker is deliberately independent of the sync code. It shares no logic with it, so a bug in how the pipeline understands a change cannot also hide in how the checker verifies it. It is slow, and that is acceptable; it runs in the test suite, not in production.

A count that matches proves nothing; only the rows do.

What this buys in production

The same at-least-once design that the tests exercise is what lets the service run over links that cannot be trusted. Drift detection and backfill in production are the checker's slower cousins: they look for the same kinds of disagreement between source and copy and repair them. When the tests pass, we know the repair paths work, because the tests forced them to run.

This is the sync service that sits behind redfly, keeping Redis in step with a database on a local link and carrying warehouse data to a central store over a remote one.

The warehouse comes first: rate limiting and priority lanes in a sync service

2026-09-25T00:00:00Z

A sync service that runs inside a warehouse shares its hardware with the system that actually runs the warehouse. The same server that answers a forklift driver's scan is the server our service reads changes from. If the sync service takes what it wants whenever it wants, the warehouse feels it, and a warehouse that slows down during a peak shift is far worse than a report that is a minute late.

So the rule inside redfly is blunt: the warehouse comes first. The sync service is a guest on that machine, and it behaves like one. This article is about how a piece of software can tell it is being greedy, and what it does about it.

Watching the host, not the queue

The obvious thing for a sync service to watch is its own backlog. That is the wrong signal. A growing backlog says "send faster", which is precisely what a busy site does not need. A service of this design has to watch the machine it lives on and the database it reads from instead, and the useful signals are whichever ones show that the warehouse itself is under strain: a busy processor, memory running short, reads that take longer than they did a moment ago.

When those signs appear, the service should assume it is part of the problem and step back. It does not know whether the load is a shift change, a month-end run or a backup job, and it does not need to know. Any of them outranks moving rows to the cloud.

Batch sizing as the throttle

The service reads changes in batches, and batch size is the natural lever for a throttle of this kind. Under quiet conditions batches can be large, so a backlog drains quickly. As load rises they should shrink and the pauses between them should lengthen, until in the worst case the service stops reading altogether and checks back later to see whether the site has calmed down.

Backing off should be gradual in both directions. A service that halved its batch size at the first sign of load and doubled it the moment load dipped would swing back and forth, hitting the database in bursts. It is better to settle slowly and stay settled. What matters is that the throttle answers to the site's load, never to the size of the backlog.

Priority lanes

Not every table matters equally, and the customer decides which matter most. When a site is set up, someone ranks the tables. Rows from the top-ranked tables travel first; the rest follow behind. In practice the tables at the front are the ones someone at head office is waiting on, such as shipments and stock movements, and the ones at the back are logs and history that nobody reads until the end of the quarter.

Lanes matter most exactly when the service is throttled. If the site is busy and the service can only move a trickle, the trickle should be the rows people are waiting for. A single queue that sends changes in the order they happened would spend that scarce budget on whatever happened to change first. The priority lane spends it on what matters. Once the site is quiet again, the lower lanes catch up, table by table, each from its own saved position.

When there is only a trickle to spend, spend it on what someone is waiting for.

Why the operational database always wins

There is a simpler design in which the sync service is just another client with a fixed schedule, and the warehouse has to live with it. We rejected it because the whole point of moving reporting to a central copy is to take load off the onsite servers. A sync service that added load back during peaks would undo its own reason for existing.

The claim we make for this setup is about resilience, not speed. Reports run against the central copy, never against the warehouse database. The sync service reads only what changed, and only when the site can spare it. Between them, that frees headroom on the onsite servers, and headroom is what keeps a warehouse system standing on a peak day when it is already running hot.

What "current within a couple of minutes" costs and buys

All of this holding back has a price. The central copy is not instantaneous. End to end, a change typically lands in the central store somewhere between thirty seconds and two minutes after it happens on site, and while the service is holding back, the lower-priority tables can lag further. If a report needs the row that was written this second, the central copy is the wrong place to read it.

What that buys is a warehouse that never has to choose between serving its own operation and feeding head office. For multi-site reporting, a couple of minutes is invisible; nobody reconciles thirty warehouses more often than that. And a site that goes quiet for an hour, because its link dropped or its servers were busy, simply catches up afterwards from where each table left off.

This is the sync service redfly runs at each site in a multi-site deployment, sending rows outward to a low-cost central store that the customer's own reports read from.

Database change notifications as a first step before adopting an API

2026-09-25T00:00:00Z

Many teams we talk to have the same hesitation. They like the idea of reads served from memory and a cache they never have to maintain, but pointing an application at a new API is a change to code that works, and code that works is not something anyone changes lightly. So we offer a smaller first step: database change notifications.

The idea is simple. The redfly Sync Service already watches the database for changes so it can keep Redis in step. Before anything reads from Redis, and before a single line of application code moves, a team can subscribe to those same changes for the tables it chooses and react to them in the systems it already has.

What a notification is

A change notification is a message that says a row in a given table was added, changed or deleted, and which row it was. It comes from the same change capture mechanism the full product uses: change tracking read on a short interval on SQL Server, logical replication streamed on PostgreSQL, change streams streamed on MongoDB.

Nothing is installed inside the database beyond what that mechanism needs. No triggers, no reader of the transaction log (the database's own journal of every write) on SQL Server, no extra table the application has to write to. The service connects with the credentials it is given and sees only the tables that connection permits and the team has chosen.

Why start here

The thing most teams are doing today, if they have any caching at all, is polling. A job wakes up every few seconds, asks the database what changed since last time, and refreshes whatever it finds. Polling is load on the database that produces nothing most of the time, and it is always a little late. Notifications replace the question "has anything changed?" with the answer "this row changed", delivered when it happens.

That swap costs very little to adopt. The application does not change. The database does not change. A small consumer (a program that receives the notifications) is written to act on them, and the existing caching, however hand-rolled it is, gets a correct signal instead of a guess.

What you can build with notifications alone

Quite a lot, as it turns out. Three common uses:

Cache refresh. The team's existing cache, whatever it is, drops or reloads an entry when the row behind it changes, instead of expiring it on a timer and hoping.
Search index updates. A product or document index is updated when the source row changes, so search stops lagging the database by however long the nightly rebuild takes. The full product can sync to Azure Search directly; notifications let a team do the same for whatever index it already runs.
Alerts and workflows. A row moving into a certain state, an order marked shipped, a stock level crossing a threshold, sends a message or starts a process, without anyone polling for it.

None of these require the redfly API. They require a subscription, a consumer, and a decision about which tables matter. That is a small piece of work, not a project.

The cheapest way to trust a system is to let it do one small thing correctly for a while.

What notifications do not give you

Being honest about the limits matters more than the list above. Notifications tell you a row changed; they do not serve the row to your application from memory. Your reads still go to the database, and the cache is still yours to keep correct, even if the signal driving it is now better. The 70% saving on the database line of the cloud bill comes from moving reads off the database, and that is the second step, not this one.

Delivery from the sync service is at-least-once, and a consumer of notifications should assume the same. It may see the same change twice, particularly after a restart or a dropped link, so it should treat a notification as "this row is now worth re-reading" rather than as an event to be counted.

The path from notifications to the API

Once a team has run notifications for a while, the second step is smaller than it looked at the start. The sync service is already connected, already authorised for the right tables, and already trusted to see every change. The next step is letting those same changes also land in Redis, the memory cache the service keeps in step with the database.

From there, reads move one place in the code at a time. A permission check, a profile load, a lookup that runs on every page: each one is switched from a hand-written query to the matching redfly entry point, Get or GetRows, and the rest stay as they were until someone gets to them. Writes keep going straight to the database throughout. If Redis is ever unreachable, reads fall back to the database on their own, so a half-migrated application is never worse off than an unmigrated one.

Notifications are the bridge redfly offers between a database you are not ready to change and an API you are not yet ready to trust; the same sync service carries you across both.

What a design partner gets, and why it is a subscription rather than a project

2026-09-25T00:00:00Z

Most software companies sell one of two things: a licence you install and forget, or a project with a start date, an end date and a handover. redfly sells neither. Today the way to adopt redfly is to become a design partner, and the relationship is a subscription with a minimum term of twelve months.

What the relationship includes

The first thing a design partner gets is an architecture review against their own schema (the layout of tables and fields in their database). We look at the shape of the data, the read and write patterns and the peaks the business has to survive, and we say where redfly helps, where it does not, and how large the footprint (the servers and memory it needs) has to be. This happens before anything is deployed, and it is part of the subscription rather than a separate engagement.

The second is the software itself: the redfly API (the single entry point an application calls for its data) and the Sync Service that keeps the memory cache in step with the database. It runs in the partner's own cloud account by preference, as a fully managed service in redfly's cloud, or on the partner's own servers. Whichever shape suits, the partner's application code contains no cache logic to write or maintain.

The third is our team's direct involvement. We set redfly up with the partner, against their database, and we stay until the numbers show up on their bill. Monitoring, tuning, schema changes and upgrades are included, with direct access to the engineers who built the product.

Influence over what gets built next

Design partners shape the product. redfly is early, and the order in which things get built is decided by what the partners in front of us need rather than by a list drawn up in advance. The order of databases is set (SQL Server now, MongoDB next, Postgres after that), but what gets built around them is shaped by the partners using it.

Once the programme closes, the product will reflect the priorities of the partners who were there. We are looking for a small number of companies whose problems are representative of the market we serve, and we would rather solve their problems thoroughly than solve everyone's problems partially.

A design partner is not a customer who arrived early; it is a company that helps decide what the product becomes.

Why a subscription and not a project

A project ends. Keeping a memory cache truthful against a live database does not. Schemas drift as features ship, traffic moves around as the business grows, and a table that was quiet in March is busy in November. If redfly were sold as a one-off build, the day after handover would be the day the partner started re-learning everything we know about running it.

So the price covers the running, not the building. Support, monitoring, tuning, product improvements and deployment are included for as long as the subscription lasts. It is not priced per seat or per request. Payment is annual at the lower rate, or monthly across the same term for a little more.

Why twelve months at minimum

The minimum term is twelve months because that is roughly how long it takes to see the whole picture. A quarter may be enough to deploy and see the first effect on the bill. It is not enough to live through a peak season, a schema change, a cloud provider incident and a couple of releases that touch the busiest tables.

A shorter term would also push both sides toward the wrong behaviour. We would optimise for a quick demonstration rather than for durable operation, and the partner would judge the relationship before it had done its job. Twelve months aligns the incentives: we are paid to keep it working, and the partner has time to see that it does.

No free trial, no free review

There is no free trial, and there is no free review either. We say this plainly because it is unusual, and because the reasons are practical rather than commercial. An architecture review against a real schema is engineering work by senior people, and much of the value early in the relationship comes from it. Giving it away would mean doing it less carefully.

What we offer instead is proof of a different kind. Live consumer products run on redfly today, in production, on a small infrastructure footprint, and anyone can open them and see how they behave. The source code for the client side is public. And the architecture review comes before anything is deployed, so a partner knows what the footprint and the expected savings look like before the system touches their traffic.

What we ask of a partner

The requests are modest. We ask for a real database with real traffic, on SQL Server, MongoDB or Postgres; SQL Server is available now, with the others following in that order. We ask for one technical contact who can grant access and make decisions. We ask for candour about what is working and what is not, because a partner who is quietly unhappy teaches us nothing.

We also ask for patience with an early product. Some rough edges exist, and design partners see them first. In return, design-partner pricing is an early-customer rate held for the length of the contract, and the programme does not stay open indefinitely.

redfly fits companies whose database has become the most expensive and fragile part of their stack, and who would rather work closely with the team that built the fix than buy a box and hope.

Cloud, managed or onsite: choosing where your data platform runs

2026-09-25T00:00:00Z

Where a piece of infrastructure runs is a business decision dressed up as a technical one. It decides who is on call, whose invoice the computing appears on, which auditor's questions you can answer, and how hard it is to leave. redfly can be deployed in three shapes, and the choice between them is usually made by the operations leader or the chief technology officer rather than by the engineers who will use it day to day. This article sets out the three shapes, who each one suits, and why we built the product so that the decision can change later.

The three shapes

The first shape, and the one we prefer, is the customer's own cloud account. The redfly API (the single entry point an application calls for its data) and the Sync Service (the process that keeps the memory cache in step with the database) run inside the customer's own account with their cloud provider, next to the database they already pay for. redfly is licensed; the hosting is the customer's.

The second is fully managed. redfly runs in our cloud as a service we operate, connected to the customer's database.

The third is onsite (on the company's own servers). The same software runs on servers in the customer's own data centre or server room, behind their firewall, against a database that never leaves the building. In all three shapes the application code is the same, and the customer's codebase contains no cache logic.

Your own cloud account: the default for most

We prefer this shape for a plain reason: it keeps the money and the control where they belong. There is no hosting markup in our price because we are not hosting anything. The customer keeps whatever discounts they have already negotiated with their cloud provider, and the computing and storage appear on the bill they already reconcile every month.

It also keeps the data inside a boundary the customer already governs. Access policies, network rules, backup schedules and audit logging are the ones the security team already signed off. redfly is another workload inside that boundary, not a new destination for the data. During the architecture review we size the footprint (the servers and memory the deployment needs), and the savings we quote are net of that footprint.

This shape suits any company that already runs a cloud account with someone responsible for it. That includes most software companies past their first few hires, and most mid-sized businesses that have moved a production database to the cloud.

Fully managed: for teams without operations staff

Some companies have a product, customers and revenue, and nobody whose job is to keep servers healthy. Asking that team to run another service in their account is asking them to do work they did not hire for. For them, fully managed is the right shape.

In this arrangement we set up, monitor, upgrade and tune the deployment, and the customer's team sees only the API. The trade is straightforward. The customer gives up some control over where the service physically runs, and in return the operational burden is ours.

Fully managed is not the premium option; it is the option for teams who would rather ship features than run infrastructure.

Onsite: for existing data centres and regulated work

Plenty of businesses own servers and intend to keep them. Manufacturers, logistics operators and long-established firms often run a data centre they have already paid for, staffed by people who know it. Others are in regulated industries where the rules, or the auditors interpreting them, require that certain data never leaves premises the company controls.

For both, onsite is the natural fit. redfly runs on the customer's hardware, against the customer's database, on the customer's network. Nothing about the product assumes a cloud. The memory cache, the sync process and the API all run as ordinary software on ordinary servers, so the operations team treats the deployment the way it treats any other internal system.

The cost picture differs from the cloud, but the mechanism is the same. Fewer reads reach the database, so the database server has more headroom, and the next hardware upgrade can often wait.

Why the choice can change

The reason all three shapes are possible is a design constraint we hold to: redfly has no dependency on any cloud vendor's proprietary services. It uses databases the customer already runs (SQL Server first, then MongoDB, then Postgres), a memory cache (Redis) that is available everywhere, and software written to run wherever those two things are.

That constraint has a consequence for the business. A company that starts fully managed because it has no operations staff can move into its own cloud account once it hires them. A company that starts onsite can move to the cloud when the data centre lease ends. A company that moves clouds for commercial reasons takes redfly with it. None of these moves needs the application rewritten, because nothing in it is tied to a particular vendor's services; the work is in the deployment, not the code.

Which cloud

Azure is our preferred cloud. SQL Server on Azure with C# goes live right away, and everything else follows a clear order of preference during onboarding. Other clouds work; they come later in that order rather than being excluded. The preference is about how quickly we can get a customer live, not about what the product can do.

redfly fits companies that want the database to cost less and hold up better under load, without handing the decision about where their data lives to the vendor supplying the fix.

Manual cache management is a 1990s tax you do not have to pay

2025-10-14T00:00:00Z

Walk into any product team that has scaled past a few thousand users and you will find the same room. A whiteboard with arrows. Someone explaining why the cache is stale. Someone else suggesting a TTL. A third engineer saying the TTL will not fix it, because the write went to a different shard. The fourth person is just tired.

This conversation has been happening in roughly the same form since the 90s. The technology around it changed. Redis got cheaper. The cloud got bigger. The conversation stayed the same.

Why does this never get solved?

Because cache invalidation is not actually a generic problem. It is a problem about your specific data, your specific writes, and your specific consistency tolerance. So every team writes their own bespoke invalidation rules. Every team ships their own bugs.

redfly takes a different position: the database knows when data changed. Generate the cache layer from the schema, listen to the right change signals, and the cache stops lying. No TTL gymnastics. No "flush on deploy" rituals.

The honest tradeoff

Yes, you give up some control. You no longer hand-roll which fields get cached and which do not. In exchange, you stop staffing a permanent cache-invalidation team. For most companies, that is the right trade.

Cache invalidation is not hard. Doing it by hand, forever, for every product, is hard.

Redis is not slow. Your integration is.

2025-09-22T00:00:00Z

Redis is not slow. Redis is a single-digit-millisecond key-value store running in RAM. When someone says "Redis is slow," they almost always mean: "the integration between my application, my ORM, my cache layer, and Redis is slow."

Where the latency actually lives

Serializing and deserializing the wrong objects
Cache-miss storms because invalidation was too aggressive
Network hops in a misconfigured topology
Lock contention in the client library
Caching at the wrong granularity (per-row instead of per-view)

Redis is the fastest part of the system. If your app feels slow, the integration is the suspect. That is uncomfortable, because the integration is something you built, and the integration is what redfly replaces with a generated, tested implementation.

What good looks like

Sub-200ms page loads on commodity database hardware. Cache hit rates above 90% on read-heavy workloads. Cache invalidation that is correct by construction, not by code review. None of that requires faster Redis. It requires a smarter cache layer between Redis and your application.

The 70%-cheaper backend: how Redis-fronted databases change cloud economics

2025-09-08T00:00:00Z

Look at your last invoice. The relational database line item is almost certainly in the top three. Often it is number one. The reason is not that databases are expensive in isolation (they are not) but that we ask them to do work they should not be doing.

The 95/5 rule

In most applications, 95% of traffic is reads and 5% is writes. The reads do not need a relational database; they need a fast view of relational data. The writes do need a relational database. So if you can serve the 95% from a much cheaper substrate, the bill collapses.

Redis is roughly an order of magnitude cheaper per request than SQL Server at scale. Move the 95% there and the math changes immediately. We routinely see 70% cost reductions on infrastructure, before any other optimization.

Why does not everyone already do this?

Because the integration is hard. We covered that in the last post. The economics are obvious. The implementation is not. That gap; the gap between "this would save us 70%" and "we shipped it"; is what redfly closes.

The point of a framework is not to be clever. The point of a framework is to make the obvious thing easy enough that everyone does it.

Cache more, optimize less: a primer on read-heavy data access patterns

2025-08-19T00:00:00Z

Engineers love query optimization. It is craftsmanship. It is fun. It is also the wrong battle for most applications.

The two questions that matter

How often does this data actually change?
How often does it actually get read?

If a piece of data is read 1,000 times for every 1 write, no amount of query tuning is the right answer. Cache it. The query that gets run zero times is infinitely fast.

The trap of "we'll cache it later"

"Later" never arrives. Caching gets bolted on when the database is already on fire. By then, the cost of getting invalidation right is so high that teams settle for half-correct solutions and ship bugs. The right time to design for caching is when you design the schema.

redfly is built around that assumption. The cache is not an afterthought. It is generated alongside the data access layer, from the same schema. By the time the first endpoint exists, the cache exists.

Why code generation beats hand-written DAL for any team under 20 engineers

2025-07-30T00:00:00Z

Small teams write data access code by hand because they think it gives them control. What they actually get is a slowly diverging set of patterns, written by different engineers on different days, none of them tested as a system.

The hidden costs

Three engineers, three opinions on transaction boundaries
Every new entity becomes a small political negotiation
Bugs cluster around the interface between hand-written code and the rest of the system
Onboarding cost grows linearly with the size of the DAL

A generated DAL has none of these. It is consistent because a compiler enforces consistency. It is testable because the same code is generated across every project. It is faster to write, by definition, because nobody is writing it.

When hand-rolling actually wins

If you have 50+ engineers and a deep platform team, hand-rolled DAL with shared libraries can match a generator. Below that, it cannot. The math does not work.

Generated code is not lazy. It is the highest-leverage form of seriousness.

Designing for the freemium user model without crashing your database

2025-07-11T00:00:00Z

Freemium is the dominant go-to-market for consumer-adjacent SaaS. It is also the fastest way to break a backend nobody designed for free traffic.

The unit-economics problem

Free users do not pay. But they generate reads. A lot of reads. If each free read costs you a database query, you are paying for users who will never pay you. The economics of free-tier reads only work if those reads cost something close to zero.

The redfly answer

Free-tier reads come out of Redis. Paid-tier reads come out of Redis. Writes hit the database. The cache layer is generated, so it is correct without engineering effort. Free traffic stops being a cost center; it becomes a flywheel.

This is not theoretical. We have seen this pattern in production at meaningful scale; including holiday spikes where read volume jumped by an order of magnitude in hours.

Sub-200ms is not a stretch goal; it is a default

2025-06-25T00:00:00Z

There is a kind of resignation built into most product roadmaps: "we will get to performance once the feature is shipped." By the time the team gets there, the performance is structural and impossible to fix without a rewrite.

Why 200ms?

Because that is the threshold at which a user stops perceiving a delay. Above it, the app feels slow even if every other thing is right. Below it, the app feels fast even if some things are wrong.

What it actually takes

Read paths served from RAM, not disk
A cache that is correct without polling
Database queries that only run on real misses
A CDN that does its job and stops there

None of this is exotic. It is what good products do. The reason most do not is the cost of getting there with hand-written code. Generated code changes the cost.

Backend-as-a-service for the post-PMF SMB

2025-05-30T00:00:00Z

There is a category of company that nobody serves well. They have product-market fit. They have customers. They have revenue. They do not have a platform team. They cannot afford to build one. They cannot afford to not have one.

Why the existing options fail

Enterprise infrastructure: priced for companies 100x their size
DIY: needs senior engineers they cannot hire
Generic BaaS: too opinionated, locks them in, performs poorly at the join boundary
Consulting: starts at $500k and ends in a code base nobody can maintain

The redfly fit

redfly is built for this buyer. The Starter, Pre-seed Growth, Seed, and SMB plans exist because the unit economics for a 5-person team are different from the unit economics for a 500-person team. Same product. Same engineering quality. Pricing that reflects scale.

If you are this company: the cheapest experiment is a design-partner conversation. The most expensive experiment is to wait six months and find out the database is still the bottleneck.