What offline support really means when a warehouse loses its link

Offline support is one of those phrases that means whatever the listener hopes it means. To a warehouse manager it may sound like the floor keeps working from a full local copy of everything. To a network engineer it may sound like a promise nobody can keep. It is neither, so this is a precise description of what the redfly sync service does when a site loses its link, and what it does not do.

The shape of the system

At each site a small service runs inside the network, next to the warehouse database. It watches the tables you have chosen and captures rows that are added, changed or deleted. On SQL Server it reads change tracking (the engine's built-in list of which rows changed) on a short interval; on PostgreSQL it uses logical replication and on MongoDB change streams, both of which stream each change to the service as it happens.

Captured changes go into a queue on site. Each batch is compressed and encrypted for that site, then sent outward over an ordinary internet connection to a central document store in your cloud account. The service opens outbound connections only; nothing connects inward.

What happens when the link drops

The capture side does not care about the link. It keeps reading changes from the database and appending them to the onsite queue for as long as the warehouse keeps working. The queue is compressed, so a site can be cut off for a long time before space becomes a concern.

The sending side notices the failure and stops. It discards nothing. When the link returns, sending resumes.

Resumption is per table. Each table remembers its own place and picks up from there. A restart of the service or the machine has the same effect; each table resumes from where it stopped rather than starting over. A long outage therefore costs you the backlog transfer time and nothing else.

Delivery is at-least-once, reconciled on arrival

Here is the part worth being precise about. When the link fails mid-batch, the service cannot always know whether the central store received that batch before the failure. The safe choice is to send it again. That means delivery is at-least-once: every change will arrive, and some changes may arrive twice.

Duplicates are reconciled on arrival. Every row is stamped with the site it came from, and the central store recognizes a change it has already seen, so a repeat does no harm. The central copy ends up correct; it gets there by tolerating repeats rather than by trying to prevent them, which over an unreliable link is the approach that actually works.

Because each table resumes on its own, one table may briefly be further along than another while a backlog catches up. A report that joins two tables during that window may see one ahead of the other until the backlog clears.

What is not promised

The onsite queue is a holding area for outbound changes. It is not a read copy. The warehouse management system cannot read from it, and no application at the site can use it as a local database while the site is cut off. The warehouse keeps working during an outage because its own database is untouched and local, not because of anything the sync service provides.

Nothing flows back in. The central store never updates the warehouse database, and the service is not built to do so. If you need changes to travel toward the site, this is not the tool.

Freshness is a couple of minutes end to end when the link is healthy, roughly thirty seconds to two minutes. During and after an outage it is however long the outage lasted plus the backlog transfer. There is no promise that any particular change is visible centrally by a particular moment.

The service holds back when the site's machines are busy. Under sustained heavy load at the site, the central copy lags further than usual until the load eases. That is by design; the warehouse always comes first.

Why it is built this way

Every one of these choices trades a stronger guarantee for a system that keeps working over links that cannot be trusted. A delivery guarantee with no repeats would require coordination that a dropped connection breaks; at-least-once with reconciliation does not. A readable local copy would put a second database at every site to install, patch and keep consistent; a queue does not.

The result has moved more than ten billion rows over ordinary public internet links in production. That is the evidence we have that the trade is the right one, and it is the reason we describe the guarantees narrowly rather than generously.

Offline support means nothing is lost and nothing has to be restarted; it does not mean the site can read a copy while cut off.

redfly runs this service at each site and keeps the central store current; the reports that read from it are yours to build.