The warehouse comes first: rate limiting and priority lanes in a sync service

A sync service that runs inside a warehouse shares its hardware with the system that actually runs the warehouse. The same server that answers a forklift driver's scan is the server our service reads changes from. If the sync service takes what it wants whenever it wants, the warehouse feels it, and a warehouse that slows down during a peak shift is far worse than a report that is a minute late.

So the rule inside redfly is blunt: the warehouse comes first. The sync service is a guest on that machine, and it behaves like one. This article is about how a piece of software can tell it is being greedy, and what it does about it.

Watching the host, not the queue

The obvious thing for a sync service to watch is its own backlog. That is the wrong signal. A growing backlog says "send faster", which is precisely what a busy site does not need. A service of this design has to watch the machine it lives on and the database it reads from instead, and the useful signals are whichever ones show that the warehouse itself is under strain: a busy processor, memory running short, reads that take longer than they did a moment ago.

When those signs appear, the service should assume it is part of the problem and step back. It does not know whether the load is a shift change, a month-end run or a backup job, and it does not need to know. Any of them outranks moving rows to the cloud.

Batch sizing as the throttle

The service reads changes in batches, and batch size is the natural lever for a throttle of this kind. Under quiet conditions batches can be large, so a backlog drains quickly. As load rises they should shrink and the pauses between them should lengthen, until in the worst case the service stops reading altogether and checks back later to see whether the site has calmed down.

Backing off should be gradual in both directions. A service that halved its batch size at the first sign of load and doubled it the moment load dipped would swing back and forth, hitting the database in bursts. It is better to settle slowly and stay settled. What matters is that the throttle answers to the site's load, never to the size of the backlog.

Priority lanes

Not every table matters equally, and the customer decides which matter most. When a site is set up, someone ranks the tables. Rows from the top-ranked tables travel first; the rest follow behind. In practice the tables at the front are the ones someone at head office is waiting on, such as shipments and stock movements, and the ones at the back are logs and history that nobody reads until the end of the quarter.

Lanes matter most exactly when the service is throttled. If the site is busy and the service can only move a trickle, the trickle should be the rows people are waiting for. A single queue that sends changes in the order they happened would spend that scarce budget on whatever happened to change first. The priority lane spends it on what matters. Once the site is quiet again, the lower lanes catch up, table by table, each from its own saved position.

When there is only a trickle to spend, spend it on what someone is waiting for.

Why the operational database always wins

There is a simpler design in which the sync service is just another client with a fixed schedule, and the warehouse has to live with it. We rejected it because the whole point of moving reporting to a central copy is to take load off the onsite servers. A sync service that added load back during peaks would undo its own reason for existing.

The claim we make for this setup is about resilience, not speed. Reports run against the central copy, never against the warehouse database. The sync service reads only what changed, and only when the site can spare it. Between them, that frees headroom on the onsite servers, and headroom is what keeps a warehouse system standing on a peak day when it is already running hot.

What "current within a couple of minutes" costs and buys

All of this holding back has a price. The central copy is not instantaneous. End to end, a change typically lands in the central store somewhere between thirty seconds and two minutes after it happens on site, and while the service is holding back, the lower-priority tables can lag further. If a report needs the row that was written this second, the central copy is the wrong place to read it.

What that buys is a warehouse that never has to choose between serving its own operation and feeding head office. For multi-site reporting, a couple of minutes is invisible; nobody reconciles thirty warehouses more often than that. And a site that goes quiet for an hour, because its link dropped or its servers were busy, simply catches up afterwards from where each table left off.

This is the sync service redfly runs at each site in a multi-site deployment, sending rows outward to a low-cost central store that the customer's own reports read from.