Why nightly exports are the wrong shape for multi-site reporting
A nightly export shows head office every site as it stood yesterday. We look at what that delay costs, and why a central copy current to within minutes changes the work.
Most companies that run several warehouses, depots or plants already centralize their reporting. The usual form is the nightly export: each site writes out a batch of files after the last shift, a scheduled job carries them to head office, and someone loads them into a reporting database before the morning meeting. It works, in the sense that numbers appear. The trouble is the shape of what arrives.
What a nightly export actually delivers
A nightly export is a photograph taken once a day. Every row in it describes the site as it stood at one moment, usually hours before anyone looks at it. By the time a regional manager reads the stock position at nine in the morning, the site has already received two trucks and shipped a dozen orders.
That gap is built in; it is not a fault in the job. Batch files are cheap to produce and easy to reason about, which is why they became the default. But the design assumes head office only needs to know about yesterday, and that assumption stopped holding once customers began asking where their goods are today.
What head office loses while it waits
The cost of the overnight run rarely shows up as a single line. It shows up as small, repeated losses spread across the day.
- A shortage at one site becomes visible only after the shift that could have covered it from another site has ended.
- A customer asks about an order, and the honest answer is a phone call to the warehouse floor.
- Two sites report different totals for the same product, and the morning goes on working out which file was late.
- When the export fails, nobody notices until the numbers look wrong, and by then a full day is missing.
Each of these pushes work back to the sites. People at the warehouse field calls, resend files and explain discrepancies, which is exactly the load a central report was supposed to take off them.
Why batch files are the wrong shape
The problem is not only lateness. Many exports send a full copy of each table every night, whether one row changed or a million did, so the traffic and the effort grow with the size of the data rather than with the day's activity.
Full copies also handle deletions badly. If a row disappears at a site, a simple load at head office may keep it, and the central picture quietly drifts from the truth. Over months, people stop trusting the central numbers.
Then there is the fragile window. The export has to finish, travel and load in the hours between the last shift and the morning meeting. One slow link or one late job, and the whole day starts blind.
A report that is always a day behind teaches people to stop reading it.
The other shape: a central copy that stays close behind
The alternative is to stop sending photographs and start sending changes. A small service runs inside each site's own network, watches the tables the business has chosen, and picks up the rows that are added, changed or deleted. Those changes queue on site, are compressed and encrypted per site, and travel outward to one central store.
With this approach the central copy is current to within a couple of minutes, roughly thirty seconds to two minutes from end to end. Because only what changed leaves the site, the traffic stays small. Because edits and deletions travel along with new rows, the central copy does not drift. And because every row is stamped with the site it came from, head office can compare sites side by side without reconciling files by hand.
What happens on the bad days
A nightly export either arrives or it does not. A steady flow of changes from each site behaves differently when things go wrong, and that matters more than how fresh the numbers are on a good day.
- When the link drops, changes wait at the site and resume from where they stopped once it returns, table by table. Nothing is lost.
- When the site's machines are busy, the service holds back, so the warehouse systems come first.
- Delivery is at-least-once, so a change may arrive twice after a reconnect. It is reconciled on arrival.
There is a second benefit. Reports run against the central copy, not against the warehouse database. That leaves the onsite servers more spare capacity, so they stay resilient during peaks.
What it does not do
It is worth being plain about the limits. The store at each site is a holding queue for outgoing changes, not a local read copy, so warehouse applications cannot read from it while a site is cut off. Each new site needs its own setup, and adding tables later is a setting someone changes on purpose. A person drives the rollout.
The reports and any portal on top of the central store are still built by the business. What changes is that they read one place, with every site in it, a couple of minutes behind rather than a day.
Choosing between them
A nightly export is still reasonable for a single site, or for figures nobody acts on before tomorrow anyway. Once several sites, several time zones and customer questions are involved, the overnight picture is simply the wrong shape for the job.
redfly builds the second shape with design partners who run more than one site: a small outbound service at each site feeding one low-cost central store, with the customer's own team building the reports on top of it.