From nightly exports to a central copy: what changes for a vendor's dashboard

Many vendors whose product runs at customer sites start with the same arrangement. Each site exports a file overnight, the file travels to your cloud, and your dashboard loads whatever arrived. It works, until the number of sites grows.

We have spent two decades keeping data in step over links that cannot be trusted. On a good day, the nightly file is fine. It is the problem on every other day, and those days add up.

What a nightly export really promises

A nightly export makes three quiet promises to the people reading your dashboard. The data is as fresh as last night. The file contains everything that happened. If something goes wrong, someone will notice and fix it.

Each promise is weaker than it sounds. A view that is a day behind is a view nobody fully trusts, so your support desk calls the site to check before acting. Many exports send only new or changed rows, so a row deleted at the site never leaves the cloud copy. And when a link drops or an export fails, recovery often means a person re-running a job by hand, if they spot the gap at all.

Multiply that by dozens of customer sites, each with its own link, its own schedule and its own way of failing, and your engineers end up maintaining collectors, retries and monitoring instead of building your product.

The alternative: one central copy, kept current

The other approach is to keep a central copy of each site's chosen tables, kept up to date from the site itself. Your dashboard stops reading files and reads that copy instead.

In the way we build it, a small service runs inside the site's own network, beside the site database. It sends a first full copy of each table once, and after that only the rows that were added, changed or deleted.

For SQL Server, it uses change tracking (a built-in feature that notes which rows changed) and reads it on a short interval. For MongoDB and PostgreSQL, it follows the change feed each database provides. Data only travels outward, and nothing at the site is replaced.

The changes land in one central store, with every row stamped with the site it came from and the time it arrived. Your team then builds the dashboards, support tools and analytics on top of that store.

Freshness: from yesterday to minutes

The most visible change is age. Instead of a picture from last night, the central copy is current to within a couple of minutes over ordinary internet links. It is not instant, but it changes how people use the dashboard.

• A support agent can look at the site's data before calling, rather than calling to ask.
• A regional view across all of a customer's sites reflects the same morning, not different nights.
• Problems show up while they can still be acted on, not in the next day's report.

You also choose and rank the tables. Important tables stay current even on a slow link, while large backlogs trickle through in the background.

Completeness: the copy matches the site

A central copy fed by changes shows what each table looks like now, including what was removed.

Edits and deletes travel too

When a row is edited at the site, it is edited centrally. When a row is deleted at the site, it is deleted centrally. Otherwise dashboards quietly overcount, because cancelled orders never leave the cloud copy.

Every row knows where it came from

Because each row carries its site stamp, many sites can feed one store while each keeps its own space. Your application can show one customer's sites together, or one site alone, without stitching files.

A dashboard is only as trustworthy as the worst day of the data behind it.

Recovery: built in rather than bolted on

This is where the difference matters most for operations. With nightly files, recovery is a process people follow. With a central copy fed from changes, recovery is part of how the system works.

1. Changes are queued on the site's disk, compressed and encrypted with a key unique to that site.
2. They are removed from the queue only once the cloud confirms receipt.
3. If the link drops, changes keep queueing and go when it is back.
4. If something restarts, each table resumes from the last change it sent.
5. If a site goes quiet, it reports its health, so a stalled site is noticed rather than discovered in a report.

Delivery is at-least-once, which means a change may occasionally arrive more than once, but nothing is lost. Even a site cut off for days resumes from where it left off when the connection returns.

What it means for the site and for your team

Moving reports to the central copy also takes reporting load off the site database. That leaves headroom, so the site servers stay resilient during peaks. Dynamic throttling (the service eases off when the database is busy) protects that database, so the operation never waits for the sync. That matters at sites where anything competing with the local database gets switched off.

There are honest limits. Each new site needs its own setup, and a person drives the rollout. Adding tables later is a setting, but someone still does the work. The central store holds current state, not a history of events.

redfly runs the site service, watches its health and handles backfill and recovery, while your team builds the dashboards and applications your customers see. We keep the data flowing from every site into one central copy you own.