IoT and fleet telemetry: high writes, flaky links, millisecond reads

A fleet is a flood of data on a bad connection. Every vehicle, sensor and robot reports position, temperature, battery, load and fault codes, over and over, all day; that stream of readings is telemetry. Between reports the link drops: a truck goes under a bridge, a depot loses its internet for an afternoon, a vessel spends a week out of range. Meanwhile somebody in an operations room is looking at a screen and expects it to show what is happening now.

Two jobs that pull in opposite directions

The write side wants durability. Every report has to land somewhere safe, in order, and stay there, because a missed fault code is a missed inspection. The read side wants speed. A dispatcher refreshing a map of four hundred vehicles does not care about the history; they care about the latest known state of each one, and they want it in milliseconds, not seconds.

Asking one database to do both is where most telemetry systems run into trouble. The writes are relentless, and every dashboard refresh lands on the same disk, competing with them. The usual answer is a bigger database, then a hand-built cache in front of it, then a team to keep that cache honest.

Writes go to the database

We keep the database as the place writes land, and only that. The ingestion code, whether it calls the redfly API or writes straight to the database, ends up putting rows in SQL Server, MongoDB or Postgres. That is the source of truth, and nothing else pretends to be.

This matters because telemetry is not just appends. The row for a vehicle gets updated with its latest position; a job row changes status from assigned to in progress to done; a decommissioned sensor is deleted. Edits and deletions have to carry through to wherever the reads come from, or the operations screen starts telling quiet lies.

Reads come from memory

Reads are served from Redis, a store that keeps data in memory and answers in about a millisecond, which the redfly sync service keeps in step with the database. The mechanism depends on the engine: on SQL Server the service reads change tracking on a short interval; on PostgreSQL it follows logical replication, streamed; on MongoDB it follows change streams, streamed. When a row changes in the database, the change reaches Redis without anyone writing invalidation code.

The application asks for data through one API, with entry points such as Get, GetRows and GetTotalRowCount, over REST or gRPC (two standard ways for programs to call a service over a network). The map screen asks for the latest state of every vehicle in a region and gets it back from memory in milliseconds. Joins and multi-field lookups are answered from Redis too, so a query like all vehicles in this depot with an open fault does not fall back to the database just because it touches two tables.

• Writes: straight to the database, unchanged
• Reads: from Redis, kept current by the sync service
• If Redis is unreachable, reads fall back to the database automatically
• No cache code, no time-to-live settings, no invalidation logic in the codebase

When the link is the problem

A vehicle that is offline does not write, so the last row it wrote is the truth until it reconnects. The operations screen should show that last known state with its timestamp, plainly, rather than hiding it or guessing. When the device comes back and its backlog lands in the database, the sync service carries those rows to Redis and the screen catches up.

Depots and remote sites are the harder case. A site often runs its own database on its own network, because the operation cannot stop when the internet does. redfly Remote Sync runs a small service inside that network, watches the tables you choose, and sends only the rows that were added, changed or deleted outward to a central store. Changes are queued on site, compressed and encrypted, and if the link drops they wait and resume from where each table left off when it returns. Nothing is lost during the offline period; it just arrives later.

The device that has been silent for an hour is not a bug in the dashboard; it is a fact the dashboard should show.

What we would not promise

We would not promise reads faster than a millisecond, and we are wary of anyone who does. A read from Redis is fast, in the low milliseconds for a typical lookup, and that is what a dispatcher notices as instant. Above that, network hops between the application and Redis matter more than the store does.

We would also not promise that the central copy is current to the second. Remote sync over the public internet lands changes in roughly thirty seconds to two minutes end to end. For fleet reporting across sites that is plenty; for a control loop that steers a machine, it is not, and that loop belongs on the device.

Where it has run

Software our team built has been used by more than one hundred enterprise customers, and the remote sync design has carried more than ten billion rows over ordinary public internet links in production, with offline periods and high latency as everyday conditions rather than surprises. Telemetry is the same shape at a smaller scale: constant writes, unreliable links, and a screen that has to be right.

redfly fits where the database is taking the writes and the dashboard is slow, or where sites and vehicles cannot stay connected and the central view has to be complete anyway.