Spotty internet: what has to run at the site when the connection drops

Every site loses its internet connection. A cold store has freezer rooms where no signal reaches. A rural depot sits at the end of a single line that a storm takes down. A yard has a corner where the scanners drop out whenever a certain trailer parks there. These are not faults to be fixed before AI can begin; they are the conditions the AI has to work in.

The design question is therefore not "will the connection drop?" but "what happens on the floor when it does?". For each integration the answer has to be written down and tested, because the floor does not stop when the cloud becomes unreachable.

Where the connection goes

Dead zones come in several kinds. Freezer rooms and thick-walled cold stores block wireless signal, so a device inside them is cut off until it comes out. Remote sites rely on one link, often consumer grade, shared with cameras, scanners and the office. Links drop during storms, during peaks when everything is busy at once, and sometimes for no reason anyone finds.

Then there are the slow links. A distant site may be connected all day and still take a noticeable moment for every request to reach the cloud and come back. An integration that asks the cloud a question for every pallet is slow there even when nothing is broken.

What can wait

A lot of useful AI work is not tied to the minute. A demand forecast for next week, a daily summary for head office, a review of yesterday's late orders, a suggestion to move stock between branches. If the link is down for two hours, these run two hours later and nobody on the floor notices.

Reading client emails into order entries usually belongs here too. The mailbox lives in the cloud, the proposed order is checked by a person before it is released, and a short delay while the link recovers costs nothing. For this kind of work, running in the cloud is fine, and the outage plan is simply to pause and resume.

What cannot wait

Other work is only useful at the moment it happens. A temperature drift warning that arrives after the product has warmed is a report, not a warning. A short pick alert that reaches the supervisor after the truck has left is noise. A dock camera that stops being watched when the link drops has stopped working during the very weather that causes the trouble.

For these, a small service runs on a server, a virtual machine or a PC inside the site's own network. It reads the local sensors, the local warehouse database and the local cameras, and it does its work there. Smaller AI models can run on modest hardware at the site. Where a larger cloud model is needed for part of the work, the local service decides what it can alone and queues the rest.

The floor comes first

Whatever runs at the site has to be a good neighbour. The warehouse system that runs the floor, the scanners and the cameras were there first, and an integration that competes with them for a thin link or leans on the warehouse database during the busiest hour is doing harm even when its answers are right.

• It eases off when the site is under pressure and catches up when the pressure lifts, rather than running at full tilt regardless.
• It never holds up the floor. If the integration is unavailable, the floor works as it did before the integration existed.
• It never needs a person to notice an outage and restart something by hand. Recovery is part of the design, not a procedure on a laminated card.
• It fails quietly and visibly: quietly in that nothing on the floor breaks, visibly in that the redfly Portal shows the integration has been out of contact since a given time.

Choosing per integration

None of this is a company policy. It is a decision made for each integration, from three questions: how quickly is the result needed, what happens on the floor if the link is down, and where does the data the integration reads already live? A forecast and a drift warning answer those questions differently and belong in different places.

The outage case is tested deliberately during the pilot. We unplug the link, or wait for the weather to do it, and watch what the integration does and what the floor sees. If anyone on the floor had to do anything, the design is not finished.

Catching up when the link returns

When the connection comes back, the site has a backlog: readings taken, emails that arrived, suggestions made locally that the redfly Portal has not seen. The integration works through them in order, so nothing is skipped and the sequence of events is kept. Alerts that are now stale are marked as such rather than fired as if they were new. Within a few minutes the Portal shows the same picture the site has had all along.

Design every integration for the day the link drops, and the floor will not notice that day when it comes.

Planning for spotty internet is part of how redfly designs each integration for warehouse and distribution companies: deciding what runs at the site and what runs in the cloud, keeping the floor working either way, and catching up in the redfly Portal when the link allows.