Surviving Black Friday with the database at 99% CPU

Every online retailer has a chart they look at once a year with their breath held. It is the CPU (processor) load line on the database server, and on the biggest shopping weekend of the year it climbs towards the top of the graph and stays there. We have been on the operations side of that weekend with the database running at 99% of its CPU and memory, and the site stayed up. This article is about why that was possible and what it takes.

What the database is actually doing at peak

A shopper who buys one item generates a single order, which is one write to the database. Before that shopper reaches checkout, they have loaded a dozen product pages, checked stock on three of them, looked at their basket twice and opened their account to confirm a delivery address. Every one of those is a read.

On a normal day the ratio is lopsided; on peak day it is extreme. The number of orders goes up several times over, but the number of reads goes up far more, because the shoppers who never buy still browse. The database spends most of its capacity answering the same questions over and over: what does this product cost, is it in stock, who is this customer.

What breaks first

The first thing to fail is not the order. It is the product page. When the database is saturated, every request waits its turn in a queue, and the pages that need the most reads to render are the ones that time out first. Product listings, stock checks and account pages go slow, then go blank.

The order path usually survives longest, because it is a small number of writes. But by then it does not matter. A shopper who cannot load the product page never reaches checkout, so the site is effectively down even though orders would technically still go through. The retailer loses revenue on the one day it planned for all year.

The database does not fall over from orders; it falls over from the browsing that happens before them.

Why buying a bigger database for one weekend is the expensive answer

The reflex is to move the database up a size for the peak. Cloud databases are sold in fixed sizes, and the next size up is roughly double the price, not ten percent more; the higher price applies to every hour it runs. Scaling down again afterwards is a second change, with its own risk, and many teams quietly never do it.

There are further problems. The larger size has to be set up and tested before the weekend, which means a rehearsal weeks ahead. The biggest size still has a ceiling, and a hot product launch can blow through it. A team that scales up every peak ends up paying for the largest size several weekends a year, then wondering why the database line on the cloud bill keeps growing.

Serving the reads from memory instead

The alternative is to stop asking the database the same questions millions of times. redfly sits between the application and the database. Reads are answered from a memory cache (Redis, a store that keeps data in memory rather than on disk, so it answers in about a millisecond). Writes, which for a retailer means orders, go straight to the database as before.

The part that makes this safe is keeping the memory copy correct. A sync service watches the database for changes; on SQL Server it reads the built-in change tracking on a short interval, and on PostgreSQL and MongoDB it streams changes as they happen. When a price changes or stock moves, the memory copy is updated shortly after. Nobody on the retailer's team writes cache code, sets expiry times or remembers to clear anything after a write.

What peak day looks like after the change

The product page, the stock check and the account page are now answered from memory. The database sees the orders and the updates that come from them, and very little else. Its CPU line still rises on peak day, but the rise is driven by the writes that actually earn revenue, and that is a far smaller number than the reads were.

If the memory cache ever becomes unreachable, reads fall back to the database automatically. The site does not switch into a degraded mode; it simply gets slower until the cache is back. That is the same position the retailer was in before, with nothing extra to lose.

The honest limits

This does not make the database infinitely large. Orders still have to be written, and a retailer whose writes alone saturate the database needs a different conversation. The memory copy lags the database by a little, so a stock count shown to a shopper can be a moment old; the order itself is still checked against the database when it is placed, which is where that check belongs.

It also takes preparation. The tables to serve from memory have to be chosen, the memory store has to be sized for them, and the application has to be pointed at the new read path. That is real work done before the first peak, not a switch flipped the night before.

Where redfly fits

redfly supplies the read path and the sync that keeps it correct, working with retailers as design partners against their own SQL Server, MongoDB or PostgreSQL database. The retailer keeps its store, its checkout and its database; the database just carries a lot less of the weekend.