What a national TV advert does to your infrastructure
175,000 concurrent users arrived in about ninety seconds. Here is what broke, what didn't, and what I would do differently.
Most traffic spikes are polite. A post does well, a newsletter goes out, and you get a curve — a rise you can watch, with time to react while it happens.
A national television advert is not a curve. It is a step function. The advert runs, and somewhere between thirty and ninety seconds later a very large number of people who have never heard of your product are all on it at once. You do not get to react. Whatever your system is at the moment the advert airs is what it will be when the traffic arrives.
We took a UK national platform through one of these to a peak of 175,000 concurrent users. Here is what I learned, in roughly the order I learned it.
The spike is shorter and sharper than you plan for
Every capacity conversation I had before launch was framed in terms of sustained load. How many users per hour, per day, at peak. That framing is comfortable and almost entirely useless for a broadcast spike.
The number that matters is arrivals per second in the first two minutes, and it is much larger than a naive division of the total suggests. People do not distribute themselves evenly across the minute after an advert. They pick up their phone during it.
Model the spike as arrival rate, not concurrency. Concurrency is what you observe afterwards; arrival rate is what breaks things.
Autoscaling is too slow, and that is fine
There is a persistent hope that autoscaling handles this. It does not. A cold pod that takes forty seconds to become ready is useless when the spike is over in ninety.
The answer is boring: pre-scale. We knew the broadcast slot. We scaled up before it, held the capacity through the window, and scaled back down afterwards. The cost of running well over capacity for two hours is trivial. The cost of failing during the two hours you spent your entire marketing budget on is not.
Your database connection pool is the real ceiling
Application instances are cheap and horizontal. Postgres connections are neither. Every new pod that comes up wants its own pool, and the ceiling you hit is not CPU on the web tier — it's connection exhaustion on the database, at which point everything fails at once rather than degrading.
Two things helped more than anything else we did:
- A connection pooler in front of the database, so instance count and connection count stopped being the same number.
- Ruthlessly moving read paths off the primary. Most of what a spike wants is the same handful of read queries; almost none of it needs to be strongly consistent.
Cache the first screen aggressively, personalise below the fold
The overwhelming majority of that 175,000 wanted exactly one thing: the page the advert told them to go to. They were not logged in. They had no personalisation to render.
Serving that page from the CDN, fully static, with the personalised parts loaded afterwards, removed most of the spike from the origin entirely. This is not a clever technique. It is just being honest about which parts of a page actually vary per user — which, on a landing page, is usually a lot less than the framework encourages you to assume.
Test beyond the number you expect
We load tested past one million concurrent — several times the traffic anyone thought was plausible. That felt excessive while we were doing it.
It was not, for two reasons. First, the headroom meant that when the real number came in higher than the forecast, nobody had to make a decision under pressure. Second, and more usefully, testing well past the expected load is how you find which component fails first. At expected load everything passes and you learn nothing. At four times expected load, the ordering of failures tells you exactly where your architecture is thin.
What I would do differently
Instrument the arrival rate specifically. We had excellent dashboards for concurrency, latency and error rate, and no single panel showing new sessions per second. That was the number everyone actually wanted to look at during the broadcast.
Rehearse the rollback. We had a rollback plan. We had not practised it under time pressure with the specific people who would be awake at that hour. A plan you have not rehearsed is a document, not a capability.
Agree the failure mode in advance. The interesting question is not "will it stay up" but "if it degrades, what do we sacrifice first?" Deciding that with your product team on a calm afternoon is much better than deciding it at 21:04 on the night.
None of this is exotic. Pre-scale, pool your connections, cache the thing everyone is asking for, test past the number you believe, and know what you will give up. The difficulty is not the techniques — it is doing them before you have the evidence that you needed to, because after the advert runs there is no second attempt.