Speed and Consistency at Scale — Caching, CDNs, Eventual Consistency, and the CAP Theorem in Plain English

The morning the results came out

Meghana had been at the company in Chennai for a little over two years when results morning happened. The company ran the online results portals for several state education boards. When a board declared its Class 12 results, students went to the portal, entered their roll number, and saw their marks. Meghana had built this year's results-lookup service. A student enters a roll number. The service queries the database. The result comes back. Simple, and she had built it well.

She had tested it thoroughly. Every roll number returned the correct marks. Invalid roll numbers returned a clean error. She had run it on the staging environment with simulated load and it held up. Her manager signed off. The board announced that Class 12 results would be declared at 10:00 AM on a Tuesday.

At 10:00 AM on that Tuesday, approximately 1.2 million students entered their roll numbers within the first few minutes. Most of them, having waited weeks in a particular kind of anxiety that anyone who has been an Indian student understands in their body, hit refresh more than once. The database, receiving a flood of near-identical read queries far beyond anything staging had simulated, slowed, then queued, then stopped responding. The portal was effectively down at the precise moment in the year it mattered most. The support phone lines lit up. A news channel ran a ticker.

A senior engineer, calm in the way the seniors in this book tend to be, told Meghana to put a cache in front of the database. She did. Within fifteen minutes the portal came back to life, and it was not just working, it was fast. For about twenty minutes she felt the specific relief of a fire going out.