Shipping Software — Version Control, the Pipeline, Environments, and the Small Disciplines That Decide Whether Shipping Is Routine or Terrifying

The Friday the mandi froze

Vikram had been at the agritech company in Nagpur for a little over a year when he froze the platform on a Friday evening. The company ran a marketplace that connected farmers across Vidarbha with buyers and showed live mandi prices. Its busiest window was the late afternoon, when the day's trading wound down and farmers and traders checked closing prices and confirmed deals before going home.

The change Vikram needed to ship was small and, importantly, correct. A new column on the listings table to store a quality grade for produce. He had written it carefully. He had tested it on his machine. The code was good. There was nothing wrong with the code.

It was 6:10 PM on a Friday. He wanted to get it in before the weekend. The change included a database migration that added the new column as NOT NULL. On his laptop, against a table with a few hundred test rows, the migration ran instantly. He merged it and triggered the deploy. He started packing his bag.

In production, the listings table had several million rows. Adding a NOT NULL column to it the way the migration was written required the database to lock the table while it rewrote every row. For the duration of that operation, no farmer could see a price and no buyer could confirm a deal. The platform was frozen for twenty-five minutes, during the single busiest window of the week, with no obvious way for Vikram to make it stop, because a migration that has begun rewriting a table is not something you cleanly cancel halfway.