APIs and Integration — REST, GraphQL, gRPC, Webhooks, Queues, Idempotency, Retries

The Wednesday that shipped twice

Nikhil had been at the two-person logistics startup in Indore for eight months when the Wednesday happened. The startup was building a delivery tracking app for small businesses, the kind of company that ships a few dozen packages a week and cannot afford an enterprise logistics platform. The app pulled shipment status from three different courier partners — DTDC, Delhivery, and a smaller local courier whose API Nikhil's co-founder had personally negotiated access to.

Two of the three integrations worked fine. The third, the local courier, was unreliable. The API would sometimes return stale data, sometimes return nothing for thirty seconds and then a 500 error, sometimes work perfectly. Nikhil's co-founder had told him, casually, "just retry it." Nikhil wrote a retry loop. Three attempts, with a small delay between each. The app appeared to work for a week.

The Wednesday started with a customer email. A small electronics shop in Bhopal had received three separate SMS notifications saying the same package had been delivered. Their warehouse had also received an automated message asking why the same shipment was being re-dispatched. Nikhil traced the bug. His retry loop was hitting an endpoint on the local courier's API that updated the delivery status on every call, not just returned it. Every retry of what he thought was a status-check was being recorded by the courier's system as a fresh delivery event.

He had retried a request that was not safe to retry. The fix would take him most of the day. The lesson would take him longer.