Somewhere in a payments-adjacent service I worked on, a customer got charged twice for the same order. Nobody wrote a bug that says “charge the customer twice.” What happened was worse, in the sense that it was more instructive: everything worked exactly as designed, and the design was wrong.
What actually happened
A Celery task called out to a payment provider, the provider processed the charge and started sending back a response, and the connection dropped before our side received it. From our task’s point of view, that’s indistinguishable from the request never arriving. Celery’s retry policy — sensible on its face — kicked in and resent the same task. The provider processed a second, entirely valid charge, because as far as it was concerned it received two distinct requests.
The failure isn’t in Celery, and it isn’t in the payment provider. It’s in treating “network call didn’t return” as “the operation didn’t happen.” Those are different facts, and at any real scale you will eventually get a timeout on a request that actually succeeded server-side.
The fix isn’t “retry less”
The instinct is to turn down retries or add more careful error handling around the HTTP call. That doesn’t solve anything — it just changes how often you get unlucky. The actual fix was making the operation idempotent from the caller’s side, not the network’s:
- Every charge request carries a client-generated idempotency key (a UUID tied to the order + attempt reason, not just a random value — you want retries of the same logical operation to reuse the key, and new operations to get a new one).
- The payment provider’s API accepts that key and guarantees that N requests with the same key produce exactly one charge, returning the original result for duplicates.
- On our side, the Celery task is written so it’s safe to run twice: it looks up whether a charge record already exists for that idempotency key before doing anything, and if Celery redelivers the task (which it will, under retry or worker-restart conditions), the second execution is a no-op that just returns the existing result.
That last point matters as much as the API-level idempotency key. Celery’s “at-least-once” delivery guarantee means your task body has to assume it might run more than once for the same logical unit of work, independent of anything the downstream API does.
Where else this bites you
Once you start looking for it, “the caller can’t tell if the write happened” shows up everywhere retries exist: webhook handlers (the sender will redeliver on any non-2xx, including a 2xx your load balancer failed to relay), background job queues, even client-side “submit” buttons on a flaky mobile connection. The pattern that generalizes:
- Give every write operation a natural or explicit idempotency key.
- Make the receiving side check-before-act on that key, not just accept-and-process.
- Assume delivery is at-least-once everywhere, because it almost always is once more than one process is involved.
None of this is exotic — it’s standard distributed systems advice. What made it click for our team wasn’t reading about it, it was tracing a duplicate charge back through three systems that were each behaving correctly.