What AI Coding Agents Still Get Wrong About Distributed Systems
Agents write correct-looking code very fast. The failures I keep seeing are not syntax or API misuse — they are assumptions about time, ordering and retries.
I use coding agents daily and they have changed how much of a day goes into typing. What has not changed is where the bugs live. The failures I keep catching in review cluster in a few places, and they are all the same kind of mistake: local reasoning applied to a system that is not local.
1. Check-then-act, everywhere
Asked to prevent duplicates, the near-universal answer is a read followed by a write:
if (!repo.existsByReference(ref)) {
repo.save(payment); // two callers both get here
}It is correct single-threaded and it reads as obviously right. The version that holds is a unique constraint, with the violation treated as the duplicate signal rather than as an error to log. Agents produce this correctly when asked directly and rarely produce it unprompted.
2. Retries without idempotency
Retry logic arrives readily — usually with sensible backoff. What arrives much less often is any statement about whether the operation being retried is safe to repeat. For a payment, that difference is the whole problem.
3. Time as if it were a clock
Instant.now() compared across services, timeouts assumed to be honoured,
ordering inferred from timestamps. Distributed timing is the area where
generated code is most confidently wrong, because the wrong version looks
completely ordinary.
4. Optimising the wrong axis
Ask for a faster batch and you tend to get concurrency: threads, parallel streams, async. Often the real win is removing round-trips — batching the access, grouping related work — which is a smaller total workload rather than an overlapped one. Agents reach for the second reflexively and the first only when told.
5. Cache invalidation as a footnote
Caches come back well-formed on the read path and thin on the write path. Multi-instance eviction, ordering between two rapid updates, what happens to an instance that was down during the publish — those need to be asked for.
How I actually work with them
None of this is an argument against agents. It is an argument about where to spend attention.
- Specify the invariant, not the feature. “Two concurrent callers must not both create a payment for this reference” produces the unique constraint. “Prevent duplicates” produces the race.
- Ask for the failure mode. “What happens if this runs twice, out of order, or half-completes?” is the single highest-yield prompt I have.
- Review concurrency and I/O boundaries by hand. Everything else I read quickly. Those two I read properly.
- Make it write the adversarial test. Agents are good at generating the concurrent test that fails; they just do not volunteer it.
The leverage is real and it is large. It sits in implementation speed, not in judgment about correctness under concurrency — and that gap is currently the most valuable thing an experienced engineer brings to the pairing.