Billing Invoice Engine

Problem

Billing file generation was vertically bound: one process, one machine, and a run time that grew with the customer base. The only lever was a bigger box, and that lever runs out.

The requirement was not just “faster” — it was capacity you can add, and a run that does not restart from zero when one shard fails.

Architecture

┌────────────┐
│   Master   │  partition · assign · track · assemble
└──────┬─────┘
     │  claims (atomic)
 ┌───┴────┬─────────┬─────────┐
 ▼        ▼         ▼         ▼
┌──────┐ ┌──────┐ ┌──────┐ ┌──────┐
│ W-1  │ │ W-2  │ │ W-3  │ │ W-n  │
└──┬───┘ └──┬───┘ └──┬───┘ └──┬───┘
 │        │        │        │
 └────────┴────┬───┴────────┘
               ▼
       ┌───────────────┐
       │  RAW segments │
       └───────┬───────┘
               ▼
        assembled output

Design decisions

Shards are claimed, not pushed. Workers atomically claim the next unclaimed shard. That makes the worker count a runtime variable — start three or thirty, the master does not change — and it removes the master as a scheduling bottleneck.

Every shard is idempotent. A shard writes to its own segment, so re-running one is safe. This is the property that makes the whole thing operable: a stuck worker is a shard to reclaim, not an incident.

Assembly is separate from generation. Segments are concatenated in a final ordered pass. Keeping generation order-free is what allows it to be parallel at all.

Leases, not heartbeats-only. A claim carries an expiry. A worker that dies silently releases its shard by timing out, without needing a supervisor to notice.

Result

Generation time became a function of worker count rather than of machine size, and the run became restartable at shard granularity. Capacity planning turned into an arithmetic question instead of a procurement one.