Payment Hub — Two-Level Cache

Problem

Reference and configuration data was read constantly on the request path and changed rarely. Every read reached the database. That is wasteful at best, and at worst it makes the database a single point of failure for traffic that did not actually need it.

A single shared Redis layer would have removed most of the load — but not the per-lookup network hop, and not the failure mode where Redis being unreachable takes the service with it.

Architecture

   request
    │
    ▼
┌─────────────┐   hit
│ Local (JCS) │─────────────► value
│  in-process │
└──────┬──────┘
     │ miss
     ▼
┌─────────────┐   hit
│    Redis    │─────────────► value ──► populate local
│   shared    │
└──────┬──────┘
     │ miss
     ▼
┌─────────────┐
│  Database   │──► value ──► populate Redis + local
└─────────────┘

invalidation:  change ──► Kafka topic ──► all instances evict

Why two levels rather than one

The two levels answer different questions.

Local (JCS) removes the network entirely for the hottest keys. It is small, short-TTL, and per-instance — which means it is allowed to be slightly stale and that is a deliberate trade, not a bug.

Redis gives every instance a shared, warm view. A newly started or newly scaled instance does not stampede the database to fill its own cache; it fills from Redis.

The database is the only source of truth, and it is reached on a cold miss.

Invalidation

Two-level caching fails on invalidation, not on lookup. A local cache in N instances means a write has to reach N processes.

Changes publish to a Kafka topic; every instance consumes it and evicts the affected keys. Kafka rather than a Redis pub/sub fan-out because it gives ordering per key, replay for an instance that was briefly down, and an audit trail of what invalidated when — which matters when you are debugging a stale read after the fact.

Local TTLs stay short regardless, so a missed message degrades into brief staleness rather than indefinite wrongness. Belt and braces, on purpose.

Result

Hot-path database reads dropped substantially, lookup latency became dominated by in-process access rather than network, and the service now survives a Redis outage in a degraded-but-serving state instead of failing.