Clustering

Cluster mode lets multiple bouine pods share cache reads, broadcast invalidations, and reduce origin load.

Choosing a mode#

listen:
  cluster: ":8443"

cluster:
  mode: strong        # "strong" (default) | "eventual"

Use this decision guide:

• Memory-constrained or large cluster (3–50+ nodes)? → strong — one copy per key, peer-fetch on miss.
• Geo-distributed or CDN edge PoPs? → eventual — independent caching, gossip-only invalidation, zero miss-latency penalty.

Cluster config#

listen:
  cluster: ":8443"

cluster:
  mode: strong         # "strong" (default) | "eventual"
  join:
    - "bouine-0.bouine-headless.default.svc.cluster.local:8443"
    - "bouine-1.bouine-headless.default.svc.cluster.local:8443"
    - "bouine-2.bouine-headless.default.svc.cluster.local:8443"
  hop_limit: 2                      # only used in strong mode
  peer_max_conns_per_host: 8        # pipelined peer connections (8 × 16 pending = 128 fetches/peer)
  peer_max_idle_conn_duration: 120s # MUST stay below admin.idle_timeout (default 300s)
  peer_fetch_concurrency: 64        # concurrent peer-fetch/put RPCs per node (1–128; default 64 since v0.6.0)
  peer_tcp_fast_open: true          # outbound TFO on peer dials (opt-in, Linux only, since v0.5.29)
  ban_ttl: 24h                      # how long invalidation bans stay active (≥ 1s when set)

The peer_max_idle_conn_duration ↔ admin.idle_timeout ordering matters: peer RPCs ride keep-alive connections pooled by the client. If the admin server reaps an idle connection before the client closes it, the next peer-fetch or peer-put fails with EOF and falls back to origin, spiking latency. Config validation rejects any explicit combination that violates the ordering.

Two sizing knobs matter under strong-mode load (both added in v0.5.11):

• peer_fetch_concurrency (default 64 since v0.6.0 — previously 16 since v0.5.29, originally 4 — range 1–128) bounds concurrent peer-fetch/peer-put RPCs per node. In strong mode most cache hits are peer hits, so this semaphore sits on the hot path — raise it together with peer_max_conns_per_host so the pipeline clients stay the binding constraint. The default was raised again in v0.6.0 because too small a pool queues peer hits behind in-flight fetches and adds tail latency on strong-mode clusters. Excess callers wait up to 100 ms for a slot, then shed to the slow path / origin fallback.
• peer_tcp_fast_open (default off, since v0.5.29) enables outbound TCP Fast Open on peer dials — with a kernel-cached TFO cookie the first RPC bytes ride in the SYN, saving one RTT on every fresh peer connection. Opt-in and Linux-only; bouine↔bouine dials use TFO end-to-end when the answering side also enables listen.tcp_fast_open.
• ban_ttl (default 24h, ≥ 1s when set; added in v0.5.20) bounds how long invalidation bans stay in the active ban list. See Cache invalidation.

On Kubernetes, gossip peer discovery requires a headless Service:

apiVersion: v1
kind: Service
metadata:
  name: bouine-headless
spec:
  clusterIP: None
  publishNotReadyAddresses: true
  selector:
    app: bouine
  ports:
    - name: cluster-tcp
      port: 8443
      protocol: TCP
    - name: cluster-udp
      port: 8443
      protocol: UDP

Required: publishNotReadyAddresses: true — without it, StatefulSet pod DNS may not resolve during startup and gossip will fail to form a cluster.

For inter-node mTLS, see TLS → Cluster TLS.

Strong mode (default)#

A consistent-hash ring (256 virtual nodes per node) determines which node owns each cache key.

Request flow:

1. Node receives a request, computes the cache key.
2. Looks up key in local store → HIT returns immediately.
3. MISS: if the key is owned by a peer, forwards a POST /v1/peer/fetch RPC to the owner.
4. Peer HIT: object returned and promoted to local hot tier.
5. Peer MISS or error: falls through to origin.

Typical peer-fetch latency: ~0.5–2 ms on the same datacenter LAN.

Invalidation: Purge and ban are delivered via HTTP fan-out to all live peers (sub-second). A secondary gossip broadcast provides redundant delivery. Refresh is forwarded to the key’s owner node only. Since v0.5.26, data-plane invalidations (POST/PUT/DELETE per RFC 9111 §4.4) also fan out through the same batching pipeline as the admin purge API — a write landing on a non-owner node no longer leaves the owner serving stale content until TTL. The data-plane hook is enqueue-only (10 ms coalescing bound), so the proxied response is never blocked on peer fan-out.

Anti-entropy: Nodes exchange ring digests on every gossip push/pull cycle. If a peer was unreachable during a rolling restart, it is automatically re-added to the ring when digests diverge.

Wire format (since v0.5.22): memberlist node metadata (peer info) and push/pull state (ring digests) are binary frames (magic + version header); receivers reject unversioned or unknown-version frames, and peer-fetch requests only accept the binary v2 format. Mixed-version clusters running a pre-v0.5.22 build will fail to exchange meta/state during a rolling upgrade — upgrade all nodes together. Also since v0.5.22 the admin API and Go SDK parse request bodies with encoding/json/v2: duplicate JSON keys and misspelled (case-mismatched) field names are rejected with 400 instead of silently accepted.

Peer-fetch resilience (v0.5.17–v0.5.29)#

The peer-fetch machinery is hardened against rolling restarts, HPA scale-down, and miss storms:

• Background reconcile pass (default every 30s) re-derives the peer set from memberlist’s live member view, healing missed leave/update notifications after a peer restart.
• Per-address failure breaker — three consecutive transport failures blacklist an address for a 30s cooldown; Fetch/Put return immediately and the handler falls back to origin. Tripped addresses surface in the bouine_peer_addr_blacklisted gauge.
• Bounded RPC budgets — all peer-fetch/put RPCs are capped at a 500 ms budget and the peer dial timeout is 200 ms, so dead peers fail fast instead of pinning fetch slots.
• Bounded queue wait + shedding — callers waiting for a peer_fetch_concurrency slot shed after 100 ms (bouine_peer_shed_total, with a direction label splitting fetch from put RPCs since v0.5.29); the queue wait itself is observed in bouine_peer_fetch_queue_wait_seconds. Since v0.5.29 a shed fast-path peer fetch carries a hint that suppresses the slow path’s duplicate peer RPC — the request goes straight to origin instead of re-queueing behind the same saturated semaphore.
• Background re-warm — a shed foreground miss schedules a bounded background refill (bouine_rewarm_fill_total), so a shed miss no longer loses the cache refill and miss storms converge back to hits.
• Config-derived Vary resolution on the cold-miss peer path (since v0.6.2, ADR-0071). On a route whose cache.key policy alone determines the Vary list (include_headers / cookie_presence), both the H1 fast path and the slow path now derive the variant key locally and ask the variant’s owner directly — one admin-port RPC instead of two. A wrong derivation falls back to the two-step; bouine_peer_fetch_derived_fallback_total counts the fallback so the degraded mode announces itself.
• Stale objects served through a peer are refreshed (since v0.6.2). In strong mode a non-owner serving the owner’s stale copy now starts the background revalidation itself and forwards the fresh object to the owner, so the object no longer stays stale for the whole SWR window.

Eventual mode#

Every node is independent — no sharding, no peer-fetch. Each node caches whatever it receives from origin.

Request flow:

1. Node receives a request, looks up in local store.
2. HIT → returns immediately. MISS → fetches from origin directly (no peer hop).

Invalidation: Purge, ban, and refresh are delivered exclusively via gossip. Convergence window: 1–5 seconds. Stale reads are possible during convergence. Data-plane invalidations (POST/PUT/DELETE) broadcast the same way (since v0.5.26). Since v0.5.26, gossip batch frames larger than the ~1.4 KiB UDP gossip window are split into standalone sub-frames at flush time (previously a full 256-event batch could never fit a gossip round and was re-queued forever, silently dropping the batch — issue #754); bouine_cluster_gossip_oversized_drops_total counts frames that could never fit and are dropped defensively.

When to use:

• CDN edge deployments where each PoP operates independently.
• Geo-distributed clusters where cross-region latency makes peer-fetch costly.
• Deployments where slightly stale reads are acceptable in exchange for zero miss latency.

Invalidation propagation summary#

Switching modes#

Mode changes require a full cluster restart (rolling restart recommended).

1. Update cluster.mode in your ConfigMap.
2. Rolling restart all pods: kubectl rollout restart statefulset/bouine.
3. Verify: curl -s http://127.0.0.1:9000/metrics | grep bouine_cluster_mode_info.

Cache state is not preserved across mode switches — nodes start with empty caches. Expect elevated miss rates for the first few minutes.

See the cluster mode operations page for detailed verification and troubleshooting procedures per mode.

Debugging peers#

kubectl exec bouine-0 -n bouine -- /bouine cluster peers
# or via the admin API:
curl http://localhost:9000/v1/cluster/peers

Should show every pod in the StatefulSet with addr set to the pod IP (not 0.0.0.0).