Clustering

Cluster mode lets multiple bouine pods share cache reads, broadcast invalidations, and reduce origin load.

Choosing a mode#

listen:
  cluster: ":8443"

cluster:
  mode: strong        # "strong" (default) | "eventual"

Use this decision guide:

• Memory-constrained or large cluster (3–50+ nodes)? → strong — one copy per key, peer-fetch on miss.
• Geo-distributed or CDN edge PoPs? → eventual — independent caching, gossip-only invalidation, zero miss-latency penalty.

Cluster config#

listen:
  cluster: ":8443"

cluster:
  mode: strong         # "strong" (default) | "eventual"
  join:
    - "bouine-0.bouine-headless.default.svc.cluster.local:8443"
    - "bouine-1.bouine-headless.default.svc.cluster.local:8443"
    - "bouine-2.bouine-headless.default.svc.cluster.local:8443"
  hop_limit: 2         # only used in strong mode

On Kubernetes, gossip peer discovery requires a headless Service:

apiVersion: v1
kind: Service
metadata:
  name: bouine-headless
spec:
  clusterIP: None
  publishNotReadyAddresses: true
  selector:
    app: bouine
  ports:
    - name: cluster-tcp
      port: 8443
      protocol: TCP
    - name: cluster-udp
      port: 8443
      protocol: UDP

Required: publishNotReadyAddresses: true — without it, StatefulSet pod DNS may not resolve during startup and gossip will fail to form a cluster.

For inter-node mTLS, see TLS → Cluster TLS.

Strong mode (default)#

A consistent-hash ring (256 virtual nodes per node) determines which node owns each cache key.

Request flow:

1. Node receives a request, computes the cache key.
2. Looks up key in local store → HIT returns immediately.
3. MISS: if the key is owned by a peer, forwards a POST /v1/peer/fetch RPC to the owner.
4. Peer HIT: object returned and promoted to local hot tier.
5. Peer MISS or error: falls through to origin.

Typical peer-fetch latency: ~0.5–2 ms on the same datacenter LAN.

Invalidation: Purge and ban are delivered via HTTP fan-out to all live peers (sub-second). A secondary gossip broadcast provides redundant delivery. Refresh is forwarded to the key’s owner node only.

Anti-entropy: Nodes exchange ring digests on every gossip push/pull cycle. If a peer was unreachable during a rolling restart, it is automatically re-added to the ring when digests diverge.

Eventual mode#

Every node is independent — no sharding, no peer-fetch. Each node caches whatever it receives from origin.

Request flow:

1. Node receives a request, looks up in local store.
2. HIT → returns immediately. MISS → fetches from origin directly (no peer hop).

Invalidation: Purge, ban, and refresh are delivered exclusively via gossip. Convergence window: 1–5 seconds. Stale reads are possible during convergence.

When to use:

• CDN edge deployments where each PoP operates independently.
• Geo-distributed clusters where cross-region latency makes peer-fetch costly.
• Deployments where slightly stale reads are acceptable in exchange for zero miss latency.

Invalidation propagation summary#

Switching modes#

Mode changes require a full cluster restart (rolling restart recommended).

1. Update cluster.mode in your ConfigMap.
2. Rolling restart all pods: kubectl rollout restart statefulset/bouine.
3. Verify: curl -s http://127.0.0.1:9000/metrics | grep bouine_cluster_mode_info.

Cache state is not preserved across mode switches — nodes start with empty caches. Expect elevated miss rates for the first few minutes.

See the cluster mode operations page for detailed verification and troubleshooting procedures per mode.

Debugging peers#

kubectl exec bouine-0 -n bouine -- /bouine cluster peers
# or via the admin API:
curl http://localhost:9000/v1/cluster/peers

Should show every pod in the StatefulSet with addr set to the pod IP (not 0.0.0.0).