Dépannage

Quick reference#

Cluster discovery#

Pods do not discover each other#

Check the headless Service:

kubectl get svc bouine-headless -n bouine -o yaml | grep publishNotReadyAddresses

Must be publishNotReadyAddresses: true. Without it, DNS does not resolve during pod startup and gossip fails.

Check peer list:

kubectl exec bouine-0 -n bouine -- /bouine cluster peers

If each pod only sees itself, check join logs:

kubectl logs bouine-1 -n bouine | grep -i 'join\|cluster'

Rolling restart produces 503/502#

See Kubernetes operations for the full zero-5xx rolling update procedure.

HIT p99 spikes to 50–100 ms under load#

Symptom#

X-Cache: HIT responses have p99 latency of 50–100 ms even though the cache is warm and the origin is healthy. The spike is reproducible under concurrent load and persists regardless of cluster mode. In Prometheus:

histogram_quantile(0.99,
  rate(bouine_request_duration_seconds_bucket{cache_result="HIT"}[1m]))

Root cause: Go GC stop-the-world pauses#

The most common cause is the Go runtime’s garbage collector running too aggressively because GOMEMLIMIT is set too low relative to actual RSS.

Confirm with:

# Check GC worst-case pause and the configured GOMEMLIMIT
curl -s http://127.0.0.1:9000/metrics | grep -E 'go_gc_duration|go_gc_gomemlimit'

go_gc_duration_seconds{quantile="1"}  0.095   ← worst-case pause ~95 ms
go_gc_gomemlimit_bytes                7.55e+07 ← GOMEMLIMIT = 72 MiB

If go_gc_duration_seconds{quantile="1"} is in the same range as your observed HIT p99, GC pauses are the cause. This typically happens when:

• GOMEMLIMIT is set well below the pod memory limit.
• The hot cache is near or exceeding GOMEMLIMIT — forcing the GC to run almost continuously and occasionally triggering a long STW cycle to reclaim enough memory to stay under the limit.

Fix: tune GOMEMLIMIT to 85 % of the pod memory limit#

Set GOMEMLIMIT to approximately 85 % of resources.limits.memory so the GC has headroom to collect lazily rather than continuously.

Kubernetes StatefulSet / Deployment:

env:
  - name: GOMEMLIMIT
    value: "82MiB"   # 85% of a 96Mi pod limit
  - name: GOGC
    value: "100"     # default; keep paired with GOMEMLIMIT

Helm chart (values.yaml):

goMemLimit: "3GiB"   # 85% of resources.limits.memory

Rule of thumb:

After the change, verify that go_gc_duration_seconds{quantile="1"} drops to < 5 ms under the same load. If the pod’s RSS still exceeds GOMEMLIMIT at peak, consider also increasing hot_max_bytes or the pod memory limit.

Other causes of HIT p99 spikes#

If GC pauses are small but HIT p99 is still elevated, check:

Cache misses#

X-Cache is always MISS#

Check:

1. The response is cacheable (Cache-Control is not no-store, private, max-age=0).
2. The route actually matches — verify with access logs (route field).
3. The origin returns a cacheable status (usually 200).
4. Request headers are not producing many variants — check bouine_vary_cap_hits_total.

curl -sI http://127.0.0.1:8080/path
kubectl logs statefulset/bouine -n bouine | grep cache_status

Purge does not propagate across cluster#

In strong mode#

• Check bouine_cluster_invalidations_http_total{type="purge"}. If zero, the admin port may be unreachable. Verify cluster.tls config and network policies.
• The gossip broadcast queue provides a secondary delivery path (check bouine_cluster_invalidations_gossip_total).

In eventual mode#

• Gossip-only convergence takes 1–5 s. Stale reads are expected during this window.
• Check peer list: curl -s http://127.0.0.1:9000/v1/cluster/peers. Should show 3+ nodes.
• The headless Service must have publishNotReadyAddresses: true.

Stale reads#

• In eventual mode: expected during gossip convergence (1–5 s). If persistent, check bouine_cluster_invalidations_gossip_total — if flat, the gossip link is broken. Restart the node.

Low hit rate in eventual mode#

Each node cold-starts independently. Over time, hit rate naturally plateaus. If load is unevenly distributed across nodes (e.g. session affinity), some nodes may have much lower hit rates. Consider strong mode.

Docker build is slow on Apple Silicon#

Use buildx and cross-compile rather than emulating amd64:

docker buildx build --platform linux/amd64 -t bouinecache/bouine:dev --load .

bouine’s Dockerfile uses BUILDPLATFORM and TARGETARCH so Go builds natively.

FD exhaustion and status-0 errors#

Symptom#

Under high connection load, clients receive responses with HTTP status 0 (connection closed without a response), and the pod approaches the file descriptor limit. In Prometheus:

process_max_fds - process_open_fds < 100

Root cause#

The fasthttp server’s concurrency and per-host connection pool can exhaust file descriptors when origin responses are slow and connections accumulate. As of v0.5.0, bouine caps fasthttp.Server.Concurrency and reduces MaxConnsPerHost to curb FD exhaustion. If you still see this under extreme load:

Fix#

1. Set listen.max_connections to bound the total concurrent data-plane connections. This directly limits FD usage.

2. Set upstream_pools[].connect.max_connections per pool to limit upstream connection count.

3. Set upstream_pools[].connect.response_header_timeout to abort slow-origin fetches that hold connections open.

4. Raise the pod FD limit if the working set genuinely requires more connections:

# values.yaml
extraVolumeMounts:
  - name: etc-security
    mountPath: /etc/security
# Or set in the container security context
securityContext:
  runAsNonRoot: true

# Verify current FD limits
kubectl exec bouine-0 -- cat /proc/1/limits | grep 'open files'

5. Check GOMEMLIMIT — the per-stream tee buffer is capped based on GOMEMLIMIT to prevent OOMKill under slow-origin conditions. If GOMEMLIMIT is too low, tee buffers are aggressively capped which can cause connection churn. See GC pauses troubleshooting.

Runbook quick reference#

For detailed procedures, see the operator runbooks in the bouine repository (docs/runbook/):

• 00-lifecycle — start, stop, reload, drain
• 10-cluster-modes — verify, diagnose, and switch modes
• 20-purge-ban — cache invalidation operations
• 30-rolling-restart — zero-5xx rolling restart