Monitoring

Prometheus metrics#

All metrics are exposed at GET /metrics on the admin port (default :9000) in Prometheus text format. No authentication is required to scrape this endpoint.

Traffic (RED)#

cache_result values: HIT, MISS, STALE, REVALIDATED, BYPASS.

source values: hot (hot in-memory tier), warm (warm disk-backed tier), peer (cluster peer via peer-fetch), origin (fetched from upstream, including errors and write-through proxy).

upstream_pool label: the matched route’s upstream pool name (a small config-bounded set — “which upstream is slow or 5xx-ing”). Pool-less routes (static files, catch-all) and unmatched traffic land on _default. The dashboard rings keep per-route attribution, so per-route views lose nothing.

v0.5.8 changes: the method label was dropped from the data-plane RED metrics (no dashboard or SLO query used it; the access log keeps the method), and the duration histogram dropped its source axis — the histogram footprint shrank ~90 %. Update any dashboard queries that filtered on method or source.

v0.5.13 update: the 2.5 s, 5 s, and 10 s tail buckets were restored on bouine_request_duration_seconds and bouine_peer_fetch_duration_seconds (the v0.5.8 1 s top bucket made a 1.01 s miss indistinguishable from a 30 s miss via PromQL). Series count per tuple grows 13 → 16 and stays well under the cardinality budget.

Hit ratio (PromQL):

sum(rate(bouine_requests_total{cache_result="HIT"}[1m]))
/
sum(rate(bouine_requests_total[1m]))

Error rate (PromQL):

sum(rate(bouine_requests_total{status=~"5.."}[1m]))
/
sum(rate(bouine_requests_total[1m]))

Per-pool error rate (PromQL) — the upstream_pool label answers which upstream is failing:

sum by (upstream_pool) (rate(bouine_requests_total{status=~"5.."}[5m]))

Native histogram cardinality#

Since v0.5.8, bouine_request_duration_seconds is registered with native-histogram support (client_golang dual representation), and since v0.5.18 every duration histogram has it — cloudflare_purge, startup, peer_fetch, warm_compaction, wal_write, and origin_request_duration included. Classic _bucket/_sum/_count series stay on the wire for layout-agnostic consumers, and the sparse-bucket native form lets Grafana Cloud / Mimir histogram_quantile work without materializing bucket series server-side. Resolution factor 1.1, capped at 80 sparse buckets, 1 h minimum reset window. None of the converted histograms are on the cache-hit path, so the zero-alloc hit-path budget is unchanged.

The native form does not reduce scrape cardinality by itself — the win requires dropping the classic _bucket series server-side. Add to your scrape config for bouine pods:

metric_relabel_configs:
  - action: drop
    regex: bouine_request_duration_seconds_bucket
    source_labels: [__name__]

Keep _sum/_count (average latency) and the native histogram (all quantiles). With the Helm chart, pass the same rule via serviceMonitor.metricRelabelings. See the native-histogram runbook for cost numbers and rollback.

Hot-tier cache storage#

Cache utilisation (PromQL):

bouine_hot_store_bytes / bouine_hot_store_max_bytes

Note. bouine_hot_store_max_bytes (and bouine_warm_store_max_bytes) are gauges exported since v0.5.1 — compute fill ratios directly. On older versions, use the configured hot_max_bytes value instead.

Streaming and load shedding#

Warm-tier storage#

Security#

Cluster#

Startup#

Cloudflare propagation#

Refresh before expiry#

These metrics are emitted when refresh_before_expiry is enabled on at least one route. All carry a route label matching the route name.

Refresh rate (PromQL):

sum(rate(bouine_refresh_total[1m])) by (route)

Refresh skip ratio (PromQL):

sum(rate(bouine_refresh_skips_total{reason="below_min_hits"}[5m]))
/
sum(rate(bouine_refresh_total[5m]))

A high below_min_hits skip ratio indicates the popularity gate (refresh_min_hits) is filtering out many objects — adjust the threshold if too many popular objects are expiring.

Go runtime#

The standard go_* and process_* metrics from the Prometheus Go client are automatically included. The most operationally relevant:

Access logs#

bouine emits a structured JSON access log line to stdout for every request.

{
  "time":         "2026-06-07T12:00:00Z",
  "level":        "INFO",
  "msg":          "access",
  "method":       "GET",
  "host":         "example.com",
  "path":         "/posts/hello/",
  "proto":        "HTTP/1.1",
  "status":       200,
  "bytes_out":    15234,
  "dur_ms":       1,
  "cache_status": "HIT",
  "remote":       "10.42.0.1:54321"
}

⚠️ Access log sampling — use Prometheus for throughput#

200 OK responses are sampled at 1:100. All other status codes (errors, redirects, unusual 2xx) are always logged.

This is intentional: at high RPS the log write would otherwise dominate the hot path. The trade-off is that you cannot compute accurate hit ratio or request rate from logs alone. Use bouine_requests_total in Prometheus for throughput and cache result distribution. Use access logs for error diagnosis and per-request debugging.

Rate accuracy from logs:  ❌ (1:100 sampling for 200s)
Error investigation:      ✅ (all non-200 always logged)
Slow request debugging:   ✅ (dur_ms present on sampled 200s)

Distributed tracing (OpenTelemetry)#

When tracing.endpoint is set, bouine exports OTLP/HTTP spans to any OpenTelemetry-compatible backend (Grafana Tempo, Jaeger, Honeycomb, etc.).

Endpoint format#

tracing.endpoint accepts either a bare host:port string or a full URL with http:// / https:// scheme. The scheme prefix is stripped automatically; WithInsecure() is used for plain HTTP.

# canonical form (host:port)
tracing:
  endpoint: "otel-collector.monitoring.svc.cluster.local:4318"
  service_name: "bouine"
  sampling_rate: 0.1

# also accepted (scheme is stripped)
tracing:
  endpoint: "http://otel-collector.monitoring.svc.cluster.local:4318"

Leave endpoint empty (the default) to disable tracing at zero overhead.

Trace structure#

Each request produces a nested span tree:

bouine.pipeline      (L2 — route matching, metrics, access log; one span per request)
  bouine.origin      (L5 — upstream fetch, miss/revalidate path only)

The bouine.origin span carries W3C TraceContext headers (traceparent, tracestate) injected into the upstream request, so the origin server can continue the trace if it also exports spans.

Since v0.5.13, the admin API also emits a bouine.admin server span for invalidation calls (POST /v1/purge, /v1/ban, /v1/refresh), joining the caller’s trace via the propagated W3C traceparent — so a distributed trace initiated by an invalidation service no longer stops at the client span.

Correlating traces with slow requests#

The data-plane middleware produces a single bouine.pipeline span per request, and bouine.origin spans (miss/revalidate path only) carry the W3C trace context forward to the origin. To jump from a slow access-log line to its trace, correlate the request timestamp and URL in your trace backend. Prometheus exemplars were removed in the v0.5.0 fasthttp migration — the histogram no longer carries per-observation trace IDs.

Admin API#

Grafana dashboard#

An official RED dashboard JSON is shipped in the bouine repository at deploy/grafana/bouine-red.json. Import it into Grafana via Dashboards → Import → Upload JSON file.

The dashboard covers five rows:

Recommended alert rules#

groups:
  - name: bouine
    rules:

    # ── Traffic ───────────────────────────────────────────────────────────
    - alert: BouineHighErrorRate
      expr: |
        sum(rate(bouine_requests_total{status=~"5.."}[5m]))
        /
        sum(rate(bouine_requests_total[5m])) > 0.01
      for: 5m
      labels: { severity: critical }
      annotations:
        summary: "bouine 5xx error rate > 1%"

    - alert: BouineHighStaleRate
      expr: |
        sum(rate(bouine_requests_total{cache_result="STALE"}[5m]))
        /
        sum(rate(bouine_requests_total[5m])) > 0.10
      for: 10m
      labels: { severity: warning }
      annotations:
        summary: "Stale serve ratio > 10% — possible origin outage"

    # ── Storage ───────────────────────────────────────────────────────────
    - alert: BouineHighEvictionRate
      expr: rate(bouine_hot_store_evictions_total[5m]) > 100
      for: 10m
      labels: { severity: warning }
      annotations:
        summary: "High SIEVE eviction rate — working set may exceed hot_max_bytes"

    # ── GC / Runtime ──────────────────────────────────────────────────────
    - alert: BouineGCPauseHigh
      expr: |
        max(rate(go_gc_duration_seconds_sum[1m])
            / rate(go_gc_duration_seconds_count[1m])) > 0.01
      for: 5m
      labels: { severity: warning }
      annotations:
        summary: "GC average pause > 10 ms — raise GOMEMLIMIT (see troubleshooting)"

    # ── Cluster ───────────────────────────────────────────────────────────
    - alert: BouineClusterModeMismatch
      expr: count(count by (mode) (bouine_cluster_mode_info == 1)) > 1
      for: 2m
      labels: { severity: critical }
      annotations:
        summary: "Pods running different cluster modes — configuration drift"

    - alert: BouinePeerFetchVariantMismatch
      expr: sum(rate(bouine_peer_fetch_variant_mismatch_total[5m])) > 0
      for: 10m
      labels: { severity: warning }
      annotations:
        summary: "Peer-fetch variant rejections — mixed-version fleet or a peer serving wrong-variant content"

    - alert: BouinePeerFetchShedding
      expr: sum(rate(bouine_peer_fetch_shed_total[5m])) > 0
      for: 5m
      labels: { severity: warning }
      annotations:
        summary: "Peer-fetch concurrency slots saturated — raise cluster.peer_fetch_concurrency"