Monitoring

Prometheus metrics#

All metrics are exposed at GET /metrics on the admin port (default :9000) in Prometheus text format. No authentication is required to scrape this endpoint.

Traffic (RED)#

cache_result values: HIT, MISS, STALE, REVALIDATED, BYPASS.

source values: hot (hot in-memory tier), warm (warm disk-backed tier), peer (cluster peer via peer-fetch), origin (fetched from upstream, including errors and write-through proxy).

route label: the name field of the matched route config entry. Falls back to host:path_prefix when name is empty, or _default for unmatched requests.

Hit ratio (PromQL):

sum(rate(bouine_requests_total{cache_result="HIT"}[1m]))
/
sum(rate(bouine_requests_total[1m]))

Error rate (PromQL):

sum(rate(bouine_requests_total{status=~"5.."}[1m]))
/
sum(rate(bouine_requests_total[1m]))

Hot-tier cache storage#

Cache utilisation (PromQL):

bouine_hot_store_bytes / <hot_max_bytes_from_config>

Note. There is no bouine_hot_store_max_bytes metric — the configured maximum is a static config value, not a gauge. Use bouine_hot_store_bytes against the known hot_max_bytes config value for utilisation calculations.

Warm-tier storage#

Security#

Cluster#

Startup#

Cloudflare propagation#

Refresh before expiry#

These metrics are emitted when refresh_before_expiry is enabled on at least one route. All carry a route label matching the route name.

Refresh rate (PromQL):

sum(rate(bouine_refresh_total[1m])) by (route)

Refresh skip ratio (PromQL):

sum(rate(bouine_refresh_skips_total{reason="below_min_hits"}[5m]))
/
sum(rate(bouine_refresh_total[5m]))

A high below_min_hits skip ratio indicates the popularity gate (refresh_min_hits) is filtering out many objects — adjust the threshold if too many popular objects are expiring.

Go runtime#

The standard go_* and process_* metrics from the Prometheus Go client are automatically included. The most operationally relevant:

Access logs#

bouine emits a structured JSON access log line to stdout for every request.

{
  "time":         "2026-06-07T12:00:00Z",
  "level":        "INFO",
  "msg":          "access",
  "method":       "GET",
  "host":         "example.com",
  "path":         "/posts/hello/",
  "proto":        "HTTP/1.1",
  "status":       200,
  "bytes_out":    15234,
  "dur_ms":       1,
  "cache_status": "HIT",
  "route":        "/",
  "remote":       "10.42.0.1:54321"
}

⚠️ Access log sampling — use Prometheus for throughput#

200 OK responses are sampled at 1:100. All other status codes (errors, redirects, unusual 2xx) are always logged.

This is intentional: at high RPS the log write would otherwise dominate the hot path. The trade-off is that you cannot compute accurate hit ratio or request rate from logs alone. Use bouine_requests_total in Prometheus for throughput and cache result distribution. Use access logs for error diagnosis and per-request debugging.

Rate accuracy from logs:  ❌ (1:100 sampling for 200s)
Error investigation:      ✅ (all non-200 always logged)
Slow request debugging:   ✅ (dur_ms present on sampled 200s)

Distributed tracing (OpenTelemetry)#

When tracing.endpoint is set, bouine exports OTLP/HTTP spans to any OpenTelemetry-compatible backend (Grafana Tempo, Jaeger, Honeycomb, etc.).

Endpoint format#

tracing.endpoint accepts either a bare host:port string or a full URL with http:// / https:// scheme. The scheme prefix is stripped automatically; WithInsecure() is used for plain HTTP.

# canonical form (host:port)
tracing:
  endpoint: "otel-collector.monitoring.svc.cluster.local:4318"
  service_name: "bouine"
  sampling_rate: 0.1

# also accepted (scheme is stripped)
tracing:
  endpoint: "http://otel-collector.monitoring.svc.cluster.local:4318"

Leave endpoint empty (the default) to disable tracing at zero overhead.

Trace structure#

Each request produces a nested span tree:

bouine.listener.http   (L1 — network accept + protocol detection)
  bouine.pipeline      (L2 — route matching, metrics, access log)
    bouine.cache       (L4 — RFC 9111 state machine)
      bouine.origin    (L5 — upstream fetch, miss/revalidate path only)

The bouine.origin span carries W3C TraceContext headers (traceparent, tracestate) injected into the upstream request, so the origin server can continue the trace if it also exports spans.

Exemplars#

When tracing is active, bouine_request_duration_seconds observations carry a Prometheus exemplar with trace_id. In Grafana, click any histogram bar and select “Query with exemplar” to jump directly from a high-latency bucket to the matching trace in Tempo.

Admin API#

Grafana dashboard#

An official RED dashboard JSON is shipped in the bouine repository at deploy/grafana/bouine-red.json. Import it into Grafana via Dashboards → Import → Upload JSON file.

The dashboard covers five rows:

Recommended alert rules#

groups:
  - name: bouine
    rules:

    # ── Traffic ───────────────────────────────────────────────────────────
    - alert: BouineHighErrorRate
      expr: |
        sum(rate(bouine_requests_total{status=~"5.."}[5m]))
        /
        sum(rate(bouine_requests_total[5m])) > 0.01
      for: 5m
      labels: { severity: critical }
      annotations:
        summary: "bouine 5xx error rate > 1%"

    - alert: BouineHighStaleRate
      expr: |
        sum(rate(bouine_requests_total{cache_result="STALE"}[5m]))
        /
        sum(rate(bouine_requests_total[5m])) > 0.10
      for: 10m
      labels: { severity: warning }
      annotations:
        summary: "Stale serve ratio > 10% — possible origin outage"

    # ── Storage ───────────────────────────────────────────────────────────
    - alert: BouineHighEvictionRate
      expr: rate(bouine_hot_store_evictions_total[5m]) > 100
      for: 10m
      labels: { severity: warning }
      annotations:
        summary: "High SIEVE eviction rate — working set may exceed hot_max_bytes"

    # ── GC / Runtime ──────────────────────────────────────────────────────
    - alert: BouineGCPauseHigh
      expr: |
        max(rate(go_gc_duration_seconds_sum[1m])
            / rate(go_gc_duration_seconds_count[1m])) > 0.01
      for: 5m
      labels: { severity: warning }
      annotations:
        summary: "GC average pause > 10 ms — raise GOMEMLIMIT (see troubleshooting)"

    # ── Cluster ───────────────────────────────────────────────────────────
    - alert: BouineClusterModeMismatch
      expr: count(count by (mode) (bouine_cluster_mode_info == 1)) > 1
      for: 2m
      labels: { severity: critical }
      annotations:
        summary: "Pods running different cluster modes — configuration drift"