监控

Prometheus metrics#

All metrics are exposed at GET /metrics on the admin port (default :9000) in Prometheus text format. No authentication is required to scrape this endpoint.

Traffic (RED)#

cache_result values: HIT, MISS, STALE, REVALIDATED, BYPASS.

source values: hot (hot in-memory tier), warm (warm disk-backed tier), peer (cluster peer via peer-fetch), origin (fetched from upstream, including errors, write-through proxy, and — since v0.5.28 — BYPASS responses, which previously landed on an unnamed default series), bouine (responses bouine synthesized itself: the only-if-cached 504 and the shed 503 — the label was the empty string before v0.5.28). The wire X-Cache-Source header stays empty for bouine-sourced responses.

upstream_pool label: the matched route’s upstream pool name (a small config-bounded set — “which upstream is slow or 5xx-ing”). Pool-less routes (static files, catch-all) and unmatched traffic land on _default. The dashboard rings keep per-route attribution, so per-route views lose nothing.

traffic_class label (since v0.5.27): always present; values come exclusively from the configured metrics.traffic_classes set plus the unclassified fallback for requests matching no class. See Traffic classes.

v0.5.8 changes: the method label was dropped from the data-plane RED metrics (no dashboard or SLO query used it; the access log keeps the method), and the duration histogram dropped its source axis — the histogram footprint shrank ~90 %. Update any dashboard queries that filtered on method or source.

v0.5.13 update: the 2.5 s, 5 s, and 10 s tail buckets were restored on bouine_request_duration_seconds and bouine_peer_fetch_duration_seconds (the v0.5.8 1 s top bucket made a 1.01 s miss indistinguishable from a 30 s miss via PromQL). Series count per tuple grows 13 → 16 and stays well under the cardinality budget.

v0.5.27 series identity reset: adding the traffic_class label changes every data-plane series’ identity — expect one rate() gap window at the upgrade boundary.

v0.5.28 series identity reset: the source label’s default slot was named bouine (previously the empty string) and BYPASS traffic moved to origin — the former source="" series and the BYPASS default series move to their new labels, so rate() queries show one gap window at the deploy boundary.

Hit ratio (PromQL):

sum(rate(bouine_requests_total{cache_result="HIT"}[1m]))
/
sum(rate(bouine_requests_total[1m]))

Error rate (PromQL):

sum(rate(bouine_requests_total{status=~"5.."}[1m]))
/
sum(rate(bouine_requests_total[1m]))

Per-pool error rate (PromQL) — the upstream_pool label answers which upstream is failing:

sum by (upstream_pool) (rate(bouine_requests_total{status=~"5.."}[5m]))

Per-traffic-class hit ratio (PromQL) — requires metrics.traffic_classes configured:

sum by (traffic_class) (rate(bouine_requests_total{cache_result="HIT"}[5m]))
/
sum by (traffic_class) (rate(bouine_requests_total[5m]))

Traffic classes#

Since v0.5.27, metrics.traffic_classes (see Configuration) declares up to 8 named traffic populations matched by request-host patterns. The traffic_class label is added — always present, values exclusively from the configured set plus the unclassified fallback — to bouine_requests_total, bouine_request_duration_seconds, and bouine_response_bytes_total, enabling per-population hit-ratio, latency, and saved-bandwidth queries inside one instance. Access logs gain a matching traffic_class attribute. The label set is closed and spoof-proof by construction.

• Declaration order is precedence — first match wins. A host pattern in a later class fully shadowed by an earlier one (it can never match) is reported at Error level at boot; boot proceeds.
• Cardinality: the label multiplies the data-plane series by 1 + #classes. bouine_requests_total crosses a 10 000-series line when pools × (1 + #classes) > 57; the overage is opt-in (no configured classes means no multiplication), and the metric_relabel_configs mitigation in the native-histogram runbook applies.
• Zero allocations on the hit path.

Native histogram cardinality#

Since v0.5.8, bouine_request_duration_seconds is registered with native-histogram support (client_golang dual representation), and since v0.5.18 every duration histogram has it — cloudflare_purge, startup, peer_fetch, warm_compaction, wal_write, and origin_request_duration included. Classic _bucket/_sum/_count series stay on the wire for layout-agnostic consumers, and the sparse-bucket native form lets Grafana Cloud / Mimir histogram_quantile work without materializing bucket series server-side. Resolution factor 1.1, capped at 80 sparse buckets, 1 h minimum reset window. None of the converted histograms are on the cache-hit path, so the zero-alloc hit-path budget is unchanged.

The native form does not reduce scrape cardinality by itself — the win requires dropping the classic _bucket series server-side. Add to your scrape config for bouine pods:

metric_relabel_configs:
  - action: drop
    regex: bouine_request_duration_seconds_bucket
    source_labels: [__name__]

Keep _sum/_count (average latency) and the native histogram (all quantiles). With the Helm chart, pass the same rule via serviceMonitor.metricRelabelings. See the native-histogram runbook for cost numbers and rollback.

Hot-tier cache storage#

Cache utilisation (PromQL):

bouine_hot_store_bytes / bouine_hot_store_max_bytes

Note. bouine_hot_store_max_bytes (and bouine_warm_store_max_bytes) are gauges exported since v0.5.1 — compute fill ratios directly. On older versions, use the configured hot_max_bytes value instead.

Streaming and load shedding#

Warm-tier storage#

Security#

Cluster#

Startup#

Cloudflare propagation#

Refresh before expiry#

These metrics are emitted when refresh_before_expiry is enabled on at least one route. All carry a route label matching the route name.

Refresh rate (PromQL):

sum(rate(bouine_refresh_total[1m])) by (route)

Refresh skip ratio (PromQL):

sum(rate(bouine_refresh_skips_total{reason="below_min_hits"}[5m]))
/
sum(rate(bouine_refresh_total[5m]))

A high below_min_hits skip ratio indicates the popularity gate (refresh_min_hits) is filtering out many objects — adjust the threshold if too many popular objects are expiring.

Go runtime#

The standard go_* and process_* metrics from the Prometheus Go client are automatically included. The most operationally relevant:

Access logs#

bouine emits a structured JSON access log line to stdout for every request.

{
  "time":         "2026-06-07T12:00:00Z",
  "level":        "INFO",
  "msg":          "access",
  "method":       "GET",
  "host":         "example.com",
  "path":         "/posts/hello/",
  "proto":        "HTTP/1.1",
  "status":       200,
  "bytes_out":    15234,
  "dur_ms":       1,
  "cache_status": "HIT",
  "traffic_class": "ssr",
  "remote":       "10.42.0.1:54321"
}

The traffic_class attribute (since v0.5.27) mirrors the metric label — the configured class name for the request host, or unclassified.

⚠️ Access log sampling — use Prometheus for throughput#

200 OK responses are sampled at 1:100. All other status codes (errors, redirects, unusual 2xx) are always logged.

This is intentional: at high RPS the log write would otherwise dominate the hot path. The trade-off is that you cannot compute accurate hit ratio or request rate from logs alone. Use bouine_requests_total in Prometheus for throughput and cache result distribution. Use access logs for error diagnosis and per-request debugging.

Rate accuracy from logs:  ❌ (1:100 sampling for 200s)
Error investigation:      ✅ (all non-200 always logged)
Slow request debugging:   ✅ (dur_ms present on sampled 200s)

Distributed tracing (OpenTelemetry)#

When tracing.endpoint is set, bouine exports OTLP/HTTP spans to any OpenTelemetry-compatible backend (Grafana Tempo, Jaeger, Honeycomb, etc.).

Endpoint format#

tracing.endpoint accepts either a bare host:port string or a full URL with http:// / https:// scheme. The scheme prefix is stripped automatically; WithInsecure() is used for plain HTTP.

# canonical form (host:port)
tracing:
  endpoint: "otel-collector.monitoring.svc.cluster.local:4318"
  service_name: "bouine"
  sampling_rate: 0.1

# also accepted (scheme is stripped)
tracing:
  endpoint: "http://otel-collector.monitoring.svc.cluster.local:4318"

Leave endpoint empty (the default) to disable tracing at zero overhead.

Trace structure#

Each request produces a nested span tree:

bouine.pipeline      (L2 — route matching, metrics, access log; one span per request)
  bouine.origin      (L5 — upstream fetch, miss/revalidate path only)

The bouine.origin span carries W3C TraceContext headers (traceparent, tracestate) injected into the upstream request, so the origin server can continue the trace if it also exports spans.

Since v0.5.22, bouine.origin spans on the miss, revalidate, invalidating-proxy, bypass, and streaming paths are parented on the request’s bouine.pipeline span instead of starting a detached root trace, and carry http.method, http.path, http.route, and upstream_pool attributes so slow fetches are filterable by route (bounded label) and pool in the trace backend. Linked spans inherit the root sampling decision — no more 50/50 random drop of origin traces under partial sampling. Background fetches (SWR revalidation, shed refill, background refresh) stay detached (the triggering request’s span is already ended) but carry the same attributes on their root spans.

Since v0.5.13, the admin API also emits a bouine.admin server span for invalidation calls (POST /v1/purge, /v1/ban, /v1/refresh), joining the caller’s trace via the propagated W3C traceparent — so a distributed trace initiated by an invalidation service no longer stops at the client span.

Correlating traces with slow requests#

The data-plane middleware produces a single bouine.pipeline span per request, and bouine.origin spans (miss/revalidate path only) carry the W3C trace context forward to the origin. To jump from a slow access-log line to its trace, correlate the request timestamp and URL in your trace backend. Prometheus exemplars were removed in the v0.5.0 fasthttp migration — the histogram no longer carries per-observation trace IDs.

Admin API#

Grafana dashboard#

An official RED dashboard JSON is shipped in the bouine repository at deploy/grafana/bouine-red.json. Import it into Grafana via Dashboards → Import → Upload JSON file.

The dashboard covers five rows:

Recommended alert rules#

groups:
  - name: bouine
    rules:

    # ── Traffic ───────────────────────────────────────────────────────────
    - alert: BouineHighErrorRate
      expr: |
        sum(rate(bouine_requests_total{status=~"5.."}[5m]))
        /
        sum(rate(bouine_requests_total[5m])) > 0.01
      for: 5m
      labels: { severity: critical }
      annotations:
        summary: "bouine 5xx error rate > 1%"

    - alert: BouineHighStaleRate
      expr: |
        sum(rate(bouine_requests_total{cache_result="STALE"}[5m]))
        /
        sum(rate(bouine_requests_total[5m])) > 0.10
      for: 10m
      labels: { severity: warning }
      annotations:
        summary: "Stale serve ratio > 10% — possible origin outage"

    # ── Storage ───────────────────────────────────────────────────────────
    - alert: BouineHighEvictionRate
      expr: rate(bouine_hot_store_evictions_total[5m]) > 100
      for: 10m
      labels: { severity: warning }
      annotations:
        summary: "High SIEVE eviction rate — working set may exceed hot_max_bytes"

    # ── GC / Runtime ──────────────────────────────────────────────────────
    - alert: BouineGCPauseHigh
      expr: |
        max(rate(go_gc_duration_seconds_sum[1m])
            / rate(go_gc_duration_seconds_count[1m])) > 0.01
      for: 5m
      labels: { severity: warning }
      annotations:
        summary: "GC average pause > 10 ms — raise GOMEMLIMIT (see troubleshooting)"

    # ── Cluster ───────────────────────────────────────────────────────────
    - alert: BouineClusterModeMismatch
      expr: count(count by (mode) (bouine_cluster_mode_info == 1)) > 1
      for: 2m
      labels: { severity: critical }
      annotations:
        summary: "Pods running different cluster modes — configuration drift"

    - alert: BouinePeerFetchVariantMismatch
      expr: sum(rate(bouine_peer_fetch_variant_mismatch_total[5m])) > 0
      for: 10m
      labels: { severity: warning }
      annotations:
        summary: "Peer-fetch variant rejections — mixed-version fleet or a peer serving wrong-variant content"

    - alert: BouinePeerFetchShedding
      expr: sum(rate(bouine_peer_shed_total[5m])) > 0
      for: 5m
      labels: { severity: warning }
      annotations:
        summary: "Peer-fetch concurrency slots saturated — raise cluster.peer_fetch_concurrency"