Streaming and live responses

bouine never buffers a response body when it does not have to. This page documents the three streaming behaviours and the two overload protections that interact with them.

Server-Sent Events (SSE)#

bouine serves Server-Sent Events as live end-to-end streams. A request announcing Accept: text/event-stream (the WHATWG client contract — what browsers, EventSource, and AI SDKs send) is served as an unbuffered stream, never cached, and never collapsed by singleflight onto another client’s stream.

routes:
  - match: { path_prefix: /chat/ }
    pool: llm
    cache:
      ttl_default: 60s

No route-level configuration is required: the Accept header is the contract. POST-based SSE (the dominant AI API shape — a request body followed by a streamed response) works the same way, and its write-method invalidation semantics are preserved: a 2xx/3xx response purges the affected cache entry at header time, not after the (endless) body.

SSE serving contract#

The last two rows are the key property: a stream whose origin keeps sending events stays open indefinitely, while a dead peer or a client that stops reading is still cut.

Non-hinted SSE#

An origin may respond Content-Type: text/event-stream to a request that did not announce the Accept header. Such responses are still streamed unbuffered, but they are bounded by the route’s fetch_timeout (the origin connection’s read deadline was armed before the response was known to be a stream). Fix the client to send the header; do not raise fetch_timeout. Concurrent non-hinted requests for the same URL are not collapsed onto one stream — nothing is buffered, so there is no shareable result.

Tuning SSE routes#

• Sparse event feeds (gaps > 10 min without heartbeats): raise nothing. The origin must send SSE comment-line heartbeats, or the stream is cut after the 10-minute idle budget and clients reconnect. This matches nginx proxy_read_timeout / Varnish between_bytes_timeout in spirit.
• Many concurrent streams: each stream holds one client connection and one origin connection for its lifetime. Raise upstream_pools[].connect.max_connections (default 64) on pools fronting SSE-heavy routes, and listen.max_connections for the data plane. max_fetch_concurrency does not need raising — streams release their fetch slot at header time.
• Hung origins: a hinted fetch whose origin accepts the connection but never sends headers pins one fetch slot for up to the 10-minute idle budget (instead of response_header_timeout). Only requests that explicitly announce stream intent take this path.

Failure modes#

Streaming misses#

Cacheable misses are streamed to the client while the body is teed to storage in the background, so the client does not wait for the full body before the first byte. The tee buffers are capped:

• Per stream: max_response_bytes (the fetch is aborted with 502 beyond it).
• Per route: max_streaming_buffer_bytes — the total bytes held in live tee buffers across concurrent miss-fetches on the route. When exceeded, new cacheable misses fall back to synchronous buffering (the client waits for the full body, the buffer is no longer live). Default derives from GOMEMLIMIT (7%), with a 64 MiB built-in floor. Watch bouine_streaming_buffer_bytes and bouine_streaming_fallback_total to see pressure.

Origin-fetch shedding#

Slow origins are the classic reverse-proxy failure mode: request goroutines park waiting for a fetch slot, pile up without bound, and the pod enters a non-recovering livelock. bouine sheds instead.

When a foreground miss cannot acquire an origin-fetch slot (bounded by max_fetch_concurrency per route) within fetch_wait_timeout (default 100 ms, validated max 1 s):

1. A stale object in scope is served stale (RFC 5861-style, within stale_if_error semantics), or
2. the client receives 503 + Retry-After: 1 — distinct from the 502 origin-failure mapping.

Singleflight followers and inflight-stream followers un-park with the leader’s shed result, so a shed affects the whole collapsed group consistently. The bouine_fetch_shed_total counter exposes the shed rate for alerting.

routes:
  - match: { path_prefix: / }
    pool: app
    cache:
      max_fetch_concurrency: 32
      fetch_wait_timeout: 100ms

The wait bound exists to absorb sub-second fetch-queue bursts, not to queue through a sustained overload: when arrival rate exceeds drain rate, no finite wait drains the queue, so a longer bound only holds goroutines (and their connections) longer before shedding them. Raise max_fetch_concurrency or scale out instead of raising fetch_wait_timeout.