Benchmarks

Disclaimer#

Benchmarks are synthetic. They measure isolated behavior in artificial conditions. Production workloads have different allocation patterns, concurrency levels, GC pressure, network conditions, and hardware. The Varnish and NGINX configurations used in these tests could likely be improved — a differently tuned Varnish or NGINX may produce different results. Benchmarking is genuinely hard, and these numbers should be treated as directional indicators, not as definitive performance claims.

Always benchmark on your own hardware with your own workload before making infrastructure decisions.

Methodology#

All benchmarks were run on the same machine (Apple M5, darwin/arm64) with Go 1.27. Proxies were run in Docker containers via docker compose, with an origin server serving synthetic responses. Load was generated by k6 at fixed RPS targets.

Proxies tested#

Scenarios#

The mixed scenario uses a realistic mix of cacheable and non-cacheable responses with varying TTLs, conditional requests (ETag revalidation), and Vary headers.

Results#

Scenario 3.2 — Hit-Only (3000 RPS, 100% cache hits)#

Pure cache-hit performance. Every request finds a fresh cached response.

Bouine and NGINX match Varnish on pure cache hits. Envoy is ~40% slower.

Scenario 3.3 — Miss Storm (1500 RPS, all misses)#

Every request goes to origin (origin returns Cache-Control: no-store).

NGINX’s proxy_cache path is not optimized for the no-store case and degrades under sustained miss traffic. Bouine and Varnish handle miss storms equally well. Envoy has low p90/p95 but higher average due to occasional spikes.

Scenario 3.6 — Mixed Realistic (3000 RPS, ~73% hit rate)#

The most production-representative scenario: a mix of cacheable and non-cacheable responses, conditional revalidation, and Vary headers.

Varnish’s 93% hit rate vs bouine’s 73% is the primary driver of the latency difference. Varnish serves stale objects unconditionally during revalidation (grace mode), while bouine’s stale-while-revalidate is RFC 5861-compliant (only serves stale within the SWR window). Aligning bouine’s grace semantics with Varnish’s would close most of this gap. Envoy’s cache filter did not cache any responses in this scenario (0% hit rate).

Evolution: bouine before and after optimization#

Go micro-benchmarks#

Unit-level benchmarks isolating specific hot-path components, from the current bench/results/current.txt gates. Handler_CacheMiss_Cacheable is held to an allocation budget of 18 in CI (currently at 13 — see the v0.5.2 changelog).

All hit-path benchmarks enforce 0 allocs/op in CI via benchmark gates. Any allocation on the hit path blocks merge.

Comparability note. The nightly load-test configuration enables experimental.h1_fast_path (since v0.5.3) and experimental.h1_reactor (since v0.5.5). Nightly proxy-comparison numbers from v0.5.2 and earlier were measured without the fast path and are not comparable.

Benchmark infrastructure#

The benchmark suite lives in bench/ in the bouine repository:

bench/
├── run.sh                    # Go micro-benchmark runner with gates
├── results/                  # Baseline and current benchmark output
└── loadtest/
    ├── docker-compose.yaml   # bouine + varnish + nginx + envoy + origin
    ├── config/               # Per-proxy configurations
    ├── scenarios/
    │   ├── 3.2_hit_only/     # Pure cache-hit scenario
    │   ├── 3.3_miss_storm/   # All-miss scenario
    │   ├── 3.6_mixed_realistic/  # Production-like mixed workload
    │   └── ...               # Cluster, chaos, and edge-case scenarios
    └── analysis/             # Python scripts for plotting and reporting

To reproduce the load-test results:

cd bench/loadtest
docker compose up -d origin bouine varnish nginx envoy

# Run a scenario
k6 run scenarios/3.2_hit_only/k6.js

# Generate a comparison report
python3 analysis/report.py results/

To run the Go micro-benchmarks:

make bench    # runs all benchmarks, checks gates, saves to bench/results/current.txt
make benchstat  # compares current vs baseline with benchstat