Batch tuning sweep

Empirical sweep of batch_size × batch_deadline_ms to inform the choice of defaults and surface the latency / throughput trade-off for operators tuning their own deployments.

Addresses the "Batching efficiency: tune batch_size / batch_deadline_ms sweet spot" item from the perf TODO in crates/weir-server/tests/load.rs:26-33.

Caveat

Results below were collected on a single shared-VM sandbox in a single session. Absolute numbers are noisy and will not match production hardware; relative ordering between configs is the takeaway.

The latency tables are single-producer. The three latency scenarios (sync / batched / buffered) and the single-thread throughput table below time one producer pushing serially, so a batch fills by reaching batch_deadline_ms, not the batch_size cap. That is why the deadline appears to dominate latency here — it is the single-threaded regime talking. Under concurrent producers the picture changes (see Tuning under concurrency): batches fill by the deadline regardless of the size cap, so batch_size does little, and the per-deadline latency figures in these tables do not carry over to concurrent throughput. Read the deadline-dominates-latency numbers as a single-threaded result, not a tuning guide for loaded deployments.

Methodology

For each (batch_size, batch_deadline_ms) point:

1. Fresh wab_dir, fresh weir-server process per trial
2. cargo test --release -p weir-server --test load -- --nocapture → 3 trials
3. Median across trials per percentile

Fixed parameters: shard_count=2, worker_count=2, payload 256 B, 1000 samples.

tests/load.rs reports three latency scenarios per run: sync (request → fsync → ack), batched (request → flush-or-deadline → ack), buffered (request → in-memory ack, no durability).

Results

sync (request → WAB fsync → ack)

batched (request → batch-flush-or-deadline → ack)

buffered (request → in-memory ack, no durability)

Interpretation

• batch_deadline_ms dominates latency: p50 tracks the deadline to within a few hundred µs. A 5 ms deadline pushes p50 to ~6 ms; a 10 ms deadline pushes it to ~11 ms.
• batch_size has a small effect at the same deadline: 256 slightly outperforms 64 on the tail (p99 / p999), and this sweep does not explain why. The explanation this bullet used to give — "more records amortise per fsync" — cannot apply to these tables, which are single-producer (see Caveat): one connection gets 1.00 records per fdatasync, because the daemon acks frame N before reading frame N+1 (crates/weir-server/src/socket/connection.rs:485-491), so there is never a second record from that connection to amortise against. Treat the gap as unexplained and possibly run-to-run noise in a 3-trial median on a shared sandbox.
• (256, 1 ms) is the sweet spot in this sweep — lowest p50 across sync / batched / buffered and tightest p99 tail in batched.
• The config/mod.rs defaults at the time of this sweep (batch_size = 1000, batch_deadline_ms = 100) were out of step with what every shipped config actually uses (the CI load job runs batch_size = 64, the bench job and smoke test use batch_deadline_ms = 1). The recommendation below was applied: the current defaults are now batch_size = 256, batch_deadline_ms = 1.

Recommendation

Change Config::from_layers defaults to batch_size = 256, batch_deadline_ms = 1. This:

• Brings defaults in line with what the project actually exercises in CI
• Lands a ~6× p50 latency improvement for any operator who runs with defaults
• Improves throughput by 3.6–7.9× across every scenario measured (computed against (1000, 10ms) in the companion sweep below, which shows no scenario where the old default beats (256, 1ms))

Applied in the commit that follows this doc.

Read those two multipliers narrowly; they travel badly. Both are medians of three trials on a shared-VM sandbox, the surface this file's own caveat calls too noisy to quote absolutely, and Interpretation attributes a smaller discrepancy to run-to-run noise in exactly this data. The direction is solid — no measured scenario favours the old pair — but "3.6–7.9×" is one sweep's spread, not a reproducible range.

The baseline matters as much as the noise. The multipliers are computed against (1000, 10 ms), the nominal default of the time, which the bullet above notes nothing actually ran: CI used batch_size = 64, and the bench job and smoke test used batch_deadline_ms = 1. So the gain is real for a hypothetical operator who took the shipped defaults untouched, and close to nothing for anyone whose config resembled what the project itself exercised. Against (64, 1 ms) the same table shows 506 vs 486 single-thread sync RPS — about 4%.

Do not quote either figure as a weir performance improvement. The change was right for a better reason: the defaults had drifted away from every configuration the project tested, and a default nobody exercises is a default nobody has debugged.

Throughput companion sweep

Same 5 configs × 3 trials, but using the baseline_*_throughput* and thundering_herd_* scenarios in tests/load.rs (5000 samples per scenario, 256 B payload). Median RPS across the 3 trials per scenario.

Single-thread

Concurrent (thundering herd)

(256, 1ms) leads or ties on every scenario; (1000, 10ms) is 3.6–6.7× worse on every concurrency level. The throughput data agrees with the latency data: no axis on which the (1000, 10ms) default beats (256, 1ms).

Tuning under concurrency

The two knobs a tuner reaches for first — batch_size and batch_deadline_ms — are largely inert under concurrent producers, and the wrong place to start. The flusher flushes when it reaches batch_size records or batch_deadline_ms elapses, whichever comes first. With many concurrent producers a batch hits the deadline long before it hits the size cap, so batch_size does little and the flush cadence is set by the deadline. The deadline-dominates-latency figures in the Results tables are single-producer measurements (see Caveat); they are not a guide to concurrent throughput. Keep the defaults (256, 1ms).

The lever that actually moves concurrent Sync throughput is group-fsync batching — how many concurrent producers' records ride a single fsync — and that is governed by connection density into few shards, not by the batch knobs. Each connection is pinned to one shard, and a shard's flusher group-fsyncs every record in its active segment at once, so more concurrent producers per shard means more records amortised per fsync. Spreading the same connections across more shards thins that batching and can push Sync throughput below a single shard — shard_count is non-monotonic, not a throughput dial. See the shard_count caveat in operations/configuration.md for the full mechanism and the connections-per-shard guidance.

This sweep does not quantify that lever: shard_count / worker_count were held fixed at (2, 2) to isolate the batching variables, and the per-machine RPS above will not transfer to other hardware. Treat this section as qualitative direction — measure the shard / connection-density trade on your own hardware before committing to a value.

A related throughput cliff: wab_segment_max_bytes

Not a batch knob, but the same throughput conversation: a small wab_segment_max_bytes (frequent segment rotation) is a throughput cliff. Each seal does an extra fsync (sentinel+footer durability commit, plus a parent-dir fsync to publish the rename), so more rotations = more seal fsyncs = lower RPS — a sweep observed sync throughput drop from ~90k to ~26k RPS when the segment cap was lowered to 1 MiB (observed on one box; treat the absolute numbers as machine-specific). The size of the cliff is also platform-specific: the parent-directory fsync that publishes each seal is File::sync_all (crates/weir-server/src/wab/segment.rs:712-714), which is a plain fsync on Linux but F_FULLFSYNC on macOS — measured at 36 µs and 3,965 µs respectively on one M3 Max. Do not carry a macOS-measured cliff to a Linux deployment or the reverse. Durability is unaffected — every accepted record is fsynced before its ack regardless of segment size; the seal fsync only publishes the sealed-segment boundary to the drain. Default is 256 MiB (SEGMENT_MAX_BYTES); see wab_segment_max_bytes.

Out of scope for this sweep

• Larger payloads. 256 B is the load-test default but production records may be larger.
• Different shard_count / worker_count. Held fixed at (2, 2) to isolate the batching variables.