Configuration reference

TL;DR — weir reads config from CLI flags, env vars, an optional TOML file, and built-in defaults, in that precedence order. Every option has a sensible default; the minimum viable config is wab_dir + the parent of socket_path existing and being writable by the daemon's user. The full reference below covers every option with its default, range, what it controls, and when to tune it.

Configuration sources

Weir merges four layers in this precedence order (first match wins):

1. CLI flags — --socket-path, --wab-dir, --batch-size, etc. Useful for one-off runs and overriding the file for tests.
2. Environment variables — WEIR_SOCKET_PATH, WEIR_WAB_DIR, WEIR_BATCH_SIZE, etc. The WEIR_ prefix + screaming-snake-case field name. Useful in container orchestrators (Kubernetes, ECS, systemd Environment=).
3. TOML file — default path /etc/weir/weir.toml, overridable via --config <path>. A missing file is not an error; the loader silently falls through to env + defaults. Unknown keys produce a warn! log but don't fail startup.
4. Built-in defaults — every option has one; running weir-server with no flags, no env vars, and no config file will start the daemon with the defaults documented here.

The merge happens once at startup. Config is not reloaded on SIGHUP (see Limitations below).

How to read this reference (knob philosophy)

The list below is long, but you can ignore most of it. Every option has a production-ready default; a real deployment typically sets only a handful — wab_dir, sink_type + sink_url, maybe shard_count and a durability-related dial. The rest are operability levers: timeouts, retry budgets, poll intervals, and safety caps that you don't touch day to day but want available for the incident where the default is wrong for your environment and a recompile isn't an option (a slow downstream, a flaky uplink, an unusually large host).

This is a deliberate design stance, not configuration sprawl:

• A knob you never change still earns its place if it's the difference between a three-line config edit and a fork-and-rebuild at 3am. The value of a safety/ tuning dial is realised exactly once, when you need it.
• Defaults are the single source of truth — they're tuned constants in the code (drain::MAX_RETRIES, drain::HEALTH_POLL_INTERVAL, …) that the knobs fall back to, so "the default" and "what the tests exercise" can't drift apart.
• Adding a knob is a backward-compatible change; removing one is breaking. So new knobs are added sparingly — only when a real deployment would plausibly need to change the value, never "just in case."

Practical takeaway: start from the defaults, set the four or five knobs your deployment actually needs, and treat the rest as documented escape hatches you'll reach for if and when an operational problem points you at one.

TOML file structure

All options live under [server]:

[server]
socket_path = "/run/weir/weir.sock"
wab_dir     = "/var/lib/weir/wab"
batch_size  = 256
# ... etc

The reference example at deploy/docker/weir.toml.example shows every option with its default value and the matching WEIR_* env var name.

Option reference

Each option below lists: type, default, valid range, CLI flag, env var, TOML key, what it controls, and when to tune.

Paths

socket_path

• Type: absolute path
• Default: /run/weir/weir.sock
• Validation: must be absolute, no .. components, no null bytes. The parent directory must exist; weir does not create it.
• CLI: --socket-path <path>
• Env: WEIR_SOCKET_PATH
• TOML: socket_path

The Unix domain socket file the daemon listens on. Producers connect here.

Operational notes:

• The parent directory must be writable only by the daemon's user (typically mode 0o700). The bind sequence relies on this — see socket-bind security analysis for the threat model.
• The socket file itself is always created with mode 0o600 via a hardened bind sequence (bind_hardened in crates/weir-server/src/socket/mod.rs).
• A stale socket at the path from a previous unclean shutdown is detected and replaced atomically. A non-socket file at the path causes startup to refuse.

When to change the default: production deployments using a non-standard runtime directory (e.g. /var/run/weir/), or per-instance socket paths when running multiple daemons on one host.

peer_uid_check

• Type: bool
• Default: true
• CLI: --peer-uid-check <true|false>
• Env: WEIR_PEER_UID_CHECK
• TOML: peer_uid_check

The application-level access control on the Unix socket path. When enabled (the default), the accept loop reads each peer's effective uid (SO_PEERCRED on Linux, getpeereid on macOS) and refuses any connection whose peer euid does not match the daemon's — closing the socket before any frame is read. It is fail-closed: if the peer credential lookup itself fails, the connection is refused. Every refusal increments weir_connection_rejected_peer_uid_total.

This is the narrow application-level complement to the socket file's 0o600 permissions: even if the socket's permissions or parent directory are looser than intended, a same-host process running as a different user cannot connect. It applies only to the Unix transport; the TCP + mutual-TLS listener authenticates clients by certificate instead.

When to change the default: set to false only when a trusted helper must connect as a different uid than the daemon (e.g. a sidecar under a separate service account) and you are deliberately relying on socket permissions alone. Leaving it on is strongly recommended.

wab_dir

• Type: absolute path to an existing directory
• Default: /var/lib/weir/wab
• Validation: must be absolute, must already exist, canonicalize()-able, no .. components, no null bytes.
• CLI: --wab-dir <path>
• Env: WEIR_WAB_DIR
• TOML: wab_dir

The write-ahead buffer directory. WAB segments and the dead-letter sub-directory live here.

Operational notes:

• The daemon does not create the WAB directory (Postgres model — the operator creates the data directory at install time). Pre-create with:

  mkdir -p /var/lib/weir/wab
  chown weir:weir /var/lib/weir/wab
  chmod 0700 /var/lib/weir/wab
  

• Subdirectories: <wab_dir>/shard_NN/ per shard (zero-padded index, e.g. shard_00), <wab_dir>/dead_letter/ for permanently-rejected records, <wab_dir>/quarantine/ for corrupted segments quarantined during crash recovery.
• Segment files are created mode 0o600. At startup, every .wab / .wab.sealed / .wab.confirmed file is audited; any with permissions ≠ 0o600 increments weir_wab_unexpected_mode_total and logs a warning.

Disk sizing: depends on batch_size, batch_deadline_ms, sink throughput, and dead_letter_max_bytes. For typical workloads, provisioning ~5× the dead-letter cap is a safe starting point. The WAB is rotated and reclaimed as the sink confirms records, so steady- state size is bounded by transient backlog, not by total throughput.

Sharding and worker pool

shard_count

• Type: usize
• Default: 1
• Range: 1–256
• CLI: --shard-count <n>
• Env: WEIR_SHARD_COUNT
• TOML: shard_count

Number of WAB shards. Each shard has its own segment file and its own flusher thread. Each connection is pinned to a shard round-robin at accept time (connection_counter % shard_count); every record on that connection goes to the same shard. Per-record shard-aware routing is future work.

When to tune: increase only if fsync throughput is your bottleneck and you have parallel disk bandwidth. On a single SSD, multiple shards do not increase fsync throughput — they fight for the same disk queue. On NVMe with high parallel write depth, 2–4 shards may improve aggregate throughput. Default 1 is correct for most deployments.

shard_count is not a throughput dial — and it is non-monotonic under concurrency. Each connection is pinned to one shard for its lifetime, round-robin at accept time (counter % shard_count, socket/mod.rs). The Durable-tier throughput lever is group-fsync batching: a shard's flusher rides every record currently in its active segment on a single fsync (wab/mod.rs flush_batch — "one group fsync covers every record still in the active segment"), so the more concurrent producers land on a shard, the more records amortise each fsync. Spreading the same set of connections across more shards does the opposite — it thins connections-per-shard, so each group fsync covers fewer records, while adding one more contending flusher thread per shard. Past the point where that trade turns over, more shards can drop Durable throughput below a single shard. "Match your cores" is therefore a trap for this knob: the optimum is governed by connections-per-shard, not core count. If you are chasing Durable throughput, prefer concentrating concurrent producers into few shards over fanning out into many. (This is qualitative — the turn-over point is workload- and hardware-specific; measure your own.)

The "few shards" advice is for the fsync tier only — it inverts for Buffered. The amortization above exists because Durable records share a single group fsync. Buffered never fsyncs — its records ack the moment they hit memory (wab/mod.rs, Durability::Buffered acks immediately, with no group fsync), so there is nothing to amortize and no benefit to piling connections onto one shard. What concentration does do there is funnel every concurrent producer through a single shard's flusher thread and active segment, serialising work that could run in parallel — in our runs this collapsed aggregate Buffered throughput by roughly 8–12× versus spreading the same connections across shards. So for a Buffered workload, spread connections across shards (and scale shard_count toward your parallel-write capacity); reserve the few-shards guidance for the Durable tier where the group fsync actually rewards density.

The daemon emits a startup advisory if shard_count / worker_count looks unusual for the host's core count. See Agent count vs cores for the empirical sweep and the heuristic the advisory is based on; the rule of thumb is max(2, cores - 2) / 2. Treat that advisory as a thread-budget sanity check (don't run far more agent threads than cores), not a throughput recommendation: as the caveat above explains, the Durable-throughput optimum follows connections-per-shard, not cores, so the cores-based heuristic does not predict the best shard_count for throughput.

worker_count

• Type: usize
• Default: min(shard_count, 64) — the resolved shard_count, capped at the field max
• Range: 1–64
• CLI: --worker-count <n>
• Env: WEIR_WORKER_COUNT
• TOML: worker_count

Number of worker threads pulling from the global queue and batching records into per-shard batch channels. Workers are stateless and pinned at startup; they exist primarily to absorb tokio spawn_blocking slots without each spawn-blocking task fighting for a queue lock.

The default was changed in 0.9.0 from the hard-coded 2 to shard_count. With the old default, the standard single-shard config had worker_count=2 but only one shard, so worker 1 was permanently idle. Defaulting to shard_count removes the idle worker and keeps the balanced invariant (worker_count == shard_count) out of the box.

When to tune: scale with max_connections / 32 as a starting point. The bottleneck is rarely worker count itself; if records appear to back up at the worker layer (rising weir_queue_depth without a corresponding rise in WAB fsync latency), worker count is the right knob.

Batching

The batch tuning data is in batch-tuning.md; an operator-facing tuning guide is planned for Phase 2. Defaults below are the sweet spot found by the empirical sweep.

These two knobs are largely inert under concurrency — keep the defaults. The flusher flushes when it reaches batch_size records or batch_deadline_ms elapses, whichever comes first (worker.rs / wab/mod.rs). With many concurrent producers a batch fills by reaching the deadline long before it reaches the size cap, so batch_size does little and batch_deadline_ms is what actually governs the flush cadence. The dramatic per-deadline figures in batch-tuning.md (e.g. "a 5 ms deadline pushes p50 to ~6 ms") come from its single-producer latency sweep; they are not a guide to throughput under concurrent load. The defaults (256, 1ms) are a good starting point at any concurrency — reach for shard_count and connection density (see its caveat) before these two when tuning concurrent Durable throughput.

batch_size

• Type: usize
• Default: 256
• Range: 1–100,000
• CLI: --batch-size <n>
• Env: WEIR_BATCH_SIZE
• TOML: batch_size

Maximum records per fsync batch. When the flusher thread has accumulated this many records or batch_deadline_ms elapses, whichever comes first, the batch is fsynced and the per-record acks are returned.

Effect: larger batches amortise fsync cost across more records (higher throughput) but increase tail latency for the first record in each batch (waiting for siblings).

When to tune: the default (256, 1ms) is a balanced point for mixed workloads. Pure-throughput workloads can push this to 1000+; strict-latency workloads down to 64 with batch_deadline_ms = 1. See batch-tuning.md for the empirical landscape.

batch_deadline_ms

• Type: u64 (milliseconds)
• Default: 1
• Range: 1–60,000
• CLI: --batch-deadline-ms <n>
• Env: WEIR_BATCH_DEADLINE_MS
• TOML: batch_deadline_ms

Maximum time the flusher thread waits to fill a batch before flushing what it has. Caps tail latency in low-traffic regimes where batch_size would not be reached for a long time.

Effect: lower values reduce p99 latency at low load; higher values reduce CPU wakeups but increase tail latency on the first record after quiet periods.

When to tune: leave at 1 unless you're running an extremely high-volume, latency-insensitive workload where you want to amortise the timer cost itself. See batch-tuning.md.

wab_segment_max_bytes

• Type: u64 (bytes)
• Default: 268435456 (256 MiB)
• Range: 4096 – 4294967296 (4 KiB – 4 GiB)
• CLI: --wab-segment-max-bytes <n>
• Env: WEIR_WAB_SEGMENT_MAX_BYTES
• TOML: wab_segment_max_bytes

The size threshold at which the WAB flusher seals the active segment and opens a fresh one. The sealed segment is forwarded to the drain for sink commit; until a segment seals, its records are durable on disk but invisible to the sink.

Low-volume trap: with both seal timers off (the default), a low-volume producer delivers nothing. With this 256 MiB cap and both wab_segment_max_age_secs and wab_segment_max_lifetime_secs at 0, a deployment that writes a handful of small records gets every record acked but the segment never reaches 256 MiB, so it never seals and the sink receives nothing until shutdown. This is the single most common "weir accepts records but my sink is empty" surprise.

Which timer fixes it depends on the producer, and picking the wrong one leaves the symptom exactly as it was: wab_segment_max_age_secs seals a segment that has gone quiet, and a producer that trickles steadily never does, so it keeps the segment open indefinitely. wab_segment_max_lifetime_secs seals on the segment's age regardless. If in doubt set both — whichever comes due first seals — or lower this cap.

Effect: smaller values trigger more frequent drain → sink activity and faster failure isolation (a corrupt segment only affects records in that window); larger values reduce the per-rotation overhead and batch more records into each Sink::commit call. The 256 MiB default balances both for production workloads.

When to tune: lower for storage-constrained deployments (e.g. embedded boxes with ≤ 1 GiB available for WAB), or in tests that need to demonstrate sink-side behaviour without pushing 256 MiB of data. Higher if you have spare disk and want to amortise the seal cost over more records.

wab_segment_max_age_secs

• Type: u64 (seconds)
• Default: 0 (disabled)
• Range: 0 – 86400
• CLI: --wab-segment-max-age-secs <n>
• Env: WEIR_WAB_SEGMENT_MAX_AGE_SECS
• TOML: wab_segment_max_age_secs

Opt-in idle seal: when > 0, the WAB flusher seals the active segment (handing it to the drain) after it has gone this many seconds with no new records, instead of waiting to reach wab_segment_max_bytes or for shutdown. 0 (default) preserves the historical behaviour — a segment seals only on the size threshold or shutdown.

Why it exists: a low-volume deployment can write a handful of small records and then go quiet; with size-only sealing those records stay durable on disk but never reach the sink until 256 MiB accumulates or the daemon restarts — which reads as "weir isn't delivering." Set this (e.g. 5–30) so a quiet segment drains promptly. On-disk format is unchanged; this only affects when a normal seal happens.

When to tune: set it on edge/low-throughput deployments where timely delivery matters more than maximal per-segment batching; leave it 0 on high-throughput deployments where segments fill quickly on their own.

This knob is named like a maximum age and is not one. It is an idle timer, and a steady trickle defeats it. The clock restarts on every flush (crates/weir-server/src/wab/mod.rs), so a producer writing one record a minute against wab_segment_max_age_secs = 300 resets it four times over before it can expire. That segment never seals on this timer at all — it grows toward wab_segment_max_bytes exactly as if the knob were 0, which at 500 bytes a minute is roughly two years before anything reaches the sink. The knob works for a producer that stops; it does nothing for one that never pauses for the whole interval.

If your producer trickles rather than bursts, you want wab_segment_max_lifetime_secs instead of, or alongside, this one.

wab_segment_max_lifetime_secs

• Type: u64 (seconds)
• Default: 0 (disabled)
• Range: 0 – 86400
• CLI: --wab-segment-max-lifetime-secs <n>
• Env: WEIR_WAB_SEGMENT_MAX_LIFETIME_SECS
• TOML: wab_segment_max_lifetime_secs

Opt-in maximum lifetime: when > 0, the WAB flusher seals the active segment (handing it to the drain) this many seconds after the segment was created, however busy the producer has been in the meantime. 0 (default) preserves the historical behaviour.

This is the knob that bounds delivery latency. Nothing restarts its clock — not a write, not a flush, not a burst — so it is the only setting that makes "records reach the sink within N seconds" a statement weir will actually keep.

Which of the two timers do I want?

They are different knobs measuring from different instants, and the names do not make that obvious. The reference point is the whole distinction:

Read the third row first. wab_segment_max_age_secs asks "has anything happened recently?"; wab_segment_max_lifetime_secs asks "how old is this segment?" Only the second question has an answer a producer cannot change by writing.

Why it exists: low volume is the normal case for several things weir is deployed as — an S3 archive, a meter, a field station, a till — and those producers trickle rather than go quiet. The idle timer is the wrong shape for all of them, and until this knob existed the only honest advice was to lower wab_segment_max_bytes and hope the arithmetic worked out.

Setting both is the usual answer, and the rule is that whichever comes due first seals. They are both upper bounds on how long a record can sit undelivered, so they compose by taking the minimum; adding one can only make delivery earlier, never later. A typical low-volume pair:

wab_segment_max_age_secs      = 5    # quiet? deliver in 5s
wab_segment_max_lifetime_secs = 300  # busy? deliver within 5 minutes regardless

A lifetime shorter than the idle interval is accepted, not rejected — it simply means the lifetime always wins and the idle timer never gets the chance to fire. That is a legitimate configuration, so weir does not second-guess it.

When to tune: lower it when delivery latency matters (a dashboard reading the sink, an archive with a freshness SLA); raise it when per-segment batching matters more (bigger objects in S3, fewer round trips to a database). Every seal is a segment file and one Sink::commit, so a 1-second lifetime on a busy shard buys latency with a lot of small segments. On-disk format is unchanged; like the idle timer this only affects when a normal seal happens.

wab_max_bytes

• Type: u64 (bytes)
• Default: 0 (disabled)
• Range: 0, or large enough that wab_max_bytes / 10 * 9 exceeds shard_count × wab_segment_max_bytes
• CLI: --wab-max-bytes <n>
• Env: WEIR_WAB_MAX_BYTES
• TOML: wab_max_bytes

A cap on total live WAB bytes on disk — the same footprint weir_wab_bytes_on_disk measures: the open active segment plus sealed segments awaiting drain. Once live bytes reach the cap, pushes on both durability tiers are rejected, including Buffered: it still writes to the WAB before it acks, so it is not exempt. This closes the case where a dead or stalled drain lets the disk fill while producers keep receiving successful acks.

Rejected pushes receive NackReason::InternalError — the same reason byte queue saturation uses, not a new one. That is deliberate: weir-client's ClientError::is_recoverable() treats InternalError as recoverable but an unknown Nack reason as fatal, so a new byte would make every client built before this feature tear down and reconnect precisely when the daemon is already under strain. The cost is that a cap rejection is indistinguishable from queue saturation by reason byte alone — weir_wab_cap_rejections_total is the only way to tell the two apart.

Once rejecting, ingest resumes only after live WAB bytes fall back below wab_max_bytes * 0.9 (hysteresis, so ingest does not flap on and off at the boundary). This state is shared daemon-wide across every listener — the Unix socket and the TCP/mTLS listener alike — so one connection observing recovery releases all of them, and one observing the cap holds all of them.

This is a soft high-water mark, not a hard limit. The value it is checked against is refreshed every 5 seconds, so the WAB can overshoot by up to 5 seconds of peak ingest. An operator who sets the cap at their exact free space will still fill the disk — leave at least that much headroom below actual free space.

0 (default) disables the cap: the WAB is unbounded, as it always was before this knob existed.

When set, the cap must be large enough to release itself. Validation requires

wab_max_bytes / 10 * 9  >  shard_count * wab_segment_max_bytes

and refuses to start otherwise, naming the minimum it would accept. The reason is that the right-hand side is a floor the daemon cannot drain away: every shard holds one open active segment, its bytes are counted by the cap, and an active segment is sealed only by a write that crosses wab_segment_max_bytes (or by wab_segment_max_age_secs, which is off by default). Once the cap starts rejecting there are no writes — so nothing seals, nothing reaches the drain, and live bytes can never fall back below the wab_max_bytes * 0.9 resume line. A cap below that bound wedges ingest until the daemon is restarted, even with a perfectly healthy sink and drain.

The practical consequence: size the cap against shard_count, not against one segment. With shard_count = 4 and the default 256 MiB segments, the floor is 1 GiB and the smallest accepted cap is 1193046480 bytes (≈1.11 GiB) — but a cap that close to the floor leaves almost no working room, so prefer several times the floor. Setting wab_segment_max_age_secs > 0 is also worth doing alongside the cap: it seals idle active segments on a timer, which lets their bytes reach the drain instead of sitting in the un-drainable floor.

Known limitation — one path can raise that floor at runtime. When a write to an active segment fails, the shard abandons that segment and opens a fresh one. The abandoned file stays on disk, keeps counting toward the cap, and is never sealed or drained; only a restart reclaims it, through recovery. Each orphan is at most wab_segment_max_bytes, so a run of write failures — a disk going bad — can lift the un-drainable floor above whatever cap validation accepted at startup and wedge ingest with a healthy sink. Watch weir_wab_fsync_failures_total and weir_wab_bytes_on_disk: a cap rejecting while weir_sink_health reads healthy and the drain is draining is this, not sink backpressure. Sizing the cap at several times the floor gives it room to absorb a few orphans.

When to tune: set it wherever the WAB shares a disk with anything else and you want a producer-visible signal (Nacks) when the sink stops draining, instead of discovering it from a full disk. Leaving it unset is not silent: the daemon separately warns when the WAB is growing while the sink is unhealthy and no cap is set (see the CHANGELOG).

wab_compression

• Type: string — "none" or "zstd"
• Default: "none"
• CLI: --wab-compression <mode>
• Env: WEIR_WAB_COMPRESSION
• TOML: wab_compression

Compress WAB record payloads with zstd. Each record is stored as its own complete zstd frame; the framing, CRC, sentinel and footer are unchanged.

Read the numbers below before enabling this. For most weir workloads the gain is small, and for small records it is negative.

What it costs

Compression runs on the flusher thread, before the group fsync, so it is on the ack path. Measured on the load suite (5-byte payloads, laptop, single run each), p50 ack latency rose consistently across all three latency scenarios below — two of which, latency_sync_d1ms and latency_batched_d1ms, now push the identical Durability::Durable tier and are kept as separate CI scenario names only so deploy/avg_benchmarks.py and the dated docs/benchmarks/ snapshots keep the columns they've always keyed off; the small gap between their numbers below is run-to-run noise, not a durability difference:

The p99 and mean figures from the same runs swung 10–30% in both directions and are single-run noise on a non-quiesced machine; only the consistent p50 rise across three independent scenarios is signal.

What it buys — and when it does not

Nothing to do with latency. At weir's batch sizes the fsync floor is dominated by per-call cost, not per-byte transfer, so fewer bytes does not mean a faster fdatasync. The benefit is WAB capacity: a given ratio is that much longer a sink outage absorbed in the same disk budget, plus proportionally fewer segment seals.

But because each record is framed independently there is no cross-record dictionary — the ratio depends entirely on redundancy within a single record. Measured at level 1 with the same zstd the daemon uses:

Rule of thumb: below roughly 200 bytes per record, leave it off — the frame overhead cancels or exceeds the gain, and tiny records expand. It pays when individual records are large enough to contain repeated structure, which in practice means batched or nested payloads of several hundred bytes and up. Already-compressed payloads (images, gzipped blobs) never benefit; zstd stores them roughly verbatim, costing CPU for nothing.

Rollback

"none" writes on-disk format v1, byte-identical to weir 1.x. "zstd" writes format v2, which a 1.x daemon refuses to read — it strands those segments loudly rather than misreading them, but they stay stranded until a 2.x daemon reads them. Treat enabling this as a one-way door for the segments written while it is on. Changing the setting does not rewrite existing segments; readers accept both versions.

Observability

weir_wab_record_logical_bytes_total / weir_wab_record_stored_bytes_total is the live ratio. The two are equal when compression is off. See monitoring.

wab_compression_level

• Type: i32
• Default: 1
• Range: 1 – 19
• CLI: --wab-compression-level <n>
• Env: WEIR_WAB_COMPRESSION_LEVEL
• TOML: wab_compression_level

zstd compression level. Only read when wab_compression = "zstd".

Capped at 19 rather than zstd's maximum of 22: levels above 19 are zstd's "ultra" tier, which needs materially more memory to decompress as well as compress — an unpleasant surprise on the drain's read path, and on any tooling reading the segment later.

The default of 1 is deliberate: this codec sits on the ack path, so speed matters more than ratio. Raise it only if you have measured that your records compress meaningfully better at a higher level and that the added ack latency is acceptable.

Not stored on disk — zstd frames are self-describing for decompression — so changing it affects only segments written afterwards, and old segments stay readable.

Connection limits

max_connections

• Type: usize
• Default: 256
• Range: 1–512
• CLI: --max-connections <n>
• Env: WEIR_MAX_CONNECTIONS
• TOML: max_connections

Maximum concurrent producer connections. Enforced by a tokio::sync::Semaphore; connections beyond the cap are dropped immediately (no Nack, the stream is closed).

Hard ceiling: weir uses spawn_blocking to push to the worker queue, so max_connections must be <= the tokio runtime's blocking thread pool size (default 512). Increasing the runtime's pool size beyond 512 requires custom runtime construction, which weir does not currently expose.

When to tune: production deployments behind a load balancer typically run with the default. Single-tenant deployments with very few producers can lower this to free resources; high-fanout deployments can raise it up to the hard ceiling.

connection_read_timeout_secs

• Type: u64 (seconds)
• Default: 30
• Range: 1–600
• CLI: --connection-read-timeout-secs <n>
• Env: WEIR_CONNECTION_READ_TIMEOUT_SECS
• TOML: connection_read_timeout_secs

How long a connection handler may sit in read_exact waiting for the next byte before being dropped. Slowloris guard: without this, a silent or extremely slow client could hold a semaphore permit indefinitely, denying service to legitimate clients.

What triggers the timeout: any of the three read_exact sites in the frame parser (header read, payload read, CRC read) blocked for longer than this value. Drops are silent (no Nack sent — the client isn't reading anyway) and increment weir_connection_idle_timeout_total.

When to tune: lower for high-throughput / short-frame deployments where 30s of silence indicates a problem (5–10s is typical). Raise for deployments with intermittent producers that legitimately go quiet between sends (60–300s).

Operational note: alert on rate(weir_connection_idle_timeout_total[5m]) > 0 to detect potential slowloris activity or buggy clients. The full alert-recipe doc is planned for Phase 2.

Payload limits

max_payload_bytes

• Type: usize
• Default: 16777216 (16 MiB)
• Range: 1 to MAX_PAYLOAD_HARD_CAP (16 MiB, compile-time)
• CLI: --max-payload-bytes <n>
• Env: WEIR_MAX_PAYLOAD_BYTES
• TOML: max_payload_bytes

Maximum size of a single record payload. The hard cap of 16 MiB is compiled into weir-core (MAX_PAYLOAD_HARD_CAP) and cannot be exceeded at runtime; raising the config value above the hard cap is a startup error.

Effect: records larger than this value are rejected with Nack(PayloadTooLarge) before any heap allocation occurs.

When to tune: lower to tighten the DoS surface in environments where you know record sizes are bounded (e.g. structured log records ≤ 64 KiB). Leave at the default for general workloads.

Memory sizing: in-flight payload buffers are sized to the actual frame, but there is no global byte budget across connections, so the worst-case transient receive memory is max_connections × max_payload_bytes (with the defaults, 256 × 16 MiB ≈ 4 GiB if every connection streams a max-size payload at once). On memory-constrained hosts, lower max_payload_bytes and/or max_connections accordingly. See the threat model.

max_batch_records

• Type: usize
• Default: 1024
• Range: 1 to MAX_BATCH_RECORDS_HARD_CAP (2048, compile-time)
• CLI: --max-batch-records <n>
• Env: WEIR_MAX_BATCH_RECORDS
• TOML: max_batch_records

Maximum number of records a single PushBatch frame may declare. Applies only to the optional batch extension; a daemon that never receives a PushBatch is unaffected by this knob.

Effect: a PushBatch declaring more than this is rejected with a Nack before anything is sized by the declared count, and the connection is closed. The rejection happens on the declared number, not on what the frame actually carries — which is the point: record_count is a u16, so a ~320 KiB frame could otherwise declare 65,535 records, all of them targeting the single partition the connection is pinned to.

The 2048 hard cap is compiled into weir-core and cannot be exceeded at runtime; raising the config value above it is a startup error. That number is not arbitrary — it is the largest cap for which the AckBatch reply (3 bytes plus ceil(N/8) of bitmap = 259 bytes) still fits under the 298-byte AckTracked bound weir already publishes, so batching introduces no new largest response and no client gains a new allocation maximum.

When to tune: lower it alongside max_payload_bytes on memory-constrained hosts — the two multiply, since a batch of N records each up to max_payload_bytes is one frame. Raising it toward 2048 trades a larger worst-case frame for fewer round trips.

Metrics endpoint

metrics_port

• Type: u16
• Default: 9185
• Range: 1–65535 (port 0 is rejected at startup with a clear error)
• CLI: --metrics-port <n>
• Env: WEIR_METRICS_PORT
• TOML: metrics_port

TCP port for the Prometheus /metrics HTTP endpoint. This is a distinct knob from the bind host (metrics_bind); the two combine into the listen socket <metrics_bind>:<metrics_port> (default 127.0.0.1:9185).

When to tune: change if 9185 conflicts with another service, or to allow multiple weir instances on one host.

metrics_bind

• Type: IP address (a bare host IpAddr — not host:port)
• Default: 127.0.0.1 (localhost only)
• Validation: parsed as an IpAddr at startup; a value that is not a valid IP address (e.g. a hostname, or a host:port pair) is a startup error.
• CLI: --metrics-bind <addr>
• Env: WEIR_METRICS_BIND
• TOML: metrics_bind

The host address the metrics server binds to. The port is a separate knob, metrics_port — metrics_bind is a host-only address and the two combine into the listen socket (<metrics_bind>:<metrics_port>, default 127.0.0.1:9185). Set the host here (e.g. 0.0.0.0 or a specific interface) and the port in metrics_port; do not put a port in metrics_bind.

The default is localhost-only, so the endpoint is not exposed off-box without an explicit decision. To scrape from another host, sidecar, or container, set 0.0.0.0 (or a specific interface) and restrict access via firewall rules or a --publish 127.0.0.1:9185:9185 port mapping in Docker. A 0.0.0.0 bind is not a security boundary — /metrics is unauthenticated.

metrics_max_connections

• Type: usize
• Default: 8
• Range: 1–1024
• CLI: --metrics-max-connections <n>
• Env: WEIR_METRICS_MAX_CONNECTIONS
• TOML: metrics_max_connections

Maximum number of concurrent scrape connections to the /metrics endpoint. Bounds the resource cost of a misbehaving or hostile scraper; the default of 8 is generous for normal Prometheus scraping.

Shutdown

shutdown_timeout_secs

• Type: u64 (seconds)
• Default: 30
• Range: 1+ (no upper cap)
• CLI: --shutdown-timeout-secs <n>
• Env: WEIR_SHUTDOWN_TIMEOUT_SECS
• TOML: shutdown_timeout_secs

After receiving SIGTERM (or Ctrl-C), how long the daemon waits for in-flight connections to finish before forcibly closing them.

Effect: clean shutdown lets connected producers complete their current Push frame and receive an Ack. Aborted shutdown drops the connection without an Ack — the producer will not know whether the record was committed (it may have been, since the WAB write may already be on disk).

When to tune: deployments with long-running batched producers should raise this to match their batch flush interval. Container orchestrators usually have their own termination-grace setting (Kubernetes terminationGracePeriodSeconds, ECS stopTimeout) — weir's shutdown_timeout_secs should be set ≤ the orchestrator grace minus a few seconds, so the daemon's own cleanup finishes before the orchestrator SIGKILLs.

Dead-letter queue

When the sink permanently rejects a record (the sink classifies it as permanent — e.g. an HTTP 4xx, or a SQL constraint violation), the record is appended to a dead-letter segment under <wab_dir>/dead_letter/. The two options below cap that directory's growth.

Dead-letter vs. stranded — they are different. Exhausting max_retries on a transient failure does not dead-letter; the segment is left in the WAB and re-attempted on the next daemon restart (it increments weir_drain_segments_stranded_total, not the dead-letter metrics). Only permanent rejections land in the dead-letter directory.

Reprocessing dead-lettered records. Use weir-ctl dl requeue once you've fixed the downstream cause. It reads the dead-letter segments under <wab_dir>/dead_letter/ via the shared weir-wab SegmentReader (CRC-verifying every record) and re-submits each record back through the daemon's Unix socket as a normal Push, so the records re-enter the pipeline and the drain re-attempts delivery. The full weir-ctl dl surface is list (count + bytes), drop (delete all), and requeue (re-submit then delete).

# Dry run (default): report what WOULD be requeued; touches nothing, connects to nothing.
weir-ctl dl requeue --wab-dir /var/lib/weir/wab

# Real run: re-submit and delete each segment once all its records are re-accepted.
weir-ctl dl requeue --wab-dir /var/lib/weir/wab --socket /run/weir/weir.sock --yes

• Defaults to a dry run, like dl drop. Without --yes it only counts the recoverable records (and lists any unreadable segments it would skip) and prints what it would do — it does not connect to the daemon or modify any file. --yes performs the real run.
• Flags: --wab-dir (env WEIR_WAB_DIR); --socket (alias --socket-path, default /run/weir/weir.sock) — the daemon socket to push back through; --durability (durable | buffered; sync and batched are accepted as legacy aliases for durable; default batched, i.e. durable) — the durability tier for the re-pushed records; --yes to actually requeue.
• At-least-once, segment-atomic. A segment is deleted only after all of its records are re-accepted (each Push is acked only once the daemon has made it durable), so an interrupted run bounds duplication to the single in-flight segment, which is re-sent on the next run. Duplicate payloads are absorbed by the sink's idempotency key. A corrupt/unreadable segment is skipped wholesale (never partially requeued) and left in place. Records that are requeued but whose segment can't be deleted are surfaced loudly (they will requeue again next run — remove them manually).

docs/monitoring.md references this command in its dead-letter runbook.

weir-ctl segments, dl list, and dl drop read the WAB directory on disk (--wab-dir / WEIR_WAB_DIR), not the daemon socket — they inspect the on-disk segments directly and never connect to a running daemon (unlike health / push, which use --socket, and metrics, which scrapes --addr). dl requeue is the exception that uses both: it reads the dead-letter segments from --wab-dir and then re-pushes them through --socket. Point --wab-dir at the daemon's configured wab_dir.

dead_letter_max_bytes

• Type: u64 (bytes)
• Default: 1073741824 (1 GiB)
• Range: 1+ (a warning is logged below 1 MiB)
• CLI: --dead-letter-max-bytes <n>
• Env: WEIR_DEAD_LETTER_MAX_BYTES
• TOML: dead_letter_max_bytes

Maximum total size of the dead-letter directory. When a new dead-letter write would push the directory over this cap, the drain transitions to BlockedDeadLetterFull and pauses all drain activity (including for healthy records) until an operator removes files or raises the cap.

Why drain blocks entirely: the alternative (drop the new dead-letter record) silently loses data that the sink already classified as un-retriable. Blocking surfaces the problem in metrics (weir_drain_state{state="blocked_dead_letter_full"} = 1) and forces operator attention.

When to tune: bigger for deployments where the operator response loop is slow (alerts → human → mitigation can be 15+ minutes); smaller when the dead-letter directory shares a disk with the WAB and you want a hard cap before WAB writes fail.

Operational note: alert on weir_drain_state{state="blocked_dead_letter_full"} == 1 (the shipped WeirDrainBlocked rule). See monitoring.md for the full alert catalogue and runbook.

dead_letter_check_interval_secs

• Type: u64 (seconds)
• Default: 30
• Range: 1–3600
• CLI: --dead-letter-check-interval-secs <n>
• Env: WEIR_DEAD_LETTER_CHECK_INTERVAL_SECS
• TOML: dead_letter_check_interval_secs

While in BlockedDeadLetterFull, how often the drain wakes to re-scan the dead-letter directory size. The re-scan catches operator- initiated deletions (lower disk usage → unblock) and external growth (some other process writing into the dir → reject the unblock).

When to tune: shorter if you want faster recovery after operator intervention; longer if the dead-letter directory is on slow storage and rescans are expensive (each rescan is a readdir + per-file stat).

health_poll_interval_secs

• Type: u64 (seconds)
• Default: 30
• Range: 1–3600
• CLI: --health-poll-interval-secs <n>
• Env: WEIR_HEALTH_POLL_INTERVAL_SECS
• TOML: health_poll_interval_secs

How often the drain re-probes sink health and rescans for stranded segments to re-drain. The probe runs on a wall-clock cadence, so it fires even while the drain is busy under sustained load (it is not gated on the channel going idle). This is also the upper bound on how long a stranded segment waits before the drain notices the sink recovered.

When to tune: lower it (e.g. 5–10 s) for faster stranded-segment recovery on a flaky uplink; raise it to probe a rate-sensitive sink less often. The health probe itself is bounded by a separate short timeout (5 s, or sink_timeout_secs if shorter), independent of the long commit backstop.

Sink selection

Weir ships with six built-in sinks. The default build compiles in noop, http, mysql, and postgres; clickhouse and s3 are compiled in only when you build with their opt-in Cargo features (clickhouse-sink, s3-sink). A default binary rejects sink_type = "clickhouse" or "s3" with a clear "built without it" error rather than an unknown-value one:

sink_type

• Type: string ("noop", "http", "mysql", "postgres", "clickhouse", or "s3")
• Default: "noop"
• CLI: --sink-type <value>
• Env: WEIR_SINK_TYPE
• TOML: sink_type

When to change: set to "http", "mysql", "postgres", or "clickhouse" once a real downstream is available; leave at "noop" until then.

sink_url

• Type: URL string
• Default: none (required when sink_type is "http", "mysql", "postgres", or "clickhouse")
• Validation: parsed at startup; invalid URLs fail fast.
• CLI: --sink-url <url>
• Env: WEIR_SINK_URL
• TOML: sink_url

For sink_type = "http": the endpoint that receives one POST per record. For sink_type = "mysql": the mysql://user:password@host:port/db connection URL. For sink_type = "postgres": the postgres://user:password@host:port/database connection URL. When the URL carries credentials, prefer setting it via WEIR_SINK_URL rather than the TOML file so the password is not written to disk — sink_url is accepted in TOML, but a credential-bearing URL placed there is stored in plaintext.

Mind the other exposure surfaces: passing --sink-url on the command line puts the credential in the process argv, visible via ps aux to any same-uid process; and even WEIR_SINK_URL is readable via /proc/<pid>/environ or ps eww. The env var is the best of the three (off disk, out of ps aux); restrict who can read the daemon's /proc entry. weir redacts credentials in its own logs and the config-debug line: both the URL userinfo password (scheme://user:<redacted>@host) and the values of known credential query parameters (?password=, ?token=, ?access_token=, ?secret=, ?api_key=/?apikey=, ?sig=/?signature=, ?passwd=/?pwd=) are scrubbed wherever a sink URL is logged, including inside transport-error strings on every retry. This is a defence-in-depth denylist, not a guarantee for arbitrary custom credential parameters — prefer a bearer token (WEIR_SINK_BEARER_TOKEN) or the userinfo form, and treat the /proc/ps surfaces above as the real boundary.

For HTTP, the body is the raw payload bytes; Content-Type is application/octet-stream. The endpoint is expected to return:

• 2xx for accepted records → committed
• 4xx (except 408, 429) for rejected records → dead-lettered
• 408, 429, 5xx for retryable failures → drain retries the whole segment with exponential backoff (up to sink_max_retries)

Network-layer failures (connect refused, DNS failure, timeout) are treated as transient.

The retry budget is configurable via sink_max_retries (default 3, range 0–100) and sink_retry_base_delay_ms (default 100, doubling on each retry) — see their sections below. Once a segment exhausts its retries it is left on disk ("stranded"), increments weir_drain_segments_stranded_total, and is re-drained automatically on the next health poll after the sink recovers (within health_poll_interval_secs), or on the next daemon restart. See monitoring.md.

Health probe (HEAD). weir periodically sends an HTTP HEAD to sink_url to populate weir_sink_health. A reachable endpoint that doesn't implement HEAD — common for POST-only ingest APIs, returning 405 or 501 — or one that issues an auth challenge (401/403) is treated as healthy: records are delivered via POST and the real health signal is commit success, so these do not flap the sink to down. Other 4xx read as degraded; other 5xx or a transport failure read as down. (An endpoint that answers HEAD with 2xx/3xx is simplest.)

Retry-After honoring: when the endpoint returns a transient status (408, 429, or 5xx) with a Retry-After: <seconds> header, the drain uses that value as the next retry delay instead of its exponential-backoff default. Only the delay-seconds form is parsed in v0; HTTP-date form is silently ignored (no header parsed → drain uses its default). The delay is capped at 5 minutes regardless of header value so a misbehaving endpoint can't stall the drain.

Idempotency: the drain guarantees at-least-once delivery per segment, so the endpoint must handle duplicates gracefully. weir helps by sending Idempotency-Key: sha256:<hex> by default (see sink_send_idempotency_key below); endpoints that prefer their own dedup scheme can disable the header.

sink_timeout_secs

• Type: u64 (seconds)
• Default: 10
• Range: 1–300
• CLI: --sink-timeout-secs <n>
• Env: WEIR_SINK_TIMEOUT_SECS
• TOML: sink_timeout_secs

Per-request timeout for the HTTP sink. Applies to the whole request (connect + send + receive). On timeout, the request is classified as a transient transport error and the drain retries.

When to tune: lower if the endpoint is local / sub-millisecond and a stuck request shouldn't block other records; higher for slow endpoints that legitimately take seconds (some logging ingesters do).

sink_max_batch_size

• Type: usize
• Default: 100
• Range: 1–10000
• CLI: --sink-max-batch-size <n>
• Env: WEIR_SINK_MAX_BATCH_SIZE
• TOML: sink_max_batch_size

Maximum records the drain hands to a single Sink::commit call. Its effect depends on the HTTP framing mode (sink_http_batch):

• per-record (none, default): the sink sends one POST per record (up to sink_http_concurrency in flight), so this caps the longest run of POSTs before the drain re-checks its shutdown signal and dead-letter state.
• ndjson: this is the number of records packed into each single newline-delimited POST. Raise it (toward 1000–10000) for high-throughput log/trace shipping, where larger batches amortise the per-POST overhead.

For the SQL sinks it is the number of rows per multi-row INSERT.

When to tune: lower for endpoints that prefer many small calls; raise it for ndjson/SQL bulk throughput. The default 100 is a balanced point for most deployments.

Keep this stable for every dedup-by-batch sink. A batch-scoped dedup key is a hash of the per-commit batch, so changing sink_max_batch_size across a restart re-splits a replayed segment into different batches carrying different keys. The endpoint no longer recognises them as duplicates and you get double-counting on replay. Pick a value and freeze it for the life of the deployment. This applies to:

• ClickHouse — insert_deduplication_token. ClickHouse deduplicates per block, so there is no per-record alternative here.
• HTTP in ndjson batch mode — one POST carries the whole batch, so the Idempotency-Key is the batch's DedupToken, not a per-record key.
• S3 — one object per batch, so the object key names the batch. See the S3 sink guide.
• Postgres / MySQL with no sink_*_id_column — the only thing written is payload bytes, so the UNIQUE constraint has nothing better to sit on.

Exempt — these key on the record, not the batch:

• HTTP in the default per_record batch mode: one POST per record, keyed by that record's RecordId.
• Postgres / MySQL with sink_postgres_id_column or sink_mysql_id_column set: the UNIQUE constraint sits on the record's RecordId.

A RecordId is the record's WAB coordinate — its segment plus its absolute index within that segment — hashed with its bytes, so it does not move when the segment is re-split. A batch-level key cannot be made to do this: it is a function of the batch, and re-batching changes the batch. If a duplicate is expensive, use a sink from the exempt list rather than freezing the knob.

If you need to raise this knob on a deployment that is not exempt, drain the backlog to empty first — a re-split can only double-deliver records that are still on disk waiting to be replayed.

sink_send_idempotency_key

• Type: bool
• Default: true
• CLI: --sink-send-idempotency-key <bool>
• Env: WEIR_SINK_SEND_IDEMPOTENCY_KEY
• TOML: sink_send_idempotency_key

Whether to send Idempotency-Key: sha256:<lowercase-hex> with each HTTP sink request.

Why it matters: the drain's at-least-once delivery contract means records can be re-POSTed if a transient failure mid-batch causes the drain to retry the whole segment. The endpoint needs to dedupe to avoid double-writes. In per-record mode (sink_http_batch = "none", the default) the key is the record's RecordId (weir_sink_sdk::RecordId), derived from the record's WAB coordinate (segment file name + index within it) and its payload — a retried record always presents the same key, but two distinct records never collide just because their payload bytes match. In ndjson mode the key is the batch's DedupToken — a pure hash of the batch's payload bytes (see sink_http_batch).

Format: sha256: prefix + 64-char lowercase hex digest. The prefix lets endpoints distinguish weir's scheme from other key sources and lets us swap algorithms in future (e.g. blake3:...) without breaking parsers.

When to disable: only if the endpoint can't tolerate the extra header (strict CORS, header allow-lists). In that case the endpoint must implement its own dedup — usually by hashing the body server-side.

Changed in 2.0.3. Through 2.0.1, the per-record key was sha256(payload) — a function of the payload bytes only, with no per-record identity. Two legitimately distinct records that happened to have byte-identical payloads (heartbeats, repeated OK/empty bodies, identical log lines, repeated metric samples) produced the same key, so an endpoint that deduped on it accepted the first and silently dropped the rest — collapsing at-least-once into at-most-once for duplicate-payload workloads. The key is now RecordId, which mixes in the record's WAB coordinate, so identical payloads at different coordinates now get different keys. Endpoints that deduplicated on the pre-2.0.3 key will not recognise a retry that spans the upgrade, same as any key-value change.

sink_http_concurrency

• Type: usize
• Default: 8
• Range: 1–1024
• CLI: --sink-http-concurrency <n>
• Env: WEIR_SINK_HTTP_CONCURRENCY
• TOML: sink_http_concurrency

How many per-record POSTs the HTTP sink keeps in flight per commit batch. The drain runs on a single thread, but the POSTs are async, so up to this many overlap their network round-trips instead of paying one RTT per record in series. The protocol is unchanged: still one POST per record, each with its own Idempotency-Key, and dead-lettering stays per-record. Results are committed in submission order.

The overlap is bounded by one commit batch, not by the segment. The drain awaits each batch of sink_max_batch_size records before reading the next records off the segment, so at the defaults a 1000-record segment is 10 sequential batches, each internally ~13 waves of 8 — not one 8-wide pipeline running the length of the segment. Widening the window means raising sink_max_batch_size alongside this knob.

Inert in ndjson mode. With sink_http_batch = "ndjson" the whole batch is a single POST that the drain awaits, so exactly one request is in flight whatever this is set to — the daemon logs concurrency = "n/a (ndjson)" at startup rather than a misleading 8.

When to tune: raise it for high-latency endpoints where the RTT dominates; lower it (or set 1 for fully serial) for endpoints that rate-limit aggressively or can't handle concurrent connections.

sink_http_batch

• Type: string (none | ndjson)
• Default: none
• CLI: --sink-http-batch <mode>
• Env: WEIR_SINK_HTTP_BATCH
• TOML: sink_http_batch

How the HTTP sink frames a commit batch into HTTP requests.

• none (default): one POST per record. Per-record Idempotency-Key, per-record dead-lettering, up to sink_http_concurrency POSTs in flight. Any endpoint that accepts a single-record POST body works — no batch format to agree on.
• ndjson: the whole commit batch (up to sink_max_batch_size records) goes in one POST, newline-delimited (one record per line, Content-Type: application/x-ndjson). This is the framing Loki, the Elasticsearch _bulk API, and similar log/trace ingesters expect. One POST per batch instead of N collapses the round-trip cost for sustained high-throughput forwarding.

What batching buys depends on the endpoint's RTT — and it gives up the per-record concurrency. ndjson trades up to sink_http_concurrency POSTs of one record for one POST of up to sink_max_batch_size records, so the win grows with the endpoint's round-trip time and shrinks toward nothing (or below) against a very fast local one. The only measurement the project has — drain-throughput.md, against a loopback mock answering in microseconds — put ndjson at ~1.25× per-record mode, and that figure is itself disputed: the window was dominated by fsync and retry timing rather than delivery work (see the caveat on that page). Don't size a deployment from it; measure against your own endpoint.

Trade-off (NDJSON is all-or-nothing). The endpoint returns a single status for the whole batch, so there is no per-record outcome: a 2xx commits every record in the batch, a permanent 4xx dead-letters the entire batch, and a transient error (408/429/5xx/transport) retries the whole segment. If you need a single bad record dead-lettered without taking its batch-mates with it, use none.

Records with embedded newlines are dead-lettered in ndjson mode — a \n/\r inside a payload can't be NDJSON-framed (the endpoint would read one record as several). Rather than silently corrupt the batch, weir routes those records straight to the dead-letter store with a clear reason (and logs a warn!); only the clean records are POSTed. If your payloads can contain newlines, use per-record mode (sink_http_batch = "none"), where each record is its own POST body and newlines are fine.

Idempotency in NDJSON mode: a single Idempotency-Key: sha256:<hex> is sent for the whole batch — the batch dedup token, the same primitive the ClickHouse sink passes as insert_deduplication_token.

Changed in 2.0. This key was previously a hash of the joined NDJSON request body. It is now derived from the length-delimited record payloads instead, so the value differs: an endpoint deduplicating on the old key will not recognise a retry that spans the upgrade, and a batch in flight across the upgrade can be delivered twice. Per-record mode is unaffected — it still sends a per-record key.

Because the key is content-derived, it inherits the same batch-stability caveat as ClickHouse's dedup token — see the note under sink_max_batch_size: changing sink_max_batch_size across a restart re-splits a replayed segment into different batches with different keys, so an endpoint deduping on the batch key would not recognise them as duplicates. Keep sink_max_batch_size frozen for the life of the deployment if you rely on batch-level dedup.

sink_max_retries

• Type: u32
• Default: 3
• Range: 0–100
• CLI: --sink-max-retries <n>
• Env: WEIR_SINK_MAX_RETRIES
• TOML: sink_max_retries

How many times the drain retries a segment on a transient sink error (with exponential backoff seeded by sink_retry_base_delay_ms) before it gives up and strands the segment on disk. A stranded segment is durable (no data loss) but undelivered; it is re-drained automatically when the sink health recovers, or on the next daemon restart. 0 strands on the first transient failure. Increments weir_drain_segments_stranded_total on strand.

When to tune: raise it to ride out longer downstream outages without stranding (each extra retry waits up to the backoff cap of 5 minutes); lower it if you'd rather strand quickly and rely on the recovery rescan / restart.

sink_retry_base_delay_ms

• Type: u64 (milliseconds)
• Default: 100
• Range: 1–60000
• CLI: --sink-retry-base-delay-ms <n>
• Env: WEIR_SINK_RETRY_BASE_DELAY_MS
• TOML: sink_retry_base_delay_ms

The first transient-retry backoff delay; it doubles on each subsequent retry (capped at 5 minutes), and a downstream Retry-After header overrides it when present. With the defaults the retry delays are 100 ms → 200 ms → 400 ms.

During shutdown each remaining backoff is clamped to 250 ms, drawn from a 1 s total budget, so a long delay (a big base, or a downstream Retry-After) can't hold the daemon open — while the retries still get real spacing rather than firing back-to-back. With the defaults the clamp is barely visible (100 ms → 200 ms → 250 ms).

WEIR_SINK_BEARER_TOKEN (env-only)

• Type: string
• Default: none
• Env: WEIR_SINK_BEARER_TOKEN
• TOML: deliberately not supported

Optional bearer token sent as Authorization: Bearer <token> with each HTTP-sink request.

Why env-only: bearer tokens are secrets. Putting them in a config file invites accidental commits to git, inclusion in container image layers, and exposure via cat /etc/weir/weir.toml to anyone who can read the file. Env vars don't have those failure modes by default. Containers should source the token from a secrets manager (Kubernetes Secret, AWS Secrets Manager, HashiCorp Vault) into the env at start.

The token is also redacted from HttpSinkConfig's Debug impl so it never reaches a log line via accidental ?config interpolation.

MySQL sink

The MySQL sink writes a whole batch with one multi-row INSERT statement. This is the IOPS-compression sink: N records pushed into the daemon become one prepared statement on one server-side commit.

Schema contract: the sink does not auto-create the target table. Provision it before pointing weir at the database. The minimal table:

CREATE TABLE weir_records (
  id        BIGINT UNSIGNED NOT NULL AUTO_INCREMENT PRIMARY KEY,
  record_id CHAR(64) NOT NULL,
  payload   VARBINARY(16384) NOT NULL,
  ingested_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
  UNIQUE KEY uq_record_id (record_id)
);

…with sink_mysql_id_column = "record_id". The UNIQUE constraint pairs with the default sink_mysql_insert_mode = "ignore": at-least-once retries that re-deliver a record are silently dropped by the server, no consumer-side dedup required.

Do not put the UNIQUE on the payload. Earlier versions of this page recommended UNIQUE KEY uq_payload (payload(255)), which is content identity and loses data: two genuinely distinct records that happen to share their first 255 bytes — a metering "one unit consumed" event, a heartbeat, any fixed-shape event — collide, and INSERT IGNORE discards the second. weir acked it, wrote it to disk and delivered it. record_id is the record's WAB coordinate hashed with its bytes, so it is unique across records with identical bytes and unchanged by a replay — including one that re-splits the segment because sink_max_batch_size changed. See sink_mysql_id_column.

sink_mysql_table

• Type: identifier ([A-Za-z_][A-Za-z0-9_]{0,63})
• Default: "weir_records"
• CLI: --sink-mysql-table <name>
• Env: WEIR_SINK_MYSQL_TABLE
• TOML: sink_mysql_table

Target table for the multi-row INSERT. Validated at startup; characters outside the allowed identifier set fail fast (the sink does not escape identifiers — it validates them, which is a strict subset of MySQL's backtick-quoted form and leaves zero SQL-injection surface).

sink_mysql_column

• Type: identifier (same rules as sink_mysql_table)
• Default: "payload"
• CLI: --sink-mysql-column <name>
• Env: WEIR_SINK_MYSQL_COLUMN
• TOML: sink_mysql_column

Column that receives the payload bytes. Must be a VARBINARY or BLOB column wide enough to hold the largest payload weir accepts (capped elsewhere by max_payload_bytes).

sink_mysql_id_column

• Type: identifier (same rules as sink_mysql_table), optional
• Default: unset — the INSERT writes the payload column only
• CLI: --sink-mysql-id-column <name>
• Env: WEIR_SINK_MYSQL_ID_COLUMN
• TOML: sink_mysql_id_column

Column that receives each record's RecordId as 64 lower-hex characters — the per-record idempotency key. CHAR(64) is the right type. Put the UNIQUE constraint here.

A RecordId is the record's WAB coordinate — the segment it was written to plus its absolute index within that segment — hashed with its bytes. That gives it the two properties a dedup key needs and a content hash cannot have together:

• Unique across records with identical bytes. A payload hash collides on a repeated heartbeat or a repeated metering event, and INSERT IGNORE then discards records weir acked, stored and delivered.
• Unchanged by a replay, including a re-batched one. The index is the record's ordinal in the segment, not in the batch, so it does not move when sink_max_batch_size changes between two delivery attempts. The batch-level DedupToken does move, which is why it cannot do this job.

The one thing it does not survive is a change of coordinate: weir-ctl dl requeue re-pushes records into a new segment, so they arrive as new ids and are inserted again. Both requeue commands say so before asking for confirmation.

Rejected at startup: an id column equal to sink_mysql_column (the generated statement would name one column twice, which MySQL rejects permanently — the drain would dead-letter every batch), and any value failing identifier validation.

Opt-in because turning it on requires a column your table does not have. With it unset every statement weir generates is byte-identical to previous releases.

sink_mysql_insert_mode

• Type: string ("ignore" or "plain")
• Default: "ignore"
• CLI: --sink-mysql-insert-mode <mode>
• Env: WEIR_SINK_MYSQL_INSERT_MODE
• TOML: sink_mysql_insert_mode

How to phrase the INSERT statement.

• "ignore" → INSERT IGNORE INTO .... Duplicate-key errors are silently dropped by the server. The recommended default: pair with a UNIQUE constraint on sink_mysql_id_column so crash-recovery retries are idempotent without consumer-side dedup. A UNIQUE on the payload instead discards distinct records that share bytes.
• "plain" → INSERT INTO .... Duplicates surface as ER_DUP_ENTRY (code 1062) and are classified as transient — the drain retries the segment. Use only if duplicate rows in the target table are tolerable.

Error classification

Credentials and TLS

The MySQL URL contains credentials. Always supply it via WEIR_SINK_URL rather than the TOML file. The MySqlSinkConfig Debug impl redacts the password before logging, but a TOML file on disk has no such protection. weir uses rustls — no system OpenSSL dependency — so TLS to a managed database "just works" with mysql+tls://... URLs.

Sink: Postgres (sink_type = "postgres")

The Postgres sink is the direct counterpart to the MySQL sink — same multi-row INSERT shape, ON CONFLICT DO NOTHING in place of INSERT IGNORE for idempotency, SQLSTATE-based error classification in place of MySQL error codes. Driver is tokio-postgres with a small connection pool (deadpool-postgres, max 4 connections).

Reference schema

CREATE TABLE weir_records (
    id BIGSERIAL PRIMARY KEY,
    record_id CHAR(64) NOT NULL,
    payload BYTEA NOT NULL,
    UNIQUE (record_id)
);

…with sink_postgres_id_column = "record_id". The UNIQUE (record_id) constraint pairs with the default sink_postgres_insert_mode = "on_conflict_do_nothing" so crash-recovery retries are idempotent: duplicate inserts are silently dropped by the server, no consumer-side dedup required.

Do not put the UNIQUE on a payload hash. Earlier versions of this page recommended a generated payload_sha256 BYTEA GENERATED ALWAYS AS (sha256(payload)) STORED with UNIQUE (payload_sha256). That is content identity and it loses data: two genuinely distinct records that happen to carry identical bytes — a metering "one unit consumed" event, a heartbeat, any fixed-shape event — collide, and ON CONFLICT DO NOTHING discards the second. weir acked it, wrote it to disk and delivered it. record_id is the record's WAB coordinate hashed with its bytes, so it is unique across records with identical bytes and unchanged by a replay — including one that re-splits the segment because sink_max_batch_size changed. See sink_postgres_id_column.

Neither column is at-rest tamper-evidence. A GENERATED ALWAYS … STORED hash is recomputed on every UPDATE, so an attacker (or a buggy migration) that rewrites payload also rewrites the hash and leaves the row internally consistent; a record_id written once by weir is likewise just a value a writer with UPDATE can change. Both anchor the bytes at insert time only. Real at-rest tamper-evidence needs an append-only table (revoke UPDATE/DELETE) or an application-maintained hash chain (each row's hash folds in the previous row's), neither of which weir provides — see the threat model ("Signed audit log": records are written verbatim, with no hash chain beyond the per-record wire CRC).

sink_postgres_table

• Type: identifier ([A-Za-z_][A-Za-z0-9_]{0,62}, ≤ 63 chars to fit Postgres's NAMEDATALEN - 1 limit)
• Default: "weir_records"
• Validation: at startup; rejected configurations include illegal characters, leading digits, length > 63.
• CLI: --sink-postgres-table <name>
• Env: WEIR_SINK_POSTGRES_TABLE
• TOML: sink_postgres_table

The strict identifier rule means the sink builds SQL via format! with no escaping logic — there is no SQL injection vector through this knob.

sink_postgres_column

• Type: identifier (same rules as sink_postgres_table)
• Default: "payload"
• CLI: --sink-postgres-column <name>
• Env: WEIR_SINK_POSTGRES_COLUMN
• TOML: sink_postgres_column

Column that receives the payload bytes. Typically BYTEA.

sink_postgres_id_column

• Type: identifier (same rules as sink_postgres_table), optional
• Default: unset — the INSERT writes the payload column only
• CLI: --sink-postgres-id-column <name>
• Env: WEIR_SINK_POSTGRES_ID_COLUMN
• TOML: sink_postgres_id_column

Column that receives each record's RecordId as 64 lower-hex characters — the per-record idempotency key. CHAR(64) is the right type. Put the UNIQUE constraint here.

A RecordId is the record's WAB coordinate — the segment it was written to plus its absolute index within that segment — hashed with its bytes. That gives it the two properties a dedup key needs and a content hash cannot have together:

• Unique across records with identical bytes. A payload hash collides on a repeated heartbeat or a repeated metering event, and ON CONFLICT DO NOTHING then discards records weir acked, stored and delivered.
• Unchanged by a replay, including a re-batched one. The index is the record's ordinal in the segment, not in the batch, so it does not move when sink_max_batch_size changes between two delivery attempts. The batch-level DedupToken does move, which is why it cannot do this job.

The one thing it does not survive is a change of coordinate: weir-ctl dl requeue re-pushes records into a new segment, so they arrive as new ids and are inserted again. Both requeue commands say so before asking for confirmation.

With this set each row takes two bind parameters instead of one. Postgres's wire protocol caps a statement at 65 535 parameters and sink_max_batch_size is already validated to 1..=10000, so the worst case is 20 000 — no new limit.

Rejected at startup: an id column equal to sink_postgres_column (the generated statement would name one column twice, which Postgres rejects as 42701 — permanent, so the drain would dead-letter every batch), and any value failing identifier validation.

Opt-in because turning it on requires a column your table does not have. With it unset every statement weir generates is byte-identical to previous releases.

sink_postgres_insert_mode

• Type: string ("on_conflict_do_nothing" or "plain")
• Default: "on_conflict_do_nothing"
• CLI: --sink-postgres-insert-mode <mode>
• Env: WEIR_SINK_POSTGRES_INSERT_MODE
• TOML: sink_postgres_insert_mode

Error classification

Credentials and TLS

The Postgres URL contains credentials. Always supply it via WEIR_SINK_URL rather than the TOML file. The PostgresSinkConfig Debug impl redacts the password before logging.

TLS support: opt-in via ?sslmode=require in the URL. The Postgres sink uses tokio-postgres-rustls (webpki-roots, ring) when the URL's sslmode query parameter is require, and NoTls otherwise. Opt-in semantics rather than auto-detect: an upgrade does not silently enable TLS on a previously-cleartext deployment.

# Cleartext (default; safe for private networks):
WEIR_SINK_URL=postgres://user:pw@10.0.0.5:5432/weir

# TLS required:
WEIR_SINK_URL=postgres://user:pw@db.example.com:5432/weir?sslmode=require

Cert verification uses the Mozilla root store bundled via webpki-roots — no host-system CA configuration required. The ring crypto provider is explicitly selected (the same one weir uses for mysql+tls and for outbound reqwest TLS), keeping the binary's code-signing surface at one provider.

For cleartext deployments without operator control over sslmode (e.g. a managed Postgres URL that omits it), the tokio-postgres default sslmode=prefer is treated as cleartext — sslmode=require is the explicit opt-in. A TLS-terminating proxy (stunnel, ghostunnel, pgbouncer with TLS upstream) remains a valid option for deployments that need certificate-pinning or client-auth — neither is wired up by the in-process connector yet.

Sink: ClickHouse (sink_type = "clickhouse")

Requires the clickhouse-sink build feature. The sink sends one HTTP INSERT INTO {database}.{table} ({column}) FORMAT RowBinary request per batch to ClickHouse's HTTP interface (default :8123), with the batch encoded as length-prefixed bytes into a single String column. N records → one request → one ClickHouse block — the same IOPS compression as the SQL sinks. sink_url carries http://[user:password@]host:8123 (credentials sent as HTTP basic auth, redacted in logs).

sink_clickhouse_database

• Type: string · Default: default
• CLI: --sink-clickhouse-database · Env: WEIR_SINK_CLICKHOUSE_DATABASE · TOML: sink_clickhouse_database

sink_clickhouse_table

• Type: string · Default: weir_records
• CLI: --sink-clickhouse-table · Env: WEIR_SINK_CLICKHOUSE_TABLE · TOML: sink_clickhouse_table

sink_clickhouse_column

• Type: string · Default: payload
• CLI: --sink-clickhouse-column · Env: WEIR_SINK_CLICKHOUSE_COLUMN · TOML: sink_clickhouse_column

Idempotency (dedup token)

The drain is at-least-once per segment, so a crash mid-commit replays the batch. ClickHouse has no ON CONFLICT; instead the sink sends a deterministic insert_deduplication_token = sha256(batch) on each insert, so a replayed byte-identical batch is deduplicated by ClickHouse provided the target table uses a dedup-capable engine — a Replicated*MergeTree (where insert_deduplicate is on by default) or a MergeTree with non_replicated_deduplication_window set. Mind the dedup window (default last ~100 blocks). If the engine isn't dedup-capable, the token is harmless and dedup falls back to your table design.

Reference schema:

CREATE TABLE weir_records (payload String)
ENGINE = MergeTree ORDER BY tuple()
SETTINGS non_replicated_deduplication_window = 100;

Error classification

Logging

log_level

• Type: string
• Default: "info"
• Valid values: trace, debug, info, warn, error, or any tracing-subscriber::EnvFilter directive (e.g. "weir_server=debug,info" to set per-module levels).
• CLI: --log-level <level>
• Env: WEIR_LOG_LEVEL
• TOML: log_level

Note: RUST_LOG is not consulted — the subscriber's EnvFilter is built only from the resolved log_level (CLI > WEIR_LOG_LEVEL > TOML > info). Set the level through one of those, not RUST_LOG.

The tracing-subscriber EnvFilter directive used to initialise the logging subscriber.

Effect: info is the production default — startup, shutdown, recovery events, and warnings are visible; per-record activity is not. debug adds per-connection events and queue activity; trace adds per-frame parser events.

When to tune: debug while diagnosing a problem; back to info once resolved. trace is verbose enough to materially impact throughput; don't leave it on in production.

Sink: S3 (sink_type = "s3")

Requires the opt-in s3-sink feature. The full guide, including the object key scheme, IAM policy, provider notes and how to point Athena or DuckDB at the bucket, is in the S3 sink page; this section is the key reference.

These are file/env only — there are no CLI flags for them.

sink_s3_bucket

• Type: string
• Default: (none — required when sink_type = "s3")
• Env: WEIR_SINK_S3_BUCKET
• TOML: sink_s3_bucket

Target bucket. Startup fails with a named error if unset.

sink_s3_region

• Type: string
• Default: us-east-1
• Env: WEIR_SINK_S3_REGION

SigV4 signing region, and the AWS endpoint's region when sink_s3_endpoint is unset. MinIO ignores it but still signs with it, so it must match on both sides.

sink_s3_endpoint

• Type: string
• Default: (empty — the AWS regional endpoint)
• Env: WEIR_SINK_S3_ENDPOINT

Scheme and host, optionally with a port. No path component — startup rejects one, because Endpoint::resolve treats everything after :// as the authority, so a path would be signed and sent differently and every request would fail SignatureDoesNotMatch with nothing naming the cause.

sink_s3_force_path_style

• Type: bool
• Default: false
• Env: WEIR_SINK_S3_FORCE_PATH_STYLE

Puts the bucket in the path (host/bucket/key) rather than the hostname (bucket.host/key). Set true for MinIO and most self-hosted gateways.

sink_s3_prefix

• Type: string
• Default: (empty)
• Env: WEIR_SINK_S3_PREFIX

Key prefix, no leading /. Rejected at startup if it contains . or .. segments, an interior //, #, ?, a backslash, or a control character — a URL parser rewrites or truncates all of those, which would make weir sign one key and write another.

sink_s3_partition

• Type: string
• Default: dt=%Y-%m-%d/hour=%H
• Env: WEIR_SINK_S3_PARTITION

Hive-style partition template. Only %Y, %m, %d and %H are recognised; anything else is literal, and an unknown %X is rejected at startup rather than passed through. "" disables partitioning.

Always rendered in UTC, and not configurable. The value comes from the WAB segment header, so it survives a replay; rendering it in local time would make the key depend on the host's TZ, and a restart across a DST change would duplicate objects.

sink_s3_framing

• Type: string — ndjson | length-prefixed
• Default: ndjson
• Env: WEIR_SINK_S3_FRAMING

ndjson writes one record per line, which every common query engine reads directly. A record containing \n or \r cannot be represented that way and is dead-lettered rather than written — framing it anyway would split one record into two lines downstream. length-prefixed (a u64 LE length before each record) is binary-safe and dead-letters nothing on content, but needs a weir-aware reader; those objects use the .weirbin extension.

sink_s3_compression

• Type: string — none | zstd | gzip
• Default: zstd
• Env: WEIR_SINK_S3_COMPRESSION

Applied to the whole framed body, so an object is a compressed NDJSON file — what query engines expect. Athena and Spark read both zstd and gzip.

sink_s3_access_key_id / sink_s3_secret_access_key

• Type: string
• Default: (empty — use the environment, IRSA, or an instance role)
• Env: WEIR_SINK_S3_ACCESS_KEY_ID, WEIR_SINK_S3_SECRET_ACCESS_KEY

Static credentials. Set both or neither — startup rejects one without the other, because a lone key id would silently fall through to the environment or an instance role and authenticate as something other than what the config names. The secret is redacted in every diagnostic that formats the config.

sink_s3_storage_class / sink_s3_sse / sink_s3_sse_kms_key_id

• Type: string
• Default: (empty — the header is omitted)
• Env: WEIR_SINK_S3_STORAGE_CLASS, WEIR_SINK_S3_SSE, WEIR_SINK_S3_SSE_KMS_KEY_ID

Become x-amz-storage-class, x-amz-server-side-encryption and x-amz-server-side-encryption-aws-kms-key-id. A rejected value here produces a 400 that weir treats as transient, so a typo strands the backlog rather than dead-lettering it.

Sizing: raise sink_max_batch_size for this sink. One object per commit batch means the default of 100 produces roughly 50 KB objects for a 500-byte record — hundreds of thousands per hour for a busy producer, expensive to PUT and pathological for every query engine. Use 10000. The daemon warns at startup below 1000.

And then never change it. For S3 the batch boundary determines the object key, so changing sink_max_batch_size after records have been written makes replayed batches land under new keys and duplicate. This is stronger than the stability rule the other dedup-by-batch sinks carry: constant for the life of the bucket, not merely across a restart.

Example minimal config

The smallest useful TOML (everything else defaults):

[server]
wab_dir = "/var/lib/weir/wab"

The daemon will:

• Listen on /run/weir/weir.sock
• Use /var/lib/weir/wab for WAB segments
• Serve metrics on 127.0.0.1:9185 (localhost only; set metrics_bind to expose)
• Use the (256, 1ms) batch defaults
• Cap connections at 256, payload at 16 MiB
• Drop idle connections after 30s
• Use the noop sink (accepts all records, forwards nothing)
• Run with log level info

For container deployments, the same config can come entirely from env vars with no file:

WEIR_WAB_DIR=/var/lib/weir/wab \
weir-server

Example production config

A more typical production TOML, with comments explaining each deviation:

[server]
# Standard paths.
socket_path = "/run/weir/weir.sock"
wab_dir     = "/var/lib/weir/wab"

# Two shards on this NVMe. Set this from your own measurement, not from this
# file: shard_count is non-monotonic (see its caveat above), and the only sweep
# weir publishes found *fewer* agents faster on a small host
# (benchmarks/agent-count-tuning.md). No published weir number supports a gain
# for two shards over one.
shard_count  = 2
worker_count = 4

# Latency-tuned: small batches, tight deadline.
batch_size        = 64
batch_deadline_ms = 1

# Match the LB connection cap.
max_connections              = 384
connection_read_timeout_secs = 10

# This deployment uses structured log records; cap accordingly to limit DoS surface.
max_payload_bytes = 1048576  # 1 MiB

# Metrics on a non-standard port to avoid colliding with node_exporter.
metrics_port = 9186

# Long-running batch producers; match the orchestrator's grace period (45s).
shutdown_timeout_secs = 40

# Bigger dead-letter cap; operator response loop can be 30+ minutes.
dead_letter_max_bytes           = 5_368_709_120  # 5 GiB
dead_letter_check_interval_secs = 60

# Forward records to an internal ingest endpoint via HTTP.
# Bearer token comes from $WEIR_SINK_BEARER_TOKEN in the unit's
# Environment= file, not from this config.
sink_type           = "http"
sink_url            = "https://ingest.internal.example.com/weir"
sink_timeout_secs   = 5
sink_max_batch_size = 200

log_level = "info"

TCP + mutual TLS

Requires --features tls — the tls feature on weir-server (and weir-client) is off by default for a plain cargo build; a binary built without it has no TCP listener and the five keys below are not recognised, and enabling tcp_bind without the feature is a startup error. This does not mean operators must build from source: as of 2.0.3, the official release binaries (Linux/macOS) and the official Docker image are already built with --features tls. There is no Windows binary — the listener layer is Unix-only, so weir-server has no ingest path there at all, and 2.0.5 stopped shipping the .exe that had implied otherwise. A Windows producer is supported and talks to a Unix daemon over this listener using weir-client --features tls; see Platform support.

TLS is mandatory on the TCP path. Setting tcp_bind without a valid TLS configuration (or without building with --features tls) is a fatal startup error — weir never exposes a plaintext TCP socket. The three cert-path keys are required whenever tcp_bind is set; omitting any one of them is a startup error.

The Unix and TCP listeners share one global connection semaphore sized max_connections. The total concurrent connections across both transports is bounded by max_connections, not 2×. Tune accordingly.

For a full operator guide including CA setup, cert rotation, and monitoring, see TCP + mutual TLS.

tcp_bind

• Type: socket address, e.g. 0.0.0.0:7100 or [::]:7100
• Default: none (TCP listener disabled; Unix socket only)
• CLI: --tcp-bind <addr>
• Env: WEIR_TCP_BIND
• TOML: tcp_bind

The address and port for the TCP listener. When unset (the default), the daemon operates Unix-socket-only and the four TLS keys below are ignored. When set, all four TLS keys become required; missing any one is a startup error.

Operational notes:

• Bind to 0.0.0.0:7100 to accept from any interface, or to a specific interface address to limit exposure. Restrict inbound access with host firewall rules independent of the bind address.
• The TCP listener runs concurrently with the Unix socket and feeds the same pipeline.
• The connection cap is shared with the Unix listener; see max_connections.

tls_cert_path

• Type: absolute path to a PEM-encoded TLS server certificate (may be a full chain)
• Default: none (required when tcp_bind is set)
• CLI: --tls-cert <path>
• Env: WEIR_TLS_CERT
• TOML: tls_cert_path

Path to the server's TLS certificate file. The file must be readable by the daemon at startup and on every SIGHUP reload.

When to tune: update the path when rotating to a new certificate chain. After updating, send SIGHUP to reload TLS material without dropping connections (see SIGHUP cert rotation in TCP + mutual TLS).

tls_key_path

• Type: absolute path to the PEM-encoded private key for tls_cert_path
• Default: none (required when tcp_bind is set)
• CLI: --tls-key <path>
• Env: WEIR_TLS_KEY
• TOML: tls_key_path

Path to the private key that pairs with tls_cert_path.

Operational notes:

• Restrict this file to mode 0o400 owned by the daemon's user. The daemon reads it at startup and on SIGHUP; no other process should be able to read it.
• Supply via env var (WEIR_TLS_KEY) in container environments where secrets management injects a file path at runtime.

tls_client_ca_path

• Type: absolute path to a PEM-encoded CA certificate
• Default: none (required when tcp_bind is set)
• CLI: --tls-client-ca <path>
• Env: WEIR_TLS_CLIENT_CA
• TOML: tls_client_ca_path

Path to the Certificate Authority that signs client certificates. Every TCP client must present a certificate signed by this CA during the TLS handshake. Anonymous or cert-less clients are rejected at the handshake level (weir_tls_handshake_failures_total{reason="no_client_cert"}).

The trust model is CA-issuance: issuing a client cert from this CA is the act of authorizing that producer. To revoke a client, rotate the CA (CRL/OCSP are out of scope; see TCP + mutual TLS for the rationale).

When to tune: when rotating the client CA as part of a revocation or re-keying procedure. Update the path and send SIGHUP.

tls_handshake_timeout_secs

• Type: u64 (seconds)
• Default: 10
• Range: 1+ (no upper cap)
• CLI: --tls-handshake-timeout-secs <n>
• Env: WEIR_TLS_HANDSHAKE_TIMEOUT_SECS
• TOML: tls_handshake_timeout_secs

Maximum time the daemon waits for a TLS handshake to complete on a new TCP connection. Slowloris guard for the TLS path: a TCP client that opens a connection but stalls during the handshake holds a semaphore permit (from max_connections) for at most this many seconds before being dropped. Drops increment weir_tls_handshake_failures_total{reason="timeout"}.

The semaphore permit is acquired before the handshake begins, so a flood of stalled TCP connections is bounded by max_connections regardless of handshake progress.

When to tune: lower for high-throughput deployments where a 10s stall indicates a problem; raise if clients are on high-latency links where legitimate handshakes approach the default. The existing per-frame connection_read_timeout_secs continues to apply after the handshake completes.

TLS metrics

Two metric families are added by the tls feature:

Alert on rate(weir_tls_handshake_failures_total[5m]) > 0 (especially reason="no_client_cert" or reason="bad_cert") to detect unauthorised connection attempts. Alert on weir_tls_config_reloads_total{outcome="failed"} to detect cert rotation failures.

Limitations

• No hot reload (except TLS material): config is read once at startup. SIGHUP reloads TLS cert/key/CA only — all other configuration changes require a daemon restart.
• No per-shard tuning: batch_size and batch_deadline_ms apply uniformly to every shard. Workloads where shards have very different load profiles cannot tune them independently.
• No CLI override of [server] table name: the TOML file must use [server] exactly; this is not configurable.

See also

• Quickstart — the fastest path to a running daemon, with a minimal config inline.
• Tuning guide — planned for Phase 2; operator-facing guide on picking values for your workload (vs the data dump in batch-tuning).
• Observability — planned for Phase 2; metrics catalogue with alert thresholds and Grafana dashboard JSON.
• Batch tuning data — the empirical sweep behind the (256, 1ms) default.
• deploy/docker/weir.toml.example — every option with its default and matching env var, kept in sync with this reference.