WAB — Write-Ahead Buffer Format

Overview

The WAB stores records as a sequence of segment files in per-shard directories under the WAB root. Each segment is an append-only binary file that is atomically renamed from .wab (active) to .wab.sealed when full or on shutdown. A .wab.confirmed sidecar is written by the drain after a segment is fully forwarded to the sink.

The daemon does not create the WAB root directory. The operator must create it before starting the daemon:

mkdir -p /path/to/wab && chmod 700 /path/to/wab

Directory layout

/path/to/wab/
├── shard_00/
│   ├── seg_00000003.wab            ← active (being written)
│   ├── seg_00000002.wab.sealed     ← sealed, awaiting drain
│   └── seg_00000001.wab.confirmed  ← drained; the .wab.sealed file was DELETED
├── shard_01/
│   └── ...
├── dead_letter/
│   └── dl_00000001.wab.sealed      ← permanently-rejected records
└── quarantine/
    ├── shard_00__seg_00000004.wab         ← mid-file-corrupt ACTIVE segment (crash recovery)
    └── shard_01__seg_00000002.wab.sealed  ← corrupt SEALED segment (the drain, or a rejected .confirmed)

Shard directories, the dead_letter/ directory, and the quarantine/ directory are created with mode 0o700. Segment files (including .confirmed sidecars and dead-letter segments) are created with mode 0o600. Permissions are set atomically at creation time via OpenOptionsExt::mode and DirBuilderExt::mode — there is no post-creation chmod.

The .wab.sealed and .wab.confirmed files do not normally coexist. On a healthy daemon the drain, after committing a sealed segment to the sink, writes the .wab.confirmed sidecar and then deletes the .wab.sealed segment (see Post-drain steady state). So a .confirmed sidecar usually has no surviving segment file beside it. The tree above shows the transient states a single segment passes through (seg_00000001 confirmed, seg_00000002 sealed-but-not-yet-drained, seg_00000003 active) — not three files for one segment.

Reserved subdirectories

Two subdirectories of the WAB root are not shards and are skipped by recovery's shard scan:

dead_letter/ — records the sink permanently rejected. "Permanent" means the sink returned a non-retryable rejection (e.g. an HTTP sink replying 4xx other than 408/429) — not that the sink was unreachable. A transient failure (sink down/unreachable, 5xx, timeout, transport error) is not dead-lettered: the whole segment is retried and, once its retry budget is exhausted, left on disk as stranded (it stays in its shard dir as seg_*.wab.sealed, surfaces via weir_drain_segments_stranded_total, and auto-resumes when the sink recovers). So a recovery tool looking for un-delivered data after a sink outage should look at stranded sealed segments in the shard dirs, not dead_letter/. Files are named dl_NNNNNNNN.wab.sealed (zero-padded u64 counter starting at 1), written by DeadLetterWriter (crates/weir-server/src/drain/dead_letter.rs). A crash mid-write can leave an orphan active partial dl_NNNNNNNN.wab (no .sealed), which recovery does not touch and the next DeadLetterWriter::open accounts for. Dead-letter files use the same WEIR segment format as the main WAB (header + records + sentinel + footer), so they are readable by the shared weir-wab SegmentReader with no separate parser — except their header carries the reserved shard id 0xFFFF as a dead-letter marker. weir-ctl dl requeue reads them this way.
quarantine/ — segments recovery could not safely recover (bad magic, unknown version, CRC mismatch), plus a .confirmed sidecar that fails its own verification during sealed-segment replay. The original file is renamed (mid-file-corrupt active segments are copied, so the valid prefix can still be truncated+sealed in place) here with a flattened name that prefixes the source shard directory and joins it with the original file name by a double underscore: <shard_name>__<file_name>, e.g. shard_00__seg_00000004.wab.sealed (see quarantine() / copy_to_quarantine() in crates/weir-server/src/wab/recovery.rs). The shard prefix prevents same-named segments from different shards clobbering each other once flattened into one directory; an exact-name collision appends .1, .2, … Quarantined files retain whatever extension they had: .wab when crash recovery preserved an active segment (the mid-file-corruption case — the one where acked records sit after the corruption), .wab.sealed when the drain (or a rejected .confirmed sidecar) preserved a sealed one, or .wab.confirmed for a sidecar that failed verification — the corresponding .wab.sealed copy sits beside it.

Reading a quarantined segment. A quarantined segment preserves the corrupt record and the (valid) records after it — the whole file is moved (or copied) untouched, nothing is truncated. weir-wab's SegmentReader stops at the first CRC error by contract: it yields every good record up to the corruption, then a single Err (for a CRC mismatch, io::ErrorKind::InvalidData), and every subsequent call returns None (see the iteration contract in crates/weir-wab/src/lib.rs) — so a plain SegmentReader recovers only the prefix before the bad record, by design (recovery and the drain depend on that stop-at-first-error contract).

To reach the records after the corruption, use weir_wab::RecoveryReader instead — a separate, forensic iterator built for exactly this file. It continues past a corrupt record (resync is free: SegmentReader::next already reads the full payload before verifying the CRC, so a failure leaves the reader positioned at the next record's boundary with no seek needed) and never fabricates data: an implausible declared length, or a run of consecutive verification failures, ends iteration with Desynced rather than guessing. weir-ctl quarantine list | inspect | requeue is built on it and is the operator-facing way to do this — see the quarantine recovery runbook for the triage procedure, including how to read RecoveryReader's "clean end" (every byte accounted for, not every record recovered) and why requeue re-sends the already-delivered prefix.

Segment file format (versions 1 and 2)

Header (24 bytes)

Offset  Size  Field       Description
──────  ────  ──────────  ──────────────────────────────────────
 0       4    magic       b"WEIR"
 4       1    version     1 or 2
 5       1    flags       v1: MUST be 0x00
                          v2: bit 0 = ZSTD; bits 1-7 reserved, MUST be 0
 6       2    shard_id    u16 little-endian
 8       8    created_at  Unix timestamp, nanoseconds, i64 LE
16       8    reserved    0x00 (padding to 24 bytes)

Format v2 — what changed, and what did not

v2 differs from v1 in one byte. Byte [5], reserved and zero-on-write since v1, is now a flags byte whose bit 0 means every record in this segment is stored as a complete zstd frame. The record framing, the end-of-records sentinel, the footer, and the .confirmed sidecar are byte-for-byte identical between the two versions.

The per-record CRC covers the STORED bytes — the zstd frame, in a compressed segment — not the logical payload. That is deliberate and it is what keeps the change small: crash recovery's torn-tail scan, verify_sealed_segment's whole-file CRC, and the footer's record_count / data_bytes cross-check all work without decompressing anything. Decompression happens in exactly one place, SegmentReader::next, after the CRC has already passed.

The footer's data_bytes is likewise stored bytes, so it is the post-compression size in a v2 ZSTD segment. For logical byte totals, read the weir_wab_record_logical_bytes_total metric.

Which version gets written

A segment is written as v1 whenever compression is off, and v2 only when it is on. With wab_compression = "none" — the default — a weir 2.0 daemon produces segments byte-identical to what 1.x wrote.

That makes upgrade-then-rollback a non-event at the default setting. Enabling compression is a one-way door: a 1.x reader rejects a v2 header outright (InvalidData), which the drain treats as "preserve for retry" and recovery quarantines — the data is stranded and loud, never misread and silent — but it stays stranded until a 2.x daemon reads it.

Reader rules

Header	Behaviour
v1, `flags == 0`	Accept, no compression
v1, `flags != 0`	Reject — v1 documented the byte as zero, so a set bit is corruption
v2, only known flag bits	Accept
v2, any unknown flag bit	Reject — written by a newer weir; reading it would misinterpret the records
version 0, or > 2	Reject — unknown format version

Rejecting an unknown flag bit rather than ignoring it is the point: a future weir that adds bit 1 (say, encryption) must not have its segments silently read as plaintext by this build.

Record size caps under compression

A compressed record's stored length can legitimately exceed MAX_PAYLOAD_HARD_CAP (16 MiB), because zstd's worst case expands slightly. So a v2 ZSTD segment accepts stored lengths up to compress_bound(MAX_PAYLOAD_HARD_CAP) = n + n/255 + 16, while v1 keeps the original cap unchanged. The plaintext is still capped at 16 MiB, enforced on the decompression output — which doubles as the decompression-bomb guard, and is an exact bound rather than a guess because the real cap is known.

Measured, on genuinely random 16 MiB input: the stored form was 394 bytes larger than the input, against an allowance of 65,809 bytes.

Records (variable, repeated)

Offset  Size       Field        Description
──────  ─────────  ───────────  ──────────────────────────────────
 0       4         payload_len  u32 little-endian
 4       4         crc32        CRC32 of the payload bytes, u32 LE
 8       payload_len  payload   raw payload bytes

A payload_len of 0 is the end-of-records sentinel (not a valid record).

payload_len is checked against MAX_PAYLOAD_HARD_CAP (16 MiB, defined in weir-core) before any heap allocation during both write and recovery replay.

Sentinel (4 bytes)

0x00 0x00 0x00 0x00 — marks the end of the record sequence.

Offset  Size  Field         Description
──────  ────  ────────────  ──────────────────────────────────────
 0       8    record_count  u64 little-endian
 8       8    data_bytes    u64 LE — total STORED payload bytes, post-compression,
                            excluding per-record overhead (see below)
16       4    file_crc32    CRC32 of all bytes before sentinel, u32 LE
20       8    sealed_at     Unix timestamp, nanoseconds, i64 LE
28       4    reserved      0x00

file_crc32 is accumulated by a running crc32fast::Hasher during writes, so no full-file re-read is needed at seal time. During crash recovery the running hasher is unavailable; the CRC is recomputed from scratch over the replayed records.

`.confirmed` sidecar format (36 bytes)

Written by the drain after a segment's records have been forwarded to the sink. A segment with a valid .confirmed file is skipped on startup replay.

Offset  Size  Field         Description
──────  ────  ────────────  ──────────────────────────────────────
 0       4    magic         b"WCON"
 4       1    version       1
 5       3    reserved      0x00 — zero on write; reserved for future flags
 8       8    sealed_at     i64 LE — copied from segment footer
16       8    record_count  u64 LE — copied from segment footer
24       8    drained_at    i64 LE — Unix timestamp, nanoseconds
32       4    crc32         CRC32 of bytes [0..32], u32 LE

The b"WCON" magic is distinct from the segment magic b"WEIR" so a misplaced segment file cannot be parsed as a confirmation file.

The sidecar's name is derived from the sealed segment by swapping the extension: seg_NNNNNNNN.wab.sealed → seg_NNNNNNNN.wab.confirmed (confirmed_path_for in crates/weir-wab/src/format.rs strips .wab.sealed and appends .wab.confirmed).

Recovery behaviour:

Magic, version (strict equality), and CRC32 all pass → skip replay of the corresponding sealed segment.
Wrong magic or CRC mismatch → quarantine both the sidecar and the sealed segment; log with specific failure reason.
Unknown version → quarantine both and log: "unknown confirmation format version N; cannot safely determine drain status — treating as unconfirmed would risk double-drain, quarantining instead."
Missing .confirmed → not an error; segment was not drained before crash. Replay normally.

Post-drain steady state

On a healthy daemon a sealed segment does not linger on disk. After the drain commits a segment to the sink, confirm_and_delete (crates/weir-server/src/drain/confirmed.rs) does two things, in order:

Writes the .wab.confirmed sidecar durably (content + directory entry fsynced).
Deletes the .wab.sealed segment file (std::fs::remove_file).

So in steady state a .wab.confirmed sidecar frequently has no surviving .wab.sealed segment beside it — the segment was deleted the moment delivery was confirmed. A tool that derives a sealed path from a .confirmed sidecar and opens it will usually hit ENOENT. Treat the .confirmed sidecar as a tombstone ("this segment was delivered and removed"), not as a pointer to a still-present segment.

The delete is deliberately second: if the .confirmed write fails, the segment is preserved (no .confirmed, segment still on disk) so recovery re-drains it on the next restart — a duplicate absorbed by the at-least-once + dedup contract, never a deleted-but-unconfirmed segment. The only window where a .sealed and its .confirmed coexist is a crash between step 1 and step 2; recovery resolves it by skipping replay (the .confirmed is valid) and the orphaned .sealed is left for the operator.

Crash recovery algorithm

On startup, before accepting connections, the daemon runs recovery in the calling thread:

Enumerate shards by directory scan, not by config. Recovery read_dirs the WAB root, skips the reserved dead_letter/ and quarantine/ subdirectories, and treats every remaining directory as a shard (recover_open_segments in crates/weir-server/src/wab/recovery.rs; the sealed-segment replay scan in crates/weir-server/src/wab/mod.rs skips the same two subdirectories) — it does not reconstruct the shard set from the shard_count config, so segments from a previous run with a different shard_count are still recovered.
Scan each shard directory for .wab (active) files.
For each active segment: a. Validate header magic and format version. Bad magic or unknown version → quarantine. b. Replay records from the header boundary, verifying per-record CRC32. c. At the first corrupt or incomplete record, record the last valid offset. d. Truncate the file at the last valid offset. e. Write sentinel + footer (record count, data bytes, running file CRC, timestamp). f. sync_all() (full fsync including metadata). g. Atomically rename .wab → .wab.sealed.
Replay sealed segments: for each .wab.sealed without a valid .wab.confirmed, send the path to the drain channel.

Key invariant: only records up to the first corrupt entry are replayed. A torn write at crash time never silently corrupts the replay stream — the trailing corrupt record is truncated, not forwarded.

Segment rotation

A segment rotates (should_rotate() == true) when bytes_written >= the configured segment_max_bytes (default 256 MiB via SEGMENT_MAX_BYTES; tunable from 4 KiB upward — see configuration.md). Rotation happens inline in ShardWriter::write_record; the write that pushes the segment past the threshold also triggers the seal. The next write opens a new segment lazily.

Naming convention

seg_{counter:08}.wab          active
seg_{counter:08}.wab.sealed   sealed, awaiting drain
seg_{counter:08}.wab.confirmed  drain-confirmed

counter is a monotonically increasing u64, zero-padded to 8 digits, starting at 1. On startup, ShardWriter::scan_and_advance_counter advances next_counter past the highest existing counter to prevent collisions with existing sealed segments.

Security model

CRC32 detects accidental corruption (bit rot, partial writes, torn records). It does not detect malicious modification: a forged WAB segment with a valid CRC32 will be accepted by recovery.

The trust boundary is the WAB directory:

The directory must be on local storage accessible only to the daemon process.
Directory permissions are 0o700; segment files are 0o600.
If the WAB is on a network filesystem or shared storage, the security model does not hold — another process could overwrite or forge segments.

This is an explicit assumption, not a weakness to be fixed. Operators who require tamper detection must add it at the storage layer (e.g. dm-verity, ZFS checksums with access control).