Monitoring weir
weir exposes Prometheus metrics at /metrics. This page is the operator's
companion: how to wire up monitoring, what each alert means and how to respond,
and a reference for every metric.
The one thing to internalise: weir is fsync-bound. weir_wab_fsync_duration_seconds
is the dominant latency signal in any durable-write deployment — fsync is
~90–98% of Durable-tier latency, the share rising as the disk gets slower
(measured: Mac NVMe ~150 µs, SATA SSD ~1.4 ms; see
docs/benchmarks/phase3-results.md and
docs/benchmarks/snapshot-2026-06-13-comparison.md). Read those two as the range
of what storage charges, not as a hardware ranking: they differ by primitive as
well as by medium — the Mac figure is macOS F_BARRIERFSYNC, a barrier rather
than a full flush, and the SATA one is an honest Linux fdatasync. Durable
throughput is set by your disk's fsync latency, full stop. The dashboard and
alerts are organised around making that obvious and actionable.
A reassurance baked into the failure modes: weir's durability guarantees are
deterministically fault-tested (the DST harness — wab::dst). A failed fsync, a
torn write, a crash between sync and rename, or a panic mid-flush all produce a
Nack, never a false ack (verified by the i1_acked_true_is_durable
invariant). So when a durability alert fires, the system already did the safe
thing — your job is to fix the underlying disk/sink, not to chase lost data.
Setup
The artifacts live under deploy/:
deploy/grafana/weir-dashboard.json— the overview Grafana dashboard (an at-a-glance health strip, then Ingest / Durability / Drain / System rows).deploy/grafana/dashboards/— per-instance dashboards (weir-min/med/high/max/ chaos) generated DRY bydeploy/grafana/gen-dashboards.py; the demo'slevels/chaosprofiles populate them.deploy/prometheus/weir-alerts.yml— the Prometheus alert rules.deploy/monitoring/— a turnkeydocker composestack (Prometheus + Grafana + weir + a load generator) to see it all live. Seedeploy/monitoring/README.md.
Existing Prometheus + Grafana: add a scrape target for weir's /metrics
(default port 9185), load the alert rules via rule_files, and import the
dashboard JSON (pick your Prometheus datasource on import). Turnkey demo:
cd deploy/monitoring && docker compose up --build, then open Grafana at
http://localhost:3000.
See what an alert does before you need it in anger. The rules are unit
tested — promtool test rules deploy/prometheus/weir-alerts_test.yml pins every
rule in both directions and runs in about a second, so you can read the suite to
see exactly which series state produces which alert.
The demo's opt-in chaos profile (docker compose --profile chaos up)
deliberately drives a permanently-failing sink, a full dead-letter dir, and
peer-UID rejections. It will not make the shipped alerts go red, and this
document used to claim otherwise. Every rule selects job=~"weir", PromQL
matchers are fully anchored, and the chaos instance is scraped as
job="weir-chaos" (deploy/monitoring/prometheus.yml), so it matches no rule —
which is deliberate, so an idle chaos profile does not trip WeirInstanceDown
(deploy/monitoring/README.md). To watch a rule fire against live data, scrape
the failing instance under a job name the rules match. Note also that
WeirSinkDown is unreachable from that profile regardless: the failmock returns
400, and sink/http.rs:645 maps any client error to Degraded, not Down.
Alert selectors assume a job="weir" scrape job — adjust job=~"weir" to match
your scrape config.
Alerts & runbook
Severity convention: critical = page-worthy (durability or total-stall);
warning = investigate during business hours. Thresholds marked [TUNE] in
the rules depend on your storage + workload — calibrate to the measured
baselines in the benchmark snapshots.
Durability (page-worthy)
WeirInstanceDown
Prometheus can't scrape weir's /metrics (up == 0). The daemon is unreachable
or dead; producers are likely failing to connect.
Respond: check the process (systemctl status weir / container state), the
socket, and recent logs. If it crashed, the WAB replays sealed-but-unconfirmed
segments on restart — no acked data is lost.
WeirFsyncFailure
fdatasync returned an error — a record could not be guaranteed on stable
storage. The producer was Nacked (no false ack — DST-verified), so nothing
was silently lost, but durability is compromised until resolved.
Respond: this is a disk/filesystem problem, not a weir bug. Check dmesg for
I/O errors, the disk's SMART health, and free space. On network-attached storage
(NFS/EFS) fdatasync can fail under contention — local SSD is strongly
recommended (it's also 5–50× faster). Producers retry under at-least-once.
WeirFlusherPanic
A WAB flusher thread panicked. The supervisor respawns it up to 10× (linear
backoff) and then takes that shard permanently offline (Nack-everything until
restart). A panic mid-flush does not produce a false ack (DST
panic_at_fsync_barrier scenario).
Respond: grab the panic payload from the logs (grep "flusher panicked").
Real flusher panics are logical bugs — capture the backtrace and restart the
daemon to clear the offline shard. Repeated panics on the same shard = file a bug.
WeirAckTimeout
A record's ack never fired within the timeout, with no panic — the wedged-flusher
signature (e.g. a stuck fsync on a hung disk).
Respond: correlate with WeirFsyncLatencyHigh and weir_queue_depth. If
fsync latency is spiking, it's storage. If a WeirFlusherPanic follows shortly,
treat it as a flusher panic. A hung daemon may need a restart.
WeirSegmentQuarantined
Crash recovery found a segment with bad magic, an unknown format version, or a
CRC mismatch and moved it (or, for mid-file corruption, a copy of it) to
<wab_dir>/quarantine/. A data-integrity event. CRC mismatches usually mean
storage corruption (failing disk / bad RAM).
Respond: weir-ctl quarantine list shows every quarantined segment;
weir-ctl quarantine inspect <segment> reports how much of it is actually
recoverable, and weir-ctl quarantine requeue re-delivers it. Full procedure —
including why a "clean end" is not "everything was recovered," and why
requeue re-sends the already-delivered prefix — in
docs/operations/quarantine-recovery.md.
For mid-file corruption specifically: the records up to the first bad one
were already recovered and delivered normally; only the corrupt record and
whatever follows it in the file went to quarantine, and quarantine's inspect/
requeue can usually recover the good records that sit after the
corruption too.
WeirFsyncLatency
(WeirFsyncLatencyHigh warning / WeirFsyncLatencyCritical critical.) Sustained
fsync p99.9 above your storage baseline. Because weir is fsync-bound, this
directly caps durable throughput.
Respond: identify storage contention (a noisy neighbour, a backup job, a
failing disk), or confirm you're not on network-attached storage. Calibrate the
thresholds: measured p99.9 is ~2.4 ms on NVMe and ~6 ms on a SATA SSD — set
High to ~3–4× your medium's baseline and Critical an order of magnitude up.
If latency is normal for your medium, raise the threshold rather than chase it.
Drain / dead-letter
WeirDrainNotConfirming
Segments are entering the WAB and none are leaving it. This is the only rule here that can see a wedged drain, so treat it as authoritative even when every other drain signal looks healthy.
weir_drain_state and weir_sink_health are written only by the drain thread
(drain/mod.rs). When that thread wedges they freeze at their last value, so
WeirDrainBlocked, WeirDrainStopped, WeirSinkDown and WeirSinkDegraded are
all silent in exactly the incident they exist for, and weir-readiness.sh prints
READY throughout. This rule rests on weir_wab_segments_total instead — a
counter incremented by the WAB flusher (sealed) and by the drain's confirm path
(confirmed) — so neither side depends on the drain publishing its own health.
Respond: do not start from the gauges, they are lying by omission. Check
whether the drain thread is alive and what it is blocked on, then the sink.
weir_wab_bytes_on_disk is a useful second opinion: it is written by the main
poll task rather than the drain, so it stays live during a wedge and should be
climbing.
Two shapes of this rule are wrong, both documented in the rule's own comment.
A raw sealed - confirmed difference is blind after a restart-with-replay, since
recovery replays segments sealed by the previous process and confirmed can
restart far above sealed — measured at 417 segments of headroom, roughly
104 GiB at the default wab_segment_max_bytes. And - rather than unless
silently never fires on a daemon that has not yet confirmed anything, because the
wab_segments family is not pre-initialised and {state="confirmed"} does not
exist at all. Both cases are pinned in weir-alerts_test.yml.
Blind spot: an idle daemon. No seals means no left-hand series, so a drain that is wedged while nothing is being produced does not fire — correctly, since nothing is at risk, but it means this rule confirms a wedge rather than ruling one out.
WeirDrainBlocked
The dead-letter directory hit its cap; all drain activity is paused and the
WAB accumulates on disk. High severity — durable buffering continues, but nothing
is being delivered.
Respond: free dead-letter space (weir-ctl dl list / weir-ctl dl drop) or
raise dead_letter_max_bytes. The drain re-checks the cap every
dead_letter_check_interval_secs and resumes automatically once there's headroom.
WeirDrainStopped
The drain supervisor has exited and will not restart itself: the respawn budget ran out after repeated panics, the dead-letter writer failed to open at start, or the daemon is shutting down. This is terminal. Records are still accepted and still acked durably — no ack becomes false — but nothing is delivered and the WAB grows until the disk fills.
weir_sink_health is not trustworthy while this fires. That gauge is written
only by the drain thread, so once the drain is gone it is frozen at its last
reading; a green sink-health panel beside a stopped drain means nothing is
probing the sink, not that the sink is fine.
Respond: restart the daemon. First check the logs for the cause —
weir_drain_panics_total climbing to max_respawns points at a panicking sink
implementation, and a dead-letter open failure names its path. Set
wab_max_bytes if it is not already set, so the next occurrence bounds the WAB
instead of filling the disk. On a deliberate shutdown this resolves when the
instance stops being scraped.
WeirSinkDown
The configured sink reports itself unhealthy. The WAB keeps buffering durably (no
data loss), but delivery is stalled.
Respond: check the downstream system (the HTTP/SQL endpoint). weir retries
transient errors with backoff; sustained down means the sink needs attention.
Records stay safe on disk until it recovers.
Detection lag — alert on more than this one signal. weir_sink_health{state}
is driven only by the periodic HEAD health probe (probe_health), not by commit
failures, and that probe runs on the health_poll_interval_secs cadence from the
Draining state. During an active outage the drain spends its time in
RetryingTransient (backing off between commit retries), where the probe does not
run — so sink_health{down} can lag the actual outage by a full poll interval or
more, while weir_drain_state{state="retrying_transient"} flips on the first
failed commit. For prompt detection, alert on the faster signals in addition to
WeirSinkDown: weir_drain_state{state="retrying_transient"} == 1 (immediate on
the first transient failure) and increase(weir_drain_segments_stranded_total[15m]) > 0
(WeirSegmentStranded, fires once retries are exhausted) — see those entries.
WeirSinkDegraded
The sink reports itself degraded — reachable but not fully healthy (e.g. the
HEAD health probe returned a 4xx other than 401/403/405; 404 is not special and
falls here too). 401/403/405/501 are treated as Healthy, since they mean the
endpoint is reachable but just rejects the unauthenticated HEAD probe (the real
authenticated POST path still works). Delivery may still work but is impaired;
the WAB keeps buffering durably. Warning, not critical.
Respond: check the downstream system before it tips to down; correlate with
WeirSegmentStranded / WeirDeadLettered. Transient flaps below 15m don't fire.
WeirSegmentStranded
The drain exhausted sink_max_retries (default 3) of transient sink failures
on a segment and left it on disk ("stranded"). The data is durable (no loss), but
delivery for that segment is paused.
Respond: fix the sink (correlate with WeirSinkDown / weir_sink_health).
Once the sink recovers, the drain automatically re-drains stranded
segments on the next health poll — watch weir_drain_segments_resumed_total
converge toward weir_drain_segments_stranded_total and the alert clear. A daemon
restart also replays them. Note: stranded − resumed is not a strict count of
currently-stuck segments — both are event counters, and a segment that strands,
resumes, then re-strands (e.g. a POST-only outage where the HEAD probe stays
healthy) bumps both, so the difference can over- or under-state the live backlog.
For the actual stuck files, list them: weir-ctl segments shows the
sealed-but-undelivered files. If the gap persists, the sink isn't healthy yet (or is
recovering then re-failing). Raise sink_max_retries / sink_retry_base_delay_ms to ride out longer
outages before stranding. Distinct from WeirDeadLettered (permanent rejections).
WeirDeadLettered
The sink permanently rejected records (e.g. a 4xx). They are written to the
dead-letter directory for manual handling — not retried.
Respond: inspect the dead-letter files and the sink's rejection reason
(a schema mismatch, auth failure, malformed payload). Fix the upstream cause,
then reprocess with weir-ctl dl requeue — it re-submits the dead-lettered
records back through the daemon's socket and deletes each segment once all its
records are re-accepted. Re-delivery is at-least-once (a record may be delivered
more than once if a requeue run is interrupted). The HTTP sink's per-record
idempotency key is derived from where the record lands in the WAB (segment +
index), not just its payload bytes, so it dedupes a retry within one delivery
attempt but not a duplicate from a separate, later dl requeue run — that
writes the record into a new WAB entry with a different key. weir-ctl dl
also offers list (inspect) and drop (discard) for when the records aren't
worth reprocessing.
Ingest / backpressure
WeirHighNackRate
Producers are being told records were not accepted durably. Causes: a shard taken offline by repeated flusher panics, fsync failures, or sustained backpressure. Respond: correlate with the durability alerts. At-least-once means producers should retry, but a sustained Nack rate means a shard or the storage is unhealthy — chase the root cause above.
WeirQueueSaturation
The work queue is backing up — the WAB flushers can't keep pace with ingest
(usually storage pressure). Producers see backpressure (slower acks).
Respond: correlate with fsync latency. If fsync is the bottleneck, you're at
the storage durability ceiling — faster disk, more shards (one flusher/disk), or
shift latency-tolerant traffic to the Buffered tier. [TUNE] the
threshold to your batch_size × shard_count and burst profile.
Security / connection
WeirUnauthorizedConnections
A process whose UID isn't allowed tried to connect over the Unix socket and was
rejected by the SO_PEERCRED check.
Respond: either a misconfigured producer (wrong user) or an unauthorized
client. Run producers as the weir user, or review who has access to the socket
directory.
WeirTlsHandshakeFailures
(tls feature) Repeated mutual-TLS handshake failures — client-cert
expiry/rotation, a CA mismatch, or a probe hitting the TCP port.
Respond: check client certificate validity and the CA bundle. A burst right
after a cert rotation suggests the new certs aren't trusted;
weir_tls_config_reloads_total{outcome="ok"} confirms whether a SIGHUP
reload landed (and outcome="failed" that it was rejected).
Integrity / tamper evidence
weir-ctl attest hash-chains sealed WAB segments so "was this segment
edited after it was written?" becomes an answerable question. It runs outside
the daemon entirely — no cost on the ack path, no FORMAT_VERSION bump, no
touch to the frozen conformance vectors. Three subcommands:
weir-ctl attest seal --wab-dir /var/lib/weir/wab [--shard N] # chain any sealed segment lacking a sidecar
weir-ctl attest verify --wab-dir /var/lib/weir/wab [--shard N] # recompute the chain and compare
weir-ctl attest head --wab-dir /var/lib/weir/wab # print the newest head per shard, for anchoring
seal writes a <segment>.wab.sealed.attest sidecar per sealed segment,
chaining each record's already-frozen RecordId into a running SHA-256 and
linking the segment's start hash to the previous segment's head. verify
re-reads every byte and recomputes the chain from nothing — seconds on a
256 MiB segment, so it is an audit operation, not a health check, and must
never be wired into a readiness probe.
Records in an active, unsealed segment are not chained at all — only
seal puts anything into the chain, and it only ever sees sealed segments.
That window is bounded by the seal cadence:
wab_segment_max_bytes
and
wab_segment_max_age_secs
are what close it. An operator who needs the window tighter lowers those, at
the usual cost of smaller/more-frequent segments.
A .attest sidecar sitting next to the segment it protects is not evidence
by itself. Anyone able to rewrite the segment can recompute a sidecar that
matches it, in the same motion. The chain only becomes evidence once its head
has been observed somewhere the editor does not control — in practice, the
line attest seal prints to its own stdout: weir.attest.head (shard,
segment name, record count, the 64-hex head).
Be precise about what that line is. It is a plain println! to a CLI
process's stdout — not a tracing event, not timestamped, not leveled, and
not shipped anywhere by itself. Nothing daemon-side is involved: the daemon
never runs seal and never sees this line. It becomes an anchor only once
the operator captures that stdout and ships it off-host — before anyone
with write access to the WAB directory could edit both the segment and its
sidecar to match. A .attest file with no independent, off-host record of
what its head used to say is a number, not an audit trail.
Nothing here runs automatically or lives in the daemon — including seal
itself. seal is the command that does the actual work: it is the only
one that writes .attest sidecars and the only one that prints the anchor
line. verify only ever checks what seal has already produced. Wiring up
verify alone (a mistake this page previously invited) means every segment
reports MISSING_SIDECAR, weir_attest_verify_failures sits at 0 forever,
and the feature has attested nothing — a clean-looking dashboard for a
WAB nobody has ever chained. Both commands need a schedule, and seal's
needs its stdout captured, not just its exit code checked.
A systemd timer pair that runs both, with seal's stdout landing in the
journal (forward that to your log pipeline the way you already ship the
daemon's own journal output):
# /etc/systemd/system/weir-attest-seal.service
[Unit]
Description=weir-attest: chain newly sealed WAB segments
[Service]
Type=oneshot
User=weir
ExecStart=/usr/local/bin/weir-ctl attest seal --wab-dir /var/lib/weir/wab
# StandardOutput=journal is what makes the weir.attest.head anchor line reach
# your log pipeline at all. Without this (or an equivalent `| systemd-cat` /
# `| logger` redirect), the line is printed once and discarded the moment
# this process exits — the chain would have no off-host anchor.
StandardOutput=journal
StandardError=journal
# /etc/systemd/system/weir-attest-seal.timer
[Unit]
Description=Run weir-attest seal periodically
[Timer]
OnBootSec=5min
OnUnitActiveSec=5min
[Install]
WantedBy=timers.target
# /etc/systemd/system/weir-attest-verify.service
[Unit]
Description=weir-attest: verify the sealed WAB chain
[Service]
Type=oneshot
User=weir
ExecStart=/usr/local/bin/weir-ctl attest verify --wab-dir /var/lib/weir/wab \
--metrics-file /var/lib/node_exporter/textfile_collector/weir_attest.prom
StandardOutput=journal
StandardError=journal
# /etc/systemd/system/weir-attest-verify.timer
[Unit]
Description=Run weir-attest verify periodically
[Timer]
OnBootSec=10min
OnUnitActiveSec=15min
[Install]
WantedBy=timers.target
Offset verify's cadence after seal's (as above), not tighter than it —
verify against segments seal has not reached yet just reports them as
MISSING_SIDECAR, but there is no reason to burn the full-reread cost (§
above) checking segments nothing has chained yet. The cron equivalent, for a
host not on systemd — with the same requirement to explicitly capture
stdout, since cron only mails it if MAILTO is configured and mail delivery
actually works, and logger is the portable way to land it in syslog/
journald regardless of that:
*/5 * * * * weir /usr/local/bin/weir-ctl attest seal --wab-dir /var/lib/weir/wab 2>&1 | logger -t weir-attest-seal
*/15 * * * * weir /usr/local/bin/weir-ctl attest verify --wab-dir /var/lib/weir/wab --metrics-file /var/lib/node_exporter/textfile_collector/weir_attest.prom 2>&1 | logger -t weir-attest-verify
Its exit codes distinguish the failure mode for that job: 0 every segment
verified, 1 at least one segment's chain diverged from what was recomputed,
or its recorded link to its predecessor did not match reality (tampering, or
corruption past what CRC32 catches), 2 at least one segment could not even
be checked (missing/unreadable segment, undecodable sidecar). Script the
cron job to alert differently on 1 than on 2 — treat 2 as "unknown,"
not "clean." And watch weir_attest_missing_sidecar (below) alongside the
exit code: a nonzero, non-falling value there over time means seal isn't
running or isn't keeping up — the one failure mode the three-outcome exit
code above cannot surface, because verify against a WAB directory nobody
has ever sealed exits 0 too.
WeirAttestVerifyFailed
weir_attest_verify_failures does not come from the daemon — the
daemon is not involved in this feature at all. It's written by a cron/timer
running weir-ctl attest verify --metrics-file, via the node_exporter
textfile collector. A firing alert means either a sealed WAB segment no
longer matches its recorded hash chain, or a segment's recorded link to its
predecessor no longer matches what that predecessor actually produced (a
broken link — reported as BROKEN_LINK in verify's text output —
which is what deleting or substituting a whole segment out of a chain looks
like). Reclaiming the oldest segments is not this: retention removes them
continuously, the oldest survivor is reported as a chain origin instead, and
weir_attest_chain_origins is where that shows up. A broken link always
names a boundary between two segments that are both still on disk.
This is an integrity incident, not a durability one. The records were acked and fsynced correctly at the time; the bytes on disk (or the set of segments present) have since stopped matching what was chained. Do not delete anything.
Respond: compare the chain head against what was captured off-host from
seal's stdout (the weir.attest.head line — see Integrity / tamper
evidence above for what that line is and is not), then run
weir-ctl attest verify --wab-dir <dir> to see which segment diverged, or
which segment boundary's link broke. Exit codes distinguish the failure
mode: 0 every segment verified with no broken links, 1 at least one
segment diverged or one link broke (tampered), 2 at least one segment
could not even be checked (unreadable segment or undecodable sidecar) —
treat 2 as "unknown," not "clean."
Capacity: surviving a sink outage
weir's pitch is "buffer durably while the uplink is down." That's bounded by disk: while the sink is unreachable, every accepted record stays in the WAB (and, for permanent rejections, the dead-letter dir), so the buffer grows at the ingest rate until storage fills. Size it deliberately:
time-to-WAB-full ≈ free_disk_bytes / (ingest_rate × avg_record_bytes)
For example, 50 GiB free and 5 MiB/s of accepted data ≈ ~2.9 hours of outage headroom before the disk fills. Once the WAB partition is full, fsync fails and producers are Nacked (no false acks — they retry/backpressure), so you lose availability, not durability.
Wire these so you see it coming, in order of urgency:
- WAB disk % — a node/host alert on the filesystem holding
wab_dir(weir doesn't measure free disk itself); page well before 100%. Cross-checkweir_wab_bytes_on_diskfor weir's own contribution. - Dead-letter vs cap —
weir_dead_letter_bytes_on_diskapproachingdead_letter_max_bytes; at the cap the drain blocks (see below). - Drain blocked —
weir_drain_state{state="blocked_dead_letter_full"} == 1(WeirDrainBlocked) means all delivery is paused. - Sink down / segments stranded —
WeirSinkDown+WeirSegmentStrandedare the leading indicators that the buffer has started growing at all.
Mitigations: raise dead_letter_max_bytes only if the disk can take it; add disk;
lower the ingest rate (producer backpressure); or fix the sink. Stranded segments
re-drain automatically on recovery (watch weir_drain_segments_resumed_total).
Metric reference
All metrics are prefixed weir_. Counters carry a _total suffix in the
exposition; histograms expose _bucket / _sum / _count.
Ingest
Changed in 2.0.0 — dashboards need a one-line edit. The
tierlabel wassync/batched/bufferedand is nowdurable/buffered.syncandbatchedwere separate series for identical behaviour; they are one series nameddurable. Any panel or alert selectingtier="sync"ortier="batched"will silently return no data until updated — panels go blank rather than erroring, so grep your dashboards rather than waiting to notice.
| Metric | Type | Meaning |
|---|---|---|
weir_records_accepted_total{tier} | counter | Records admitted for processing, by durability tier (durable/buffered). |
weir_records_ack_total{tier} | counter | Records durably acked to the producer, by durability tier (durable/buffered). The gap accepted − ack is in-flight or failed (Nacked) work; it is only ~0, and the amortization ratio only meaningful, while weir_records_nack_total stays flat. |
weir_records_nack_total{tier,reason} | counter | Records Nacked (not durably accepted), by durability tier and reason. Should be ~0. reason is one of bad_magic, version_mismatch, bad_header_crc, payload_too_large, bad_payload_crc, internal_error, empty_payload, unknown_message, reserved_flags_set (the wire NackReason variant names, lowercased). internal_error is the only transient reason (queue saturation / ack timeout / write-fsync error / the wab_max_bytes cap rejecting — connection stays open, producer retries); the rest are permanent protocol/payload errors that close the connection. The cap is the one cause with its own counter — weir_wab_cap_rejections_total — so subtracting it from internal_error separates "the WAB is full" from the other three. |
weir_batch_records | histogram | Records carried by one PushBatch frame. Only observed for the optional batch extension, so it stays empty on a daemon no client batches against. Its shape is the question "are producers actually batching?" — a distribution pinned near 1 means the batch path is being paid for without being used. |
weir_batch_partial | counter | PushBatch frames answered with at least one record not durable. Should be ~0. A partial batch is not a protocol error: the frame was valid and the connection stays open, and the clear bits are the same at-least-once signal a single-record Nack(InternalError) carries. Rising values track the same underlying causes as internal_error — queue saturation, ack timeout, a write or fsync that did not succeed, the wab_max_bytes cap — so read it alongside weir_records_nack_total{reason="internal_error"} rather than on its own. |
weir_accept_latency_seconds | histogram | Socket-accept → connection-handler spawn latency (independent of fsync). |
weir_queue_depth | gauge | In-flight records between the socket layer and workers. |
These counters are per-process and reset to 0 on every restart (they are in-memory atomics, not persisted). After a crash/restart,
accepted − ackis not a lifetime loss figure: records that were durably written before the restart but not yet drained are replayed from the WAB on startup and counted underweir_recovery_records_replayed_total(Durability section), not re-counted underaccepted/ack. So a fresh process legitimately showsaccepted ≈ acknear zero while the recovery counter carries the pre-restart backlog. Userate(...)/increase(...)over a window (which tolerate counter resets) rather than raw cumulative differences across a restart boundary.
Durability (WAB)
| Metric | Type | Meaning |
|---|---|---|
weir_wab_fsync_duration_seconds | histogram | The dominant latency signal. Per-fsync duration. |
weir_wab_fsync_failures_total | counter | fsync returned an error. Must be 0. The failing batch is nacked, and the active segment is retired so no later batch can be acked over the hole. Nothing at runtime rescans an unsealed .wab, so any already-acked records left in that segment (up to wab_segment_max_bytes, 256 MiB by default) are stranded until the daemon restarts and crash recovery seals them. Stranded, not lost — but clear the storage fault and restart to release them. |
weir_wab_group_commit_records | histogram | Records covered by one WAB group fsync — the fsync amortisation factor. 1.0 means every record paid for its own fsync, which is the condition that makes a round-trip-bound producer slow for a reason batching cannot fix, and the condition under which adding concurrency helps far more than adding batching. Read it together with weir_wab_fsync_duration_seconds: the two multiply into the per-record durable cost. |
weir_wab_flusher_panics_total | counter | Flusher thread panics. Must be 0. |
weir_drain_panics_total | counter | weir-drain thread panics caught and respawned by its supervisor. Must be 0; sustained values indicate a sink/drain logic bug, and exhausting the respawn budget stops delivery. |
weir_wab_segments_total{state} | counter | Segment lifecycle transitions: open → sealed → confirmed; quarantined on corruption. A segment is counted sealed whichever way it was sealed — rotation, idle-seal, shutdown-seal, or crash recovery (recovery finalises an unsealed .wab and renames it, which is a real transition and is counted). A mid-file-corrupt segment counts both quarantined (the preserved forensic copy) and sealed (the truncated valid prefix, which is still delivered). Backlog caveat across a restart — see the note below. |
weir_wab_bytes_on_disk | gauge | Bytes used by live WAB shard segments: the open active segment (.wab) plus sealed segments awaiting drain (.wab.sealed). It scans on a 5 s cadence, so it trails real-time by up to that interval. It is not a total-disk-usage gauge: it excludes drained-marker files (.wab.confirmed) and the dead_letter/ (own gauge: weir_dead_letter_bytes_on_disk) and quarantine/ subdirs, and it doesn't measure the filesystem's free space — for the "disk filling up" signal use a node/host filesystem alert on the partition holding wab_dir (see Capacity). |
weir_wab_record_logical_bytes_total | counter | Cumulative record payload bytes as accepted from producers, before any compression. Equal to weir_wab_record_stored_bytes_total when wab_compression = "none". Counts payload bytes only — not the 8-byte per-record framing, the segment header, or the footer — so it is not a file-size measure. |
weir_wab_record_stored_bytes_total | counter | Cumulative record payload bytes as written to disk, after compression. Divide the logical counter by this one for the live compression ratio (see Compression ratio below). Bumped only after a successful write, so a rejected record moves neither counter. |
weir_wab_unexpected_mode_total | counter | Segment file found with unexpected permissions (tampering guard). |
weir_wab_cap_rejections_total | counter | Pushes Nacked because live WAB bytes exceeded wab_max_bytes. These surface to clients as NackReason::InternalError — the same byte queue saturation uses — so this counter is the only way to distinguish a cap rejection from queue-saturation InternalError Nacks. Zero when wab_max_bytes is unset (the default). |
weir_recovery_records_replayed_total | counter | Records replayed from sealed-but-unconfirmed segments on startup. |
weir_recovery_segments_quarantined_total | counter | Corrupt segments quarantined during active-segment recovery (bad header, or mid-file corruption with a copied tail). Covers every segment quarantine: active-segment recovery (bad header, or mid-file corruption with a copied tail), sealed-segment replay when a .confirmed sidecar fails verification, and drain-time quarantine. It moves in lockstep with weir_wab_segments{state="quarantined"}. The one thing it deliberately does not count is the .confirmed sidecar file itself when it is quarantined alongside its segment — that would double-count a single incident. weir_quarantine_bytes_on_disk is the signal for how much is sitting there. Recover with weir-ctl quarantine requeue — see the runbook. |
weir_recovery_segments_failed_total | counter | Segments whose active-segment recovery (recover_segment) failed and left the segment at its original path, so nothing will ever reach it. Deliberately excludes quarantined segments — a header-level quarantine (bad magic / unknown version / too-short header) moves the file and is counted by weir_recovery_segments_quarantined_total instead, because it is handled, not abandoned. (Such a segment is listed by weir-ctl quarantine list, though inspect cannot parse it — the header is what's corrupt.) Does NOT fire for the mid-file-corruption case this toolchain exists for: recovery returns Ok there, having sealed and delivered the valid prefix. Non-zero therefore means a genuine I/O failure left a segment stranded — check the startup logs for the path and cause. |
weir_quarantine_bytes_on_disk | gauge | Total bytes under <wab_dir>/quarantine/. The one signal that moves on every quarantine event, including the .confirmed-sidecar case neither counter above catches. Survives a restart (unlike the counters), refreshed on the same cadence as weir_dead_letter_bytes_on_disk. Recover with weir-ctl quarantine requeue. |
weir_recovery_quarantine_copy_failed_total | counter | Mid-file-corrupt segments whose quarantine copy failed (disk full / read-only / inode exhaustion); recovery left the segment un-truncated to preserve acked-durable tail records and will retry. Non-zero means recovery is stuck on that segment — clear the disk/read-only state and restart. |
weir_ack_timeout_total | counter | Acks that never fired within the timeout (wedged flusher). |
weir_stage_{queue,bridge_wait,write,total}_seconds | histogram | Per-stage latency decomposition (bench-trace builds only — diagnostic). |
Using
weir_wab_segments_totalfor drain backlog. Within a single process run the family conserves: every segment countedsealedis later countedconfirmed(orquarantined), so a growingsealed − confirmedreally does mean the drain is falling behind. It does not survive a restart. Like every other counter here it is a per-process in-memory atomic, and segments that were sealed by the previous process and replayed on startup are countedconfirmedby this one with no matchingsealed— a replay is not a transition, so it is deliberately not re-counted (weir_recovery_records_replayed_totalis the signal for that backlog). So after a restart the raw difference can go negative and stay negative. For the live backlog useweir_wab_bytes_on_disk, or list the files withweir-ctl segments; userate(...)/increase(...)over a window for the trend.
Drain / sink
| Metric | Type | Meaning |
|---|---|---|
weir_drain_state{state} | gauge | One of draining / retrying_transient / blocked_dead_letter_full / stopped is 1. stopped is terminal: the drain supervisor has exited (clean shutdown, the respawn budget exhausted, or the dead-letter writer failing to open at start) and nothing will be delivered until the daemon restarts — the WAB grows until then. Alert on weir_drain_state{state="stopped"} == 1 on a daemon you expect to be running. |
weir_sink_commit_records_total{outcome} | counter | committed / retried / dead_lettered per record. |
weir_sink_commit_duration_seconds | histogram | Sink commit() call duration. |
weir_sink_health{state} | gauge | Current sink health: exactly one of state=healthy / degraded / down is 1. Alert on weir_sink_health{state="down"} == 1 (WeirSinkDown); degraded is an early warning (WeirSinkDegraded). This gauge is driven solely by the periodic HEAD health probe, not by commit failures — see the lag note below. It is not a delivery-success signal. A sink that answers HEAD with 2xx (or with 401/403/405/501, which read as healthy) but then 4xx-es every POST reads healthy here while dead-lettering all traffic — the only metric that rises is weir_sink_commit_records_total{outcome="dead_lettered"}. Alert on the dead-lettered outcome rate (rate(weir_sink_commit_records_total{outcome="dead_lettered"}[5m]) > 0, the shipped WeirDeadLettered rule) in addition to weir_sink_health — health alone will not catch a HEAD-healthy / POST-rejecting endpoint. |
weir_sink_info{sink_type} | gauge | The configured sink type, set to 1 for the active one. sink_type="noop" means records are acked then DISCARDED (not forwarded) — alert/indicate on weir_sink_info{sink_type="noop"} == 1 in any non-soak-test deployment. |
weir_dead_letter_bytes_on_disk | gauge | Dead-letter directory size. |
weir_dead_letter_full_total | counter | Count of distinct BlockedDeadLetterFull episodes (each entry into the blocked state). For the current-blocked boolean use weir_drain_state{state="blocked_dead_letter_full"}. |
weir_dead_letter_blocked_duration_seconds | gauge | Seconds since the drain entered BlockedDeadLetterFull (resets to 0 on exit); alert when it exceeds your threshold. |
weir_drain_segments_stranded_total | counter | Segments left on disk after exhausting sink_max_retries transient sink failures. The data is durable; delivery is paused. Alert on increase(weir_drain_segments_stranded_total[15m]) > 0 and correlate with weir_sink_health{state="down"}. They are re-drained automatically when the sink recovers (see below) or on restart. This is an event counter, not a live gauge of distinct stuck segments: the same segment can strand, auto-resume on the next healthy probe, and re-strand — each strand increments it — so it can exceed the number of distinct stuck segments. That happens notably in a POST-only outage: deliveries (commit) keep failing while the HEAD health probe stays healthy, so each recovery edge re-queues the segment, it re-fails, and it strands again. Distinct from weir_dead_letter_full_total (permanent rejections). |
weir_drain_segments_resumed_total | counter | Stranded segments re-queued for delivery after a sink health recovery (down→up). Convergence with weir_drain_segments_stranded_total means an outage's backlog has been picked back up; a persistent gap means segments are still stranded (sink not healthy yet). |
System / security
| Metric | Type | Meaning |
|---|---|---|
weir_connection_rejected_peer_uid_total | counter | Connections rejected by the SO_PEERCRED UID check. |
weir_accept_resource_exhaustion_total | counter | accept(2) failures from resource exhaustion (EMFILE/ENFILE/ENOBUFS/ENOMEM); the accept loop backs off on each. A rising value means the daemon is near an fd/memory limit. |
weir_connection_idle_timeout_total | counter | Connections dropped for read-idle (slowloris guard). |
weir_connections_aborted_at_shutdown_total | counter | Connections force-closed at shutdown after the grace period. |
weir_tls_handshake_failures_total | counter | (tls) Mutual-TLS handshake failures. |
weir_tls_config_reloads_total{outcome} | counter | (tls) SIGHUP TLS cert/key/CA reload attempts, by outcome (ok / failed). Alert on outcome="failed" — a failed reload means the daemon keeps serving the old certificate. |
Integrity / tamper evidence (weir-attest)
Unlike every other row on this page, these do not come from the daemon's
/metrics. They are written by a cron-runweir-ctl attest verify --metrics-file <path>onto the node_exporter textfile collector — see Integrity / tamper evidence above for the full picture, including thatverifyalone attests nothing:sealmust be scheduled too. All four are gauges, overwritten wholesale on each run, not counters: they carry no_totalsuffix on purpose.
| Metric | Type | Meaning |
|---|---|---|
weir_attest_segments | gauge | Sealed segments examined by the last weir-ctl attest verify run. |
weir_attest_verify_failures | gauge | Segments that diverged, or whose recorded link to their predecessor did not match reality (a broken link — see above), or that could not be verified in the last run. Must be 0 — a firing alert here is an integrity incident, not a durability one. See WeirAttestVerifyFailed. |
weir_attest_chain_origins | gauge | Segments observed with no chain predecessor in the last run — the first segment of a shard, the oldest to survive retention (confirm_and_delete reclaims drained sealed segments continuously, so this is the steady-state case, not a rare one), one whose predecessor was quarantined, or one whose sidecar declares itself an origin. Never counted as a failure. Verify cannot tell a reclaimed predecessor from a deleted one — after either, the predecessor is simply gone — so an origin is not evidence of tampering and an unexpected origin is; only your own record of a shard's history distinguishes them. Alert on this changing unexpectedly (a jump with no reclamation or quarantine behind it), not on it being nonzero. |
weir_attest_missing_sidecar | gauge | Sealed segments with no .attest sidecar in the last run — seal has never chained them. Not counted in weir_attest_verify_failures (a missing sidecar is not tampering by itself — and since the next segment's prev_head commits to this one's content, verify recomputes the head of a sidecar-less segment and still checks that link, so deleting a sidecar to hide an edit breaks the chain rather than erasing it) — but a nonzero, non-falling value here over time means seal isn't running or isn't keeping up, which the other three metrics cannot tell you: a WAB directory nobody has ever sealed reports verify_failures 0 forever. |
Compression ratio
When wab_compression = "zstd" is enabled, the live ratio is:
rate(weir_wab_record_logical_bytes_total[5m])
/ rate(weir_wab_record_stored_bytes_total[5m])
A value of 1.0 means compression is achieving nothing on your payloads; below
1.0 means it is actively expanding them, which happens when records are small
enough that zstd's frame overhead exceeds the savings. Each record is compressed
independently, so the ratio reflects redundancy within a single record, not
across the stream — see
wab_compression for measured
figures by payload size and the rule of thumb for when to leave it off.
This is the number to watch when sizing wab_dir for a sink outage: it is the
multiplier on how long an outage the same disk can absorb.