The S3 sink

Writes each commit batch as one object to S3-compatible storage. Targets the S3 API, not AWS specifically — set sink_s3_endpoint and MinIO, Cloudflare R2, Backblaze B2 and Ceph all work on the same path.

Built from the separately published weir-sink-s3 crate, behind the opt-in s3-sink Cargo feature:

cargo build --release -p weir-server --features s3-sink

Quick start

# /etc/weir/weir.toml
sink_type           = "s3"
sink_s3_bucket      = "my-archive"
sink_s3_region      = "eu-west-1"
sink_s3_prefix      = "weir/events"
sink_max_batch_size = 10000        # see "Object sizing" below

Credentials come from the environment, an IRSA role, or an EC2 instance role — see Credentials. Nothing secret needs to touch the config file.

For MinIO or another self-hosted gateway, add:

sink_s3_endpoint         = "http://minio:9000"
sink_s3_force_path_style = true

What an object looks like

my-archive/weir/events/dt=2026-09-06/hour=14/18d2c17997b60400-9f86d081…08-10000.ndjson.zst
└─ bucket ┘└─ prefix ─┘└──── partition ────┘└ segment time ┘└─ WAB coordinate ┘└ count ┘

The body is newline-delimited records, one per line, zstd-compressed by default. That is deliberately the plainest thing a query engine can read — see Reading the bucket.

Why the key is shaped this way

weir's drain is at-least-once: after a crash it re-commits a byte-identical batch. The key is therefore built entirely from values that do not change under that replay, so a replayed batch overwrites its own object rather than creating a second copy of the same records.

The practical consequence is the strongest idempotency story of any weir sink: the downstream needs no dedup support at all. No ON CONFLICT, no Idempotency-Key, no engine-side dedup window.

What this does and does not guarantee

The property is real and worth stating exactly: weir never writes different records to a key it has already used. Every component of the name is read from disk or derived from a WAB coordinate, the WAB itself is append-only, and a replay re-derives the same name from the same bytes. Combined with at-least-once delivery, an object either does not exist or holds the batch its name describes.

It is not tamper-evidence, and must not be sold as one. The name carries the first RecordId and the count — .first(), at crates/weir-sink-s3/src/lib.rs. At sink_max_batch_size = 10000, anyone holding s3:PutObject on the prefix can alter record #7, leave record #1 and the count alone, and the modified body takes the original key as an ordinary last-write-wins overwrite. Nothing in the key covers it.

weir does send x-amz-checksum-sha256, and S3 stores and returns it, so corruption in transit or at rest is detectable. A deliberate rewrite is not: whoever replaces the object computes a correct checksum for the new bytes. If you need the stronger property, it comes from the bucket, not from weir — see Object Lock below, and grant s3:PutObject to nothing but the daemon.

One precondition. sink_max_batch_size must be constant for the life of the bucket, not merely stable across a restart. The drain re-reads it on every call, so changing it re-splits batches at different boundaries and produces different keys for the same records. Changing it duplicates; it does not lose. But the duplicates are silent and permanent.

Why that precondition is specific to this sink

It follows from one object per commit batch, not from anything about weir's identity model. An object key names a batch, and re-batching produces different batches — so the key changes, and S3 has no way to notice that the new object's contents overlap an existing one. The same is true of any batch-level key: the DedupToken, ClickHouse's insert_deduplication_token, an HTTP Idempotency-Key on an NDJSON body. It is not a limitation a different digest removes.

weir's per-record identity does not have this problem. A RecordId is built from the record's WAB coordinate — its segment plus its absolute index within that segment — so it is unchanged when the segment is re-split. Sinks that key on it survive a changed sink_max_batch_size:

If a duplicate is expensive — a billing row, a metering event — put the records in a store that can dedup per record and treat the bucket as the archive it is. If the bucket is the system of record, set sink_max_batch_size once, at the value you want forever, before the first object lands.

Configuration

Every key is available as a TOML key and as a WEIR_-prefixed environment variable. These are file/env only — there are no CLI flags for them.

sink_timeout_secs, sink_max_batch_size, sink_max_retries and sink_retry_base_delay_ms apply as they do to every sink.

Things rejected at startup, not at first commit

By the time a record reaches the sink it is already acked and waiting, so a configuration mistake must stop the daemon starting rather than surface later as a stalled drain. Rejected up front:

• A sink_s3_prefix or sink_s3_partition containing . or .. segments, an interior //, #, ?, a backslash, or a control character. A URL parser rewrites or truncates all of these, which would make weir sign one key and write another.
• A sink_s3_endpoint with a path component, for the same reason.
• An unknown sink_s3_framing or sink_s3_compression value.
• sink_s3_access_key_id without sink_s3_secret_access_key, or vice versa — a lone key id would silently fall through to the environment or an instance role and authenticate as something other than what the config names.

Object sizing

One object per commit batch. At the default sink_max_batch_size of 100, a 500-byte record produces roughly a 50 KB object — hundreds of thousands per hour for a busy producer, which is expensive to PUT and pathological for every query engine. Set sink_max_batch_size = 10000 and leave it there.

The daemon logs a startup WARN below 1000 saying so, and repeating the constant-for-the-life-of-the-bucket rule, because the operator raising the value is exactly the person who needs to know it must not change again.

At low volume the WAB decides, not sink_max_batch_size

sink_max_batch_size is an upper bound. The sink is handed whatever a sealed WAB segment contains, so below that bound the object size is set by how often a segment seals — and an S3 archive is the archetypal low-volume deployment, so this is the common case here rather than the exotic one.

Set wab_segment_max_age_secs. It defaults to 0, which means idle-seal is off, and a segment then seals only on reaching wab_segment_max_bytes (256 MiB by default) or on shutdown. A producer writing a few small records and going quiet gets every record acked and delivers nothing to S3. The bucket stays empty while the daemon reports healthy; nothing is lost, but nothing arrives either. See the low-volume trap for the general form of this.

wab_segment_max_age_secs = 300   # seal after 5 minutes with no new records

Read that comment literally: it is an idle timer, not a maximum age. The clock restarts on every flush (the last_write_at reset in crates/weir-server/src/wab/mod.rs), so it fires only when the producer actually stops. A steady trickle — one record a minute against a 300-second setting — resets the clock before it ever expires, and the segment grows toward 256 MiB exactly as if idle-seal were off. At 500 bytes a minute that is roughly two years.

So size the interval against your write gap, not against the delivery latency you would like:

If a steady trickle is your shape and a sub-write-gap interval is too chatty, lower wab_segment_max_bytes instead — that is the knob that bounds a segment for a producer that never idles.

Once a seal does fire, its interval is what sets the object size at low volume: whatever accumulated between seals is one object, however large sink_max_batch_size is. The two knobs bound the object from opposite ends — the seal decides when a batch becomes available, and sink_max_batch_size caps how much of an available segment goes into a single object. Raising sink_max_batch_size cannot make a low-volume object bigger.

Credentials

Resolved in this order, first match wins:

1. sink_s3_access_key_id + sink_s3_secret_access_key
2. AWS_ACCESS_KEY_ID + AWS_SECRET_ACCESS_KEY (+ optional AWS_SESSION_TOKEN)
3. Web identity / IRSA — AWS_WEB_IDENTITY_TOKEN_FILE + AWS_ROLE_ARN
4. IMDSv2 — the EC2 instance role

Temporary credentials refresh five minutes before expiry. If a refresh fails while the current credential is still valid, weir keeps using it and warns rather than stranding the backlog.

Not implemented, deliberately: IMDSv1 (an unauthenticated GET, and so reachable through any SSRF bug in a co-located process), ~/.aws/credentials profiles, and SSO. ECS task roles and EKS Pod Identity are detected and refused rather than ignored — silently continuing would authenticate as the node role instead of the pod's, which is a broader identity. A half-configured IRSA setup fails for the same reason.

Minimum IAM policy

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": "s3:PutObject",
    "Resource": "arn:aws:s3:::my-archive/weir/events/*"
  }]
}

s3:PutObject alone is enough to deliver. weir's health probe uses HeadBucket, which additionally needs s3:ListBucket; without it the probe returns 403 and weir reports Degraded, not Down, and keeps delivering. Add s3:ListBucket on the bucket ARN if you want a clean health signal.

With sink_s3_sse = "aws:kms" that policy is not enough. An SSE-KMS PutObject needs kms:GenerateDataKey on the key, and every reader needs kms:Decrypt:

{
  "Effect": "Allow",
  "Action": ["kms:GenerateDataKey", "kms:DescribeKey"],
  "Resource": "arn:aws:kms:eu-west-1:111122223333:key/<key-id>"
}

Without it every PUT returns 403. weir classifies that as transient, which is the right call for a 403 in general — an expired IRSA token and a rotated key present identically — but it means a day-one KMS policy mistake strands the backlog indefinitely rather than failing loudly. The symptom is a growing weir_drain_segments_stranded_total with a bucket that stays empty and a health probe that keeps saying Degraded-or-better, because HeadBucket never touches KMS. If you have configured SSE-KMS and nothing is arriving, check the key policy before anything else.

weir sets no Object Lock retention headers itself, so no s3:PutObjectRetention is required even against a locked bucket — a bucket default retention is applied by S3, not by the caller.

Failure handling

The rule: dead-letter on record-level rejection, strand on connection-level rejection. A record is dead-lettered only when that record is the problem. Anything about the credential, the endpoint or the bucket applies equally to every record in the backlog, so it strands — which is recoverable, and visible through WeirSegmentStranded and the wab_max_bytes cap — rather than dead-lettering, which silently displaces acked data.

Stranded and retried: all 5xx, throttling (SlowDown, 429), every authentication and authorization failure including 403 AccessDenied and ExpiredToken, 404 NoSuchBucket, clock skew, transport-level 400s (RequestTimeout, BadDigest, IncompleteBody), KMS throttling, and any status this sink does not recognise.

Dead-lettered: 400 for a malformed request, 411, and 413 EntityTooLarge — request-shape faults where the same bytes will be rejected again. Plus records that NDJSON cannot represent (see below).

This departs from the ClickHouse sink, which maps auth 4xx to permanent. The departure is deliberate: a day-one IAM misconfiguration, an expired IRSA token and a rotated key all present as 403, and dead-lettering a whole backlog over any of them converts a recoverable operator error into displaced data.

Records NDJSON cannot represent

A record containing \n or \r cannot be a single NDJSON line and is dead-lettered rather than written — framing it anyway would split one record into two lines downstream, silently corrupting the shape of the data. For binary payloads set sink_s3_framing = "length-prefixed", which is a u64 little-endian length before each record and dead-letters nothing on content. Those objects use the .weirbin extension and need a weir-aware reader.

Requeue produces duplicates

weir-ctl dl requeue and weir-ctl quarantine requeue re-push records through the daemon socket into a new segment, so the object key changes by design and you get a second object holding the same records. That is the at-least-once contract behaving as documented; deduplicate downstream if it matters.

Object Lock and versioned buckets

A compliance archive is the obvious use for this sink, so it is worth being precise about what happens when the bucket is locked down.

weir sends x-amz-checksum-sha256 on every PutObject. S3 rejects any PUT landing inside an Object Lock retention period unless it carries Content-MD5 or an x-amz-checksum-* header — including a bucket default retention, which the caller never mentions and no individual record knows about. The sink's first release, 2.1.0, sent neither, so under a default retention S3 answered 400 InvalidRequest, weir classified it Permanent, and every acked record was dead-lettered while health() still reported Healthy, because HeadBucket kept succeeding. Both halves are fixed: the checksum is sent, and InvalidRequest now strands rather than dead-letters, so an unfamiliar provider rejecting a request under that code cannot displace acked data either.

Object Lock requires versioning, and versioning changes what a replay does. Everywhere else in this document a replayed batch "overwrites its own object". In a versioned bucket nothing is overwritten — the PUT adds a new version under the same key, and under Compliance mode no one, including the account root, can delete it before its retention expires.

The headline property survives this. ListObjectsV2 and an unqualified GET return current versions only, so Athena, DuckDB, Spark and Glue still read each batch exactly once and still need no dedup. What you pay is storage: noncurrent versions are billed, and under Compliance mode they are undeletable until retention lapses. A lifecycle rule can expire noncurrent versions, but not before the lock allows it.

Budget for the stranded-segment case. When a sink recovers from a strand the segment is reprocessed from the beginning — the progress cursor is deliberately dropped at the strand boundary (crates/weir-server/src/drain/mod.rs:828-840), because persisting it would need a crash-safe sidecar. In an ordinary bucket that is invisible: each re-PUT overwrites the object it already wrote. In a versioned bucket it adds one noncurrent version per batch already committed in that segment. A 256 MiB segment at sink_max_batch_size = 10000 is a few hundred batches, so a single strand-and-recover cycle can add a few hundred undeletable copies. Prefer smaller wab_segment_max_bytes on a locked bucket, and treat WeirSegmentStranded as a cost alert as well as an availability one.

Provider notes

Only AWS S3 and MinIO are covered by the integration suite. An unrecognised error from any provider strands rather than dead-letters, so an untested one fails safe.

Reading the bucket

The default layout is Hive-partitioned NDJSON, which every common engine reads directly.

Athena:

CREATE EXTERNAL TABLE weir_events (payload string)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\n'
LOCATION 's3://my-archive/weir/events/'
TBLPROPERTIES ('projection.enabled'='false');

MSCK REPAIR TABLE weir_events;   -- discovers dt= / hour= partitions

DuckDB:

SELECT * FROM read_ndjson_auto('s3://my-archive/weir/events/**/*.ndjson.zst');

Partition pruning works on dt and hour because the key is Hive-shaped; that is the whole reason the partition exists rather than a flat content-addressed layout.

Testing against MinIO

bash deploy/run-sink-integration-tests.sh

Brings up MinIO alongside the SQL services and runs five #[ignore] tests. Two of them carry the design and both are required:

• s3_sink_replay_is_an_idempotent_overwrite — a crash-replayed batch must produce one object.
• s3_sink_distinct_batches_of_identical_records_produce_distinct_objects — two distinct batches of byte-identical records must produce two.

Either alone passes a broken key scheme. The first is equally true when the sink is overwriting its own data; the second when it duplicates on every replay.