Sink integration tests

The MySQL, Postgres, and ClickHouse sinks each ship a unit-test suite that covers the parts that don't need a real backend — identifier validation, SQL generation, error classification, password redaction. What unit tests can't verify is the part that actually matters in production: does the sink correctly commit a multi-row batch to a real database, does SQLSTATE / error-code mapping work end-to-end, do crash-recovery re-commits actually de-duplicate via the schema's UNIQUE constraint (or, for ClickHouse, the per-batch insert_deduplication_token)?

For that, we ship #[ignore]-marked tests in crates/weir-server/tests/system.rs:

  • mysql_sink_end_to_end — pushes 100 Sync records, verifies the sink committed them all and did so as ≥10 records per commit() call (the IOPS-compression claim).
  • postgres_sink_end_to_end — same assertion shape for the Postgres sink.
  • clickhouse_sink_end_to_end — same assertion shape for the ClickHouse sink, verifying the end-to-end RowBinary HTTP insert path and the IOPS-compression ratio. Gated behind the clickhouse-sink cargo feature (#[cfg(feature = "clickhouse-sink")]), so it only compiles when that feature is enabled.

The S3 sink, against MinIO

weir-sink-s3 is exercised by a five-test group against a real S3 API, not a mock — deliberately, because the sink hand-rolls SigV4 and a mock that accepts any signature would leave the job green while testing nothing:

  • s3_sink_end_to_end
  • s3_sink_replay_is_an_idempotent_overwrite — pins replay stability.
  • s3_sink_distinct_batches_of_identical_records_produce_distinct_objects — pins collision freedom. Run these two as a group: either alone passes a broken key scheme (the first is equally true when the sink overwrites its own data, the second when it duplicates on every replay).
  • s3_sink_keeps_delivering_when_head_bucket_is_denied — a least-privilege IAM policy reads as Degraded, and a Degraded sink must still drain.
  • s3_sink_an_unreachable_endpoint_strands_rather_than_dead_letters

Gated behind the s3-sink cargo feature, and pointed at the rig by WEIR_TEST_S3_ENDPOINT plus AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY — the sink reads credentials from the environment as a production deployment would, so they never touch the generated config file.

The MinIO image is not the upstream one. MinIO withdrew its public images: every tag on quay.io/minio/{minio,mc} now answers 401 UNAUTHORIZED to an anonymous pull and docker.io/minio/minio answers object not found. The rig uses bitnamilegacy/minio, which is the same MinIO server, so SigV4 is still genuinely verified. See the comment in deploy/docker/test/docker-compose.yml for the bar a future replacement has to clear.

Why they are all gated

Every test above is #[ignore]-marked because it needs a real backend: WEIR_TEST_MYSQL_URL / WEIR_TEST_POSTGRES_URL / WEIR_TEST_CLICKHOUSE_URL / WEIR_TEST_S3_ENDPOINT. The runner script below brings up all four backends in containers — MySQL, Postgres, ClickHouse and MinIO — and runs every one of these tests in one shot, nine in total plus sql_sink_content_keyed_schema_loses_a_distinct_duplicate_record.

Quick start

# Pre-requisite: Docker installed, `docker compose` plugin available.
# Brings up the stack, waits for healthchecks, runs all three tests, tears down.
bash deploy/run-sink-integration-tests.sh

# Release build (matches CI):
RELEASE=1 bash deploy/run-sink-integration-tests.sh

The script exits 0 iff all three tests pass. The ClickHouse test is run with --features clickhouse-sink so its sink is compiled in. The container stack is torn down on exit (including failures), so the runner is safe to re-invoke without leftover state.

What it does

  1. Runs docker compose -f deploy/docker/test/docker-compose.yml up -d. The compose file spins up:
    • mysql:8.0 on 127.0.0.1:33306, with the schema from deploy/docker/test/init-mysql.sql pre-seeded.
    • postgres:16 on 127.0.0.1:55432, with the schema from deploy/docker/test/init-postgres.sql pre-seeded.
    • clickhouse/clickhouse-server:24-alpine on 127.0.0.1:18123 (HTTP), with the schema from deploy/docker/test/init-clickhouse.sql pre-seeded.
  2. Waits up to 120 s per service for the compose healthchecks (mysqladmin ping, pg_isready, and ClickHouse's HTTP /ping) to report healthy. Each container entrypoint gates the external listener on init-script completion, so a healthy status implies the weir_records table exists.
  3. Exports the three WEIR_TEST_*_URL env vars pointing at the containers' published ports.
  4. Runs the three ignored tests via cargo test -- --ignored --exact, adding --features clickhouse-sink for the ClickHouse test.
  5. Tears down with docker compose down -v on exit.

Schemas

All three schemas live in deploy/docker/test/ and mirror the reference schemas documented in docs/operations/configuration.md. The SQL schemas include the UNIQUE constraint the default insert mode (INSERT IGNORE for MySQL, ON CONFLICT DO NOTHING for Postgres) keys against; the ClickHouse schema is a MergeTree with a non-zero non_replicated_deduplication_window, which dedups inserts by the per-batch insert_deduplication_token the sink sends. Either way the tests exercise the idempotent re-commit path the drain relies on under crash-recovery.

SQL-sink dedup keys on the whole payload bytes — a byte collision collapses distinct records. The reference schemas dedup on the payload (or its sha256), and the ClickHouse insert_deduplication_token is a hash of the batch's bytes — there is no per-record identity in either case. Two legitimately distinct events that happen to have byte-identical payloads (heartbeats, repeated OK bodies, identical log lines) collapse to one row, even though weir acks both — the drop is invisible to weir. If distinct events can share bytes, embed a unique field (event id, timestamp, sequence) in the payload so each record is byte-distinct. (As of 2.0.3 this no longer applies to the HTTP sink's default per-record Idempotency-Key — it is derived from the record's WAB coordinate, not the payload alone; see sink_send_idempotency_key. The HTTP sink's NDJSON batch-mode key is still a pure content hash and collides the same way ClickHouse's token does.)

weir_sink_commit_records_total{outcome="committed"} counts records sent to the INSERT, not rows persisted. With INSERT IGNORE / ON CONFLICT DO NOTHING (and ClickHouse dedup), the server silently drops duplicate-key rows, so the table can hold fewer rows than the committed count. weir can't see that server-side drop — the statement succeeded from its side. For a dedupping sink, reconcile row counts against the database, not the committed counter.

A whole-batch transient outage yields zero observable duplicates. The drain retries at the segment granularity, so if every record in a batch fails transiently, nothing commits and the entire segment is re-sent — the duplicate is absorbed by the UNIQUE/dedup path and no duplicate row is ever observable. To actually observe a re-delivered (and de-duplicated) record you need a partial failure: a committed prefix followed by a transient error, so the retry re-sends rows that did persist the first time and the dedup constraint swallows them. The end-to-end tests force this partial-failure shape deliberately; a clean all-or-nothing outage would exercise the retry path but leave the dedup path unobserved.

Manual setup

If docker compose isn't available, the MySQL and Postgres test docstrings include single-docker run setup recipes (the ClickHouse docstring uses the compose recipe). Adjust the URL accordingly and run one test at a time:

WEIR_TEST_MYSQL_URL=mysql://root:test@127.0.0.1:3306/weir_test \
  cargo test -p weir-server --test system -- --ignored --exact \
  mysql_sink_end_to_end

Out of scope

  • CI. The runner script ships locally-runnable. Wiring this into GitHub Actions needs a service-container setup and is enough complexity that it deserves its own PR — .github/workflows/sink-integration.yml is a clean follow-up when someone wants it.
  • TLS-enabled containers. The Postgres sink supports TLS via ?sslmode=require (see Configuration → Postgres, but the test compose stack runs without TLS so the runner doesn't need a CA-trusted cert. A TLS-enabled postgres-tls service alongside the existing postgres one is a clean follow-up that would exercise the handshake path end-to-end.