Ledger engine: crash-safe, resumable drill_events migration — no Redis - #7
Open
ar2rsawseen wants to merge 42 commits into
Open
Ledger engine: crash-safe, resumable drill_events migration — no Redis#7ar2rsawseen wants to merge 42 commits into
ar2rsawseen wants to merge 42 commits into
Conversation
…A/B vs classic) Adds a second migration engine behind MIGRATION_ENGINE=ledger for A/B testing against the current architecture (classic remains the default and is untouched). Ledger engine design: - Progress state = one MongoDB ledger row per cd-bounded chunk (pending → in_progress → written → attaching → done). No Redis anywhere; MongoDB + ClickHouse are the only dependencies. - Each chunk copies into its own staging table (clone of the live DDL), verified by read-tally vs exact ClickHouse count(), then promoted via verify-then-ATTACH PARTITION per partition (INSERT SELECT fallback), then dropped. The live table only ever receives whole verified chunks. - Crash recovery never trusts the ledger: in_progress chunks are dropped and redone; written chunks are recounted; attaching chunks verify each partition against the live table before attaching (no double-attach). - Synchronous inserts (errors surface; dedup token effective) + startup dedup canary that measures whether the token works on the target engine. - Error classifier: permanent ClickHouse data errors fail immediately instead of burning the 8x retry backoff; transient errors keep retrying. - Pipelined reads (prefetch) + bounded concurrent insert window. - Fixes the page-boundary double-read (inclusive min() re-returns the previous page's last doc) — the classic engine exhibits this on main: measured 25 duplicate rows + 1 lost boundary doc per 250k clean run. A/B harness in bench/: seed script, kill-drill (random SIGKILL until convergence, verifies zero loss + zero duplicates), instructions. Measured on the same 250k-doc dataset, same machine: - classic: 44s, 250,024 rows / 249,999 unique (dups + loss on a clean run) - ledger: 12s, 250,000 / 250,000 exact; 4x SIGKILL drill converges exact Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replaces the Redis-fed viz for the ledger engine: data comes from the chunk ledger (MongoDB) + in-process engine counters, polled every 2s. Brand tokens sampled from countly.com (#21B566 green, #24292E ink, Plus Jakarta Sans + Inter). Shows live counters, per-collection progress, a chunk map colored by ledger status (newest-first), dedup-canary and engine badges, and failed chunks with their errors. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ure, monitor, dry-run, report Closes every gap the initial POC deliberately skipped, so the branch is a complete solution rather than a proof of concept: - Bisection → doc-level DLQ (mig_dlq_docs): permanent insert errors are halved-and-retried down to the exact offending documents, which are stored WITH their full raw source doc. Every unmigratable doc (invalid ts, missing fields, transform errors) is likewise captured — accounted for and replayable, never silently dropped. - DLQ replay (POST /control/replay-dlq): re-transforms stored raw docs under the current TRANSFORM_VERSION and inserts into the live table; still-broken docs stay pending with updated errors. Never re-reads the source collection. - Circuit breaker: pauses the engine when >LEDGER_BREAKER_PCT% of a chunk's docs fail (systematic bug) or after N consecutive failed chunks. Resume via POST /control/resume. - ClickHouse backpressure: TTL-cached sampler (never 3 system queries per batch); waits out parts pressure between pages. - Streaming reads (C3): one long-lived cursor per chunk instead of a fresh find() per page, reopened from the last position on cursor death; kills the per-page boundary re-read class entirely. - Multi-pod lease reclaim tick: expired claims are recovered during the work loop, not only at collection start. - Invariant monitor: background spot checks of done chunks against live-table counts; violation → pause + chunk flagged. - Dry-run mode (DRY_RUN=1): ≤5% stratified sample against a Null-engine clone — full parse/type validation, nothing stored; DLQ + coercions become the pre-run report. - Coercion policy (two-tier): Countly-owned c clamped to UInt32; customer sg/custom/cmp/up values that can't survive the numeric path stringified losslessly (zero-copy when clean). Every coercion counted with samples. - GET /report: chunk status, skips by reason, coercions per key, DLQ summary. - Tests: 12 new (classifier, coercions, ledger claims/leases/transitions, end-to-end pipeline with poisoned docs → DLQ with raw docs, replay). Validated: 250k-doc run exact (250,000/250,000) with the full feature set at identical speed to the POC; SIGKILL drill converges exact; typecheck clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…rely The team compares branches (main vs this one), so this branch carries only the new implementation. Removed: BatchRunner, collection/range orchestration, Redis hot state, collection locks, global progress, async batch writer, manifest batch store, coverage math, legacy ClickHouse writer, legacy HTTP routes, GC controller, process metrics, their tests and helpers, the engine switch, all classic-only config (Redis, rerun modes, range-parallel, GC, async-write, lock tuning), and the ioredis dependency. Dependencies are now MongoDB + ClickHouse, full stop. Added the one capability only the legacy engine had: a null-cd sweep — a dedicated chunk (sentinel bounds) pages by _id over documents without a cd value, with id-based verify-then-attach (no cd window exists for them) and a monitor mode that stays sound when null-cd rows land inside regular chunks' cd windows. Validated: 12/12 tests (incl. null-cd end-to-end), 250k straight run exact in 8s, SIGKILL drill (2 kills) converges exact — zero loss, zero duplicates. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Completion bug: the work loop treated "no pending chunks" as collection complete, silently skipping chunks still leased by a dead pod (SIGKILL orphan under an unexpired lease) — a run could report completed with a hole. Complete now means NO non-terminal chunks: single-pod recovers orphans immediately; multi-pod waits and reclaims on lease expiry (reclaim tick capped at 30s cadence). - POST /control/retry-failed: resets failed chunks to pending and resumes; chunks that were already promoted (e.g. flagged by the invariant monitor) get their live cd window purged first so redo is clean. - Circuit-breaker path now drops its staging table. - bench/seed-failures.ts: seeds breaker-burst / scattered-DLQ / coercion scenarios for failure drills. Drill verified end-to-end on 100k docs: SIGKILL → breaker trip (805 docs DLQ'd with raw docs) → deliberate live-table corruption caught by the invariant monitor in seconds → retry-failed purge+redo → orphan recovery → final 100,003/100,003 exact. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… harness (D4, #6) Vendors the differential harness from countly-platform (corpus.json 71 fixtures, goldens.json generated from the LIVE ingestion normalization, decode/canonicalize, hash-tied sync contract) into tests/differential/, adds the repo's first CI workflow (typecheck + harness, service-free), and adopts the shared normalization spec in src/transform: - normalize.ts/validators.ts rewritten to the spec (platform branch claude/jovial-shannon-b3dd29 is the source of truth): existing non-blank doc.n wins over sg-derived names (dedup identity with live rows), clampUInt32/clampDateTime64 for Countly-owned fields, sanitizeJsonValue for customer bags — stringify ONLY what JSON cannot carry (NaN/±Infinity, bigint, BSON Decimal128/Long). Notably this DROPS the earlier >2^53-stringify rule: live ingestion keeps finite large doubles numeric, and matching live is the whole point — the harness caught that divergence. - CoercionCounter re-threaded as pure accounting (optional param, zero behavior change): clamp + stringify events counted per (rule, bag) with samples for the /report endpoint. 86/86 tests green (71 differential fixtures + engine suite); 100k end-to-end run exact after the transform change. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Found by the 3-pod drill: concurrent pods probing a shared canary table name race on CREATE/DROP and false-flag dedup as inert. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A document that kills the process on every touch (OOM-class, not a clean insert rejection) previously crash-looped until the whole multi-million-doc chunk was quarantined. Now: after 3 crash-retries a splittable chunk is bisected into 4 sub-chunks instead of retried — repeated splitting converges on a <=1-minute window around the poison doc, quarantined as a tiny failed chunk while everything else migrates. Originals become 'superseded' (terminal); the null-cd sentinel and <=1-min windows quarantine directly. Includes a gated chaos hook (LEDGER_TEST_CRASH_ID) and bench/poison-drill.ts. Drill result: 20k docs + 1 poison -> converged in 25 restarts / 7 split generations to a 0.5-min 2-doc window, 19,999/20,001 migrated with the poison active, exact 20,001/20,001 after the operator fix + retry. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
POST /control/waive-dlq (optionally {ids}): explicitly accept that pending
DLQ docs will not migrate. Waived is terminal but reversible; raw docs stay
in the DLQ permanently as the record of what was excluded. Sign-off requires
pending = 0 — every entry must end resolved (fixed+replayed) or waived.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The dashboard now covers the complete operator workflow, not just state:
- Action buttons wired to the /control endpoints (pause, resume, retry
failed chunks, replay DLQ, waive pending DLQ) with confirmation prompts
on the destructive ones and toast receipts that distinguish success from
HTTP errors (an error response no longer masquerades as success).
- Dead-letter queue panel: pending/resolved/waived pills with the sign-off
gate spelled out (pending must reach 0), top errors table, and expandable
per-doc samples showing the stored raw source document.
- Coercions panel: per-(rule, field) counts with before→after samples.
- New GET /api/dlq feeding the panel.
Fixes from driving it in a real browser: POST fetches now send '{}' with the
JSON content-type (Fastify 400s an empty JSON body — the button clicks were
silently failing), and toasts report non-2xx responses as failures.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…, verification
Turns the dashboard into a console a self-hosted customer can migrate with,
not just watch:
- Migration Guide tab: the runbook as a guided checklist. Automated phases
report their own status (index coverage, dry-run state, live progress);
manual phases (Prepare, Cutover) are persistent checkboxes (localStorage).
Sign-off is three explicit gates: all chunks done, DLQ pending = 0, full
verification passed.
- Preflight (GET /api/preflight + button): MongoDB reachability, per-
collection {cd,_id} index coverage, doc estimates, ClickHouse target
existence, dedup-canary verdict, dry-run status — read-only, run anytime.
- One-click verification (GET /api/verify + button): every completed chunk
recounted against the live table, plus table totals and duplicate check.
Exact; feeds the sign-off gate.
- Help & Recovery tab: the runbook's incident scenarios as expandable
entries with the relevant action buttons inline (incl. a cross-tab jump
to verification).
- Two-step confirmation replaces native confirm() dialogs: first click arms
the button (auto-disarms after 4s), second click fires. Testable,
consistent, no browser dialogs.
Every element driven and verified in a real browser: tabs, preflight,
checkbox persistence across reload, verify (6 chunks, 0 duplicates, gates
updating truthfully mid-incident), DLQ raw-doc expander, arm/disarm/fire on
waive (receipt {"waived":505}, gate flip to ready-for-sign-off), replay,
pause, resume receipts.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Self-service & scaling:
- Pods panel (GET /api/pods): per-pod chunks done/active/last-seen with
alive/gone pills; README gains a scaling guide (pods scale across
machines — one pod is CPU-bound; find the ceiling by adding pods and
watching per-pod docs/s).
- Preflight became actionable: POST /control/build-indexes builds missing
{cd,_id} indexes server-side with live progress (GET /api/index-progress,
incl. $currentOp build percentage); POST /control/dry-run runs the sampled
rehearsal in-process with its own reader (guarded while migrating);
both wired into the Guide tab.
- New preflight checks: replica-set detection with a secondaryPreferred
suggestion (source is frozen after cutover — secondary reads are exact),
MongoDB and ClickHouse disk headroom (the #1 preventable incident).
Chaos-verified (scratch containers + TCP chaos proxy; shared dev services
untouched), all with exact final counts:
- mongod hard-killed 8s mid-run (OOM/crash surface): self-healed, driver
retry layer alone absorbed it
- mongod CPU-starved to 0.15 cores mid-run: slowed, completed exact
- ClickHouse unreachable 8s mid-run: insert retries rode it out
Disk-full stance documented in the classifier: capacity errors are
transient → retries → attempts → breaker pause → operator frees space →
retry-failed.
Found & fixed by the chaos run: estimatedDocumentCount resets after an
unclean mongod shutdown, which collapsed chunk sizing to one mega-chunk.
Chunk count now also floors by time span (LEDGER_MAX_CHUNK_DAYS, default 7)
so a bad estimate can never produce a whole-collection chunk.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ge, state location GET /api/config surfaces the sizing knobs (chunk target, max chunk days, page size, insert window, lease, breaker, read preference) with current vs default values and guidance, plus where progress state physically lives (mig_ranges / mig_dlq_docs in MANIFEST_DB) and the recovery stance. Rendered as a Guide-tab card with 'changed' pills on non-default values. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…by review) Walking the state machine for 'can any crash point produce duplicates or missing data' surfaced a real gap, no crash required: the null-cd sweep chunk had the highest idx, so newest-first claiming ran it FIRST — and its rows carry cd derived from ts, landing inside regular chunks' cd windows. A regular chunk attaching afterwards saw rows in its window during verify-then-attach and skipped attaching a never-attached partition: silently missing data. (The invariant monitor would flag it, but the retry purge would then also delete sweep rows in that window.) Fixes: - The sweep is now gated: claimNext excludes the sentinel until every regular chunk of the collection is terminal (multi-pod safe — the gate counts in-flight chunks too). - retry-failed on a regular chunk of a collection whose sweep already ran also resets the sweep, purging its remaining rows precisely by id (it has no cd window of its own). - Orphaned staging tables (crash between done and drop) are swept at collection completion. - Regression test: a null-cd doc whose derived cd lands inside regular windows — exact totals now; would silently lose data before this fix. Residual (documented): on dedup-inert targets only, an ack-lost crash during DLQ REPLAY can duplicate one replay batch; the canary identifies such targets and /api/verify's uniqExact check detects it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…t, null-cd preflight count - docker-compose.yml: the migrator only, connecting to YOUR Mongo/ClickHouse (previous file still shipped Redis and a bundled MongoDB from the legacy architecture — misleading for setup). - .env.example: current variables only (required trio up top, common, scaling, sizing, rehearsal), replica-set read-preference note included. - README 'Setup & run': prerequisites, two start paths, then hand over to the dashboard. Explicit split: README = everything BEFORE the dashboard exists (install, env, start, automation reference); the UI = everything after (guide, actions, troubleshooting, verification); RUNBOOK.md = the cross-system cutover procedure and incident tables. - Preflight now counts null-cd outliers per collection (pass when zero, which is the expected case) and labels the doc-count estimate as span-guarded. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
No reason an operator should know a path fragment; the dashboard is the product's front door. Docs updated to plain http://host:port. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Found while smoke-testing the new root route: pointing MONGO_DB at a
database without drill_events collections (or any orchestrator startup
error) hit process.exit(1) — taking the dashboard down with it, so a
fresh operator with a config typo saw a dead process instead of the UI.
Now the crash marks the run failed and the console stays up:
- red 'Engine stopped' banner on the dashboard with the actual error and
'fix env + restart, state is untouched' guidance
- /healthz returns {status:'error', error} for orchestration/probes
- /stats carries fatalError; status badge shows FAILED
Verified live: bad MONGO_DB → console at / renders the banner, healthz
reports the error, run resumes normally once config is fixed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…d-window queries
Two things, the second found while designing the first.
1) SCOPING FIX (latent multi-collection bug). Every test and drill so far
used ONE source collection; production has many drill_events{hash}
collections all overlapping in wall-clock time, landing in one CH table
partitioned by month only. Four live-table cd-window queries were
collection-agnostic, so on real deployments:
- retry-failed's window purge DELETED sibling collections' rows
- crash-during-attach recovery could see sibling rows in the same
partition+window and skip a never-attached partition (silent loss)
- the invariant monitor and verify would false-alarm (live > expected)
Chunks now persist their ClickHouse row identity at creation
(scope_a/scope_e/scope_n — custom events map to e='[CLY]_custom' with
the name in n, so their scope is (a,e,n); internal events (a,e)):
- countLiveInCdRange / deleteLiveCdRange take the scope
- attach-recovery now checks staged row ids universally (precise for
the chunk regardless of siblings; window-count check removed)
- unresolvable collections in multi-collection runs purge by Mongo ids
and are skipped by per-window equality checks (global totals still
verify); single-collection runs keep exact unscoped semantics
Regression suite: two hashed sibling collections over the same time
range — exact migration, scoped verify clean, retry-failed leaves the
sibling untouched.
2) LEDGER REBUILD (operator request): regenerate mig_ranges from the data
itself when progress state is lost. Frozen source ⇒ chunk grid is
re-derivable; per window, exact Mongo count vs scoped live CH count:
equal→done, zero→pending, partial→failed (redo purges first). Post-
cutover live ingestion is untouchable by construction (newer cd than
every window); the tool's own null-cd sweep rows are attributed by id
and subtracted per window. Guarded: not while copying, not with other
pods active, force required to replace an existing ledger.
- POST /control/rebuild-ledger {force} + GET /api/rebuild
- Help & Recovery: 'Migration progress state lost' scenario with
two-step Rebuild button, force-overwrite path, live progress and
per-collection summary table (verified in-browser end to end)
- tests: rebuild after wipe (all done, live rows ignored), partial
window → failed → retry+resume heals exactly
Suite: 91 tests green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The engine was already Kubernetes-shaped — pod identity defaults to the hostname (= pod name), coordination is Mongo chunk leases with no shared service, /healthz exists, and abrupt kills are the designed recovery path — but the repo shipped no manifests, so only Docker had a concrete artifact. - k8s/migration.yaml: ConfigMap + Secret + Deployment + Service. Pods stay up after completion so the dashboard remains available for verification and sign-off; any pod shows the whole run (state is in MongoDB). - k8s/job.yaml: batch Job variant with EXIT_ON_COMPLETE=true (pods exit 0 when every chunk is terminal), generous backoffLimit since crash-redo is normal operation. - README: Kubernetes subsection in Setup & run. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Decision context: the countly-platform eventTransformer rewrite (claude/jovial-shannon-b3dd29) will not merge for now. Audited every divergence between this tool's transform and platform MAIN: - The tool has no code dependency on platform — it writes ClickHouse directly. Ledger, DLQ, rebuild, verify, UI: all unaffected. - Cross-query semantics already agree with main's live pipeline: custom events e='[CLY]_custom' + name in n (confirmed), uid_canon left to identity machinery on both sides, cd = history vs receive-time. - Everything else in the spec (NaN/Decimal128/Long stringification, ts heuristics, clamps, skip rules) concerns BSON-only shapes that JSON SDK ingestion can never produce — divergence is unobservable. - The rebuild's non-overlap assumption is GUARANTEED by main's behavior (cd always re-stamped to now for live rows). One real hazard documented as a guardrail instead of a code change: replaying historical drill docs through platform ingestion on main re-stamps cd to insert time → history duplicated at today's date. Added to RUNBOOK incident table and the DLQ Help scenario: replay only via the tool's Replay DLQ. normalize.ts header + differential README no longer claim a two-repo CI lock; the goldens are this repo's frozen spec, the platform PR is optional platform-side hardening. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The cd fix was split out of the shelved transformer-spec branch into its own minimal PR — it affects LIVE rows (Kafka offset replay and connector redelivery re-date events), not just doc replay. Guardrail wording fixed: platform-side replay of already-migrated docs is off-limits regardless of that fix, since the live table does not dedup by _id. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Live ingestion on platform main is at-least-once (connector exactlyOnce=false); ordinary redelivery leaves a handful of duplicate _ids until the platform's nightly EventDeduplicationJob cleans them. Verify's global count-vs-uniqExact check surfaced those as bare 'duplicates: N', indistinguishable from a migration defect at sign-off. Verify now samples duplicate groups with their cd spread and classifies each against the EXACT migrated-data boundary (max chunk upper_cd from the ledger): groups entirely above it are live at-least-once artifacts (reported, do NOT fail verification — the nightly job cleans them); any group reaching below it involves migrated data and fails verification for investigation. UI verify panel shows the attribution. Also the written record of the compatibility audit against platform main-as-deployed: ingestor owns the [CLY]_custom/n mapping and cd stamping, EventDeduplicationJob's 26h/7d cd window can never scan or delete historical-cd migrated rows, and cross-cutover SDK-retry dups resolve to the older (migrated) copy when the job sees both. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Arturs' question 'can new incoming data mix with migrated data in these checks?' had a real yes: cd-window checks can't mix (live cd is always newer than every migrated window), but ID-BASED checks could. An SDK retry straddling cutover lands the same _id in both stacks, in the SAME ts-month partition — so attach-recovery's staged-ids sample could see the live retry copy, conclude 'partition already attached', and skip a never-attached partition (silent loss). Duplicate attribution by cd boundary was likewise heuristic at the edge. Now provenance is a column, not an inference: - connect() adds 'migrated Bool DEFAULT false' to the live table (metadata-only ALTER, instant at any size; live inserts default false) - the INSERT layer stamps migrated=true on every row (staging + DLQ replay); the transform/goldens stay unaware — it's transport metadata - every migration-side query filters on it: staged-ids attach recovery, window counts (verify/monitor/rebuild), window purges, by-id purges, null-cd sweep attribution - verify's duplicate attribution is now exact with three verdicts: 0 migrated copies = live at-least-once artifact (nightly platform job cleans), 1 = cross-cutover retry (benign, reported), 2+ = migration defect (fails sign-off) - guard: resuming a run whose completed chunks predate the flag fails fast with the backfill recipe (checks would otherwise see zero rows) New precision test pins the loss vector: staged-ids check returns 0 when only a live retry copy of a staged _id exists, 1 once the migrated copy is live. 93 tests green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… the migrated column Team direction: no new column on the production drill_events table. The column is gone entirely (no ALTER, nothing for future rows to inherit) and the 100% migrated/live distinction is preserved by construction: cd IS the provenance marker. Migrated rows carry historical cd from the source; live rows are stamped at post-cutover insert time. A cross- cutover SDK retry shares _id with its migrated twin but can never share cd — so where an id alone is ambiguous, checks match (_id, cd) pairs: - attach-recovery (the loss vector): staged (_id, cd) pairs vs live — the retry copy is invisible to it, the chunk's own promoted rows are matched exactly - purges: deleteLiveByPairs (parallel arrays zipped server-side — Array(Tuple) params don't parse over HTTP); the null-cd sweep purge reconstructs its ts-derived cd values so even it is pair-exact - verify's duplicate attribution: classified against the ledger's end-of-migrated-data boundary (max chunk upper_cd) — same three verdicts (live artifact / cross-cutover retry / migration defect) - window counts need no pairs at all: the historical cd range excludes live rows and is minmax-index-accelerated (cheaper than any flag) New preflight check guards the one assumption this rests on: 'Source frozen & clocks sane' fails if the newest source cd is within 60s of ClickHouse server time (source still ingesting, or skewed clocks would blur the boundary). Cost note: pair matching runs only on rare recovery/purge paths where the partition scan dominates either way; the hot checks use the cd minmax index. Nothing is stamped on future live rows, ever. 93 tests green (attach-recovery precision test now pins pair semantics). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…e 26.4) CI previously ran only typecheck + the pure differential harness; the 93-test suite (engine e2e, multi-collection scoping, rebuild, duplicate attribution, pair-based recovery) now runs against service containers pinned to the production ClickHouse version (26.4, per countly-platform deploy/compose/images.standard.env). 26.4 requires a password for non-localhost clients, so the tests accept TEST_CLICKHOUSE_URL/ TEST_CLICKHOUSE_PASSWORD (defaults unchanged for local runs). Verified on GKE the same day: image runs on a dedicated cluster against CH 26.4 — 120,200 docs exact, all chunks promoted via real ATTACH, 2-pod run with a SIGKILL mid-flight converged exact (zero loss, zero duplicates). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Live-review findings from Arturs walking the dashboard: - docs/second kept 'declining' after completion — the elapsed clock never froze, so the card showed the run average decaying while the finished engine idled. finishedAt now freezes the clock (completed/stopped/ fatal); the card shows the true run average with an 'avg' suffix, or a dash when this process copied nothing. - 'Docs migrated' now prefers the durable ledger sum (done chunks' rows_expected) over process-local counters, so a restarted engine shows 80,001 — not 0 — for a completed run. - DLQ panel now answers 'where do I run the update?': names the fix location (<manifestDb>.mig_dlq_docs — Replay re-transforms the STORED raw_doc, never the source) and each entry carries its source collection plus a copy-pasteable updateOne targeting its dlq _id. - expanded DLQ entries survive the 2s re-render (open-state preserved by dlq _id). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Review findings from Arturs (what happens at a billion DLQ docs; why is the tool ASKING the operator to set read preference when it can decide): Scale: - /api/chunks now ships an O(collections) aggregation summary (status counts, docs done, remaining estimates per collection); full chunk details only under 2,000 chunks, else just pending/active/failed capped at 500 — a 10TB run no longer streams tens of thousands of chunk docs to the browser every 2s. Cards, bars, ETA and gates all compute from the summary; the chunk map notes when done-cells are summarized away. - DLQ panel paginates (8 per page, stable _id order, Prev/Next with 'x–y of N pending'); counts stay index-served aggregates. - CoercionCounter caps distinct (rule, field) keys at 10k with an overflow bucket — totals stay exact under pathological field-name cardinality. Self-driving checks: - MONGO_READ_PREFERENCE defaults to 'auto': the engine probes hello at startup and picks secondaryPreferred on replica sets itself (frozen source ⇒ secondary reads exact). Explicit env still wins; preflight and the config card show '(auto-selected)'. - New preflight check 'Old ingestion stopped (source frozen)': double probe of newest cd + estimated counts 4s apart — any advance fails the check and names the still-growing collections. - New preflight check 'New ingestion flowing into ClickHouse': rows with cd in the last 15 min (pass with count, warn when zero — traffic may legitimately be zero). Both checks are topology-agnostic: they read only the source handle and the target handle, so new-cluster and same-cluster migrations behave identically. 96 tests (coercion cap, frozen-probe detection, DLQ pagination added). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…, full DLQ drain
Second pass of the billion-document audit, this time below the UI:
- verifyMigration recounted every window SEQUENTIALLY inside one HTTP
request — hours and a guaranteed timeout at tens of thousands of
chunks. It now runs as a background task (POST /control/verify +
GET /api/verify with {status, progress, result}), counts windows with
bounded concurrency (8), and reports live progress; the UI button
polls and shows 'checked X/Y · phase'.
- The global uniqExact(_id) + whole-table GROUP BY duplicate check can
exhaust ClickHouse memory at billions of distinct ids. Replaced with
duplicateStats(): partition-by-partition scans (external group-by
enabled) — exact for every duplicate class we act on, because copies
of the same document share their ts month (a retry RESENDS the same
event ⇒ same ts ⇒ same partition), and memory-bounded per month.
Dead countAndUniq/duplicateSample removed.
- replayDlq silently processed only the first 10,000 pending entries
(listPending's default limit) — one click on a large DLQ reported
success while replaying a fraction. Now a keyset drain (pages of 500
by _id) processes the entire queue; still-failing entries stay
pending but sort behind the advancing cursor, so it terminates.
Batch dedup tokens are keyed by the page's first _id (stable across
retries, unlike positional counters over a shifting list).
- Test fixture correction that validates the partition assumption: a
cross-cutover retry shares the original event's ts (a retry resends
the same event) — the earlier fixture gave the copy a fresh ts, which
no real duplicate has.
96 tests green; async verify exercised live end-to-end.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ot three Setup story now matches reality: MONGO_URI and CLICKHOUSE_URL are the only variables an operator must set. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Gap flagged by Arturs: everything so far used STATIC post-cutover rows — no test ever ran a continuous live writer against drill_events while the migrator was copying, attaching, and monitoring. New suite runs a 40-rows/30ms writer for the entire migration (starting before, ending after), adversarially aimed: half the rows land in the SAME (a,e,n) scope being migrated, and the source's newest slice carries recent ts with historical cd so its chunk ATTACHes into the CURRENT month partition — the exact partition live traffic is inserting into. The invariant monitor runs hot (150ms) throughout. Asserts: engine completes unpaused (no false invariant trip from live rows), migrated counts exact (30k, zero dups), every live-written row survives untouched, the hot partition really contains both populations, and full verify passes with live data present. 97 tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…hen-replay dedup
Arturs: 'what happens if we have more problematic docs than 10k? at 10B
scale that is under 1% and we HAVE had those.' The 10k replay cap was
fixed earlier today; this closes what the question actually exposes:
- GLOBAL DLQ PAUSE GUARD (LEDGER_DLQ_PAUSE_THRESHOLD, default 1M, 0
disables): the per-chunk breaker (5% of one chunk) never trips on
evenly-spread failure — 1% of every chunk on a 10B run would silently
accumulate ~100M raw docs into the manifest DB. The engine now checks
total pending after each chunk (cheap in-process pre-filter) and at
run start (so a resumed run inheriting a mass DLQ pauses immediately),
and pauses for an operator decision.
- REPLAY IS A BACKGROUND TASK (POST /control/replay-dlq starts, GET
/api/replay has {status, progress, result}) — millions of entries are
not one HTTP request's work. DLQ panel shows live progress.
- REDO-THEN-REPLAY CANNOT DUPLICATE: replay now skips entries whose
rows are already live as (_id, cd) pairs and marks them resolved
('already live — no insert'). This is the safe bulk path for mass
DLQ: fix the transform, Retry failed chunks (redo re-reads the
source), then Replay resolves the stale entries without inserting.
- DLQ counts/topErrors aggregations cached 15s in the route — the 2s UI
poll stays harmless against a 100M-doc DLQ collection.
Runbook row added with the mass-DLQ playbook. 99 tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The 1M pause is a decision point — give the operator the numbers the decision needs: the guard's pause log now includes the DLQ collection's storage size and the manifest DB's disk-free %, and the DLQ panel shows a storage pill (rides the 15s aggregate cache). Raising the threshold is now an informed 'the disk can afford it', not a guess. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replay became a background task; the button receipt now points at the live progress line instead of claiming completion. Found in the full UI button sweep (all 20+ interactions re-tested in-browser: tabs, pause/resume, two-step retry/waive, background replay with progress, DLQ pagination + storage pill, preflight with the new automated checks, index build, dry run, async verify with gate, checkbox persistence across reload, rebuild refuse→force flow, Help-tab cross-pane actions). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The console can pause runs, purge windows, and rebuild the ledger — it must not sit open on a reachable port at a customer site. One env var enables HTTP Basic on everything except /healthz (constant-time compare, any username, browser prompts natively — zero UI changes). Unset = open, for localhost/port-forward setups. Verified live: 401 without/with wrong password, 200 with correct, /healthz stays open for probes. Also probed 1,000-collection scale end to end (real deployments have thousands of hashed collections; tests used two): exact migration, 1,000 chunks, dashboard summary query 29ms. Two quantified limits, no code change needed yet: preflight is ~17s at 1k collections (sequential per-collection probes — parallelize if 5k+ deployments appear) and per-collection fixed overhead is ~140ms (irrelevant when collections hold real data volumes). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Arturs' call: deployments run the console on non-reachable ports (localhost / port-forward / cluster-internal), so DASHBOARD_PASSWORD is complexity without a user. The 1,000-collection probe findings from the reverted commit's message still stand (they were code-free). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Verified live: 2 containers via docker compose --scale against one ledger — 16 chunks split 8/8 by container-hostname pod ids, 120,000 rows exact with zero duplicates, both containers exit 0 on completion (EXIT_ON_COMPLETE). Recipe documents the two real-world caveats: scale across machines (CPU-bound per container) and drop the fixed published port when scaling on one host for tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…mparison Arturs' question: commit gates are count-based, not doc-per-doc — what can that miss and how do we recover? The honest map has two blind spots that deserved tooling, not documentation: 1. SELF-CONSISTENT UNDER-READ: a reader that silently loses cursor tail produces tally == staging count == live count — every existing check passes while docs are missing, because they all compare against the tally. 'Audit vs source' (rebuild machinery in checkOnly mode) is the defense: recount every window directly against MongoDB and report disagreeing windows WITHOUT touching the ledger. The source is the truth, not the tally. 2. RIGHT COUNT, WRONG CONTENT: a transform bug produces the correct number of corrupted rows — invisible to any count forever. 'Content sample audit' re-transforms random source docs (deterministic index-served probes, no $sample scan) and compares them field-by-field with their live rows: scalar columns exact, JSON columns by top-level key set (value-level JSON equality is the differential harness's job — ClickHouse normalizes encodings). Both are background tasks with Guide-phase-6 buttons and result rendering; recovery for anything flagged is the standard invariant — purge the scoped window, redo from the frozen source. Pinning test choreography proves complementarity: wrong-uid row (same _id/cd/counts) → content audit catches with field attribution while source audit stays green; deleted row → source audit flags the exact window; restore → both green; checkOnly verifiably leaves the ledger byte-identical. 100 tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Arturs asked whether the audits add load and whether they should run per commit. Answering it properly surfaced two things: 1. PER-COMMIT UNDER-READ GUARD (LEDGER_SOURCE_COUNT_CHECK, default on): the audits stay on-demand, but the cheapest tally-independent check — 'how many docs does the source say this window holds?' — costs one indexed countDocuments (~1% of chunk duration) and closes the self-consistent-under-read blind spot AT COMMIT TIME instead of at sign-off. Mismatch flags the chunk failed for standard purge+redo. 2. SCALE BUG: fetchRowsByIds / fetchLiveCdByIds / deleteLiveByPairs filtered ClickHouse by bare _id — not in the ORDER BY, so each call full-scanned the _id column (mass DLQ replay did that PER 500-DOC BATCH: days instead of minutes at 10B rows). Every caller knows its rows' cd values; all three now take cd min/max bounds and prune to the relevant partitions (replay batches, content-audit samples, rebuild sweep attribution via ts-derived cds, pair purges). Two fixes found by the new test running against the changes: checkOnly no longer reports pending (live=0) windows as disagreements, and toString(cd) AS cd alias-shadowed the WHERE bound column in ClickHouse (String vs DateTime64 type error) — aliases renamed. 100 tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Found by live-clicking the new Audit-vs-source button (CI could not see this — the demo state had WAIVED docs): a window whose only source-live difference is its own DLQ'd docs was flagged as a disagreement, and the render's advice (rebuild + retry) would just re-DLQ them — an operator confusion loop. DLQ entries now carry cd_ms (derived from raw_doc.cd, ts fallback, null when unparseable) with an index; the source audit and rebuild classification subtract known-unmigrated (non-resolved) docs per window: live + unresolved == source → agreement/done. Entries written before this field can't be attributed and count zero (documented). The per-commit source-count guard needs no such adjustment — it compares against docs READ, which includes later-skipped docs. Verified in-browser: the previously-flagged demo window now passes; new pinning test (waived doc absent from live → audit green, restore → still green). 101 tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
CI runner finished the migration faster than the local box, so the writer landed 920 rows against a hardcoded >1000 threshold. Assert the structural property instead: batches landed DURING the run (>100 rows before completion) and the writer continued after — magnitude varies with runner speed and proves nothing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
A ground-up replacement of the migration engine, built for the three problems that block large (10TB-class) drill_events migrations today: instability under bad data, throughput, and the operational risk of long ingestion downtime.
This is the complete solution, not a slimmed-down PoC — every improvement from the redesign is included (chunk ledger, DLQ with raw docs, verify-then-attach, poison-pill quarantine, circuit breaker, error classifier + bisection, sampled dry run, preflight incl. clock-skew check, index builds from the UI, multi-pod scaling, ledger rebuild, duplicate attribution, operator dashboard with embedded runbook, k8s manifests, CI). What the branch does NOT carry is the legacy engine: no Redis code paths, no dual-engine switches, no dead config. Reviewing against
mainis therefore a clean either/or comparison of two complete engines rather than a diff tangled through both.Architecture in one paragraph
Work is cut into cd-bounded chunks tracked in a MongoDB ledger (
mig_ranges) — ~50-100 tiny documents, claimed atomically with leases, safe for N pods with zero coordination infrastructure (Redis is gone entirely). Each chunk copies into its own ClickHouse staging table (sync inserts), is count-verified there, then promoted into the live table via verify-then-ATTACH per partition (INSERT SELECT fallback). Recovery never trusts the ledger: in-flight chunks are redone, promoted chunks are recounted, half-attached chunks are checked by staged(_id, cd)pairs. Unmigratable documents land in a DLQ with their full raw source (replay / waive from the UI); repeated crashers are auto-bisected down to the poison document; a circuit breaker pauses on systematic failure. A built-in dashboard (/) carries the whole runbook: preflight, index builds, sampled dry-run, live progress, verification gates, incident recovery — self-hosted customers can run this without us.Key correctness properties (each has a pinning test)
uniqExactcheck; continuous invariant monitor(a, e, n)identity — overlapping hashed collections cannot corrupt each other (purge/verify/recovery)cdconstruction +(_id, cd)pair matching; cross-cutover SDK retries are harmless; nothing added to the production tableRebuild ledger from dataregeneratesmig_rangesby recounting windows (Mongo vs scoped ClickHouse)Evidence
What reviewers should look at first
src/runtime/chunk-orchestrator.ts— the chunk lifecycle + every recovery pathsrc/state/ledger-store.ts— claim/lease/transition semanticssrc/target/staging-manager.ts— staging lifecycle, verify-then-attach, scoped queriesdocs/RUNBOOK.md— the cutover choreography and incident tabletests/integration/— the correctness contract, in executable formNot in this PR
Platform-side items tracked separately: countly-platform#1105 (cd passthrough, draft — interacts with EventDeduplicationJob), #722 (dedup job replay resilience). This tool depends on neither.
🤖 Generated with Claude Code