Move flattening to the client and compaction to a merge

[zmaril]

A contributed build now arrives as the public tables for that build — nine parquet files framed into one compressed bundle — and compaction combines them by copying encoded row groups. It never decodes a value, never holds a row, and needs no parquet codec at all.

This is not deployable by merging alone. See "Deploying this" at the bottom.

Why

Compaction read every stored session, parsed it, flattened it, and re-encoded the result — holding every row in a 128 MB isolate. Measured on 400 staged files holding 684,800 rows:

	reading rows	copying row groups
time	~480 ms	16 ms
memory	~152 MB	12 MB

152 MB does not fit in the isolate. The ceiling in the ops notes stops existing, because memory is now flat in row count — only row-group metadata accumulates.

One object per build, not nine

Parquet holds one schema per file, so a build is nine files. Measured on a real session:

	bytes	vs the JSON it replaces
nine separate files	67,750	4.28×
one bundle	28,386	1.79×

A quarter of the nine-file version is footers. Framed together and compressed once, the outer compressor sees what the tables have in common — run_id is in every one of them. It also drops nine Class A writes per build to one.

The framing is deliberately dull, since nothing reads a staged object except compaction, which slices files out by offset. There is a test asserting the unframed bytes are views into the buffer rather than copies, because a copy would quietly undo the point.

object_key is gone from sessions

A merge cannot write a value the client did not, and the client cannot know its own object key — the file exists before the upload. The column named a staged object, which is now transient: merged, then deleted. Rows are identified by run_id.

Schema fingerprints

Row groups only copy between files whose schemas match to the byte, annotations included — I found this by merging our writer's output with DuckDB's, which differed only by converted_type: INT_64 versus NONE. So the client declares a fingerprint in x-cratebank-schema and ingest files objects under it.

That value is attacker-controlled, so it is reduced to at most 32 hexadecimal digits before it touches a key, falling back to unknown.

Ingest now requires application/zstd

Header-only, so the body still streams past untouched and a 100 MB upload still survives a 128 MB isolate. This endpoint is unauthenticated by design, and 294 junk objects accumulated in a day and a half — that is what was failing every deploy. A scanner sends form-encoded or nothing.

What got deleted

ingest.ts, compact.ts, compact.test.ts, the old wrangler config, and the client's entire JSON transport. ship.rs is now one function that posts bytes — which also deletes the second, discarded compression it did on every successful run, about 44 ms measured.

Net: +825 / −1,452.

The Workers move to wrangler

OpenTofu cannot upload a Worker with a separate wasm module — its multipart upload omits the wasm part, and the runtime refuses to instantiate a module from bytes, so it cannot be inlined either. wrangler deploy with worker-build does the whole thing.

wrangler owns the scripts, bindings, cron, and workers.dev subdomain; OpenTofu keeps the bucket, data.cratebank.io, and the custom domains, which reference scripts by name. This also fixes the provider drift the infra README complains about, since script content leaves state entirely.

removed blocks make OpenTofu forget the scripts rather than destroy them. Without those, the first apply after this merges would take production down.

Verification

The client path and the compaction path produce identical published tables. Two real sessions, every row of all nine tables hashed and summed:

sessions	units	phases	timeline	unit_flags	artifacts	edges	compiler_units	build_config
✓	✓	✓	✓	✓	✓	✓	✓	✓

Not equivalent — identical. That check lives in examples/edge.rs and is the thing to run when either half changes.

Also: 5 test suites green, clippy clean with all features, both wasm crates building, tofu validate passing, and a real build through the local collector reporting nine tables and 55 rows from its footers alone.

The client's cold build goes 16s → 22s with parquet in the tree, 185 → 227 crates. Paid once, on install.

Deploying this

In order:

1. cargo install worker-build
2. cd crates/worker-ingest && npx wrangler deploy
3. cd crates/worker-compact && npx wrangler deploy
4. Let a deploy run so OpenTofu processes the removed blocks — before any apply that does not have them
5. The removed blocks can be deleted after that apply

Already done: COMPACT_SECRET is set on the compact Worker via wrangler and verified against the live endpoint, and the bucket has been reset — nothing reads the old JSON format any more.

client/latest.json went with the reset and comes back on the next deploy.

🤖 Generated with Claude Code

https://claude.ai/code/session_01361quprG7tUgaVFKAmKtJZ

[github-actions]

cratebank preview

Preview removed because this pull request is closed.