Skip to content

Replace a span's attribute rows and search entry only when an export repeats it - #9

Draft
RhysSullivan wants to merge 1 commit into
mainfrom
fix/indexed-span-replacement
Draft

RhysSullivan wants to merge 1 commit into
mainfrom
fix/indexed-span-replacement

Conversation

@RhysSullivan

Copy link
Copy Markdown

Each trace export ran DELETE FROM span_operation_fts WHERE (trace_id, span_id) IN (…) for every span it carried. trace_id and span_id are UNINDEXED FTS5 columns, so that delete read every search entry in the store: each export cost more as the store grew. It also deleted and reinserted each span's attribute rows, one statement per span.

A span is stored again only when an export repeats it. The store now checks the span's primary key first, and only for a repeated span deletes its attribute rows and search entry. New spans, nearly all of them, skip both. An export's attribute rows go in one statement. When one export lists a span twice, only its last listing is kept; before, both operation names stayed searchable.

Executor's e2e Cloud suite runs one collector for 40+ concurrent scenarios. The collector refused 540–1,828 exports per CI run as queueFull (16 pending), and spans the scenarios assert went missing.

Measured with 4,000 exports captured from that suite (16 concurrent senders, a fresh workerd collector each run, interleaved, n=4 each):

workerd CPU per export (median) exports/s (median)
main 8.9 ms 93
this PR 6.5 ms 119

Throughput per 1,000 exports also no longer falls as the store grows.

Verified: bun test (218 pass; the daemon health timing test fails intermittently on this machine, also on main), typecheck. The test is in the PR above this one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant