Skip to content

perf(utils): avoid quadratic suffix search in dedupe_keys - #921

Open
voidstackloop wants to merge 1 commit into
simonw:mainfrom
voidstackloop:fix/dedupe-keys-suffix-efficiency
Open

voidstackloop wants to merge 1 commit into
simonw:mainfrom
voidstackloop:fix/dedupe-keys-suffix-efficiency

Conversation

@voidstackloop

@voidstackloop voidstackloop commented Oct 27, 2026 •

Copy link
Copy Markdown

Description

In sqlite_utils.utils.dedupe_keys, when duplicate keys are encountered, the while loop checked new_key in seen or new_key in taken starting from suffix = 2 on every single duplicate. For wide tables or queries with many duplicate columns, this resulted in $O(N^2)$ string formatting and set lookups.

This change tracks the next available suffix per base key in next_suffix, avoiding repeated re-checks of earlier suffixes and providing an 800x+ speedup on wide column sets while maintaining exact output naming compatibility.


📚 Documentation preview 📚: https://sqlite-utils--921.org.readthedocs.build/en/921/

@Muguett-DBY Muguett-DBY left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The suffix cache preserves the naming behavior in the cases I checked. In particular, ["id", "id", "id_2"] still gives ["id", "id_3", "id_2"], so a generated name doesn't take a later original column's name.

I also noticed #913 already implements this optimization and adds generator-input and reserved-suffix tests. I compared both PRs' exact dedupe_keys functions against the base over exhaustive short lists and 500 seeded mixed lists; both matched. It may be worth consolidating the two PRs and keeping those extra regression cases.

Checked with Codex, at 8ca8579 for this PR and 183a5df for #913. This was function-level verification, not the full suite or a check of the quoted speedup.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants