Sort completion candidates by frecency from history

[rolandwalker]

Description

• calculate frecency for tokens in recent history, simply tokenizing and ignoring literals
• recalculate frecency in a background thread every N queries at the REPL
• sort completion candidates by frecency

A good followup could be making the sort-tuple returned by completion_sort_key() configurable.

Fixes #351.

Checklist

• I added this contribution to the changelog.md file.
• I added my name to the AUTHORS file (or it's already there).
• To lint and format the code, I ran

  uv run ruff check && uv run ruff format && uv run mypy --install-types .

[scottnemes]

Should this have similar if/else logic as frecency_history_entries above?

[scottnemes]

Potential for caching/memoization if you want to reduce load on larger schemas:

(mycli) arthlo@CAMELOT-2025:~/git/official-mycli$ git diff
diff --git a/mycli/packages/ptoolkit/history.py b/mycli/packages/ptoolkit/history.py
index ba14d38..78b2ba0 100644
--- a/mycli/packages/ptoolkit/history.py
+++ b/mycli/packages/ptoolkit/history.py
@@ -1,5 +1,6 @@
 from collections import defaultdict
 from collections.abc import Iterable, Mapping
+from functools import lru_cache
 from itertools import islice
 import logging
 import os
@@ -56,12 +57,18 @@ def _calculate_frecency(
     return dict(frecency)


-def frecency_score(text: str, frecency: Mapping[str, float]) -> float:
+@lru_cache(maxsize=8192)
+def _frecency_tokens(text: str) -> tuple[str, ...]:
+    """Tokenize a completion candidate. Cached: candidates repeat on every keystroke."""
     try:
         tokens = tokenize(text, dialect='mysql')
     except TokenError:
-        return 0.0
-    normalized_tokens = [normalized for token in tokens if (normalized := _normalize_frecency_token(token))]
+        return ()
+    return tuple(normalized for token in tokens if (normalized := _normalize_frecency_token(token)))
+
+
+def frecency_score(text: str, frecency: Mapping[str, float]) -> float:
+    normalized_tokens = _frecency_tokens(text)
     if not normalized_tokens:
         return 0.0
     return sum(frecency.get(token, 0.0) for token in normalized_tokens) / len(normalized_tokens)

[scottnemes]

Added two comments for potential improvements if desired