[jamalkamaladdin]

Fixes

Fixes #352

Summary

AhocorasickTokenizer.get_extractors returned a set. Set order depends on PYTHONHASHSEED. For 37 T.C. at 155, one full cite token and two short cite tokens share one span. tokenize takes the first of them.

get_extractors now sorts selected extractors by their index in EXTRACTORS. Base Tokenizer uses the same order. 37 T.C. at 155 now yields a ShortCaseCitation on every seed.

New test test_extractor_order_is_stable in tests/test_TokenizeTest.py checks extractor order and the short cite token. Added a CHANGES.md entry.

AI Disclosure

• No AI tools were used to create the content of this PR.
• Parts of this PR were created with the help of an AI tool, and I have carefully reviewed all of its content and take full responsibility for it.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[CLAassistant]

All committers have signed the CLA.

[lokkju]

Tested the head (f54f57f) against eyecite 2.7.8's 86 T.C. at 561 case: ShortCaseCitation under PYTHONHASHSEED 0 to 3, where 2.7.8 alternates with FullCaseCitation. One thought for review, detailed in #352: this makes identical-span ties reproducible, but the winner is still whichever extractor comes first in EXTRACTORS. An explicit tie-break (keep the short-cite token when a short and a full token share a span) would make the outcome principled rather than order-dependent.