Repository navigation
Fix leaks and round-trip failures found by fuzzing - #89
Merged
Merged
Conversation
A randomized round-trip and leak fuzzer (documents with ranks, names in
all cases and letter cases, numbers, dates, non-breaking spaces, tabs,
CRLF and hard line wraps) found these; each has a test in
tests/test_fuzz_findings.py.
Names left unmasked:
- A full name before a rank followed by more words ("Петренко Іван
Іванович доповів капітану про виконання") was not masked: the parser
took name anchors only after the first rank of the line.
- A line break inside a name left part of it open. Lines are joined for
parsing when a name with a patronymic, or a rank and its name, crosses
the break; the break comes back after masking.
- Initials and a surname on adjacent lines; runs of initials and
surnames ("Ю.А. Зінченко Н.І. Савчук"), now resolved by taking the set
of matches that covers the most names.
- Double surnames in capitals and surnames with an apostrophe before or
after initials; a heading in capitals before a name; adjective
surnames in oblique cases; the vocative "Любове".
Unmasking did not restore the original:
- "генерал-майор медичної служби" as a mask (rank boundaries now exclude
a hyphen).
- Non-breaking spaces or tabs inside a rank whose mask had a different
number of words; ranks broken across lines.
- An open rank "... медичної служби" used as the mask of another rank.
- A patronymic before initials masked as a surname, then masked again.
- Two forms with the same mask in one line restored the wrong way round:
masks in a line are now computed in text order.
- A brigade name broken across lines lost its tail in the masked text.
Masks: naval ranks get naval masks; the feminine genitive of an
adjective surname shares the stem of its other forms.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XT6iUWaQgahXDB9TWX9Bq7
Generated-With: Claude Code 2.1.42
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A randomized fuzzer (documents with ranks, names in every case and letter case, numbers, dates, non-breaking spaces, tabs, CRLF and hard line wraps; round trip, leak and over-masking checks) found the problems below. All of them except the hang were already on
main. Each one has a test intests/test_fuzz_findings.py: 56 of its tests fail onmain, and the rest are guards against over-masking.Names left unmasked
Петренко Іван Іванович доповів капітану про виконання.The parser looked for name anchors only after the first rank in the line.Петренка\nІвана Івановича,Петренка Івана\nІвановича,Шановний Віталію\nРомановичу!). When a name with a patronymic, or a rank and its name, crosses the break, the two lines are now parsed together, and the break is put back after masking.Ю.А. Зінченко Н.І. Савчук, only the middle pair used to be masked. The masker now picks the set of matches that covers the most names.ПЕТРЕНКО-КОВАЛЬ І.М.) and surnames with an apostrophe (ПИЛИП'ЮК О.І.).ВИТЯГ З НАКАЗУ Коваль …), an adjective surname in the genitive (Рапорт Білого Ярослава Миколайовича), and the vocativeЛюбове.Unmasking did not restore the original
генерал-майор медичної службиwas not restored. Rank boundaries in the unmasker now exclude a hyphen.… медичної служби(one without a name next to it) could become the mask of another rank.Мороз Ольги Богданівни Л.А. Мороз, the patronymic before the initials was masked as a surname and then masked again.ІванеandІвану(both masked asІгорю) were restored the wrong way round.Masks
капітан 1-го рангуused to becomeмайор).Білоїnow gets the same synthetic stem asБілаandБілій.Checks
--encryptand--re-mask 2.main, the only mask change ininput_example.txtis a naval rank. In the generated documents, the changes are the fixes listed above.Changes are under
## [Unreleased]in CHANGELOG.md. The version is not bumped, per the release-time bump rule.Generated by Claude Code