Repository navigation
Remove and clean-room reimplement code without relicensing consent (#154) - #201
Merged
Merged
Conversation
Frank Fischer declined consent to the license change proposed in #154 and asked for all of his changes to be rewritten. This removes what remains of his 2011 commits ahead of an independent reimplementation: - bfaa188: the dynamic field writer branch of Entry#method_missing and the matching clause in Entry#respond_to_missing? - 8eb3979: '/' in the lexer's name, spaced_name and key patterns (including the key patterns, which inherited it) - 7778f42: the body of Bibliography#save_to, which now raises NotImplementedError The branch is intentionally red until the reimplementation lands: 24 minitest errors from field writers (e.g. entry.address = ...), 2 from save_to, and the cucumber scenario features/issues/slash_keys.feature:6 (DBLP:journals/dq/RossinK99).
Record the lexer's token stream for every fixture and for each section of errors.bib, under six lexer configurations, before the identifier character set is widened (R2). Also pin the parsed results: entry counts, keys, field names, @string names, error traces, and the ParseError raised for errors.bib as a whole. Refs #154
Cover entry.foo = value adding or replacing a field with the same value conversion as Entry#add, respond_to? returning exactly true for Symbol and String writer names, "=" alone not being a writer, and the existing readers, convert_/from_ ghost methods, parent delegation, real writers and operators staying as they are. Refs #154
Exercise every example character from REQUIREMENTS.md in field names, @string names and macro references, in strict and non-strict mode, with and without :strip_comments. Cover % handling, @ inside a name, names starting with a digit, multi-word field names in non-strict mode, and keys containing / with :strip_comments and allow_missing_keys. Add a cucumber feature with the parsing examples and a step to parse with lexer options. Refs #154
Cover writing to_s with { } as the default :quotes, honouring options,
UTF-8 output regardless of the default external encoding, round-trips
through BibTeX.open, overwriting, String and Pathname paths, returning
self, and leaving an existing file intact when serialisation raises.
Refs #154
Serialise the bibliography with #to_s before touching the file system, so an exception raised while writing out an element propagates and leaves an existing file untouched and does not create a missing one. Values are enclosed in braces unless the caller passes a non-nil :quotes; all other options go to #to_s unchanged, and the caller's hash is not modified. The content is always UTF-8, independent of Encoding.default_external, so it reads back through BibTeX.open. A symlink is resolved first. A regular file that may be written, or a missing one, in a writable directory is replaced atomically: the content goes to a temporary file with a short fixed name in the same directory, which is flushed, fsynced (skipped where fsync is not implemented) and renamed over the target, and removed again if anything fails, so a full disk or a crash never leaves a partial file. The file keeps its permission bits without setuid, setgid and sticky; a new one gets 0666 minus the umask. Everything else (a read-only file or directory, a device, FIFO or socket, a dangling symlink) is written directly, so the OS behaves and fails as for File.write. Both paths write in text mode. Hard links and directory fsync are out of scope. Refs #154
Entry#method_missing now treats any name of the form <field>= as a writer for that field and stores the value through #add, so it gets the same Value conversion as #add, #[]= and the generated writers of the required and name fields. The writer pattern requires a field name that does not end in '=', so :'=' (and names made only of '=') stay unknown methods and raise NoMethodError. A wrong number of arguments raises ArgumentError, as a real writer would. The writer and conversion patterns are private constants shared by method_missing and respond_to_missing?, which now uses Regexp#match? and so answers true or false rather than a match position. Field readers and delegation to a cross-referenced parent are unchanged. Refs #154
Field names, @string names and macro references now accept what bibtex.web's id_class allows: any printable ASCII character except white space and " # % ' ( ) , = { }. The project's extensions stay: non-ASCII letters and digits, and '%' unless :strip_comments makes it start a comment. The illegal characters are listed once; a helper derives the character class with or without '%' (escaped, as it is spliced into a class), and the :name and :spaced_name patterns (and their _without_percent variants) are built from it. A name still may not start with a digit. '@' may start a name only when it cannot start an object. Identifiers cannot contain { ( = , } #, so the character after '@word' decides: a name if it is = , } or # (after white space, and TeX comments when they are stripped), an object start otherwise. This keeps error recovery after an unterminated object, which relies on '@' starting the next one, in strict and non-strict mode. The words of a non-strict multi-word name use the same rule, so only the last one can start with '@', and as before a word may start with a digit. A word that starts with '@' ends before the next '@', so @A@b = x is not a name (a@b and a@b@c still are). Otherwise the lookahead would be retried from every '@' of a long run like @A@a@a... and rescan the rest of it each time, which made lexing quadratic. Keys accept '/' in addition to the characters accepted before; their character list is also kept in one place and shared by the four key patterns. All Lexer.patterns keys keep their names, and the recorded token streams of the fixtures are unchanged. Refs #154
The clean-room commits listed their changes relative to the stripped snapshot, so some bullets described existing behaviour as new (field writers, '/' in keys) or claimed a respond_to? change that users never saw: Ruby already returned true or false. Keep only what changes for users since 6.2.0. Refs #154
The clean-room tests and one lexer comment referred to the requirements document (R1-R3), which is not part of the gem, and to an issue number. Like the rest of the suite, comments and test names now describe what is tested; the issues stay in the commit messages and History.txt. Refs #154
tmaier
force-pushed
the
rewrite-154-fischer
branch
from
November 14, 2026 22:08
dbbd73c to
b9bf376
Compare
On Windows save_to writes in text mode, so the file has CRLF line endings, as before. Comparing the raw bytes with to_s failed there; reading the file back as UTF-8 text checks the encoding on every platform, and the bytes are still checked for UTF-8 umlauts. Refs #154
A mutation test of the lexer showed which regressions the suite misses. These tests catch them directly, without a recorded token stream: - test_error_recovery.rb takes over the errors.bib checks section by section, in strict and non-strict mode, and adds an unterminated braced value and an indented entry after free text in non-strict mode - test_parser.rb pins how meta content between objects is kept, a TeX comment before the key with :strip_comments, and the rejection of invalid characters in entry types and of a line break or tab before the key's comma - test_bibtex.rb gets the decoret and bibdesk checks it was missing, and test_roundtrip and test_construct are real tests again; the dead test_errors stub is gone Refs #154
test/fixtures/lexer_tokens.json recorded the full token stream of every fixture under six lexer settings. Two thirds of the streams were copies; it could not be regenerated; String#inspect made it depend on the locale for non-ASCII text; and the fixtures lack the input that the new identifier rules affect, so it guarded less than it seemed. Of 14 lexer mutations, the targeted tests from the previous commit catch every one the token pins caught, and five that nothing caught before. Refs #154
21 of 25 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of the relicensing in #154. A 2011 contributor declined consent and asked for all of his changes to be rewritten. This PR removes what remained of his three commits and reimplements that behaviour independently. While doing so, it fixes two real
save_tobugs and accepts the full BibTeX identifier syntax.What was removed (e75779a)
entry.foo = v,respond_to?(:foo=))Entry#method_missing,#respond_to_missing?/in identifiersname,spaced_nameandkeylexer patternsFile.openblockBibliography#save_toAfter the removal, 26 minitest errors and the DBLP slash-key cucumber scenario failed, as expected.
How it was reimplemented
id_classinbibtex.web, BibTeX's own source.File.write(path, content, encoding: …)and the key character list. The judge's quality findings, such as read-only targets and quadratic lexing on adversarial input, were fixed by the implementer, not by anyone who had seen the history.Refs #154in each message. The tree was byte-identical to the clean room's, apart fromREQUIREMENTS.md, which is not shipped. Two maintainer commits follow:REQUIREMENTS.mdand an issue number from comments and test names, following Remove issue numbers from code comments and test names #200.normalize_monthfrom Stop Entry::MONTHS_FILTER from growing with every parse #196 andwrite_fieldlanding at the same place inEntry;StringScanner#skip;Changes for users (History.txt)
@stringnames and macro references accept every character BibTeX allows, e.g.x;y,p[1],a|b,a<b>,a@band@foo. These used to raise an unexpected-token error.@followed by a name and{or(still starts a new entry, so error recovery is unchanged (covered bytest_error_recovery.rb).save_to/savereplace the file atomically. A failure while serialising or writing now leaves the existing file untouched; before, it could be left empty. Symlinked targets and permission bits are kept. Devices, pipes and files in read-only directories are written directly.save_tono longer adds:quotesto the options hash you pass in.Field writers,
/in names and keys, UTF-8 output and Pathname support work as before.Test changes after review
The clean-room test author also recorded a golden master,
test/fixtures/lexer_tokens.json(143 KB): the exact token stream of every fixture and everyerrors.bibsection under six lexer settings. It had several problems:String#inspectmade it depend on the locale for non-ASCII text;An independent review ran 14 lexer mutations against the suite. Based on that, the file is replaced with targeted tests:
1db7c6dadds the new tests:test/bibtex/test_error_recovery.rbtakes over theerrors.bibchecks section by section, in strict and non-strict mode. It adds an unterminated braced value and an indented entry after free text in non-strict mode.test_parser.rbcovers meta content between objects, a TeX comment before the key with:strip_comments, and the rejection of invalid entry-type characters and of a line break or tab before the key's comma.test_bibtex.rbgets the missing decoret and bibdesk checks, andtest_roundtripandtest_constructare real tests again.ecdc1fedeletes the golden master and its test.The new suite catches every mutation the golden master caught, plus five that nothing caught before. Two remain uncaught deliberately: allowing a newline between
@and the entry type, which BibTeX accepts, and trailing whitespace after a comma, which another token already consumes. The file stays in the history because this PR is merged with a merge commit, but it is not in the PR's diff.Evidence
rake test: 907 runs, 2093 assertions, 0 failures, 0 errors, 2 skips (the existing RDF skips), on Ruby 3.4.12 and 4.0.6. The count is lower than before because the 126 token-stream tests and their parse duplicates are gone.cucumber: 141 scenarios and 414 steps passed, on Ruby 3.4.12 and 4.0.6.BibTeX.parse: 70–72 ms on master, 63–64 ms on this branch, with 404,021 objects instead of 406,021.'@' + 'a@' * 50_000 + ' {'and similar): 0.07–0.09 s. These inputs are now linear-time regression tests.Known limits and follow-ups (not in this PR)
@ends before the next@, so@a@b = xis not a name. BibTeX would accept it.Entry.newdeletes keys from the attribute hash passed to it (lib/bibtex/entry.rb:72), so it modifies the caller's hash. This is an existing bug, like the:quotesone fixed insave_to;test_constructpasses a copy.META_CONTENTinLexer::BACKTRACE_BOUNDARIESappears to be unused: error recovery always reachesATfirst.