Skip to content

Remove and clean-room reimplement code without relicensing consent (#154) - #201

Merged
tmaier merged 13 commits into
masterfrom
rewrite-154-fischer
Nov 14, 2026
Merged

tmaier merged 13 commits into
masterfrom
rewrite-154-fischer

Conversation

@tmaier

@tmaier tmaier commented Nov 14, 2026 •

Copy link
Copy Markdown
Collaborator

Part of the relicensing in #154. A 2011 contributor declined consent and asked for all of his changes to be rewritten. This PR removes what remained of his three commits and reimplements that behaviour independently. While doing so, it fixes two real save_to bugs and accepts the full BibTeX identifier syntax.

What was removed (e75779a)

Commit Change Removed from
bfaa188 dynamic field writers (entry.foo = v, respond_to?(:foo=)) Entry#method_missing, #respond_to_missing?
8eb3979 / in identifiers all eight name, spaced_name and key lexer patterns
7778f42 File.open block body of Bibliography#save_to

After the removal, 26 minitest errors and the DBLP slash-key cucumber scenario failed, as expected.

How it was reimplemented

  1. Requirements. The behaviour was written down from public sources only: README usage, and the identifier class id_class in bibtex.web, BibTeX's own source.
  2. Clean room. A copy of the removal commit with no git history and no remote was given to two separate agents. No copy of bibtex-ruby was installed on the machine.
    • The test author wrote black-box tests from the requirements, including a recording of the exact token streams of every fixture (see "Test changes after review" below).
    • The implementer saw only the requirements, the snapshot and those tests.
  3. Judge. A separate agent on a different model compared every hunk with the removed commits and with every historic version of the touched code, across all branches and the 91 release tags. It ruled on each hunk in three rounds. Result: no suspicious hunks. Two are convergent by necessity: File.write(path, content, encoding: …) and the key character list. The judge's quality findings, such as read-only targets and quadratic lexing on adversarial input, were fixed by the implementer, not by anyone who had seen the history.
  4. Integration. The clean-room commits were applied unchanged, apart from Refs #154 in each message. The tree was byte-identical to the clean room's, apart from REQUIREMENTS.md, which is not shipped. Two maintainer commits follow:
  5. Rebase. The branch is rebased onto master, which now includes Stop Entry::MONTHS_FILTER from growing with every parse #196–Remove issue numbers from code comments and test names #200. All conflicts were mechanical, with no change to the clean-room logic:
  6. Test changes after review. Three maintainer commits on top, described in the next section.

Changes for users (History.txt)

  • Field names, @string names and macro references accept every character BibTeX allows, e.g. x;y, p[1], a|b, a<b>, a@b and @foo. These used to raise an unexpected-token error.
  • An @ followed by a name and { or ( still starts a new entry, so error recovery is unchanged (covered by test_error_recovery.rb).
  • save_to/save replace the file atomically. A failure while serialising or writing now leaves the existing file untouched; before, it could be left empty. Symlinked targets and permission bits are kept. Devices, pipes and files in read-only directories are written directly.
  • save_to no longer adds :quotes to the options hash you pass in.

Field writers, / in names and keys, UTF-8 output and Pathname support work as before.

Test changes after review

The clean-room test author also recorded a golden master, test/fixtures/lexer_tokens.json (143 KB): the exact token stream of every fixture and every errors.bib section under six lexer settings. It had several problems:

  • two thirds of its 126 streams were copies of another setting's stream;
  • there was no way to re-record it;
  • String#inspect made it depend on the locale for non-ASCII text;
  • the fixtures contain none of the characters the new identifier rules affect, so it guarded less than it seemed.

An independent review ran 14 lexer mutations against the suite. Based on that, the file is replaced with targeted tests:

  • 1db7c6d adds the new tests:
    • test/bibtex/test_error_recovery.rb takes over the errors.bib checks section by section, in strict and non-strict mode. It adds an unterminated braced value and an indented entry after free text in non-strict mode.
    • test_parser.rb covers meta content between objects, a TeX comment before the key with :strip_comments, and the rejection of invalid entry-type characters and of a line break or tab before the key's comma.
    • test_bibtex.rb gets the missing decoret and bibdesk checks, and test_roundtrip and test_construct are real tests again.
  • ecdc1fe deletes the golden master and its test.

The new suite catches every mutation the golden master caught, plus five that nothing caught before. Two remain uncaught deliberately: allowing a newline between @ and the entry type, which BibTeX accepts, and trailing whitespace after a comma, which another token already consumes. The file stays in the history because this PR is merged with a merge commit, but it is not in the PR's diff.

Evidence

  • rake test: 907 runs, 2093 assertions, 0 failures, 0 errors, 2 skips (the existing RDF skips), on Ruby 3.4.12 and 4.0.6. The count is lower than before because the 126 token-stream tests and their parse duplicates are gone.
  • cucumber: 141 scenarios and 414 steps passed, on Ruby 3.4.12 and 4.0.6.
  • RuboCop: 52 files, no offenses.
  • 1,000-entry synthetic corpus, Ruby 3.4.12, against current master (with Reduce allocations in BibTeX.parse #199):
    • lexer: 32–34 ms on master, 27–28 ms on this branch, with identical allocations (120,529);
    • BibTeX.parse: 70–72 ms on master, 63–64 ms on this branch, with 404,021 objects instead of 406,021.
  • Adversarial input ('@' + 'a@' * 50_000 + ' {' and similar): 0.07–0.09 s. These inputs are now linear-time regression tests.

Known limits and follow-ups (not in this PR)

  • The atomic path is path-based. A target swapped between the checks and the rename gets replaced; this only matters in directories writable by untrusted users. The file's owner becomes the caller, and only permission bits are kept, not ACLs or xattrs.
  • On Windows, the rename replaces the target, but atomicity is not guaranteed.
  • A name starting with @ ends before the next @, so @a@b = x is not a name. BibTeX would accept it.
  • The linear-time tests use a 5 s wall-clock bound. A ratio check (time at n against time at 4n) would be more robust.
  • The non-strict multi-word field-name pattern from tag name with spaces break parser #131 is still quadratic on adversarial input, as it was before this change. That needs a separate issue.
  • Entry.new deletes keys from the attribute hash passed to it (lib/bibtex/entry.rb:72), so it modifies the caller's hash. This is an existing bug, like the :quotes one fixed in save_to; test_construct passes a copy.
  • META_CONTENT in Lexer::BACKTRACE_BOUNDARIES appears to be unused: error recovery always reaches AT first.

Copilot AI balanced review requested due to automatic review settings November 14, 2026 21:59

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

Frank Fischer declined consent to the license change proposed in #154
and asked for all of his changes to be rewritten. This removes what
remains of his 2011 commits ahead of an independent reimplementation:

- bfaa188: the dynamic field writer branch of Entry#method_missing and
  the matching clause in Entry#respond_to_missing?
- 8eb3979: '/' in the lexer's name, spaced_name and key patterns
  (including the key patterns, which inherited it)
- 7778f42: the body of Bibliography#save_to, which now raises
  NotImplementedError

The branch is intentionally red until the reimplementation lands:
24 minitest errors from field writers (e.g. entry.address = ...),
2 from save_to, and the cucumber scenario
features/issues/slash_keys.feature:6 (DBLP:journals/dq/RossinK99).
Record the lexer's token stream for every fixture and for each section
of errors.bib, under six lexer configurations, before the identifier
character set is widened (R2). Also pin the parsed results: entry
counts, keys, field names, @string names, error traces, and the
ParseError raised for errors.bib as a whole.

Refs #154
Cover entry.foo = value adding or replacing a field with the same value
conversion as Entry#add, respond_to? returning exactly true for Symbol
and String writer names, "=" alone not being a writer, and the existing
readers, convert_/from_ ghost methods, parent delegation, real writers
and operators staying as they are.

Refs #154
Exercise every example character from REQUIREMENTS.md in field names,
@string names and macro references, in strict and non-strict mode, with
and without :strip_comments. Cover % handling, @ inside a name, names
starting with a digit, multi-word field names in non-strict mode, and
keys containing / with :strip_comments and allow_missing_keys.

Add a cucumber feature with the parsing examples and a step to parse
with lexer options.

Refs #154
Cover writing to_s with { } as the default :quotes, honouring options,
UTF-8 output regardless of the default external encoding, round-trips
through BibTeX.open, overwriting, String and Pathname paths, returning
self, and leaving an existing file intact when serialisation raises.

Refs #154
Serialise the bibliography with #to_s before touching the file system,
so an exception raised while writing out an element propagates and
leaves an existing file untouched and does not create a missing one.
Values are enclosed in braces unless the caller passes a non-nil
:quotes; all other options go to #to_s unchanged, and the caller's
hash is not modified. The content is always UTF-8, independent of
Encoding.default_external, so it reads back through BibTeX.open.

A symlink is resolved first. A regular file that may be written, or a
missing one, in a writable directory is replaced atomically: the
content goes to a temporary file with a short fixed name in the same
directory, which is flushed, fsynced (skipped where fsync is not
implemented) and renamed over the target, and removed again if
anything fails, so a full disk or a crash never leaves a partial file.
The file keeps its permission bits without setuid, setgid and sticky;
a new one gets 0666 minus the umask. Everything else (a read-only file
or directory, a device, FIFO or socket, a dangling symlink) is written
directly, so the OS behaves and fails as for File.write. Both paths
write in text mode. Hard links and directory fsync are out of scope.

Refs #154
Entry#method_missing now treats any name of the form <field>= as a
writer for that field and stores the value through #add, so it gets the
same Value conversion as #add, #[]= and the generated writers of the
required and name fields. The writer pattern requires a field name that
does not end in '=', so :'=' (and names made only of '=') stay unknown
methods and raise NoMethodError. A wrong number of arguments raises
ArgumentError, as a real writer would.

The writer and conversion patterns are private constants shared by
method_missing and respond_to_missing?, which now uses Regexp#match? and
so answers true or false rather than a match position. Field readers
and delegation to a cross-referenced parent are unchanged.

Refs #154
Field names, @string names and macro references now accept what
bibtex.web's id_class allows: any printable ASCII character except
white space and " # % ' ( ) , = { }. The project's extensions stay:
non-ASCII letters and digits, and '%' unless :strip_comments makes it
start a comment.

The illegal characters are listed once; a helper derives the character
class with or without '%' (escaped, as it is spliced into a class), and
the :name and :spaced_name patterns (and their _without_percent
variants) are built from it. A name still may not start with a digit.

'@' may start a name only when it cannot start an object. Identifiers
cannot contain { ( = , } #, so the character after '@word' decides: a
name if it is = , } or # (after white space, and TeX comments when they
are stripped), an object start otherwise. This keeps error recovery
after an unterminated object, which relies on '@' starting the next
one, in strict and non-strict mode. The words of a non-strict
multi-word name use the same rule, so only the last one can start with
'@', and as before a word may start with a digit.

A word that starts with '@' ends before the next '@', so @A@b = x is
not a name (a@b and a@b@c still are). Otherwise the lookahead would be
retried from every '@' of a long run like @A@a@a... and rescan the
rest of it each time, which made lexing quadratic.

Keys accept '/' in addition to the characters accepted before; their
character list is also kept in one place and shared by the four key
patterns. All Lexer.patterns keys keep their names, and the recorded
token streams of the fixtures are unchanged.

Refs #154
The clean-room commits listed their changes relative to the stripped
snapshot, so some bullets described existing behaviour as new (field
writers, '/' in keys) or claimed a respond_to? change that users never
saw: Ruby already returned true or false. Keep only what changes for
users since 6.2.0.

Refs #154
The clean-room tests and one lexer comment referred to the requirements
document (R1-R3), which is not part of the gem, and to an issue number.
Like the rest of the suite, comments and test names now describe what
is tested; the issues stay in the commit messages and History.txt.

Refs #154
@tmaier
tmaier force-pushed the rewrite-154-fischer branch from dbbd73c to b9bf376 Compare November 14, 2026 22:08
On Windows save_to writes in text mode, so the file has CRLF line
endings, as before. Comparing the raw bytes with to_s failed there;
reading the file back as UTF-8 text checks the encoding on every
platform, and the bytes are still checked for UTF-8 umlauts.

Refs #154
A mutation test of the lexer showed which regressions the suite misses.
These tests catch them directly, without a recorded token stream:

- test_error_recovery.rb takes over the errors.bib checks section by
  section, in strict and non-strict mode, and adds an unterminated
  braced value and an indented entry after free text in non-strict mode
- test_parser.rb pins how meta content between objects is kept, a TeX
  comment before the key with :strip_comments, and the rejection of
  invalid characters in entry types and of a line break or tab before
  the key's comma
- test_bibtex.rb gets the decoret and bibdesk checks it was missing, and
  test_roundtrip and test_construct are real tests again; the dead
  test_errors stub is gone

Refs #154
test/fixtures/lexer_tokens.json recorded the full token stream of every
fixture under six lexer settings. Two thirds of the streams were copies;
it could not be regenerated; String#inspect made it depend on the locale
for non-ASCII text; and the fixtures lack the input that the new
identifier rules affect, so it guarded less than it seemed.

Of 14 lexer mutations, the targeted tests from the previous commit
catch every one the token pins caught, and five that nothing caught
before.

Refs #154
@tmaier
tmaier merged commit 6bbaf30 into master Nov 14, 2026
11 checks passed
@tmaier
tmaier deleted the rewrite-154-fischer branch November 14, 2026 23:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants