Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions History.txt
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,10 @@ Unreleased
* Enable frozen_string_literal in the generated parsers and stop mutating strings that Ruby 3.4+ will freeze (Symbol#to_s in Entry#digest, the name parser's word buffer); these emitted deprecation warnings and would raise FrozenError in a future Ruby
* Allocate fewer objects when parsing: the lexer no longer builds a method name per token or copies matched punctuation, and the grammar drops redundant actions
* Fix a memory leak in month parsing: Entry::MONTHS_FILTER cached every parsed month and never released it, so memory use and parse time grew with each parse (#196). The constant is removed; months are normalised by Entry#parse_month. Entries no longer share month Values; padded months like " Jan " are recognised, while "0" and "00" are no longer mistaken for months
* Accept every character BibTeX allows in identifiers in field names, @string names and macro references, e.g. x;y, p[1], a|b, a^b, a~b, a<b>, a@b and @foo; these used to raise an unexpected-token error. An '@' followed by a name and '{' or '(' still starts a new entry, and a name that starts with '@' ends before the next '@'
* Bibliography#save_to (and #save) replace the file atomically, so a failure while serialising or writing leaves the existing file untouched instead of empty or partly written; symlinked targets and permission bits are kept, and devices, pipes and files in read-only directories are written directly
* Bibliography#save_to no longer adds :quotes to the options hash passed in
* Reimplement Entry's dynamic field writers, Bibliography#save_to and the lexer's identifier and key rules from a written specification, without the code they replace (#154)

6.2.0 / 2025-07-20
==================
Expand Down
102 changes: 102 additions & 0 deletions features/identifiers.feature
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
Feature: Identifier characters in field names, @string names and macro references
As a hacker who works with bibliographies
I want to parse every identifier that BibTeX itself accepts
Because exported bibliographies use field names like "a/b" or "p[1]"

Scenario Outline: A field name containing <name> (<mode>)
When I parse the following file with options <options>:
"""
@misc{key, <name> = {value}, title = {Title}}
@misc{other, title = {Other}}
"""
Then my bibliography should contain 2 misc
And the entry with key "key" should have a field "<name>" with the value "value"
And the entry with key "key" should have a field "title" with the value "Title"

Examples:
| name | mode | options |
| a/b | strict | strict: true |
| x;y | strict | strict: true |
| p[1] | strict | strict: true |
| a\|b | strict | strict: true |
| a^b | strict | strict: true |
| a~b | strict | strict: true |
| a<b> | strict | strict: true |
| a@b | strict | strict: true |
| a/b | non-strict | strict: false |
| x;y | non-strict | strict: false |
| p[1] | non-strict | strict: false |
| a\|b | non-strict | strict: false |
| a^b | non-strict | strict: false |
| a~b | non-strict | strict: false |
| a<b> | non-strict | strict: false |
| a@b | non-strict | strict: false |
| a/b | strict with :strip_comments | strip_comments: true |
| a@b | strict with :strip_comments | strip_comments: true |
| p[1] | non-strict with :strip_comments | strict: false, strip_comments: true |
| a<b> | non-strict with :strip_comments | strict: false, strip_comments: true |

Scenario Outline: A @string named <name> and a reference to it (<mode>)
When I parse the following file with options <options>:
"""
@string{ <name> = "Journal of Examples" }
@article{key, journal = <name>, title = {Title}}
"""
And I replace all strings in my bibliography
Then the string "<name>" should be "Journal of Examples"
And the entry with key "key" should have a field "journal" with the value "Journal of Examples"

Examples:
| name | mode | options |
| a/b | strict | strict: true |
| x;y | strict | strict: true |
| p[1] | strict | strict: true |
| a\|b | strict | strict: true |
| a^b | strict | strict: true |
| a~b | strict | strict: true |
| a<b> | strict | strict: true |
| a@b | strict | strict: true |
| a@b | non-strict | strict: false |
| x;y | strict with :strip_comments | strip_comments: true |

Scenario: Percent signs in names depend on :strip_comments
When I parse the following file with options strict: true:
"""
@string{ a/b%c = "with percent" }
@misc{key, x%y = a/b%c}
"""
And I replace all strings in my bibliography
Then the entry with key "key" should have a field "x%y" with the value "with percent"

Scenario: Percent signs start a comment with :strip_comments
When I parse the following file with options strip_comments: true:
"""
@string{ a/b = "no percent" } % a comment
@misc{key,
x;y = a/b % a comment
}
"""
And I replace all strings in my bibliography
Then the entry with key "key" should have a field "x;y" with the value "no percent"

Scenario: Field names made of several words in non-strict mode
When I parse the following file with options strict: false:
"""
@misc{key, Early/Access Date[1] = {NOV}}
"""
Then the entry with key "key" should have a field "early/access_date[1]" with the value "NOV"

Scenario Outline: Keys containing '/' with <options>
When I parse the following file with options <options>:
"""
@article{DBLP:journals/dq/RossinK99, title = {Data Errors}}
"""
Then my bibliography should contain an article with id "DBLP:journals/dq/RossinK99"

Examples:
| options |
| strict: true |
| strict: false |
| strip_comments: true |
| allow_missing_keys: true |
| strip_comments: true, allow_missing_keys: true |
10 changes: 10 additions & 0 deletions features/step_definitions/bibtex_steps.rb
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,16 @@
@bibliography = BibTeX.parse(string)
end

# Options are written as "name: value" pairs separated by commas, e.g.
# "strict: false, strip_comments: true"; values are true or false.
When(/^I parse the following file with options (.+):$/) do |options, string|
options = options.split(/\s*,\s*/).to_h do |pair|
name, value = pair.split(/\s*:\s*/, 2)
[name.to_sym, value == 'true']
end
@bibliography = BibTeX.parse(string, options)
end

When(/^I search for "([^"]*)"$/) do |query|
@result = @bibliography.query(query)
end
Expand Down
84 changes: 78 additions & 6 deletions lib/bibtex/bibliography.rb
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,9 @@
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#++

require 'fileutils'
require 'tempfile'

module BibTeX
#
# The Bibliography class models a BibTeX bibliography;
Expand Down Expand Up @@ -137,14 +140,24 @@ def save(options = {})
save_to(@path, options)
end

# Saves the bibliography to a file at the given path. Returns the bibliography.
# Saves the bibliography to a file at the given path (a String or a
# Pathname), replacing a regular file atomically where it can and
# writing directly otherwise. The options are passed on to #to_s;
# values are enclosed in braces unless :quotes says otherwise.
# Returns the bibliography.
def save_to(path, options = {})
options[:quotes] ||= %w[{ }]

File.open(path, 'w:UTF-8') do |f|
f.write(to_s(options))
# Serialise before touching the file system, so an error in #to_s
# leaves no trace. A nil :quotes means the default, as an absent key
# does. Always UTF-8, whatever Encoding.default_external is, so the
# file reads back through BibTeX.open.
content = to_s(options.merge(quotes: options[:quotes] || %w[{ }])).encode(Encoding::UTF_8)
# Replace the file a symlink points to, not the link.
target = File.exist?(path) ? File.realpath(path) : path
if replace_atomically?(target)
write_atomically(target, content)
else
File.write(target, content, encoding: Encoding::UTF_8)
end

self
end

Expand Down Expand Up @@ -568,6 +581,65 @@ def uniq(*_arguments)

private

# A short, fixed prefix: a name derived from the target's could exceed
# the file system's limit for long target names.
TEMP_NAME = ['.bibtex-save', '.tmp'].freeze
private_constant :TEMP_NAME

# Whether #save_to may replace target atomically: only a regular file
# (or a missing one) that may be written, in a directory where the
# temporary file can be created. Anything else -- a read-only file,
# a read-only directory, a device, FIFO or socket, a dangling symlink --
# is written directly, so the OS behaves and fails as for File.write.
def replace_atomically?(target)
return false unless File.writable?(File.dirname(target))

stat = File.lstat(target)
stat.file? && stat.writable?
rescue Errno::ENOENT
true
end

# Writes content to a temporary file in the target's directory (so the
# rename stays on one file system), syncs it and renames it over the
# target: a failed write never leaves a partial file. Renaming over an
# existing file is atomic on POSIX; on Windows File.rename replaces the
# target but is not guaranteed to be atomic. The directory is not
# fsynced, so the rename itself may not survive a crash; IO#fsync can
# raise NotImplementedError on some platforms, and is skipped there.
#
# The file keeps its permission bits (without setuid, setgid and sticky)
# and a new one gets 0666 minus the umask rather than the temporary
# file's 0600. Hard links are out of scope: the target gets a new inode,
# so other links to the old file keep the old content (as do its owner,
# ACLs and xattrs). Text mode, as for File.write, keeps the platform's
# newline conversion.
def write_atomically(target, content)
mode = File.exist?(target) ? File.stat(target).mode & 0o777 : 0o666 & ~File.umask
temp = Tempfile.create(TEMP_NAME, File.dirname(target), encoding: Encoding::UTF_8)
begin
temp.write(content)
temp.flush
fsync(temp)
temp.chmod(mode)
temp.close
File.rename(temp.path, target)
temp = nil
ensure
if temp
temp.close
FileUtils.rm_f(temp.path)
end
end
end

def fsync(file)
file.fsync
rescue NotImplementedError
# No fsync on this platform; the data has still been flushed.
nil
end

def query_handler(selector)
case selector.to_s
when /first|distinct|detect/i
Expand Down
31 changes: 24 additions & 7 deletions lib/bibtex/entry.rb
Original file line number Diff line number Diff line change
Expand Up @@ -259,13 +259,21 @@ def inherited_fields
names
end

def method_missing(name, *args, &)
# Ghost methods: a writer `<field>=` for any non-empty field name that
# does not itself end in '=' (so :'=', :'==' and :'a==' stay unknown
# methods), and the convert_<filter>, convert_<filter>! and
# from_<filter> conversions.
FIELD_WRITER = /\A(?<field>.*[^=])=\z/m
CONVERSION = /\A(?:convert|from)_(?<filter>[a-z]+)(?<bang>!)?\z/
private_constant :FIELD_WRITER, :CONVERSION

def method_missing(name, *, &)
if fields.key?(name)
fields[name]
elsif name.to_s =~ /^(.+)=$/
send(:add, Regexp.last_match(1).to_sym, args[0])
elsif name =~ /^(?:convert|from)_([a-z]+)(!)?$/
Regexp.last_match(2) ? convert!(Regexp.last_match(1), &) : convert(Regexp.last_match(1), &)
elsif (writer = FIELD_WRITER.match(name))
write_field(writer[:field], *)
elsif (conversion = CONVERSION.match(name))
conversion[:bang] ? convert!(conversion[:filter], &) : convert(conversion[:filter], &)
elsif has_parent? && parent.provides?(name)
parent.provide(name)
else
Expand All @@ -274,8 +282,9 @@ def method_missing(name, *args, &)
end

def respond_to_missing?(method, include_private = false)
provides?(method.to_sym) || method.to_s.match(/=$/) ||
method =~ /^(?:convert|from)_([a-z]+)(!)?$/ ||
provides?(method.to_sym) ||
FIELD_WRITER.match?(method) ||
CONVERSION.match?(method) ||
(has_parent? && parent.respond_to?(method, include_private)) || super
end

Expand Down Expand Up @@ -692,6 +701,14 @@ def normalize_month(month)
end
end

# Backs the ghost writers; checks the arity a real writer would have.
def write_field(name, *values)
raise ::ArgumentError, "wrong number of arguments (given #{values.length}, expected 1)" unless values.length == 1

# Same conversion as #add, like the generated writers above.
add(name, values[0])
end

# Escapes every & not already preceded by a backslash, unless the field
# is verbatim, a BibDesk URL field, or a howpublished containing \url.
#
Expand Down
79 changes: 69 additions & 10 deletions lib/bibtex/lexer.rb
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,67 @@ class Lexer
strip_comments: false
}

# bibtex.web's id_class: an identifier (field, @string or macro name)
# may hold any printable ASCII character except white space and these
# (illegal_id_char).
ILLEGAL_ID_CHARS = %q{"#%'(),={}}

# Characters of a citation key: those accepted before, plus '/' as in
# DBLP keys (DBLP:journals/dq/RossinK99). Braces only with a key that
# may not be empty. '-' comes last as it is spliced into a class.
KEY_CHAR = %q([:alnum:] :_!$?.+;&*'"/-)
private_constant :ILLEGAL_ID_CHARS, :KEY_CHAR

class << self
private

# A character class for identifiers. Non-ASCII letters and digits are
# a project extension, as is '%' unless :strip_comments makes it start
# a comment.
def id_char(percent:, at: true)
illegal = percent ? ILLEGAL_ID_CHARS.delete('%') : ILLEGAL_ID_CHARS
illegal += '@' unless at
# Escaped because the list is spliced into a character class, where
# ']', '\\', '^' or '-' would otherwise change its meaning.
"[[:alnum:]!-~&&[^#{Regexp.escape(illegal)}]]"
end

# One word of a name. id_class allows '@' anywhere, but '@' also
# starts the next object, which error recovery after an unterminated
# object relies on. Identifiers cannot contain = , } # { or (, so the
# character after '@word' decides unambiguously: a name if (after
# white space, and TeX comments when they are stripped) it is one of
# = , } #, an object start otherwise, e.g. at '{' or '('.
#
# A word that starts with '@' ends before the next '@'. Otherwise the
# lookahead would be retried from every '@' of a long run like
# @a@a@a... and rescan the rest of it each time, which is quadratic.
# So @a@b = x is not a name; a@b and a@b@c still are.
def word_pattern(percent:)
gap = percent ? '\s*' : '(?:\s|%[^\n]*)*'
char = id_char(percent:)
"(?:(?!@)#{char}+|@#{id_char(percent:, at: false)}+(?=#{gap}[=,}#]))"
end

# A name must not start with a digit (bibtex.web).
def name_pattern(percent:)
/(?![[:digit:]])#{word_pattern(percent:)}/
end

# Several words followed by '=', e.g. Web of Science's `Early Access
# Date'. The words are joined into one name; as before, a word
# may start with a digit. By the rule above only the last word, which
# is followed by '=', can start with '@'.
def spaced_name_pattern(percent:)
word = word_pattern(percent:)
/#{word}(?:[\t ]+#{word})+(?=\s*=)/
end

def key_pattern(extra, optional:)
/\s*[#{extra}#{KEY_CHAR}]#{optional ? '*' : '+'},/
end
end

# Patterns Cache (#37: MacRuby does not cache regular expressions)
@patterns = {
space: /\s+/o,
Expand All @@ -49,12 +110,10 @@ class Lexer
eq: /\s*=\s*/o,
comma: /\s*,\s*/o,
number: /[[:digit:]]+/o,
name: %r{[[:alpha:][:digit:]/:_!$?.%+&*-]+}io,
# Several space-separated words followed by `=', e.g. the
# `Early Access Date' fields exported by Web of Science.
spaced_name: %r{[[:alpha:][:digit:]/:_!$?.%+&*-]+(?:[\t ]+[[:alpha:][:digit:]/:_!$?.%+&*-]+)+(?=\s*=)}io,
spaced_name_without_percent: %r{[[:alpha:][:digit:]/:_!$?.+&*-]+(?:[\t ]+[[:alpha:][:digit:]/:_!$?.+&*-]+)+(?=\s*=)}io,
name_without_percent: %r{[[:alpha:][:digit:]/:_!$?.+&*-]+}io,
name: name_pattern(percent: true),
name_without_percent: name_pattern(percent: false),
spaced_name: spaced_name_pattern(percent: true),
spaced_name_without_percent: spaced_name_pattern(percent: false),
tex_comment: /%[^\n]*/o,
tex_comments: /(?:\s*%[^\n]*)+/o,
quote: /\s*"/o,
Expand All @@ -68,10 +127,10 @@ class Lexer
string: /string/io,
comment: /comment\b/io,
preamble: /preamble\b/io,
key: %r{\s*[[:alpha:][:digit:] /:_!$?.%+;&*'"{}-]+,}io,
optional_key: %r{\s*[[:alpha:][:digit:] /:_!$?.%+;&*'"-]*,}io,
key_without_percent: %r{\s*[[:alpha:][:digit:] /:_!$?.+;&*'"{}-]+,}io,
optional_key_without_percent: %r{\s*[[:alpha:][:digit:] /:_!$?.+;&*'"-]*,}io
key: key_pattern('%{}', optional: false),
optional_key: key_pattern('%', optional: true),
key_without_percent: key_pattern('{}', optional: false),
optional_key_without_percent: key_pattern('', optional: true)
}

MODE = Hash.new(:meta).merge(
Expand Down
Loading
Loading