Skip to content

Add autotag lexicon data type for pattern entries - #40

Open
apmoore1 wants to merge 14 commits into
UCREL:masterfrom
apmoore1:wildcard-lexicon-feature
Open

apmoore1 wants to merge 14 commits into
UCREL:masterfrom
apmoore1:wildcard-lexicon-feature

Conversation

@apmoore1

Copy link
Copy Markdown
Member

Pattern entries (lemma or pos containing * or [...]) were mixed with single word entries where the same characters are literal, so tools such as PyMUSAS could not tell which to read as patterns. Move them into a new autotag data type, with its own *_autotag.tsv file per language.

  • Add autotag to language_resources.json, test_collection.py, txt_to_tsv.py, lexicon_statistics.py, create_pos_tagsets.py and duplicate_entires.py. Autotag files need a pos pattern that is not * and valid [...] classes (backslash escapes supported).
  • Move pattern entries out of the English and Danish single lexicons.
  • Sort generated_pos_tagset.tsv rows alphabetically by POS tag.
  • Document the autotag format and the pattern syntax in the README and CONTRIBUTING.md; note that spaces_to_tabs.py does not support autotag.

Closes #36

Pattern entries (lemma or pos containing `*` or `[...]`) were mixed with
single word entries where the same characters are literal, so tools such
as PyMUSAS could not tell which to read as patterns. Move them into a new
`autotag` data type, with its own `*_autotag.tsv` file per language.

- Add `autotag` to language_resources.json, test_collection.py,
  txt_to_tsv.py, lexicon_statistics.py, create_pos_tagsets.py and
  duplicate_entires.py. Autotag files need a pos pattern that is not `*`
  and valid `[...]` classes (backslash escapes supported).
- Move pattern entries out of the English and Danish single lexicons.
- Sort generated_pos_tagset.tsv rows alphabetically by POS tag.
- Document the autotag format and the pattern syntax in the README and
  CONTRIBUTING.md; note that spaces_to_tabs.py does not support autotag.

Closes UCREL#36
@apmoore1 apmoore1 self-assigned this Oct 19, 2026
POS tags, lemmas and MWE templates in some lexicons contain stray
whitespace (e.g. `Noun ` in Finnish, non-breaking spaces between
tokens in Russian MWE templates) or malformed MWE tokens (e.g. `_*`).
These entries cannot match as intended, and PyMUSAS assumes MWE
template tokens are separated by exactly one space.

- Add `find_whitespace_problems` and `find_mwe_template_problems` to
  test_collection.py, and reject entries that fail them in `check_file`:
  `lemma`, `token` and `pos` values cannot start or end with whitespace,
  and `mwe_template` tokens must be `{...}` or `<word>_<POS>` with a
  non-empty word and POS, separated by single spaces.
- Add find_formatting_issues.py, which lists every entry that fails
  these checks with its .txt line number and, where the fix is
  mechanical, a suggested fix. Its output is in
  formatting_issues_report.md (185 entries across Danish, Finnish,
  Russian, Swedish, Urdu and Welsh).
- Document the rules in the README and CONTRIBUTING.md.

The existing lexicon data is not fixed in this commit, so CI fails
until the entries in formatting_issues_report.md are resolved.

Refs UCREL#41
Apply the fixes found by find_formatting_issues.py so that every lexicon
passes the whitespace and MWE template checks.

- Add fix_formatting_issues.py, which applies the mechanical fixes for a
  language (--language) or file (--file), with --skip-line and --dry-run.
  It skips fixes that would create a duplicate entry, keeps comments and
  line endings, regenerates the TSV, and reports whether the file passes
  the CI check. Move the shared fix logic into find_formatting_issues.py.
- Remove stray whitespace from 173 entries: Finnish (50 lemma, 4 pos),
  Swedish (29 lemma), Urdu (13 lemma) and Russian (77 MWE templates,
  mostly non-breaking spaces between tokens).
- Move the 12 MWE entries that need a language expert to review files
  that are not in language_resources.json: Danish/review_mwe_dk.txt (1),
  Russian/review_mwe_rus.txt (10) and Welsh/review_mwe_cy.txt (1).
- Regenerate the TSV files, generated_pos_tagset.tsv files (Finnish
  `noun ` and `adverb ` are merged into `noun` and `adverb`) and
  lexicon_statistics.md.
- Document fix_formatting_issues.py in the README and Changelog, and
  stop referring to a saved formatting_issues_report.md.

Urdu `ضرورت` (NN1) and `ضروری` (VV0) now each appear twice with
different semantic tags and need merging by an Urdu speaker.

Closes UCREL#41
Move the 446 entries in Portuguese/semantic_lexicon_pt.txt whose lemma
contains an asterisk (*), e.g. `**** Papa` or `****afiado`, into the new
file Portuguese/review_semantic_lexicon_pt.txt so that a language expert
can review them later. The review file keeps the lexicon header and the
original CRLF line endings, and it is not listed in
language_resources.json, so it is not converted to TSV or checked by CI.

The Portuguese TSV lexicon and lexicon_statistics.md have been
regenerated, and the Portuguese entry count drops from 13,942 to 13,496.

Refs UCREL#5
Move Portuguese lexicon entries with asterisk tokens into a review file
…S tagset whereby the UPOS tagset was generated using CLAUDE which used the POS mappers we have in this repository, the comments made by CLAUDE about the conversion are in the MD file
…nisation and the addition of common weights and measurement entries
A literal [ or ] can only occur within a character class and has to be
escaped (\[ and \]), following how PyMUSAS parses autotag lexicons.
test_collection.py now raises a ValueError for an unescaped [ within a
character class, e.g. [a[], and for a ] that does not close a class,
e.g. a]b. Both were previously accepted.

Documented the escaping rule in the README pattern syntax table and the
test file format checks, and in the CONTRIBUTING autotag entry guidance.
@apmoore1

apmoore1 commented Nov 9, 2026 •

Copy link
Copy Markdown
Member Author

This will also close #5 see apmoore1#2, close #41 see apmoore1#1, and has a potential solution for #42

The reason those pull requests from apmoore1 are mentioned as the solution as they are part of this pull request through merging earlier on in this pull request.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Separate wildcard (*) single word lexicon entries into a new autotag data type

1 participant