Repository navigation
Conversation
Pattern entries (lemma or pos containing `*` or `[...]`) were mixed with single word entries where the same characters are literal, so tools such as PyMUSAS could not tell which to read as patterns. Move them into a new `autotag` data type, with its own `*_autotag.tsv` file per language. - Add `autotag` to language_resources.json, test_collection.py, txt_to_tsv.py, lexicon_statistics.py, create_pos_tagsets.py and duplicate_entires.py. Autotag files need a pos pattern that is not `*` and valid `[...]` classes (backslash escapes supported). - Move pattern entries out of the English and Danish single lexicons. - Sort generated_pos_tagset.tsv rows alphabetically by POS tag. - Document the autotag format and the pattern syntax in the README and CONTRIBUTING.md; note that spaces_to_tabs.py does not support autotag. Closes UCREL#36
POS tags, lemmas and MWE templates in some lexicons contain stray
whitespace (e.g. `Noun ` in Finnish, non-breaking spaces between
tokens in Russian MWE templates) or malformed MWE tokens (e.g. `_*`).
These entries cannot match as intended, and PyMUSAS assumes MWE
template tokens are separated by exactly one space.
- Add `find_whitespace_problems` and `find_mwe_template_problems` to
test_collection.py, and reject entries that fail them in `check_file`:
`lemma`, `token` and `pos` values cannot start or end with whitespace,
and `mwe_template` tokens must be `{...}` or `<word>_<POS>` with a
non-empty word and POS, separated by single spaces.
- Add find_formatting_issues.py, which lists every entry that fails
these checks with its .txt line number and, where the fix is
mechanical, a suggested fix. Its output is in
formatting_issues_report.md (185 entries across Danish, Finnish,
Russian, Swedish, Urdu and Welsh).
- Document the rules in the README and CONTRIBUTING.md.
The existing lexicon data is not fixed in this commit, so CI fails
until the entries in formatting_issues_report.md are resolved.
Refs UCREL#41
Apply the fixes found by find_formatting_issues.py so that every lexicon passes the whitespace and MWE template checks. - Add fix_formatting_issues.py, which applies the mechanical fixes for a language (--language) or file (--file), with --skip-line and --dry-run. It skips fixes that would create a duplicate entry, keeps comments and line endings, regenerates the TSV, and reports whether the file passes the CI check. Move the shared fix logic into find_formatting_issues.py. - Remove stray whitespace from 173 entries: Finnish (50 lemma, 4 pos), Swedish (29 lemma), Urdu (13 lemma) and Russian (77 MWE templates, mostly non-breaking spaces between tokens). - Move the 12 MWE entries that need a language expert to review files that are not in language_resources.json: Danish/review_mwe_dk.txt (1), Russian/review_mwe_rus.txt (10) and Welsh/review_mwe_cy.txt (1). - Regenerate the TSV files, generated_pos_tagset.tsv files (Finnish `noun ` and `adverb ` are merged into `noun` and `adverb`) and lexicon_statistics.md. - Document fix_formatting_issues.py in the README and Changelog, and stop referring to a saved formatting_issues_report.md. Urdu `ضرورت` (NN1) and `ضروری` (VV0) now each appear twice with different semantic tags and need merging by an Urdu speaker. Closes UCREL#41
Move the 446 entries in Portuguese/semantic_lexicon_pt.txt whose lemma contains an asterisk (*), e.g. `**** Papa` or `****afiado`, into the new file Portuguese/review_semantic_lexicon_pt.txt so that a language expert can review them later. The review file keeps the lexicon header and the original CRLF line endings, and it is not listed in language_resources.json, so it is not converted to TSV or checked by CI. The Portuguese TSV lexicon and lexicon_statistics.md have been regenerated, and the Portuguese entry count drops from 13,942 to 13,496. Refs UCREL#5
Move Portuguese lexicon entries with asterisk tokens into a review file
…S tagset whereby the UPOS tagset was generated using CLAUDE which used the POS mappers we have in this repository, the comments made by CLAUDE about the conversion are in the MD file
…nisation and the addition of common weights and measurement entries
…d Claude autotag lexicons Refs UCREL#42
…d Claude autotag lexicons Refs UCREL#42
A literal [ or ] can only occur within a character class and has to be escaped (\[ and \]), following how PyMUSAS parses autotag lexicons. test_collection.py now raises a ValueError for an unescaped [ within a character class, e.g. [a[], and for a ] that does not close a class, e.g. a]b. Both were previously accepted. Documented the escaping rule in the README pattern syntax table and the test file format checks, and in the CONTRIBUTING autotag entry guidance.
Member
Author
|
This will also close #5 see apmoore1#2, close #41 see apmoore1#1, and has a potential solution for #42 The reason those pull requests from apmoore1 are mentioned as the solution as they are part of this pull request through merging earlier on in this pull request. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pattern entries (lemma or pos containing
*or[...]) were mixed with single word entries where the same characters are literal, so tools such as PyMUSAS could not tell which to read as patterns. Move them into a newautotagdata type, with its own*_autotag.tsvfile per language.autotagto language_resources.json, test_collection.py, txt_to_tsv.py, lexicon_statistics.py, create_pos_tagsets.py and duplicate_entires.py. Autotag files need a pos pattern that is not*and valid[...]classes (backslash escapes supported).Closes #36