Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 8 additions & 2 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,13 +11,19 @@ The steps to take when wanting to contribute a lexicon resource:
1. When constructing the resource it must be written following the format of the resource you want to construct:
1. For single word lexicon files, follow the [single word lexicon file format described in the README.](./README.md#single-word-lexicon-file-format)
2. For Multi Word Expression (MWE) lexicon files, follow the [MWE lexicon file format described in the README,](./README.md#multi-word-expression-mwe-lexicon-file-format)
3. For autotag lexicon files, follow the [autotag lexicon file format described in the README.](./README.md#autotag-lexicon-file-format) Autotag files use the single word lexicon file format, but their `lemma` and `pos` values are patterns, which support `*` (zero or more of any character), `[...]` (a character class that matches exactly one character, e.g. `[0-9]`), and `[...]*` (zero or more characters from that class). Any single word entry whose `lemma` or `pos` is a pattern, e.g. `*-year-old`, should go in an autotag file named `semantic_lexicon_LANG_autotag.tsv`, whereby `LANG` is the same language suffix as the language's single word lexicon file, rather than in the single word lexicon file, as in a single word lexicon file `*`, `[` and `]` in the `lemma` are always literal. When writing autotag entries note that:
* `[...]*` also matches **zero** characters, e.g. `[0-9]*ish` matches `9ish` and `10ish`, but also `ish`. If at least one character from the class is required, add a non-optional copy of the class before it, e.g. `[0-9][0-9]*ish`.
* A literal `[` or `]` can only be written within a character class and has to be escaped with a `\`, e.g. `[\[]` matches `[` and `[\]]` matches `]`. An unescaped `[` within a character class, or a `]` that does not close a class, is invalid.
* Every autotag entry needs a POS pattern, a `pos` value of just `*` is **not** allowed, however `*` alongside other characters is, e.g. `N*`.
* Entries are matched in file order and only the first matching entry is used, so put more specific patterns before more general ones, see the [autotag lexicon file format section of the README](./README.md#autotag-lexicon-file-format).
4. For all lexicon files, the `lemma`, `token`, and `pos` values cannot start or end with whitespace, e.g. `Noun ` is **not** allowed. For MWE lexicon files, each `{token}_{pos}` (or curly brace, e.g. `{N*/P*/R*}`) part of the `mwe_template` has to be separated by a single space (not a non-breaking space or tab), the `mwe_template` cannot start or end with whitespace, and neither the token nor the POS can be empty, e.g. `_*` is **not** allowed.

In both cases these files are described as TSV files, **however** these files at this stage should be **text files**. They should follow TSV files with respect to having header names and tabs separating the columns. The main differences is that with these files you can also include comments using a `#` symbol at the start of each comment line **but** the comment line cannot include a tab character.
In all cases these files are described as TSV files, **however** these files at this stage should be **text files**. They should follow TSV files with respect to having header names and tabs separating the columns. The main differences is that with these files you can also include comments using a `#` symbol at the start of each comment line **but** the comment line cannot include a tab character.

2. Once you have constructed your text file version of the lexicon file, **fork** this GitHub repository so that you can make a **pull request** in step 7, and then clone your fork to your computer.
3. Set up the development environment so that you can run the Python scripts in this repository, either locally or through a dev container, as described in the [Development Environment section of the README](./README.md#development-environment).
4. Within your forked version of this repository, save your constructed text version of the lexicon file to the relevant language name folder, if your language name folder does not exist create one.
5. Add your resource to the meta data file, [./language_resources.json](./language_resources.json), of which the [USAS Lexicon Meta Data section of the README](./README.md#usas-lexicon-meta-data) should be a good guide on how to add your resource to the meta data file. **NOTE** that the file path to your lexicon file should have the `.tsv` file extension rather than the expected `.txt`, as in the next step you will create a `TSV` file from your text file version of the lexicon.
5. Add your resource to the meta data file, [./language_resources.json](./language_resources.json), of which the [USAS Lexicon Meta Data section of the README](./README.md#usas-lexicon-meta-data) should be a good guide on how to add your resource to the meta data file, e.g. an autotag lexicon file is added with `"data type": "autotag"`. **NOTE** that the file path to your lexicon file should have the `.tsv` file extension rather than the expected `.txt`, as in the next step you will create a `TSV` file from your text file version of the lexicon.
6. From the root of the repository, run the following commands:

``` bash
Expand Down
4 changes: 3 additions & 1 deletion Changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,4 +16,6 @@ MWE = Multi Word Expression
12. Added [CONTRIBUTING guidelines](./CONTRIBUTING.md) for contributing a lexicon resource.
13. Added [GitHub action](./.github/workflows/ci.yml) which will convert the lexicon resources created in text file format, following [CONTRIBUTING guidelines](./CONTRIBUTING.md), to TSV format. After conversion it will check that TSV files are formatted correctly, and if so it will add and commit the TSV files into the repository.
14. Added a [dev container](./.devcontainer/devcontainer.json) to allow developers to easily setup a working environment. The Dev container only contains Python 3.12.
15. Spanish lexicons have been updated. (11th of November 2025).
15. Spanish lexicons have been updated. (11th of November 2025).
16. Added the `autotag` lexicon data type, which uses the single word lexicon file format but whereby the `lemma` and `pos` values are patterns (`*`, `[...]`, and `[...]*`). The pattern entries in the [English](./English/semantic_lexicon_en.txt) and [Danish](./Danish/semantic_lexicon_dk.txt) single word lexicons, e.g. `*-year-old`, have been moved into [English/semantic_lexicon_en_autotag.txt](./English/semantic_lexicon_en_autotag.txt) and [Danish/semantic_lexicon_dk_autotag.txt](./Danish/semantic_lexicon_dk_autotag.txt), so that `*`, `[` and `]` in single word lexicon files are always literal. The `generated_pos_tagset.tsv` files are now sorted alphabetically by POS tag. See [issue #36](https://github.com/UCREL/Multilingual-USAS/issues/36).
17. Added checks to [test_collection.py](./test_collection.py) so that `lemma`, `token`, and `pos` values cannot start or end with whitespace, and `mwe_template` values have to be `{token}_{pos}` (or `{...}`) tokens separated by a single space. Added the [find_formatting_issues.py script](./find_formatting_issues.py), which outputs a Markdown report of all the lexicon entries that fail these checks, and the [fix_formatting_issues.py script](./fix_formatting_issues.py), which applies the mechanical fixes from that report. See [issue #41](https://github.com/UCREL/Multilingual-USAS/issues/41).
24 changes: 12 additions & 12 deletions Chinese/generated_pos_tagset.tsv
Original file line number Diff line number Diff line change
@@ -1,20 +1,20 @@
POS Count
noun 47450
num 900
pron 356
pnoun 8904
adj 9494
adv 5971
verb 31412
conj 428
det 411
fw 15
idiom 10
intj 272
loc 371
conj 428
mark 192
msr 763
noun 47450
num 900
ono 1
part 3778
adj 9494
pnoun 8904
prep 1614
intj 272
mark 192
idiom 10
fw 15
ono 1
pron 356
punc 144
verb 31412
14 changes: 7 additions & 7 deletions Czech/generated_pos_tagset.tsv
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
POS Count
j 71
v 6050
r 66
p 54
d 1774
a 6670
c 58
d 1774
i 61
j 71
n 13057
a 6670
p 54
r 66
t 67
i 61
v 6050
x 233
24 changes: 12 additions & 12 deletions Danish/generated_pos_tagset.tsv
Original file line number Diff line number Diff line change
@@ -1,20 +1,20 @@
POS Count
cconj 488
noun 37686
part 455
num 430
* 1753
adj 12208
adp 3893
propn 11218
sconj 182
adv 4536
aux 106
pron 1114
cconj 488
conj 75
det 1964
intj 234
noun 37685
num 430
part 455
pron 1114
propn 11216
punct 14
sconj 182
sym 73
verb 12855
x 174
adj 12208
sym 73
* 1754
conj 75
punct 14
1 change: 0 additions & 1 deletion Danish/mwe-dk.tsv
Original file line number Diff line number Diff line change
Expand Up @@ -16390,7 +16390,6 @@ Bronzealderen_NOUN analys*_NOUN Q3
Eurotunnelen_NOUN book*_NOUN Q4.1
Kanaløerne_PROPN messaging_NOUN Q1.3
Fællesmarkedet_NOUN mining_NOUN Y2/Q3
Sankt_PROPN *_* _PROPN kors_NOUN Y2/Q3
Uafhængighedserklæringen_NOUN conservator*_NOUN C1/S2mf
Deep_PROPN South_PROPN C1
Østindien_PROPN designer*_NOUN C1/S2mf
Expand Down
1 change: 0 additions & 1 deletion Danish/mwe-dk.txt
Original file line number Diff line number Diff line change
Expand Up @@ -16390,7 +16390,6 @@ Bronzealderen_NOUN analys*_NOUN Q3
Eurotunnelen_NOUN book*_NOUN Q4.1
Kanaløerne_PROPN messaging_NOUN Q1.3
Fællesmarkedet_NOUN mining_NOUN Y2/Q3
Sankt_PROPN *_* _PROPN kors_NOUN Y2/Q3
Uafhængighedserklæringen_NOUN conservator*_NOUN C1/S2mf
Deep_PROPN South_PROPN C1
Østindien_PROPN designer*_NOUN C1/S2mf
Expand Down
2 changes: 2 additions & 0 deletions Danish/review_mwe_dk.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
mwe_template semantic_tags
Sankt_PROPN *_* _PROPN kors_NOUN Y2/Q3
31 changes: 0 additions & 31 deletions Danish/semantic_lexicon_dk.tsv
Original file line number Diff line number Diff line change
Expand Up @@ -28,37 +28,6 @@ puha INTJ Z4
'un NUM N1
har VERB A9+ A2.2 Z5
* X Z5
*% NUM N5
*'C NOUN O4.6
*-hektar NOUN N3.6
*-seng ADJ N3.2/H4
*-medarbejder ADJ N3.2 N5
*-fod ADJ N3.7
*-fuld ADJ N3.4
*-time ADJ T1.3
*-tommer NOUN N3.7
*-måler NOUN N3.7
*-mil NOUN N3.3
*-minut ADJ T1.3
*-måned ADJ T1.3
*-måned NOUN T1.3
*-ulige ADJ N5 T3
*-side ADJ Q1.2
*-plus ADJ N5 T3
*-punkt ADJ N3.1
*-noget NOUN N5 T3
*-etagers ADJ H1/N3.7
*-stærk ADJ N3.2
*-ugentlig ADJ N6+
*-år ADJ T1.3 T3
*-årige NOUN T3
*hektar NOUN N3.6
*cms NOUN N3.7
*ha NOUN N3.6
*KBps NOUN N3.8/Y2 N3.8/Q1.3
*kph NOUN N3.8
*MB NOUN N3.2/Y2
*MW NOUN N3/O3
+ ADP Z5
-baseret ADJ M6 A2.2
-opført ADJ I2.1
Expand Down
31 changes: 0 additions & 31 deletions Danish/semantic_lexicon_dk.txt
Original file line number Diff line number Diff line change
Expand Up @@ -28,37 +28,6 @@ puha INTJ Z4
'un NUM N1
har VERB A9+ A2.2 Z5
* X Z5
*% NUM N5
*'C NOUN O4.6
*-hektar NOUN N3.6
*-seng ADJ N3.2/H4
*-medarbejder ADJ N3.2 N5
*-fod ADJ N3.7
*-fuld ADJ N3.4
*-time ADJ T1.3
*-tommer NOUN N3.7
*-måler NOUN N3.7
*-mil NOUN N3.3
*-minut ADJ T1.3
*-måned ADJ T1.3
*-måned NOUN T1.3
*-ulige ADJ N5 T3
*-side ADJ Q1.2
*-plus ADJ N5 T3
*-punkt ADJ N3.1
*-noget NOUN N5 T3
*-etagers ADJ H1/N3.7
*-stærk ADJ N3.2
*-ugentlig ADJ N6+
*-år ADJ T1.3 T3
*-årige NOUN T3
*hektar NOUN N3.6
*cms NOUN N3.7
*ha NOUN N3.6
*KBps NOUN N3.8/Y2 N3.8/Q1.3
*kph NOUN N3.8
*MB NOUN N3.2/Y2
*MW NOUN N3/O3
+ ADP Z5
-baseret ADJ M6 A2.2
-opført ADJ I2.1
Expand Down
32 changes: 32 additions & 0 deletions Danish/semantic_lexicon_dk_autotag.tsv
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
lemma pos semantic_tags
*% NUM N5
*'C NOUN O4.6
*-hektar NOUN N3.6
*-seng ADJ N3.2/H4
*-medarbejder ADJ N3.2 N5
*-fod ADJ N3.7
*-fuld ADJ N3.4
*-time ADJ T1.3
*-tommer NOUN N3.7
*-måler NOUN N3.7
*-mil NOUN N3.3
*-minut ADJ T1.3
*-måned ADJ T1.3
*-måned NOUN T1.3
*-ulige ADJ N5 T3
*-side ADJ Q1.2
*-plus ADJ N5 T3
*-punkt ADJ N3.1
*-noget NOUN N5 T3
*-etagers ADJ H1/N3.7
*-stærk ADJ N3.2
*-ugentlig ADJ N6+
*-år ADJ T1.3 T3
*-årige NOUN T3
*hektar NOUN N3.6
*cms NOUN N3.7
*ha NOUN N3.6
*KBps NOUN N3.8/Y2 N3.8/Q1.3
*kph NOUN N3.8
*MB NOUN N3.2/Y2
*MW NOUN N3/O3
32 changes: 32 additions & 0 deletions Danish/semantic_lexicon_dk_autotag.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
lemma pos semantic_tags
*% NUM N5
*'C NOUN O4.6
*-hektar NOUN N3.6
*-seng ADJ N3.2/H4
*-medarbejder ADJ N3.2 N5
*-fod ADJ N3.7
*-fuld ADJ N3.4
*-time ADJ T1.3
*-tommer NOUN N3.7
*-måler NOUN N3.7
*-mil NOUN N3.3
*-minut ADJ T1.3
*-måned ADJ T1.3
*-måned NOUN T1.3
*-ulige ADJ N5 T3
*-side ADJ Q1.2
*-plus ADJ N5 T3
*-punkt ADJ N3.1
*-noget NOUN N5 T3
*-etagers ADJ H1/N3.7
*-stærk ADJ N3.2
*-ugentlig ADJ N6+
*-år ADJ T1.3 T3
*-årige NOUN T3
*hektar NOUN N3.6
*cms NOUN N3.7
*ha NOUN N3.6
*KBps NOUN N3.8/Y2 N3.8/Q1.3
*kph NOUN N3.8
*MB NOUN N3.2/Y2
*MW NOUN N3/O3
12 changes: 6 additions & 6 deletions Dutch/generated_pos_tagset.tsv
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
POS Count
adj 769
noun 2061
verb 1000
adv 208
art 2
conj 33
interj 21
prep 33
adv 208
intj 1
noun 2061
num 39
prep 33
pron 53
intj 1
art 2
verb 1000
22 changes: 11 additions & 11 deletions English/generated_pos_tagset.tsv
Original file line number Diff line number Diff line change
@@ -1,19 +1,19 @@
POS Count
* 8303
adj 14581
adp 3597
adv 5089
aux 103
cconj 452
det 3027
intj 267
noun 42478
part 455
num 621
part 455
pron 605
propn 11202
adp 3597
punct 24
sconj 200
adv 5089
aux 103
pron 605
det 3027
sym 111
verb 11601
x 226
adj 14581
sym 111
intj 267
* 8303
punct 24
36 changes: 0 additions & 36 deletions English/semantic_lexicon_en.tsv
Original file line number Diff line number Diff line change
Expand Up @@ -29,42 +29,6 @@ lemma pos semantic_tags
'un NUM N1
've VERB A9+ A2.2 Z5
* X Z5
*% NUM N5
*'C NOUN O4.6
*-a-year NOUN N5 I1
*-acre NOUN N3.6
*-bed ADJ N3.2/H4
*-employee ADJ N3.2 N5
*-foot ADJ N3.7
*-full ADJ N3.4
*-hour ADJ T1.3
*-inch NOUN N3.7
*-metre NOUN N3.7
*-mile NOUN N3.3
*-minute ADJ T1.3
*-month ADJ T1.3
*-month NOUN T1.3
*-odd ADJ N5 T3
*-page ADJ Q1.2
*-plus ADJ N5 T3
*-point ADJ N3.1
*-something NOUN N5 T3
*-somethings NOUN N5 T3
*-storey ADJ H1/N3.7
*-strong ADJ N3.2
*-weekly ADJ N6+
*-year ADJ T1.3 T3
*-year-old ADJ T3
*-year-old NOUN T3
*-year-olds NOUN T3
*KBps NOUN N3.8/Y2 N3.8/Q1.3
*MB NOUN N3.2/Y2
*MW NOUN N3/O3
*acre NOUN N3.6
*cms NOUN N3.7
*ha NOUN N3.6
*hectare NOUN N3.6
*kph NOUN N3.8
+ ADP Z5
-Reuter PROPN Z3c Z1mf
-based ADJ M6 A2.2
Expand Down
Loading
Loading