IceBERT PoS Tagging Interface

A high-level Python interface for PoS tagging Icelandic text using the IceBERT-PoS model with classical tokenization.

TODOs

• Proper device handling (GPU) for tensors

Installation

# This package is currently not available on PyPI, so you need to install it directly from the source repository.

# Without PyTorch (lighter, but model inference won't work)
# Allows you to control the PyTorch version
pip install git+ssh://git@github.com/mideind/IceBERT-PoS.git@main  # Installs directly from main branch

# With PyTorch support (required for model inference) - RECOMMENDED
pip install "git+ssh://git@github.com/mideind/IceBERT-PoS.git[torch]"  # @main implied

Note: The [torch] extra is required for model inference. Then why package it separately? To avoid pinning to specific versions of PyTorch and allow the user to install the latest version compatible with their system.

Versioning

Package versioning is done via Git tags. To install a specific version, use the @ syntax:

pip install "git+ssh://git@github.com/mideind/IceBERT-PoS.git@<version>"

Version History

• v0.4.0: Token text is now the UNEDITED source slice (text[char_start:char_end] always holds) — previously the classical tokenizer's normalized form leaked in, fabricating characters (e.g. '27.' for '27' inside date composites, shifting the following offsets) or dropping them (soft hyphens). Composites are split on the raw slice's whitespace. Token/TaggedToken gain kind (tokenizer token kind) and parent (the underlying tokenizer Tok when 1:1, None for split parts; runtime-only) plus as_dict() for serialization. Breaking: the new dataclass fields are required, and JSON/CLI output now includes kind.
• v0.3.0: Loosen transformers version requirement from >=4.46.3,<5.0 to >=4.46.3,<6.0. In other words, add support for transformers version 5.
• v0.2.0: First stable release

Features

• Classical Tokenization: Uses the Miðeind tokenizer for Icelandic tokenziation
• Character Positions: Preserves exact character start/end positions in original text
• Sentence-Aware Processing: Maintains sentence boundaries and processes them in batches
• Dual Format Output: Provides both IFD tags and structured category/features
• Caller-owned Model: Load model once, reuse for multiple calls
• Batch Processing: Efficient processing of multiple sentences

Usage

Command Line Interface

After installation, you can use the icebert-pos command:

# Basic POS tagging with full IFD tags
icebert-pos "Þetta er stutt sýnidæmi."
# Þetta[fahen] er[sfg3en] stutt[lhensf] sýnidæmi[nhen].[pl]

# Get only POS categories (without detailed features)
icebert-pos --only-category "Þetta er stutt sýnidæmi."
# Þetta[fa] er[sf] stutt[l] sýnidæmi[n].[pl]

# Get structured json output
icebert-pos --json "Þetta er stutt sýnidæmi."
# [
#   [
#     {
#       "text": "Þetta",
#       "char_start": 0,
#       "char_end": 5,
#       "kind": "WORD",
#       "category": "fa",
#       "features": [
#         "neut",
#         "sing",
#         "nom"
#       ],
#       "ifd_tag": "fahen"
#     },
#     ...
#     {
#       "text": ".",
#       "char_start": 23,
#       "char_end": 24,
#       "kind": "PUNCTUATION",
#       "category": "pl",
#       "features": [],
#       "ifd_tag": "pl"
#     }
#   ]
# ]

# Default behavior is to split composite tokens (like "samskipta- og kynningarstýra") into individual tokens
icebert-pos "samskipta- og kynningarstýra"
# 3 tokens:
# samskipta-[kt] og[c] kynningarstýra[nven]
icebert-pos --keep-composite-tokens "samskipta- og kynningarstýra"
# 1 token:
# samskipta- og kynningarstýra[nven]

# Enable debug logging
icebert-pos --debug "Þetta er stutt sýnidæmi."
# lots of output

There are some additional command line options available, run icebert-pos --help to see them.

Python API

Simple Usage

from icebert_pos import pos_tag_text, TaggedToken
from transformers import AutoModel, AutoTokenizer
import torch

# Load model and tokenizer, you need to have trust_remote_code=True to load the custom model code.
# You can check the model repository for details: https://huggingface.co/mideind/IceBERT-PoS
model = AutoModel.from_pretrained("mideind/IceBERT-PoS", trust_remote_code=True)
# set the model to evaluation mode - otherwise the output will be stochastic
model.eval()
# place the model on the appropriate device (CPU/GPU)
model.to("cuda" if torch.cuda.is_available() else "cpu")
tokenizer = AutoTokenizer.from_pretrained("mideind/IceBERT-PoS")

text = "Þetta er stutt sýnidæmi."
# POS tag text - returns List[List[TaggedToken]]
sentence_results = pos_tag_text(text, model, tokenizer)
# Each token's `text` is the exact source slice (text[char_start:char_end]); `kind` is the
# classical tokenizer's token kind. `as_dict()` gives the serializable fields — it excludes
# `parent`, the runtime-only handle to the underlying tokenizer token (None for tokens that
# were split out of a composite; `parent.txt` holds the tokenizer-normalized form).
assert [token.as_dict() for token in sentence_results[0]] == [
    {"text": "Þetta", "char_start": 0, "char_end": 5, "kind": "WORD", "category": "fa", "features": ["neut", "sing", "nom"], "ifd_tag": "fahen"},
    {"text": "er", "char_start": 6, "char_end": 8, "kind": "WORD", "category": "sf", "features": ["sing", "act", "3", "pres"], "ifd_tag": "sfg3en"},
    {"text": "stutt", "char_start": 9, "char_end": 14, "kind": "WORD", "category": "l", "features": ["neut", "sing", "nom", "strong", "pos"], "ifd_tag": "lhensf"},
    {"text": "sýnidæmi", "char_start": 15, "char_end": 23, "kind": "WORD", "category": "n", "features": ["neut", "sing", "nom"], "ifd_tag": "nhen"},
    {"text": ".", "char_start": 23, "char_end": 24, "kind": "PUNCTUATION", "category": "pl", "features": [], "ifd_tag": "pl"},
]

Batch Processing for Efficiency

# For processing multiple sentences efficiently
# The Miðeind tokenizer will split this string into 3 sentences and process them in batches
texts = ["Fyrsti texti.", "Annar texti.", "Þriðji texti."]
# The batching is done automatically by the pos_tag_text function and this will call model.forward twice
sentence_results = pos_tag_text("\n".join(texts), model, tokenizer, batch_size=2)
assert len(sentence_results) == 3  # Should return 3 sentences

Advanced Usage with Lower-Level Functions

from icebert_pos import (
    segment_text_to_sentences,
    prepare_sentence,
    batch_sentences,
    predict_sentences
)
# Same example as before
text = "Þetta er stutt sýnidæmi."
# Segment text into sentences
sentences = segment_text_to_sentences(text)

# Prepare individual sentences
sentence_tensors = []
for sentence in sentences:
    tensors = prepare_sentence(sentence, model, tokenizer, truncate=True)
    sentence_tensors.append(tensors)

# Batch multiple sentences for efficient processing
batch_input_ids, batch_attention_mask, batch_word_mask = batch_sentences(
    sentence_tensors, tokenizer
)

# Get raw predictions
predictions = predict_sentences(
    batch_input_ids, batch_attention_mask, batch_word_mask, model
)

# predictions is List[List[Tuple[str, List[str]]]]
# - List of sentences
# - Each sentence has List of (category, features) tuples for each word
assert predictions == [
    [
        ("fa", ["neut", "sing", "nom"]),
        ("sf", ["sing", "act", "3", "pres"]),
        ("l", ["neut", "sing", "nom", "strong", "pos"]),
        ("n", ["neut", "sing", "nom"]),
        ("pl", [])
    ]
]

Data Structures

Token

Basic token with text and position:

• text: The UNEDITED source slice — always exactly source_text[char_start:char_end]. The classical tokenizer's normalizations (e.g. the ordinal period it fabricates inside date composites, or soft-hyphen stripping) never leak into it.
• char_start: Start position in original text
• char_end: End position in original text
• kind: The classical tokenizer's token kind (TOK.descr, e.g. "WORD", "DATEABS", "AMOUNT"). Tokens split out of a whitespace composite inherit the composite's kind.
• parent: The underlying tokenizer Tok when the token maps 1:1 to it; None when the token was split out of a composite (the composite Tok describes the whole phrase, not the part). parent.txt is the tokenizer-NORMALIZED form — the right form for dictionary lookups. Runtime-only: serialize via as_dict(), which excludes it.

Sentence

Collection of tokens representing a sentence:

• tokens: List of Token objects

TaggedToken

Token with POS tagging information (extends Token, so it also carries kind and parent):

• text: The token text (exact source slice, as above)
• char_start: Start position in original text
• char_end: End position in original text
• category: POS category (e.g., "fp", "sfg")
• features: List of morphological features (e.g., ["1", "sing", "nom"])
• ifd_tag: Full IFD POS tag (e.g., "fp1en", "sfg3en")

API Reference

High-Level Functions

• pos_tag_text(text, model, tokenizer, batch_size=1, split_composite_tokens=True, truncate=False) - Main function for POS tagging
• segment_text_to_sentences(text, split_composite_tokens=True) - Segment text into sentences using classical tokenization

Parameters

• batch_size: Number of sentences to process in each batch for efficiency (default: 1)
• split_composite_tokens: Whether to split composite tokens (like "samskipta- og kynningarstýra") into individual tokens on whitespace (default: True)
• truncate: Whether to truncate input sequences that exceed the model's maximum length. If False, long sentences may cause errors (default: False)

Lower-Level Functions

• prepare_sentence(sentence, model, tokenizer, truncate=False) - Prepare tensors for a single sentence
• batch_sentences(sentence_tensors, tokenizer) - Batch multiple sentence tensors
• predict_sentences(input_ids, attention_mask, word_mask, model) - Get raw predictions from model

When using the lower-level functions you can control more of the processing but will also need to handle device placement and batching manually.

License

MIT

Copyright (C) Miðeind ehf.