overwrite00 · Nov 11, 2026 · Nov 11, 2026
diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md
@@ -25,7 +25,7 @@ It supports two privacy export modes:
 
 | 📄 File | ✨ Responsibility |
 | ------ | ----------------- |
-| `NullifyPDF.py` | Main PySide6 UI, PDF rendering, AI scan orchestration, redaction, export, reconstruction, OCR integration |
+| `NullifyPDF.py` | Main PySide6 UI, PDF rendering, AI scan orchestration (including the Italian-model consent step), redaction, export, reconstruction, OCR integration, Info window |
 | `update_check.py` | Qt-free update logic: version comparison (SemVer, stable/beta channels), GitHub release lookup, per-OS asset selection, trusted-host download, startup-check preference (`~/.nullifypdf/update_check.json`). Only runs on user click or if the user enabled the startup check |
 | `pii_detection.py` | Qt-free AI detection helpers: word-box text index (neighbour-aware rectangle clamping, ligatures, letter-spaced and hyphenated words), offset-to-rect mapping and partial-match policy, per-type thresholds, pattern validation, evidence-based PERSON/LOCATION scoring, accepted/suggested states, allowlist by word identity, whole-word propagation, entity selection persistence |
 | `privacy_core.py` | Privacy mode enum, placeholder registry (random / sequential / generic / keyed styles), encrypted restore-map payloads (version 3, optional per-occurrence text style; scrypt, PBKDF2 maps still readable), password strength check, restore-map parsing |
@@ -34,7 +34,9 @@ It supports two privacy export modes:
 | `export_engine.py` | Qt-free export pipeline (`export_document`), neutral file names, private temp handling, audit record, local-data wipe |
 | `review_dialog.py` | Qt dialog to accept or reject suggested detections |
 | `model_manager.py` | Pinned Hugging Face model catalog, SHA-256 verification, local model store and hardware recommendation |
-| `model_dialog.py` | Italian model selection, consent and progress-aware downloads |
+| `model_dialog.py` | Local ONNX model dialog: catalog, details, consent and a threaded download whose progress fills the download button (second click cancels; list and Close are locked meanwhile) |
+| `spacy_models.py` | The spaCy Italian pipeline (CC BY-NC-SA 3.0) downloaded on demand: consent text, pinned size and SHA-256, safe unpacking into `~/.nullifypdf/spacy_models/`, blank-pipeline fallback |
+| `licences_info.py` | Licence of NullifyPDF and of every component and model with official links, used by the Info window and the README table |
 | `onnx_ner.py` | CPU ONNX token-classification adapter with character-offset mapping and label mapping |
 | `word_import.py` | Safe DOCX OOXML text/table extraction into an internal PyMuPDF document; no macro execution |
 | `help_guide.py` | In-app guide (Italian, F1) and the tooltip table applied to every widget |
@@ -45,7 +47,7 @@ It supports two privacy export modes:
 | `PDF_Checker.py` | Heuristic post-export inspection helper |
 | `build_local.py` | PyInstaller build script |
 | `scripts/download_ocr_data.py` | Downloads EN/IT Tesseract `tessdata_fast` files bundled into every build |
-| `tests/` | Unit and smoke tests for validation, privacy primitives, OCR config, and build configuration |
+| `tests/` | Unit, integration and offscreen GUI tests (see [Tests](#-tests)) |
 
 ---
 
@@ -107,8 +109,8 @@ Export pipeline writes a cleaned PDF copy
 2. The language is always Italian (`IT`); the user optionally enables:
    - `OCR PDF scansionati`
    - `Oscura Immagini`
-3. `cmd_auto_ai()` checks OCR configuration if OCR is enabled.
-4. `AIWorker.run_scan()` processes pages in a background thread.
+3. `cmd_auto_ai()` calls `_prepare_italian_model()`: if the spaCy Italian pipeline is neither installed as a package nor in `~/.nullifypdf/spacy_models/`, a dialog shows its licence and asks for consent (download, continue with rules only, or cancel); then it checks OCR configuration if OCR is enabled.
+4. `AIWorker.run_scan()` processes pages in a background thread; Presidio loads only the Italian pipeline (`AIWorker.italian_model`), and installed local ONNX models are loaded once per scan and add suggestions only.
 5. For each page:
    - the analysis text is built by `pii_detection.build_page_index()` from `page.get_text("words")`, recording each word's rectangle so character offsets map back to exact boxes. Each rectangle is trimmed against the words of neighbouring lines (a word box spans ascender to descender and would otherwise overlap them); ligatures are expanded, letter-spaced runs become one word and a word hyphenated across two lines is joined
    - if the page looks scanned, OCR is run with `page.get_textpage_ocr()` and the same index is built from the OCR words; on text pages with embedded images only the images are OCR'd and merged
@@ -229,7 +231,18 @@ Runtime behavior:
 4. OCR text is used for Presidio detection.
 5. OCR search rectangles are used to place redactions on scanned pages.
 
-Every release bundles EN/IT OCR data.
+Every release bundles EN/IT OCR data. The documents analysed are Italian, so `ita` is the OCR language; `eng` is added only to read English words inside Italian pages.
+
+---
+
+## 🧠 Models
+
+| Model | Where it comes from | Role |
+| ----- | ------------------- | ---- |
+| spaCy `it_core_news_md` 3.8.0 | Downloaded on first use after consent (`spacy_models.py`); never bundled, because its licence (CC BY-NC-SA 3.0) is non-commercial | Statistical recognition of names and places through Presidio |
+| ONNX models (MiniLMv2 EN/IT, PiiScan, OpenMed Italian FastClinical) | Optional, downloaded from pinned Hugging Face revisions (`model_manager.py`), every file checked against size and SHA-256 | Extra suggestions only; never applied automatically |
+
+The development environment installs the Italian spaCy package so the tests can run; the PyInstaller specs exclude it, together with `pymupdf4llm` and `pymupdf.layout`.
 
 ---
 
@@ -277,6 +290,8 @@ Beta releases (`v*-beta.N` tags) and stable releases (`vX.Y.Z` tags) are each co
 
 **Linux packages** are built by the reusable workflow `build-linux.yml`, shared by the stable, beta and test builds. The PyInstaller binary and the `.rpm` are built once inside a `rockylinux:9` container (glibc 2.34, rpm 4.16), the `.deb` is wrapped from the same binary on `ubuntu-24.04`, and each package is then installed and started under Xvfb in clean Ubuntu 24.04, Ubuntu 26.04, Debian 13, Rocky 9, Rocky 10 and Fedora containers. A release is published only if this smoke test passes. Runners are pinned, so the move of `ubuntu-latest` to a new Ubuntu release does not change the build. When a distribution approaches end of support, update the matrix in `build-linux.yml` (and the list in the README).
 
+Pull requests also run the unit tests against the latest PyMuPDF, a guard that only `develop` may be merged into `main`, project automation, and GitHub code scanning (CodeQL on pull requests to `main`).
+
 Expected artifact names include:
 
 ```text
@@ -292,14 +307,13 @@ NullifyPDF_vX.Y.Z_Fedora.rpm
 
 Current tests cover:
 
-- list persistence
-- resource path resolution
-- OCR configuration helpers
-- privacy placeholder registry
-- encrypted restore-map round trip
-- reconstruction of a pseudonymized PDF from its restore map
-- build configuration
-- OCR data inclusion behavior
+- detection pipeline, geometry (rectangle clamping, glyph-deletion rule), scoring and the Italian lexicon (`test_pii_detection.py`, `test_detection_pipeline.py`, `test_geometry.py`)
+- redaction engine, export sanitiser and export verification (`test_redaction_engine.py`, `test_export_sanitize.py`, `test_privacy_hardening.py`)
+- placeholders, encrypted restore maps and reconstruction (`test_privacy_core.py`, `test_reconstruct.py`)
+- encrypted dictionaries and the key store (`test_secure_store.py`)
+- offscreen GUI flows: review, export, reconstruction, guide and tooltips, Info window, model dialog (`test_gui_flow.py`, `test_gui_help.py`, `test_model_dialog.py`)
+- model catalog, on-demand Italian model download, DOCX import, update check (`test_model_manager.py`, `test_spacy_models.py`, `test_word_import.py`, `test_update_check.py`)
+- licence consistency (no non-commercial component in the installer) and build configuration (`test_licences.py`, `test_build_config.py`, `test_validation.py`)
 
 Run:
 
@@ -313,6 +327,7 @@ pytest tests/ -v
 
 - Detection is probabilistic. Presidio, spaCy, and OCR can miss data or create false positives. Heuristics that reject label-like single-word names trade a little recall (for example a lone first name that starts a line) for precision; such values can be marked manually.
 - OCR depends on Tesseract language data and scan quality.
+- Only Italian documents are supported. Without the Italian spaCy model (download declined) only rules and dictionaries run and names are found less often.
 - Handwriting is not reliably supported.
 - Complex PDFs may contain structures not covered by automated cleanup.
 - Pseudonymized PDFs are still personal-data processing when the restore map exists.

diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -24,7 +24,7 @@ git remote add upstream https://github.com/overwrite00/NullifyPDF.git
 python setup_env.py
 ```
 
-This creates isolated venv, installs dependencies, and downloads NLP models.
+This creates an isolated venv, installs dependencies and installs the Italian spaCy model used by the tests (the released program downloads it on demand).
 
 Then enable the repo's git hooks (one-time per clone) so the `Last updated` footer in docs stays in sync automatically:
 
@@ -219,7 +219,7 @@ thread when scanning large PDFs concurrently."
 ### Before Opening PR
 
 1. ✅ Run tests: `pytest tests/ -v`
-2. ✅ Type check (blocking in CI): `mypy NullifyPDF.py pii_detection.py privacy_core.py update_check.py PDF_Checker.py`
+2. ✅ Type check (blocking in CI): `mypy NullifyPDF.py pii_detection.py privacy_core.py update_check.py PDF_Checker.py export_engine.py pdf_sanitize.py review_dialog.py lexicon_it.py secure_store.py word_lists.py dictionary_dialog.py help_guide.py spacy_models.py licences_info.py`
 3. ✅ Build locally: `python build_local.py`
 4. ✅ Update CHANGELOG.md with your changes
 5. ✅ Self-review your code (would you understand this in 6 months?)

diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md
@@ -105,32 +105,44 @@ If the GUI opens → **You're ready to develop!** 🎉
 ```
 NullifyPDF/
 |-- NullifyPDF.py              # Main PySide6 application
-|-- model_manager.py           # Pinned local model catalog and integrity checks
-|-- model_dialog.py             # Model consent and download UI
-|-- onnx_ner.py                 # ONNX token-classification adapter
-|-- word_import.py              # Safe DOCX text/table importer
-|-- privacy_core.py            # Privacy modes and encrypted restore maps
+|-- pii_detection.py           # Qt-free detection: word index, geometry, scoring, partial-match policy
+|-- lexicon_it.py              # Italian word lists; lexicon_data_it.py is generated (ISTAT, Wikidata)
+|-- privacy_core.py            # Privacy modes, placeholders and encrypted restore maps
+|-- export_engine.py           # Qt-free export pipeline, audit record, local-data wipe
+|-- pdf_sanitize.py            # Export sanitiser and output verification
+|-- review_dialog.py           # Review of suggested detections
+|-- secure_store.py            # Master key protection (DPAPI / keyring / key file), AES-GCM sealing
+|-- word_lists.py              # Encrypted blocklist/allowlist with expiry
+|-- dictionary_dialog.py       # Dictionary editor
+|-- help_guide.py              # In-app guide (Italian) and tooltips
+|-- licences_info.py           # Licences and links shown in the Info window and README
+|-- spacy_models.py            # Italian spaCy pipeline downloaded on demand after consent
+|-- model_manager.py           # Pinned local ONNX model catalog and integrity checks
+|-- model_dialog.py            # Local model dialog with in-button download progress
+|-- onnx_ner.py                # ONNX token-classification adapter
+|-- word_import.py             # Safe DOCX text/table importer
 |-- update_check.py            # Qt-free update check: versions, GitHub releases, asset choice, download
-|-- pii_detection.py           # Qt-free AI detection helpers (offsets, filters, whole-word matching)
 |-- PDF_Checker.py             # Heuristic verification utility
 |-- setup_env.py               # Environment setup script
 |-- build_local.py             # PyInstaller build script
 |-- requirements.txt           # Python dependencies
+|-- LICENSE, THIRD_PARTY_NOTICES.md
 |-- scripts/
-|   `-- download_ocr_data.py   # EN/IT OCR data downloader
-|-- tests/
-|   |-- test_validation.py
-|   |-- test_privacy_core.py
-|   |-- test_pii_detection.py
-|   |-- test_reconstruct.py
-|   `-- test_build_config.py
+|   |-- download_ocr_data.py   # EN/IT OCR data downloader
+|   |-- build_lexicon_data.py  # Generates lexicon_data_it.py (development only)
+|   |-- extract_release_notes.py
+|   `-- update_docs_dates.py
+|-- tests/                     # pytest suite (unit, integration and offscreen GUI tests)
 |-- images/
 |   `-- NullifyPDF.png
 `-- .github/
     `-- workflows/
-        |-- test_build.yml
-        |-- beta-release.yml
-        `-- release.yml
+        |-- test_build.yml     # unit tests and full build matrix
+        |-- beta-release.yml   # v*-beta.N tags
+        |-- release.yml        # vX.Y.Z tags
+        |-- build-linux.yml    # reusable: .deb/.rpm build and smoke tests
+        |-- guard-main-source.yml
+        `-- project-automation.yml
 ```
 
 ---
@@ -145,25 +157,32 @@ NullifyPDF/
 | **PyMuPDF**           | 2.1     | PDF manipulation and OCR bridge |
 | **presidio-analyzer** | 2.2.366 | PII detection                |
 | **spaCy**             | 3.8.18  | NLP for entity recognition   |
-| **cryptography**      | 50.0.2  | Encrypted restore maps       |
+| **cryptography**      | 50.0.2  | Encrypted restore maps and dictionaries |
+| **keyring**           | 25.7.0  | OS key store (macOS and Linux only) |
 | **onnxruntime**       | 1.31.1  | CPU inference for optional local PII models |
 | **tokenizers**        | 0.23.3  | Fast local tokenizer with character offsets |
 | **python-docx**       | —       | Not required; DOCX import uses the standard-library OOXML reader |
 | **pytest**            | 9.2.0   | Testing framework            |
+| **mypy**              | 1.20.2  | Type checking (blocking in CI) |
+| **PyInstaller**       | 6.22.3  | Builds the executables       |
 
 ### Language Models
 
 | Model             | Size  | Purpose     | Licence | In the installer |
 | ----------------- | ----- | ----------- | ------- | ---------------- |
 | `it_core_news_md` | 45 MB | Italian NER | CC BY-NC-SA 3.0 | **no**, downloaded by the app on demand (`spacy_models.py`) |
 
-`setup_env.py` installs both in the development environment (the Italian tests
+`setup_env.py` installs it in the development environment (the Italian tests
 need it). The PyInstaller specs exclude `it_core_news_md`, `pymupdf4llm` and
 `pymupdf.layout`. At run time the app asks for consent, downloads the Italian
 pipeline from the publisher's release, checks its size and SHA-256 and stores it
 under `~/.nullifypdf/spacy_models/`. If the user declines, a blank Italian
 pipeline is used and only rule-based detection is available.
 
+The optional ONNX models (MiniLMv2 EN/IT, PiiScan, OpenMed Italian FastClinical)
+are described in `model_manager.py`; they are downloaded by the user from pinned
+Hugging Face revisions into `~/.nullifypdf/models/` and add suggestions only.
+
 ---
 
 ## 🔍 Understanding the Code
@@ -242,20 +261,23 @@ Opens `htmlcov/index.html` in browser.
 CI runs `mypy` as a blocking step. Run the same check locally before opening a PR:
 
 ```bash
-mypy NullifyPDF.py pii_detection.py privacy_core.py update_check.py PDF_Checker.py
+mypy NullifyPDF.py pii_detection.py privacy_core.py update_check.py PDF_Checker.py export_engine.py pdf_sanitize.py review_dialog.py lexicon_it.py secure_store.py word_lists.py dictionary_dialog.py help_guide.py spacy_models.py licences_info.py
 ```
 
 PyMuPDF is imported as `pymupdf` (aliased to `fitz`) so it is really type-checked; its dynamically defined constants are read once near the top of `NullifyPDF.py`.
 
 ### What's Tested
 
-- ✅ **PDFListManager** — File I/O, persistence
-- ✅ **OCR Config** — Tesseract language selection and tessdata discovery
-- ✅ **Privacy Core** — Placeholder mapping and encrypted restore maps (v1/v2/v3)
-- ✅ **AI Detection Helpers** — Offset-to-rectangle mapping, thresholds, label/heading filtering, whole-word matching
-- ✅ **Reconstruction** — Placeholder restoration, original font/size/colour on native PDFs
-- ✅ **Build Config** — OCR data bundling behavior
-- ✅ **Resource Paths** — PyInstaller compatibility
+- ✅ **Detection** — word index, rectangle clamping and the glyph-deletion rule, scoring, partial-match policy, Italian lexicon
+- ✅ **Redaction and export** — redaction engine, sanitiser, export verification (survivor and leak checks)
+- ✅ **Privacy Core** — placeholders and encrypted restore maps (v1 to v4 formats, Argon2id envelope, recovery key), reconstruction with the original font
+- ✅ **Local data** — encrypted dictionaries with expiry and the secure key store
+- ✅ **GUI flows** — offscreen tests of review, export, reconstruction, guide and tooltips, Info window, model dialog and Italian-model consent
+- ✅ **Models and import** — model catalog and integrity checks, on-demand Italian model download, DOCX import, update check
+- ✅ **Licences and build** — licence consistency, OCR data bundling, PyInstaller configuration, resource paths
+
+> [!TIP]
+> GUI tests run offscreen (`QT_QPA_PLATFORM=offscreen`) and never touch the OS keyring (`NULLIFYPDF_DISABLE_KEYRING=1`).
 
 ---
 
@@ -469,6 +491,10 @@ isort NullifyPDF.py
 | `pdf_sanitize.py`  | Export sanitiser and output verification |
 | `export_engine.py` | Qt-free export pipeline, audit record, local-data hygiene |
 | `review_dialog.py` | Review dialog for suggested detections |
+| `spacy_models.py`  | Italian spaCy pipeline downloaded on demand (consent, SHA-256, fallback) |
+| `licences_info.py` | Licences and official links shown in the Info window and README (a test keeps README in step) |
+| `model_manager.py` / `model_dialog.py` / `onnx_ner.py` | Optional ONNX models: catalog, verified download, adapter |
+| `word_import.py`   | DOCX import |
 | `secure_store.py`  | Master key protection (DPAPI / keyring / key file), AES-GCM sealing |
 | `word_lists.py`    | Encrypted blocklist/allowlist with expiry |
 | `dictionary_dialog.py` | Dictionary editor dialog |
@@ -595,6 +621,13 @@ Ready to contribute? See [CONTRIBUTING.md](./CONTRIBUTING.md) for:
 - 💬 **GitHub Discussions:** [Discussions](https://github.com/overwrite00/NullifyPDF/discussions)
 - 🐛 **GitHub Issues:** [Issues](https://github.com/overwrite00/NullifyPDF/issues)
 
+### Release procedure
+
+- **Beta**: work lands on `develop` through squash-merged pull requests; an annotated tag `vX.Y.Z-beta.N` on `develop` starts `beta-release.yml`. Set `__version_prerelease__` in `NullifyPDF.py` to the beta suffix.
+- **Stable**: set `__version_prerelease__` to `""`, move the `[Unreleased]` changelog entries under `## [X.Y.Z] - date` (leave `[Unreleased]` empty), merge `develop` into `main` with a merge commit through a pull request (one approving review is required), then tag `vX.Y.Z` on `main`. `release.yml` checks the changelog and the version before building.
+- Documentation-only changes do not need a release.
+- When dependencies change, update `THIRD_PARTY_NOTICES.md` and `licences_info.py`; `tests/test_licences.py` fails if a non-commercial component would be bundled.
+
 ### Release notes and the update check
 
 The release workflows fill each GitHub release body with `scripts/extract_release_notes.py`: the `## [X.Y.Z]` section for stable tags, `## [Unreleased]` for beta tags. The in-app update check displays that body, so keep the changelog sections accurate before tagging.