You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit beef340
Browse filesBrowse the repository at this point in the historyBrowse files
Copy file name to clipboardExpand all lines: ARCHITECTURE.md
+29-14Lines changed: 29 additions & 14 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -25,7 +25,7 @@ It supports two privacy export modes:
25
25
26
26
| 📄 File | ✨ Responsibility |
27
27
| ------ | ----------------- |
28
-
|`NullifyPDF.py`| Main PySide6 UI, PDF rendering, AI scan orchestration, redaction, export, reconstruction, OCR integration |
28
+
|`NullifyPDF.py`| Main PySide6 UI, PDF rendering, AI scan orchestration (including the Italian-model consent step), redaction, export, reconstruction, OCR integration, Info window|
29
29
|`update_check.py`| Qt-free update logic: version comparison (SemVer, stable/beta channels), GitHub release lookup, per-OS asset selection, trusted-host download, startup-check preference (`~/.nullifypdf/update_check.json`). Only runs on user click or if the user enabled the startup check |
30
30
|`pii_detection.py`| Qt-free AI detection helpers: word-box text index (neighbour-aware rectangle clamping, ligatures, letter-spaced and hyphenated words), offset-to-rect mapping and partial-match policy, per-type thresholds, pattern validation, evidence-based PERSON/LOCATION scoring, accepted/suggested states, allowlist by word identity, whole-word propagation, entity selection persistence |
|`review_dialog.py`| Qt dialog to accept or reject suggested detections |
36
36
|`model_manager.py`| Pinned Hugging Face model catalog, SHA-256 verification, local model store and hardware recommendation |
37
-
|`model_dialog.py`| Italian model selection, consent and progress-aware downloads |
37
+
|`model_dialog.py`| Local ONNX model dialog: catalog, details, consent and a threaded download whose progress fills the download button (second click cancels; list and Close are locked meanwhile) |
38
+
|`spacy_models.py`| The spaCy Italian pipeline (CC BY-NC-SA 3.0) downloaded on demand: consent text, pinned size and SHA-256, safe unpacking into `~/.nullifypdf/spacy_models/`, blank-pipeline fallback |
39
+
|`licences_info.py`| Licence of NullifyPDF and of every component and model with official links, used by the Info window and the README table |
38
40
|`onnx_ner.py`| CPU ONNX token-classification adapter with character-offset mapping and label mapping |
39
41
|`word_import.py`| Safe DOCX OOXML text/table extraction into an internal PyMuPDF document; no macro execution |
40
42
|`help_guide.py`| In-app guide (Italian, F1) and the tooltip table applied to every widget |
@@ -45,7 +47,7 @@ It supports two privacy export modes:
|`scripts/download_ocr_data.py`| Downloads EN/IT Tesseract `tessdata_fast` files bundled into every build |
48
-
|`tests/`| Unit and smoke tests for validation, privacy primitives, OCR config, and build configuration|
50
+
|`tests/`| Unit, integration and offscreen GUI tests (see [Tests](#-tests))|
49
51
50
52
---
51
53
@@ -107,8 +109,8 @@ Export pipeline writes a cleaned PDF copy
107
109
2. The language is always Italian (`IT`); the user optionally enables:
108
110
-`OCR PDF scansionati`
109
111
-`Oscura Immagini`
110
-
3.`cmd_auto_ai()` checks OCR configuration if OCR is enabled.
111
-
4.`AIWorker.run_scan()` processes pages in a background thread.
112
+
3.`cmd_auto_ai()`calls `_prepare_italian_model()`: if the spaCy Italian pipeline is neither installed as a package nor in `~/.nullifypdf/spacy_models/`, a dialog shows its licence and asks for consent (download, continue with rules only, or cancel); then it checks OCR configuration if OCR is enabled.
113
+
4.`AIWorker.run_scan()` processes pages in a background thread; Presidio loads only the Italian pipeline (`AIWorker.italian_model`), and installed local ONNX models are loaded once per scan and add suggestions only.
112
114
5. For each page:
113
115
- the analysis text is built by `pii_detection.build_page_index()` from `page.get_text("words")`, recording each word's rectangle so character offsets map back to exact boxes. Each rectangle is trimmed against the words of neighbouring lines (a word box spans ascender to descender and would otherwise overlap them); ligatures are expanded, letter-spaced runs become one word and a word hyphenated across two lines is joined
114
116
- if the page looks scanned, OCR is run with `page.get_textpage_ocr()` and the same index is built from the OCR words; on text pages with embedded images only the images are OCR'd and merged
@@ -229,7 +231,18 @@ Runtime behavior:
229
231
4. OCR text is used for Presidio detection.
230
232
5. OCR search rectangles are used to place redactions on scanned pages.
231
233
232
-
Every release bundles EN/IT OCR data.
234
+
Every release bundles EN/IT OCR data. The documents analysed are Italian, so `ita` is the OCR language; `eng` is added only to read English words inside Italian pages.
235
+
236
+
---
237
+
238
+
## 🧠 Models
239
+
240
+
| Model | Where it comes from | Role |
241
+
| ----- | ------------------- | ---- |
242
+
| spaCy `it_core_news_md` 3.8.0 | Downloaded on first use after consent (`spacy_models.py`); never bundled, because its licence (CC BY-NC-SA 3.0) is non-commercial | Statistical recognition of names and places through Presidio |
243
+
| ONNX models (MiniLMv2 EN/IT, PiiScan, OpenMed Italian FastClinical) | Optional, downloaded from pinned Hugging Face revisions (`model_manager.py`), every file checked against size and SHA-256 | Extra suggestions only; never applied automatically |
244
+
245
+
The development environment installs the Italian spaCy package so the tests can run; the PyInstaller specs exclude it, together with `pymupdf4llm` and `pymupdf.layout`.
233
246
234
247
---
235
248
@@ -277,6 +290,8 @@ Beta releases (`v*-beta.N` tags) and stable releases (`vX.Y.Z` tags) are each co
277
290
278
291
**Linux packages** are built by the reusable workflow `build-linux.yml`, shared by the stable, beta and test builds. The PyInstaller binary and the `.rpm` are built once inside a `rockylinux:9` container (glibc 2.34, rpm 4.16), the `.deb` is wrapped from the same binary on `ubuntu-24.04`, and each package is then installed and started under Xvfb in clean Ubuntu 24.04, Ubuntu 26.04, Debian 13, Rocky 9, Rocky 10 and Fedora containers. A release is published only if this smoke test passes. Runners are pinned, so the move of `ubuntu-latest` to a new Ubuntu release does not change the build. When a distribution approaches end of support, update the matrix in `build-linux.yml` (and the list in the README).
279
292
293
+
Pull requests also run the unit tests against the latest PyMuPDF, a guard that only `develop` may be merged into `main`, project automation, and GitHub code scanning (CodeQL on pull requests to `main`).
- placeholders, encrypted restore maps and reconstruction (`test_privacy_core.py`, `test_reconstruct.py`)
313
+
- encrypted dictionaries and the key store (`test_secure_store.py`)
314
+
- offscreen GUI flows: review, export, reconstruction, guide and tooltips, Info window, model dialog (`test_gui_flow.py`, `test_gui_help.py`, `test_model_dialog.py`)
315
+
- model catalog, on-demand Italian model download, DOCX import, update check (`test_model_manager.py`, `test_spacy_models.py`, `test_word_import.py`, `test_update_check.py`)
316
+
- licence consistency (no non-commercial component in the installer) and build configuration (`test_licences.py`, `test_build_config.py`, `test_validation.py`)
303
317
304
318
Run:
305
319
@@ -313,6 +327,7 @@ pytest tests/ -v
313
327
314
328
- Detection is probabilistic. Presidio, spaCy, and OCR can miss data or create false positives. Heuristics that reject label-like single-word names trade a little recall (for example a lone first name that starts a line) for precision; such values can be marked manually.
315
329
- OCR depends on Tesseract language data and scan quality.
330
+
- Only Italian documents are supported. Without the Italian spaCy model (download declined) only rules and dictionaries run and names are found less often.
316
331
- Handwriting is not reliably supported.
317
332
- Complex PDFs may contain structures not covered by automated cleanup.
318
333
- Pseudonymized PDFs are still personal-data processing when the restore map exists.
This creates isolated venv, installs dependencies, and downloads NLP models.
27
+
This creates an isolated venv, installs dependencies and installs the Italian spaCy model used by the tests (the released program downloads it on demand).
28
28
29
29
Then enable the repo's git hooks (one-time per clone) so the `Last updated` footer in docs stays in sync automatically:
30
30
@@ -219,7 +219,7 @@ thread when scanning large PDFs concurrently."
219
219
### Before Opening PR
220
220
221
221
1. ✅ Run tests: `pytest tests/ -v`
222
-
2. ✅ Type check (blocking in CI): `mypy NullifyPDF.py pii_detection.py privacy_core.py update_check.py PDF_Checker.py`
PyMuPDF is imported as `pymupdf` (aliased to `fitz`) so it is really type-checked; its dynamically defined constants are read once near the top of `NullifyPDF.py`.
249
268
250
269
### What's Tested
251
270
252
-
- ✅ **PDFListManager** — File I/O, persistence
253
-
- ✅ **OCR Config** — Tesseract language selection and tessdata discovery
- ✅ **Detection** — word index, rectangle clamping and the glyph-deletion rule, scoring, partial-match policy, Italian lexicon
272
+
- ✅ **Redaction and export** — redaction engine, sanitiser, export verification (survivor and leak checks)
273
+
- ✅ **Privacy Core** — placeholders and encrypted restore maps (v1 to v4 formats, Argon2id envelope, recovery key), reconstruction with the original font
274
+
- ✅ **Local data** — encrypted dictionaries with expiry and the secure key store
275
+
- ✅ **GUI flows** — offscreen tests of review, export, reconstruction, guide and tooltips, Info window, model dialog and Italian-model consent
276
+
- ✅ **Models and import** — model catalog and integrity checks, on-demand Italian model download, DOCX import, update check
277
+
- ✅ **Licences and build** — licence consistency, OCR data bundling, PyInstaller configuration, resource paths
278
+
279
+
> [!TIP]
280
+
> GUI tests run offscreen (`QT_QPA_PLATFORM=offscreen`) and never touch the OS keyring (`NULLIFYPDF_DISABLE_KEYRING=1`).
259
281
260
282
---
261
283
@@ -469,6 +491,10 @@ isort NullifyPDF.py
469
491
|`pdf_sanitize.py`| Export sanitiser and output verification |
-**Beta**: work lands on `develop` through squash-merged pull requests; an annotated tag `vX.Y.Z-beta.N` on `develop` starts `beta-release.yml`. Set `__version_prerelease__` in `NullifyPDF.py` to the beta suffix.
627
+
-**Stable**: set `__version_prerelease__` to `""`, move the `[Unreleased]` changelog entries under `## [X.Y.Z] - date` (leave `[Unreleased]` empty), merge `develop` into `main` with a merge commit through a pull request (one approving review is required), then tag `vX.Y.Z` on `main`. `release.yml` checks the changelog and the version before building.
628
+
- Documentation-only changes do not need a release.
629
+
- When dependencies change, update `THIRD_PARTY_NOTICES.md` and `licences_info.py`; `tests/test_licences.py` fails if a non-commercial component would be bundled.
630
+
598
631
### Release notes and the update check
599
632
600
633
The release workflows fill each GitHub release body with `scripts/extract_release_notes.py`: the `## [X.Y.Z]` section for stable tags, `## [Unreleased]` for beta tags. The in-app update check displays that body, so keep the changelog sections accurate before tagging.
0 commit comments