Release 2.3.0

.github/workflows/beta-release.yml

@@ -47,7 +47,7 @@ jobs:
 
       - name: Type Check (mypy)
         shell: bash
-        run: mypy NullifyPDF.py --ignore-missing-imports --no-error-summary 2>&1 | head -50 || true
+        run: mypy NullifyPDF.py pii_detection.py privacy_core.py PDF_Checker.py
 
       - name: Resolve beta suffix from tag
         if: startsWith(github.ref, 'refs/tags/')

.github/workflows/test_build.yml

@@ -47,7 +47,7 @@ jobs:
 
       - name: Type Check (mypy)
         shell: bash
-        run: mypy NullifyPDF.py --ignore-missing-imports --no-error-summary 2>&1 | head -50 || true
+        run: mypy NullifyPDF.py pii_detection.py privacy_core.py PDF_Checker.py
 
       - name: Build
         run: python build_local.py

ARCHITECTURE.md

@@ -26,7 +26,8 @@ It supports two privacy export modes:
 | 📄 File | ✨ Responsibility |
 | ------ | ----------------- |
 | `NullifyPDF.py` | Main PySide6 UI, PDF rendering, AI scan orchestration, redaction, export, reconstruction, OCR integration |
-| `privacy_core.py` | Privacy mode enum, placeholder registry, encrypted restore-map payloads, restore-map parsing |
+| `pii_detection.py` | Qt-free AI detection helpers: word-box text index, offset-to-rect mapping, per-type score thresholds, generic-word filtering, whole-word propagation, entity selection persistence |
+| `privacy_core.py` | Privacy mode enum, placeholder registry, encrypted restore-map payloads (version 3, optional per-occurrence text style), restore-map parsing |
 | `PDF_Checker.py` | Heuristic post-export inspection helper |
 | `build_local.py` | PyInstaller build script |
 | `scripts/download_ocr_data.py` | Downloads EN/IT Tesseract `tessdata_fast` files bundled into every build |
@@ -88,18 +89,20 @@ Export pipeline writes a cleaned PDF copy
 
 ### 🤖 AI Scan
 
-1. User selects language: `EN`, `IT`, or `BOTH`.
-2. User optionally enables:
+1. User clicks **Auto Redact (AI)** and, in a dialog, picks which data types to look for (names, locations, email, phone, IBAN, cards, crypto, fiscal code, VAT, licence, ID card, passport). The selection is remembered in `~/.nullifypdf/ai_entities.json`.
+2. User selects language (`EN`, `IT`, or `BOTH`) and optionally enables:
    - `OCR PDF scansionati`
    - `Oscura Immagini`
 3. `cmd_auto_ai()` checks OCR configuration if OCR is enabled.
 4. `AIWorker.run_scan()` processes pages in a background thread.
 5. For each page:
-   - digital text is extracted with `page.get_text()`
-   - if the page looks scanned, OCR is run with `page.get_textpage_ocr()`
-   - Presidio analyzes the extracted/OCR text
-   - detected entities are emitted with text, entity type, source, and optional OCR rectangles
-6. `apply_ai_to_page()` runs on the UI thread and creates pending redaction annotations.
+   - the analysis text is built by `pii_detection.build_page_index()` from `page.get_text("words")`, recording each word's rectangle so character offsets map back to exact boxes
+   - if the page looks scanned, OCR is run with `page.get_textpage_ocr()` and the same index is built from the OCR words
+   - Presidio analyzes the text for the enabled entity types
+   - results are filtered by per-type confidence thresholds; NER spans are split at line breaks; generic PERSON/LOCATION hits (titles, months, salutations, lowercase, role and identifier words) are dropped; single-word hits that look like labels or headings (followed by a colon, list items, lone heading lines, words also written in lowercase on the page, capitalized line starts) are rejected; a name glued to a following address is cut at the street word; overlapping spans are resolved
+   - each surviving value is located by its offsets, and its other occurrences on the page are added as **whole words only**
+   - detections are emitted with text, entity type, score, source, and their rectangles
+6. `apply_ai_to_page()` runs on the UI thread and creates pending redaction annotations from the received rectangles. Blocklist and allowlist terms are matched as whole words too.
 
 ### 🖊️ Manual Review
 
@@ -185,6 +188,8 @@ The restore map contains:
 - original values
 - entity types
 - page numbers
+- the redaction box of every occurrence
+- for native (non-scanned) PDFs, the typography of the redacted text: font, size, colour, flags and baseline (map version 3, optional)
 
 The map is encrypted with `cryptography` using Fernet and a key derived from the user password with PBKDF2-HMAC-SHA256.
 
@@ -292,7 +297,7 @@ pytest tests/ -v
 
 ## ⚠️ Known Limits
 
-- Detection is probabilistic. Presidio, spaCy, and OCR can miss data or create false positives.
+- Detection is probabilistic. Presidio, spaCy, and OCR can miss data or create false positives. Heuristics that reject label-like single-word names trade a little recall (for example a lone first name that starts a line) for precision; such values can be marked manually.
 - OCR depends on Tesseract language data and scan quality.
 - Handwriting is not reliably supported.
 - Complex PDFs may contain structures not covered by automated cleanup.
@@ -304,5 +309,5 @@ pytest tests/ -v
 
 ---
 
-*Last updated: 2026-09-18*  
+*Last updated: 2026-09-19*  
 *[Back to README](./README.md) | [Development →](./DEVELOPMENT.md)*

CHANGELOG.md

@@ -7,6 +7,26 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [2.3.0] - 2026-09-19
+
+Validated through the 2.3.0-beta.1/beta.2/beta.3 prereleases before this stable release, promoted from `v2.3.0-beta.3` without rebuilding; only the final stable version gets its own heading here.
+
+### ✨ Added
+
+- **Choose what the AI scan looks for**: clicking "Auto Redact (AI)" now opens a dialog with one checkbox per data type (names, locations, email, phone, IBAN, credit cards, crypto wallets, fiscal code, VAT number, driver licence, ID card, passport). The choice is remembered in `~/.nullifypdf/ai_entities.json`.
+- **Reconstruction keeps the original typography on native PDFs**: for PDFs not produced by scanning (Word, Office, ...), the pseudonymization export now records the font, size, colour and baseline of each redacted value, and "Ricostruisci PDF" writes it back with that size, colour and position, reusing the original embedded font when present with all needed glyphs and otherwise the closest Base-14 font (serif/sans/mono, bold/italic). Restore maps are now version 3 (versions 1 and 2 still load); scanned pages keep the fitted-box behaviour.
+
+### 🐛 Fixed
+
+- **AI scan flagged labels and headings as names or places** ("Telefono:", "- Presente", "Configurazione"): NER hits are now judged line by line (a span no longer crosses a line break, and bullets/dashes are ignored), and a single-word PERSON/LOCATION hit is dropped when it is a field label followed by a colon, a list item, a lone heading line, a word also written in lowercase elsewhere on the page, or a capitalized line start followed by a lowercase word. A list of common label, section and job-title words (IT/EN) is also ignored. Multi-word names such as "Mario Rossi" are unaffected.
+- **AI scan crashed in packaged builds when an email was found**: Presidio's email recognizer needs `tldextract`'s `.tld_set_snapshot` data file, which PyInstaller did not bundle, so the scan aborted with `FileNotFoundError` under `_MEI...	ldextract`. The package is now collected by the spec and by `build_local.py`.
+- **AI scan selected ordinary text and parts of words**: detections were reduced to bare strings and then re-searched page-wide with a case-insensitive substring search, so one NER hit on "Rossi" also redacted "Rossini", and a stray tag spread to every occurrence on the page. Detections now use Presidio's exact character offsets mapped to word boxes, other occurrences are matched as whole words only, weak recognizers are filtered by per-type confidence thresholds, and generic PERSON/LOCATION hits (titles, months, salutations, lowercase text) are dropped. Blocklist and allowlist matching is whole-word too. A name that the NER model glues to the following address ("Rossini Marco Via Garibaldi") is now cut at the street word or number; the address part is kept as a location only when that type is enabled.
+- **A value wrapped over several boxes is restored per box**: reconstruction now restores the text that actually sat inside each box instead of writing the whole value into each one.
+
+### 🔧 Changed
+
+- **Type checking is now a blocking CI step**: `mypy` runs on `NullifyPDF.py`, `pii_detection.py`, `privacy_core.py` and `PDF_Checker.py` without `--ignore-missing-imports` or `|| true`. PyMuPDF is imported as `pymupdf` (the `fitz` alias ships no type marker), so its usage is really type-checked.
+
 ## [2.2.0] - 2026-09-18
 
 Validated through the 2.2.0-beta.1/beta.2/beta.3/beta.4 prereleases before this stable release, promoted from `v2.2.0-beta.4` without rebuilding; only the final stable version gets its own heading here.

CONTRIBUTING.md

@@ -219,9 +219,10 @@ thread when scanning large PDFs concurrently."
 ### Before Opening PR
 
 1. ✅ Run tests: `pytest tests/ -v`
-2. ✅ Build locally: `python build_local.py`
-3. ✅ Update CHANGELOG.md with your changes
-4. ✅ Self-review your code (would you understand this in 6 months?)
+2. ✅ Type check (blocking in CI): `mypy NullifyPDF.py pii_detection.py privacy_core.py PDF_Checker.py`
+3. ✅ Build locally: `python build_local.py`
+4. ✅ Update CHANGELOG.md with your changes
+5. ✅ Self-review your code (would you understand this in 6 months?)
 5. ✅ Check links in documentation
 
 ### PR Description Template
@@ -378,5 +379,5 @@ Every contribution — code, docs, bug reports, ideas — helps make NullifyPDF
 
 ---
 
-*Last updated: 2026-09-18*  
+*Last updated: 2026-09-19*  
 *← [Back to README](./README.md) | [Architecture →](./ARCHITECTURE.md)*

DEVELOPMENT.md

@@ -106,6 +106,7 @@ If the GUI opens → **You're ready to develop!** 🎉
 NullifyPDF/
 |-- NullifyPDF.py              # Main PySide6 application
 |-- privacy_core.py            # Privacy modes and encrypted restore maps
+|-- pii_detection.py           # Qt-free AI detection helpers (offsets, filters, whole-word matching)
 |-- PDF_Checker.py             # Heuristic verification utility
 |-- setup_env.py               # Environment setup script
 |-- build_local.py             # PyInstaller build script
@@ -115,6 +116,7 @@ NullifyPDF/
 |-- tests/
 |   |-- test_validation.py
 |   |-- test_privacy_core.py
+|   |-- test_pii_detection.py
 |   |-- test_reconstruct.py
 |   `-- test_build_config.py
 |-- images/
@@ -223,11 +225,23 @@ pytest tests/ --cov=. --cov-report=html
 
 Opens `htmlcov/index.html` in browser.
 
+### Type Checking
+
+CI runs `mypy` as a blocking step. Run the same check locally before opening a PR:
+
+```bash
+mypy NullifyPDF.py pii_detection.py privacy_core.py PDF_Checker.py
+```
+
+PyMuPDF is imported as `pymupdf` (aliased to `fitz`) so it is really type-checked; its dynamically defined constants are read once near the top of `NullifyPDF.py`.
+
 ### What's Tested
 
 - ✅ **PDFListManager** — File I/O, persistence
 - ✅ **OCR Config** — Tesseract language selection and tessdata discovery
-- ✅ **Privacy Core** — Placeholder mapping and encrypted restore maps
+- ✅ **Privacy Core** — Placeholder mapping and encrypted restore maps (v1/v2/v3)
+- ✅ **AI Detection Helpers** — Offset-to-rectangle mapping, thresholds, label/heading filtering, whole-word matching
+- ✅ **Reconstruction** — Placeholder restoration, original font/size/colour on native PDFs
 - ✅ **Build Config** — OCR data bundling behavior
 - ✅ **Resource Paths** — PyInstaller compatibility
 
@@ -438,6 +452,7 @@ isort NullifyPDF.py
 | ------------------ | ----------------------- |
 | `NullifyPDF.py`    | Main app, GUI, OCR, export logic |
 | `privacy_core.py`  | Placeholder and restore-map logic |
+| `pii_detection.py` | AI detection precision: word-box index, thresholds, filters, entity selection |
 | `setup_env.py`     | Environment setup |
 | `build_local.py`   | PyInstaller build |
 | `PDF_Checker.py`   | Post-processing utility |
@@ -519,11 +534,9 @@ Never hardcode API keys or passwords.
 ### Q: Where do I add new AI detections?
 
 **A:**
-In `AIWorker.run_scan()`:
-1. Use Presidio analyzer for regex patterns
-2. Use spaCy models for entity recognition
-3. Merge and deduplicate results
-4. Filter through allowlist
+1. Add the Presidio entity type (and its dialog label) to `ENTITY_GROUPS` in `pii_detection.py`, plus a threshold in `ENTITY_SCORE_THRESHOLDS`.
+2. `AIWorker.run_scan()` analyzes the page text built by `build_page_index()`. Filtering (`filter_results`, `clean_candidates`, `resolve_overlaps`) and offset-to-rectangle mapping (`span_to_rects`, `propagate_whole_word`) live in `pii_detection.py`, which has no Qt dependency and is unit-tested in `tests/test_pii_detection.py`.
+3. Detections are filtered through the allowlist in the worker.
 
 See [ARCHITECTURE.md](./ARCHITECTURE.md) for AI pipeline details.
 
@@ -563,5 +576,5 @@ Ready to contribute? See [CONTRIBUTING.md](./CONTRIBUTING.md) for:
 
 ---
 
-*Last updated: 2026-09-18*  
+*Last updated: 2026-09-19*  
 *← [Troubleshooting](./TROUBLESHOOTING.md) | [Back to README →](./README.md)*