Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/beta-release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ jobs:

- name: Type Check (mypy)
shell: bash
run: mypy NullifyPDF.py pii_detection.py privacy_core.py update_check.py PDF_Checker.py export_engine.py pdf_sanitize.py review_dialog.py lexicon_it.py secure_store.py word_lists.py dictionary_dialog.py help_guide.py
run: mypy NullifyPDF.py pii_detection.py privacy_core.py update_check.py PDF_Checker.py export_engine.py pdf_sanitize.py review_dialog.py lexicon_it.py secure_store.py word_lists.py dictionary_dialog.py help_guide.py spacy_models.py licences_info.py

- name: Resolve beta suffix from tag
if: startsWith(github.ref, 'refs/tags/')
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/build-linux.yml
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ jobs:
run: pytest tests/ -v

- name: Type check (mypy)
run: mypy NullifyPDF.py pii_detection.py privacy_core.py update_check.py PDF_Checker.py export_engine.py pdf_sanitize.py review_dialog.py lexicon_it.py secure_store.py word_lists.py dictionary_dialog.py help_guide.py
run: mypy NullifyPDF.py pii_detection.py privacy_core.py update_check.py PDF_Checker.py export_engine.py pdf_sanitize.py review_dialog.py lexicon_it.py secure_store.py word_lists.py dictionary_dialog.py help_guide.py spacy_models.py licences_info.py

- name: Resolve beta suffix from tag
if: inputs.is_beta && startsWith(github.ref, 'refs/tags/')
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@ jobs:

- name: Type Check (mypy)
shell: bash
run: mypy NullifyPDF.py pii_detection.py privacy_core.py update_check.py PDF_Checker.py export_engine.py pdf_sanitize.py review_dialog.py lexicon_it.py secure_store.py word_lists.py dictionary_dialog.py help_guide.py
run: mypy NullifyPDF.py pii_detection.py privacy_core.py update_check.py PDF_Checker.py export_engine.py pdf_sanitize.py review_dialog.py lexicon_it.py secure_store.py word_lists.py dictionary_dialog.py help_guide.py spacy_models.py licences_info.py

- name: Build
run: python build_local.py
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/test_build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ jobs:

- name: Type Check (mypy)
shell: bash
run: mypy NullifyPDF.py pii_detection.py privacy_core.py update_check.py PDF_Checker.py export_engine.py pdf_sanitize.py review_dialog.py lexicon_it.py secure_store.py word_lists.py dictionary_dialog.py help_guide.py
run: mypy NullifyPDF.py pii_detection.py privacy_core.py update_check.py PDF_Checker.py export_engine.py pdf_sanitize.py review_dialog.py lexicon_it.py secure_store.py word_lists.py dictionary_dialog.py help_guide.py spacy_models.py licences_info.py

- name: Build
run: python build_local.py
Expand Down
7 changes: 2 additions & 5 deletions ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,7 @@ Export pipeline writes a cleaned PDF copy
### 🤖 AI Scan

1. User clicks **Auto Redact** and, in a dialog, picks which data types to look for (names, locations, email, phone, IBAN, cards, crypto, fiscal code, VAT, licence, ID card, passport). The selection is remembered in `~/.nullifypdf/ai_entities.json`.
2. User selects language (`EN`, `IT`, or `BOTH`) and optionally enables:
2. The language is always Italian (`IT`); the user optionally enables:
- `OCR PDF scansionati`
- `Oscura Immagini`
3. `cmd_auto_ai()` checks OCR configuration if OCR is enabled.
Expand Down Expand Up @@ -224,10 +224,7 @@ Runtime behavior:
- bundled `ocr/tessdata` via `resource_path()`
- PyMuPDF's `fitz.get_tessdata()`
- common platform locations
2. `ocr_language_for_choice()` maps UI language to Tesseract codes:
- `EN` -> `eng`
- `IT` -> `ita`
- `BOTH` -> `eng+ita`
2. `ocr_language_for_choice()` returns the Tesseract codes: `ita`, plus `eng` when its data is installed (so English words inside Italian documents are read well).
3. `AIWorker._page_needs_ocr()` runs OCR only when text extraction is missing or very poor.
4. OCR text is used for Presidio detection.
5. OCR search rectangles are used to place redactions on scanned pages.
Expand Down
8 changes: 8 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

### 🔄 Changed

- **Italian only**: the English model and the EN/IT/BOTH selector are gone. The program analyses Italian documents; the sidebar shows "Lingua dei documenti: italiano". The English OCR data is still used, when installed, only to read English words inside Italian pages.
- **Model download progress in the button**: in the local-models window the "Scarica modello" button now fills like a progress bar with the percentage, a second click cancels, the list and the Close button are locked while downloading, and the download no longer blocks the window. The empty grey button of the old progress window is gone.
- **New licence: AGPL-3.0-or-later** (previously MIT), because the program uses PyMuPDF, which is AGPL-3.0 or commercial. The program stays free and is not sold. Full third-party licence list in `THIRD_PARTY_NOTICES.md`.
- **Licence notice in the Info window and README**: the Info window now lists the licence of NullifyPDF and of every component and model it uses, with clickable links to the official texts.
- **Italian name-recognition model downloaded on demand**: its licence (CC BY-NC-SA 3.0, non-commercial) cannot be bundled in an AGPL installer, so the first Italian scan shows the licence and asks for consent, then downloads and verifies the model (SHA-256). Detection quality is unchanged once installed; if you decline, the scan continues with rules and dictionaries only and the log says so.

### ✨ Added

- **Local PII models**: optional, user-downloaded ONNX models from pinned Hugging Face revisions are stored under `~/.nullifypdf/models/`, verified file-by-file with SHA-256, and contribute suggestions only. The model dialog shows the source, license, revision and download progress; unverified experimental entries remain blocked.
Expand Down
6 changes: 5 additions & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -379,5 +379,9 @@ Every contribution — code, docs, bug reports, ideas — helps make NullifyPDF

---

*Last updated: 2026-11-08*
*Last updated: 2026-11-11*
*← [Back to README](./README.md) | [Architecture →](./ARCHITECTURE.md)*

## Licence of contributions

NullifyPDF is licensed under the AGPL-3.0-or-later (see [LICENSE](./LICENSE)). By submitting a contribution you agree that it is released under the same licence.
20 changes: 12 additions & 8 deletions DEVELOPMENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ Before starting, verify you have:
| **Python 3.13** | 150 MB | [Download](https://www.python.org/downloads/) |
| **Git** | 50 MB | [Download](https://git-scm.com/) |
| **Virtual Environment** | 2 GB | `.venv/` auto-created by setup script |
| **Disk Space** | 3 GB | Dependencies + spaCy models |
| **Disk Space** | 3 GB | Dependencies + Italian model |
| **RAM** | 4 GB | 8 GB recommended |

---
Expand Down Expand Up @@ -51,7 +51,7 @@ python setup_env.py
This automatically:
- ✅ Creates `.venv/` virtual environment
- ✅ Installs dependencies
- ✅ Downloads spaCy models (EN + IT)
- ✅ Installs the Italian spaCy model (development only)

### 4️⃣ Activate Virtual Environment

Expand Down Expand Up @@ -151,14 +151,18 @@ NullifyPDF/
| **python-docx** | — | Not required; DOCX import uses the standard-library OOXML reader |
| **pytest** | 9.2.0 | Testing framework |

### Language Models (Auto-Downloaded)
### Language Models

| Model | Size | Purpose |
| ----------------- | ----- | ----------- |
| `en_core_web_md` | 40 MB | English NER |
| `it_core_news_md` | 45 MB | Italian NER |
| Model | Size | Purpose | Licence | In the installer |
| ----------------- | ----- | ----------- | ------- | ---------------- |
| `it_core_news_md` | 45 MB | Italian NER | CC BY-NC-SA 3.0 | **no**, downloaded by the app on demand (`spacy_models.py`) |

These are downloaded automatically by `setup_env.py`.
`setup_env.py` installs both in the development environment (the Italian tests
need it). The PyInstaller specs exclude `it_core_news_md`, `pymupdf4llm` and
`pymupdf.layout`. At run time the app asks for consent, downloads the Italian
pipeline from the publisher's release, checks its size and SHA-256 and stores it
under `~/.nullifypdf/spacy_models/`. If the user declines, a blank Italian
pipeline is used and only rule-based detection is available.

---

Expand Down
Loading
Loading