From beef340ba7674d7d55f4e29f70152a8cfe643a2c Mon Sep 17 00:00:00 2001 From: 0verwrite <31691645+overwrite00@users.noreply.github.com> Date: Wed, 11 Nov 2026 11:33:22 +0100 Subject: [PATCH] docs: align all documentation with the 2.4.0 code (#138) --- ARCHITECTURE.md | 43 ++++++++++++++------- CONTRIBUTING.md | 4 +- DEVELOPMENT.md | 85 +++++++++++++++++++++++++++++------------- OCR_SETUP.md | 4 +- README.md | 36 ++++++++++++------ SECURITY.md | 27 +++++++------- THIRD_PARTY_NOTICES.md | 3 ++ TROUBLESHOOTING.md | 22 +++++++++-- USER_GUIDE.md | 14 ++++--- help_guide.py | 1 + licences_info.py | 20 ++++++++++ 11 files changed, 182 insertions(+), 77 deletions(-) diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index f9e324d..107ac99 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -25,7 +25,7 @@ It supports two privacy export modes: | ๐Ÿ“„ File | โœจ Responsibility | | ------ | ----------------- | -| `NullifyPDF.py` | Main PySide6 UI, PDF rendering, AI scan orchestration, redaction, export, reconstruction, OCR integration | +| `NullifyPDF.py` | Main PySide6 UI, PDF rendering, AI scan orchestration (including the Italian-model consent step), redaction, export, reconstruction, OCR integration, Info window | | `update_check.py` | Qt-free update logic: version comparison (SemVer, stable/beta channels), GitHub release lookup, per-OS asset selection, trusted-host download, startup-check preference (`~/.nullifypdf/update_check.json`). Only runs on user click or if the user enabled the startup check | | `pii_detection.py` | Qt-free AI detection helpers: word-box text index (neighbour-aware rectangle clamping, ligatures, letter-spaced and hyphenated words), offset-to-rect mapping and partial-match policy, per-type thresholds, pattern validation, evidence-based PERSON/LOCATION scoring, accepted/suggested states, allowlist by word identity, whole-word propagation, entity selection persistence | | `privacy_core.py` | Privacy mode enum, placeholder registry (random / sequential / generic / keyed styles), encrypted restore-map payloads (version 3, optional per-occurrence text style; scrypt, PBKDF2 maps still readable), password strength check, restore-map parsing | @@ -34,7 +34,9 @@ It supports two privacy export modes: | `export_engine.py` | Qt-free export pipeline (`export_document`), neutral file names, private temp handling, audit record, local-data wipe | | `review_dialog.py` | Qt dialog to accept or reject suggested detections | | `model_manager.py` | Pinned Hugging Face model catalog, SHA-256 verification, local model store and hardware recommendation | -| `model_dialog.py` | Italian model selection, consent and progress-aware downloads | +| `model_dialog.py` | Local ONNX model dialog: catalog, details, consent and a threaded download whose progress fills the download button (second click cancels; list and Close are locked meanwhile) | +| `spacy_models.py` | The spaCy Italian pipeline (CC BY-NC-SA 3.0) downloaded on demand: consent text, pinned size and SHA-256, safe unpacking into `~/.nullifypdf/spacy_models/`, blank-pipeline fallback | +| `licences_info.py` | Licence of NullifyPDF and of every component and model with official links, used by the Info window and the README table | | `onnx_ner.py` | CPU ONNX token-classification adapter with character-offset mapping and label mapping | | `word_import.py` | Safe DOCX OOXML text/table extraction into an internal PyMuPDF document; no macro execution | | `help_guide.py` | In-app guide (Italian, F1) and the tooltip table applied to every widget | @@ -45,7 +47,7 @@ It supports two privacy export modes: | `PDF_Checker.py` | Heuristic post-export inspection helper | | `build_local.py` | PyInstaller build script | | `scripts/download_ocr_data.py` | Downloads EN/IT Tesseract `tessdata_fast` files bundled into every build | -| `tests/` | Unit and smoke tests for validation, privacy primitives, OCR config, and build configuration | +| `tests/` | Unit, integration and offscreen GUI tests (see [Tests](#-tests)) | --- @@ -107,8 +109,8 @@ Export pipeline writes a cleaned PDF copy 2. The language is always Italian (`IT`); the user optionally enables: - `OCR PDF scansionati` - `Oscura Immagini` -3. `cmd_auto_ai()` checks OCR configuration if OCR is enabled. -4. `AIWorker.run_scan()` processes pages in a background thread. +3. `cmd_auto_ai()` calls `_prepare_italian_model()`: if the spaCy Italian pipeline is neither installed as a package nor in `~/.nullifypdf/spacy_models/`, a dialog shows its licence and asks for consent (download, continue with rules only, or cancel); then it checks OCR configuration if OCR is enabled. +4. `AIWorker.run_scan()` processes pages in a background thread; Presidio loads only the Italian pipeline (`AIWorker.italian_model`), and installed local ONNX models are loaded once per scan and add suggestions only. 5. For each page: - the analysis text is built by `pii_detection.build_page_index()` from `page.get_text("words")`, recording each word's rectangle so character offsets map back to exact boxes. Each rectangle is trimmed against the words of neighbouring lines (a word box spans ascender to descender and would otherwise overlap them); ligatures are expanded, letter-spaced runs become one word and a word hyphenated across two lines is joined - if the page looks scanned, OCR is run with `page.get_textpage_ocr()` and the same index is built from the OCR words; on text pages with embedded images only the images are OCR'd and merged @@ -229,7 +231,18 @@ Runtime behavior: 4. OCR text is used for Presidio detection. 5. OCR search rectangles are used to place redactions on scanned pages. -Every release bundles EN/IT OCR data. +Every release bundles EN/IT OCR data. The documents analysed are Italian, so `ita` is the OCR language; `eng` is added only to read English words inside Italian pages. + +--- + +## ๐Ÿง  Models + +| Model | Where it comes from | Role | +| ----- | ------------------- | ---- | +| spaCy `it_core_news_md` 3.8.0 | Downloaded on first use after consent (`spacy_models.py`); never bundled, because its licence (CC BY-NC-SA 3.0) is non-commercial | Statistical recognition of names and places through Presidio | +| ONNX models (MiniLMv2 EN/IT, PiiScan, OpenMed Italian FastClinical) | Optional, downloaded from pinned Hugging Face revisions (`model_manager.py`), every file checked against size and SHA-256 | Extra suggestions only; never applied automatically | + +The development environment installs the Italian spaCy package so the tests can run; the PyInstaller specs exclude it, together with `pymupdf4llm` and `pymupdf.layout`. --- @@ -277,6 +290,8 @@ Beta releases (`v*-beta.N` tags) and stable releases (`vX.Y.Z` tags) are each co **Linux packages** are built by the reusable workflow `build-linux.yml`, shared by the stable, beta and test builds. The PyInstaller binary and the `.rpm` are built once inside a `rockylinux:9` container (glibc 2.34, rpm 4.16), the `.deb` is wrapped from the same binary on `ubuntu-24.04`, and each package is then installed and started under Xvfb in clean Ubuntu 24.04, Ubuntu 26.04, Debian 13, Rocky 9, Rocky 10 and Fedora containers. A release is published only if this smoke test passes. Runners are pinned, so the move of `ubuntu-latest` to a new Ubuntu release does not change the build. When a distribution approaches end of support, update the matrix in `build-linux.yml` (and the list in the README). +Pull requests also run the unit tests against the latest PyMuPDF, a guard that only `develop` may be merged into `main`, project automation, and GitHub code scanning (CodeQL on pull requests to `main`). + Expected artifact names include: ```text @@ -292,14 +307,13 @@ NullifyPDF_vX.Y.Z_Fedora.rpm Current tests cover: -- list persistence -- resource path resolution -- OCR configuration helpers -- privacy placeholder registry -- encrypted restore-map round trip -- reconstruction of a pseudonymized PDF from its restore map -- build configuration -- OCR data inclusion behavior +- detection pipeline, geometry (rectangle clamping, glyph-deletion rule), scoring and the Italian lexicon (`test_pii_detection.py`, `test_detection_pipeline.py`, `test_geometry.py`) +- redaction engine, export sanitiser and export verification (`test_redaction_engine.py`, `test_export_sanitize.py`, `test_privacy_hardening.py`) +- placeholders, encrypted restore maps and reconstruction (`test_privacy_core.py`, `test_reconstruct.py`) +- encrypted dictionaries and the key store (`test_secure_store.py`) +- offscreen GUI flows: review, export, reconstruction, guide and tooltips, Info window, model dialog (`test_gui_flow.py`, `test_gui_help.py`, `test_model_dialog.py`) +- model catalog, on-demand Italian model download, DOCX import, update check (`test_model_manager.py`, `test_spacy_models.py`, `test_word_import.py`, `test_update_check.py`) +- licence consistency (no non-commercial component in the installer) and build configuration (`test_licences.py`, `test_build_config.py`, `test_validation.py`) Run: @@ -313,6 +327,7 @@ pytest tests/ -v - Detection is probabilistic. Presidio, spaCy, and OCR can miss data or create false positives. Heuristics that reject label-like single-word names trade a little recall (for example a lone first name that starts a line) for precision; such values can be marked manually. - OCR depends on Tesseract language data and scan quality. +- Only Italian documents are supported. Without the Italian spaCy model (download declined) only rules and dictionaries run and names are found less often. - Handwriting is not reliably supported. - Complex PDFs may contain structures not covered by automated cleanup. - Pseudonymized PDFs are still personal-data processing when the restore map exists. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 5f101e2..d97e0ad 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -24,7 +24,7 @@ git remote add upstream https://github.com/overwrite00/NullifyPDF.git python setup_env.py ``` -This creates isolated venv, installs dependencies, and downloads NLP models. +This creates an isolated venv, installs dependencies and installs the Italian spaCy model used by the tests (the released program downloads it on demand). Then enable the repo's git hooks (one-time per clone) so the `Last updated` footer in docs stays in sync automatically: @@ -219,7 +219,7 @@ thread when scanning large PDFs concurrently." ### Before Opening PR 1. โœ… Run tests: `pytest tests/ -v` -2. โœ… Type check (blocking in CI): `mypy NullifyPDF.py pii_detection.py privacy_core.py update_check.py PDF_Checker.py` +2. โœ… Type check (blocking in CI): `mypy NullifyPDF.py pii_detection.py privacy_core.py update_check.py PDF_Checker.py export_engine.py pdf_sanitize.py review_dialog.py lexicon_it.py secure_store.py word_lists.py dictionary_dialog.py help_guide.py spacy_models.py licences_info.py` 3. โœ… Build locally: `python build_local.py` 4. โœ… Update CHANGELOG.md with your changes 5. โœ… Self-review your code (would you understand this in 6 months?) diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index 36287ac..5cd66b1 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -105,32 +105,44 @@ If the GUI opens โ†’ **You're ready to develop!** ๐ŸŽ‰ ``` NullifyPDF/ |-- NullifyPDF.py # Main PySide6 application -|-- model_manager.py # Pinned local model catalog and integrity checks -|-- model_dialog.py # Model consent and download UI -|-- onnx_ner.py # ONNX token-classification adapter -|-- word_import.py # Safe DOCX text/table importer -|-- privacy_core.py # Privacy modes and encrypted restore maps +|-- pii_detection.py # Qt-free detection: word index, geometry, scoring, partial-match policy +|-- lexicon_it.py # Italian word lists; lexicon_data_it.py is generated (ISTAT, Wikidata) +|-- privacy_core.py # Privacy modes, placeholders and encrypted restore maps +|-- export_engine.py # Qt-free export pipeline, audit record, local-data wipe +|-- pdf_sanitize.py # Export sanitiser and output verification +|-- review_dialog.py # Review of suggested detections +|-- secure_store.py # Master key protection (DPAPI / keyring / key file), AES-GCM sealing +|-- word_lists.py # Encrypted blocklist/allowlist with expiry +|-- dictionary_dialog.py # Dictionary editor +|-- help_guide.py # In-app guide (Italian) and tooltips +|-- licences_info.py # Licences and links shown in the Info window and README +|-- spacy_models.py # Italian spaCy pipeline downloaded on demand after consent +|-- model_manager.py # Pinned local ONNX model catalog and integrity checks +|-- model_dialog.py # Local model dialog with in-button download progress +|-- onnx_ner.py # ONNX token-classification adapter +|-- word_import.py # Safe DOCX text/table importer |-- update_check.py # Qt-free update check: versions, GitHub releases, asset choice, download -|-- pii_detection.py # Qt-free AI detection helpers (offsets, filters, whole-word matching) |-- PDF_Checker.py # Heuristic verification utility |-- setup_env.py # Environment setup script |-- build_local.py # PyInstaller build script |-- requirements.txt # Python dependencies +|-- LICENSE, THIRD_PARTY_NOTICES.md |-- scripts/ -| `-- download_ocr_data.py # EN/IT OCR data downloader -|-- tests/ -| |-- test_validation.py -| |-- test_privacy_core.py -| |-- test_pii_detection.py -| |-- test_reconstruct.py -| `-- test_build_config.py +| |-- download_ocr_data.py # EN/IT OCR data downloader +| |-- build_lexicon_data.py # Generates lexicon_data_it.py (development only) +| |-- extract_release_notes.py +| `-- update_docs_dates.py +|-- tests/ # pytest suite (unit, integration and offscreen GUI tests) |-- images/ | `-- NullifyPDF.png `-- .github/ `-- workflows/ - |-- test_build.yml - |-- beta-release.yml - `-- release.yml + |-- test_build.yml # unit tests and full build matrix + |-- beta-release.yml # v*-beta.N tags + |-- release.yml # vX.Y.Z tags + |-- build-linux.yml # reusable: .deb/.rpm build and smoke tests + |-- guard-main-source.yml + `-- project-automation.yml ``` --- @@ -145,11 +157,14 @@ NullifyPDF/ | **PyMuPDF** | 2.1 | PDF manipulation and OCR bridge | | **presidio-analyzer** | 2.2.366 | PII detection | | **spaCy** | 3.8.18 | NLP for entity recognition | -| **cryptography** | 50.0.2 | Encrypted restore maps | +| **cryptography** | 50.0.2 | Encrypted restore maps and dictionaries | +| **keyring** | 25.7.0 | OS key store (macOS and Linux only) | | **onnxruntime** | 1.31.1 | CPU inference for optional local PII models | | **tokenizers** | 0.23.3 | Fast local tokenizer with character offsets | | **python-docx** | โ€” | Not required; DOCX import uses the standard-library OOXML reader | | **pytest** | 9.2.0 | Testing framework | +| **mypy** | 1.20.2 | Type checking (blocking in CI) | +| **PyInstaller** | 6.22.3 | Builds the executables | ### Language Models @@ -157,13 +172,17 @@ NullifyPDF/ | ----------------- | ----- | ----------- | ------- | ---------------- | | `it_core_news_md` | 45 MB | Italian NER | CC BY-NC-SA 3.0 | **no**, downloaded by the app on demand (`spacy_models.py`) | -`setup_env.py` installs both in the development environment (the Italian tests +`setup_env.py` installs it in the development environment (the Italian tests need it). The PyInstaller specs exclude `it_core_news_md`, `pymupdf4llm` and `pymupdf.layout`. At run time the app asks for consent, downloads the Italian pipeline from the publisher's release, checks its size and SHA-256 and stores it under `~/.nullifypdf/spacy_models/`. If the user declines, a blank Italian pipeline is used and only rule-based detection is available. +The optional ONNX models (MiniLMv2 EN/IT, PiiScan, OpenMed Italian FastClinical) +are described in `model_manager.py`; they are downloaded by the user from pinned +Hugging Face revisions into `~/.nullifypdf/models/` and add suggestions only. + --- ## ๐Ÿ” Understanding the Code @@ -242,20 +261,23 @@ Opens `htmlcov/index.html` in browser. CI runs `mypy` as a blocking step. Run the same check locally before opening a PR: ```bash -mypy NullifyPDF.py pii_detection.py privacy_core.py update_check.py PDF_Checker.py +mypy NullifyPDF.py pii_detection.py privacy_core.py update_check.py PDF_Checker.py export_engine.py pdf_sanitize.py review_dialog.py lexicon_it.py secure_store.py word_lists.py dictionary_dialog.py help_guide.py spacy_models.py licences_info.py ``` PyMuPDF is imported as `pymupdf` (aliased to `fitz`) so it is really type-checked; its dynamically defined constants are read once near the top of `NullifyPDF.py`. ### What's Tested -- โœ… **PDFListManager** โ€” File I/O, persistence -- โœ… **OCR Config** โ€” Tesseract language selection and tessdata discovery -- โœ… **Privacy Core** โ€” Placeholder mapping and encrypted restore maps (v1/v2/v3) -- โœ… **AI Detection Helpers** โ€” Offset-to-rectangle mapping, thresholds, label/heading filtering, whole-word matching -- โœ… **Reconstruction** โ€” Placeholder restoration, original font/size/colour on native PDFs -- โœ… **Build Config** โ€” OCR data bundling behavior -- โœ… **Resource Paths** โ€” PyInstaller compatibility +- โœ… **Detection** โ€” word index, rectangle clamping and the glyph-deletion rule, scoring, partial-match policy, Italian lexicon +- โœ… **Redaction and export** โ€” redaction engine, sanitiser, export verification (survivor and leak checks) +- โœ… **Privacy Core** โ€” placeholders and encrypted restore maps (v1 to v4 formats, Argon2id envelope, recovery key), reconstruction with the original font +- โœ… **Local data** โ€” encrypted dictionaries with expiry and the secure key store +- โœ… **GUI flows** โ€” offscreen tests of review, export, reconstruction, guide and tooltips, Info window, model dialog and Italian-model consent +- โœ… **Models and import** โ€” model catalog and integrity checks, on-demand Italian model download, DOCX import, update check +- โœ… **Licences and build** โ€” licence consistency, OCR data bundling, PyInstaller configuration, resource paths + +> [!TIP] +> GUI tests run offscreen (`QT_QPA_PLATFORM=offscreen`) and never touch the OS keyring (`NULLIFYPDF_DISABLE_KEYRING=1`). --- @@ -469,6 +491,10 @@ isort NullifyPDF.py | `pdf_sanitize.py` | Export sanitiser and output verification | | `export_engine.py` | Qt-free export pipeline, audit record, local-data hygiene | | `review_dialog.py` | Review dialog for suggested detections | +| `spacy_models.py` | Italian spaCy pipeline downloaded on demand (consent, SHA-256, fallback) | +| `licences_info.py` | Licences and official links shown in the Info window and README (a test keeps README in step) | +| `model_manager.py` / `model_dialog.py` / `onnx_ner.py` | Optional ONNX models: catalog, verified download, adapter | +| `word_import.py` | DOCX import | | `secure_store.py` | Master key protection (DPAPI / keyring / key file), AES-GCM sealing | | `word_lists.py` | Encrypted blocklist/allowlist with expiry | | `dictionary_dialog.py` | Dictionary editor dialog | @@ -595,6 +621,13 @@ Ready to contribute? See [CONTRIBUTING.md](./CONTRIBUTING.md) for: - ๐Ÿ’ฌ **GitHub Discussions:** [Discussions](https://github.com/overwrite00/NullifyPDF/discussions) - ๐Ÿ› **GitHub Issues:** [Issues](https://github.com/overwrite00/NullifyPDF/issues) +### Release procedure + +- **Beta**: work lands on `develop` through squash-merged pull requests; an annotated tag `vX.Y.Z-beta.N` on `develop` starts `beta-release.yml`. Set `__version_prerelease__` in `NullifyPDF.py` to the beta suffix. +- **Stable**: set `__version_prerelease__` to `""`, move the `[Unreleased]` changelog entries under `## [X.Y.Z] - date` (leave `[Unreleased]` empty), merge `develop` into `main` with a merge commit through a pull request (one approving review is required), then tag `vX.Y.Z` on `main`. `release.yml` checks the changelog and the version before building. +- Documentation-only changes do not need a release. +- When dependencies change, update `THIRD_PARTY_NOTICES.md` and `licences_info.py`; `tests/test_licences.py` fails if a non-commercial component would be bundled. + ### Release notes and the update check The release workflows fill each GitHub release body with `scripts/extract_release_notes.py`: the `## [X.Y.Z]` section for stable tags, `## [Unreleased]` for beta tags. The in-app update check displays that body, so keep the changelog sections accurate before tagging. diff --git a/OCR_SETUP.md b/OCR_SETUP.md index 35a0755..5e31b02 100644 --- a/OCR_SETUP.md +++ b/OCR_SETUP.md @@ -21,6 +21,8 @@ Install or download the language data you need: - `eng.traineddata` for English OCR. - `ita.traineddata` for Italian OCR. +NullifyPDF analyses Italian documents, so `ita` is the language that matters; `eng` is used in addition, when present, to read English words inside Italian pages. + Release builds always bundle these two files automatically. Developers can prepare a local build with: @@ -37,5 +39,5 @@ processing scanned pages without OCR. --- -*Last updated: 2026-09-18* +*Last updated: 2026-11-11* *[Back to README](./README.md) | [Architecture โ†’](./ARCHITECTURE.md)* diff --git a/README.md b/README.md index 57f2912..2055fb8 100644 --- a/README.md +++ b/README.md @@ -23,7 +23,7 @@ NullifyPDF goes beyond simple text covering. It uses **Natural Language Processi | ๐ŸŽฏ Feature | โœจ Benefit | | -------------------- | ------------------------------------- | -| ๐Ÿง  **Auto-detection** | Bilingual (EN/IT) automatic detection | +| ๐Ÿง  **Auto-detection** | Automatic detection tuned for Italian documents | | ๐Ÿ” **Local-first** | No cloud uploads during PDF processing | | โšก **Responsive UI** | Background scanning with live preview | | ๐Ÿ›ก๏ธ **Privacy Export** | Irreversible redaction or reversible placeholders | @@ -32,14 +32,15 @@ NullifyPDF goes beyond simple text covering. It uses **Natural Language Processi ## โœจ Key Features -- ๐Ÿง  **Automatic Redaction** โ€” Automatic bilingual (EN/IT) detection of PII: names, locations, emails, phones, IBANs, credit cards, crypto addresses, Italian fiscal codes and IDs. A dialog lets you choose which data types to scan for, and matching is whole-word so ordinary text is not caught +- ๐Ÿง  **Automatic Redaction** โ€” Automatic detection of PII in Italian documents: names, locations, emails, phones, IBANs, credit cards, crypto addresses, Italian fiscal codes and IDs. A dialog lets you choose which data types to scan for, and matching is whole-word so ordinary text is not caught - ๐Ÿ”„ **Update Check** โ€” Optional, user-triggered check for new releases (stable builds see stable only, beta builds see both) showing the release notes and a one-click download into your Downloads folder; can be enabled at every start from the Info dialog (off by default) - ๐Ÿ—„๏ธ **Fluid UI & Thread-Safe** โ€” PySide6 modern dark-mode interface with zero UI freezing. Text extraction in worker thread with QMutex serialization - ๐Ÿ“– **Persistent Dictionaries** โ€” Blocklist and Allowlist synchronized to disk (`~/.nullifypdf`) with O(1) fast-path matching - ๐Ÿ›ก๏ธ **Privacy Export Modes** โ€” Choose irreversible removal or reversible pseudonymization with a separately encrypted restore map; the exported file is sanitised (annotations, attachments, scripts, metadata, hidden text) and verified - ๐Ÿ“– **Built-in guide and tooltips** โ€” Press F1 for an Italian guide to every function; every control shows a description on hover - ๐Ÿ” **Review of uncertain hits** โ€” Detections without enough evidence are suggested for review instead of being applied or lost silently -- ๐Ÿง  **Optional local models** โ€” Reviewed ONNX PII models can be downloaded by the user from pinned Hugging Face revisions; files are SHA-256 verified and models only add review suggestions +- ๐Ÿ‡ฎ๐Ÿ‡น **Italian name recognition on demand** โ€” the spaCy Italian model (CC BY-NC-SA 3.0, non-commercial) is not in the installer: the first Auto Redact shows its licence, asks for consent and downloads it (SHA-256 verified). Declining keeps rules and dictionaries only +- ๐Ÿง  **Optional local models** โ€” Three reviewed ONNX PII models (MiniLMv2 EN/IT, PiiScan, OpenMed Italian FastClinical) can be downloaded by the user from pinned Hugging Face revisions; files are SHA-256 verified, the download progress is shown inside the button, and models only add review suggestions - ๐Ÿ“„ **Word import** โ€” DOCX paragraphs and tables can be imported locally into the analysis workflow; legacy binary DOC files are not supported - ๐Ÿ”Ž **OCR Support** โ€” Bundled EN/IT Tesseract language data for scanned PDFs - ๐Ÿ–ผ๏ธ **Blindfold Mode** โ€” One-click image/logo censoring with professional placeholder: `[ IMAGE REMOVED ]` @@ -71,7 +72,7 @@ To keep NullifyPDF lightweight, 100% offline, and secure, be aware of these tech ``` โœ… Python 3.13 (required for PyMuPDF wheel compatibility) -โœ… 2 GB disk space (dependencies + spaCy models) +โœ… 2 GB disk space (dependencies + Italian model) โœ… 4 GB RAM minimum (8 GB recommended for large PDFs) ``` @@ -199,12 +200,13 @@ pytest tests/ -v ``` **Test coverage:** -- PDFListManager (blocklist/allowlist persistence) -- Resource path resolution -- OCR configuration helpers -- Privacy placeholder and encrypted restore-map primitives -- AI detection helpers (offset mapping, filtering, whole-word matching) and reconstruction with the original font -- Build configuration +- Detection pipeline: word index, rectangle geometry, scoring, partial-match policy, Italian lexicon +- Redaction engine, export sanitiser and export verification (survivor and leak checks) +- Privacy placeholders, encrypted restore maps (Argon2id envelope, recovery key) and reconstruction +- Encrypted dictionaries with expiry and the secure key store +- Offscreen GUI flows (review, export, reconstruction, guide and tooltips, model dialog, Info window) +- Local model catalog, on-demand Italian model download, DOCX import, update check +- Licence consistency and build configuration @@ -221,13 +223,16 @@ pytest tests/ -v | [OCR_SETUP.md](./OCR_SETUP.md) | OCR data setup and build notes | | [TROUBLESHOOTING.md](./TROUBLESHOOTING.md) | Common issues & solutions | | [CHANGELOG.md](./CHANGELOG.md) | Release history & updates | +| [SECURITY.md](./SECURITY.md) | Security model & responsible disclosure | +| [THIRD_PARTY_NOTICES.md](./THIRD_PARTY_NOTICES.md) | Licences of data and components | +| [LICENSE](./LICENSE) | AGPL-3.0-or-later licence text | --- ## ๐Ÿ”’ Security & Privacy โœ… **100% Local Processing** โ€” All analysis happens on your machine -โœ… **No Cloud Uploads** โ€” PDF processing stays local +โœ… **No Cloud Uploads** โ€” PDF content never leaves your computer; the only network access is the optional update check and the model downloads you approve โœ… **Open Source** โ€” Full code transparency โœ… **No Telemetry** โ€” Zero user tracking @@ -246,7 +251,8 @@ See [SECURITY.md](./SECURITY.md) for responsible disclosure and privacy details. | **PySide6 (Qt6)** | Modern dark-mode GUI with multi-threading | | **PyMuPDF (fitz)** | High-performance PDF manipulation | | **Microsoft Presidio** | PII (Personally Identifiable Information) detection | -| **spaCy** | NLP for entity recognition (bilingual EN/IT) | +| **spaCy** | NLP for entity recognition (Italian model, downloaded on demand) | +| **ONNX Runtime** | CPU inference for the optional local PII models | | **cryptography** | Encrypted pseudonymization restore maps | --- @@ -278,6 +284,12 @@ components are listed in [THIRD_PARTY_NOTICES.md](./THIRD_PARTY_NOTICES.md). | [Presidio](https://github.com/microsoft/presidio/blob/main/LICENSE) | MIT | personal-data detection | | [spaCy](https://github.com/explosion/spaCy/blob/master/LICENSE) | MIT | language processing | | [spaCy it_core_news_md](https://creativecommons.org/licenses/by-nc-sa/3.0/) | CC BY-NC-SA 3.0 (non-commercial use only) | Italian model, downloaded after consent | +| [MiniLMv2 EN/IT NER (optional download)](https://huggingface.co/jkomyno/minilmv2-l6-h384-ner-en-it-q8) | MIT | optional local model by Alberto Schiabel (jkomyno) | +| [PiiScan (optional download)](https://huggingface.co/vectorsense/piiscan) | Apache-2.0 (training data: Ai4Privacy OpenPII, CC BY 4.0) | optional local model | +| [OpenMed PII Italian FastClinical Small (optional download)](https://huggingface.co/OpenMed/OpenMed-PII-Italian-FastClinical-Small-82M-v1-onnx-android) | Apache-2.0 | optional local model | +| [MiniLMv2 EN/IT NER (optional download)](https://huggingface.co/jkomyno/minilmv2-l6-h384-ner-en-it-q8) | MIT | optional local model by Alberto Schiabel (jkomyno) | +| [PiiScan (optional download)](https://huggingface.co/vectorsense/piiscan) | Apache-2.0 (training data: Ai4Privacy OpenPII, CC BY 4.0) | optional local model | +| [OpenMed PII Italian FastClinical Small (optional download)](https://huggingface.co/OpenMed/OpenMed-PII-Italian-FastClinical-Small-82M-v1-onnx-android) | Apache-2.0 | optional local model | | [onnxruntime](https://github.com/microsoft/onnxruntime/blob/main/LICENSE) | MIT | running local models | | [tokenizers](https://github.com/huggingface/tokenizers/blob/main/LICENSE) | Apache-2.0 | tokenizing for local models | | [cryptography](https://github.com/pyca/cryptography/blob/main/LICENSE) | Apache-2.0 or BSD-3-Clause | encryption | diff --git a/SECURITY.md b/SECURITY.md index 54bd5b9..1999867 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -16,7 +16,7 @@ NullifyPDF follows a **privacy-first architecture**: | Principle | Implementation | | --------------------------- | ------------------------------------------------- | | **100% Local** | All PDF processing happens on your machine | -| **No Cloud** | No file uploads; the only network access is the optional update check (Info โ†’ Controlla aggiornamenti, or at every start if the user enables it; off by default), which sends no document data | +| **No Cloud** | No file uploads; the only network access is the optional update check (Info โ†’ Controlla aggiornamenti, or at every start if the user enables it; off by default) and the model downloads the user approves (Italian spaCy model from the publisher's GitHub release, optional ONNX models from Hugging Face). None of them sends document data | | **No Telemetry** | Zero user tracking or analytics | | **Open Source** | Full code transparency, auditable by anyone | | **Privacy Export** | Irreversible redaction or encrypted reversible pseudonymization | @@ -40,7 +40,7 @@ When you export a PDF with redactions: 1. โœ… **Redactions applied** โ€” selected text/image areas are removed from the content stream, not just covered 2. โœ… **Sanitiser** โ€” annotations, embedded files, external links, bookmarks, JavaScript, tagged-PDF alternative text, thumbnails, document and XMP metadata, invisible text, text outside the visible page box and the trailer ID are removed; hidden layers are shown before the scan so their text is analysed 3. โœ… **Verification** โ€” every redacted word is checked glyph by glyph (the number of characters actually deleted must match the measured deletion rule, and characters a thin box cannot reach are reported), then the written file is re-opened and checked for any removed value in the text layer, every object and decompressed stream, annotations, attachments, bookmarks and metadata; findings list locations, never the values -4. โœ… **Pseudonymization maps encrypted** โ€” restore maps are stored separately and encrypted with a user password (scrypt key derivation, authenticated encryption) +4. โœ… **Pseudonymization maps encrypted** โ€” restore maps are stored separately and encrypted with AES-256-GCM under a data key wrapped by a password-derived key (Argon2id) and, optionally, a recovery key 5. โš™๏ธ **Optional width masking** โ€” the space left by removed text can be rounded so its width is not recoverable (off by default: it can shift the following text slightly) Limits you should know about: @@ -49,6 +49,10 @@ Limits you should know about: - Text drawn as part of an image is removed only where the image is redacted; OCR is used to find it. - Signatures, stamps and photos are not detected automatically; use "Oscura Immagini" or draw a redaction manually. +### Downloaded models + +Models are untrusted data until verified. The Italian spaCy pipeline is downloaded over HTTPS only after the user has seen its licence and agreed; its size and SHA-256 are pinned in the program, the archive is unpacked only into `~/.nullifypdf/spacy_models/` (paths that escape that folder are rejected) and an incomplete or altered download is discarded. Optional ONNX models come from pinned Hugging Face revisions and every file is checked for size and SHA-256 before it is installed. Models only produce suggestions; the user decides what is redacted. + ### Temporary Files - The working copy used during export is kept **in memory**. Only very large documents use a private per-user folder (`~/.nullifypdf/tmp`), emptied after the export and at the next start-up. @@ -84,7 +88,7 @@ NullifyPDF runs locally; the person or organisation using it is the controller o ### File Type Verification -- โœ… PDF files only (blocked: `.exe`, `.zip`, etc.) +- โœ… PDF and Word DOCX files only (blocked: `.exe`, `.zip`, etc.); DOCX macros, external links and embedded objects are never executed - โœ… Password-protected PDFs are blocked before processing - โœ… Encryption detection (blocks password-protected PDFs) - โœ… Extension and existence checks before opening PDFs @@ -92,7 +96,6 @@ NullifyPDF runs locally; the person or organisation using it is the controller o ### User Input Validation - โœ… Page number bounds checking -- โœ… Language selection validation (EN/IT/BOTH only) - โœ… File path sanitization - โœ… Type hints on all functions @@ -177,16 +180,12 @@ The codebase is open source and welcomes security audits: We use Python static analysis tools: ```bash -# Type checking -mypy NullifyPDF.py - -# Linting -pylint NullifyPDF.py - -# Security scanning -bandit NullifyPDF.py +# Type checking (blocking in CI; full file list in DEVELOPMENT.md) +mypy NullifyPDF.py pii_detection.py privacy_core.py ``` +GitHub code scanning (CodeQL) analyses every pull request to `main`, and Dependabot proposes dependency updates. + ### Testing Security-relevant tests cover: @@ -238,6 +237,7 @@ NullifyPDF uses trusted, actively-maintained libraries: | **pymupdf** | PDF manipulation | โœ… Actively maintained | | **presidio-analyzer** | PII detection | โœ… Maintained by Microsoft | | **spacy** | NLP engine | โœ… Actively maintained | +| **onnxruntime** | Local model inference | โœ… Actively maintained | | **cryptography** | Restore-map encryption | โœ… Actively maintained | ### Vulnerability Scanning @@ -245,6 +245,7 @@ NullifyPDF uses trusted, actively-maintained libraries: We monitor dependencies for CVEs: - GitHub Dependabot alerts enabled +- GitHub code scanning (CodeQL) on pull requests to `main` - Security updates applied promptly - Community reports welcomed @@ -299,5 +300,5 @@ For **security vulnerabilities only**: --- -*Last updated: 2026-11-10* +*Last updated: 2026-11-11* *โ† [Contributing](./CONTRIBUTING.md) | [Back to README โ†’](./README.md)* diff --git a/THIRD_PARTY_NOTICES.md b/THIRD_PARTY_NOTICES.md index 3a92a62..39825a2 100644 --- a/THIRD_PARTY_NOTICES.md +++ b/THIRD_PARTY_NOTICES.md @@ -41,6 +41,9 @@ the pinned versions in `requirements.txt` and the publishers' pages. | presidio-analyzer | MIT | | | spaCy, thinc | MIT | | | spaCy `it_core_news_md` 3.8.0 | CC BY-NC-SA 3.0 | **Not bundled.** Downloaded by the user on request after showing the licence (non-commercial, attribution, share-alike). Sources: UD Italian ISDT (CC BY-NC-SA 3.0), WikiNER (CC BY 4.0), fastText vectors (CC0). Publisher: Explosion, https://github.com/explosion/spacy-models | +| MiniLMv2 EN/IT NER (optional download) | MIT | By Alberto Schiabel (jkomyno), https://huggingface.co/jkomyno/minilmv2-l6-h384-ner-en-it-q8. Base model `nreimers/MiniLMv2-L6-H384-distilled-from-RoBERTa-Large` (MIT); the card asks for attribution to the author. | +| PiiScan (optional download) | Apache-2.0 | https://huggingface.co/vectorsense/piiscan. Trained on data derived from Ai4Privacy OpenPII (CC BY-4.0, Ai Suisse SA), which requires attribution. | +| OpenMed PII Italian FastClinical Small (optional download) | Apache-2.0 | https://huggingface.co/OpenMed/OpenMed-PII-Italian-FastClinical-Small-82M-v1-onnx-android. The model card does not state its training data or their licences; the project has not verified them. | | onnxruntime | MIT | | | tokenizers | Apache-2.0 | | | cryptography | Apache-2.0 or BSD-3-Clause | | diff --git a/TROUBLESHOOTING.md b/TROUBLESHOOTING.md index 6102577..cc1dd75 100644 --- a/TROUBLESHOOTING.md +++ b/TROUBLESHOOTING.md @@ -43,7 +43,6 @@ Text extraction from large PDFs is CPU-intensive. This is **normal**, not a cras **Speed it up:** - Close other applications (frees RAM) -- Use language selection (**EN** or **IT**, not **BOTH**) - Split document: Redact first 100 pages, then next 100 ### Still Not Working? @@ -55,6 +54,21 @@ If log isn't updating: --- +## ๐Ÿ‡ฎ๐Ÿ‡น The Italian Model Does Not Download + +### Symptom +On the first **Auto Redact** the download of the Italian model fails ("Download non riuscito"), or you are asked again every time. + +### Solution + +- A failed download usually means no connection, a proxy or a firewall blocking `github.com`. Check the message, then run Auto Redact again and choose **Accetta e scarica**. +- If you choose **Continua senza modello** the scan runs with rules and dictionaries only and the question comes back on the next Auto Redact; accept the download once and it is never asked again. +- The model is verified against a pinned size and SHA-256: a corrupted or altered file is discarded and nothing is installed. +- Offline machines: install `it_core_news_md` 3.8.0 in the Python environment (development setups), or copy the unpacked pipeline folder `it_core_news_md-3.8.0` (it contains `meta.json` and `ner/`) into `~/.nullifypdf/spacy_models/`. +- The model is for non-commercial use only (CC BY-NC-SA 3.0); the licence is shown before the download. + +--- + ## ๐Ÿ”ด Export Fails or Crashes ### Symptom @@ -149,7 +163,7 @@ Then try export again. Upload `~/.nullifypdf/logs/nullifypdf.log` to GitHub issu Names, emails, or other PII should be detected but aren't highlighted. ### Root Cause -- Wrong language selected +- The Italian model was not downloaded (rules and dictionaries only), or the document is not in Italian - Text is in an image and OCR is disabled or not configured - Unusual formatting - Text is handwritten @@ -436,7 +450,7 @@ When opening a GitHub issue, include: 1. **Environment:** - OS (Windows 10, macOS 13, Ubuntu 22.04, etc.) - Python version: `python --version` - - NullifyPDF version (see title bar) + - NullifyPDF version (see the Info window) 2. **What Happened:** - Exact error message or description @@ -459,7 +473,7 @@ The more details, the faster we can fix it! ๐Ÿš€ | --------------------------- | --------------------------------- | | Keep original PDF as backup | Delete original after export | | Test on copy first | Redact production file first time | -| Use correct language | Use BOTH if EN or IT works | +| Use Italian documents | Expect names in other languages | | Check logs when stuck | Restart without checking why | | Enable debug mode for bugs | Submit bugs without logs | | Save to common folder | Save to network drive (slow) | diff --git a/USER_GUIDE.md b/USER_GUIDE.md index 53a1ca0..08c3eb9 100644 --- a/USER_GUIDE.md +++ b/USER_GUIDE.md @@ -12,7 +12,7 @@ Your step-by-step guide to redacting sensitive data from PDFs safely and securel | ๐ŸŽฏ Step | โฑ๏ธ Time | What You'll Do | | ----------------- | ------- | ----------------------------- | | **1๏ธโƒฃ Load** | 30 sec | Open a PDF or Word DOCX file | -| **2๏ธโƒฃ Configure** | 30 sec | Select document language | +| **2๏ธโƒฃ Language** | 30 sec | Italian model (first use only) | | **3๏ธโƒฃ Auto-Redact** | 1-2 min | Run AI to find sensitive data | | **4๏ธโƒฃ Review** | 1-2 min | Manually fix any mistakes | | **5๏ธโƒฃ Blindfold** | 30 sec | Hide images (optional) | @@ -34,7 +34,7 @@ Your step-by-step guide to redacting sensitive data from PDFs safely and securel ### Optional local models -Open **Modelli locali** in the sidebar to view the reviewed shortlist. Models are optional and are not included in the installer. Before downloading, the dialog shows the Hugging Face source, license, pinned revision and total size. The progress bar reports the overall percentage across all model files. +Open **Modelli locali** in the sidebar to view the reviewed shortlist (MiniLMv2 EN/IT, PiiScan, OpenMed Italian FastClinical, plus experimental entries that cannot be downloaded). Models are optional and are not included in the installer. Before downloading, the dialog shows the Hugging Face source, license, pinned revision and total size. After you confirm, the **Scarica modello** button itself turns into a progress bar with the overall percentage across all model files; click it again to cancel. The list and the Close button stay locked until the download ends. This is separate from the Italian name-recognition model of Step 2, which is downloaded on the first Auto Redact. Only entries with a complete application manifest can be downloaded. The manifest fixes the revision, allowed files, expected sizes and SHA-256 values. Experimental entries without those checks remain unavailable. Installed models run locally and add suggestions for review; they never apply a redaction automatically. @@ -194,6 +194,10 @@ Your redaction preferences are saved automatically: โ”œโ”€โ”€ audit/ โ† Export records (counts and hashes, no values) โ”œโ”€โ”€ logs/ โ”‚ โ””โ”€โ”€ nullifypdf.log +โ”œโ”€โ”€ ai_entities.json โ† Data types chosen in the Auto Redact dialog +โ”œโ”€โ”€ spacy_models/ โ† Italian name-recognition model (downloaded after your consent) +โ”œโ”€โ”€ models/ โ† Optional local ONNX models you downloaded +โ””โ”€โ”€ tmp/ โ† Private temporary files for very large exports (emptied after use) ``` ### Editing the dictionaries @@ -216,7 +220,7 @@ same term to both. 1. **Backup Original** โ€” Always keep the original PDF before redacting 2. **Test on Copy** โ€” Redact a test copy first, verify results 3. **Verify Export** โ€” Open exported PDF to confirm redactions look correct -4. **Choose Language Carefully** โ€” Wrong language = missed detections +4. **Italian documents only** โ€” Other languages are not analysed, so names and places in them are missed 5. **Use Allowlist Sparingly** โ€” Only add words you're 100% sure aren't sensitive --- @@ -270,7 +274,7 @@ python3.13 NullifyPDF.py - **Encoding:** UTF-8 - **Location:** `~/.nullifypdf/logs/nullifypdf.log` -- **Rotation:** not currently implemented; remove old logs manually if needed +- **Rotation:** the file rotates at about 512 KB (3 older files kept) and logs older than 30 days are deleted at start-up --- @@ -289,7 +293,7 @@ python3.13 NullifyPDF.py **A:** Irreversible anonymization cannot be undone from the exported PDF. Pseudonymization can be restored using **"Ricostruisci PDF"** together with the encrypted restore map and its password (see [Step 7](#step-7%EF%B8%8F%E2%83%A3--reconstruct-a-pseudonymized-pdf) below). ### Q: Does NullifyPDF send data to the cloud? -**A:** **No.** PDF processing is local and no document data ever leaves your machine. The only network access is the optional update check (see below), and OCR data is downloaded at build time, not while processing your PDFs. +**A:** **No.** PDF processing is local and no document data ever leaves your machine. The only network access is the optional update check (see below) and the downloads you approve: the Italian name-recognition model (from the publisher's GitHub release) and the optional local models (from Hugging Face), each verified with a pinned SHA-256. OCR data is downloaded at build time, not while processing your PDFs. ### Q: How do I check for new versions? **A:** Open **Info** and click **Controlla aggiornamenti**. NullifyPDF asks the public GitHub releases of this repository whether a newer version exists (no PDF content or personal data is sent). If one does, a window shows the version, the link to the release, the **release notes** (taken from the matching section of `CHANGELOG.md`) and a **Download** button that saves the installer for your operating system into your system **Downloads** folder (the location configured in your OS), after checking its size and the SHA-256 published by GitHub. Nothing is installed automatically. On Linux distributions other than Debian/Ubuntu and Fedora/RHEL-based ones the window only offers the release link, where you can pick the file yourself. diff --git a/help_guide.py b/help_guide.py index 67cbf3d..63dc1eb 100644 --- a/help_guide.py +++ b/help_guide.py @@ -269,6 +269,7 @@ def widgets_without_tooltip(root: QWidget) -> List[str]:
  • Ogni modello deve avere una licenza, una revisione e un SHA-256 verificati.
  • Il programma non carica codice remoto, file pickle o modelli arbitrari.
  • Il download richiede consenso esplicito e i pesi vengono conservati nella cartella dell'utente, non nell'installer.
    • +
  • Dopo il consenso, il pulsante Scarica modello diventa una barra di avanzamento con la percentuale; un secondo clic annulla il download. Elenco e Close restano bloccati finchรฉ il download non termina.
  • La qualitร  dei modelli va misurata sul tipo di documento concreto: nessun modello garantisce anonimizzazione o copertura completa.
    • """), diff --git a/licences_info.py b/licences_info.py index 6686e2c..ff716f1 100644 --- a/licences_info.py +++ b/licences_info.py @@ -48,6 +48,26 @@ class Component(NamedTuple): Component("spaCy it_core_news_md", "CC BY-NC-SA 3.0 (solo uso non commerciale)", CC_NC_SA_URL, "modello italiano, scaricato dopo il consenso", "Italian model, downloaded after consent", "CC BY-NC-SA 3.0 (non-commercial use only)"), + Component("MiniLMv2 EN/IT NER (optional download)", "MIT", + "https://huggingface.co/jkomyno/minilmv2-l6-h384-ner-en-it-q8", + "modello locale opzionale, di Alberto Schiabel (jkomyno)", "optional local model by Alberto Schiabel (jkomyno)"), + Component("PiiScan (optional download)", "Apache-2.0 (dati: Ai4Privacy OpenPII, CC BY 4.0)", + "https://huggingface.co/vectorsense/piiscan", + "modello locale opzionale", "optional local model", + "Apache-2.0 (training data: Ai4Privacy OpenPII, CC BY 4.0)"), + Component("OpenMed PII Italian FastClinical Small (optional download)", "Apache-2.0", + "https://huggingface.co/OpenMed/OpenMed-PII-Italian-FastClinical-Small-82M-v1-onnx-android", + "modello locale opzionale", "optional local model"), + Component("MiniLMv2 EN/IT NER (optional download)", "MIT", + "https://huggingface.co/jkomyno/minilmv2-l6-h384-ner-en-it-q8", + "modello locale opzionale, di Alberto Schiabel (jkomyno)", "optional local model by Alberto Schiabel (jkomyno)"), + Component("PiiScan (optional download)", "Apache-2.0 (dati: Ai4Privacy OpenPII, CC BY 4.0)", + "https://huggingface.co/vectorsense/piiscan", + "modello locale opzionale", "optional local model", + "Apache-2.0 (training data: Ai4Privacy OpenPII, CC BY 4.0)"), + Component("OpenMed PII Italian FastClinical Small (optional download)", "Apache-2.0", + "https://huggingface.co/OpenMed/OpenMed-PII-Italian-FastClinical-Small-82M-v1-onnx-android", + "modello locale opzionale", "optional local model"), Component("onnxruntime", "MIT", "https://github.com/microsoft/onnxruntime/blob/main/LICENSE", "esecuzione dei modelli locali", "running local models"), Component("tokenizers", "Apache-2.0", "https://github.com/huggingface/tokenizers/blob/main/LICENSE",