Skip to content

ARCHITECTURE.md

Latest commit

 

History

History
342 lines (248 loc) · 19.2 KB

ARCHITECTURE.md

File metadata and controls

342 lines (248 loc) · 19.2 KB
Raw

🏗️ Architecture — NullifyPDF

This document explains how NullifyPDF is structured today: application layers, privacy export flow, OCR support, release packaging, and the main security boundaries that matter during maintenance.

Tip

Read this after the README if you want the technical view of how detection, redaction, pseudonymization, and packaging work together.

📋 Current Scope

NullifyPDF is a local-first desktop application for preparing PDFs before they are shared with third parties or uploaded to AI systems.

It supports two privacy export modes:

  • Irreversible anonymization — selected content is redacted and removed from the exported PDF
  • Reversible pseudonymization — selected content is replaced with stable placeholders, while original values are stored in a separate encrypted restore map. The same restore map can later be used to reconstruct the original values back into a pseudonymized PDF.

Note

Reversible mode is pseudonymization, not true anonymization, because the encrypted restore map can re-identify the original values.

🧩 High-Level Components

📄 File ✨ Responsibility
NullifyPDF.py Main PySide6 UI, PDF rendering, AI scan orchestration (including the Italian-model consent step), redaction, export, reconstruction, OCR integration, Info window
update_check.py Qt-free update logic: version comparison (SemVer, stable/beta channels), GitHub release lookup, per-OS asset selection, trusted-host download, startup-check preference (~/.nullifypdf/update_check.json). Only runs on user click or if the user enabled the startup check
pii_detection.py Qt-free AI detection helpers: word-box text index (neighbour-aware rectangle clamping, ligatures, letter-spaced and hyphenated words), offset-to-rect mapping and partial-match policy, per-type thresholds, pattern validation, evidence-based PERSON/LOCATION scoring, accepted/suggested states, allowlist by word identity, whole-word propagation, entity selection persistence
privacy_core.py Privacy mode enum, placeholder registry (random / sequential / generic / keyed styles), encrypted restore-map payloads (version 3, optional per-occurrence text style; scrypt, PBKDF2 maps still readable), password strength check, restore-map parsing
lexicon_it.py Italian lexicon: certificate/legal verb forms, medical and administrative words, given names, regions and provinces (used to reject ordinary words and to confirm names/places)
pdf_sanitize.py Export sanitiser (annotations, attachments, scripts, hidden layers/text, off-page text, metadata, optional TJ-gap quantisation) and verify_export()
export_engine.py Qt-free export pipeline (export_document), neutral file names, private temp handling, audit record, local-data wipe
review_dialog.py Qt dialog to accept or reject suggested detections
model_manager.py Pinned Hugging Face model catalog, SHA-256 verification, local model store and hardware recommendation
model_dialog.py Local ONNX model dialog: catalog, details, consent and a threaded download whose progress fills the download button (second click cancels; list and Close are locked meanwhile)
spacy_models.py The spaCy Italian pipeline (CC BY-NC-SA 3.0) downloaded on demand: consent text, pinned size and SHA-256, safe unpacking into ~/.nullifypdf/spacy_models/, blank-pipeline fallback
licences_info.py Licence of NullifyPDF and of every component and model with official links, used by the Info window and the README table
onnx_ner.py CPU ONNX token-classification adapter with character-offset mapping and label mapping
word_import.py Safe DOCX OOXML text/table extraction into an internal PyMuPDF document; no macro execution
help_guide.py In-app guide (Italian, F1) and the tooltip table applied to every widget
secure_store.py Master key protected by DPAPI / OS keyring / key file; AES-GCM sealing; atomic private writes
word_lists.py Encrypted blocklist/allowlist with expiry (PDFListManager)
dictionary_dialog.py Qt editor for both lists with dates and default expiry
lexicon_data_it.py Generated by scripts/build_lexicon_data.py from ISTAT open data (municipalities, provinces, regions) and, with --wikidata, Wikidata (given names, surnames); see THIRD_PARTY_NOTICES.md; imported lazily and used only to confirm names and places
PDF_Checker.py Heuristic post-export inspection helper
build_local.py PyInstaller build script
scripts/download_ocr_data.py Downloads EN/IT Tesseract tessdata_fast files bundled into every build
tests/ Unit, integration and offscreen GUI tests (see Tests)

🧠 Runtime Architecture

User
  |
  v
PySide6 UI (NullifyPDF)
  |
  |-- PDFView renders the current page with PyMuPDF pixmaps
  |-- Manual redaction draws pending redaction annotations
  |-- Export dialog selects anonymization or pseudonymization
  |
  v
AIWorker (QThread)
  |
  |-- Extracts digital text from pages
  |-- Runs OCR on scanned/image-only pages when enabled
  |-- Runs Presidio + spaCy entity detection
  |-- Emits detections back to UI thread
  |
  v
UI thread applies pending redaction annotations
  |
  v
Export pipeline writes a cleaned PDF copy

🏛️ Main Classes

🧱 Class 📍 Location 🎯 Purpose
PDFListManager word_lists.py Loads and saves the blocklist/allowlist as encrypted files (*.v2.enc) with per-entry expiry; migrates plaintext files
AIWorker NullifyPDF.py Background NLP/OCR worker that emits page-level detections
PDFView NullifyPDF.py Custom QGraphicsView for rendering pages, drawing rectangles, and zooming
NullifyPDF NullifyPDF.py Main window and application controller
PlaceholderRegistry privacy_core.py Creates stable placeholders such as PERSON_9F2A (random), PERSON_001 (sequential), DATA_9F2A (generic) or an HMAC-derived one (keyed)

🔄 Data Flow

📂 Load

  1. User opens or drops a PDF.
  2. NullifyPDF.load_path() validates the path and extension.
  3. PyMuPDF opens the document.
  4. Password-protected PDFs are rejected.
  5. The first page is rendered in PDFView.
  6. _warn_if_scanned_pdf() warns when pages look image-only and may need OCR.

🤖 AI Scan

  1. User clicks Auto Redact and, in a dialog, picks which data types to look for (names, locations, email, phone, IBAN, cards, crypto, fiscal code, VAT, licence, ID card, passport). The selection is remembered in ~/.nullifypdf/ai_entities.json.
  2. The language is always Italian (IT); the user optionally enables:
    • OCR PDF scansionati
    • Oscura Immagini
  3. cmd_auto_ai() calls _prepare_italian_model(): if the spaCy Italian pipeline is neither installed as a package nor in ~/.nullifypdf/spacy_models/, a dialog shows its licence and asks for consent (download, continue with rules only, or cancel); then it checks OCR configuration if OCR is enabled.
  4. AIWorker.run_scan() processes pages in a background thread; Presidio loads only the Italian pipeline (AIWorker.italian_model), and installed local ONNX models are loaded once per scan and add suggestions only.
  5. For each page:
    • the analysis text is built by pii_detection.build_page_index() from page.get_text("words"), recording each word's rectangle so character offsets map back to exact boxes. Each rectangle is trimmed against the words of neighbouring lines (a word box spans ascender to descender and would otherwise overlap them); ligatures are expanded, letter-spaced runs become one word and a word hyphenated across two lines is joined
    • if the page looks scanned, OCR is run with page.get_textpage_ocr() and the same index is built from the OCR words; on text pages with embedded images only the images are OCR'd and merged
    • Presidio analyzes the text for the enabled entity types; date of birth and health identifiers are found by context-anchored patterns
    • classify_page() filters the results: per-type thresholds; pattern hits are validated (token boundaries, context words, uppercase form, plausibility); NER spans are split at line breaks and names on consecutive lines are merged; ordinary Italian words are stripped from the edges of a span; fragments and OCR noise are rejected; field labels and list items are rejected; each remaining PERSON/LOCATION hit gets an evidence score and becomes accepted, suggested or is dropped (in health mode never dropped)
    • build_detections() resolves each candidate to words with the partial-match policy (NER hits must not be a fragment of a longer word; a checksum-valid identifier glued to other text is redacted whole), removes words protected by the allowlist (by word identity), and adds other occurrences as whole words only
    • detections are emitted with text, entity type, score, state, reasons, source and their rectangles
  6. apply_ai_to_page() runs on the UI thread: accepted detections become pending redaction annotations (a rectangle already covered by at least 60% is not added twice); suggested ones are queued and reviewed in ReviewDialog. Blocklist terms are matched as whole words.

🖊️ Manual Review

Redactions remain annotations until export. The original in-memory document is not flattened while the user is reviewing it.

Manual drawing over text:

  • adds a redaction annotation
  • records the clipped text in annotation metadata when available
  • adds the text to the blocklist

Clicking an existing redaction:

  • removes the pending annotation
  • adds the clipped text to the allowlist when available

🛡️ Privacy Export Pipeline

cmd_export() is the main export entry point.

  1. If suggestions are unreviewed, the user is asked to review them or continue without them.
  2. User chooses privacy mode:
    • irreversible removal
    • reversible pseudonymization
  3. User chooses the output path (default name contains nothing from the source name; a warning appears if the chosen name contains a removed value).
  4. For pseudonymization only:
    • user chooses the placeholder style (random, sequential, generic, keyed)
    • user chooses the .nullifypdf-map path (a warning appears for the PDF's folder or a synchronised folder)
    • user enters a password twice; it must pass a strength check (minimum 12 characters)
  5. export_engine.export_document() works on a copy of the document kept in memory (or, above a size threshold, in a private per-user temp folder):
    • links overlapping redactions are deleted
    • redactions are applied with PyMuPDF (image pixels blanked, not whole images)
    • widgets are removed
    • pdf_sanitize.sanitize_document() removes annotations, attachments, scripts, bookmarks, structure, metadata, hidden/invisible/off-page text and external links
    • the final PDF is saved with cleanup options
  6. Before the redactions are applied, every band that stores the whole box of its word is checked; after applying them, the number of glyphs that vanished from each line must match the measured deletion rule (a glyph is deleted when the rectangle reaches its box inset by 10%), and target glyphs the band cannot reach are counted. Any survivor is reported without the value.
  7. pdf_sanitize.verify_export() re-opens the written file and checks it; a failure is reported and the file can be deleted.
  8. For pseudonymization, the restore map is encrypted and written with owner-only permissions.
  9. An audit record (no values) is written to ~/.nullifypdf/audit/.

The original open document remains editable after export because the export works on a copy.

🔒 Anonymization Mode

Irreversible anonymization applies redaction annotations directly. The exported PDF does not contain a restore map.

This mode should be used when the output PDF must not contain recoverable personal data from the selected redaction areas.

Warning

Residual risk still exists for complex PDFs. NullifyPDF removes selected redacted content and common metadata, form, and link structures, but the output should still be reviewed before high-risk sharing.

🔐 Pseudonymization Mode

Reversible pseudonymization replaces each selected value with a placeholder.

Example placeholders:

PERSON_001
EMAIL_ADDRESS_001
IBAN_CODE_001
PHONE_NUMBER_001
DATA_001

The mapping is stored in a separate encrypted file:

document.nullifypdf-map

The restore map contains:

  • source filename
  • source SHA-256 when available
  • output SHA-256
  • placeholder entries
  • original values
  • entity types
  • page numbers
  • the redaction box of every occurrence
  • for native (non-scanned) PDFs, the typography of the redacted text: font, size, colour, flags and baseline (map version 3, optional)

The map is an envelope (version 3): a random data key encrypts the payload with AES-256-GCM and is wrapped by a key derived from the password with Argon2id (t=3, m=64 MiB, p=4; scrypt if Argon2id is unavailable) and, optionally, by a 256-bit recovery key. Maps created with scrypt + Fernet (version 2) or PBKDF2-HMAC-SHA256 (version 1) still open; the parameters read from a file are bounded before use. Each occurrence may carry the compact label that was drawn when the full placeholder did not fit the box (map version 4).

Important

The restore map is never embedded in the pseudonymized PDF.

🔎 OCR Architecture

OCR is implemented through PyMuPDF's integrated Tesseract support.

Runtime behavior:

  1. find_tessdata_dir() resolves Tesseract language data in this order:
    • TESSDATA_PREFIX
    • bundled ocr/tessdata via resource_path()
    • PyMuPDF's fitz.get_tessdata()
    • common platform locations
  2. ocr_language_for_choice() returns the Tesseract codes: ita, plus eng when its data is installed (so English words inside Italian documents are read well).
  3. AIWorker._page_needs_ocr() runs OCR only when text extraction is missing or very poor.
  4. OCR text is used for Presidio detection.
  5. OCR search rectangles are used to place redactions on scanned pages.

Every release bundles EN/IT OCR data. The documents analysed are Italian, so ita is the OCR language; eng is added only to read English words inside Italian pages.

🧠 Models

Model Where it comes from Role
spaCy it_core_news_md 3.8.0 Downloaded on first use after consent (spacy_models.py); never bundled, because its licence (CC BY-NC-SA 3.0) is non-commercial Statistical recognition of names and places through Presidio
ONNX models (MiniLMv2 EN/IT, PiiScan, OpenMed Italian FastClinical) Optional, downloaded from pinned Hugging Face revisions (model_manager.py), every file checked against size and SHA-256 Extra suggestions only; never applied automatically

The development environment installs the Italian spaCy package so the tests can run; the PyInstaller specs exclude it, together with pymupdf4llm and pymupdf.layout.

🧵 Threading Model

The UI thread owns rendering and document mutation.

The worker thread performs text extraction, OCR, and NLP analysis. Access to the shared PyMuPDF document is serialized with QMutex.

Important rules:

  • AIWorker should not mutate page annotations
  • apply_ai_to_page() runs on the UI thread
  • export happens after detection and review, not during the worker scan

This separation prevents UI freezes and reduces concurrent document-mutation risks.

📦 Build

Every build bundles OCR language data:

ocr/tessdata/eng.traineddata
ocr/tessdata/ita.traineddata

If the files are missing, build_local.py automatically downloads them from tesseract-ocr/tessdata_fast through scripts/download_ocr_data.py.

The .traineddata files are ignored by git and are not stored in the repository.

🚀 CI/CD

GitHub Actions build one artifact per supported OS:

  • Windows
  • macOS
  • Ubuntu / Debian (.deb)
  • Fedora / RHEL (.rpm)

Beta releases (v*-beta.N tags) and stable releases (vX.Y.Z tags) are each compiled from their own tag, because the application embeds its version string at build time. The stable workflow first checks that CHANGELOG.md has a [X.Y.Z] entry with an empty [Unreleased] section and that NullifyPDF.py declares that exact version with no prerelease suffix.

Linux packages are built by the reusable workflow build-linux.yml, shared by the stable, beta and test builds. The PyInstaller binary and the .rpm are built once inside a rockylinux:9 container (glibc 2.34, rpm 4.16), the .deb is wrapped from the same binary on ubuntu-24.04, and each package is then installed and started under Xvfb in clean Ubuntu 24.04, Ubuntu 26.04, Debian 13, Rocky 9, Rocky 10 and Fedora containers. A release is published only if this smoke test passes. Runners are pinned, so the move of ubuntu-latest to a new Ubuntu release does not change the build. When a distribution approaches end of support, update the matrix in build-linux.yml (and the list in the README).

Pull requests also run the unit tests against the latest PyMuPDF, a guard that only develop may be merged into main, project automation, and GitHub code scanning (CodeQL on pull requests to main).

Expected artifact names include:

NullifyPDF_vX.Y.Z_Windows.exe
NullifyPDF_vX.Y.Z_macOS.zip
NullifyPDF_vX.Y.Z_Ubuntu.deb
NullifyPDF_vX.Y.Z_Fedora.rpm

🧪 Tests

Current tests cover:

  • detection pipeline, geometry (rectangle clamping, glyph-deletion rule), scoring and the Italian lexicon (test_pii_detection.py, test_detection_pipeline.py, test_geometry.py)
  • redaction engine, export sanitiser and export verification (test_redaction_engine.py, test_export_sanitize.py, test_privacy_hardening.py)
  • placeholders, encrypted restore maps and reconstruction (test_privacy_core.py, test_reconstruct.py)
  • encrypted dictionaries and the key store (test_secure_store.py)
  • offscreen GUI flows: review, export, reconstruction, guide and tooltips, Info window, model dialog (test_gui_flow.py, test_gui_help.py, test_model_dialog.py)
  • model catalog, on-demand Italian model download, DOCX import, update check (test_model_manager.py, test_spacy_models.py, test_word_import.py, test_update_check.py)
  • licence consistency (no non-commercial component in the installer) and build configuration (test_licences.py, test_build_config.py, test_validation.py)

Run:

pytest tests/ -v

⚠️ Known Limits

  • Detection is probabilistic. Presidio, spaCy, and OCR can miss data or create false positives. Heuristics that reject label-like single-word names trade a little recall (for example a lone first name that starts a line) for precision; such values can be marked manually.
  • OCR depends on Tesseract language data and scan quality.
  • Only Italian documents are supported. Without the Italian spaCy model (download declined) only rules and dictionaries run and names are found less often.
  • Handwriting is not reliably supported.
  • Complex PDFs may contain structures not covered by automated cleanup.
  • Pseudonymized PDFs are still personal-data processing when the restore map exists.
  • Digital signatures are invalidated by privacy export.

Warning

For high-risk workflows, exported PDFs should be reviewed independently before sharing.

Last updated: 2026-11-11
Back to README | Development →