diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index d6a1523..1973c39 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -419,6 +419,8 @@ jobs: smoke-linux-beta: if: github.ref == 'refs/heads/develop' needs: [get-version-beta, build-linux-beta, build-rpm-beta] + permissions: + contents: read uses: ./.github/workflows/smoke-packages.yml with: expected_version: ${{ needs.get-version-beta.outputs.base_version }} @@ -810,6 +812,8 @@ jobs: smoke-linux-stable: if: github.ref == 'refs/heads/main' needs: [get-version, build-linux-stable, build-rpm-stable] + permissions: + contents: read uses: ./.github/workflows/smoke-packages.yml with: expected_version: ${{ needs.get-version.outputs.version }} diff --git a/CHANGELOG.md b/CHANGELOG.md index 9c0f311..959b9f9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,12 +9,108 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) — ### Roadmap -- [ ] **CSV/JSON Export** — Export metadata for the current file or folder - [ ] **Batch Edit** — Apply a field change to multiple selected files at once - [ ] **Search/Filter Bar** — Filter the file list by name or extension --- +## [0.5.0] — 2026-11-04 + +### Added +- **Exit button.** An **Exit** button at the right of the toolbar closes MetaLens: it stops the background + process that reads and writes the metadata (with everything it started), cancels a download in progress and + releases its memory and port. It waits while a save is in progress. (**File → Quit** does the same.) +- **Add a position.** The Batch dialog puts a latitude and longitude on the selected pictures, or looks up the + position of each picture by its capture time along a GPX track (with a camera-clock offset and a maximum gap); + pictures of formats that cannot hold a position are reported. GPS entries can now also be added by hand. +- **Legacy Office (DOC, XLS, PPT):** the document properties (author, company, manager, custom ones...) can be + removed in place, without changing the size of the file; Undo restores it exactly. +- **Canon CR3 and Fujifilm RAF** raw files: EXIF, GPS and the preview picture metadata are read. +- **PDF:** the XMP metadata stream is now shown and can be removed as a whole (its edit history and identifiers + were previously kept); **TIFF:** the IPTC record can be removed as a whole. +- **Other languages of XMP texts** (title, description...) are listed next to the main text (`xmp:dc:title@de`), + read only. **Camera maker data:** the serial number, owner name, lens and similar values stored in Canon, Nikon + and Sony camera notes are shown (`maker:*`, read only) and counted as device or identity data when preparing a + file for sharing; removing the camera note removes them. +- **Search by common names.** The names used by the search box are also served by the sidecar (`GET /aliases`). +- **Canon CR3 and Fujifilm RAF** files now say why they are read only. +- **Operations on several files.** A *Batch* button in the file list works on the selected files: prepare them + for sharing, shift their dates (minutes, hours or days, forwards or backwards), copy the metadata of the first + file to the others, rename from metadata (template such as `{exif:DateTimeOriginal|%Y%m%d}_{name}`) and, for the + folder, export the metadata of every file to CSV or JSON. Each action shows a preview first; every file is saved + and can be undone on its own. +- **Prepare for sharing.** A new *Share* tab removes what a file reveals before it is sent to someone: choose a + preset (location, personal data, everything shown) or individual categories, see exactly which fields will + be removed and which cannot be, then apply it in one step that can be undone (`POST /sanitize`). +- **Editing and adding metadata in TIFF, DNG and RAW (TIFF-based), HEIC/HEIF/AVIF and MP4/MOV files.** + EXIF entries and XMP properties can be changed or added (a longer value is stored at the end of the file and + the entry points to it; nothing that exists is moved, the picture data is verified unchanged and every change + can be undone). In videos the creation/modification dates, location, device, software and text entries can be + changed or created, and XMP can be edited or created; a longer value is stored by growing the movie atom + (the media data and its offsets are kept consistent). RAW support is experimental. +- **PNG and WebP can now be edited:** EXIF and XMP fields are written (WebP files gain the extended header + they need), PNG text chunks can be removed, the picture data is verified unchanged on every save (and the + save is refused otherwise) and every change can be undone. +- **Office documents (DOCX, XLSX, PPTX): application and custom properties** (company, manager, template, + custom fields...) can now be removed, one by one or with *Prepare for sharing*, and the change can be undone. +- **OpenDocument and EPUB files can be edited:** texts, dates and custom fields can be rewritten or added, any + field can be removed (an EPUB keeps its identifier, title and language) and the change can be undone; the rest + of the package is copied unchanged and signed documents stay untouched. +- **Matroska and WebM (MKV, WEBM, MKA):** the file's own fields are now read (title, muxer and writing + application, date, tags, attachments, tracks) instead of the generic reader, and the title, tags, dates and + attachments can be removed or, for texts, rewritten when they fit (also with *Prepare for sharing*). Nothing + is moved, the audio and video data is verified unchanged and every change can be undone byte for byte. +- **HEIC, HEIF and AVIF: removal of metadata.** EXIF, GPS and XMP can be removed in place (also with *Prepare + for sharing*); the image items are verified unchanged and the change can be undone byte for byte. +- **TIFF, DNG and TIFF-based RAW files: removal of metadata.** Camera, owner, software, dates, GPS and the other + EXIF entries (and XMP properties) can be removed without moving anything in the file: the removed values are + zeroed, the picture data and the maker note stay valid and unchanged (verified on every save), and the change + can be undone byte for byte. RAW support is marked experimental. Nothing can be added or edited in these files. +- **MP4, MOV and M4A: removal of structure and XMP.** Location, device, dates, text atoms and XMP of videos can + be removed (also with *Prepare for sharing*). Nothing in the file is moved and the audio/video data is + verified unchanged; the change can be undone byte for byte. +- **XMP data not shown** (edit history, document identifiers, derived-from...) is now listed as one field + (`xmp:@hidden`) and is removed together with the other personal data by *Prepare for sharing*. +- **Privacy indicators.** Every field is classified by what it can reveal when the file is shared (location, + identity, device, dates, software, thumbnail, free text). The metadata views show a badge on each sensitive + field and a line summarizing what sharing the file would reveal. The API exposes `sensitivity` per field and + `sensitivity_summary` per file. +- **Video and audio structure (MP4, MOV, M4A).** Brand, creation and modification dates, track list (type, + language, size, rotation, codec, frame rate, sample rate, channels), QuickTime keys (including the location + and device written by phones), text atoms and the XMP packet, all read-only (`qt:` fields). +- **HEIC, HEIF and AVIF** images: size, rotation, EXIF, GPS, XMP and ICC profile. +- **PNG:** XMP is now shown as regular `xmp:` fields; image header (bit depth, colour type, interlace), last + modification time and ICC profile. +- **TIFF and DNG:** embedded XMP, IPTC and ICC profile. +- **JPEG:** JFIF header, comments, Multi-Picture index and ICC profile (read-only). +- **RAW:** Olympus ORF and Panasonic RW2/RWL files are read, as well as more TIFF-based RAW extensions. +- **OpenDocument (ODT, ODS, ODP, ODG) and EPUB** metadata (read-only). +- **PDF:** version and linearization are shown. +- **Audio:** MPEG layer/version/channel mode/bitrate mode/encoder, FLAC stream details, encoder vendor of + Vorbis files; pictures and general objects show their type and description. +- **Files without an extension** are recognized from their content and handled like their format (a JPEG + named `photo` can now be read and edited like a `.jpg`). + +### Security +- The Batch operations (prepare for sharing, shift dates, copy tags, add a position) no longer return the text of + an internal error for a file that failed: each failed file carries its structured `errors` list only (the + `detail` field of those results is gone). +- GitHub Actions: the package smoke-test jobs now run with read-only repository permissions. +- Development dependency `minimatch` (used by the packaging tools) updated to 11.0.4 where the packager uses it. + +### Changed +- The About window is wider (and shows longer release notes), so it no longer looks narrow and tall when an update is available. +- Keys: `png:XML:com.adobe.xmp` is replaced by the `xmp:` fields it contains; PNG text in several languages + is `png:@` instead of `png:#2`; files without an extension use their format's + keys (e.g. `exif:*`) instead of `hachoir:*`; MKV and WebM files use `mkv:*` keys instead of `hachoir:*`. +- The temporary copy used while saving now has the same name as the file (inside the private temporary folder). +- A tag editor that cannot open a movie shows its structure read-only instead of reporting a read error. +- An Exif 3.0 UTF-8 text tag no longer makes the whole EXIF block non-rewritable. +- Code quality: circular imports between the image and HEIC handlers removed, unused code and empty error + handlers cleaned up, a few tests strengthened (they now compare the restored bytes after Undo). + +--- + ## [0.4.1] — 2026-11-03 This release contains no change to the application itself: it hardens the build and release pipeline diff --git a/README.md b/README.md index 1c5314c..28bcd8c 100644 --- a/README.md +++ b/README.md @@ -119,19 +119,31 @@ Explore the sidecar API in [API.md](docs/API.md). | Category | Formats | Read | Write | |---|---|---|---| -| Images | JPEG — EXIF/GPS, XMP, IPTC-IIM | ✅ | ✅ | -| Images | TIFF, PNG (text chunks, `eXIf`), WebP (EXIF, XMP), GIF (comment), BMP, ICO | ✅ | — | -| Images (RAW) | CR2, NEF, ARW, DNG, ORF, RW2, PEF, SRW (EXIF read for TIFF-based RAW) | ✅ | — | +| Images | JPEG — EXIF/GPS, XMP, IPTC-IIM (also JFIF header, comments, Multi-Picture index and ICC profile, read-only) | ✅ | ✅ | +| Images | PNG (text, `eXIf`, XMP, ICC, header), WebP (EXIF, XMP, ICC), TIFF (EXIF, XMP, IPTC, ICC) — EXIF/XMP can be edited and added, PNG text and the IPTC record of a TIFF removed | ✅ | ✅ | +| Images | GIF (comment), BMP, ICO | ✅ | — | +| Images | HEIC, HEIF, AVIF — size, rotation, EXIF/GPS, XMP, ICC (EXIF/XMP edited, added and removed in place) | ✅ | ✅ | +| Images (RAW) | CR2, NEF, ARW, DNG, ORF, RW2, RWL, PEF, SRW and other TIFF-based RAW (EXIF, XMP, IPTC) — changes are experimental and never move existing data | ✅ | ✅ | +| Images (RAW) | Canon CR3 and Fujifilm RAF (EXIF, GPS, preview metadata) | ✅ | — | | Audio | MP3, WAV, AIFF, TTA (ID3) · FLAC, OGG, OGA, Opus, SPX (Vorbis comments) · WMA (ASF) · APE, WV, MPC (APEv2) · M4A (MP4 atoms) | ✅ | ✅ | | Audio | AAC (ADTS/ADIF) | ✅ | — | -| Video | MP4, M4V, MOV, M4B, M4R (MP4 atoms) | ✅ | ✅ | -| Video | MKV, WebM, AVI, WMV, FLV, TS, MTS, M2TS (via hachoir) | ✅ | — | -| Documents | PDF — document info + XMP (not encrypted, not digitally signed) | ✅ | ✅ | -| Office | DOCX, XLSX, PPTX, DOCM, XLSM, PPTM — core properties (not digitally signed); `app.xml`/`custom.xml` read-only | ✅ | ✅ | -| Office (legacy) | DOC, XLS, PPT, DOT, XLT, POT (OLE property sets) | ✅ | — | +| Video | MP4, M4V, MOV, M4B, M4R (MP4 atoms; structure: brand, dates, tracks, rotation, codec, frame rate; dates, location, device and text entries and XMP can be edited, added and removed) | ✅ | ✅ | +| Video | MKV, WebM, MKA (title, tags, attachments, tracks; removed or rewritten in place) | ✅ | ✅ | +| Video | AVI, WMV, FLV, TS, MTS, M2TS (via hachoir) | ✅ | — | +| Documents | PDF — document info + XMP (the XMP stream can be removed as a whole; not encrypted, not digitally signed) | ✅ | ✅ | +| Office | DOCX, XLSX, PPTX, DOCM, XLSM, PPTM — core properties (not digitally signed); `app.xml`/`custom.xml` values can be removed | ✅ | ✅ | +| OpenDocument | ODT, ODS, ODP, ODG and templates (`meta.xml`) | ✅ | ✅ | +| E-books | EPUB (package metadata; identifier, title and language stay) | ✅ | ✅ | +| Office (legacy) | DOC, XLS, PPT, DOT, XLT, POT (OLE property sets; properties can be removed, not edited) | ✅ | 🗑️ remove | | Any file | Filesystem: `fs:mtime` writable; size, other timestamps, permissions/owner (POSIX), Windows attributes read-only; xattr (Linux: `user.` writable; macOS: read-only) | ✅ | ✅ (mtime, xattr) | | Other | Many more formats recognized from their content via hachoir | ✅ | — | +Files without an extension are recognized from their content. Every change is verified (the picture, audio or video data must come out identical) and can be undone. + +### 🛡️ Sharing files safely + +The **Share** tab removes what a file reveals before you send it (location, identity, device, dates, software, thumbnails, edit history), with a preview and Undo. The **Batch** button of the file list does the same on many files and also shifts dates, copies metadata between files, adds a position (fixed or from a GPX track), renames from metadata and exports a folder to CSV or JSON. + ### 🔑 File Integrity Compute cryptographic hashes on demand from the **Metadata** panel — no impact on browsing performance: @@ -246,5 +258,5 @@ Distributed with MIT License · [View License](LICENSE) --- -*Last updated: 2026-11-02* +*Last updated: 2026-11-03* *← [Requirements](docs/REQUIREMENTS.md) | [Docs →](docs/)* diff --git a/docs/API.md b/docs/API.md index 70ae6c9..e1c58a4 100644 --- a/docs/API.md +++ b/docs/API.md @@ -26,11 +26,13 @@ browser fallback is disabled. ## 📚 Table of Contents 1. [GET /health](#get-health) +2. [GET /aliases](#get-aliases) 2. [GET /list](#get-list) 3. [GET /read](#get-read) 4. [POST /write](#post-write) 5. [POST /delete](#post-delete) -6. [POST /undo](#post-undo) +6. [POST /sanitize](#post-sanitize) +7. [POST /undo](#post-undo) 7. [GET /diff](#get-diff) 8. [GET /hash](#get-hash) 9. [⚠️ Errors](#-errors) @@ -51,6 +53,17 @@ Health check + version, used by Electron to know the sidecar is ready (it checks --- +## GET `/aliases` + +Other names under which common fields are known (`"date taken"` for `exif:DateTimeOriginal`…), used by the search box. + +**Response** +```json +{ "aliases": { "exif:DateTimeOriginal": ["date taken", "capture date", "shot date", "create date"] } } +``` + +--- + ## GET `/list` Lists the contents of a directory with the detected format handler per file (directories first, then files, by name). @@ -168,6 +181,49 @@ Only fields present in the file with `deletable: true` can be removed; other key --- +## POST `/sanitize` + +Prepares a file for sharing: removes every deletable field whose [sensitivity](#metadatafield) category is chosen, +in one atomic save with one Undo token. + +**Body** +```json +{ "path": "/home/user/Pictures/photo.jpg", "categories": ["location", "device"], "preview": true, "keep": ["exif:Make"] } +``` + +`categories`: one or more of `location`, `identity`, `device`, `thumbnail`, `history`, `text`, `datetime`, `software` (unknown values: **422**). +`preview` (default `true`): only report, change nothing. `keep`: keys that stay in the file. + +**Response** +```json +{ "ok": true, "preview": false, + "will_remove": [{ "key": "gps:Latitude", "label": "Latitude", "category": "location" }], + "removed": [{ "key": "gps:Latitude", "label": "Latitude", "category": "location" }], + "remaining": [{ "key": "exif:Artist", "label": "Artist", "category": "identity", "reason": "..." }], + "undo_token": "Zt8x…", "undo_unavailable": null, "warnings": [] } +``` + +`remaining` lists the sensitive fields of the chosen categories still in the file (not removable for this kind of file, or +still present after the save). `removed` is only present when `preview` is `false`. + +--- + +## Operations on several files + +`POST /sanitize-many` `{paths, categories, preview, keep}` · `POST /shift-dates` `{paths, seconds, keys?, preview}` · +`POST /copy-tags` `{source, targets, keys?, preview}` · `POST /rename` `{path, template, preview}` · +`GET /export?path=&format=json|csv&recursive=false` · `POST /geotag` `{paths, latitude, longitude, altitude?, gpx?, utc_offset_minutes, max_gap_seconds, preview}` +(fixed coordinates, or the position looked up along a GPX track by the capture time of each picture). + +The first three answer `{ok, preview, results: [{path, ok, ..., undo_token}], undo_tokens}`: every file is its own +verified save with its own Undo token, a failing file does not stop the others (they are not atomic across files). +A failed file is `{path, ok: false, status, errors: [{code, key, message}]}` (no `detail`; the UI joins the messages). +At most 500 files per request (2000 for the export). `preview` defaults to `true` and changes nothing. +The rename template accepts `{name}`, `{ext}` and `{key}` / `{key|strftime}` for date-time keys, never replaces an existing +file (409) and cannot be undone. CSV cells that start with a formula character are prefixed with `'`. + +--- + ## POST `/undo` Reverts one earlier `/write` or `/delete` exactly, using the raw pre-images kept by the sidecar. @@ -304,7 +360,8 @@ Defined in `python/core/models.py`. | `length` | int \| null | Fixed list size for tuples (e.g. track `[3, 10]`, GPS coordinate of 3 rationals) | | `choices` | array | Allowed values for `enum` fields; always present, empty otherwise | | `binary` | object \| array \| null | For `binary` fields: the `{size, sha256, mime}` descriptor (a list when `multi`) | -| `source` | string | Group shown in the UI: `exif`, `gps`, `xmp`, `iptc`, `image`, `png`, `gif`, `id3`, `vorbis`, `flac`, `asf`, `ape`, `mp4`, `audio`, `video`, `pdf`, `office`, `ole`, `hachoir`, `filesystem`, `xattr` | +| `sensitivity` | string \| null | What the field can reveal when the file is shared: `location`, `identity`, `device`, `thumbnail`, `history`, `text`, `datetime`, `software`; `null` for technical fields. Informative only: it never changes `editable`/`deletable` | +| `source` | string | Group shown in the UI: `exif`, `gps`, `xmp`, `iptc`, `image`, `png`, `gif`, `id3`, `vorbis`, `flac`, `asf`, `ape`, `mp4`, `qt`, `mkv`, `heif`, `jfif`, `jpeg`, `mpf`, `icc`, `audio`, `video`, `pdf`, `office`, `odf`, `epub`, `ole`, `hachoir`, `filesystem`, `xattr` | ### `MetadataRecord` | Field | Type | Notes | @@ -322,6 +379,7 @@ Defined in `python/core/models.py`. | `supports_write` | bool | Legacy: format **or** filesystem level writable | | `supports_delete` | bool | Legacy: format **or** filesystem level deletable | | `addable_fields` | `MetadataField[]` | Schemas (value `null`) of fields that can be added with `/write` | +| `sensitivity_summary` | object | Number of fields per sensitivity category, most serious first (categories without fields are omitted) | If a handler ever produces two fields with the same key, the later ones are renamed `key#2`, `key#3`…, made read-only and reported in `warnings`. @@ -371,26 +429,37 @@ persisted anywhere (the Undo journal is in memory), so they may change between v | Namespace | Handler / formats | Native id | Writable | |---|---|---|---| -| `img:` | image (all) | `format`, `mode`, `width`, `height`, `dpi` (`[x, y]`, from JFIF/EXIF/Pillow) | no (computed) | -| `exif:` | JPEG, TIFF, TIFF-based RAW (e.g. CR2, NEF, ARW, DNG), PNG `eXIf`, WebP `EXIF` | tag name (`exif:Artist`, `exif:DateTimeOriginal`); a tag outside its usual IFD is `exif::`; unnamed tags `exif:tag_` | JPEG only; image-structure tags, pointers, the thumbnail and `UNDEFINED` binaries (e.g. `MakerNote`) are read-only, binaries deletable | +| `img:` | image (all) | `format`, `mode`, `width`, `height`, `dpi` (`[x, y]`, from JFIF/EXIF/Pillow); PNG also `bit_depth`, `color_type`, `interlace` | no (computed) | +| `exif:` | JPEG, PNG `eXIf`, WebP `EXIF` (editable); TIFF, TIFF-based RAW (entries can be changed, added or deleted; nothing existing is moved) (e.g. CR2, NEF, ARW, DNG, ORF, RW2), PNG `eXIf`, WebP `EXIF` | tag name (`exif:Artist`, `exif:DateTimeOriginal`); a tag outside its usual IFD is `exif::`; unnamed tags `exif:tag_` | JPEG only; image-structure tags, pointers, the thumbnail and `UNDEFINED` binaries (e.g. `MakerNote`) are read-only, binaries deletable | | `gps:` | as `exif:` (GPS IFD) | tag name without the `GPS` prefix (`gps:Latitude`, `gps:LatitudeRef`) | JPEG only | -| `xmp:` | JPEG (APP1 XMP), WebP `XMP ` | `xmp::` with canonical prefixes (`xmp:dc:title`, `xmp:dc:subject`, `xmp:xmp:Rating`, `xmp:photoshop:City`, `xmp:Iptc4xmpCore:Location`…) | JPEG only, for simple values, language alternatives (`x-default`) and Bag/Seq of simple items; structs and qualifiers are not shown; extended XMP is read-only | -| `iptc:` | JPEG (APP13 `Photoshop 3.0`, resource 0x0404) | mapped IIM name (`iptc:Keywords`, `iptc:By-line`, `iptc:Caption-Abstract`…) or `iptc::` | JPEG only; repeatable datasets are lists; written as UTF-8 (`1:90 = ESC % G`); a legacy (non-UTF-8) charset makes IPTC read-only; an edit or delete of a dataset is mirrored to its XMP twin (and vice versa) when the twin is already in the file: Keywords↔`dc:subject`, By-line↔`dc:creator`, ObjectName↔`dc:title`, Caption-Abstract↔`dc:description`, CopyrightNotice↔`dc:rights`, Headline, City, Province-State, Country, Credit, Source, SpecialInstructions, TransmissionReference, Sub-location, Country code ↔ the matching `photoshop:`/`Iptc4xmpCore:` property. The mirror is part of the same transaction and Undo, never creates a twin, never overrides a key sent in the same request, and a value the twin cannot store is skipped with a `warnings` note | -| `png:` | PNG `tEXt`/`zTXt`/`iTXt` | keyword | no | +| `xmp:` | JPEG (APP1 XMP), WebP `XMP `, PNG (`XML:com.adobe.xmp` iTXt), TIFF/DNG (tag 700), MP4/MOV (XMP `uuid`) | `xmp::` with canonical prefixes (`xmp:dc:title`, `xmp:dc:subject`, `xmp:xmp:Rating`, `xmp:photoshop:City`, `xmp:Iptc4xmpCore:Location`…) | JPEG, PNG and WebP, for simple values, language alternatives (`x-default`; the other languages are listed read only as `@`, e.g. `xmp:dc:title@de`) and Bag/Seq of simple items; structs and qualifiers are not shown individually: `xmp:@hidden` lists them (computed, removable as a whole); extended XMP is read-only | +| `iptc:` | JPEG (APP13 `Photoshop 3.0`, resource 0x0404), TIFF/DNG (tag 33723, read-only) | mapped IIM name (`iptc:Keywords`, `iptc:By-line`, `iptc:Caption-Abstract`…) or `iptc::` | JPEG only; repeatable datasets are lists; written as UTF-8 (`1:90 = ESC % G`); a legacy (non-UTF-8) charset makes IPTC read-only; an edit or delete of a dataset is mirrored to its XMP twin (and vice versa) when the twin is already in the file: Keywords↔`dc:subject`, By-line↔`dc:creator`, ObjectName↔`dc:title`, Caption-Abstract↔`dc:description`, CopyrightNotice↔`dc:rights`, Headline, City, Province-State, Country, Credit, Source, SpecialInstructions, TransmissionReference, Sub-location, Country code ↔ the matching `photoshop:`/`Iptc4xmpCore:` property. The mirror is part of the same transaction and Undo, never creates a twin, never overrides a key sent in the same request, and a value the twin cannot store is skipped with a `warnings` note | +| `heif:` | HEIC / HEIF / AVIF (`.heic .heif .hif .avif`; EXIF, XMP and ICC are also served as `exif:`, `gps:`, `xmp:`, `icc:`) | `MajorBrand`, `CompatibleBrands`, `Rotation` | no (read-only; the EXIF / GPS / XMP fields of these files can be deleted in place) | +| `jfif:` | JPEG (APP0 JFIF) | `Version`, `ResolutionUnit`, `XResolution`, `YResolution` | no | +| `jpeg:` | JPEG | `Comment` (COM segments; a list when there are several) | no | +| `maker:` | JPEG/TIFF/RAW EXIF with a Canon, Nikon or Sony camera note | `maker::` (`maker:canon:OwnerName`, `maker:nikon:SerialNumber`…) | no (computed; removing `exif:MakerNote` removes them) | +| `mpf:` | JPEG with a Multi-Picture index (MPO) | `NumberOfImages`, `image:Type`, `image:Size` (computed) | no | +| `icc:` | JPEG (APP2, split profiles are joined), PNG `iCCP`, TIFF/DNG (tag 34675), WebP `ICCP` | `Version`, `ProfileClass`, `ColorSpace`, `ConnectionSpace`, `CreationDate`, `Platform`, `RenderingIntent`, `Description`, `Copyright` | no | +| `png:` | PNG `tEXt`/`zTXt`/`iTXt` | keyword; an `iTXt` with a language tag is `keyword@language` | no | +| `pngchunk:` | PNG chunks that are not text | `pngchunk:tIME` (last modification, UTC) | no | | `gif:` | GIF | `comment` | no | | `tag:` | ID3 (MP3, WAV, AIFF, TTA), Vorbis comments (FLAC, Ogg Vorbis/Opus/Speex/FLAC), ASF (WMA), APEv2 (APE, WavPack, Musepack) | ID3 FrameID (`tag:TPE1`, `tag:TXXX:`, `tag:COMM::`, URL frames grouped by FrameID); lowercase Vorbis name (`tag:artist`); ASF attribute name; APE item key | yes, except AAC and other read-only formats; pictures, private and structured frames (APIC, GEOB, PRIV, POPM, PCNT, SYLT…) are read-only but deletable | | `flac:` | FLAC | `picture` (picture blocks) | read-only, deletable | +| `qt:` | MP4 family: M4A/M4B/M4R, MP4/M4V/MOV | `qt:MajorBrand`, `qt:CompatibleBrands`, `qt:CreateDate`/`qt:ModifyDate` (movie header, UTC), `qt:TimeScale`, per track by track ID `qt:track:HandlerType/CreateDate/Language/MediaTimeScale/Width/Height/Rotation/CodecID/Channels/SampleRate/FrameRate`, QuickTime keys `qt:mdta:` (e.g. `com.apple.quicktime.location.ISO6709`), `qt:udta:` | dates (`CreateDate`, `ModifyDate`, per track), `udta` text atoms and text `mdta` keys can be changed, added and deleted; the other values are read-only | | `mp4:` | MP4 family: M4A/M4B/M4R (audio), MP4/M4V/MOV (video) | `ilst` atom (`mp4:©ART`, `mp4:trkn`, `mp4:covr`, freeform `mp4:----::`) | yes; `covr` and non-UTF-8 freeform values read-only but deletable | | `audio:` | audio files, first audio track of MP4 | `duration`, `bitrate`, `sample_rate`, `channels`, `bits_per_sample`, `codec` | no (computed) | | `video:` | MP4 family | `duration` | no (computed) | -| `pdf:` | PDF | `/Info` entry without the slash (`pdf:Title`, `pdf:CreationDate`); computed `pdf:page_count`, `pdf:encrypted`, `pdf:signed` (an `/Info` entry with one of those names is `pdf:#info`) | text and date entries, unless the PDF is encrypted or signed; `/Trapped`, numbers and booleans read-only but deletable | -| `office:` | DOCX/XLSX/PPTX/DOCM/XLSM/PPTM | core property (`office:title`, `office:author`, `office:created`…); `office:app.` (docProps/app.xml), `office:custom.` (docProps/custom.xml) | core properties only, unless the package is signed; `app.`/`custom.` read-only | -| `ole:` | DOC/XLS/PPT/DOT/XLT/POT | SummaryInformation and DocumentSummaryInformation property (`ole:author`, `ole:company`…), user-defined `ole:custom:` | no | -| `hachoir:` | fallback for other files; MKV/WebM/AVI/WMV/FLV/TS/MTS/M2TS (video handler) | `hachoir::` (group 0 = root metadata, then one group per stream) | no | +| `pdf:` | PDF (`pdf:@xmp` = the XMP metadata stream, computed, removable as a whole) | `/Info` entry without the slash (`pdf:Title`, `pdf:CreationDate`); computed `pdf:page_count`, `pdf:encrypted`, `pdf:signed`, `pdf:PDFVersion`, `pdf:Linearized` (an `/Info` entry with one of those names is `pdf:#info`) | text and date entries, unless the PDF is encrypted or signed; `/Trapped`, numbers and booleans read-only but deletable | +| `office:` | DOCX/XLSX/PPTX/DOCM/XLSM/PPTM | core property (`office:title`, `office:author`, `office:created`…); `office:app.` (docProps/app.xml), `office:custom.` (docProps/custom.xml) | core properties editable unless the package is signed; `app.`/`custom.` values are read-only but can be deleted | +| `odf:` | OpenDocument (`.odt .ods .odp .odg` and templates) | `title`, `subject`, `description`, `initial-creator` (author), `creator` (last modified by), `keyword`, dates, `editing-cycles`, `editing-duration`, `stat.`, `custom.` | no (read-only) | +| `epub:` | EPUB | `version`, `title`, `creator`, `language`, `identifier`, `publisher`, `date`, `subject`, `description`, `rights`, `modified`… | no (read-only) | +| `ole:` | DOC/XLS/PPT/DOT/XLT/POT | SummaryInformation and DocumentSummaryInformation property (`ole:author`, `ole:company`…), user-defined `ole:custom:` | read-only, but deletable (the property is emptied in place; the file keeps its size) | +| `mkv:` | Matroska / WebM (`.mkv .webm .mka`) | `mkv:DocType`, `mkv:Info:Title/MuxingApp/WritingApp/DateUTC/Duration`, `mkv:tag:`, `mkv:attachment:`, `mkv:track:Type/CodecID/Language/Name` | titles, apps and tag texts can be rewritten when they fit; those and the date, tags and attachments can be deleted (in place) | +| `hachoir:` | fallback for other files; AVI/WMV/FLV/TS/MTS/M2TS (video handler) | `hachoir::` (group 0 = root metadata, then one group per stream) | no | | `fs:` | every file | `size`, `mtime`, `ctime` (creation time on Windows, metadata change on POSIX), `atime`, `win_attrs` (Windows), `permissions`/`owner`/`group` (POSIX) | `fs:mtime` only | | `xattr:` | every file (Linux, macOS) | attribute name (`xattr:user.comment`) | Linux `user.` attributes with UTF-8 values (non-UTF-8 values binary, read-only, deletable); other namespaces and macOS read-only | --- -*Last updated: 2026-11-02* +*Last updated: 2026-11-04* *← [Usage](./USAGE.md) | [Back to README →](../README.md)* diff --git a/docs/USAGE.md b/docs/USAGE.md index 26a225a..9a73dd7 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -14,10 +14,12 @@ Learn how to browse files, inspect and edit metadata, compute hashes, and compar 3. [📄 Select a File](#-select-a-file) 4. [👁️ View Metadata](#-view-metadata) 5. [✏️ Edit Metadata](#-edit-metadata) -6. [🗑️ Delete Metadata Fields](#-delete-metadata-fields) -7. [🔑 Compute File Hashes](#-compute-file-hashes) -8. [⚖️ Compare Two Files (Diff)](#-compare-two-files-diff) -9. [↩️ Undo](#-undo) +6. [🛡️ Prepare a File for Sharing](#-prepare-a-file-for-sharing) +7. [🗑️ Delete Metadata Fields](#-delete-metadata-fields) +8. [🔑 Compute File Hashes](#-compute-file-hashes) +9. [⚖️ Compare Two Files (Diff)](#-compare-two-files-diff) +10. [↩️ Undo](#-undo) +11. [🚪 Exit MetaLens](#-exit-metalens) --- @@ -47,6 +49,7 @@ The **center panel** (`FilePanel`) shows every file in the currently selected fo The **detail panel** (`DetailPanel`) opens on the **Metadata** tab by default, showing a read-only table (`MetadataTable`) of every field MetaLens extracted, grouped by source (`exif`, `gps`, `xmp`, `iptc`, `id3`, `vorbis`, `mp4`, `pdf`, `office`, `hachoir`, `filesystem`, …). - Multi-valued fields (artists, keywords, …) are shown as lists, never joined into one string. +- A line above the table tells what sharing the file would reveal (location, identity, device, dates...), and each sensitive field carries a badge with its category. - Binary values (cover art, pictures, MakerNote, thumbnails) are shown as size, MIME type and SHA-256, not as their bytes. - Notices above the table explain why the format metadata is read-only (e.g. signed PDF, AAC, legacy Office), report read errors and show warnings (e.g. an EXIF MakerNote that is moved on rewrite, duplicated metadata segments). @@ -63,6 +66,18 @@ Switch to the **Edit** tab to open `MetadataEditor`: --- +## 🛡️ Prepare a File for Sharing + +The **Share** tab removes what a file reveals before you send it to someone. + +1. Choose a preset (*Remove location*, *Remove personal data*, *Remove everything shown*) or tick the categories: location, identity, device, thumbnail, history (edit history and other data that is stored but not shown), free text, dates, software. +2. The preview lists the fields that will be removed and those that cannot be removed from this kind of file, with the reason. +3. *Remove* saves once; the change can be undone. + +The **Batch** button of the file list (select files with Ctrl/Shift) applies the same to many files and also: shifts dates, copies metadata of the first selected file to the others, adds a position (latitude and longitude, or the position along a GPX track by capture time), renames from metadata and exports the metadata of the folder to CSV or JSON. Every action previews first and every file is saved and undone on its own. The search box above the metadata table finds fields by name, another common name or value. + +--- + ## 🗑️ Delete Metadata Fields Individual fields marked as **deletable** can be removed from the Edit tab, including binary ones such as cover art (which cannot be edited). Deletions are part of the same verified atomic save as edits. Computed values (image size, stream info, file size, timestamps) cannot be deleted. @@ -107,6 +122,12 @@ MetaLens keeps an undo stack of your last **50** saves for the current session. --- +## 🚪 Exit MetaLens + +Click **Exit** at the right of the toolbar (or **File → Quit**, or close the window). MetaLens stops its background metadata process together with everything that process started, cancels any update download in progress and then closes, so no process or network port is left behind and the memory is released. The **Exit** button is disabled while a save is in progress. The undo stack is not kept: after you exit, saves can no longer be undone. + +--- + ## ℹ️ Checking Your Version Open **Help → About** to see the running app version (the packaged Electron app version, including any `-beta.N` suffix) — useful when reporting a bug. The dialog also links to the project repository on GitHub. @@ -129,5 +150,5 @@ The preferences (auto-check on/off, ignored version) are stored in `settings.jso --- -*Last updated: 2026-11-02* +*Last updated: 2026-11-04* *← [Installation](./INSTALLATION.md) | [API Reference →](./API.md)* diff --git a/electron/main.js b/electron/main.js index 2cfe8c9..e0f2d88 100644 --- a/electron/main.js +++ b/electron/main.js @@ -9,6 +9,7 @@ const { isTrustedFrame, secureWindow } = require('./security-policy') const { waitForSidecar } = require('./sidecar-health') const { handleSquirrelEvent } = require('./squirrel-events') const { spawnTree, killProcessTree } = require('./process-tree') +const { createShutdown } = require('./shutdown') const { buildMenuTemplate } = require('./menu') const { REPO_URL, checkForUpdates, downloadAsset, cleanupStaleDownloads, createSettingsStore, @@ -103,6 +104,13 @@ async function startSidecar() { } } +const shutdown = createShutdown({ + app, + stopSidecar: () => stopSidecar(), + cancelDownload: () => installAbort?.abort(), + closeWindows: () => BrowserWindow.getAllWindows().forEach(win => win.destroy()), +}) + function stopSidecar() { if (sidecarProcess) { killProcessTree(sidecarProcess) @@ -197,6 +205,9 @@ function setupIPC() { handle('get-app-version', () => app.getVersion()) + // Exit button: stop everything the app started, then quit (the renderer passes nothing) + handle('quit-app', () => { shutdown.quit() }) + handle('open-repo', () => shell.openExternal(REPO_URL)) handle('updates-get-settings', () => settingsStore.get()) handle('updates-set-settings', (_, patch) => settingsStore.update(patch)) @@ -306,4 +317,6 @@ app.on('window-all-closed', () => { if (process.platform !== 'darwin') app.quit() }) +app.on('before-quit', stopSidecar) app.on('will-quit', stopSidecar) +process.on('exit', stopSidecar) // last resort: never leave the sidecar running after the app is gone diff --git a/electron/package-lock.json b/electron/package-lock.json index f532f64..59ef4ae 100644 --- a/electron/package-lock.json +++ b/electron/package-lock.json @@ -1,12 +1,12 @@ { "name": "metalens", - "version": "0.4.1", + "version": "0.5.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "metalens", - "version": "0.4.1", + "version": "0.5.0", "license": "MIT", "devDependencies": { "@electron-forge/cli": "^8.0.1", @@ -458,16 +458,17 @@ } }, "node_modules/@electron/packager/node_modules/minimatch": { - "version": "10.2.6", - "resolved": "https://registry.npmjs.org/minimatch/-/minimatch-10.2.6.tgz", - "integrity": "sha512-vpLQEs+VLCr1nU0BXS07maYoFwlDAH0gngQuuttxIwutDFEMHq2blX+8vpgxDdK3J1PwjCJiep77OitTZ4Ll1A==", + "version": "11.0.4", + "resolved": "https://registry.npmjs.org/minimatch/-/minimatch-11.0.4.tgz", + "integrity": "sha512-/UJ1dEoVQTjPNDh6SGlUo+ueLMmoHTas4q+8yaAR/5t6nLAIiBDJtoYUh20IYleix7ehiQkgczb84PE3uzHkxg==", "dev": true, "license": "BlueOak-1.0.0", "dependencies": { - "brace-expansion": "^5.0.8" + "@isaacs/nonrecursive-glob-match": "^1.0.3", + "brace-expansion": "^5.0.12" }, "engines": { - "node": "18 || 20 || >=22" + "node": "20 || >=22" }, "funding": { "url": "https://github.com/sponsors/isaacs" @@ -545,16 +546,17 @@ } }, "node_modules/@electron/universal/node_modules/minimatch": { - "version": "10.2.6", - "resolved": "https://registry.npmjs.org/minimatch/-/minimatch-10.2.6.tgz", - "integrity": "sha512-vpLQEs+VLCr1nU0BXS07maYoFwlDAH0gngQuuttxIwutDFEMHq2blX+8vpgxDdK3J1PwjCJiep77OitTZ4Ll1A==", + "version": "11.0.4", + "resolved": "https://registry.npmjs.org/minimatch/-/minimatch-11.0.4.tgz", + "integrity": "sha512-/UJ1dEoVQTjPNDh6SGlUo+ueLMmoHTas4q+8yaAR/5t6nLAIiBDJtoYUh20IYleix7ehiQkgczb84PE3uzHkxg==", "dev": true, "license": "BlueOak-1.0.0", "dependencies": { - "brace-expansion": "^5.0.8" + "@isaacs/nonrecursive-glob-match": "^1.0.3", + "brace-expansion": "^5.0.12" }, "engines": { - "node": "18 || 20 || >=22" + "node": "20 || >=22" }, "funding": { "url": "https://github.com/sponsors/isaacs" @@ -610,6 +612,13 @@ "node": ">=18.0.0" } }, + "node_modules/@isaacs/nonrecursive-glob-match": { + "version": "1.0.3", + "resolved": "https://registry.npmjs.org/@isaacs/nonrecursive-glob-match/-/nonrecursive-glob-match-1.0.3.tgz", + "integrity": "sha512-SI7kyKHq4XXDRozw+r9GlRZkia1ZkFrpLW98r2IC2Bpr8YkFGE3fvjzmxCG0x78A++69HmDNrDvSMYBZmHGpKA==", + "dev": true, + "license": "BlueOak-1.0.0" + }, "node_modules/@malept/cross-spawn-promise": { "version": "2.0.0", "resolved": "https://registry.npmjs.org/@malept/cross-spawn-promise/-/cross-spawn-promise-2.0.0.tgz", diff --git a/electron/package.json b/electron/package.json index 1954856..13fda85 100644 --- a/electron/package.json +++ b/electron/package.json @@ -1,6 +1,6 @@ { "name": "metalens", - "version": "0.4.1", + "version": "0.5.0", "description": "Universal File Metadata Manager", "author": "Graziano Mariella", "license": "MIT", @@ -17,5 +17,10 @@ "@electron-forge/maker-squirrel": "^8.0.1", "@electron-forge/maker-zip": "^8.0.1", "electron": "^45.1.0" + }, + "overrides": { + "@electron/asar@4": { + "minimatch": "^11.0.4" + } } } diff --git a/electron/preload.js b/electron/preload.js index bd5773e..8d445fa 100644 --- a/electron/preload.js +++ b/electron/preload.js @@ -11,6 +11,9 @@ contextBridge.exposeInMainWorld('electronAPI', { getPort: () => ipcRenderer.invoke('get-port'), getSidecarConnection: () => ipcRenderer.invoke('get-sidecar-connection'), + // Exit button: stops the sidecar and quits the whole app + quitApp: () => ipcRenderer.invoke('quit-app'), + // Packaged app version (reflects beta suffix, unlike the sidecar's /health version) getAppVersion: () => ipcRenderer.invoke('get-app-version'), diff --git a/electron/shutdown.js b/electron/shutdown.js new file mode 100644 index 0000000..094458c --- /dev/null +++ b/electron/shutdown.js @@ -0,0 +1,28 @@ +// Orderly shutdown, kept free of Electron imports so it can be unit tested. +// +// `quit()` is what the Exit button, the File > Exit menu item and "window closed" all lead to: +// it cancels a running update download, stops the sidecar's whole process tree (releasing its port and +// memory), closes the windows and asks Electron to quit. It is idempotent. If Electron has not exited +// after `forceAfterMs` (a hung renderer, a blocked will-quit), `app.exit()` ends the process anyway +// so nothing is left running. + +function createShutdown({ app, stopSidecar, cancelDownload = () => {}, closeWindows = () => {}, + forceAfterMs = 5000, setTimer = setTimeout }) { + let started = false + + function quit() { + if (started) return false + started = true + try { cancelDownload() } catch { /* the download is abandoned with the process anyway */ } + try { stopSidecar() } catch { /* the sidecar is stopped again on will-quit and on process exit */ } + try { closeWindows() } catch { /* windows are destroyed by app.quit() */ } + const timer = setTimer(() => app.exit(0), forceAfterMs) + if (timer && typeof timer.unref === 'function') timer.unref() + app.quit() + return true + } + + return { quit, get started() { return started } } +} + +module.exports = { createShutdown } diff --git a/electron/test/shutdown.test.js b/electron/test/shutdown.test.js new file mode 100644 index 0000000..59cc027 --- /dev/null +++ b/electron/test/shutdown.test.js @@ -0,0 +1,45 @@ +const test = require('node:test') +const assert = require('node:assert/strict') +const { createShutdown } = require('../shutdown') + +function setup(overrides = {}) { + const calls = [] + const timers = [] + const app = { quit: () => calls.push('quit'), exit: code => calls.push(`exit:${code}`) } + const shutdown = createShutdown({ + app, + stopSidecar: () => calls.push('sidecar'), + cancelDownload: () => calls.push('download'), + closeWindows: () => calls.push('windows'), + setTimer: (fn, ms) => { timers.push({ fn, ms }); return { unref() {} } }, + ...overrides, + }) + return { calls, timers, shutdown } +} + +test('quit stops everything the app started, then quits', () => { + const { calls, shutdown } = setup() + assert.equal(shutdown.quit(), true) + assert.deepEqual(calls, ['download', 'sidecar', 'windows', 'quit']) +}) + +test('quit is idempotent', () => { + const { calls, shutdown } = setup() + shutdown.quit() + assert.equal(shutdown.quit(), false) + assert.equal(calls.filter(c => c === 'sidecar').length, 1) +}) + +test('a failing step does not stop the others', () => { + const { calls, shutdown } = setup({ cancelDownload: () => { throw new Error('x') }, stopSidecar: () => { throw new Error('y') } }) + shutdown.quit() + assert.deepEqual(calls, ['windows', 'quit']) +}) + +test('the process is ended by force if Electron does not exit in time', () => { + const { calls, timers, shutdown } = setup({ forceAfterMs: 1234 }) + shutdown.quit() + assert.equal(timers[0].ms, 1234) + timers[0].fn() + assert.equal(calls.at(-1), 'exit:0') +}) diff --git a/frontend/package-lock.json b/frontend/package-lock.json index b40f7af..11ab764 100644 --- a/frontend/package-lock.json +++ b/frontend/package-lock.json @@ -1,12 +1,12 @@ { "name": "metalens-frontend", - "version": "0.4.1", + "version": "0.5.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "metalens-frontend", - "version": "0.4.1", + "version": "0.5.0", "dependencies": { "@fontsource/inter": "^5.3.0", "@fontsource/jetbrains-mono": "^5.3.0", diff --git a/frontend/package.json b/frontend/package.json index a9d0bc1..40372dc 100644 --- a/frontend/package.json +++ b/frontend/package.json @@ -1,6 +1,6 @@ { "name": "metalens-frontend", - "version": "0.4.1", + "version": "0.5.0", "private": true, "type": "module", "scripts": { diff --git a/frontend/src/App.jsx b/frontend/src/App.jsx index 0d1de78..4242e5b 100644 --- a/frontend/src/App.jsx +++ b/frontend/src/App.jsx @@ -8,7 +8,7 @@ import { useUndoStack } from './hooks/useUndoStack' import { useUpdates } from './hooks/useUpdates' import { metalens } from './api/client' import { isTextEditable, isUndoShortcut } from './lib/keyboard' -import { FolderOpen, RotateCcw, GitCompare, Layers, CircleArrowUp } from 'lucide-react' +import { FolderOpen, RotateCcw, GitCompare, Layers, CircleArrowUp, LogOut } from 'lucide-react' import './styles/globals.css' const baseName = path => (path ? String(path).split(/[\\/]/).pop() : 'the file') @@ -18,6 +18,7 @@ export default function App() { const [selectedFile, setSelectedFile] = useState(null) const [fileB, setFileB] = useState(null) const [showAbout, setShowAbout] = useState(false) + const [exiting, setExiting] = useState(false) const [undoMsg, setUndoMsg] = useState(null) const { push: undoPush, undo, canUndo, runExclusive, busy } = useUndoStack() const updates = useUpdates() @@ -86,6 +87,13 @@ export default function App() { if (path) handleSelectPath(path) } + // Exit: the main process stops the sidecar (its whole process tree) and quits the app + async function exitApp() { + if (busy || exiting) return + setExiting(true) + try { await window.electronAPI.quitApp() } catch { setExiting(false) } + } + // ── Undo ────────────────────────────────────────────────────────────────── function showUndoMsg(message, ms) { clearTimeout(undoMsgTimer.current) @@ -194,6 +202,12 @@ export default function App() { )} + {window.electronAPI?.quitApp && ( + + )} @@ -215,6 +229,9 @@ export default function App() { reloadKey={filesReloadKey} onSelectFile={handleSelectFile} onSelectTwo={handleSelectTwo} + onUndoPush={undoPush} + runExclusive={runExclusive} + onChanged={() => setFilesReloadKey(value => value + 1)} /> diff --git a/frontend/src/api/client.js b/frontend/src/api/client.js index 23830b6..87023eb 100644 --- a/frontend/src/api/client.js +++ b/frontend/src/api/client.js @@ -91,6 +91,14 @@ async function api(method, path, body = null) { // The server reads only key and value: permission/type flags are never sent. const writeFields = fields => (fields ?? []).map(({ key, value }) => ({ key, value })) +/** A response body as text (for downloads such as the CSV export). */ +async function apiText(path) { + const { port, token } = await getConnection() + const res = await fetch(`http://127.0.0.1:${port}${path}`, { headers: { Authorization: `Bearer ${token}` } }) + if (!res.ok) throw apiError(res.status, res.statusText, await readJson(res).catch(() => null)) + return res.text() +} + export const metalens = { health: () => api('GET', '/health'), list: (path) => api('GET', `/list?path=${encodeURIComponent(path)}`), @@ -105,6 +113,20 @@ export const metalens = { // Rejects with status 410 `undo_expired` or 409 `concurrent_modification` when the // entry can no longer be applied; other errors (403/422/500/network) are retryable. undo: (token) => api('POST', '/undo', { token }), + // Prepare for sharing: remove what the categories reveal. With `preview` nothing is changed. + sanitize: (path, categories, { preview = true, keep = [] } = {}) => + api('POST', '/sanitize', { path, categories: [...categories], preview, keep: [...keep] }), + // Several files at once: one verified save and one undo token per file (not atomic across files). + sanitizeMany: (paths, categories, { preview = true, keep = [] } = {}) => + api('POST', '/sanitize-many', { paths: [...paths], categories: [...categories], preview, keep: [...keep] }), + shiftDates: (paths, seconds, { preview = true, keys = null } = {}) => + api('POST', '/shift-dates', { paths: [...paths], seconds, preview, ...(keys ? { keys: [...keys] } : {}) }), + copyTags: (source, targets, { preview = true, keys = null } = {}) => + api('POST', '/copy-tags', { source, targets: [...targets], preview, ...(keys ? { keys: [...keys] } : {}) }), + geotag: (paths, position, { preview = true } = {}) => api('POST', '/geotag', { paths: [...paths], preview, ...position }), + rename: (path, template, { preview = true } = {}) => api('POST', '/rename', { path, template, preview }), + exportFolder: (path, format = 'csv', recursive = false) => + apiText(`/export?path=${encodeURIComponent(path)}&format=${format}&recursive=${recursive}`), diff: (a, b) => api('GET', `/diff?a=${encodeURIComponent(a)}&b=${encodeURIComponent(b)}`), hash: (path, algorithms) => api('GET', `/hash?path=${encodeURIComponent(path)}&algorithms=${encodeURIComponent(algorithms)}`), } diff --git a/frontend/src/components/AboutDialog.jsx b/frontend/src/components/AboutDialog.jsx index 61ce229..d466dd3 100644 --- a/frontend/src/components/AboutDialog.jsx +++ b/frontend/src/components/AboutDialog.jsx @@ -32,7 +32,7 @@ function UpdateSection({ updates }) { {update.ignored && (ignored)} {update.notes && ( - 
{update.notes}
)} @@ -105,7 +105,7 @@ export function AboutDialog({ onClose, updates }) {
e.stopPropagation()}> diff --git a/frontend/src/components/BatchDialog.jsx b/frontend/src/components/BatchDialog.jsx new file mode 100644 index 0000000..8174054 --- /dev/null +++ b/frontend/src/components/BatchDialog.jsx @@ -0,0 +1,218 @@ +import { useState } from 'react' +import { X, ShieldCheck, CalendarClock, Copy, Pencil, Download, MapPin } from 'lucide-react' +import { Button } from './ui/Button' +import { Spinner } from './ui/Spinner' +import { metalens } from '../api/client' +import { PRESETS } from './SharePanel' + +const ACTIONS = [ + { id: 'share', label: 'Prepare for sharing', Icon: ShieldCheck }, + { id: 'dates', label: 'Shift dates', Icon: CalendarClock }, + { id: 'copy', label: 'Copy tags', Icon: Copy }, + { id: 'place', label: 'Add location', Icon: MapPin }, + { id: 'rename', label: 'Rename', Icon: Pencil }, + { id: 'export', label: 'Export', Icon: Download }, +] +const UNITS = { minutes: 60, hours: 3600, days: 86400 } +const baseName = path => String(path).split(/[\\/]/).pop() + +/** Save `text` as a file through the browser (no network: the content is already in memory). */ +export function download(name, text, type) { + const url = URL.createObjectURL(new Blob([text], { type })) + const link = document.createElement('a') + link.href = url + link.download = name + link.click() + setTimeout(() => URL.revokeObjectURL(url), 1000) +} + +function Results({ response, describe }) { + if (!response) return null + return ( +
    + {response.results.map(r => ( +
  • + {baseName(r.path)}{' '} + {r.ok ? describe(r, response.preview) : `not done: ${(r.errors ?? []).map(e => e.message).join('; ') || 'error'}`} +
    • + ))} +
+ ) +} + +export function BatchDialog({ paths, folder, onClose, onUndoPush, runExclusive = fn => fn(), onChanged }) { + const [action, setAction] = useState('share') + const [preset, setPreset] = useState(PRESETS[1]) + const [amount, setAmount] = useState(1) + const [unit, setUnit] = useState('hours') + const [template, setTemplate] = useState('{exif:DateTimeOriginal|%Y%m%d_%H%M%S}_{name}') + const [format, setFormat] = useState('csv') + const [latitude, setLatitude] = useState('') + const [longitude, setLongitude] = useState('') + const [gpx, setGpx] = useState('') + const [offset, setOffset] = useState(0) + const [response, setResponse] = useState(null) + const [busy, setBusy] = useState(false) + const [error, setError] = useState(null) + + const seconds = Math.round(Number(amount) * UNITS[unit]) + + const run = async (preview) => { + setBusy(true) + setError(null) + try { + let body + if (action === 'share') body = await metalens.sanitizeMany(paths, preset.categories, { preview }) + else if (action === 'dates') body = await metalens.shiftDates(paths, seconds, { preview }) + else if (action === 'place') { + body = await metalens.geotag(paths, gpx.trim() + ? { gpx: gpx.trim(), utc_offset_minutes: Math.round(Number(offset) * 60) } + : { latitude: Number(latitude), longitude: Number(longitude) }, { preview }) + } else if (action === 'copy') body = await metalens.copyTags(paths[0], paths.slice(1), { preview }) + else if (action === 'rename') { + const results = [] + for (const path of paths) { + try { results.push({ path, ok: true, ...(await metalens.rename(path, template, { preview })) }) } + catch (e) { results.push({ path, ok: false, detail: e.message }) } + } + body = { ok: results.every(r => r.ok), results, undo_tokens: [] } + } + setResponse({ ...body, preview }) + if (!preview) { + for (const r of body.results ?? []) { + if (r.undo_token) onUndoPush?.({ filePath: r.path, token: r.undo_token, label: ACTIONS.find(a => a.id === action).label }) + } + onChanged?.() + } + } catch (e) { + setError(e.message) + } finally { + setBusy(false) + } + } + const runExclusiveApply = () => runExclusive(() => run(false)) + + const exportFolder = async () => { + setBusy(true) + setError(null) + try { + const text = await metalens.exportFolder(folder, format) + download(`metadata.${format}`, text, format === 'csv' ? 'text/csv' : 'application/json') + } catch (e) { + setError(e.message) + } finally { + setBusy(false) + } + } + + const needsTwo = (action === 'copy' && paths.length < 2) + || (action === 'place' && !gpx.trim() && (latitude === '' || longitude === '')) + const describe = { + share: (r, preview) => preview ? `${r.will_remove?.length ?? 0} to remove${r.remaining?.length ? `, ${r.remaining.length} cannot be removed` : ''}` + : `removed ${r.removed?.length ?? 0}`, + dates: r => r.changes?.length ? `${r.changes.length} date${r.changes.length > 1 ? 's' : ''}` : 'no dates to change', + place: r => r.will_write?.length ? `position ${r.position?.slice(0, 2).join(', ')}` : (r.reason ?? 'no position'), + copy: r => r.will_copy?.length ? `${r.will_copy.length} field${r.will_copy.length > 1 ? 's' : ''}` : 'nothing to copy', + rename: r => (r.renamed ? 'renamed to ' : 'would become ') + baseName(r.new ?? ''), + }[action] + + return ( +
+
+
+ + {action === 'export' ? 'Export the metadata of the folder' : `${paths.length} file${paths.length !== 1 ? 's' : ''} selected`} + + +
+ +
+ {ACTIONS.map(({ id, label, Icon }) => ( + + ))} +
+ + {action === 'share' && ( +
+ {PRESETS.map(p => ( + + ))} +
+ )} + {action === 'dates' && ( +
+ + + negative values move the dates back +
+ )} + {action === 'copy' && ( +

The metadata of {baseName(paths[0] ?? '')} (the first + selected file) is copied to the other {Math.max(paths.length - 1, 0)} selected file(s) where the field exists.

+ )} + {action === 'place' && ( +
+
+ + +
+ + {gpx.trim() && ( + + )} +
+ )} + {action === 'rename' && ( + + )} + {action === 'export' && ( +
+ + +
+ )} + + {error &&
{error}
} + + {action !== 'export' && ( + <> + +
+ + +
+ {response && !response.preview &&
Done.
} + + )} +
+
+ ) +} diff --git a/frontend/src/components/DetailPanel.jsx b/frontend/src/components/DetailPanel.jsx index a212ea9..ba672aa 100644 --- a/frontend/src/components/DetailPanel.jsx +++ b/frontend/src/components/DetailPanel.jsx @@ -1,9 +1,10 @@ import { useState, useEffect, useCallback, useRef } from 'react' -import { Eye, Pencil, GitCompare, AlertTriangle, ShieldCheck, Copy, Check, Hash } from 'lucide-react' +import { Eye, Pencil, Share2, GitCompare, AlertTriangle, ShieldCheck, Copy, Check, Hash } from 'lucide-react' import { Spinner } from './ui/Spinner' import { MetadataTable } from './MetadataTable' import { RecordNotices } from './RecordNotices' import { MetadataEditor } from './MetadataEditor' +import { SharePanel } from './SharePanel' import { DiffView } from './DiffView' import { metalens } from '../api/client' @@ -112,6 +113,7 @@ function HashSection({ filePath }) { const TABS = [ { id: 'view', label: 'Metadata', Icon: Eye }, { id: 'edit', label: 'Edit', Icon: Pencil }, + { id: 'share', label: 'Share', Icon: Share2 }, { id: 'diff', label: 'Diff', Icon: GitCompare }, ] @@ -208,6 +210,17 @@ export function DetailPanel({ filePath, filePathB, onUndoPush, runExclusive, mut mutationBusy={mutationBusy} /> )} + {!loading && !error && filePath && tab === 'share' && ( + setRevision(value => value + 1)} + onUndoPush={onUndoPush} + runExclusive={runExclusive} + mutationBusy={mutationBusy} + /> + )} {tab === 'diff' && ( )} diff --git a/frontend/src/components/FilePanel.jsx b/frontend/src/components/FilePanel.jsx index b84e707..6d9a01c 100644 --- a/frontend/src/components/FilePanel.jsx +++ b/frontend/src/components/FilePanel.jsx @@ -1,4 +1,5 @@ import { useState, useEffect, useRef } from 'react' +import { BatchDialog } from './BatchDialog' import { File, Image, Music, Film, FileText, FileCode, Archive, Aperture, Sheet, Presentation, AlertCircle } from 'lucide-react' import { Spinner } from './ui/Spinner' import { metalens } from '../api/client' @@ -40,11 +41,12 @@ function formatDate(ts) { return new Date(ts * 1000).toLocaleString() } -export function FilePanel({ currentPath, reloadKey = 0, onSelectFile, onSelectTwo }) { +export function FilePanel({ currentPath, reloadKey = 0, onSelectFile, onSelectTwo, onUndoPush, runExclusive, onChanged }) { const [items, setItems] = useState([]) const [loading, setLoading] = useState(false) const [error, setError] = useState(null) const [selected, setSelected] = useState(new Set()) + const [batchOpen, setBatchOpen] = useState(false) const [sortKey, setSortKey] = useState('name') const [sortAsc, setSortAsc] = useState(true) @@ -177,7 +179,17 @@ export function FilePanel({ currentPath, reloadKey = 0, onSelectFile, onSelectTw {sorted.length} file{sorted.length !== 1 ? 's' : ''} {selected.size > 0 && ` · ${selected.size} selected`} {selected.size === 2 && — ready to compare} + {currentPath && ( + + )}
+ {batchOpen && ( + setBatchOpen(false)} + onUndoPush={onUndoPush} runExclusive={runExclusive} onChanged={onChanged} /> + )}
) } diff --git a/frontend/src/components/MetadataEditor.jsx b/frontend/src/components/MetadataEditor.jsx index 7d15c6a..6d86ce8 100644 --- a/frontend/src/components/MetadataEditor.jsx +++ b/frontend/src/components/MetadataEditor.jsx @@ -3,6 +3,7 @@ import { Save, RotateCcw, Trash2, Plus, X } from 'lucide-react' import { Button } from './ui/Button' import { FieldInput } from './FieldInput' import { FieldValue } from './MetadataTable' +import { PrivacySummary, SensitivityBadge } from './PrivacyIndicators' import { RecordNotices } from './RecordNotices' import { metalens } from '../api/client' import { emptyInput, fromInput, inputKind, toInput } from '../lib/serializeValue' @@ -23,9 +24,9 @@ function describeSave(edited, added, deleted) { // exclusively from the server's `editable` / `deletable` flags. const GROUPS = [ ['exif', 'EXIF'], ['gps', 'GPS'], ['img', 'Image'], ['xmp', 'XMP'], ['iptc', 'IPTC'], - ['png', 'PNG'], ['gif', 'GIF'], ['tag', 'Tags'], ['vorbis', 'Vorbis'], ['flac', 'FLAC'], - ['mp4', 'MP4'], ['asf', 'ASF'], ['ape', 'APE'], ['audio', 'Audio'], ['video', 'Video'], - ['pdf', 'PDF'], ['office', 'Office'], ['ole', 'OLE'], ['hachoir', 'Hachoir'], + ['png', 'PNG'], ['pngchunk', 'PNG chunks'], ['heif', 'HEIF'], ['jfif', 'JFIF'], ['jpeg', 'JPEG'], ['mpf', 'MPF'], ['icc', 'ICC profile'], ['maker', 'Camera maker'], ['gif', 'GIF'], ['tag', 'Tags'], ['vorbis', 'Vorbis'], ['flac', 'FLAC'], + ['mp4', 'MP4'], ['qt', 'QuickTime'], ['mkv', 'Matroska'], ['asf', 'ASF'], ['ape', 'APE'], ['audio', 'Audio'], ['video', 'Video'], + ['pdf', 'PDF'], ['office', 'Office'], ['odf', 'OpenDocument'], ['epub', 'EPUB'], ['ole', 'OLE'], ['hachoir', 'Hachoir'], ['other', 'Other'], ['fs', 'Filesystem'], ['xattr', 'Extended attributes'], ] const GROUP_LABELS = Object.fromEntries(GROUPS) @@ -192,6 +193,7 @@ export function MetadataEditor({ record, filePath, onSaved, onUndoPush, runExclu + {/* Fields */}
@@ -249,6 +251,7 @@ function FieldRow({ field: f, draft, isNew, isChanged, isDeleted, error, disable
{f.label || f.key} {isNew && new} +
{isDeleted diff --git a/frontend/src/components/MetadataTable.jsx b/frontend/src/components/MetadataTable.jsx index ff8b948..db1b1a8 100644 --- a/frontend/src/components/MetadataTable.jsx +++ b/frontend/src/components/MetadataTable.jsx @@ -1,16 +1,22 @@ import { Badge } from './ui/Badge' +import { PrivacySummary, SensitivityBadge } from './PrivacyIndicators' +import { matchesQuery } from '../lib/aliases' +import { useState } from 'react' import { formatItems, isBinaryDescriptor, uniqueKeys, valueTitle } from '../lib/formatValue' const SOURCE_VARIANTS = { - exif: 'cyan', gps: 'cyan', iptc: 'cyan', xmp: 'cyan', png: 'cyan', gif: 'cyan', + exif: 'cyan', gps: 'cyan', iptc: 'cyan', xmp: 'cyan', png: 'cyan', gif: 'cyan', pngchunk: 'cyan', id3: 'purple', vorbis: 'purple', mp4: 'purple', asf: 'purple', ape: 'purple', audio: 'purple', flac: 'purple', pdf: 'warning', office: 'warning', ole: 'warning', filesystem: 'default', xattr: 'default', - hachoir: 'default', video: 'purple', image: 'cyan', + hachoir: 'default', video: 'purple', image: 'cyan', qt: 'purple', mkv: 'purple', odf: 'warning', epub: 'warning', heif: 'cyan', icc: 'cyan', maker: 'cyan', jfif: 'cyan', jpeg: 'cyan', mpf: 'cyan', } -export function MetadataTable({ fields = [] }) { - if (!fields.length) { +export function MetadataTable({ fields: allFields = [] }) { + const [query, setQuery] = useState('') + const needle = query.trim().toLowerCase() + const fields = allFields.filter(f => matchesQuery(f, needle)) + if (!allFields.length) { return (
No metadata fields @@ -28,6 +34,13 @@ export function MetadataTable({ fields = [] }) { return (
+ +
+ setQuery(e.target.value)} + className="w-full bg-cyber-bg border border-cyber-border rounded px-2 py-0.5 text-[11px] font-mono" /> +
+ {!fields.length &&
No field matches
} {Object.entries(groups).map(([src, groupFields]) => { const keys = uniqueKeys(groupFields, f => f.key) return ( @@ -44,7 +57,7 @@ export function MetadataTable({ fields = [] }) { {f.label || f.key}
- +
{!f.editable && ro} diff --git a/frontend/src/components/PrivacyIndicators.jsx b/frontend/src/components/PrivacyIndicators.jsx new file mode 100644 index 0000000..37824d6 --- /dev/null +++ b/frontend/src/components/PrivacyIndicators.jsx @@ -0,0 +1,48 @@ +import { ShieldAlert } from 'lucide-react' +import { Badge } from './ui/Badge' + +// What a field can reveal when the file is shared (classified by the sidecar: `field.sensitivity`). +export const SENSITIVITY = { + location: { label: 'Location', variant: 'danger', hint: 'Reveals where the file was made' }, + identity: { label: 'Identity', variant: 'danger', hint: 'Reveals who made or owns the file' }, + device: { label: 'Device', variant: 'warning', hint: 'Reveals the device or unique identifiers' }, + thumbnail: { label: 'Thumbnail', variant: 'warning', hint: 'Contains an embedded copy of the content' }, + history: { label: 'History', variant: 'warning', hint: 'Edit history and other data stored but not shown' }, + datetime: { label: 'Date', variant: 'default', hint: 'Reveals when the file was made or changed' }, + software: { label: 'Software', variant: 'default', hint: 'Reveals the program used' }, + text: { label: 'Text', variant: 'default', hint: 'Free text written by a person' }, +} + +const ORDER = ['location', 'identity', 'device', 'thumbnail', 'history', 'text', 'datetime', 'software'] + +export function SensitivityBadge({ field }) { + const info = SENSITIVITY[field?.sensitivity] + if (!info) return null + return {info.label} +} + +/** Counts of fields per sensitivity category, most serious first. */ +export function summarizeSensitivity(fields = []) { + const counts = {} + for (const f of fields) if (SENSITIVITY[f.sensitivity]) counts[f.sensitivity] = (counts[f.sensitivity] || 0) + 1 + return ORDER.filter(c => counts[c]).map(c => ({ category: c, count: counts[c] })) +} + +/** One line telling what sharing this file would reveal. */ +export function PrivacySummary({ fields = [] }) { + const summary = summarizeSensitivity(fields).filter(s => SENSITIVITY[s.category].variant !== 'default' + || s.category === 'datetime') + if (!summary.length) return null + return ( +
+
+ ) +} diff --git a/frontend/src/components/SharePanel.jsx b/frontend/src/components/SharePanel.jsx new file mode 100644 index 0000000..5f10877 --- /dev/null +++ b/frontend/src/components/SharePanel.jsx @@ -0,0 +1,154 @@ +import { useEffect, useMemo, useState } from 'react' +import { ShieldCheck, Eye } from 'lucide-react' +import { Button } from './ui/Button' +import { Spinner } from './ui/Spinner' +import { Badge } from './ui/Badge' +import { SENSITIVITY, summarizeSensitivity } from './PrivacyIndicators' +import { metalens } from '../api/client' + +// Presets of "Prepare for sharing": which categories are removed. +export const PRESETS = [ + { id: 'location', label: 'Remove location', categories: ['location'] }, + { id: 'personal', label: 'Remove personal data', categories: ['location', 'identity', 'device', 'thumbnail', 'history'] }, + { id: 'everything', label: 'Remove everything shown', + categories: ['location', 'identity', 'device', 'thumbnail', 'history', 'text', 'datetime', 'software'] }, +] + +const immediately = fn => fn() + +export function SharePanel({ record, filePath, onSaved, onUndoPush, runExclusive = immediately, mutationBusy = false }) { + const [categories, setCategories] = useState(new Set(PRESETS[1].categories)) + const [preview, setPreview] = useState(null) + const [result, setResult] = useState(null) + const [busy, setBusy] = useState(false) + const [error, setError] = useState(null) + + const present = useMemo(() => summarizeSensitivity(record?.fields ?? []), [record]) + const list = useMemo(() => [...categories].sort(), [categories]) + + useEffect(() => { + let current = true + setPreview(null) + setError(null) + if (!list.length) return undefined + setBusy(true) + metalens.sanitize(filePath, list, { preview: true }) + .then(body => { if (current) setPreview(body) }) + .catch(e => { if (current) setError(e.message) }) + .finally(() => { if (current) setBusy(false) }) + return () => { current = false } + }, [filePath, list, record]) + + const toggle = category => { + setResult(null) + setCategories(prev => { + const next = new Set(prev) + if (next.has(category)) next.delete(category) + else next.add(category) + return next + }) + } + const choose = preset => { + setCategories(new Set(preset.categories)) + setResult(null) + } + + const apply = async () => { + setBusy(true) + setError(null) + try { + await runExclusive(async () => { + const body = await metalens.sanitize(filePath, list, { preview: false }) + if (body?.undo_token) onUndoPush?.({ filePath, token: body.undo_token, label: 'Prepare for sharing' }) + setResult(body) + }) + onSaved?.() + } catch (e) { + setError(e.message) + } finally { + setBusy(false) + } + } + + const removable = preview?.will_remove?.length ?? 0 + const blocked = preview?.remaining ?? [] + const removed = result?.removed?.length ?? 0 + + return ( +
+
+
+

+ Choose what to remove before you send the file to someone. You see what will be removed first, and the + change can be undone. +

+ +
+ {PRESETS.map(p => ( + + ))} +
+ +
+ Remove + {Object.entries(SENSITIVITY).map(([id, info]) => { + const count = present.find(s => s.category === id)?.count ?? 0 + return ( + + ) + })} +
+ + {error &&
{error}
} + + {preview && !result && ( +
+
+
+ {removable === 0 + ?
Nothing to remove for the chosen categories.
+ : ( +
    + {preview.will_remove.map(e => ( +
  • {e.label || e.key} ({e.key})
    • + ))} +
+ )} + {blocked.length > 0 && ( +
+ {blocked.length} sensitive field{blocked.length > 1 ? 's' : ''} cannot be removed from this file: +
    + {blocked.map(e =>
  • {e.label || e.key}: {e.reason}
    • )} +
+
+ )} +
+ )} + + {result && ( +
+ Removed {removed} field{removed === 1 ? '' : 's'}. + {result.remaining?.length > 0 && ( + {result.remaining.length} could not be removed. + )} + {result.warnings?.map((w, i) =>
{w}
)} +
+ )} + +
+ +
+
+ ) +} diff --git a/frontend/src/components/ui/Badge.jsx b/frontend/src/components/ui/Badge.jsx index 4c3c4a3..766ff24 100644 --- a/frontend/src/components/ui/Badge.jsx +++ b/frontend/src/components/ui/Badge.jsx @@ -1,4 +1,4 @@ -export function Badge({ children, variant = 'default', className = '' }) { +export function Badge({ children, variant = 'default', className = '', title }) { const variants = { default: 'bg-cyber-tertiary text-cyber-muted border-cyber-border', cyan: 'bg-cyan-950/50 text-cyber-cyan border-cyber-cyan/30', @@ -8,7 +8,7 @@ export function Badge({ children, variant = 'default', className = '' }) { danger: 'bg-red-950/50 text-cyber-danger border-cyber-danger/30', } return ( - {children} diff --git a/frontend/src/lib/aliases.js b/frontend/src/lib/aliases.js new file mode 100644 index 0000000..518ffbf --- /dev/null +++ b/frontend/src/lib/aliases.js @@ -0,0 +1,54 @@ +// Other names under which a field is commonly known: used to find it in the search box of the metadata +// table. The keys themselves never change. +export const ALIASES = { + 'exif:DateTime': ['modify date', 'modified', 'date modified'], + 'exif:DateTimeOriginal': ['date taken', 'capture date', 'shot date', 'create date'], + 'exif:DateTimeDigitized': ['create date', 'digitized'], + 'exif:ISOSpeedRatings': ['iso'], + 'exif:ExposureBiasValue': ['exposure compensation'], + 'exif:PixelXDimension': ['exif image width', 'width'], + 'exif:PixelYDimension': ['exif image height', 'height'], + 'exif:FNumber': ['aperture', 'f-number'], + 'exif:ExposureTime': ['shutter speed'], + 'exif:Make': ['camera maker', 'manufacturer'], + 'exif:Model': ['camera model'], + 'exif:Artist': ['author', 'photographer', 'creator'], + 'exif:Copyright': ['rights'], + 'exif:Software': ['program', 'application'], + 'exif:ImageDescription': ['caption', 'description'], + 'gps:Latitude': ['gps latitude', 'position', 'location'], + 'gps:Longitude': ['gps longitude', 'position', 'location'], + 'gps:Altitude': ['gps altitude', 'elevation'], + 'xmp:dc:creator': ['author', 'artist'], + 'xmp:dc:title': ['title'], + 'xmp:dc:subject': ['keywords', 'tags'], + 'xmp:dc:description': ['caption', 'description'], + 'xmp:xmp:CreateDate': ['date created', 'create date'], + 'xmp:xmp:CreatorTool': ['software', 'program'], + 'iptc:By-line': ['author', 'creator', 'photographer'], + 'iptc:Keywords': ['tags'], + 'iptc:Caption-Abstract': ['caption', 'description'], + 'tag:TIT2': ['title', 'song'], + 'tag:TPE1': ['artist', 'performer'], + 'tag:TALB': ['album'], + 'tag:TDRC': ['year', 'recording date'], + 'mp4:©nam': ['title'], + 'mp4:©ART': ['artist'], + 'pdf:Author': ['creator', 'owner'], + 'pdf:CreationDate': ['create date', 'created'], + 'pdf:ModDate': ['modify date', 'modified'], + 'pdf:Producer': ['software', 'pdf producer'], + 'office:author': ['creator', 'owner'], + 'office:last_modified_by': ['editor', 'modified by'], + 'office:created': ['create date', 'date created'], + 'qt:CreateDate': ['create date', 'creation date', 'date taken'], + 'qt:mdta:com.apple.quicktime.location.ISO6709': ['gps', 'position', 'location'], +} + +/** Whether `field` matches the lowercase `query` by key, label, value or one of its other names. */ +export function matchesQuery(field, query) { + if (!query) return true + const haystack = [field.key, field.label, ...(ALIASES[field.key] ?? []), + ...(Array.isArray(field.value) ? field.value : [field.value]).filter(v => typeof v === 'string' || typeof v === 'number')] + return haystack.some(text => String(text ?? '').toLowerCase().includes(query)) +} diff --git a/frontend/test/batch.test.jsx b/frontend/test/batch.test.jsx new file mode 100644 index 0000000..b5c9791 --- /dev/null +++ b/frontend/test/batch.test.jsx @@ -0,0 +1,99 @@ +import { fireEvent, render, screen, waitFor, within } from '@testing-library/react' +import { beforeEach, describe, expect, test, vi } from 'vitest' +import { metalens } from '../src/api/client' +import { BatchDialog } from '../src/components/BatchDialog' + +vi.mock('../src/api/client', () => ({ + metalens: { sanitizeMany: vi.fn(), shiftDates: vi.fn(), copyTags: vi.fn(), rename: vi.fn(), exportFolder: vi.fn() }, +})) + +const paths = ['C:/p/a.jpg', 'C:/p/b.jpg'] +const previewBody = { ok: true, results: [ + { path: paths[0], ok: true, will_remove: [{ key: 'gps:Latitude' }], remaining: [], undo_token: null }, + { path: paths[1], ok: false, status: 403, errors: [{ code: 'not_writable', key: null, message: 'file is read-only' }] }, +], undo_tokens: [] } + +beforeEach(() => { vi.resetAllMocks() }) + +describe('Batch dialog', () => { + test('previews every file first and shows a failing file without stopping', async () => { + metalens.sanitizeMany.mockResolvedValue(previewBody) + render( {}} />) + expect(screen.getByRole('button', { name: /Apply to/ }).disabled).toBe(true) // nothing is applied unseen + fireEvent.click(screen.getByRole('button', { name: 'Preview' })) + const results = await screen.findByLabelText('Results') + expect(within(results).getByText('a.jpg')).toBeTruthy() + expect(results.textContent).toContain('1 to remove') + expect(results.textContent).toContain('not done: file is read-only') + expect(metalens.sanitizeMany.mock.calls[0][2]).toEqual({ preview: true }) + }) + + test('applying pushes one undo entry per file', async () => { + const onUndoPush = vi.fn() + metalens.sanitizeMany.mockImplementation(async (_p, _c, { preview }) => preview ? previewBody + : { ok: true, results: [{ path: paths[0], ok: true, removed: [{ key: 'x' }], remaining: [], undo_token: 'tok-a' }], undo_tokens: ['tok-a'] }) + render( {}} onUndoPush={onUndoPush} />) + fireEvent.click(screen.getByRole('button', { name: 'Preview' })) + await screen.findByLabelText('Results') + fireEvent.click(screen.getByRole('button', { name: /Apply to 2 file/ })) + await screen.findByRole('status') + expect(onUndoPush).toHaveBeenCalledWith({ filePath: paths[0], token: 'tok-a', label: 'Prepare for sharing' }) + }) + + test('dates are shifted by the chosen amount of the chosen unit', async () => { + metalens.shiftDates.mockResolvedValue({ ok: true, results: [{ path: paths[0], ok: true, changes: [{ key: 'k' }], skipped: [] }], undo_tokens: [] }) + render( {}} />) + fireEvent.click(screen.getByRole('tab', { name: /Shift dates/ })) + fireEvent.change(screen.getByLabelText('Amount'), { target: { value: '-2' } }) + fireEvent.change(screen.getByLabelText('Unit'), { target: { value: 'days' } }) + fireEvent.click(screen.getByRole('button', { name: 'Preview' })) + await waitFor(() => expect(metalens.shiftDates).toHaveBeenCalled()) + expect(metalens.shiftDates.mock.calls[0][1]).toBe(-2 * 86400) + }) + + test('copying needs at least two files and uses the first as the source', async () => { + metalens.copyTags.mockResolvedValue({ ok: true, results: [{ path: paths[1], ok: true, will_copy: ['exif:Artist'], skipped: [] }], undo_tokens: [] }) + const { rerender } = render( {}} />) + fireEvent.click(screen.getByRole('tab', { name: /Copy tags/ })) + expect(screen.getByRole('button', { name: 'Preview' }).disabled).toBe(true) + rerender( {}} />) + fireEvent.click(screen.getByRole('button', { name: 'Preview' })) + await waitFor(() => expect(metalens.copyTags).toHaveBeenCalledWith(paths[0], [paths[1]], { preview: true })) + }) + + test('export saves the folder metadata with the chosen format', async () => { + metalens.exportFolder.mockResolvedValue('file,handler\n') + const click = vi.fn() + URL.createObjectURL = vi.fn(() => 'blob:x') + URL.revokeObjectURL = vi.fn() + const create = document.createElement.bind(document) + vi.spyOn(document, 'createElement').mockImplementation(tag => { + const element = create(tag) + if (tag === 'a') element.click = click + return element + }) + render( {}} />) + fireEvent.click(screen.getByRole('tab', { name: /Export/ })) + fireEvent.click(screen.getByRole('button', { name: /Save the metadata/ })) + await waitFor(() => expect(click).toHaveBeenCalled()) + expect(metalens.exportFolder).toHaveBeenCalledWith('C:/p', 'csv') + }) +}) + +describe('Add location', () => { + test('fixed coordinates are sent as numbers, a GPX track with the clock offset', async () => { + metalens.geotag = vi.fn().mockResolvedValue({ ok: true, results: [{ path: paths[0], ok: true, will_write: ['gps:Latitude'], position: [1, 2, null] }], undo_tokens: [] }) + render( {}} />) + fireEvent.click(screen.getByRole('tab', { name: /Add location/ })) + expect(screen.getByRole('button', { name: 'Preview' }).disabled).toBe(true) + fireEvent.change(screen.getByLabelText('Latitude'), { target: { value: '45.5' } }) + fireEvent.change(screen.getByLabelText('Longitude'), { target: { value: '9.25' } }) + fireEvent.click(screen.getByRole('button', { name: 'Preview' })) + await waitFor(() => expect(metalens.geotag).toHaveBeenCalledWith(paths, { latitude: 45.5, longitude: 9.25 }, { preview: true })) + fireEvent.change(screen.getByLabelText('GPX file'), { target: { value: 'C:/t/track.gpx' } }) + fireEvent.change(screen.getByLabelText('Clock offset'), { target: { value: '2' } }) + fireEvent.click(screen.getByRole('button', { name: 'Preview' })) + await waitFor(() => expect(metalens.geotag).toHaveBeenLastCalledWith( + paths, { gpx: 'C:/t/track.gpx', utc_offset_minutes: 120 }, { preview: true })) + }) +}) diff --git a/frontend/test/exit.test.jsx b/frontend/test/exit.test.jsx new file mode 100644 index 0000000..01ebb9c --- /dev/null +++ b/frontend/test/exit.test.jsx @@ -0,0 +1,45 @@ +import { fireEvent, render, screen, waitFor } from '@testing-library/react' +import { afterEach, beforeEach, expect, test, vi } from 'vitest' +import App from '../src/App' + +vi.mock('../src/api/client', () => ({ metalens: {} })) +vi.mock('../src/components/FolderPanel', () => ({ FolderPanel: () => null })) +vi.mock('../src/components/FilePanel', () => ({ FilePanel: () => null })) +vi.mock('../src/components/DetailPanel', () => ({ DetailPanel: () => null })) + +let api + +beforeEach(() => { + api = { + onMenuOpenFolder: vi.fn(), onMenuUndo: vi.fn(), onMenuAbout: vi.fn(), removeAllListeners: vi.fn(), + onMenuCheckUpdates: vi.fn(() => vi.fn()), onUpdateProgress: vi.fn(() => vi.fn()), + getAppVersion: vi.fn().mockResolvedValue('0.5.0'), + getUpdateSettings: vi.fn(async () => ({ autoCheckUpdates: false, ignoredVersion: null })), + quitApp: vi.fn().mockResolvedValue(undefined), + } + window.electronAPI = api +}) +afterEach(() => { delete window.electronAPI }) + +test('the Exit button asks the main process to quit, once', async () => { + render() + const button = screen.getByRole('button', { name: /exit/i }) + fireEvent.click(button) + fireEvent.click(button) + await waitFor(() => expect(api.quitApp).toHaveBeenCalledOnce()) + expect(button.disabled).toBe(true) +}) + +test('the Exit button is enabled again when quitting fails', async () => { + api.quitApp.mockRejectedValueOnce(new Error('boom')) + render() + const button = screen.getByRole('button', { name: /exit/i }) + fireEvent.click(button) + await waitFor(() => expect(button.disabled).toBe(false)) +}) + +test('there is no Exit button outside Electron', () => { + delete window.electronAPI + render() + expect(screen.queryByRole('button', { name: /exit/i })).toBeNull() +}) diff --git a/frontend/test/privacy.test.jsx b/frontend/test/privacy.test.jsx new file mode 100644 index 0000000..8da31c6 --- /dev/null +++ b/frontend/test/privacy.test.jsx @@ -0,0 +1,69 @@ +import { render, screen } from '@testing-library/react' +import { describe, expect, test } from 'vitest' +import { MetadataTable } from '../src/components/MetadataTable' +import { PrivacySummary, SensitivityBadge, summarizeSensitivity } from '../src/components/PrivacyIndicators' + +const f = (key, sensitivity, extra = {}) => ({ key, label: key.split(':').pop(), value: 'x', raw_value: 'x', + editable: false, deletable: false, field_type: 'str', source: key.split(':')[0], choices: [], multi: false, + length: null, binary: null, sensitivity, ...extra }) + +const fields = [ + f('gps:Latitude', 'location'), f('gps:Longitude', 'location'), f('exif:Make', 'device'), + f('exif:DateTime', 'datetime'), f('exif:Orientation', null), +] + +describe('privacy indicators', () => { + test('categories are counted and ordered by seriousness', () => { + expect(summarizeSensitivity(fields)).toEqual([ + { category: 'location', count: 2 }, { category: 'device', count: 1 }, { category: 'datetime', count: 1 }]) + expect(summarizeSensitivity([f('img:width', null)])).toEqual([]) + }) + + test('the summary names what sharing reveals', () => { + render() + const note = screen.getByRole('note', { name: 'Privacy summary' }) + expect(note.textContent).toContain('Sharing this file reveals') + expect(note.textContent).toContain('Location · 2') + expect(note.textContent).toContain('Device · 1') + }) + + test('no summary for a file without sensitive fields', () => { + const { container } = render() + expect(container.innerHTML).toBe('') + }) + + test('each sensitive field carries its badge, technical ones do not', () => { + const { rerender } = render() + expect(screen.getByText('Location')).toBeTruthy() + rerender() + expect(screen.queryByText('Location')).toBeNull() + }) + + test('the table shows the summary and the badges', () => { + render() + expect(screen.getByRole('note', { name: 'Privacy summary' })).toBeTruthy() + expect(screen.getAllByText('Location').length).toBeGreaterThanOrEqual(2) + }) +}) + +describe('search', () => { + test('finds a field by its key, its label, its value or another name', async () => { + const { fireEvent } = await import('@testing-library/react') + const list = [ + { ...f('exif:DateTimeOriginal', 'datetime'), value: '2026-01-02T03:04:05', label: 'DateTimeOriginal' }, + { ...f('exif:Make', 'device'), value: 'Canon' }, + { ...f('exif:ISOSpeedRatings', null), value: 200, label: 'ISOSpeedRatings' }, + ] + render() + const search = screen.getByLabelText('Search fields') + fireEvent.change(search, { target: { value: 'date taken' } }) // another name + expect(screen.queryByText('DateTimeOriginal')).toBeTruthy() + expect(screen.queryByText('Make')).toBeNull() + fireEvent.change(search, { target: { value: 'canon' } }) // a value + expect(screen.queryByText('Make')).toBeTruthy() + fireEvent.change(search, { target: { value: 'iso' } }) + expect(screen.queryByText('ISOSpeedRatings')).toBeTruthy() + fireEvent.change(search, { target: { value: 'zzz' } }) + expect(screen.getByText('No field matches')).toBeTruthy() + }) +}) diff --git a/frontend/test/share.test.jsx b/frontend/test/share.test.jsx new file mode 100644 index 0000000..c4c4564 --- /dev/null +++ b/frontend/test/share.test.jsx @@ -0,0 +1,65 @@ +import { fireEvent, render, screen, waitFor } from '@testing-library/react' +import { beforeEach, describe, expect, test, vi } from 'vitest' +import { metalens } from '../src/api/client' +import { PRESETS, SharePanel } from '../src/components/SharePanel' + +vi.mock('../src/api/client', () => ({ metalens: { sanitize: vi.fn() } })) + +const field = (key, sensitivity) => ({ key, label: key.split(':').pop(), value: 'x', sensitivity }) +const record = { fields: [field('gps:Latitude', 'location'), field('exif:Make', 'device'), field('img:width', null)] } +const previewBody = { ok: true, preview: true, undo_token: null, warnings: [], remaining: [], + will_remove: [{ key: 'gps:Latitude', label: 'Latitude', category: 'location' }] } + +beforeEach(() => { + vi.resetAllMocks() + metalens.sanitize.mockResolvedValue(previewBody) +}) + +describe('Prepare for sharing', () => { + test('previews first, with the personal-data preset selected', async () => { + render() + await screen.findByLabelText('Preview') + const [, categories, options] = metalens.sanitize.mock.calls[0] + expect(categories).toEqual([...PRESETS[1].categories].sort()) + expect(options).toEqual({ preview: true }) + expect(screen.getByText('Latitude')).toBeTruthy() + }) + + test('presets change the categories and the preview is requested again', async () => { + render() + await screen.findByLabelText('Preview') + fireEvent.click(screen.getByRole('button', { name: 'Remove location' })) + await waitFor(() => expect(metalens.sanitize.mock.calls.at(-1)[1]).toEqual(['location'])) + }) + + test('applying removes, pushes an undo entry and reports the result', async () => { + const onUndoPush = vi.fn() + const onSaved = vi.fn() + metalens.sanitize.mockImplementation(async (_path, _cats, { preview }) => preview + ? previewBody + : { ...previewBody, preview: false, undo_token: 'tok', removed: previewBody.will_remove, remaining: [] }) + render() + await screen.findByLabelText('Preview') + fireEvent.click(screen.getByRole('button', { name: /Remove 1 field/ })) + await screen.findByRole('status') + expect(onUndoPush).toHaveBeenCalledWith({ filePath: 'a.jpg', token: 'tok', label: 'Prepare for sharing' }) + expect(onSaved).toHaveBeenCalled() + expect(screen.getByRole('status').textContent).toContain('Removed 1 field') + }) + + test('what cannot be removed is listed with its reason', async () => { + metalens.sanitize.mockResolvedValue({ ...previewBody, will_remove: [], + remaining: [{ key: 'gps:Latitude', label: 'Latitude', category: 'location', reason: 'read-only format' }] }) + render() + const preview = await screen.findByLabelText('Preview') + expect(preview.textContent).toContain('cannot be removed from this file') + expect(preview.textContent).toContain('read-only format') + expect(screen.getByRole('button', { name: 'Remove' }).disabled).toBe(true) + }) + + test('a server error is shown', async () => { + metalens.sanitize.mockRejectedValue(new Error('boom')) + render() + expect((await screen.findByRole('alert')).textContent).toContain('boom') + }) +}) diff --git a/python/SECURITY.md b/python/SECURITY.md index 37e98f8..2cfb672 100644 --- a/python/SECURITY.md +++ b/python/SECURITY.md @@ -31,7 +31,7 @@ remain public wrappers that open their own transaction (`_atomic_write`). READONLY attribute) or no temporary directory can be created beside it. 2. Serialize in-process writers and create an exclusively allocated private temporary directory (`.metalens-*`) on the same filesystem as the target. -3. Copy the file (`.ml_tmp`). On Windows, clear HIDDEN/SYSTEM/READONLY on the +3. Copy the file (same name as the target, inside the private directory). On Windows, clear HIDDEN/SYSTEM/READONLY on the copy while the callback runs (libraries recreate files by name, which Windows refuses for hidden files). 4. Validate and apply all changes to the copy, then read it back: written keys must hold the diff --git a/python/api/geotag.py b/python/api/geotag.py new file mode 100644 index 0000000..0d3d767 --- /dev/null +++ b/python/api/geotag.py @@ -0,0 +1,146 @@ +"""Put a position on pictures: fixed coordinates, or the time of each picture looked up along a GPX track.""" +from __future__ import annotations + +import bisect +import xml.etree.ElementTree as ET +from datetime import datetime, timezone +from typing import Any + +from fastapi import APIRouter +from pydantic import BaseModel, Field + +from api import routes +from api.operations import _item, _paths +from api.routes import ApiCode, ApiError, PATH_LOCKS, stable_read +from core import values +from core.path_security import PathSecurityError, validate_file_path + +router = APIRouter() +MAX_GPX_BYTES = 20 * 1024 * 1024 + + +class GeotagRequest(BaseModel): + paths: list[str] = Field(min_length=1) + latitude: float | None = Field(default=None, ge=-90, le=90) + longitude: float | None = Field(default=None, ge=-180, le=180) + altitude: float | None = Field(default=None, ge=-1000, le=100000) + gpx: str | None = None # a GPX track: the position is taken from the time of each picture + utc_offset_minutes: int = Field(default=0, ge=-840, le=840) # the camera clock minus UTC + max_gap_seconds: int = Field(default=600, ge=1, le=86400) + preview: bool = True + + +def _read_track(gpx_path: str) -> list[tuple[float, float, float, float | None]]: + """``(epoch seconds, latitude, longitude, altitude)`` of the track points of a GPX file, in time order.""" + try: + path = validate_file_path(gpx_path, must_exist=True) + except PathSecurityError as exc: + raise routes._path_error(exc) from exc + if path.stat().st_size > MAX_GPX_BYTES: + raise ApiError.one(422, ApiCode.INVALID_REQUEST, "the GPX file is too large") + raw = path.read_bytes() + if b" 0 else None + after = track[i] if i < len(track) else None + if before and after and after[0] > before[0] and after[0] - before[0] <= 2 * max_gap and before[0] <= when <= after[0]: + f = (when - before[0]) / (after[0] - before[0]) + alt = None if before[3] is None or after[3] is None else before[3] + f * (after[3] - before[3]) + return before[1] + f * (after[1] - before[1]), before[2] + f * (after[2] - before[2]), alt + near = min((p for p in (before, after) if p), key=lambda p: abs(p[0] - when), default=None) + if near is not None and abs(near[0] - when) <= max_gap: + return near[1], near[2], near[3] + return None + + +def _dms(value: float) -> list[str]: + value = abs(value) + degrees = int(value) + minutes = int((value - degrees) * 60) + seconds = round((value - degrees - minutes / 60) * 3600 * 10000) + if seconds >= 600000: # rounding reached a whole minute + seconds, minutes = 0, minutes + 1 + if minutes >= 60: + minutes, degrees = 0, degrees + 1 + return [f"{degrees}/1", f"{minutes}/1", f"{seconds}/10000"] + + +def _gps_writes(latitude: float, longitude: float, altitude: float | None) -> list[dict]: + writes = [{"key": "gps:LatitudeRef", "value": "N" if latitude >= 0 else "S"}, + {"key": "gps:Latitude", "value": _dms(latitude)}, + {"key": "gps:LongitudeRef", "value": "E" if longitude >= 0 else "W"}, + {"key": "gps:Longitude", "value": _dms(longitude)}] + if altitude is not None: + writes += [{"key": "gps:AltitudeRef", "value": 0 if altitude >= 0 else 1}, + {"key": "gps:Altitude", "value": f"{round(abs(altitude) * 100)}/100"}] + return writes + + +def _geotag_one(path: str, req: GeotagRequest, track) -> dict[str, Any]: + fpath = routes._valid_file(path) + with PATH_LOCKS.reading(fpath): + ctx = stable_read(fpath, lambda: routes._read_full_record(fpath)) + latitude, longitude, altitude = req.latitude, req.longitude, req.altitude + if track is not None: + shot = next((f for f in ctx.record.fields if f.key == "exif:DateTimeOriginal" and f.value), None) + if shot is None: + return {"position": None, "will_write": [], "reason": "the picture has no capture time"} + when = values.parse_localdatetime(shot.value).replace(tzinfo=timezone.utc).timestamp() \ + - req.utc_offset_minutes * 60 + found = _position_at(track, when, req.max_gap_seconds) + if found is None: + return {"position": None, "will_write": [], "reason": "no track point near the time of the picture"} + latitude, longitude, altitude = found + writes = _gps_writes(latitude, longitude, altitude) + handler = ctx.handler_of(routes.FORMAT) + usable = {f.key for f in ctx.record.fields if f.editable} | {f.key for f in handler.writable_fields(fpath, ctx.record)} \ + | {f.key for f in ctx.record.addable_fields} + if any(w["key"] not in usable for w in writes): + return {"position": [latitude, longitude, altitude], "will_write": [], + "reason": "this kind of file cannot take a position"} + out: dict[str, Any] = {"position": [round(latitude, 6), round(longitude, 6), altitude], + "will_write": [w["key"] for w in writes], "undo_token": None} + if not req.preview: + saved = routes._save(path, [routes.WriteField(key=w["key"], value=w["value"]) for w in writes], []) + out.update(undo_token=saved["undo_token"], undo_unavailable=saved["undo_unavailable"], + warnings=saved["warnings"]) + return out + + +@router.post("/geotag") +def geotag(req: GeotagRequest): + track = None + if req.gpx: + track = _read_track(req.gpx) + elif req.latitude is None or req.longitude is None: + raise ApiError.one(422, ApiCode.INVALID_REQUEST, "give a latitude and a longitude, or a GPX file") + results = [_item(p, lambda p=p: _geotag_one(p, req, track)) for p in _paths(req.paths)] + return {"ok": all(r["ok"] for r in results), "preview": req.preview, "results": results, + "undo_tokens": [r["undo_token"] for r in results if r.get("undo_token")]} diff --git a/python/api/operations.py b/python/api/operations.py new file mode 100644 index 0000000..8f116eb --- /dev/null +++ b/python/api/operations.py @@ -0,0 +1,286 @@ +"""Operations on several files or on dates: batch preparation for sharing, date shift, tag copy, export, rename. + +Every change goes through the same single-file save as ``/write`` (one verified transaction and one Undo token +per file); a batch is therefore NOT atomic across files, and says which files were saved and which were not. +""" +from __future__ import annotations + +import csv +import io +import json +import os +import re +from datetime import timedelta +from pathlib import Path +from typing import Any + +from fastapi import APIRouter, Query +from fastapi.responses import Response +from pydantic import BaseModel, Field + +from api import routes +from api.routes import ApiCode, ApiError, PATH_LOCKS, SanitizeRequest, stable_read +from core import values +from core.errors import ErrorCode, MetadataValueErrors, error_entry +from core.models import MetadataField +from core.path_security import PathSecurityError, validate_directory_path + +router = APIRouter() +MAX_FILES = 500 +MAX_EXPORT_FILES = 2000 + + +def _entries(errors: list[dict]) -> list[dict]: + """Copy of ``errors`` rebuilt from its three fields (code, key, message): nothing else of the + exception travels into the response.""" + return [error_entry(str(e.get("code", "")), e.get("key"), str(e.get("message", ""))) for e in errors] + + +def _item(path: str, work) -> dict[str, Any]: + """Result of ``work()`` for one file: ``{"path", "ok", ...}``; an error never stops the batch. + + A failed item carries only the status and the structured ``errors`` list (the UI builds the + summary from their messages); the text of the exception itself is never returned.""" + try: + return {"path": path, "ok": True, **work()} + except ApiError as exc: + return {"path": path, "ok": False, "status": exc.status, "errors": _entries(exc.errors)} + except MetadataValueErrors as exc: + return {"path": path, "ok": False, "status": 422, "errors": _entries(exc.to_dicts())} + + +def _paths(paths: list[str]) -> list[str]: + if len(paths) > MAX_FILES: + raise ApiError.one(422, ApiCode.INVALID_REQUEST, f"at most {MAX_FILES} files per request") + return paths + + +# ─────────────────────────── prepare many files for sharing ─────────────────────────── + +class SanitizeManyRequest(BaseModel): + paths: list[str] = Field(min_length=1) + categories: list[str] = Field(min_length=1) + preview: bool = True + keep: list[str] = Field(default_factory=list) + + +@router.post("/sanitize-many") +def sanitize_many(req: SanitizeManyRequest): + results = [_item(p, lambda p=p: routes.sanitize_metadata( + SanitizeRequest(path=p, categories=req.categories, preview=req.preview, keep=req.keep))) + for p in _paths(req.paths)] + return {"ok": all(r["ok"] for r in results), "preview": req.preview, "results": results, + "undo_tokens": [r["undo_token"] for r in results if r.get("undo_token")]} + + +# ─────────────────────────── date shift ─────────────────────────── + +class ShiftRequest(BaseModel): + paths: list[str] = Field(min_length=1) + seconds: int = Field(ge=-10 * 365 * 86400, le=10 * 365 * 86400) + keys: list[str] | None = None # default: every editable date-time field of the file + preview: bool = True + + +def _shifted(field: MetadataField, seconds: int) -> str | None: + """The shifted canonical value of a date-time field, ``None`` when it cannot be shifted.""" + try: + if field.field_type == "localdatetime": + return values.format_datetime(values.parse_localdatetime(field.value) + timedelta(seconds=seconds)) + if field.field_type == "datetime": + return values.format_datetime(values.parse_datetime(field.value) + timedelta(seconds=seconds)) + except (ValueError, OverflowError, values.MetadataValueError): + return None + return None + + +def _shift_one(path: str, req: ShiftRequest) -> dict[str, Any]: + fpath = routes._valid_file(path) + with PATH_LOCKS.reading(fpath): + ctx = stable_read(fpath, lambda: routes._read_full_record(fpath)) + wanted = set(req.keys) if req.keys else None + changes, skipped = [], [] + for f in ctx.record.fields: + if f.field_type not in ("datetime", "localdatetime") or f.multi or routes._is_filesystem_key(f.key): + continue + if wanted is not None and f.key not in wanted: + continue + new = _shifted(f, req.seconds) + if not f.editable or new is None: + skipped.append({"key": f.key, "label": f.label, "reason": "this date cannot be changed"}) + else: + changes.append({"key": f.key, "label": f.label, "from": f.value, "to": new}) + out: dict[str, Any] = {"changes": changes, "skipped": skipped, "undo_token": None} + if not req.preview and changes: + saved = routes._save(path, [routes.WriteField(key=c["key"], value=c["to"]) for c in changes], []) + out.update(undo_token=saved["undo_token"], undo_unavailable=saved["undo_unavailable"], + warnings=saved["warnings"]) + return out + + +@router.post("/shift-dates") +def shift_dates(req: ShiftRequest): + results = [_item(p, lambda p=p: _shift_one(p, req)) for p in _paths(req.paths)] + return {"ok": all(r["ok"] for r in results), "preview": req.preview, "results": results, + "undo_tokens": [r["undo_token"] for r in results if r.get("undo_token")]} + + +# ─────────────────────────── copy tags between files ─────────────────────────── + +class CopyRequest(BaseModel): + source: str + targets: list[str] = Field(min_length=1) + keys: list[str] | None = None # default: every copyable field of the source + preview: bool = True + + +def _copy_one(source_fields: dict[str, MetadataField], target: str, req: CopyRequest) -> dict[str, Any]: + fpath = routes._valid_file(target) + with PATH_LOCKS.reading(fpath): + ctx = stable_read(fpath, lambda: routes._read_full_record(fpath)) + schema = {f.key: f for f in ctx.record.fields if f.editable} + schema.update({f.key: f for f in ctx.record.addable_fields}) + handler = ctx.handler_of(routes.FORMAT) + writes, skipped = [], [] + for key, field in source_fields.items(): + if key not in schema: + skipped.append({"key": key, "reason": "the target has no such field"}) + continue + target_field = schema[key] + if target_field.field_type != field.field_type or bool(target_field.multi) != bool(field.multi): + skipped.append({"key": key, "reason": "the type of the field is different"}) + continue + try: + handler.canonical(target_field, field.value) + except Exception as exc: # a value the target cannot hold + skipped.append({"key": key, "reason": f"the value is not accepted ({type(exc).__name__})"}) + continue + writes.append({"key": key, "value": field.value}) + out: dict[str, Any] = {"will_copy": [w["key"] for w in writes], "skipped": skipped, "undo_token": None} + if not req.preview and writes: + saved = routes._save(target, [routes.WriteField(key=w["key"], value=w["value"]) for w in writes], []) + out.update(undo_token=saved["undo_token"], undo_unavailable=saved["undo_unavailable"], + warnings=saved["warnings"]) + return out + + +@router.post("/copy-tags") +def copy_tags(req: CopyRequest): + source = routes._valid_file(req.source) + with PATH_LOCKS.reading(source): + record = stable_read(source, lambda: routes._read_full_record(source)).record + wanted = set(req.keys) if req.keys else None + fields = {f.key: f for f in record.fields + if not routes._is_filesystem_key(f.key) and f.field_type != "binary" and not f.computed + and (wanted is None or f.key in wanted)} + results = [_item(t, lambda t=t: _copy_one(fields, t, req)) for t in _paths(req.targets)] + return {"ok": all(r["ok"] for r in results), "preview": req.preview, "results": results, + "undo_tokens": [r["undo_token"] for r in results if r.get("undo_token")]} + + +# ─────────────────────────── export ─────────────────────────── + +_FORMULA = re.compile(r"^[=+\-@\t\r]") + + +def _cell(value: Any) -> str: + text = json.dumps(value, ensure_ascii=False) if isinstance(value, (list, dict)) else \ + ("" if value is None else str(value)) + return "'" + text if _FORMULA.match(text) else text + + +@router.get("/export") +def export_folder(path: str = Query(..., description="Absolute directory path"), + format: str = Query("json", pattern="^(json|csv)$"), recursive: bool = False): + try: + folder = validate_directory_path(path, must_exist=True) + except PathSecurityError as exc: + raise routes._path_error(exc) from exc + files = sorted(p for p in (folder.rglob("*") if recursive else folder.iterdir()) + if p.is_file() and not routes.is_active_transaction_dir(p.parent)) + if len(files) > MAX_EXPORT_FILES: + raise ApiError.one(422, ApiCode.INVALID_REQUEST, f"at most {MAX_EXPORT_FILES} files can be exported at once") + records = [] + for file in files: + try: + with PATH_LOCKS.reading(file): + record = stable_read(file, lambda f=file: routes._read_full_record(f)).record + records.append({"file": str(file.relative_to(folder)), "handler": record.handler_name, + "fields": {f.key: f.to_dict()["value"] for f in record.fields}, + "warnings": record.warnings, "read_errors": record.read_errors}) + except Exception as exc: # one unreadable file does not stop the export + records.append({"file": str(file.relative_to(folder)), "handler": None, "fields": {}, + "warnings": [], "read_errors": [routes.describe_error(exc)]}) + if format == "json": + return Response(json.dumps(records, ensure_ascii=False, indent=1), media_type="application/json", + headers={"Content-Disposition": 'attachment; filename="metadata.json"'}) + columns = sorted({key for r in records for key in r["fields"]}) + buffer = io.StringIO() + writer = csv.writer(buffer) + writer.writerow(["file", "handler", *columns]) + for r in records: + writer.writerow([_cell(r["file"]), _cell(r["handler"]), *(_cell(r["fields"].get(c)) for c in columns)]) + return Response("" + buffer.getvalue(), media_type="text/csv; charset=utf-8", + headers={"Content-Disposition": 'attachment; filename="metadata.csv"'}) + + +# ─────────────────────────── rename ─────────────────────────── + +class RenameRequest(BaseModel): + path: str + template: str = Field(min_length=1, max_length=200) # e.g. "{exif:DateTimeOriginal|%Y%m%d_%H%M%S}_{name}" + preview: bool = True + + +_TOKEN = re.compile(r"\{([^{}|]+)(?:\|([^{}]*))?\}") +_BAD_NAME = re.compile(r'[<>:"/\\|?*\x00-\x1f]') +_RESERVED = {"con", "prn", "aux", "nul", *(f"com{i}" for i in range(1, 10)), *(f"lpt{i}" for i in range(1, 10))} + + +def _new_name(path: Path, record, template: str) -> str: + by_key = {f.key: f for f in record.fields} + + def fill(match: re.Match) -> str: + key, fmt = match.group(1), match.group(2) + if key == "name": + return path.stem + if key == "ext": + return path.suffix.lstrip(".") + field = by_key.get(key) + if field is None or field.value in (None, "", []): + raise ApiError.one(422, ApiCode.INVALID_REQUEST, f"the file has no value for {{{key}}}") + value = field.value[0] if isinstance(field.value, list) else field.value + if field.field_type in ("datetime", "localdatetime"): + stamp = values.parse_datetime(value) if field.field_type == "datetime" else values.parse_localdatetime(value) + return stamp.strftime(fmt or "%Y%m%d_%H%M%S") + return str(value) + stem = _BAD_NAME.sub("_", _TOKEN.sub(fill, template)).strip(" .") + if not stem or stem.lower() in _RESERVED: + raise ApiError.one(422, ApiCode.INVALID_REQUEST, "the template gives an empty or reserved name") + if "{" in stem or "}" in stem: + raise ApiError.one(422, ApiCode.INVALID_REQUEST, "unknown or unbalanced placeholder in the template") + return stem + path.suffix + + +@router.post("/rename") +def rename_file(req: RenameRequest): + path = routes._valid_file(req.path) + with PATH_LOCKS.reading(path): + record = stable_read(path, lambda: routes._read_full_record(path)).record + new = path.with_name(_new_name(path, record, req.template)) + body = {"ok": True, "preview": req.preview, "old": str(path), "new": str(new), "renamed": False} + if new == path or req.preview: + return body + with routes._MUTATION_LOCK, PATH_LOCKS.writing(path): + if new.exists(): + raise ApiError.one(409, ErrorCode.CONCURRENT_MODIFICATION, f"{new.name} already exists") + try: + os.link(path, new) if os.name != "nt" else os.rename(path, new) + if os.name != "nt": + os.unlink(path) + except FileExistsError as exc: + raise ApiError.one(409, ErrorCode.CONCURRENT_MODIFICATION, f"{new.name} already exists") from exc + except OSError as exc: + raise ApiError.one(403, ApiCode.ACCESS_DENIED, routes.describe_error(exc)) from exc + body["renamed"] = True + return body diff --git a/python/api/routes.py b/python/api/routes.py index 07fccf2..fcf6096 100644 --- a/python/api/routes.py +++ b/python/api/routes.py @@ -89,7 +89,7 @@ MetadataValueErrors, TargetNotWritableError, describe_error, error_entry) from core.handlers.filesystem_handler import MTIME_KEY, FilesystemHandler, write_block_reason from core.models import MetadataField, MetadataRecord -from core import path_security +from core import path_security, sensitivity from core.path_security import (validate_file_path, validate_directory_path, secure_atomic_write, FileBusyError, NothingToCommit, PathSecurityError, is_active_transaction_dir, raise_if_unreadable, winerror_of) @@ -270,6 +270,13 @@ def health(): return {"status": "ok", "version": settings.version, "app": settings.app_name} +@router.get("/aliases") +def aliases(): + """Other names of common fields, for searching by what a person would type.""" + from core.aliases import ALIASES + return {"aliases": ALIASES} + + # ──────────────────────────────── /hash ────────────────────────────────────── _SUPPORTED_ALGORITHMS = {"md5", "sha1", "sha256", "sha512", "blake2b"} @@ -988,6 +995,17 @@ def _mutate(ctx: _Context, plan: _Plan) -> _Outcome: return _mutate_once(ctx, plan) +def _media_digest(ctx: _Context, path: Path) -> str | None: + """G5: digest of the media data as the format handler defines it (``None`` = not checked).""" + if ctx.fs_primary: + return None + try: + return ctx.handler.content_digest(path) + except Exception: + logger.warning("Media digest of %s failed", path, exc_info=True) + return None + + def _mutate_once(ctx: _Context, plan: _Plan) -> _Outcome: """ONE transaction: hooks on the temporary copy, verification, replace. @@ -1003,6 +1021,7 @@ def _mutate_once(ctx: _Context, plan: _Plan) -> _Outcome: def mutate(tmp: Path) -> None: before = {lay: _layer_values(ctx, tmp, lay)[0] for lay in layers} + media_before = _media_digest(ctx, tmp) if FORMAT in layers else None try: outcome.pre = {lay: _snapshot(ctx.handler_of(lay), tmp, plan.keys(lay)) for lay in layers} except Exception as exc: @@ -1018,6 +1037,9 @@ def mutate(tmp: Path) -> None: _call_hook(handler, "apply_write", tmp, plan.writes[lay], [f.key for f in plan.writes[lay]]) problems: list[MetadataValueError] = [] changed = False + if media_before is not None and _media_digest(ctx, tmp) != media_before: + problems.append(MetadataValueError(None, ErrorCode.LOSSY_WRITE, + "the save would have changed the media data of the file")) for lay in layers: after, after_errors = _layer_values(ctx, tmp, lay) written, deleted = {f.key for f in plan.writes[lay]}, set(plan.deletes[lay]) @@ -1148,6 +1170,65 @@ def delete_metadata(req: DeleteRequest): return _save(req.path, [], req.keys) +# ──────────────────────────────── /sanitize ────────────────────────────────── + +class SanitizeRequest(BaseModel): + """Prepare a file for sharing: remove what the chosen categories reveal.""" + path: str + categories: list[str] = Field(min_length=1) + preview: bool = True # only report what would be removed + keep: list[str] = Field(default_factory=list) # keys the user wants to keep anyway + + +def _sanitize_entry(f: MetadataField) -> dict: + return {"key": f.key, "label": f.label, "category": sensitivity.classify(f.key)} + + +def _sanitize_scan(ctx: _Context, categories: set[str], keep: set[str]) -> tuple[list[MetadataField], list[dict]]: + """Fields that can be removed and those of the categories that cannot (with the reason).""" + removable: list[MetadataField] = [] + remaining: list[dict] = [] + reason = ctx.record.read_only_reason + for f in ctx.record.fields: + if sensitivity.classify(f.key) not in categories or f.key in keep: + continue + if f.deletable and not ctx.not_writable: + removable.append(f) + else: + why = ctx.not_writable or (reason if not _is_filesystem_key(f.key) else None) \ + or "This field cannot be removed from this kind of file" + remaining.append({**_sanitize_entry(f), "reason": why}) + return removable, remaining + + +@router.post("/sanitize") +def sanitize_metadata(req: SanitizeRequest): + unknown = [c for c in req.categories if c not in sensitivity.CATEGORIES] + if unknown: + raise ApiError.one(422, ApiCode.INVALID_REQUEST, f"unknown categories: {', '.join(unknown)}") + categories, keep = set(req.categories), set(req.keep) + path = _valid_file(req.path) + with PATH_LOCKS.reading(path): + ctx = stable_read(path, lambda: _read_full_record(path)) + removable, remaining = _sanitize_scan(ctx, categories, keep) + body: dict = {"ok": True, "preview": req.preview, "will_remove": [_sanitize_entry(f) for f in removable], + "remaining": remaining, "undo_token": None, "undo_unavailable": None, "warnings": []} + if req.preview or not removable: + return body + saved = _save(req.path, [], [f.key for f in removable]) + # Check the result: what the chosen categories still reveal after the save. + with PATH_LOCKS.reading(path): + after = stable_read(path, lambda: _read_full_record(path)) + left_removable, left = _sanitize_scan(after, categories, keep) + body.update(undo_token=saved["undo_token"], undo_unavailable=saved["undo_unavailable"], + warnings=saved["warnings"], remaining=left, + removed=[_sanitize_entry(f) for f in removable if f.key not in {g.key for g in left_removable}]) + if left_removable: + body["warnings"] = [*body["warnings"], + "Some fields could not be removed: " + ", ".join(f.key for f in left_removable)] + return body + + # ──────────────────────────────── /undo ────────────────────────────────────── class UndoRequest(BaseModel): diff --git a/python/config.py b/python/config.py index fced168..2865369 100644 --- a/python/config.py +++ b/python/config.py @@ -1,7 +1,7 @@ from dataclasses import dataclass -VERSION = "0.4.1" +VERSION = "0.5.0" APP_NAME = "MetaLens" SIDECAR_HOST = "127.0.0.1" diff --git a/python/core/PATH_SECURITY_USAGE.md b/python/core/PATH_SECURITY_USAGE.md index c7940f6..c082e45 100644 --- a/python/core/PATH_SECURITY_USAGE.md +++ b/python/core/PATH_SECURITY_USAGE.md @@ -29,7 +29,7 @@ on the temporary path inside the callback. Hooks must modify only the path they must not open another transaction. Outside the API, `self._atomic_write(path, callback)` (used by the public `write`/`delete` wrappers) delegates to the same function. -The callback receives a temporary copy named `.ml_tmp` (original extension kept), +The callback receives a temporary copy named the same name as the target (inside the private `.metalens-*` directory; `temp_suffix` is kept only for compatibility), inside a private `.metalens-*` directory beside the target. It must modify only that copy and propagate failures. Never construct predictable temporary names or write directly to the original. diff --git a/python/core/aliases.py b/python/core/aliases.py new file mode 100644 index 0000000..0fef249 --- /dev/null +++ b/python/core/aliases.py @@ -0,0 +1,46 @@ +"""Other names under which a field is commonly known (search box of the metadata table). Keys never change.""" + +ALIASES: dict[str, list[str]] = { + "exif:DateTime": ["modify date", "modified", "date modified"], + "exif:DateTimeOriginal": ["date taken", "capture date", "shot date", "create date"], + "exif:DateTimeDigitized": ["create date", "digitized"], + "exif:ISOSpeedRatings": ["iso"], + "exif:ExposureBiasValue": ["exposure compensation"], + "exif:PixelXDimension": ["exif image width", "width"], + "exif:PixelYDimension": ["exif image height", "height"], + "exif:FNumber": ["aperture", "f-number"], + "exif:ExposureTime": ["shutter speed"], + "exif:Make": ["camera maker", "manufacturer"], + "exif:Model": ["camera model"], + "exif:Artist": ["author", "photographer", "creator"], + "exif:Copyright": ["rights"], + "exif:Software": ["program", "application"], + "exif:ImageDescription": ["caption", "description"], + "gps:Latitude": ["gps latitude", "position", "location"], + "gps:Longitude": ["gps longitude", "position", "location"], + "gps:Altitude": ["gps altitude", "elevation"], + "xmp:dc:creator": ["author", "artist"], + "xmp:dc:title": ["title"], + "xmp:dc:subject": ["keywords", "tags"], + "xmp:dc:description": ["caption", "description"], + "xmp:xmp:CreateDate": ["date created", "create date"], + "xmp:xmp:CreatorTool": ["software", "program"], + "iptc:By-line": ["author", "creator", "photographer"], + "iptc:Keywords": ["tags"], + "iptc:Caption-Abstract": ["caption", "description"], + "tag:TIT2": ["title", "song"], + "tag:TPE1": ["artist", "performer"], + "tag:TALB": ["album"], + "tag:TDRC": ["year", "recording date"], + "mp4:©nam": ["title"], + "mp4:©ART": ["artist"], + "pdf:Author": ["creator", "owner"], + "pdf:CreationDate": ["create date", "created"], + "pdf:ModDate": ["modify date", "modified"], + "pdf:Producer": ["software", "pdf producer"], + "office:author": ["creator", "owner"], + "office:last_modified_by": ["editor", "modified by"], + "office:created": ["create date", "date created"], + "qt:CreateDate": ["create date", "creation date", "date taken"], + "qt:mdta:com.apple.quicktime.location.ISO6709": ["gps", "position", "location"], +} diff --git a/python/core/base_handler.py b/python/core/base_handler.py index 99a6466..9e1f605 100644 --- a/python/core/base_handler.py +++ b/python/core/base_handler.py @@ -78,6 +78,11 @@ def delete(self, path: Path, keys: list[str]) -> None: def can_handle(self, path: Path) -> bool: return path.suffix.lower() in self.EXTENSIONS + def content_digest(self, path: Path) -> str | None: + """Digest of the media data of ``path`` (the picture, the audio, the video), ``None`` when + the handler cannot tell. A metadata write must never change it: the save is refused otherwise (G5).""" + return None + def writable_field(self, path: Path, key: str) -> MetadataField | None: """Server-owned schema for known writable fields absent from a file.""" return None diff --git a/python/core/ebml.py b/python/core/ebml.py new file mode 100644 index 0000000..44bcefd --- /dev/null +++ b/python/core/ebml.py @@ -0,0 +1,197 @@ +"""Bounded reader of Matroska / WebM (EBML) files: header values, tags, attachments, tracks. + +Elements are read where they are; a removed element is turned into a ``Void`` element of the same total size +(its payload zeroed), a text is rewritten over the old one when it fits and the rest is voided: nothing moves, +the clusters (the audio and video) are never touched. Files whose segment or clusters have an unknown size +(live recordings) are read-only. +""" +from __future__ import annotations + +import hashlib +from dataclasses import dataclass +from pathlib import Path + +EBML, SEGMENT, SEEKHEAD, INFO, TRACKS, CLUSTER, CUES, TAGS, ATTACHMENTS, CHAPTERS, VOID = ( + 0x1A45DFA3, 0x18538067, 0x114D9B74, 0x1549A966, 0x1654AE6B, 0x1F43B675, 0x1C53BB6B, 0x1254C367, 0x1941A469, + 0x1043A770, 0xEC) +TITLE, MUXING_APP, WRITING_APP, DATE_UTC, DURATION, TIMECODE_SCALE = 0x7BA9, 0x4D80, 0x5741, 0x4461, 0x4489, 0x2AD7B1 +TAG, SIMPLE_TAG, TAG_NAME, TAG_STRING, TAG_LANGUAGE = 0x7373, 0x67C8, 0x45A3, 0x4487, 0x447A +ATTACHED_FILE, FILE_NAME, FILE_MIME, FILE_DATA = 0x61A7, 0x466E, 0x4660, 0x465C +TRACK_ENTRY, TRACK_NUMBER, TRACK_TYPE, CODEC_ID, LANGUAGE, NAME = 0xAE, 0xD7, 0x83, 0x86, 0x22B59C, 0x536E +MAX_ELEMENTS = 200_000 +TEXT_INFO = {TITLE: "Title", MUXING_APP: "MuxingApp", WRITING_APP: "WritingApp"} + + +@dataclass(frozen=True) +class Element: + id: int + start: int + data_start: int + end: int + unknown_size: bool = False + + @property + def size(self) -> int: + return self.end - self.start + + +class EbmlError(ValueError): + pass + + +def _vint(buf, pos: int, keep_marker: bool) -> tuple[int | None, int]: + """(value, length) of the variable-length integer at ``pos``; ``None`` for the "unknown size" value.""" + if pos >= len(buf): + raise EbmlError("truncated") + first = buf[pos] + length = 1 + while length <= 8 and not first & (0x80 >> (length - 1)): + length += 1 + if length > 8 or pos + length > len(buf): + raise EbmlError("bad variable-length integer") + value = first if keep_marker else first & (0xFF >> length) + for b in buf[pos + 1:pos + length]: + value = (value << 8) | b + if not keep_marker and value == (1 << (7 * length)) - 1: + return None, length + return value, length + + +def children(buf, start: int, end: int) -> list[Element]: + """The elements laid out in ``buf[start:end]`` (an unreadable tail is dropped).""" + out: list[Element] = [] + pos = start + while pos < end and len(out) < MAX_ELEMENTS: + try: + ident, id_len = _vint(buf, pos, True) + size, size_len = _vint(buf, pos + id_len, False) + except EbmlError: + break + data_start = pos + id_len + size_len + unknown = size is None + stop = end if unknown else data_start + size + if stop > end: + break + out.append(Element(ident, pos, data_start, stop, unknown)) + pos = stop + return out + + +def _find(buf, parent_range: tuple[int, int], ident: int) -> list[Element]: + return [e for e in children(buf, *parent_range) if e.id == ident] + + +def _uint(buf, e: Element) -> int: + return int.from_bytes(buf[e.data_start:e.end], "big") + + +def _text(buf, e: Element) -> str: + return bytes(buf[e.data_start:e.end]).rstrip(b"\x00").decode("utf-8", "replace") + + +def _float(buf, e: Element) -> float | None: + import struct + raw = bytes(buf[e.data_start:e.end]) + if len(raw) == 4: + return struct.unpack(">f", raw)[0] + if len(raw) == 8: + return struct.unpack(">d", raw)[0] + return None + + +class Segment: + """The top-level structure of a file: the Segment and the elements the metadata lives in.""" + + def __init__(self, buf) -> None: + self.buf = buf + top = children(buf, 0, len(buf)) + if not top or top[0].id != EBML: + raise EbmlError("not a Matroska file") + seg = next((e for e in top if e.id == SEGMENT), None) + if seg is None: + raise EbmlError("no Segment") + self.segment = seg + self.items = children(buf, seg.data_start, seg.end) + self.live = seg.unknown_size or any(e.unknown_size for e in self.items) + header = children(buf, top[0].data_start, top[0].end) + doc = next((e for e in header if e.id == 0x4282), None) + self.doc_type = _text(buf, doc) if doc else "matroska" + + def first(self, ident: int) -> Element | None: + return next((e for e in self.items if e.id == ident), None) + + def all(self, ident: int) -> list[Element]: + return [e for e in self.items if e.id == ident] + + +def void_patch(buf, element: Element): + """``(offset, original, replacement)`` turning ``element`` into a Void element of the same total size.""" + total = element.size + for size_len in range(1, 9): + payload = total - 1 - size_len + if payload >= 0 and payload < (1 << (7 * size_len)) - 1: + marker = 1 << (7 * size_len) + header = bytes([VOID]) + (marker | payload).to_bytes(size_len, "big") + return element.start, bytes(buf[element.start:element.end]), header + bytes(payload) + raise EbmlError("the element is too small to be voided") + + +def info_values(buf, seg: Segment) -> dict[int, Element]: + info = seg.first(INFO) + return {e.id: e for e in children(buf, info.data_start, info.end)} if info else {} + + +def simple_tags(buf, seg: Segment) -> list[tuple[str, str, Element]]: + """(name, string, element) of every SimpleTag inside the Tags elements.""" + out = [] + for tags in seg.all(TAGS): + for tag in _find(buf, (tags.data_start, tags.end), TAG): + for simple in _find(buf, (tag.data_start, tag.end), SIMPLE_TAG): + kids = {e.id: e for e in children(buf, simple.data_start, simple.end)} + if TAG_NAME in kids and TAG_STRING in kids: + out.append((_text(buf, kids[TAG_NAME]), _text(buf, kids[TAG_STRING]), simple)) + return out + + +def attachments(buf, seg: Segment) -> list[tuple[str, str, bytes, Element]]: + out = [] + for block in seg.all(ATTACHMENTS): + for attached in _find(buf, (block.data_start, block.end), ATTACHED_FILE): + kids = {e.id: e for e in children(buf, attached.data_start, attached.end)} + name = _text(buf, kids[FILE_NAME]) if FILE_NAME in kids else "" + mime = _text(buf, kids[FILE_MIME]) if FILE_MIME in kids else "" + data = bytes(buf[kids[FILE_DATA].data_start:kids[FILE_DATA].end]) if FILE_DATA in kids else b"" + out.append((name, mime, data, attached)) + return out + + +def tracks(buf, seg: Segment) -> list[dict]: + out = [] + block = seg.first(TRACKS) + for entry in (_find(buf, (block.data_start, block.end), TRACK_ENTRY) if block else []): + kids = {e.id: e for e in children(buf, entry.data_start, entry.end)} + out.append({"number": _uint(buf, kids[TRACK_NUMBER]) if TRACK_NUMBER in kids else len(out) + 1, + "type": _uint(buf, kids[TRACK_TYPE]) if TRACK_TYPE in kids else 0, + "codec": _text(buf, kids[CODEC_ID]) if CODEC_ID in kids else "", + "language": _text(buf, kids[LANGUAGE]) if LANGUAGE in kids else "", + "name": _text(buf, kids[NAME]) if NAME in kids else ""}) + return out + + +def media_digest(path: Path) -> str | None: + """SHA-256 of the clusters (the audio and video data).""" + digest = hashlib.sha256() + with open(path, "rb") as stream: + try: + import mmap + buf = mmap.mmap(stream.fileno(), 0, access=mmap.ACCESS_READ) + except (ValueError, OSError): + return None + try: + for e in Segment(buf).all(CLUSTER): + digest.update(bytes(buf[e.start:e.end])) + return digest.hexdigest() + except EbmlError: + return None + finally: + buf.close() diff --git a/python/core/exif_codec.py b/python/core/exif_codec.py index 5982924..86d20be 100644 --- a/python/core/exif_codec.py +++ b/python/core/exif_codec.py @@ -35,7 +35,8 @@ from core import values from core.errors import ErrorCode, MetadataValueError -TYPE_SIZES = {1: 1, 2: 1, 3: 2, 4: 4, 5: 8, 6: 1, 7: 1, 8: 2, 9: 4, 10: 8, 11: 4, 12: 8, 13: 4} +# 129 = UTF-8 text (Exif 3.0): read as text, kept byte for byte on a rewrite. +TYPE_SIZES = {1: 1, 2: 1, 3: 2, 4: 4, 5: 8, 6: 1, 7: 1, 8: 2, 9: 4, 10: 8, 11: 4, 12: 8, 13: 4, 129: 1} _INT_FORMATS = {1: "B", 3: "H", 4: "L", 6: "b", 8: "h", 9: "l", 13: "L"} _INT_RANGES = {1: (0, 255), 3: (0, 65535), 4: (0, 2**32 - 1), 6: (-128, 127), 8: (-32768, 32767), 9: (-2**31, 2**31 - 1), 13: (0, 2**32 - 1)} @@ -105,6 +106,8 @@ ("0th", 0x010E), ("0th", 0x010F), ("0th", 0x0110), ("0th", 0x0131), ("0th", 0x013B), ("0th", 0x8298), ("0th", 0x0132), ("Exif", 0x9003), ("Exif", 0x9004), ("0th", 0x0112), ("Exif", 0x9286), *sorted(XP_TAGS), + # position: reference letters, coordinates (degrees, minutes, seconds) and altitude + ("GPS", 0x0001), ("GPS", 0x0002), ("GPS", 0x0003), ("GPS", 0x0004), ("GPS", 0x0005), ("GPS", 0x0006), ) _BINARY_COUNT_LIMIT = 64 @@ -222,15 +225,22 @@ def spec_type(ifd: str, tag: int) -> int | None: # ──────────────────────── parse ────────────────────────────────────────────── -def parse(tiff: bytes, *, with_ifd1: bool = True) -> ExifBlock: +# Magic of TIFF-based raw files whose IFDs are laid out like TIFF (Olympus ORF, Panasonic RW2/RWL). +RAW_MAGICS = {b"IIRO": "<", b"IIRS": "<", b"MMOR": ">", b"IIU\x00": "<"} + + +def parse(tiff: bytes, *, with_ifd1: bool = True, raw: bool = False) -> ExifBlock: """Parse a TIFF structure (``Exif\\0\\0`` already stripped). + ``raw=True`` also accepts the headers of ORF/RW2 raw files (read only: such a + block is never rewritten). + Raises ``ExifFormatError`` and nothing else, whatever the input bytes: a malformed block is a read error of the caller, never an unhandled exception. """ view = tiff if isinstance(tiff, memoryview) else memoryview(tiff) try: - return _parse(view, with_ifd1) + return _parse(view, with_ifd1, raw) except ExifFormatError: raise except (struct.error, IndexError, KeyError, ValueError, TypeError, OverflowError) as exc: @@ -248,7 +258,7 @@ def _pointer_value(entry: Entry, bo: str, what: str) -> int: return struct.unpack(bo + _INT_FORMATS[entry.type], entry.data)[0] -def _parse(data: memoryview, with_ifd1: bool) -> ExifBlock: +def _parse(data: memoryview, with_ifd1: bool, raw: bool = False) -> ExifBlock: size = len(data) if size < 8: raise ExifFormatError("EXIF block too short") @@ -257,6 +267,8 @@ def _parse(data: memoryview, with_ifd1: bool) -> ExifBlock: bo = "<" elif head == b"MM\x00*": bo = ">" + elif raw and head in RAW_MAGICS: + bo = RAW_MAGICS[head] else: raise ExifFormatError("invalid TIFF header") block = ExifBlock(bo) @@ -513,6 +525,8 @@ def decode(ifd: str, tag: int, entry: Entry, bo: str) -> Decoded: except UnicodeDecodeError: return binary return Decoded("str", text.rstrip("\x00"), editable=editable, kind="xp") + if t == 129: + return Decoded("str", values.decode_text(entry.data.rstrip(b"\x00"), ("utf-8",)), editable=False, kind="utf8") if t == 2: raw = entry.data.rstrip(b"\x00") if where in DATETIME_TAGS: @@ -670,7 +684,7 @@ def _encode_utf16(key: str, text: str, encoding: str) -> bytes: def swap_entry(entry: Entry, bo_from: str, bo_to: str) -> Entry: """Same entry in another byte order (snapshot restored into a block with another order).""" - if bo_from == bo_to or entry.type in (1, 2, 6, 7) or entry.type not in TYPE_SIZES: + if bo_from == bo_to or entry.type in (1, 2, 6, 7, 129) or entry.type not in TYPE_SIZES: return entry fmt = {3: "H", 4: "L", 5: "LL", 8: "h", 9: "l", 10: "ll", 11: "f", 12: "d", 13: "L"}[entry.type] vals = struct.unpack(bo_from + fmt * entry.count, entry.data) diff --git a/python/core/handlers/__init__.py b/python/core/handlers/__init__.py index 2c7b3e6..8abb349 100644 --- a/python/core/handlers/__init__.py +++ b/python/core/handlers/__init__.py @@ -5,6 +5,9 @@ from core.handlers.image_handler import ImageHandler from core.handlers.audio_handler import AudioHandler from core.handlers.video_handler import VideoHandler +from core.handlers.epub_handler import EpubHandler +from core.handlers.heif_handler import HeifHandler +from core.handlers.odf_handler import OdfHandler from core.handlers.pdf_handler import PdfHandler from core.handlers.office_handler import OfficeHandler from core.handlers.ole_handler import OleHandler @@ -14,6 +17,9 @@ HandlerRegistry.register(ImageHandler()) HandlerRegistry.register(AudioHandler()) HandlerRegistry.register(VideoHandler()) +HandlerRegistry.register(HeifHandler()) +HandlerRegistry.register(OdfHandler()) +HandlerRegistry.register(EpubHandler()) HandlerRegistry.register(PdfHandler()) HandlerRegistry.register(OfficeHandler()) HandlerRegistry.register(OleHandler()) diff --git a/python/core/handlers/_tag_codecs.py b/python/core/handlers/_tag_codecs.py index c291db1..a4f6058 100644 --- a/python/core/handlers/_tag_codecs.py +++ b/python/core/handlers/_tag_codecs.py @@ -498,6 +498,11 @@ def fields(self) -> list[MetadataField]: out.append(_text_field(key, label, urls, self.SOURCE, multi=isinstance(frame, id3.UrlFrameU))) elif fid in ("APIC", "GEOB"): + if fid == "APIC": + kind = getattr(frame.type, "name", str(frame.type)).replace("_", " ").capitalize() + label = f"{label} - {kind}" + (f" ({frame.desc})" if frame.desc else "") + elif frame.filename: + label = f"{label} - {frame.filename}" out.append(MetadataField.binary_field(key, label, frame.data, mime=frame.mime or None, source=self.SOURCE, raw_value=frame.data)) elif fid == "UFID": diff --git a/python/core/handlers/audio_handler.py b/python/core/handlers/audio_handler.py index 9cf950a..8cc2cda 100644 --- a/python/core/handlers/audio_handler.py +++ b/python/core/handlers/audio_handler.py @@ -100,6 +100,33 @@ def add(key, label, value, field_type): codec = getattr(info, "codec", None) if isinstance(codec, str) and codec: add(f"{prefix}:codec", "Codec", codec, "str") + # Format-specific stream details (MPEG audio, FLAC, Ogg): read-only, none changes with a tag write. + mpeg_version = getattr(info, "version", None) + layer = getattr(info, "layer", None) + if isinstance(mpeg_version, (int, float)) and not isinstance(mpeg_version, bool) and type(layer) is int: + add(f"{prefix}:mpeg_version", "MPEG Version", str(mpeg_version), "str") + add(f"{prefix}:mpeg_layer", "MPEG Layer", layer, "int") + mode = getattr(info, "mode", None) + if type(mode) is int: + add(f"{prefix}:channel_mode", "Channel Mode", + {0: "Stereo", 1: "Joint stereo", 2: "Dual channel", 3: "Mono"}.get(mode, str(mode)), "str") + if isinstance(getattr(info, "protected", None), bool): + add(f"{prefix}:crc_protected", "CRC Protected", info.protected, "bool") + bitrate_mode = getattr(info, "bitrate_mode", None) + if bitrate_mode is not None and getattr(bitrate_mode, "name", ""): + add(f"{prefix}:bitrate_mode", "Bitrate Mode", bitrate_mode.name, "str") + encoder = getattr(info, "encoder_info", None) + if isinstance(encoder, str) and encoder: + add(f"{prefix}:encoder", "Encoder", encoder, "str") + for attr, key, label in (("min_blocksize", "min_blocksize", "Min Block Size"), + ("max_blocksize", "max_blocksize", "Max Block Size"), + ("total_samples", "total_samples", "Total Samples")): + v = getattr(info, attr, None) + if type(v) is int and v > 0: + add(f"{prefix}:{key}", label, v, "int") + signature = getattr(info, "md5_signature", None) + if type(signature) is int and signature: + add(f"{prefix}:md5_signature", "Audio MD5", f"{signature:032x}", "str") return out @@ -118,16 +145,29 @@ def _open_codec(self, path: Path): def read(self, path: Path) -> MetadataRecord: fields: list[MetadataField] = [] errors: list[str] = [] + notes: list[str] = [] writable = False try: import mutagen + import mutagen.mp4 audio = mutagen.File(path, easy=False) if audio is None: errors.append("mutagen could not parse file") else: fields += stream_fields(audio.info, "audio") + if isinstance(audio, mutagen.mp4.MP4): + from core.handlers.qt_fields import qt_fields + try: + structure, notes = qt_fields(path) + fields += structure + except Exception as e: # extra information: never fail the read for it + notes.append(f"QuickTime structure not read ({type(e).__name__})") codec = codec_for(audio) writable = codec is not None and isinstance(audio, _writable_classes()) + vendor = getattr(audio.tags, "vendor", None) + if isinstance(vendor, str) and vendor: + fields.append(MetadataField("audio:vendor", "Encoder Vendor", vendor, raw_value=vendor, + editable=False, deletable=False, source="audio", computed=True)) if codec is not None: tag_fields = codec.fields() if not writable: @@ -137,7 +177,7 @@ def read(self, path: Path) -> MetadataRecord: except Exception as e: errors.append(f"Audio read error: {e}") writable = False - record = self._make_record(path, read_errors=errors, + record = self._make_record(path, read_errors=errors, warnings=notes, supports_write=writable and not errors, supports_delete=writable and not errors) if errors: diff --git a/python/core/handlers/epub_handler.py b/python/core/handlers/epub_handler.py new file mode 100644 index 0000000..a909c0c --- /dev/null +++ b/python/core/handlers/epub_handler.py @@ -0,0 +1,145 @@ +"""EPUB books: read-only metadata from the OPF package document.""" +from __future__ import annotations + +import xml.etree.ElementTree as ET +from pathlib import Path +from zipfile import BadZipFile, ZipFile + +from core import values +from core.base_handler import BaseMetadataHandler +from core.handlers.xml_package import Mutations +from core.models import MetadataField, MetadataRecord +from core.zipxml import PackageError, local, read_xml + +NS_CONTAINER = "{urn:oasis:names:tc:opendocument:xmlns:container}" +NS_DC = "http://purl.org/dc/elements/1.1/" + +_DC = {"title": "Title", "creator": "Author", "contributor": "Contributor", "publisher": "Publisher", + "language": "Language", "identifier": "Identifier", "subject": "Subject", "description": "Description", + "rights": "Rights", "date": "Publication Date", "type": "Type", "source": "Source", "relation": "Relation", + "coverage": "Coverage", "format": "Format"} +_MULTI = {"title", "creator", "contributor", "subject", "identifier", "language"} + + +def _field(key: str, label: str, value, field_type: str = "str", multi: bool = False) -> MetadataField: + return MetadataField(f"epub:{key}", label, value, raw_value=value, editable=False, deletable=False, + field_type=field_type, source="epub", multi=multi) + + +def _opf_path(archive: ZipFile) -> str: + container = read_xml(archive, "META-INF/container.xml") + for node in container.iter(f"{NS_CONTAINER}rootfile"): + path = node.get("full-path") + if path and ".." not in path.split("/") and not path.startswith("/"): + return path + raise PackageError("container.xml names no package document") + + +def opf_fields(root) -> list[MetadataField]: + fields: list[MetadataField] = [] + version = root.get("version") + if version: + fields.append(_field("version", "EPUB Version", version)) + metadata = next((c for c in root if local(c.tag) == "metadata"), None) + if metadata is None: + return fields + collected: dict[str, list[str]] = {} + modified = None + for node in metadata: + text = (node.text or "").strip() + if node.tag.startswith("{" + NS_DC + "}") and text: + collected.setdefault(local(node.tag), []).append(text) + elif local(node.tag) == "meta" and node.get("property") == "dcterms:modified" and text: + modified = text + for name, label in _DC.items(): + if name in collected: + many = name in _MULTI or len(collected[name]) > 1 + fields.append(_field(name, label, collected[name] if many else collected[name][0], "str", many)) + if modified: + try: + fields.append(_field("modified", "Last Modified", + values.format_datetime(values.parse_datetime(modified)), "datetime")) + except Exception: + fields.append(_field("modified", "Last Modified", modified)) + return fields + + +_REQUIRED = {"version", "identifier", "title", "language"} # a book without them is not valid + + +def _metadata_node(root): + return next((c for c in root if local(c.tag) == "metadata"), None) + + +class EpubHandler(Mutations, BaseMetadataHandler): + NAME = "epub" + EXTENSIONS = frozenset({".epub"}) + PRIORITY = 17 + + # ── where a key lives in the package document ── + def part_name(self, archive) -> str: + return _opf_path(archive) + + def removable(self, key: str) -> bool: + return key[len("epub:"):] not in _REQUIRED + + def locate(self, root, key: str) -> list[tuple]: + meta = _metadata_node(root) + short = key[len("epub:"):] + hits = [] + for node in (meta if meta is not None else []): + if short == "modified" and local(node.tag) == "meta" and node.get("property") == "dcterms:modified": + hits.append((meta, node, None)) + elif node.tag == "{" + NS_DC + "}" + short: + hits.append((meta, node, None)) + return hits + + def make(self, root, key: str, value) -> list: + meta = _metadata_node(root) + short = key[len("epub:"):] + if meta is None or short == "version": + return [] + out = [] + for v in (value if isinstance(value, list) else [value]): + if short == "modified": + element = ET.Element("{http://www.idpf.org/2007/opf}meta", {"property": "dcterms:modified"}) + elif short in _DC: + element = ET.Element("{%s}%s" % (NS_DC, short)) + else: + return [] + element.text = str(v) + out.append((meta, element)) + return out + + def read(self, path: Path) -> MetadataRecord: + fields: list[MetadataField] = [] + errors: list[str] = [] + warnings: list[str] = [] + try: + with ZipFile(path) as archive: + fields = opf_fields(read_xml(archive, _opf_path(archive))) + except (BadZipFile, PackageError, OSError) as exc: + errors.append(f"EPUB read error: {exc}") + writable = not errors + for f in fields: + short = f.key[len("epub:"):] + f.deletable = writable and short not in _REQUIRED + f.editable = writable and short != "version" and f.field_type == "str" + record = self._make_record(path, read_errors=errors, warnings=warnings, + supports_write=writable, supports_delete=writable) + record.fields = fields + record.dedupe_keys() + return record + + def writable_fields(self, path: Path, record) -> list[MetadataField]: + if not record.format_supports_write or record.read_errors: + return [] + present = {f.key for f in record.fields} + return [MetadataField(f"epub:{name}", label, None, source="epub", multi=name in _MULTI) + for name, label in _DC.items() if f"epub:{name}" not in present and name not in _REQUIRED] + + def write(self, path: Path, fields: list[MetadataField]) -> None: + self._atomic_write(path, lambda tmp: self.apply_write(tmp, fields)) + + def delete(self, path: Path, keys: list[str]) -> None: + self._atomic_write(path, lambda tmp: self.apply_delete(tmp, keys)) diff --git a/python/core/handlers/heif_handler.py b/python/core/handlers/heif_handler.py new file mode 100644 index 0000000..8693202 --- /dev/null +++ b/python/core/handlers/heif_handler.py @@ -0,0 +1,194 @@ +"""HEIC / HEIF / AVIF images: read-only metadata from the ISO base media structure. + +The still image is never decoded. The ``meta`` box is walked for the item table +(``iinf``), the item locations (``iloc``) and the image properties (``ipco`` / +``ipma``): size and rotation of the primary image, EXIF and XMP items and the ICC +profile. Item payloads are read from their ``iloc`` extents, bounded in size. +""" +from __future__ import annotations + +import struct +from pathlib import Path + +from core import exif_codec, icc, xmp +from core.base_handler import BaseMetadataHandler +from core.handlers.image_handler import ImageHandler, _HeifState, _exif_fields, _xmp_fields +from core.heif import MAX_EDIT_BYTES, _XMP_MIME, _items, _meta_children, _payload, _primary_properties, metadata_ranges +from core.isobmff import Reader, open_reader +from core.models import MetadataField, MetadataRecord + + +class HeifHandler(BaseMetadataHandler): + NAME = "heif" + EXTENSIONS = frozenset({".heic", ".heif", ".hif", ".avif"}) + PRIORITY = 12 + + def read(self, path: Path) -> MetadataRecord: + fields: list[MetadataField] = [] + errors: list[str] = [] + warnings: list[str] = [] + try: + fields = self._fields(path, warnings) + except Exception as exc: # reading never raises: damaged data is a read error + errors.append(f"HEIF read error: {type(exc).__name__}: {exc}") + removable = self._editable(path) and not errors + record = self._make_record(path, read_errors=errors, warnings=warnings, + supports_write=removable, supports_delete=removable) + record.fields = fields + if not removable: + record.read_only_reason = "The metadata of this HEIF/AVIF file is not stored in a way that can be changed" + for f in fields: + f.editable = f.deletable = False + else: + try: + record.addable_fields = self.writable_fields(path, record) + except Exception: # the schema is optional: never fail the read for it + record.addable_fields = [] + record.dedupe_keys() + return record + + @staticmethod + def _editable(path: Path) -> bool: + """The Exif or XMP item is stored in one piece, so it can be changed.""" + if path.stat().st_size > MAX_EDIT_BYTES: + return False + return bool(metadata_ranges(path.read_bytes())) + + # ── mutation hooks: EXIF entries and XMP are changed in place (the image items never move) ── + def apply_delete(self, tmp: Path, keys: list[str]) -> None: + ImageHandler().apply_delete(tmp, keys) + + def snapshot(self, tmp: Path, keys: list[str]): + return ImageHandler().snapshot(tmp, keys) + + def restore(self, tmp: Path, snap) -> None: + ImageHandler().restore(tmp, snap) + + def canonical(self, field: MetadataField, value): + return ImageHandler().canonical(field, value) + + def apply_write(self, tmp: Path, fields: list[MetadataField]) -> None: + ImageHandler().apply_write(tmp, fields) + + def writable_fields(self, path: Path, record) -> list[MetadataField]: + """EXIF entries (and XMP properties, when the file has an XMP item) that can be added.""" + from core import jpeg_segments + if record.read_errors or path.stat().st_size > MAX_EDIT_BYTES: + return [] + try: + state = _HeifState.load(path.read_bytes()) + except jpeg_segments.JpegFormatError: + return [] + present = {f.key for f in record.fields} + return ImageHandler._tiff_addable(ImageHandler()._addable(state, present)) + + def writable_field(self, path: Path, key: str) -> MetadataField | None: + record = self.read(path) + return next((f for f in record.addable_fields if f.key == key), None) + + def content_digest(self, path: Path) -> str | None: + """SHA-256 of the data of the image items (everything except the Exif and XMP items), wherever + the metadata items are stored.""" + import hashlib + data = Path(path).read_bytes() + rd = Reader(data) + meta, children = _meta_children(rd) + if meta is None: + return None + idat = next((b for b in children if b.type == b"idat"), None) + digest = hashlib.sha256() + for item in sorted(_items(rd, children).values(), key=lambda i: i.item_id): + if item.kind == "Exif" or (item.kind == "mime" and item.content_type == _XMP_MIME): + continue + payload = _payload(rd, item, idat) + if payload is None and item.extents: + return None + digest.update(struct.pack(">II", item.item_id, len(payload or b""))) + digest.update(payload or b"") + return digest.hexdigest() + + def _fields(self, path: Path, warnings: list[str]) -> list[MetadataField]: + fields: list[MetadataField] = [] + + def add(key, label, value, field_type="str", multi=False): + fields.append(MetadataField(key, label, value, raw_value=value, editable=False, deletable=False, + field_type=field_type, source="heif", multi=multi)) + + with open_reader(path) as rd: + ftyp = rd.find(None, b"ftyp") + if ftyp is not None and ftyp.end - ftyp.data_start >= 8: + major = bytes(rd.buf[ftyp.data_start:ftyp.data_start + 4]).decode("latin-1").strip() + add("heif:MajorBrand", "Major Brand", major) + brands = [bytes(rd.buf[p:p + 4]).decode("latin-1").strip() + for p in range(ftyp.data_start + 8, ftyp.end - 3, 4)] + if brands: + add("heif:CompatibleBrands", "Compatible Brands", brands, multi=True) + meta, children = _meta_children(rd) + if meta is None: + warnings.append("No metadata box found") + return fields + props = _primary_properties(rd, children) + brands = {f.value for f in fields if f.key == "heif:MajorBrand"} + for f in fields: + if f.key == "heif:CompatibleBrands": + brands |= set(f.value) + fmt = "AVIF" if brands & {"avif", "avis"} else "HEIF" + fields.insert(0, MetadataField("img:format", "Format", fmt, raw_value=fmt, editable=False, + deletable=False, source="image")) + if "width" in props: + fields.append(MetadataField("img:width", "Width (px)", props["width"], editable=False, + deletable=False, field_type="int", source="image")) + fields.append(MetadataField("img:height", "Height (px)", props["height"], editable=False, + deletable=False, field_type="int", source="image")) + if "rotation" in props: + add("heif:Rotation", "Rotation (°)", props["rotation"], "int") + idat = next((b for b in children if b.type == b"idat"), None) + items = _items(rd, children) + edited = self._state_fields(path, warnings) + for item in ([] if edited is not None else items.values()): + data = None + if item.kind == "Exif": + data = _payload(rd, item, idat) + if data and len(data) > 4: + skip = struct.unpack_from(">I", data, 0)[0] + try: + fields += _exif_fields(exif_codec.parse(data[4 + skip:], with_ifd1=False), False) + except exif_codec.ExifFormatError as exc: + warnings.append(f"EXIF not readable: {exc}") + elif item.kind == "mime" and item.content_type == _XMP_MIME: + data = _payload(rd, item, idat) + if data: + try: + fields += _xmp_fields(xmp.XmpPacket(data), False) + except xmp.XmpFormatError as exc: + warnings.append(f"XMP not readable: {exc}") + if edited is not None: + fields += edited + for prop in (props.get("properties") or []): + if prop.type == b"colr" and prop.end - prop.data_start > 4: + if bytes(rd.buf[prop.data_start:prop.data_start + 4]) in (b"prof", b"rICC"): + fields += icc.fields(bytes(rd.buf[prop.data_start + 4:prop.end])) + return fields + + @staticmethod + def _state_fields(path: Path, warnings: list[str]) -> list[MetadataField] | None: + """EXIF / GPS / XMP fields with their removable flags, or ``None`` when the layout cannot be + changed in place (the fields are then read straight from the items, read-only).""" + from core import jpeg_segments + if path.stat().st_size > MAX_EDIT_BYTES: + return None + data = path.read_bytes() + if not metadata_ranges(data): + return None + try: + state = _HeifState.load(data) + except jpeg_segments.JpegFormatError: + return None + warnings += state.warnings + return ImageHandler._state_fields(state, True) + + def write(self, path: Path, fields: list[MetadataField]) -> None: + self._atomic_write(path, lambda tmp: self.apply_write(tmp, fields)) + + def delete(self, path: Path, keys: list[str]) -> None: + self._atomic_write(path, lambda tmp: self.apply_delete(tmp, keys)) diff --git a/python/core/handlers/image_handler.py b/python/core/handlers/image_handler.py index e82bccd..be2a7c0 100644 --- a/python/core/handlers/image_handler.py +++ b/python/core/handlers/image_handler.py @@ -34,17 +34,28 @@ import struct import zlib from dataclasses import dataclass, field +from datetime import datetime, timezone from pathlib import Path +from types import SimpleNamespace from typing import Any -from core import exif_codec, iptc, jpeg_segments, values, xmp +from core import exif_codec, icc, image_containers, iptc, jpeg_segments, makernote, sniff, tiff_inplace, values, xmp from core.base_handler import ABSENT, BaseMetadataHandler, Snapshot from core.errors import ErrorCode, MetadataValueError, MetadataValueErrors +from core.heif import is_heif_brand, metadata_layout from core.models import MetadataField, MetadataRecord -_READ_ONLY_EXTS = frozenset({".cr2", ".nef", ".arw", ".dng", ".orf", ".rw2", ".pef", ".srw"}) +_READ_ONLY_EXTS = frozenset({".cr2", ".nef", ".arw", ".dng", ".orf", ".rw2", ".pef", ".srw", + ".rwl", ".nrw", ".srf", ".sr2", ".3fr", ".erf", ".mos", ".kdc", ".dcr", ".mef", ".iiq", + ".cr3", ".raf"}) JPEG_EXTS = frozenset({".jpg", ".jpeg"}) + +def _jpeg_name(path: Path) -> bool: + """A JPEG by name: ``.jpg``/``.jpeg`` or a file without extension whose bytes are a JPEG.""" + suffix = path.suffix.lower() + return suffix in JPEG_EXTS or (suffix == "" and sniff.detect(path) == "jpeg") + MAKERNOTE_WARNING = ("EXIF MakerNote present: it is kept byte for byte at its original offset when " "EXIF is rewritten; vendor data stored outside the MakerNote itself is not tracked") @@ -209,6 +220,8 @@ def _namespace(key: str) -> str: return "xmp" if key.startswith("iptc:"): return "iptc" + if key.startswith("png:"): + return "png" raise MetadataValueError(key, ErrorCode.UNKNOWN_FIELD) @@ -419,6 +432,572 @@ def to_bytes(self, size_keys: dict[str, str | None]) -> bytes: return self.jpeg.to_bytes() +class _PngState(_JpegState): + """Metadata blocks of a PNG: EXIF in ``eXIf``, XMP in an ``iTXt`` chunk; no IPTC. + + Reuses the editing model of ``_JpegState`` (``exif`` / ``xmp`` blocks, ``touched``, ``exact``): only the + container I/O differs. Text chunks are removed through ``png`` (never edited).""" + png: image_containers.PngFile + + @classmethod + def load(cls, data: bytes, *, tolerant: bool = False) -> "_PngState": + try: + png = image_containers.PngFile.parse(data) + except image_containers.ContainerError as exc: + raise jpeg_segments.JpegFormatError(f"PNG structure error: {exc}") from exc + state = cls(jpeg_segments.JpegFile(b"")) + state.png = png + state.iptc_writable = False + state._load_exif() + state._load_xmp() + return state + + def read_only_reason(self) -> str | None: + return None + + def segment(self, ns: str) -> tuple[bytes | None, int | None]: + if ns == "exif": + found = self.png.index_of(b"eXIf") + return (self.png.chunks[found[0]][1], found[0]) if found else (None, None) + if ns == "xmp": + found = self.png.xmp_indexes() + return (self.png.xmp_packet(found[0]), found[0]) if found else (None, None) + return None, None + + def _load_exif(self) -> None: + found = self.png.index_of(b"eXIf") + if not found: + return + if len(found) > 1: + self.warnings.append(f"{len(found)} EXIF chunks: only the first is used; the duplicates are " + "removed on the next EXIF save") + try: + self.exif = exif_codec.parse(_strip_exif_header(self.png.chunks[found[0]][1])) + except exif_codec.ExifFormatError as exc: + self.errors.append(f"EXIF read error: {exc}") + self.exif_writable = False + return + if self.exif.rewrite_problems: + self.exif_writable = False + self.warnings.append("EXIF is read-only: " + "; ".join(self.exif.rewrite_problems)) + + def _load_xmp(self) -> None: + found = self.png.xmp_indexes() + if not found: + return + if len(found) > 1: + self.xmp_writable = False + self.warnings.append(f"{len(found)} XMP chunks: XMP is read-only") + packet = self.png.xmp_packet(found[0]) + try: + if packet is None: + raise xmp.XmpFormatError("the chunk cannot be decoded") + self.xmp = xmp.XmpPacket(packet) + except xmp.XmpFormatError as exc: + self.xmp_writable = False + self.warnings.append(f"XMP not readable, left untouched: {exc}") + + def _load_iptc(self) -> None: + return + + def to_bytes(self, size_keys: dict[str, str | None]) -> bytes: + if "exif" in self.touched: + tiff = self.exact.get("exif") + if tiff is None and self.exif is not None and (any(self.exif.ifds.values()) or self.exif.thumbnail): + tiff = exif_codec.serialize(self.exif) + self.png.set_exif(tiff, self.hints.get("exif")) + if "xmp" in self.touched and self.xmp is not None: + packet = self.exact.get("xmp") + if packet is None and not self.xmp.is_blank(): + packet = self.xmp.to_bytes() + self.png.set_xmp(packet, self.hints.get("xmp")) + return self.png.to_bytes() + + +class _WebpState(_JpegState): + """Metadata blocks of a WebP: ``EXIF`` and ``XMP `` chunks; no IPTC.""" + webp: image_containers.WebpFile + + @classmethod + def load(cls, data: bytes, *, tolerant: bool = False) -> "_WebpState": + try: + webp = image_containers.WebpFile.parse(data) + except image_containers.ContainerError as exc: + raise jpeg_segments.JpegFormatError(f"WebP structure error: {exc}") from exc + state = cls(jpeg_segments.JpegFile(b"")) + state.webp = webp + state.iptc_writable = False + state._load_exif() + state._load_xmp() + return state + + def read_only_reason(self) -> str | None: + if self.webp.find(b"ANIM") or self.webp.find(b"ANMF"): + return "Animated WebP files are read-only" + return None + + def segment(self, ns: str) -> tuple[bytes | None, int | None]: + found = self.webp.find({"exif": b"EXIF", "xmp": b"XMP "}.get(ns, b"")) + return (self.webp.chunks[found[0]][1], found[0]) if found else (None, None) + + def _load_exif(self) -> None: + found = self.webp.find(b"EXIF") + if not found: + return + if len(found) > 1: + self.warnings.append(f"{len(found)} EXIF chunks: only the first is used") + try: + self.exif = exif_codec.parse(_strip_exif_header(self.webp.chunks[found[0]][1])) + except exif_codec.ExifFormatError as exc: + self.errors.append(f"EXIF read error: {exc}") + self.exif_writable = False + return + if self.exif.rewrite_problems: + self.exif_writable = False + self.warnings.append("EXIF is read-only: " + "; ".join(self.exif.rewrite_problems)) + + def _load_xmp(self) -> None: + found = self.webp.find(b"XMP ") + if not found: + return + if len(found) > 1: + self.xmp_writable = False + self.warnings.append(f"{len(found)} XMP chunks: XMP is read-only") + try: + self.xmp = xmp.XmpPacket(self.webp.chunks[found[0]][1]) + except xmp.XmpFormatError as exc: + self.xmp_writable = False + self.warnings.append(f"XMP not readable, left untouched: {exc}") + + def _load_iptc(self) -> None: + return + + def to_bytes(self, size_keys: dict[str, str | None]) -> bytes: + try: + if "exif" in self.touched: + tiff = self.exact.get("exif") + if tiff is None and self.exif is not None and (any(self.exif.ifds.values()) or self.exif.thumbnail): + tiff = exif_codec.serialize(self.exif) + self.webp.set_exif(tiff) + if "xmp" in self.touched and self.xmp is not None: + packet = self.exact.get("xmp") + if packet is None and not self.xmp.is_blank(): + packet = self.xmp.to_bytes() + self.webp.set_xmp(packet) + except image_containers.ContainerError as exc: + raise MetadataValueError(None, ErrorCode.UNSUPPORTED, f"WebP cannot take this metadata: {exc}") from exc + return self.webp.to_bytes() + + +_XMP_UUID = bytes.fromhex("be7acfcb97a942e89c71999491e3afac") + + +def _mp4_xmp_location(data: bytes) -> tuple[int, int] | None: + """(start, end) of the XMP packet of an MP4-family file: a ``uuid`` atom at the root or an ``XMP_`` atom in ``udta``.""" + from core.isobmff import Reader + rd = Reader(data) + for box in rd.children(): + if box.type == b"uuid" and box.usertype == _XMP_UUID: + return box.data_start, box.end + moov = rd.find(None, b"moov") + udta = rd.find(moov, b"udta") if moov is not None else None + if udta is not None: + for box in rd.children(udta.data_start, udta.end): + if box.type == b"XMP_": + return box.data_start, box.end + return None + + +class _Mp4State(_JpegState): + """The XMP packet of an MP4-family file. Changed in place: the packet never grows, the atoms around it + stay where they are, so no media offset changes.""" + data: bytes + location: tuple[int, int] | None + + @classmethod + def load(cls, data: bytes, *, tolerant: bool = False) -> "_Mp4State": + state = cls(jpeg_segments.JpegFile(b"")) + state.data = data + state.iptc_writable = False + state.exif_writable = False # EXIF is not stored in these files + state.location = _mp4_xmp_location(data) + state._load_xmp() + if state.location is None: + state.xmp_writable = True # no packet yet: a first property is stored in a new atom + return state + + def read_only_reason(self) -> str | None: + return None + + def segment(self, ns: str) -> tuple[bytes | None, int | None]: + if ns == "xmp" and self.location is not None: + return self.data[self.location[0]:self.location[1]], 0 + return None, None + + def _load_exif(self) -> None: + return + + def _load_iptc(self) -> None: + return + + def _load_xmp(self) -> None: + if self.location is None: + self.xmp_writable = False + return + try: + self.xmp = xmp.XmpPacket(self.data[self.location[0]:self.location[1]]) + except xmp.XmpFormatError as exc: + self.xmp_writable = False + self.warnings.append(f"XMP not readable, left untouched: {exc}") + + def to_bytes(self, size_keys: dict[str, str | None]) -> bytes: + if "xmp" in self.touched and self.xmp is not None and self.location is None and not self.xmp.is_blank(): + return self._appended(self.xmp.to_bytes(), clear=None) + if "xmp" not in self.touched or self.xmp is None or self.location is None: + return self.data + start, end = self.location + packet = self.exact.get("xmp") + if packet is None: + packet = (xmp.XmpPacket.new() if self.xmp.is_blank() else self.xmp).to_bytes() + if len(packet) > end - start: + # does not fit: a new atom at the end of the file, the old one is cleared (nothing moves) + return self._appended(packet, clear=(start, end)) + return self.data[:start] + packet + b" " * (end - start - len(packet)) + self.data[end:] + + def _appended(self, packet: bytes, clear: tuple[int, int] | None) -> bytes: + out = bytearray(self.data) + if clear is not None: + start, end = clear + # the old atom becomes a ``free`` atom (so that it is not found again) and its content is cleared + header = start - 24 if bytes(out[start - 20:start - 16]) == b"uuid" else start - 8 + out[header + 4:header + 8] = b"free" + out[header + 8:end] = bytes(end - header - 8) + out += struct.pack(">I", 24 + len(packet)) + b"uuid" + _XMP_UUID + packet + return bytes(out) + + +_TIFF_EXTS = frozenset({".tif", ".tiff"}) | _READ_ONLY_EXTS + + +class _TiffState(_JpegState): + """A TIFF / DNG / TIFF-based RAW: entries are removed in place and the XMP packet (tag 700) is reduced + in place, so nothing in the file moves. Nothing is added or edited.""" + buf: bytearray + + @classmethod + def load(cls, data: bytes, *, tolerant: bool = False) -> "_TiffState": + state = cls(jpeg_segments.JpegFile(b"")) + state.buf = bytearray(data) + state.iptc_writable = False + try: + tiff_inplace.parse(state.buf) + except (tiff_inplace.TiffError, struct.error) as exc: + raise jpeg_segments.JpegFormatError(f"TIFF structure error: {exc}") from exc + state._load_exif() + state._load_xmp() + state._original = {name: dict(entries) for name, entries in state.exif.ifds.items()} if state.exif else {} + return state + + def read_only_reason(self) -> str | None: + return None + + def _location(self) -> tuple[int, int] | None: + _, ifds = tiff_inplace.parse(self.buf) + entry = next((e for e in ifds["0th"].entries if e.tag == 700), None) + if entry is None or entry.data_at is None or entry.type not in (1, 7): + return None + return entry.data_at, entry.data_at + entry.size + + def segment(self, ns: str) -> tuple[bytes | None, int | None]: + if ns == "xmp": + location = self._location() + if location is not None: + return bytes(self.buf[location[0]:location[1]]), 0 + return None, None + + def _load_exif(self) -> None: + try: + self.exif = exif_codec.parse(bytes(self.buf), with_ifd1=False, raw=True) + except exif_codec.ExifFormatError as exc: + self.errors.append(f"EXIF read error: {exc}") + self.exif_writable = False + + def _load_xmp(self) -> None: + location = self._location() + if location is None: + self.xmp_writable = not any(e.tag == 700 for e in tiff_inplace.parse(self.buf)[1]["0th"].entries) + return + try: + self.xmp = xmp.XmpPacket(bytes(self.buf[location[0]:location[1]])) + except xmp.XmpFormatError as exc: + self.xmp_writable = False + self.warnings.append(f"XMP not readable, left untouched: {exc}") + + def _load_iptc(self) -> None: + return + + def _apply_exif_changes(self) -> None: + """Write the entries that differ from the file: over the old value when it fits, else appended.""" + if self.exif is None: + return + try: + for name in ("0th", "Exif", "GPS", "Interop"): + for tag, entry in sorted(self.exif.ifds[name].items()): + if self._original.get(name, {}).get(tag) != entry: + if not tiff_inplace.editable(name, tag): + raise MetadataValueError(exif_codec.key_for(name, tag), ErrorCode.READ_ONLY) + tiff_inplace.set_entry(self.buf, name, tag, entry.type, entry.count, entry.data) + except tiff_inplace.TiffError as exc: + raise MetadataValueError(None, ErrorCode.UNSUPPORTED, f"the EXIF change cannot be stored: {exc}") from exc + + def _apply_xmp(self) -> None: + packet = self.exact.get("xmp") + if packet is None and self.xmp.is_blank(): + tiff_inplace.remove(self.buf, "0th", {700}, force=True) # nothing left: drop the packet + return + if packet is None: + packet = self.xmp.to_bytes() + location = self._location() + try: + if location is not None and len(packet) <= location[1] - location[0]: + start, end = location + self.buf[start:end] = packet + b" " * (end - start - len(packet)) + else: + tiff_inplace.set_entry(self.buf, "0th", 700, 1, len(packet), packet) + except tiff_inplace.TiffError as exc: + raise MetadataValueError(None, ErrorCode.UNSUPPORTED, f"the XMP change cannot be stored: {exc}") from exc + + def to_bytes(self, size_keys: dict[str, str | None]) -> bytes: + if "exif" in self.touched: + self._apply_exif_changes() + if "xmp" in self.touched and self.xmp is not None: + self._apply_xmp() + return bytes(self.buf) + + +class _HeifState(_TiffState): + """HEIC / HEIF / AVIF: the TIFF part of the Exif item and the XMP item are changed where they are; an item + that outgrows its place is written again at the end of the file as a new ``mdat`` atom and its entry in the + item location table is pointed there (the old bytes are zeroed). Image items and every other offset stay.""" + data: bytearray + layout: dict + + @classmethod + def load(cls, data: bytes, *, tolerant: bool = False) -> "_HeifState": + layout = metadata_layout(data) + if not layout: + raise jpeg_segments.JpegFormatError("HEIF structure error: no Exif or XMP item stored in one piece") + state = cls(jpeg_segments.JpegFile(b"")) + state.data = bytearray(data) + state.layout = layout + exif = layout.get("exif") + state.buf = bytearray(data[exif["range"][0]:exif["range"][1]]) if exif else bytearray() + state.iptc_writable = False + if exif: + state._load_exif() + state._original = {name: dict(entries) for name, entries in state.exif.ifds.items()} if state.exif else {} + else: + state._original = {} + state._load_xmp() + return state + + def _xmp_range(self) -> tuple[int, int] | None: + item = self.layout.get("xmp") + return item["range"] if item else None + + def _location(self) -> tuple[int, int] | None: + return self._xmp_range() + + def segment(self, ns: str) -> tuple[bytes | None, int | None]: + if ns == "xmp" and self._xmp_range() is not None: + start, end = self._xmp_range() + return bytes(self.data[start:end]), 0 + return None, None + + def _load_xmp(self) -> None: + located = self._xmp_range() + if located is None: + self.xmp_writable = False # no XMP item: this file has nowhere to put XMP properties + return + try: + self.xmp = xmp.XmpPacket(bytes(self.data[located[0]:located[1]])) + except xmp.XmpFormatError as exc: + self.xmp_writable = False + self.warnings.append(f"XMP not readable, left untouched: {exc}") + + def _move_item(self, name: str, payload: bytes) -> None: + """Write ``payload`` as the new content of the item ``name`` at the end of the file.""" + entry = self.layout[name] + offset_at, offset_size, length_at, length_size, base = entry["loc"] + start = len(self.data) + 8 + if start - base >= 1 << (8 * offset_size) or len(payload) >= 1 << (8 * length_size): + raise MetadataValueError(None, ErrorCode.TOO_LARGE, "the item location table cannot address the new item") + old_start, old_end = entry["item"] + self.data.extend(struct.pack(">I", 8 + len(payload)) + b"mdat" + payload) + self.data[old_start:old_end] = bytes(old_end - old_start) + self.data[offset_at:offset_at + offset_size] = (start - base).to_bytes(offset_size, "big") + self.data[length_at:length_at + length_size] = len(payload).to_bytes(length_size, "big") + entry["item"] = (start, start + len(payload)) + entry["range"] = (start + (4 if name == "exif" else 0), start + len(payload)) + + def to_bytes(self, size_keys: dict[str, str | None]) -> bytes: + if "exif" in self.touched: + self._apply_exif_changes() + exif = self.layout.get("exif") + if exif is not None and (bytes(self.buf) != bytes(self.data[exif["range"][0]:exif["range"][1]])): + start, end = exif["range"] + if len(self.buf) <= end - start: + self.data[start:end] = bytes(self.buf).ljust(end - start, b"\x00") + else: + self._move_item("exif", struct.pack(">I", 0) + bytes(self.buf)) + if "xmp" in self.touched and self.xmp is not None: + located = self._xmp_range() + packet = self.exact.get("xmp") + if packet is None: + packet = (xmp.XmpPacket.new() if self.xmp.is_blank() else self.xmp).to_bytes() + if located is None: + raise MetadataValueError(None, ErrorCode.UNSUPPORTED, + "this file has no XMP item: XMP cannot be added to it") + if len(packet) <= located[1] - located[0]: + self.data[located[0]:located[1]] = packet + b" " * (located[1] - located[0] - len(packet)) + else: + self._move_item("xmp", packet) + return bytes(self.data) + + +IPTC_BLOCK_KEY = "iptc:@block" + + +def _tiff_iptc_block(buf, writable: bool) -> list[MetadataField]: + """The IPTC-NAA record of a TIFF as one removable field (its datasets are not edited one by one).""" + try: + _, ifds = tiff_inplace.parse(buf) + entry = next((e for e in ifds["0th"].entries if e.tag == 33723), None) + except (tiff_inplace.TiffError, struct.error): + return [] + if entry is None or entry.data_at is None: + return [] + data = bytes(buf[entry.data_at:entry.data_at + entry.size]) + field = MetadataField.binary_field(IPTC_BLOCK_KEY, "IPTC record (all its fields)", data, mime="application/x-iptc", + deletable=writable, source="iptc") + field.computed = True + return [field] + + +def _tiff_deletable(fields: list[MetadataField], writable: bool) -> None: + """Mark which EXIF/GPS entries of a TIFF-based file can be removed; nothing can be edited in place.""" + for f in fields: + if f.key.startswith(("exif:", "gps:")): + where = exif_codec.parse_key(f.key) + f.deletable = bool(writable and where and tiff_inplace.removable(*where)) + f.editable = bool(writable and where and f.editable and f.field_type != "binary" + and tiff_inplace.editable(*where)) + if where in (("0th", 700), ("0th", 33723)): + f.computed = True # the raw XMP / IPTC bytes change with their interpreted fields + continue + elif f.key == IPTC_BLOCK_KEY: + f.deletable = bool(writable) + elif f.key.startswith(("xmp:", "iptc:")): + f.deletable = bool(writable and f.key.startswith("xmp:") and f.deletable) + f.editable = bool(writable and f.key.startswith("xmp:") and f.editable) + else: + f.editable = False + + +def _container_kind(path: Path) -> str | None: + """``jpeg`` / ``png`` / ``webp`` when the file may be edited as such: its bytes say so and its + extension agrees (or it has none).""" + suffix = path.suffix.lower() + kind = sniff.detect(path) + if kind == "tiff" or (kind is None and suffix in _READ_ONLY_EXTS and _read_head(path)[:4] in exif_codec.RAW_MAGICS): + return "tiff" if suffix in _TIFF_EXTS else None + if kind not in ("jpeg", "png", "webp"): + return None + allowed = {"jpeg": JPEG_EXTS, "png": {".png"}, "webp": {".webp"}}[kind] + return kind if suffix in allowed or suffix == "" else None + + +def _state_for(data: bytes) -> "_JpegState": + """Metadata state of the container the bytes are (JPEG, PNG or WebP).""" + if data[:8] == _PNG_SIGNATURE: + return _PngState.load(data) + if data[:4] == b"RIFF" and data[8:12] == b"WEBP": + return _WebpState.load(data) + if data[4:8] == b"ftyp": + return _HeifState.load(data) if is_heif_brand(data) else _Mp4State.load(data) + if data[:4] in (b"II*\x00", b"MM\x00*") or data[:4] in exif_codec.RAW_MAGICS: + return _TiffState.load(data) + return _JpegState.load(data) + + +def _png_text_key(chunk: list) -> str | None: + """Field key (``png:`` or ``png:@``) of a text chunk.""" + found = image_containers.PngFile.text_keyword(chunk) + if found is None: + return None + keyword, language = found + return f"png:{keyword}@{language}" if language else f"png:{keyword}" + + +def _png_text_fields(png: "image_containers.PngFile", writable: bool) -> list[MetadataField]: + """PNG text chunks: shown, and removable (never edited in place).""" + texts, _ = _png_metadata(png.to_bytes(), [], None) + return [MetadataField(f"png:{k}", k, v, editable=False, deletable=writable, + field_type="text" if "\n" in v else "str", source="png") + for k, v in texts] + + +_CANON_UUID = bytes.fromhex("85c0b687820f11e08111f4ce462b6a48") + + +def _cr3_fields(path: Path, warnings: list[str]) -> list[MetadataField]: + """EXIF / GPS of a Canon CR3: the tables CMT1 (main), CMT2 (Exif) and CMT4 (GPS) of the Canon ``uuid`` atom.""" + from core.isobmff import open_reader + block = exif_codec.ExifBlock.empty() + found = False + with open_reader(path) as rd: + moov = rd.find(None, b"moov") + canon = next((b for b in (rd.children(moov.data_start, moov.end) if moov else []) + if b.type == b"uuid" and b.usertype == _CANON_UUID), None) + if canon is None: + warnings.append("No Canon metadata atom found") + return [] + for atom in rd.children(canon.data_start, canon.end): + target = {b"CMT1": "0th", b"CMT2": "Exif", b"CMT4": "GPS"}.get(atom.type) + if target is None: + continue + try: + part = exif_codec.parse(rd.payload(atom), with_ifd1=False) + except exif_codec.ExifFormatError as exc: + warnings.append(f"{atom.type.decode()} not readable: {exc}") + continue + block.byteorder = part.byteorder if not found else block.byteorder + block.ifds[target] = part.ifds["0th"] + found = True + return _exif_fields(block, False) if found else [] + + +def _raf_fields(path: Path, warnings: list[str]) -> list[MetadataField]: + """EXIF / XMP / IPTC of the JPEG preview stored in a Fujifilm RAF file.""" + with open(path, "rb") as stream: + head = stream.read(0x5C) + if len(head) < 0x5C: + return [] + offset, length = struct.unpack(">II", head[0x54:0x5C]) + stream.seek(offset) + jpeg = stream.read(min(length, 64 * 1024 * 1024)) + if not jpeg_segments.is_jpeg(jpeg[:4]): + warnings.append("The preview picture of this RAF could not be found") + return [] + try: + state = _JpegState.load(jpeg, tolerant=True) + except jpeg_segments.JpegFormatError as exc: + warnings.append(f"RAF preview not readable: {exc}") + return [] + warnings.extend(state.warnings) + return ImageHandler._state_fields(state, False) + + def _linked_fields_agree(state: _JpegState) -> bool: """Every IPTC dataset that has an XMP twin in the file holds the same value as the twin.""" fields = {f.key: f for f in ImageHandler._state_fields(state, False)} @@ -448,6 +1027,7 @@ def _exif_fields(block: exif_codec.ExifBlock, writable: bool) -> list[MetadataFi if tag in exif_codec.HIDDEN_TAGS: continue fields.append(_exif_field(block, ifd, tag, writable)) + fields += _makernote_fields(block) if block.thumbnail is not None: fields.append(MetadataField.binary_field( "exif:Thumbnail", "Thumbnail", block.thumbnail, mime="image/jpeg", @@ -455,6 +1035,22 @@ def _exif_fields(block: exif_codec.ExifBlock, writable: bool) -> list[MetadataFi return fields +def _makernote_fields(block: exif_codec.ExifBlock) -> list[MetadataField]: + """Known values of the vendor MakerNote, read only (removing the MakerNote removes them).""" + note = block.ifds["Exif"].get(exif_codec.MAKERNOTE[1]) + if note is None: + return [] + make = block.ifds["0th"].get(0x010F) + make_text = make.data.split(b"\x00", 1)[0].decode("ascii", "replace") if make else "" + try: + found = makernote.read(make_text, note.data, block.byteorder, block.pins.get(exif_codec.MAKERNOTE)) + except (struct.error, ValueError, IndexError): + return [] + return [MetadataField(f"maker:{v.vendor}:{v.name}", f"{v.vendor.title()} {v.label}", v.value, editable=False, + deletable=False, field_type="int" if isinstance(v.value, int) else "str", source="maker", + computed=True) for v in found] + + def _exif_field(block, ifd: str, tag: int, writable: bool) -> MetadataField: entry = block.ifds[ifd][tag] key = exif_codec.key_for(ifd, tag) @@ -520,8 +1116,12 @@ def _xmp_key_map(packet: xmp.XmpPacket) -> dict[str, tuple[str, str]]: return out +XMP_HIDDEN_KEY = "xmp:@hidden" + + def _xmp_fields(packet: xmp.XmpPacket, writable: bool) -> list[MetadataField]: fields: list[MetadataField] = [] + hidden: list[xmp.Prop] = [] seen: set[tuple[str, str]] = set() for prop in packet.properties(): name = (prop.uri, prop.local) @@ -535,9 +1135,35 @@ def _xmp_fields(packet: xmp.XmpPacket, writable: bool) -> list[MetadataField]: f = _unmapped_xmp_field(packet, prop, writable) if f is not None: fields.append(f) + fields += _xmp_language_fields(packet, prop, f) + else: + hidden.append(prop) + if hidden: + # Structured data (edit history, document ids, derived-from...) that has no field of its own: + # listed so that it is not a blind spot, and removable as a whole. + names = [f"{xmp.KNOWN_PREFIXES.get(p.uri) or p.prefix or 'ns'}:{p.local}" for p in hidden] + fields.append(MetadataField(XMP_HIDDEN_KEY, "XMP data not shown", names, raw_value=names, editable=False, + deletable=writable, field_type="str", source="xmp", multi=True, computed=True)) return fields +def _xmp_language_fields(packet: xmp.XmpPacket, prop: xmp.Prop, main: MetadataField) -> list[MetadataField]: + """The other languages of a text with alternatives, one read-only field each (``@``). They go + with the main field: removing it removes them.""" + out = [] + for lang, text in packet.other_languages(prop): + out.append(MetadataField(f"{main.key}@{lang}", f"{main.label} ({lang})", text, editable=False, + deletable=False, field_type="text", source="xmp", computed=True)) + return out + + +def _xmp_hidden(packet: xmp.XmpPacket) -> list[tuple[str, str]]: + """(uri, local) of the properties of the packet that have no field of their own.""" + return [(p.uri, p.local) for p in packet.properties() + if ((_mapped_xmp_field(packet, p, _XMP_BY_NAME[(p.uri, p.local)], True) + if (p.uri, p.local) in _XMP_BY_NAME else _unmapped_xmp_field(packet, p, True)) is None)] + + def _mapped_xmp_field(packet, prop, key, writable) -> MetadataField | None: uri, local, form, kind, label = XMP_MAP[key] value = packet.read_value(prop, form) @@ -662,6 +1288,7 @@ def _png_chunks(data: bytes): pos += 12 + length +_PNG_SIGNATURE = b"\x89PNG\r\n\x1a\n" _MAX_TEXT = 1 << 20 @@ -674,11 +1301,35 @@ def _inflate(data: bytes) -> bytes | None: return None -def _png_metadata(data: bytes, warnings: list[str]) -> tuple[list[tuple[str, str]], bytes | None]: +_PNG_COLOR_TYPES = {0: "Grayscale", 2: "RGB", 3: "Palette", 4: "Grayscale + alpha", 6: "RGB + alpha"} +_PNG_XMP_KEYWORD = "XML:com.adobe.xmp" + + +@dataclass +class _PngExtras: + """What a PNG holds besides its text chunks and EXIF.""" + xmp: bytes | None = None + icc: bytes | None = None + header: tuple[int, int, int] | None = None # bit depth, colour type, interlace method + modified: tuple[int, ...] | None = None # tIME: year, month, day, hour, minute, second + + +def _png_metadata(data: bytes, warnings: list[str], + extras: "_PngExtras | None" = None) -> tuple[list[tuple[str, str]], bytes | None]: + """Text chunks (``keyword`` or ``keyword@language``) and the EXIF chunk; the XMP packet, + the header and the modification time go to ``extras``.""" texts: list[tuple[str, str]] = [] exif = None + extras = extras if extras is not None else _PngExtras() for ctype, body in _png_chunks(data): - if ctype == b"tEXt" and b"\x00" in body: + if ctype == b"IHDR" and len(body) == 13 and extras.header is None: + extras.header = (body[8], body[9], body[12]) + elif ctype == b"iCCP" and extras.icc is None and b"\x00" in body: + _name, _, rest = body.partition(b"\x00") + extras.icc = _inflate(rest[1:]) + elif ctype == b"tIME" and len(body) == 7 and extras.modified is None: + extras.modified = (int.from_bytes(body[0:2], "big"), *body[2:7]) + elif ctype == b"tEXt" and b"\x00" in body: kw, _, text = body.partition(b"\x00") texts.append((kw.decode("latin-1"), text.decode("latin-1"))) elif ctype == b"zTXt" and b"\x00" in body: @@ -693,19 +1344,99 @@ def _png_metadata(data: bytes, warnings: list[str]) -> tuple[list[tuple[str, str if len(rest) < 2: continue compressed, rest = rest[0], rest[2:] - _lang, _, rest = rest.partition(b"\x00") + lang, _, rest = rest.partition(b"\x00") _tkw, _, text = rest.partition(b"\x00") if compressed: text = _inflate(text) if text is None: warnings.append(f"PNG iTXt {kw.decode('latin-1')!r} not decoded") continue - texts.append((kw.decode("latin-1"), values.decode_text(text, ("utf-8",)))) + name = kw.decode("latin-1") + if name == _PNG_XMP_KEYWORD: + if extras.xmp is None: + extras.xmp = text + continue + tag = lang.decode("ascii", "replace").strip() + texts.append((f"{name}@{tag}" if tag else name, values.decode_text(text, ("utf-8",)))) elif ctype == b"eXIf" and exif is None: exif = body return texts, exif +_JFIF_UNITS = {0: "None (aspect ratio)", 1: "Inches", 2: "Centimeters"} +_MPF_TYPES = {0x000000: "Undefined", 0x010001: "Large thumbnail (VGA)", 0x010002: "Large thumbnail (full HD)", + 0x020001: "Multi-frame panorama", 0x020002: "Multi-frame disparity", + 0x020003: "Multi-frame multi-angle", 0x030000: "Baseline MP primary image"} + + +def _jpeg_structure_fields(state: "_JpegState") -> list[MetadataField]: + """Read-only JFIF header, comments, MPF index and ICC profile of a JPEG.""" + fields: list[MetadataField] = [] + + def add(key, label, value, field_type="str", source="jpeg", **extra): + fields.append(MetadataField(key, label, value, raw_value=value, editable=False, deletable=False, + field_type=field_type, source=source, **extra)) + + segments = state.jpeg.segments + for seg in segments: + if seg.is_jfif and seg.payload[:5] == b"JFIF\x00" and len(seg.payload) >= 12: + p = seg.payload + add("jfif:Version", "JFIF Version", f"{p[5]}.{p[6]:02d}", source="jfif") + add("jfif:ResolutionUnit", "JFIF Resolution Unit", _JFIF_UNITS.get(p[7], str(p[7])), source="jfif") + add("jfif:XResolution", "JFIF X Resolution", int.from_bytes(p[8:10], "big"), "int", source="jfif") + add("jfif:YResolution", "JFIF Y Resolution", int.from_bytes(p[10:12], "big"), "int", source="jfif") + break + comments = [values.decode_text(seg.payload, ("utf-8",)).rstrip("\x00") + for seg in segments if seg.marker == jpeg_segments.COM and seg.payload is not None] + if len(comments) == 1: + add("jpeg:Comment", "JPEG Comment", comments[0], "text" if "\n" in comments[0] else "str") + elif comments: + add("jpeg:Comment", "JPEG Comment", comments, "str", multi=True) + for seg in segments: + if seg.is_mpf: + try: + index = jpeg_segments.parse_mp_index(seg.payload) + except jpeg_segments.MpfError: + break + add("mpf:NumberOfImages", "MPF Number of Images", len(index.entries), "int", source="mpf") + for number, entry in enumerate(index.entries, start=1): + kind = entry[0] & 0xFFFFFF + add(f"mpf:image{number}:Type", f"MPF Image {number} Type", _MPF_TYPES.get(kind, f"0x{kind:06X}"), + source="mpf") + add(f"mpf:image{number}:Size", f"MPF Image {number} Size (bytes)", entry[1], "int", + source="mpf", computed=True) + break + profile = icc.assemble([s.payload for s in segments if s.is_app(2, b"ICC_PROFILE\x00") and s.payload]) + if profile is not None: + fields += icc.fields(profile) + return fields + + +def _png_structure_fields(extras: "_PngExtras") -> list[MetadataField]: + """Read-only header and last-modification fields of a PNG.""" + fields: list[MetadataField] = [] + if extras.header is not None: + depth, color, interlace = extras.header + fields += [ + MetadataField("img:bit_depth", "Bit Depth", depth, editable=False, deletable=False, + field_type="int", source="image"), + MetadataField("img:color_type", "PNG Color Type", _PNG_COLOR_TYPES.get(color, str(color)), + editable=False, deletable=False, source="image"), + MetadataField("img:interlace", "Interlaced", bool(interlace), editable=False, deletable=False, + field_type="bool", source="image"), + ] + if extras.modified is not None: + year, month, day, hour, minute, second = extras.modified + try: + stamp = values.format_datetime(datetime(year, month, day, hour, minute, second, tzinfo=timezone.utc)) + except ValueError: + stamp = None + if stamp: + fields.append(MetadataField("pngchunk:tIME", "PNG Last Modification (UTC)", stamp, editable=False, + deletable=False, field_type="datetime", source="png")) + return fields + + def _riff_chunks(data: bytes): if data[:4] != b"RIFF" or data[8:12] != b"WEBP": return @@ -732,6 +1463,8 @@ class ImageHandler(BaseMetadataHandler): ".jpg", ".jpeg", ".tif", ".tiff", ".png", ".bmp", ".gif", ".webp", ".ico", ".cr2", ".nef", ".arw", ".dng", ".orf", ".rw2", ".pef", ".srw", + ".rwl", ".nrw", ".srf", ".sr2", ".3fr", ".erf", ".mos", ".kdc", ".dcr", ".mef", ".iiq", + ".cr3", ".raf", }) PRIORITY = 10 @@ -740,38 +1473,97 @@ def read(self, path: Path) -> MetadataRecord: fields: list[MetadataField] = [] errors: list[str] = [] warnings: list[str] = [] - fields += self._pillow_fields(path, errors) + pillow_errors: list[str] = [] + fields += self._pillow_fields(path, pillow_errors) suffix = path.suffix.lower() writable = False reason = None + raw_reason = None try: head = _read_head(path) except OSError as exc: errors.append(f"read error: {exc}") head = b"" + raw_file = head[:4] in exif_codec.RAW_MAGICS or (suffix in _READ_ONLY_EXTS and bool(pillow_errors)) + if raw_file and pillow_errors: + # The image library does not know this raw format: its properties are simply not shown. + warnings.append("Image properties not read: format not supported by the image library") + else: + errors += pillow_errors try: if jpeg_segments.is_jpeg(head): - writable = suffix in JPEG_EXTS + writable = _jpeg_name(path) more, reason = self._jpeg_fields(path, writable, errors, warnings) fields += more - elif head[:4] in (b"II*\x00", b"MM\x00*"): - fields += self._tiff_fields(path, errors) + elif head[:4] in (b"II*\x00", b"MM\x00*") or head[:4] in exif_codec.RAW_MAGICS: + writable = _container_kind(path) == "tiff" + fields += self._tiff_fields(path, errors, writable) + if writable and suffix in _READ_ONLY_EXTS: + warnings.append("Raw file: metadata changes are experimental (nothing existing is moved); " + "check the result with your raw software") elif head[:8] == b"\x89PNG\r\n\x1a\n": - texts, exif = _png_metadata(path.read_bytes(), warnings) - fields += [MetadataField(f"png:{k}", k, v, editable=False, deletable=False, - field_type="text" if "\n" in v else "str", source="png") - for k, v in texts] - if exif is not None: - fields += self._exif_bytes_fields(_strip_exif_header(exif), errors) - elif head[:4] == b"RIFF" and head[8:12] == b"WEBP": - for fourcc, body in _riff_chunks(path.read_bytes()): - if fourcc == b"EXIF": - fields += self._exif_bytes_fields(_strip_exif_header(body), errors) - elif fourcc == b"XMP ": + data = path.read_bytes() + extras = _PngExtras() + texts, exif = _png_metadata(data, warnings, extras) + fields += _png_structure_fields(extras) + if extras.icc is not None: + fields += icc.fields(extras.icc) + state = None + if _container_kind(path) == "png": + try: + state = _PngState.load(data) + except jpeg_segments.JpegFormatError as exc: + warnings.append(f"{exc}: metadata is shown but read-only") + if state is not None: + errors += state.errors + warnings += state.warnings + writable = True + fields += self._state_fields(state, writable) + else: + fields += [MetadataField(f"png:{k}", k, v, editable=False, deletable=False, + field_type="text" if "\n" in v else "str", source="png") + for k, v in texts] + if exif is not None: + fields += self._exif_bytes_fields(_strip_exif_header(exif), errors) + if extras.xmp is not None: try: - fields += _xmp_fields(xmp.XmpPacket(body), False) + fields += _xmp_fields(xmp.XmpPacket(extras.xmp), False) except xmp.XmpFormatError as exc: warnings.append(f"XMP not readable: {exc}") + elif head[:4] == b"RIFF" and head[8:12] == b"WEBP": + data = path.read_bytes() + for fourcc, body in _riff_chunks(data): + if fourcc == b"ICCP": + fields += icc.fields(body) + state = None + if _container_kind(path) == "webp": + try: + state = _WebpState.load(data) + except jpeg_segments.JpegFormatError as exc: + warnings.append(f"{exc}: metadata is shown but read-only") + if state is not None: + errors += state.errors + warnings += state.warnings + reason = state.read_only_reason() + if reason: + warnings.append(reason) + writable = True + fields += self._state_fields(state, writable and not reason) + else: + for fourcc, body in _riff_chunks(data): + if fourcc == b"EXIF": + fields += self._exif_bytes_fields(_strip_exif_header(body), errors) + elif fourcc == b"XMP ": + try: + fields += _xmp_fields(xmp.XmpPacket(body), False) + except xmp.XmpFormatError as exc: + warnings.append(f"XMP not readable: {exc}") + elif head[4:12] == b"ftypcrx " or (suffix == ".cr3" and head[4:8] == b"ftyp"): + fields += _cr3_fields(path, warnings) + raw_reason = "Canon CR3 files can be read but not changed" + elif head[:15] == b"FUJIFILMCCD-RAW": + fields += _raf_fields(path, warnings) + raw_reason = "Fujifilm RAF files can be read but not changed" elif suffix in _READ_ONLY_EXTS: warnings.append("EXIF of this RAW format is not read") except Exception as exc: # read never raises: unexpected data is a read error, read-only @@ -781,8 +1573,10 @@ def read(self, path: Path) -> MetadataRecord: writable = False record = self._make_record(path, read_errors=errors, warnings=warnings, supports_write=writable, supports_delete=writable) - if reason and suffix in JPEG_EXTS and not errors: + if reason and _container_kind(path) and not errors: record.read_only_reason = reason + elif raw_reason and not errors: + record.read_only_reason = raw_reason record.fields = fields if not writable or errors: for f in fields: @@ -846,7 +1640,7 @@ def _jpeg_fields(self, path, writable, errors, warnings) -> tuple[list[MetadataF reason = state.read_only_reason() if reason: warnings.append(reason) - return _jpeg_dpi(state) + self._state_fields(state, writable and not reason), reason + return _jpeg_dpi(state) + _jpeg_structure_fields(state) + self._state_fields(state, writable and not reason), reason @staticmethod def _state_fields(state: _JpegState, writable: bool) -> list[MetadataField]: @@ -857,16 +1651,44 @@ def _state_fields(state: _JpegState, writable: bool) -> list[MetadataField]: fields += _xmp_fields(state.xmp, writable and state.xmp_writable) if state.iim is not None: fields += _iptc_fields(state, writable and state.iptc_writable) + if isinstance(state, _PngState): + fields += _png_text_fields(state.png, writable) + if isinstance(state, _TiffState): + fields += _tiff_iptc_block(state.buf, writable) + _tiff_deletable(fields, writable) return fields - def _tiff_fields(self, path: Path, errors: list[str]) -> list[MetadataField]: + def _tiff_fields(self, path: Path, errors: list[str], writable: bool = False) -> list[MetadataField]: try: with open(path, "rb") as fh, mmap.mmap(fh.fileno(), 0, access=mmap.ACCESS_READ) as mm: - block = exif_codec.parse(mm, with_ifd1=False) + block = exif_codec.parse(mm, with_ifd1=False, raw=True) except (exif_codec.ExifFormatError, OSError, ValueError) as exc: errors.append(f"EXIF read error: {exc}") return [] - return _exif_fields(block, False) + fields = _exif_fields(block, writable) + entries = block.ifds["0th"] + if 34675 in entries: # embedded ICC profile + fields += icc.fields(entries[34675].data) + if 700 in entries: # XMP packet (Lightroom, many editors) + try: + fields += _xmp_fields(xmp.XmpPacket(entries[700].data), writable) + except xmp.XmpFormatError: + pass # the raw packet stays visible as its binary field + if 33723 in entries: # IPTC-NAA (IIM) record + if writable: + fields += _tiff_iptc_block(Path(path).read_bytes(), True) + try: + datasets, _ = iptc.parse_iim(entries[33723].data) + utf8 = iptc.charset_marker(datasets) in (None, iptc.UTF8_MARKER) + datasets_fields = _iptc_fields(SimpleNamespace(iim=datasets, iptc_utf8=utf8), False) + for f in datasets_fields: + f.computed = True # they disappear with the record + fields += datasets_fields + except iptc.IptcFormatError: + pass # unreadable datasets: the block itself is still listed + if writable: + _tiff_deletable(fields, True) + return fields def _exif_bytes_fields(self, data: bytes, errors: list[str]) -> list[MetadataField]: try: @@ -881,17 +1703,48 @@ def _is_canonical(self, f: MetadataField) -> bool: except MetadataValueError: return False + def content_digest(self, path: Path) -> str | None: + """SHA-256 of the picture data: JPEG scan data, PNG image chunks, WebP bitstream chunks.""" + import hashlib + data = Path(path).read_bytes() + digest = hashlib.sha256() + if jpeg_segments.is_jpeg(data[:4]): + try: + digest.update(_JpegState.load(data, tolerant=True).jpeg.tail) + except jpeg_segments.JpegFormatError: + return None + elif data[:8] == _PNG_SIGNATURE: + for ctype, body in _png_chunks(data): + if ctype in (b"IHDR", b"PLTE", b"tRNS", b"IDAT", b"acTL", b"fcTL", b"fdAT"): + digest.update(ctype + body) + elif data[:4] == b"RIFF" and data[8:12] == b"WEBP": + for fourcc, body in _riff_chunks(data): + if fourcc in (b"VP8 ", b"VP8L", b"ALPH", b"ANMF"): + digest.update(fourcc + body) + elif data[:4] in (b"II*\x00", b"MM\x00*") or data[:4] in exif_codec.RAW_MAGICS: + return tiff_inplace.media_digest(data) + else: + return None + return digest.hexdigest() + # ── schema ── def writable_fields(self, path: Path, record: MetadataRecord) -> list[MetadataField]: - if path.suffix.lower() not in JPEG_EXTS or record.read_errors: + if not _container_kind(path) or record.read_errors: return [] try: - state = _JpegState.load(Path(path).read_bytes()) + state = _state_for(Path(path).read_bytes()) except (jpeg_segments.JpegFormatError, OSError): return [] if state.errors or state.read_only_reason(): return [] - return self._addable(state, {f.key for f in record.fields}) + addable = self._addable(state, {f.key for f in record.fields}) + return self._tiff_addable(addable) if isinstance(state, _TiffState) else addable + + @staticmethod + def _tiff_addable(addable: list[MetadataField]) -> list[MetadataField]: + """Of the metadata that can be added, what a TIFF-structured block may take.""" + return [f for f in addable if f.key.startswith("xmp:") or + ((where := exif_codec.parse_key(f.key)) is not None and tiff_inplace.editable(*where))] def _addable(self, state: _JpegState, present: set[str]) -> list[MetadataField]: out: list[MetadataField] = [] @@ -1071,15 +1924,20 @@ def snapshot(self, tmp: Path, keys: list[str]) -> Snapshot: def restore(self, tmp: Path, snap: Snapshot) -> None: state = self._load_for_mutation(tmp) - groups: dict[str, list] = {"exif": [], "xmp": [], "iptc": []} + groups: dict[str, list] = {"exif": [], "xmp": [], "iptc": [], "png": []} for key, raw in snap.items(): groups[_namespace(key)].append((key, raw)) for key, raw in groups["exif"]: self._put_raw(state, key, raw) if groups["xmp"]: self._restore_xmp(state, groups["xmp"]) - if groups["iptc"]: + if isinstance(state, _TiffState): + for key, raw in groups["iptc"]: + self._put_raw(state, key, raw) # the whole IPTC record + elif groups["iptc"]: self._restore_iptc(state, groups["iptc"]) + for key, raw in groups["png"]: + self._put_raw(state, key, raw) for ns, items in groups.items(): carrier = next((raw for _, raw in items if isinstance(raw, _SegmentRaw) and raw.segment is not None), None) @@ -1115,7 +1973,14 @@ def _restore_xmp(self, state: _JpegState, items) -> None: batch = [] for key, raw in items: raw = _unwrap(raw) - if raw is ABSENT: + if key == XMP_HIDDEN_KEY: + if raw is ABSENT: + if state.xmp is not None: + for uri, local in _xmp_hidden(state.xmp): + batch.append((uri, local, None)) + else: + batch.extend((uri, local, occurrences) for uri, local, occurrences in raw[3]) + elif raw is ABSENT: name = self._xmp_name(state, key) if name is not None: batch.append((*name, None)) @@ -1149,12 +2014,12 @@ def _restore_iptc(self, state: _JpegState, items) -> None: state.touched.add("iptc") def write(self, path: Path, fields: list[MetadataField]) -> None: - if path.suffix.lower() not in JPEG_EXTS: + if not _container_kind(path): raise NotImplementedError(f"Metadata write not supported for {path.suffix}") self._atomic_write(path, lambda tmp: self.apply_write(tmp, fields)) def delete(self, path: Path, keys: list[str]) -> None: - if path.suffix.lower() not in JPEG_EXTS: + if not _container_kind(path): raise NotImplementedError(f"Metadata delete not supported for {path.suffix}") self._atomic_write(path, lambda tmp: self.apply_delete(tmp, keys)) @@ -1162,9 +2027,9 @@ def delete(self, path: Path, keys: list[str]) -> None: @staticmethod def _load_for_mutation(path: Path) -> _JpegState: try: - state = _JpegState.load(Path(path).read_bytes()) + state = _state_for(Path(path).read_bytes()) except jpeg_segments.JpegFormatError as exc: - raise MetadataValueError(None, ErrorCode.UNSUPPORTED, f"JPEG structure error: {exc}") from exc + raise MetadataValueError(None, ErrorCode.UNSUPPORTED, str(exc)) from exc if state.errors: raise MetadataValueError(None, ErrorCode.UNSUPPORTED, "; ".join(state.errors)) reason = state.read_only_reason() @@ -1237,6 +2102,10 @@ def _raw(self, state: _JpegState, key: str) -> Any: return _SegmentRaw(raw, payload, index) def _raw_value(self, state: _JpegState, key: str) -> Any: + if isinstance(state, _TiffState) and (key.startswith(("exif:", "gps:")) or key == IPTC_BLOCK_KEY): + where = ("0th", 33723) if key == IPTC_BLOCK_KEY else exif_codec.parse_key(key) + saved = tiff_inplace.snapshot(state.buf, *where) if where else None + return ("tiff", saved) if saved is not None else ABSENT if key.startswith(("exif:", "gps:")): where = exif_codec.parse_key(key) if where is None: @@ -1245,6 +2114,12 @@ def _raw_value(self, state: _JpegState, key: str) -> Any: if entry is None: return ABSENT return ("exif", state.exif.byteorder, entry.tag, entry.type, entry.count, entry.data) + if key == XMP_HIDDEN_KEY: + hidden = _xmp_hidden(state.xmp) if state.xmp is not None else [] + if not hidden: + return ABSENT + kept = tuple((uri, local, state.xmp.raw(uri, local)) for uri, local in hidden) + return _PositionedRaw(("xmp", XMP_HIDDEN_KEY, None, kept), ("xmp-hidden", tuple(hidden))) if key.startswith("xmp:"): name = self._xmp_name(state, key) raw = state.xmp.raw(*name) if state.xmp is not None and name else None @@ -1260,10 +2135,28 @@ def _raw_value(self, state: _JpegState, key: str) -> Any: if not occ: return ABSENT return _PositionedRaw(("iptc", tuple(occ)), ("iptc", tuple(data for _, data in occ))) + if key.startswith("png:") and isinstance(state, _PngState): + found = [(i, bytes(c[0]), bytes(c[1])) for i, c in enumerate(state.png.chunks) + if _png_text_key(c) == key] + return ("png", tuple(found)) if found else ABSENT raise MetadataValueError(key, ErrorCode.UNKNOWN_FIELD) def _put_raw(self, state: _JpegState, key: str, raw: Any) -> None: raw = _unwrap(raw) + if isinstance(state, _TiffState) and (key.startswith(("exif:", "gps:")) or key == IPTC_BLOCK_KEY): + where = ("0th", 33723) if key == IPTC_BLOCK_KEY else exif_codec.parse_key(key) + if where is None or not (tiff_inplace.editable(*where) or where == ("0th", 33723)): + raise MetadataValueError(key, ErrorCode.NOT_DELETABLE) + try: + if raw is ABSENT: + if tiff_inplace.remove(state.buf, where[0], {where[1]}, force=True): + state.touched.add("tiff") + else: + tiff_inplace.put_entry(state.buf, raw[1]) + state.touched.add("tiff") + except tiff_inplace.TiffError as exc: + raise MetadataValueError(key, ErrorCode.LOSSY_WRITE, f"{key}: {exc}") from exc + return if key.startswith(("exif:", "gps:")): if not state.exif_writable: raise MetadataValueError(key, ErrorCode.READ_ONLY) @@ -1279,6 +2172,17 @@ def _put_raw(self, state: _JpegState, key: str, raw: Any) -> None: state.exif.ifds[ifd][tag] = entry state.touched.add("exif") return + if key == XMP_HIDDEN_KEY: + if not state.xmp_writable: + raise MetadataValueError(key, ErrorCode.READ_ONLY) + if raw is ABSENT: + if state.xmp is not None: + for uri, local in _xmp_hidden(state.xmp): + state.xmp.remove(uri, local) + state.touched.add("xmp") + return + self._restore_xmp(state, [(key, raw)]) + return if key.startswith("xmp:"): if not state.xmp_writable: raise MetadataValueError(key, ErrorCode.READ_ONLY) @@ -1304,6 +2208,15 @@ def _put_raw(self, state: _JpegState, key: str, raw: Any) -> None: if removed is not None or raw is not ABSENT: state.touched.add("iptc") return + if key.startswith("png:") and isinstance(state, _PngState): + wanted = [i for i, c in enumerate(state.png.chunks) if _png_text_key(c) == key] + for i in reversed(wanted): + del state.png.chunks[i] + if raw is not ABSENT: + state.png.put_text([(index, [kind, body]) for index, kind, body in raw[1]]) + if wanted or raw is not ABSENT: + state.touched.add("png") + return raise MetadataValueError(key, ErrorCode.UNKNOWN_FIELD) @staticmethod @@ -1337,8 +2250,10 @@ def _new_exif_schema(ifd: str, tag: int) -> MetadataField | None: field_type = {"datetime": "localdatetime", "int": "int", "rational": "rational"}.get(kind, "str") if choices: field_type = "enum" + triple = (ifd, tag) in (("GPS", 2), ("GPS", 4)) # degrees, minutes, seconds return MetadataField(key, label, None, source="gps" if ifd == "GPS" else "exif", - field_type=field_type, choices=list(choices or [])) + field_type=field_type, choices=list(choices or []), + multi=triple, length=3 if triple else None) def _canonical_exif(field: MetadataField, value: Any) -> Any: diff --git a/python/core/handlers/mkv_fields.py b/python/core/handlers/mkv_fields.py new file mode 100644 index 0000000..8fdd0e9 --- /dev/null +++ b/python/core/handlers/mkv_fields.py @@ -0,0 +1,197 @@ +"""Matroska / WebM fields (``mkv:`` namespace) and their removal / in-place edit.""" +from __future__ import annotations + +import mmap +from datetime import datetime, timedelta, timezone +from pathlib import Path + +from core import ebml, values +from core.models import MetadataField + +_EPOCH_2001 = datetime(2001, 1, 1, tzinfo=timezone.utc) +_TRACK_TYPES = {1: "Video", 2: "Audio", 3: "Complex", 16: "Logo", 17: "Subtitle", 18: "Buttons", 32: "Control"} +INFO_KEYS = {ebml.TITLE: "mkv:Info:Title", ebml.MUXING_APP: "mkv:Info:MuxingApp", ebml.WRITING_APP: "mkv:Info:WritingApp"} + + +def _field(key, label, value, field_type="str", *, deletable=False, editable=False, multi=False, computed=False): + return MetadataField(key, label, value, raw_value=value, editable=editable, deletable=deletable, + field_type=field_type, source="mkv", multi=multi, computed=computed) + + +def _open(path: Path): + stream = open(path, "rb") + try: + return stream, mmap.mmap(stream.fileno(), 0, access=mmap.ACCESS_READ) + except (ValueError, OSError): + stream.close() + raise ebml.EbmlError("empty file") + + +def _plan(path: Path): + """``({key: [Element or patch tuple]}, segment, buffer, stream)`` helper shared by read and edit.""" + stream, buf = _open(path) + return stream, buf, ebml.Segment(buf) + + +def fields(path: Path, writable: bool = True) -> tuple[list[MetadataField], list[str], bool]: + """``(fields, notices, editable)`` of a Matroska file; ``editable`` is False for live recordings.""" + stream, buf = _open(path) + try: + seg = ebml.Segment(buf) + notes: list[str] = [] + editable = writable and not seg.live + if seg.live: + notes.append("This recording has no final size: its metadata is read-only") + out = [_field("mkv:DocType", "Document Type", seg.doc_type)] + info = ebml.info_values(buf, seg) + for ident, key in INFO_KEYS.items(): + if ident in info: + out.append(_field(key, key.rsplit(":", 1)[1], ebml._text(buf, info[ident]), + deletable=editable, editable=editable)) + if ebml.DATE_UTC in info and info[ebml.DATE_UTC].end - info[ebml.DATE_UTC].data_start == 8: + nanos = int.from_bytes(buf[info[ebml.DATE_UTC].data_start:info[ebml.DATE_UTC].end], "big", signed=True) + try: + out.append(_field("mkv:Info:DateUTC", "Date (UTC)", + values.format_datetime(_EPOCH_2001 + timedelta(microseconds=nanos // 1000)), + "datetime", deletable=editable)) + except OverflowError: + pass # a date outside the representable range is not shown + scale = ebml._uint(buf, info[ebml.TIMECODE_SCALE]) if ebml.TIMECODE_SCALE in info else 1_000_000 + if ebml.DURATION in info: + duration = ebml._float(buf, info[ebml.DURATION]) + if duration is not None and duration >= 0: + out.append(_field("mkv:Info:Duration", "Duration (s)", round(duration * scale / 1e9, 3), "duration", + computed=True)) + seen: set[str] = set() + for name, text, _element in ebml.simple_tags(buf, seg): + if name in seen: + continue + seen.add(name) + out.append(_field(f"mkv:tag:{name}", name, text, deletable=editable, editable=editable)) + for name, mime, data, _element in ebml.attachments(buf, seg): + if name and f"mkv:attachment:{name}" not in seen: + seen.add(f"mkv:attachment:{name}") + out.append(MetadataField.binary_field(f"mkv:attachment:{name}", f"Attachment {name}", data, + mime=mime or None, deletable=editable, source="mkv")) + for t in ebml.tracks(buf, seg): + n = t["number"] + out.append(_field(f"mkv:track{n}:Type", f"Track {n} Type", _TRACK_TYPES.get(t["type"], str(t["type"])))) + if t["codec"]: + out.append(_field(f"mkv:track{n}:CodecID", f"Track {n} Codec", t["codec"])) + if t["language"]: + out.append(_field(f"mkv:track{n}:Language", f"Track {n} Language", t["language"])) + if t["name"]: + out.append(_field(f"mkv:track{n}:Name", f"Track {n} Name", t["name"])) + return out, notes, editable + finally: + buf.close() + stream.close() + + +def removal_plan(path: Path) -> dict[str, tuple]: + """Patches ``(offset, original, replacement)`` that remove each removable field, by key.""" + stream, buf = _open(path) + try: + seg = ebml.Segment(buf) + if seg.live: + return {} + plan: dict[str, tuple] = {} + info = ebml.info_values(buf, seg) + for ident, key in INFO_KEYS.items(): + if ident in info: + plan[key] = (ebml.void_patch(buf, info[ident]),) + if ebml.DATE_UTC in info: + plan["mkv:Info:DateUTC"] = (ebml.void_patch(buf, info[ebml.DATE_UTC]),) + for name, _text, element in ebml.simple_tags(buf, seg): + plan.setdefault(f"mkv:tag:{name}", ()) + plan[f"mkv:tag:{name}"] += (ebml.void_patch(buf, element),) + for name, _mime, _data, element in ebml.attachments(buf, seg): + if name: + plan[f"mkv:attachment:{name}"] = (ebml.void_patch(buf, element),) + return plan + finally: + buf.close() + stream.close() + + +def apply_patches(path: Path, patches, restore: bool = False) -> None: + with open(path, "r+b") as stream: + for offset, original, replacement in patches: + stream.seek(offset) + stream.write(original if restore else replacement) + + +def _vint_size(n: int) -> bytes: + for length in range(1, 9): + if n < (1 << (7 * length)) - 1: + return ((1 << (7 * length)) | n).to_bytes(length, "big") + raise ebml.EbmlError("too large") + + +def set_text(path: Path, key: str, text: str) -> None: + """Write a string over the old one; what is left of the old element becomes a Void element.""" + stream, buf = _open(path) + try: + seg = ebml.Segment(buf) + if seg.live: + raise ebml.EbmlError("a recording without a final size cannot be changed") + target = None + parent = None + if key in INFO_KEYS.values(): + info = ebml.info_values(buf, seg) + ident = next(i for i, k in INFO_KEYS.items() if k == key) + target = info.get(ident) + block = seg.first(ebml.INFO) + parent = (block.data_start, block.end) + elif key.startswith("mkv:tag:"): + name = key[len("mkv:tag:"):] + for tag_name, _t, simple in ebml.simple_tags(buf, seg): + if tag_name == name: + kids = {e.id: e for e in ebml.children(buf, simple.data_start, simple.end)} + target = kids[ebml.TAG_STRING] + parent = (simple.data_start, simple.end) + break + if target is None: + raise ebml.EbmlError("the field was not found") + id_bytes = bytes(buf[target.start:target.start + _id_length(buf[target.start])]) + total, start = target.size, target.start + # the Void elements that follow it are free room (what an earlier, shorter value left behind) + following = [e for e in ebml.children(buf, *parent) if e.start >= target.end] + edge = target.end + for e in following: + if e.id != ebml.VOID or e.start != edge: + break + total += e.size + edge = e.end + finally: + buf.close() + stream.close() + raw = text.encode("utf-8") + for pad in range(0, 3): + payload = raw + bytes(pad) + element = id_bytes + _vint_size(len(payload)) + payload + rest = total - len(element) + if rest == 0 or rest >= 2: + filler = b"" + if rest: + filler = _void_bytes(rest) + with open(path, "r+b") as handle: + handle.seek(start) + handle.write(element + filler) + return + raise ebml.EbmlError("the new text is longer than the room the old one leaves") + + +def _id_length(first: int) -> int: + length = 1 + while length <= 4 and not first & (0x80 >> (length - 1)): + length += 1 + return length + + +def _void_bytes(total: int) -> bytes: + for size_len in range(1, 9): + payload = total - 1 - size_len + if payload >= 0 and payload < (1 << (7 * size_len)) - 1: + return bytes([ebml.VOID]) + ((1 << (7 * size_len)) | payload).to_bytes(size_len, "big") + bytes(payload) + raise ebml.EbmlError("too small to be voided") diff --git a/python/core/handlers/odf_handler.py b/python/core/handlers/odf_handler.py new file mode 100644 index 0000000..2ca764a --- /dev/null +++ b/python/core/handlers/odf_handler.py @@ -0,0 +1,210 @@ +"""OpenDocument files (.odt .ods .odp .odg and templates): read-only metadata from ``meta.xml``.""" +from __future__ import annotations + +import re +import xml.etree.ElementTree as ET +from pathlib import Path +from zipfile import BadZipFile, ZipFile + +from core import values +from core.base_handler import BaseMetadataHandler +from core.handlers.xml_package import Mutations +from core.models import MetadataField, MetadataRecord +from core.zipxml import PackageError, local, read_xml + +NS_DC = "http://purl.org/dc/elements/1.1/" +NS_META = "urn:oasis:names:tc:opendocument:xmlns:meta:1.0" + +_TEXT = { # (namespace, local name) -> (key, label, multi) + (NS_DC, "title"): ("title", "Title", False), + (NS_DC, "subject"): ("subject", "Subject", False), + (NS_DC, "description"): ("description", "Description", False), + (NS_DC, "creator"): ("creator", "Last Modified By", False), + (NS_DC, "language"): ("language", "Language", False), + (NS_META, "initial-creator"): ("initial-creator", "Author", False), + (NS_META, "generator"): ("generator", "Generator", False), + (NS_META, "keyword"): ("keyword", "Keywords", True), +} +_DATES = {(NS_DC, "date"): ("date", "Modified"), (NS_META, "creation-date"): ("creation-date", "Created"), + (NS_META, "print-date"): ("print-date", "Last Printed")} +_DURATION = re.compile(r"P(?:(\d+)D)?T?(?:(\d+)H)?(?:(\d+)M)?(?:(\d+(?:\.\d+)?)S)?", re.ASCII) + + +def _field(key: str, label: str, value, field_type: str = "str", multi: bool = False) -> MetadataField: + return MetadataField(f"odf:{key}", label, value, raw_value=value, editable=False, deletable=False, + field_type=field_type, source="odf", multi=multi) + + +def _date_field(key: str, label: str, text: str) -> MetadataField: + for kind, parse in (("datetime", values.parse_datetime), ("localdatetime", values.parse_localdatetime)): + try: + return _field(key, label, values.format_datetime(parse(text)), kind) + except Exception: + continue + return _field(key, label, text) + + +def _duration(text: str) -> float | None: + m = _DURATION.fullmatch(text.strip()) + if not m or not any(m.groups()): + return None + days, hours, minutes, seconds = (float(g) if g else 0.0 for g in m.groups()) + return days * 86400 + hours * 3600 + minutes * 60 + seconds + + +def meta_fields(root, warnings: list[str]) -> list[MetadataField]: + meta = next((c for c in root if local(c.tag) == "meta"), None) + if meta is None: + return [] + fields: list[MetadataField] = [] + texts: dict[str, list[str]] = {} + for node in meta: + uri, _, name = node.tag[1:].partition("}") if node.tag.startswith("{") else ("", "", node.tag) + text = (node.text or "").strip() + ident = (uri, name) + if ident in _TEXT and text: + texts.setdefault(_TEXT[ident][0], []).append(text) + elif ident in _DATES and text: + fields.append(_date_field(*_DATES[ident], text)) + elif ident == (NS_META, "editing-cycles") and text.isdigit(): + fields.append(_field("editing-cycles", "Revision", int(text), "int")) + elif ident == (NS_META, "editing-duration"): + seconds = _duration(text) + if seconds is not None: + fields.append(_field("editing-duration", "Total Editing Time", seconds, "duration")) + elif ident == (NS_META, "document-statistic"): + for attr, value in sorted(node.attrib.items()): + short = attr.rsplit("}", 1)[-1] + if value.isdigit(): + fields.append(_field(f"stat.{short}", f"Statistic {short}", int(value), "int")) + elif ident == (NS_META, "user-defined"): + label = next((v for k, v in node.attrib.items() if local(k) == "name"), "") + if label: + fields.append(_field(f"custom.{label}", f"Custom {label}", text)) + ordered = [] + for (uri, name), (key, label, multi) in _TEXT.items(): + if key in texts: + ordered.append(_field(key, label, texts[key] if multi else texts[key][0], "str", multi)) + return ordered + fields + + +_EDITABLE = {"title", "subject", "description", "creator", "initial-creator", "language", "generator", "keyword", + "creation-date", "date", "print-date", "editing-cycles"} + + +def _meta_node(root): + return next((c for c in root if local(c.tag) == "meta"), None) + + +def _name_of(node) -> tuple[str, str]: + return node.tag[1:].partition("}")[::2] if node.tag.startswith("{") else ("", node.tag) + + +class OdfHandler(Mutations, BaseMetadataHandler): + NAME = "odf" + EXTENSIONS = frozenset({".odt", ".ods", ".odp", ".odg", ".odf", ".ott", ".ots", ".otp", ".otg"}) + PRIORITY = 16 + + # ── where a key lives in meta.xml ── + def part_name(self, archive) -> str: + return "meta.xml" + + def locate(self, root, key: str) -> list[tuple]: + meta = _meta_node(root) + short = key[len("odf:"):] if key.startswith("odf:") else key + hits = [] + for node in (meta if meta is not None else []): + ident = _name_of(node) + if ident in _TEXT and _TEXT[ident][0] == short or ident in _DATES and _DATES[ident][0] == short \ + or ident == (NS_META, short) and short in ("editing-cycles", "editing-duration"): + hits.append((meta, node, None)) + elif ident == (NS_META, "document-statistic") and short.startswith("stat."): + attribute = next((a for a in node.attrib if a.rsplit("}", 1)[-1] == short[5:]), None) + if attribute is not None: + hits.append((meta, node, attribute)) + elif ident == (NS_META, "user-defined") and short.startswith("custom.") \ + and next((v for k, v in node.attrib.items() if local(k) == "name"), None) == short[7:]: + hits.append((meta, node, None)) + return hits + + def make(self, root, key: str, value) -> list: + meta = _meta_node(root) + short = key[len("odf:"):] + if meta is None: + return [] + values_ = value if isinstance(value, list) else [value] + if short.startswith("custom."): + elements = [ET.Element("{%s}user-defined" % NS_META, {"{%s}name" % NS_META: short[7:]})] + elements[0].text = str(values_[0]) + return [(meta, e) for e in elements] + for (uri, name), (text_key, _label, _multi) in _TEXT.items(): + if text_key == short: + made = [] + for v in values_: + element = ET.Element("{%s}%s" % (uri, name)) + element.text = str(v) + made.append((meta, element)) + return made + for (uri, name), (date_key, _label) in _DATES.items(): + if date_key == short: + element = ET.Element("{%s}%s" % (uri, name)) + element.text = str(values_[0]) + return [(meta, element)] + if short == "editing-cycles": + element = ET.Element("{%s}editing-cycles" % NS_META) + element.text = str(values_[0]) + return [(meta, element)] + return [] + + def read(self, path: Path) -> MetadataRecord: + fields: list[MetadataField] = [] + errors: list[str] = [] + warnings: list[str] = [] + try: + with ZipFile(path) as archive: + root = read_xml(archive, "meta.xml", required=False) + if root is not None: + fields = meta_fields(root, warnings) + else: + warnings.append("The document has no meta.xml") + if any(n.startswith("META-INF/") and "signatures" in n for n in archive.namelist()): + warnings.append("Digitally signed document") + except (BadZipFile, PackageError, OSError) as exc: + errors.append(f"ODF read error: {exc}") + signed = any("signed" in w for w in warnings) + writable = not errors and not signed + for f in fields: + short = f.key[len("odf:"):] + f.deletable = writable + f.editable = writable and (short in _EDITABLE or (short.startswith("custom.") and f.field_type == "str")) + record = self._make_record(path, read_errors=errors, warnings=warnings, + supports_write=writable, supports_delete=writable) + record.fields = fields + if not writable and not errors: + record.read_only_reason = "Digitally signed document: editing would invalidate the signature" + record.dedupe_keys() + return record + + def writable_fields(self, path: Path, record) -> list[MetadataField]: + if not record.format_supports_write or record.read_errors: + return [] + present = {f.key for f in record.fields} + return [MetadataField(f"odf:{key}", label, None, source="odf", + field_type="localdatetime" if key in ("creation-date", "date", "print-date") else + "int" if key == "editing-cycles" else "str", multi=multi) + for (_uri, _name), (key, label, multi) in _TEXT.items() if f"odf:{key}" not in present] + + def canonical(self, field: MetadataField, value): + from core import values + result = values.canonicalize(field, value) + for item in (result if isinstance(result, list) else [result]): + if isinstance(item, str) and "\x00" in item: + from core.errors import ErrorCode, MetadataValueError + raise MetadataValueError(field.key, ErrorCode.INVALID_VALUE, f"{field.key}: NUL is not allowed") + return result + + def write(self, path: Path, fields: list[MetadataField]) -> None: + self._atomic_write(path, lambda tmp: self.apply_write(tmp, fields)) + + def delete(self, path: Path, keys: list[str]) -> None: + self._atomic_write(path, lambda tmp: self.apply_delete(tmp, keys)) diff --git a/python/core/handlers/office_handler.py b/python/core/handlers/office_handler.py index bad9cd6..466e4d1 100644 --- a/python/core/handlers/office_handler.py +++ b/python/core/handlers/office_handler.py @@ -293,6 +293,9 @@ def read(self, path: Path) -> MetadataRecord: errors.append(f"Office read error: {exc}") fields = [] writable = not errors and not signed + for field in fields: + if field.key.startswith(("office:app.", "office:custom.")): + field.deletable = writable record = self._make_record(path, read_errors=errors, supports_write=writable, supports_delete=writable) if not writable: @@ -355,6 +358,12 @@ def apply_write(self, tmp: Path, fields: list[MetadataField]) -> None: raise MetadataValueError(f"office:{name}", ErrorCode.LOSSY_WRITE) def apply_delete(self, tmp: Path, keys: list[str]) -> None: + extra = [k for k in keys if _extra_part(k)] + keys = [k for k in keys if not _extra_part(k)] + if extra: + self._delete_extra(tmp, extra) + if not keys: + return root = _open_core(tmp) errors: list[MetadataValueError] = [] names = [] @@ -375,13 +384,26 @@ def apply_delete(self, tmp: Path, keys: list[str]) -> None: def snapshot(self, tmp: Path, keys: list[str]) -> Snapshot: with ZipFile(tmp) as archive: root = _load_core(archive) + parts = {part: _parse_part(archive, part, required=False) for part in (_APP_PART, _CUSTOM_PART)} snap: Snapshot = {} for key in keys: + located = _extra_part(key) + if located: + part_root = parts[located[0]] + nodes = _extra_nodes(part_root, *located) if part_root is not None else [] + snap[key] = tuple((list(part_root).index(n), _node_bytes(n)) for n in nodes) if nodes else ABSENT + continue nodes = _nodes(root, _property_name(key)) snap[key] = tuple(_node_bytes(n) for n in nodes) if nodes else ABSENT return snap def restore(self, tmp: Path, snap: Snapshot) -> None: + extra = {k: v for k, v in snap.items() if _extra_part(k)} + snap = {k: v for k, v in snap.items() if not _extra_part(k)} + if extra: + self._restore_extra(tmp, extra) + if not snap: + return root = _open_core(tmp) for key, raw in snap.items(): name = _property_name(key) @@ -395,6 +417,43 @@ def restore(self, tmp: Path, snap: Snapshot) -> None: root.append(node) _rewrite_core(tmp, root) + @staticmethod + def _part_roots(tmp: Path) -> dict: + with ZipFile(tmp) as archive: + if _is_signed(archive): + raise MetadataValueError(None, ErrorCode.READ_ONLY, + "Editing digitally signed Office packages is not supported") + return {part: _parse_part(archive, part, required=False) for part in (_APP_PART, _CUSTOM_PART)} + + def _delete_extra(self, tmp: Path, keys: list[str]) -> None: + roots = self._part_roots(tmp) + errors = [] + for key in keys: + part, name = _extra_part(key) + nodes = _extra_nodes(roots[part], part, name) if roots[part] is not None else [] + if not nodes: + errors.append(MetadataValueError(key, ErrorCode.UNKNOWN_FIELD)) + continue + for node in nodes: + roots[part].remove(node) + _raise_collected(errors) + _rewrite_parts(tmp, {part: ET.tostring(root, encoding="utf-8", xml_declaration=True) + for part, root in roots.items() if root is not None}) + + def _restore_extra(self, tmp: Path, snap: Snapshot) -> None: + roots = self._part_roots(tmp) + for key, raw in snap.items(): + part, name = _extra_part(key) + if roots[part] is None: + continue + for node in _extra_nodes(roots[part], part, name): + roots[part].remove(node) + if raw is not ABSENT: + for index, data in sorted(raw): + roots[part].insert(min(index, len(roots[part])), ET.fromstring(data)) + _rewrite_parts(tmp, {part: ET.tostring(root, encoding="utf-8", xml_declaration=True) + for part, root in roots.items() if root is not None}) + # ── public wrappers (own transaction) ──────────────────────────────────── def write(self, path: Path, fields: list[MetadataField]) -> None: @@ -411,14 +470,18 @@ def _rewrite_core(path: Path, root) -> None: shapes and vendor extensions. The result is written back into the same file opened ``r+b`` (never recreated: Windows refuses ``wb`` on hidden files). """ - raw = ET.tostring(root, encoding="utf-8", xml_declaration=True) + _rewrite_parts(path, {_CORE_PART: ET.tostring(root, encoding="utf-8", xml_declaration=True)}) + + +def _rewrite_parts(path: Path, replacements: dict[str, bytes]) -> None: + """Replace the named parts of the package, streaming every other member unchanged.""" with tempfile.TemporaryFile(dir=path.parent) as output: with ZipFile(path) as source: with ZipFile(output, "w") as destination: destination.comment = source.comment for info in source.infolist(): - if info.filename == _CORE_PART: - destination.writestr(copy(info), raw) + if info.filename in replacements: + destination.writestr(copy(info), replacements[info.filename]) else: with source.open(info) as reader, destination.open(copy(info), "w") as writer: shutil.copyfileobj(reader, writer) @@ -426,3 +489,20 @@ def _rewrite_core(path: Path, root) -> None: with path.open("r+b") as target: shutil.copyfileobj(output, target) target.truncate() + + +# -- application / custom properties: removable one by one -- + +def _extra_part(key: str) -> tuple[str, str] | None: + """(part, property name) of an ``office:app.`` / ``office:custom.`` key.""" + if key.startswith("office:app."): + return _APP_PART, key[len("office:app."):] + if key.startswith("office:custom."): + return _CUSTOM_PART, key[len("office:custom."):] + return None + + +def _extra_nodes(root, part: str, name: str) -> list: + if part == _APP_PART: + return [c for c in root if _local(c.tag) == (_APP_NS, name)] + return [c for c in root if c.tag == "{" + _CUSTOM_NS + "}property" and c.get("name") == name] diff --git a/python/core/handlers/ole_handler.py b/python/core/handlers/ole_handler.py index a53610e..ef1dbaa 100644 --- a/python/core/handlers/ole_handler.py +++ b/python/core/handlers/ole_handler.py @@ -408,6 +408,62 @@ def _heading_pairs(value: Any) -> Any: return pairs +def _keys_of(sec: Section) -> dict[int, str]: + """Field key of each property of a section (the same keys ``_section_fields`` gives).""" + if sec.fmtid == FMTID_SUMMARY: + return {pid: _SUMMARY_PROPS[pid][0] for pid in sec.props if pid in _SUMMARY_PROPS} + if sec.fmtid == FMTID_DOC_SUMMARY: + return {pid: _DOC_SUMMARY_PROPS[pid][0] for pid in sec.props if pid in _DOC_SUMMARY_PROPS} + if sec.fmtid == FMTID_USER_DEFINED: + return {pid: f"ole:custom:{sec.names.get(pid) or f'pid{pid}'}" for pid in sec.props + if pid >= 2 and not pid & 0x80000000} + return {} + + +def removal_plan(streams: dict[str, bytes]) -> dict[str, list[tuple[str, int, bytes, bytes]]]: + """``{key: [(stream, offset, original, replacement)]}``: a property is removed by giving its value the type + "empty" and zeroing the rest of the room it had, so nothing in the stream moves.""" + plan: dict[str, list[tuple[str, int, bytes, bytes]]] = {} + for name, data in streams.items(): + try: + sections = parse_property_set(data) + count = struct.unpack_from(" len(data): + continue + plan.setdefault(key, []).append((name, lo, bytes(data[lo:hi]), bytes(hi - lo))) + return plan + + +def _read_streams(path: Path) -> dict[str, bytes]: + import olefile + out = {} + with olefile.OleFileIO(str(path)) as ole: + for name in (SUMMARY_STREAM, DOC_SUMMARY_STREAM): + if ole.exists(name): + out[name] = ole.openstream(name).read() + return out + + +def _write_streams(path: Path, changes: dict[str, bytes]) -> None: + import olefile + with olefile.OleFileIO(str(path), write_mode=True) as ole: + for name, data in changes.items(): + ole.write_stream(name, data) # same size as before: only values are overwritten + + class OleHandler(BaseMetadataHandler): NAME = "ole" EXTENSIONS = _OLE_EXTS @@ -437,17 +493,59 @@ def read(self, path: Path) -> MetadataRecord: except Exception as e: errors.append(f"OLE read error: {e}") + writable = not errors + for f in fields: + f.deletable = writable record = self._make_record(path, read_errors=errors, warnings=warnings, - supports_write=False, supports_delete=False) + supports_write=False, supports_delete=writable) record.fields = fields record.dedupe_keys() return record + # ── mutation hooks: properties are removed in place (the values are never edited) ── + def apply_delete(self, tmp: Path, keys: list[str]) -> None: + from core.errors import ErrorCode, MetadataValueError, MetadataValueErrors + streams = _read_streams(tmp) + plan = removal_plan(streams) + missing = [k for k in keys if k not in plan] + if missing: + raise MetadataValueErrors([MetadataValueError(k, ErrorCode.UNKNOWN_FIELD) for k in missing]) + changed = {name: bytearray(data) for name, data in streams.items()} + for key in keys: + for name, offset, original, replacement in plan[key]: + changed[name][offset:offset + len(replacement)] = replacement + _write_streams(tmp, {name: bytes(data) for name, data in changed.items() if bytes(data) != streams[name]}) + + def snapshot(self, tmp: Path, keys: list[str]): + from core.base_handler import ABSENT + from core.handlers.qt_edit import QtRaw + plan = removal_plan(_read_streams(tmp)) + current = {f.key: f for f in self.read(tmp).fields} + return {k: (QtRaw(current[k].value, tuple(plan.get(k, ()))) if k in current and k in plan else ABSENT) + for k in keys} + + def restore(self, tmp: Path, snap) -> None: + from core.base_handler import ABSENT + streams = _read_streams(tmp) + changed = {name: bytearray(data) for name, data in streams.items()} + plan = removal_plan(streams) + for key, raw in snap.items(): + if raw is ABSENT: + for name, offset, _original, replacement in plan.get(key, ()): + changed[name][offset:offset + len(replacement)] = replacement + continue + for name, offset, original, _replacement in raw.patches: + changed[name][offset:offset + len(original)] = original + _write_streams(tmp, {name: bytes(data) for name, data in changed.items() if bytes(data) != streams[name]}) + + def canonical(self, field, value): + raise NotImplementedError("OLE values are not edited") + def write(self, path: Path, fields: list[MetadataField]) -> None: - raise NotImplementedError("OLE write not supported — use docx/xlsx/pptx format") + raise NotImplementedError("OLE values are not edited; they can only be removed") def delete(self, path: Path, keys: list[str]) -> None: - raise NotImplementedError("OLE delete not supported — use docx/xlsx/pptx format") + self._atomic_write(path, lambda tmp: self.apply_delete(tmp, keys)) def _section_fields(sec: Section, warnings: list[str]) -> list[MetadataField]: diff --git a/python/core/handlers/pdf_handler.py b/python/core/handlers/pdf_handler.py index 2ed2d62..59b8e5d 100644 --- a/python/core/handlers/pdf_handler.py +++ b/python/core/handlers/pdf_handler.py @@ -214,6 +214,20 @@ def _key(name: str) -> str: return f"pdf:{local}#info" if local in _COMPUTED else f"pdf:{local}" +XMP_STREAM_KEY = "pdf:@xmp" + + +def _xmp_bytes(root) -> bytes | None: + """Decoded bytes of the document /Metadata stream (any filter pypdf understands), None when absent.""" + meta = root.get("/Metadata") + if meta is None: + return None + try: + return bytes(meta.get_object().get_data()) + except Exception: # noqa: BLE001 - not a stream / undecodable: left alone + return None + + def _name(key: str) -> str: if not key.startswith("pdf:") or len(key) <= 4 or key[4:] in _COMPUTED or key.endswith("#info"): raise MetadataValueError(key, ErrorCode.UNKNOWN_FIELD, f"{key}: not a PDF information entry") @@ -387,6 +401,45 @@ def _attributed(exc: MetadataValueError, key: str) -> MetadataValueError: # ──────────────────────── handler ──────────────────────────────────────────── +def _xmp_stream_fields(root, removable: bool) -> list[MetadataField]: + """The XMP properties of the document (read-only) and the whole stream (removable, computed).""" + data = _xmp_bytes(root) + if not data: + return [] + from core import xmp as core_xmp + from core.handlers.image_handler import _xmp_fields + out: list[MetadataField] = [] + try: + out += [f for f in _xmp_fields(core_xmp.XmpPacket(data), False) if f.key != "xmp:@hidden"] + for f in out: + f.computed = True # they disappear with the stream + except core_xmp.XmpFormatError: + pass # a malformed XMP stream is reported elsewhere; here it just adds no fields + stream = MetadataField.binary_field(XMP_STREAM_KEY, "XMP metadata stream", data, mime="application/rdf+xml", + deletable=removable, source="pdf") + stream.computed = True # the entries of the information dictionary are mirrored into it + return out + [stream] + + +def _file_structure_fields(path: Path) -> list[MetadataField]: + """Version of the file header and whether it is linearized (computed: a rewrite changes them).""" + try: + with open(path, "rb") as stream: + head = stream.read(2048) + except OSError: + return [] + out: list[MetadataField] = [] + start = head.find(b"%PDF-") + if start >= 0: + version = head[start + 5:start + 8].decode("ascii", "replace") + if len(version) == 3 and version[0].isdigit() and version[1] == "." and version[2].isdigit(): + out.append(MetadataField("pdf:PDFVersion", "PDF Version", version, raw_value=version, editable=False, + deletable=False, field_type="str", source="pdf", computed=True)) + out.append(MetadataField("pdf:Linearized", "Linearized", b"/Linearized" in head, editable=False, + deletable=False, field_type="bool", source="pdf", computed=True)) + return out + + class PdfHandler(BaseMetadataHandler): NAME = "pdf" EXTENSIONS = frozenset({".pdf"}) @@ -434,6 +487,9 @@ def _read(self, path: Path) -> MetadataRecord: warning = _xmp_of(reader.trailer["/Root"].get_object())[2] if warning: warnings.append(warning) + if not encrypted: + fields += _xmp_stream_fields(reader.trailer["/Root"].get_object(), signed is None) + fields += _file_structure_fields(path) fields.append(MetadataField("pdf:page_count", "Page Count", len(reader.pages), editable=False, deletable=False, field_type="int", source="pdf")) fields.append(MetadataField("pdf:encrypted", "Encrypted", encrypted, @@ -528,6 +584,17 @@ def apply_delete(self, tmp: Path, keys: list[str]) -> None: doc = _Mutation(tmp) errors: list[MetadataValueError] = [] names = [] + if XMP_STREAM_KEY in keys: + keys = [k for k in keys if k != XMP_STREAM_KEY] + if "/Metadata" not in doc.writer._root_object: + raise MetadataValueError(XMP_STREAM_KEY, ErrorCode.UNKNOWN_FIELD) + try: + doc.writer._root_object["/Metadata"].get_object().set_data(b"") # an orphaned object is still written + except Exception: # noqa: BLE001 - not a stream: only the entry goes + pass + del doc.writer._root_object["/Metadata"] + doc.xmp = doc.xmp_stream = None + doc.changed = True for key in keys: try: name = _name(key) @@ -552,6 +619,10 @@ def snapshot(self, tmp: Path, keys: list[str]) -> Snapshot: xmp = None if reader.is_encrypted else _xmp_of(reader.trailer["/Root"].get_object())[0] snap: Snapshot = {} for key in keys: + if key == XMP_STREAM_KEY: + data = None if reader.is_encrypted else _xmp_bytes(reader.trailer["/Root"].get_object()) + snap[key] = ABSENT if data is None else ("xmp-stream", data) + continue name = _name(key) info_raw = _raw(info[name]) if name in info else None occurrences = xmp.snapshot(name) if xmp is not None and name in XMP_PROPERTIES else () @@ -562,6 +633,21 @@ def snapshot(self, tmp: Path, keys: list[str]) -> Snapshot: def restore(self, tmp: Path, snap: Snapshot) -> None: doc = _Mutation(tmp) for key, raw in snap.items(): + if key == XMP_STREAM_KEY: + from pypdf.generic import DecodedStreamObject, NameObject + root = doc.writer._root_object + if raw is ABSENT: + if "/Metadata" in root: + del root["/Metadata"] + else: + stream = DecodedStreamObject() + stream.set_data(raw[1]) + stream[NameObject("/Type")] = NameObject("/Metadata") + stream[NameObject("/Subtype")] = NameObject("/XML") + root[NameObject("/Metadata")] = doc.writer._add_object(stream) + doc.xmp = doc.xmp_stream = None + doc.changed = True + continue name = _name(key) info_raw, occurrences = (None, ()) if raw is ABSENT else raw if info_raw is None: diff --git a/python/core/handlers/qt_edit.py b/python/core/handlers/qt_edit.py new file mode 100644 index 0000000..3ac4544 --- /dev/null +++ b/python/core/handlers/qt_edit.py @@ -0,0 +1,303 @@ +"""Editing and adding QuickTime / MP4 structure fields (the ``qt:`` namespace). + +Dates are written over the old ones (fixed width). Text atoms (``udta`` ``(c)xyz``, ``(c)mak``...) and text +values of the ``mdta`` keys are rebuilt inside ``moov`` and ``moov`` is resized. Nothing else is touched, and a +change that would move the media data is refused: the size of ``moov`` may change only when nothing that is addressed by an +offset follows it or when the freed bytes can be turned into a ``free`` atom (a shorter value). +""" +from __future__ import annotations + +import re +import struct +from dataclasses import dataclass, field +from datetime import datetime, timezone +from pathlib import Path + +from core.isobmff import Box, Reader + +_EPOCH_1904 = datetime(1904, 1, 1, tzinfo=timezone.utc) +UDTA_ADDABLE = {"©xyz": "Location", "©mak": "Make", "©mod": "Model", "©swr": "Software", + "©cmt": "Comment"} +MDTA_ADDABLE = {"com.apple.quicktime.location.ISO6709": "Location", "com.apple.quicktime.make": "Make", + "com.apple.quicktime.model": "Model", "com.apple.quicktime.software": "Software", + "com.apple.quicktime.creationdate": "Creation date"} +DATE_KEYS = ("qt:CreateDate", "qt:ModifyDate") + + +class LayoutError(ValueError): + """The file cannot take this change without moving its media data.""" + + +def seconds_from_iso(value: str) -> int: + """Seconds since 1904 (UTC) of a canonical date-time with an offset.""" + stamp = datetime.fromisoformat(value) + seconds = int((stamp - _EPOCH_1904).total_seconds()) + if not 0 < seconds < 1 << 32: + raise LayoutError("QuickTime dates range from 1904 to 2040 in this file") + return seconds + + +def _splice(blob: bytearray, start: int, end: int, new: bytes, container: Box) -> bytearray: + """Replace ``blob[start:end]`` (inside ``container``) and add the size change to ``container`` and to every + atom that contains it.""" + delta = len(new) - (end - start) + chain = [] + rd = Reader(bytes(blob)) + level = rd.children() + while True: + hit = next((b for b in level if b.start <= container.start and container.end <= b.end), None) + if hit is None: + break + if hit.data_start - hit.start != 8: + raise LayoutError("an atom with a 64-bit size cannot be resized") + chain.append(hit) + if hit.start == container.start and hit.end == container.end: + break + level = rd.children(hit.data_start + (4 if hit.type == b"meta" and _is_full_meta(rd, hit) else 0), hit.end) + if not chain or (chain[-1].start, chain[-1].end) != (container.start, container.end): + raise LayoutError("the atom to change was not found") + out = bytearray(blob[:start]) + new + blob[end:] + for box in chain: + out[box.start:box.start + 4] = struct.pack(">I", box.size + delta) + return out + + +def _is_full_meta(rd: Reader, meta: Box) -> bool: + return bytes(rd.buf[meta.data_start:meta.data_start + 4]) == bytes(4) + + +def _moov(path: Path) -> tuple[Box, bool, int]: + from core.isobmff import open_reader + with open_reader(path) as rd: + boxes = rd.children() + moov = next((b for b in boxes if b.type == b"moov"), None) + if moov is None: + raise LayoutError("the file has no movie atom") + if moov.data_start - moov.start != 8: + raise LayoutError("the movie atom has a 64-bit size") + if any(b.type in (b"moof", b"sidx") for b in boxes): + raise LayoutError("fragmented files are not supported") + after = boxes[boxes.index(moov) + 1:] + # nothing after ``moov`` is addressed by an offset when it is empty or only holds atoms found by scanning + return moov, all(b.type in (b"uuid", b"free", b"skip") for b in after), len(rd.buf) + + +def _write_back(path: Path, moov: Box, last: bool, new: bytes) -> None: + old = moov.size + if len(new) == old: + pass + elif last: + with open(path, "rb") as stream: # what follows is only found by scanning: it can move + stream.seek(moov.end) + tail = stream.read() + new = new + tail + with open(path, "r+b") as stream: + stream.seek(moov.start) + stream.write(new) + stream.truncate(moov.start + len(new)) + return + elif len(new) < old and old - len(new) >= 8: + new = new + struct.pack(">I", old - len(new)) + b"free" + bytes(old - len(new) - 8) + elif len(new) > old: + _grow_before_media(path, moov, new) + return + else: + raise LayoutError("a value 1 to 7 bytes shorter cannot be absorbed: change it by more or by nothing") + with open(path, "r+b") as stream: + stream.seek(moov.start) + stream.write(new) + + +def _read_moov(path: Path, moov: Box) -> bytearray: + with open(path, "rb") as stream: + stream.seek(moov.start) + return bytearray(stream.read(moov.size)) + + +def _text_atom(name: str, text: str) -> bytes: + raw = text.encode("utf-8") + if len(raw) > 0xFFFF: + raise LayoutError("the text is too long") + return struct.pack(">I", 12 + len(raw)) + name.encode("latin-1") + struct.pack(">HH", len(raw), 0x15C7) + raw + + +def set_udta_text(path: Path, name: str, text: str) -> None: + moov, last, _ = _moov(path) + blob = _read_moov(path, moov) + rd = Reader(bytes(blob)) + root = rd.children()[0] + atom = _text_atom(name, text) + udta = rd.find(root, b"udta") + if udta is None: + new = _splice(blob, root.end, root.end, struct.pack(">I", 8 + len(atom)) + b"udta" + atom, root) + else: + old = next((b for b in rd.children(udta.data_start, udta.end) if b.type.decode("latin-1") == name), None) + if old is not None: + new = _splice(blob, old.start, old.end, atom, udta) + else: + new = _splice(blob, udta.end, udta.end, atom, udta) + _write_back(path, moov, last, bytes(new)) + + +def _mdta_item(index: int, text: str) -> bytes: + raw = text.encode("utf-8") + data = struct.pack(">I", 16 + len(raw)) + b"data" + struct.pack(">II", 1, 0) + raw + return struct.pack(">I", 8 + len(data)) + struct.pack(">I", index) + data + + +def set_mdta_text(path: Path, name: str, text: str) -> None: + moov, last, _ = _moov(path) + blob = _read_moov(path, moov) + rd = Reader(bytes(blob)) + root = rd.children()[0] + meta = rd.find(root, b"meta") + if meta is None: + # an ISO full-box ``meta`` with an ``mdta`` handler, the key and its item + handler = struct.pack(">I", 33) + b"hdlr" + bytes(4) + bytes(4) + b"mdta" + bytes(12) + b"\x00" + keys = struct.pack(">I", 24 + len(name)) + b"keys" + bytes(4) + struct.pack(">I", 1) \ + + struct.pack(">I", 8 + len(name)) + b"mdta" + name.encode("utf-8") + ilst = struct.pack(">I", 8 + len(_mdta_item(1, text))) + b"ilst" + _mdta_item(1, text) + body = bytes(4) + handler + keys + ilst + new = _splice(blob, root.end, root.end, struct.pack(">I", 8 + len(body)) + b"meta" + body, root) + _write_back(path, moov, last, bytes(new)) + return + start = meta.data_start + (4 if _is_full_meta(rd, meta) else 0) + keys = ilst = None + for child in rd.children(start, meta.end): + if child.type == b"keys": + keys = child + elif child.type == b"ilst": + ilst = child + if keys is None or ilst is None: + raise LayoutError("the metadata atom has no key table") + count = struct.unpack_from(">I", blob, keys.data_start + 4)[0] + names, pos = [], keys.data_start + 8 + for _ in range(count): + size = struct.unpack_from(">I", blob, pos)[0] + names.append(bytes(blob[pos + 8:pos + size]).decode("utf-8", "replace")) + pos += size + if name in names: + index = names.index(name) + 1 + item = next((c for c in rd.children(ilst.data_start, ilst.end) + if struct.unpack(">I", c.type)[0] == index), None) + new = _splice(blob, item.start, item.end, _mdta_item(index, text), ilst) if item is not None \ + else _splice(blob, ilst.end, ilst.end, _mdta_item(index, text), ilst) + else: + entry = struct.pack(">I", 8 + len(name.encode("utf-8"))) + b"mdta" + name.encode("utf-8") + blob = _splice(blob, ilst.end, ilst.end, _mdta_item(count + 1, text), ilst) # later offsets first + blob[keys.data_start + 4:keys.data_start + 8] = struct.pack(">I", count + 1) + new = _splice(blob, keys.end, keys.end, entry, keys) + _write_back(path, moov, last, bytes(new)) + + +def set_date(path: Path, key: str, seconds: int) -> None: + """Write the creation / modification time of the movie header or of a track (tkhd and mdhd).""" + from core.isobmff import open_reader + with open_reader(path) as rd: + moov = rd.find(None, b"moov") + if moov is None: + raise LayoutError("the file has no movie atom") + targets: list[tuple[Box, int]] = [] + if key in DATE_KEYS: + mvhd = rd.find(moov, b"mvhd") + if mvhd is not None: + targets.append((mvhd, 0 if key == "qt:CreateDate" else 1)) + else: + track = int(key.split(":")[1][len("track"):]) + for trak in rd.find_all(moov, b"trak"): + tkhd = rd.find(trak, b"tkhd") + width = 8 if tkhd is not None and rd.buf[tkhd.data_start] == 1 else 4 + if tkhd is not None and rd.u("I", tkhd.data_start + 4 + 2 * width) == (track,): + targets += [(tkhd, 0), (tkhd, 1)] + mdhd = rd.find(trak, b"mdia", b"mdhd") + if mdhd is not None: + targets += [(mdhd, 0), (mdhd, 1)] + patches = [] + for box, which in targets: + width = 8 if rd.buf[box.data_start] == 1 else 4 + if seconds >= 1 << (8 * width): + raise LayoutError("the date does not fit in this file") + patches.append((box.data_start + 4 + which * width, seconds.to_bytes(width, "big"))) + if not patches: + raise LayoutError("the header to change was not found") + with open(path, "r+b") as stream: + for offset, raw in patches: + stream.seek(offset) + stream.write(raw) + + +_ISO6709 = re.compile(r"[+-][0-9]{2}(\.[0-9]+)?[+-][0-9]{3}(\.[0-9]+)?([+-][0-9]+(\.[0-9]+)?)?/") + + +@dataclass(frozen=True) +class QtRaw: + """Pre-image of one ``qt:`` key. Equality is the value only: the patches (offset, original, replacement) + are a hint to put a removed atom back exactly where it was.""" + value: object + patches: tuple = field(default=(), compare=False) + + +def text_of(key: str, value: str) -> str: + """The checked text of a ``qt:udta:`` / ``qt:mdta:`` value.""" + if not isinstance(value, str) or "\x00" in value: + raise LayoutError("the value must be text without NUL characters") + name = key.split(":", 2)[2] + if name in ("\u00a9xyz", "com.apple.quicktime.location.ISO6709") and not _ISO6709.fullmatch(value): + raise LayoutError("a location looks like +45.4642+009.1900+120.500/") + return value + + +def set_value(path: Path, key: str, value) -> None: + """Write the value of a ``qt:`` key: a date, or the text of a ``udta`` / ``mdta`` entry.""" + if key in DATE_KEYS or (key.startswith("qt:track") and key.endswith(":CreateDate")): + set_date(path, key, seconds_from_iso(value)) + elif key.startswith("qt:udta:"): + set_udta_text(path, key.split(":", 2)[2], text_of(key, value)) + elif key.startswith("qt:mdta:"): + set_mdta_text(path, key.split(":", 2)[2], text_of(key, value)) + else: + raise LayoutError("this field cannot be written") + + +_CHUNK = 8 * 1024 * 1024 + + +def _fix_offsets(blob: bytes, old_end: int, delta: int) -> bytes: + """The chunk offset tables (``stco`` / ``co64``) of a movie atom, with every offset past ``old_end`` moved.""" + out = bytearray(blob) + rd = Reader(bytes(blob)) + root = rd.children()[0] + for trak in rd.find_all(root, b"trak"): + stbl = rd.find(trak, b"mdia", b"minf", b"stbl") + if stbl is None: + continue + for kind, width, code in ((b"stco", 4, ">I"), (b"co64", 8, ">Q")): + for table in rd.find_all(stbl, kind): + count = struct.unpack_from(">I", out, table.data_start + 4)[0] + for i in range(count): + at = table.data_start + 8 + width * i + value = struct.unpack_from(code, out, at)[0] + if value >= old_end: + if kind == b"stco" and value + delta >= 1 << 32: + raise LayoutError("a 32-bit chunk offset would overflow: the file needs 64-bit offsets") + struct.pack_into(code, out, at, value + delta) + return bytes(out) + + +def _grow_before_media(path: Path, moov: Box, new: bytes) -> None: + """The movie atom grows while media follows it: the rest of the file moves up and the chunk offsets follow.""" + delta = len(new) - moov.size + fixed = _fix_offsets(new, moov.end, delta) + with open(path, "r+b") as stream: + stream.seek(0, 2) + size = stream.tell() + pos = size + while pos > moov.end: # from the end, so nothing is overwritten + start = max(moov.end, pos - _CHUNK) + stream.seek(start) + chunk = stream.read(pos - start) + stream.seek(start + delta) + stream.write(chunk) + pos = start + stream.seek(moov.start) + stream.write(fixed) diff --git a/python/core/handlers/qt_fields.py b/python/core/handlers/qt_fields.py new file mode 100644 index 0000000..86cbf10 --- /dev/null +++ b/python/core/handlers/qt_fields.py @@ -0,0 +1,293 @@ +"""Read-only QuickTime / ISO base media structure fields (``qt:`` namespace). + +Shared by the MP4 family handlers: brand, movie and track headers, handler, +codec, language, rotation, frame rate, QuickTime ``mdta`` keys (where phones +store the location and capture time), ``udta`` text atoms and an XMP packet. +Offsets, sizes and sample tables are never exposed: they change with any +rewrite and would be reported as changed by the post-write verification. +""" +from __future__ import annotations + +import math +import struct +from datetime import datetime, timedelta, timezone +from pathlib import Path + +from core.isobmff import Box, Reader, open_reader +from core.models import MetadataField +from core.values import format_datetime + +_EPOCH_1904 = datetime(1904, 1, 1, tzinfo=timezone.utc) +_XMP_UUID = bytes.fromhex("be7acfcb97a942e89c71999491e3afac") +_HANDLER_NAMES = {b"vide": "Video", b"soun": "Audio", b"text": "Text", b"sbtl": "Subtitle", b"meta": "Metadata", + b"hint": "Hint", b"tmcd": "Timecode", b"pict": "Picture"} + + +def _field(key: str, label: str, value, field_type: str, *, multi: bool = False) -> MetadataField: + return MetadataField(key, label, value, raw_value=value, editable=False, deletable=False, + field_type=field_type, source="qt", multi=multi) + + +def _text(raw: bytes) -> str: + try: + return raw.decode("utf-8") + except UnicodeDecodeError: + return raw.decode("latin-1") + + +def _date(seconds: int) -> str | None: + """Mac epoch (1904) seconds to canonical UTC, ``None`` when unset (0) or out of range.""" + if not seconds: + return None + try: + return format_datetime(_EPOCH_1904 + timedelta(seconds=seconds)) + except (OverflowError, ValueError): + return None + + +def _language(code: int) -> str | None: + if code == 0 or code == 0x7FFF or code < 0x400: # unset or a legacy Macintosh code + return None + letters = [((code >> shift) & 31) + 0x60 for shift in (10, 5, 0)] + if not all(0x61 <= c <= 0x7A for c in letters): + return None + word = "".join(map(chr, letters)) + return None if word == "und" else word + + +def _rotation(matrix: tuple[int, ...]) -> int | None: + """Rotation in degrees (0/90/180/270) from the a/b entries of the matrix, ``None`` if it is not one.""" + a, b = matrix[0] / 65536, matrix[1] / 65536 + if a == 0 and b == 0: + return None + angle = math.degrees(math.atan2(b, a)) % 360 + nearest = round(angle / 90) * 90 + return int(nearest % 360) if abs(angle - nearest) < 1 else None + + +def _ftyp(rd: Reader, out: list[MetadataField]) -> None: + box = rd.find(None, b"ftyp") + if box is None or box.end - box.data_start < 8: + return + major = bytes(rd.buf[box.data_start:box.data_start + 4]) + minor = rd.u("I", box.data_start + 4) + out.append(_field("qt:MajorBrand", "Major Brand", _text(major).strip(), "str")) + if minor and minor[0]: + out.append(_field("qt:MinorVersion", "Minor Version", minor[0], "int")) + brands = [_text(bytes(rd.buf[p:p + 4])).strip() for p in range(box.data_start + 8, box.end - 3, 4)] + brands = [b for b in brands if b] + if brands: + out.append(_field("qt:CompatibleBrands", "Compatible Brands", brands, "str", multi=True)) + + +def _mvhd(rd: Reader, moov: Box, out: list[MetadataField]) -> None: + box = rd.find(moov, b"mvhd") + if box is None: + return + base = box.data_start + version = rd.buf[base] if base < box.end else 0 + if version == 1: + head = rd.u("QQI", base + 4) + rate = rd.u("IH", base + 32) + else: + head = rd.u("III", base + 4) + rate = rd.u("IH", base + 20) + if head is None: + return + created, modified, scale = head + for key, label, seconds in (("qt:CreateDate", "Create Date", created), ("qt:ModifyDate", "Modify Date", modified)): + stamp = _date(seconds) + if stamp: + out.append(_field(key, label, stamp, "datetime")) + if scale: + out.append(_field("qt:TimeScale", "Time Scale", scale, "int")) + if rate is not None: + out.append(_field("qt:PreferredRate", "Preferred Rate", rate[0] / 65536, "float")) + out.append(_field("qt:PreferredVolume", "Preferred Volume", rate[1] / 256, "float")) + + +def _track(rd: Reader, trak: Box, out: list[MetadataField]) -> None: + tkhd = rd.find(trak, b"tkhd") + if tkhd is None: + return + base = tkhd.data_start + version = rd.buf[base] if base < tkhd.end else 0 + if version == 1: + ids = rd.u("QQI", base + 4) + matrix_at = base + 4 + 8 + 8 + 4 + 4 + 8 + 8 + 8 + else: + ids = rd.u("III", base + 4) + matrix_at = base + 4 + 4 + 4 + 4 + 4 + 4 + 8 + 8 + if ids is None: + return + track_id = ids[2] + prefix = f"qt:track{track_id}" + label = f"Track {track_id}" + matrix = rd.u("9i", matrix_at) + size = rd.u("II", matrix_at + 36) + created = _date(ids[0]) + if created: + out.append(_field(f"{prefix}:CreateDate", f"{label} Create Date", created, "datetime")) + + mdia = rd.find(trak, b"mdia") + handler = None + scale = duration = None + language = None + if mdia is not None: + hdlr = rd.find(mdia, b"hdlr") + if hdlr is not None: + found = rd.u("4s", hdlr.data_start + 8) + handler = bytes(found[0]) if found else None + mdhd = rd.find(mdia, b"mdhd") + if mdhd is not None: + v = rd.buf[mdhd.data_start] if mdhd.data_start < mdhd.end else 0 + m = rd.u("QQIQH", mdhd.data_start + 4) if v == 1 else rd.u("IIIIH", mdhd.data_start + 4) + if m is not None: + scale, duration, language = m[2], m[3], _language(m[4]) + if handler: + out.append(_field(f"{prefix}:HandlerType", f"{label} Handler", + _HANDLER_NAMES.get(handler, _text(handler).strip()), "str")) + if language: + out.append(_field(f"{prefix}:Language", f"{label} Language", language, "str")) + if scale: + out.append(_field(f"{prefix}:MediaTimeScale", f"{label} Media Time Scale", scale, "int")) + if matrix is not None and handler == b"vide": + rotation = _rotation(matrix) + if rotation is not None: + out.append(_field(f"{prefix}:Rotation", f"{label} Rotation (°)", rotation, "int")) + if size is not None and handler == b"vide" and size[0] and size[1]: + out.append(_field(f"{prefix}:Width", f"{label} Width", size[0] >> 16, "int")) + out.append(_field(f"{prefix}:Height", f"{label} Height", size[1] >> 16, "int")) + + stbl = rd.find(mdia, b"minf", b"stbl") if mdia is not None else None + if stbl is None: + return + stsd = rd.find(stbl, b"stsd") + if stsd is not None: + entry = stsd.data_start + 8 + head = rd.u("I4s", entry) + if head is not None: + out.append(_field(f"{prefix}:CodecID", f"{label} Codec", _text(bytes(head[1])).strip(), "str")) + if handler == b"soun": + audio = rd.u("HHHHI", entry + 24) + if audio is not None: + if audio[0]: + out.append(_field(f"{prefix}:Channels", f"{label} Channels", audio[0], "int")) + if audio[4] >> 16: + out.append(_field(f"{prefix}:SampleRate", f"{label} Sample Rate (Hz)", audio[4] >> 16, "int")) + stts = rd.find(stbl, b"stts") + if stts is not None and handler == b"vide" and scale and duration: + count = rd.u("I", stts.data_start + 4) + if count and count[0] * 8 + 8 <= stts.end - stts.data_start: + samples = sum(rd.u("II", stts.data_start + 8 + 8 * i)[0] for i in range(count[0])) + if samples: + out.append(_field(f"{prefix}:FrameRate", f"{label} Frame Rate", + round(samples * scale / duration, 3), "float")) + + +def _mdta_value(kind: int, raw: bytes): + """(value, field_type) of a ``data`` atom, ``None`` when the type is not understood.""" + if kind == 1: + return _text(raw), "str" + if kind in (21, 22) and len(raw) in (1, 2, 4, 8): + return int.from_bytes(raw, "big", signed=(kind == 21)), "int" + if kind == 23 and len(raw) == 4: + value = struct.unpack(">f", raw)[0] + return (float(value), "float") if math.isfinite(value) else None + if kind == 24 and len(raw) == 8: + value = struct.unpack(">d", raw)[0] + return (float(value), "float") if math.isfinite(value) else None + return None + + +def _mdta(rd: Reader, moov: Box, out: list[MetadataField]) -> None: + meta = rd.find(moov, b"meta") + if meta is None: + return + start = meta.data_start + if bytes(rd.buf[start:start + 4]) == b"\x00\x00\x00\x00": # ISO full box; QuickTime has none + start += 4 + keys = ilst = None + for child in rd.children(start, meta.end): + if child.type == b"keys": + keys = child + elif child.type == b"ilst": + ilst = child + if keys is None or ilst is None: + return + count = rd.u("I", keys.data_start + 4) + names: list[str] = [] + pos = keys.data_start + 8 + for _ in range(min(count[0], 10_000) if count else 0): + entry = rd.u("I4s", pos) + if entry is None or entry[0] < 8 or pos + entry[0] > keys.end: + break + names.append(_text(bytes(rd.buf[pos + 8:pos + entry[0]]))) + pos += entry[0] + seen: set[str] = set() + for item in rd.children(ilst.data_start, ilst.end): + index = struct.unpack(">I", item.type)[0] + if not 1 <= index <= len(names) or names[index - 1] in seen: + continue + name = names[index - 1] + data = rd.find(item, b"data") + head = rd.u("I", data.data_start) if data is not None else None + if head is None: + continue + raw = bytes(rd.buf[data.data_start + 8:data.end]) + seen.add(name) + parsed = _mdta_value(head[0] & 0xFFFFFF, raw) + if parsed is not None: + out.append(_field(f"qt:mdta:{name}", f"QuickTime {name}", parsed[0], parsed[1])) + elif raw: + out.append(MetadataField.binary_field(f"qt:mdta:{name}", f"QuickTime {name}", raw, + deletable=False, source="qt")) + + +def _udta(rd: Reader, moov: Box, out: list[MetadataField]) -> None: + udta = rd.find(moov, b"udta") + if udta is None: + return + for child in rd.children(udta.data_start, udta.end): + if child.type[:1] != b"\xa9": + continue + head = rd.u("HH", child.data_start) + if head is None: + continue + text = _text(bytes(rd.buf[child.data_start + 4:min(child.end, child.data_start + 4 + head[0])])) + name = child.type.decode("latin-1") + out.append(_field(f"qt:udta:{name}", f"QuickTime {name}", text.rstrip("\x00"), "str")) + + +def _xmp(rd: Reader, moov: Box | None, out: list[MetadataField], warnings: list[str], writable: bool = False) -> None: + from core import xmp + from core.handlers.image_handler import _xmp_fields + packets = [b for b in rd.children() if b.type == b"uuid" and b.usertype == _XMP_UUID] + if moov is not None: + udta = rd.find(moov, b"udta") + if udta is not None: + packets += [b for b in rd.children(udta.data_start, udta.end) if b.type == b"XMP_"] + for box in packets[:1]: + try: + out.extend(_xmp_fields(xmp.XmpPacket(rd.payload(box)), writable)) + except xmp.XmpFormatError as exc: + warnings.append(f"XMP not readable: {exc}") + + +def qt_fields(path: Path, xmp_writable: bool = False) -> tuple[list[MetadataField], list[str]]: + """Structure fields of an MP4-family file and the notices to show with them.""" + out: list[MetadataField] = [] + warnings: list[str] = [] + with open_reader(path) as rd: + _ftyp(rd, out) + moov = rd.find(None, b"moov") + if moov is not None: + _mvhd(rd, moov, out) + for trak in rd.find_all(moov, b"trak")[:64]: + _track(rd, trak, out) + _mdta(rd, moov, out) + _udta(rd, moov, out) + _xmp(rd, moov, out, warnings, xmp_writable) + if any(f.key in ("qt:CreateDate", "qt:ModifyDate") for f in out): + warnings.append("QuickTime dates are UTC by specification, but many cameras store local time there.") + return out, warnings diff --git a/python/core/handlers/qt_patch.py b/python/core/handlers/qt_patch.py new file mode 100644 index 0000000..40b461c --- /dev/null +++ b/python/core/handlers/qt_patch.py @@ -0,0 +1,133 @@ +"""Size-preserving removal of QuickTime / MP4 structure fields (the ``qt:`` namespace). + +Nothing is ever moved: a removed atom is turned into a ``free`` atom of the same size with its payload +zeroed, a removed date is set to 0 (which means "not set"). The offsets of the media data therefore stay +valid and the media is never touched. ``plan`` computes the patches of the keys that can be removed, +``apply`` writes them; the original bytes are kept in the patch so that an Undo can write them back. +""" +from __future__ import annotations + +import struct +from dataclasses import dataclass +from pathlib import Path + +from core.isobmff import Box, Reader, open_reader + + +@dataclass(frozen=True) +class Patch: + offset: int + original: bytes + replacement: bytes + + +def _zero(rd: Reader, offset: int, size: int) -> Patch: + return Patch(offset, bytes(rd.buf[offset:offset + size]), bytes(size)) + + +def _free(rd: Reader, box: Box) -> Patch | None: + """The whole atom as a ``free`` atom of the same size (None when its header is not the plain 8 bytes).""" + if box.data_start - box.start != 8: + return None + original = bytes(rd.buf[box.start:box.end]) + return Patch(box.start, original, struct.pack(">I", len(original)) + b"free" + bytes(len(original) - 8)) + + +def _times(rd: Reader, box: Box) -> list[Patch]: + """Creation and modification time of a ``mvhd`` / ``tkhd`` / ``mdhd`` atom.""" + base = box.data_start + if base >= box.end: + return [] + width = 8 if rd.buf[base] == 1 else 4 + if base + 4 + 2 * width > box.end: + return [] + return [_zero(rd, base + 4, width), _zero(rd, base + 4 + width, width)] + + +def _mdta(rd: Reader, moov: Box, out: dict[str, list[Patch]]) -> None: + meta = rd.find(moov, b"meta") + if meta is None: + return + start = meta.data_start + if bytes(rd.buf[start:start + 4]) == bytes(4): + start += 4 + keys = ilst = None + for child in rd.children(start, meta.end): + if child.type == b"keys": + keys = child + elif child.type == b"ilst": + ilst = child + if keys is None or ilst is None: + return + count = rd.u("I", keys.data_start + 4) + names: list[tuple[str, int, int]] = [] # name, offset of the name bytes, length + pos = keys.data_start + 8 + for _ in range(min(count[0], 10_000) if count else 0): + entry = rd.u("I4s", pos) + if entry is None or entry[0] < 8 or pos + entry[0] > keys.end: + break + raw = bytes(rd.buf[pos + 8:pos + entry[0]]) + try: + text = raw.decode("utf-8") + except UnicodeDecodeError: + text = raw.decode("latin-1") + names.append((text, pos + 8, entry[0] - 8)) + pos += entry[0] + seen: set[str] = set() + for item in rd.children(ilst.data_start, ilst.end): + index = struct.unpack(">I", item.type)[0] + if not 1 <= index <= len(names): + continue + name, name_at, name_len = names[index - 1] + if name in seen: + continue + seen.add(name) + free = _free(rd, item) + if free is not None: + out[f"qt:mdta:{name}"] = [free, _zero(rd, name_at, name_len)] + + +def plan(path: Path) -> dict[str, list[Patch]]: + """Patches that remove each removable ``qt:`` field of the file, by key.""" + out: dict[str, list[Patch]] = {} + with open_reader(path) as rd: + moov = rd.find(None, b"moov") + if moov is None: + return out + mvhd = rd.find(moov, b"mvhd") + if mvhd is not None: + times = _times(rd, mvhd) + if times: + out["qt:CreateDate"], out["qt:ModifyDate"] = [times[0]], [times[1]] + for trak in rd.find_all(moov, b"trak")[:64]: + tkhd = rd.find(trak, b"tkhd") + if tkhd is None: + continue + base = tkhd.data_start + width = 8 if rd.buf[base] == 1 else 4 + track = rd.u("I", base + 4 + 2 * width) + if track is None: + continue + patches = _times(rd, tkhd) + mdhd = rd.find(trak, b"mdia", b"mdhd") + if mdhd is not None: + patches += _times(rd, mdhd) + if patches: + out[f"qt:track{track[0]}:CreateDate"] = patches + _mdta(rd, moov, out) + udta = rd.find(moov, b"udta") + if udta is not None: + for child in rd.children(udta.data_start, udta.end): + if child.type[:1] == b"\xa9": + free = _free(rd, child) + if free is not None: + out[f"qt:udta:{child.type.decode('latin-1')}"] = [free] + return out + + +def apply(path: Path, patches: list[Patch], restore: bool = False) -> None: + """Write the patches (or their original bytes when ``restore``) into the file, in place.""" + with open(path, "r+b") as stream: + for patch in patches: + stream.seek(patch.offset) + stream.write(patch.original if restore else patch.replacement) diff --git a/python/core/handlers/video_handler.py b/python/core/handlers/video_handler.py index 2eaa13a..209259d 100644 --- a/python/core/handlers/video_handler.py +++ b/python/core/handlers/video_handler.py @@ -8,50 +8,328 @@ from __future__ import annotations from pathlib import Path +from core import sniff from core.base_handler import BaseMetadataHandler from core.handlers._mp4_codec import Mp4Codec from core.handlers._tag_codecs import TagMutations +from core.handlers import mkv_fields, qt_edit, qt_patch +from core.handlers.qt_fields import qt_fields from core.handlers.hachoir_handler import extract_fields from core.models import MetadataField, MetadataRecord _RW_EXTS = frozenset({".mp4", ".m4v", ".mov", ".m4b", ".m4r"}) -_RO_EXTS = frozenset({".mkv", ".webm", ".avi", ".wmv", ".flv", ".ts", ".mts", ".m2ts"}) +_MP4_FORMATS = frozenset({"mp4", "mov", "m4a"}) + + +def _is_rw(path: Path) -> bool: + """An MP4-family file by name, or without extension but with MP4-family bytes.""" + suffix = path.suffix.lower() + return suffix in _RW_EXTS or (suffix == "" and sniff.detect(path) in _MP4_FORMATS) + + +_MKV_EXTS = frozenset({".mkv", ".webm", ".mka"}) +_RO_EXTS = frozenset({".avi", ".wmv", ".flv", ".ts", ".mts", ".m2ts"}) + + +def _is_mkv(path: Path) -> bool: + """A Matroska / WebM file by name, or without extension but with Matroska bytes.""" + suffix = path.suffix.lower() + return suffix in _MKV_EXTS or (suffix == "" and sniff.detect(path) == "mkv") class VideoHandler(TagMutations, BaseMetadataHandler): NAME = "video" - EXTENSIONS = _RW_EXTS | _RO_EXTS + EXTENSIONS = _RW_EXTS | _MKV_EXTS | _RO_EXTS PRIORITY = 25 def _open_codec(self, path: Path): - if path.suffix.lower() not in _RW_EXTS: + if not _is_rw(path): return None, False import mutagen.mp4 return Mp4Codec(mutagen.mp4.MP4(path)), True def read(self, path: Path) -> MetadataRecord: ext = path.suffix.lower() - if ext in _RW_EXTS: - fields, errors = _read_mp4(path) - writable = not errors + if _is_mkv(path): + return self._read_mkv(path) + if _is_rw(path): + fields, errors, notes, reason = _read_mp4(path) + writable = not errors and reason is None else: fields, errors = _read_hachoir(path) + notes, reason = [], None writable = False - record = self._make_record(path, read_errors=errors, + record = self._make_record(path, read_errors=errors, warnings=notes, supports_write=writable, supports_delete=writable) - if errors: + if writable: + self._mark_removable(path, fields) + if errors or not writable: for f in fields: f.editable = f.deletable = False + if errors: + pass elif not writable: - record.read_only_reason = f"{ext or 'this'} container is read-only" + record.read_only_reason = reason or f"{ext or 'this'} container is read-only" record.fields = fields return record + def _read_mkv(self, path: Path) -> MetadataRecord: + errors: list[str] = [] + try: + fields, notes, editable = mkv_fields.fields(path) + except Exception as exc: # damaged file: shown as an error, read-only + fields, notes, editable = [], [], False + errors.append(f"Matroska read error: {exc}") + record = self._make_record(path, read_errors=errors, warnings=notes, + supports_write=editable and not errors, supports_delete=editable and not errors) + record.fields = fields + if errors or not editable: + for f in fields: + f.editable = f.deletable = False + if not errors: + record.read_only_reason = "This recording cannot be changed" + record.dedupe_keys() + return record + + @staticmethod + def _mark_removable(path: Path, fields: list[MetadataField]) -> None: + """QuickTime structure fields and XMP can be removed (in place, nothing moves); only the + mutagen tags are editable.""" + try: + removable = set(qt_patch.plan(path)) + except Exception: + removable = set() + for f in fields: + if f.key.startswith("qt:"): + f.deletable = f.key in removable + f.editable = f.key in removable and ( + f.key in qt_edit.DATE_KEYS or f.key.endswith(":CreateDate") or f.key.startswith("qt:udta:") + or (f.key.startswith("qt:mdta:") and f.field_type == "str")) + + # ── mutation hooks: tags through mutagen, structure and XMP by in-place patches ── + @staticmethod + def _split(keys): + qt = [k for k in keys if k.startswith("qt:")] + xmp_keys = [k for k in keys if k.startswith("xmp:")] + return qt, xmp_keys, [k for k in keys if k not in qt and k not in xmp_keys] + + def _mkv_split(self, keys): + mkv = [k for k in keys if k.startswith("mkv:")] + return mkv, [k for k in keys if k not in mkv] + + def apply_write(self, tmp: Path, fields: list[MetadataField]) -> None: + if _is_mkv(tmp): + from core import ebml + from core.errors import ErrorCode, MetadataValueError, MetadataValueErrors + current = {f.key: f.value for f in mkv_fields.fields(tmp)[0]} + errors = [] + for f in fields: + if current.get(f.key) == f.value: + continue + try: + mkv_fields.set_text(tmp, f.key, f.value) + except (ebml.EbmlError, ValueError) as exc: + errors.append(MetadataValueError(f.key, ErrorCode.UNSUPPORTED, f"{f.key}: {exc}")) + if errors: + raise MetadataValueErrors(errors) + return + from core.errors import ErrorCode, MetadataValueError, MetadataValueErrors + from core.handlers.image_handler import ImageHandler + qt = [f for f in fields if f.key.startswith("qt:")] + xmp_fields = [f for f in fields if f.key.startswith("xmp:")] + rest = [f for f in fields if f not in qt and f not in xmp_fields] + errors = [] + current = {x.key: x.value for x in qt_fields(tmp)[0]} if qt else {} + for f in qt: + if current.get(f.key) == f.value: + continue # already what it should be: the file is left alone + try: + qt_edit.set_value(tmp, f.key, f.value) + except (qt_edit.LayoutError, ValueError) as exc: + errors.append(MetadataValueError(f.key, ErrorCode.UNSUPPORTED, f"{f.key}: {exc}")) + if errors: + raise MetadataValueErrors(errors) + if rest: + super().apply_write(tmp, rest) + if xmp_fields: + ImageHandler().apply_write(tmp, xmp_fields) + + def writable_fields(self, path: Path, record) -> list[MetadataField]: + """Tags (mutagen), QuickTime dates / text entries and XMP properties that can be added.""" + from core.handlers.image_handler import ImageHandler, _Mp4State + out = list(super().writable_fields(path, record)) + present = {f.key for f in record.fields} + if record.read_errors or not _is_rw(path): + return out + try: + fields = {f.key for f in qt_fields(path)[0]} + with_header = any(k in fields for k in ("qt:TimeScale",)) + except Exception: + return out + if with_header: + for key, label in (("qt:CreateDate", "Create Date"), ("qt:ModifyDate", "Modify Date")): + if key not in present: + out.append(MetadataField(key, label, None, field_type="datetime", source="qt")) + for atom, label in qt_edit.UDTA_ADDABLE.items(): + if f"qt:udta:{atom}" not in present: + out.append(MetadataField(f"qt:udta:{atom}", f"QuickTime {atom}", None, source="qt")) + for name, label in qt_edit.MDTA_ADDABLE.items(): + if f"qt:mdta:{name}" not in present: + out.append(MetadataField(f"qt:mdta:{name}", f"QuickTime {name}", None, source="qt")) + try: + state = _Mp4State.load(Path(path).read_bytes()) + out += ImageHandler()._addable(state, present) + except Exception: + pass # a file that cannot be parsed simply offers no fields to add + return out + + def writable_field(self, path: Path, key: str) -> MetadataField | None: + record = self.read(path) + found = next((f for f in record.addable_fields if f.key == key), None) + return found or super().writable_field(path, key) + + def apply_delete(self, tmp: Path, keys: list[str]) -> None: + if _is_mkv(tmp): + from core.errors import ErrorCode, MetadataValueError, MetadataValueErrors + plan = mkv_fields.removal_plan(tmp) + missing = [k for k in keys if k not in plan] + if missing: + raise MetadataValueErrors([MetadataValueError(k, ErrorCode.NOT_DELETABLE) for k in missing]) + for key in keys: + mkv_fields.apply_patches(tmp, plan[key]) + return + from core.handlers.image_handler import ImageHandler + qt, xmp_keys, rest = self._split(keys) + if qt: + patches = qt_patch.plan(tmp) + missing = [k for k in qt if k not in patches] + if missing: + from core.errors import ErrorCode, MetadataValueError, MetadataValueErrors + raise MetadataValueErrors([MetadataValueError(k, ErrorCode.NOT_DELETABLE) for k in missing]) + for key in qt: + qt_patch.apply(tmp, patches[key]) + if xmp_keys: + ImageHandler().apply_delete(tmp, xmp_keys) + if rest: + super().apply_delete(tmp, rest) + + @staticmethod + def _removed_in_place(path: Path, patches) -> bool: + """The bytes of a removed field are still the zeroed / free atom this code wrote.""" + with open(path, "rb") as stream: + for offset, original, replacement in patches: + stream.seek(offset) + if stream.read(len(replacement)) != replacement: + return False + return True + + def snapshot(self, tmp: Path, keys: list[str]): + from core.base_handler import ABSENT + if _is_mkv(tmp): + plan = mkv_fields.removal_plan(tmp) + current = {f.key: f for f in mkv_fields.fields(tmp)[0]} + return {k: (qt_edit.QtRaw(current[k].value, plan.get(k, ())) if k in current else ABSENT) for k in keys} + from core.handlers.image_handler import ImageHandler + qt, xmp_keys, rest = self._split(keys) + snap = super().snapshot(tmp, rest) if rest else {} + if xmp_keys: + snap.update(ImageHandler().snapshot(tmp, xmp_keys)) + if qt: + patches = qt_patch.plan(tmp) + current = {f.key: f for f in qt_fields(tmp)[0]} + for key in qt: + if key not in current: + snap[key] = ABSENT + continue + plan = tuple((p.offset, p.original, p.replacement) for p in patches.get(key, ())) + snap[key] = qt_edit.QtRaw(current[key].value, plan) + return snap + + def restore(self, tmp: Path, snap) -> None: + from core.base_handler import ABSENT + if _is_mkv(tmp): + plan = mkv_fields.removal_plan(tmp) + current = {f.key: f for f in mkv_fields.fields(tmp)[0]} + for key, raw in snap.items(): + if raw is ABSENT: + if key in plan: + mkv_fields.apply_patches(tmp, plan[key]) + elif key in current and current[key].value == raw.value: + continue + elif key not in current and raw.patches and self._removed_in_place(tmp, raw.patches): + mkv_fields.apply_patches(tmp, raw.patches, restore=True) + else: + mkv_fields.set_text(tmp, key, raw.value) + return + from core.handlers.image_handler import ImageHandler + qt, xmp_keys, rest = self._split(list(snap)) + if rest: + super().restore(tmp, {k: snap[k] for k in rest}) + if xmp_keys: + ImageHandler().restore(tmp, {k: snap[k] for k in xmp_keys}) + if qt: + current = {f.key: f for f in qt_fields(tmp)[0]} + patches = qt_patch.plan(tmp) + for key in qt: + raw = snap[key] + if raw is ABSENT: + if key in patches: + qt_patch.apply(tmp, patches[key]) # added by the change: take it out again + continue + if key in current and current[key].value == raw.value: + continue + if key not in current and raw.patches and self._removed_in_place(tmp, raw.patches): + with open(tmp, "r+b") as stream: + for offset, original, _replacement in raw.patches: + stream.seek(offset) + stream.write(original) + continue + qt_edit.set_value(tmp, key, raw.value) + + def canonical(self, field: MetadataField, value): + if field.key.startswith("mkv:"): + from core import values + return values.canonicalize(field, value) + if field.key.startswith("qt:"): + from core import values + result = values.canonicalize(field, value) + if field.field_type == "datetime": + from datetime import timezone + result = values.format_datetime(values.parse_datetime(result).astimezone(timezone.utc)) + if field.key.startswith(("qt:udta:", "qt:mdta:")): + try: + qt_edit.text_of(field.key, result) + except qt_edit.LayoutError as exc: + from core.errors import ErrorCode, MetadataValueError + raise MetadataValueError(field.key, ErrorCode.INVALID_VALUE, f"{field.key}: {exc}") from exc + return result + if field.key.startswith("xmp:"): + from core.handlers.image_handler import ImageHandler + return ImageHandler().canonical(field, value) + return super().canonical(field, value) + + def content_digest(self, path: Path) -> str | None: + """SHA-256 of the ``mdat`` atoms (the audio and video data).""" + if _is_mkv(path): + from core import ebml + return ebml.media_digest(path) + if not _is_rw(path): + return None + import hashlib + from core.isobmff import open_reader + digest = hashlib.sha256() + with open_reader(path) as rd: + for box in rd.children(): + if box.type == b"mdat": + for start in range(box.data_start, box.end, 1 << 20): + digest.update(bytes(rd.buf[start:min(start + (1 << 20), box.end)])) + return digest.hexdigest() + def write(self, path: Path, fields: list[MetadataField]) -> None: self._atomic_write(path, lambda tmp: self._write_tags(tmp, fields)) def _write_tags(self, path: Path, fields: list[MetadataField]) -> None: - if path.suffix.lower() not in _RW_EXTS: + if not _is_rw(path): raise NotImplementedError(f"Write not supported for {path.suffix}") self.apply_write(path, fields) @@ -59,15 +337,16 @@ def delete(self, path: Path, keys: list[str]) -> None: self._atomic_write(path, lambda tmp: self._delete_tags(tmp, keys)) def _delete_tags(self, path: Path, keys: list[str]) -> None: - if path.suffix.lower() not in _RW_EXTS: + if not _is_rw(path): raise NotImplementedError(f"Delete not supported for {path.suffix}") self.apply_delete(path, keys) -def _read_mp4(path: Path) -> tuple[list[MetadataField], list[str]]: +def _read_mp4(path: Path) -> tuple[list[MetadataField], list[str], list[str], str | None]: from core.handlers.audio_handler import stream_fields fields: list[MetadataField] = [] errors: list[str] = [] + notes: list[str] = [] try: import mutagen.mp4 mp4 = mutagen.mp4.MP4(path) @@ -81,7 +360,20 @@ def _read_mp4(path: Path) -> tuple[list[MetadataField], list[str]]: fields += Mp4Codec(mp4).fields() except Exception as e: errors.append(f"MP4 read error: {e}") - return fields, errors + try: + structure, notes = qt_fields(path, xmp_writable=True) + fields += structure + except Exception as e: # the structure is extra information: never fail the read for it + notes.append(f"QuickTime structure not read ({type(e).__name__})") + structure = [] + reason = None + if errors and structure: + # The tag library cannot open this file (e.g. a phone movie), but its structure was read: + # show it as a read-only file instead of a failed read. + reason = "The tags of this file cannot be edited yet" + notes.append(f"Tags not read: {errors[0]}") + errors = [] + return fields, errors, notes, reason # Moved to the hachoir handler; kept under the old name for existing callers. diff --git a/python/core/handlers/xml_package.py b/python/core/handlers/xml_package.py new file mode 100644 index 0000000..425f35f --- /dev/null +++ b/python/core/handlers/xml_package.py @@ -0,0 +1,173 @@ +"""Changing the metadata of OpenDocument and EPUB packages: an XML part of a ZIP is edited and written back. + +The part is parsed, the elements of the field are removed / rewritten / put back, and the package is written again +with every other member copied unchanged (the same way as for Office documents). A ``Mutations`` subclass says which +part holds the metadata and how a key is found in it. +""" +from __future__ import annotations + +import copy +import xml.etree.ElementTree as ET +from dataclasses import dataclass, field +from pathlib import Path +from zipfile import ZipFile + +from core.base_handler import ABSENT +from core.errors import ErrorCode, MetadataValueError, MetadataValueErrors +from core.handlers.office_handler import _rewrite_parts +from core.zipxml import read_xml + +for _prefix, _uri in { + "office": "urn:oasis:names:tc:opendocument:xmlns:office:1.0", "meta": "urn:oasis:names:tc:opendocument:xmlns:meta:1.0", + "dc": "http://purl.org/dc/elements/1.1/", "xlink": "http://www.w3.org/1999/xlink", + "grddl": "http://www.w3.org/2003/g/data-view#", "ooo": "http://openoffice.org/2004/office", + "opf": "http://www.idpf.org/2007/opf", "dcterms": "http://purl.org/dc/terms/"}.items(): + ET.register_namespace(_prefix, _uri) + + +@dataclass(frozen=True) +class Placed: + """One saved element: equality is its content; the position is only where to put it back.""" + content: bytes + index: int = field(default=0, compare=False) + + +@dataclass(frozen=True) +class AttrSaved: + owner: str # local name of the element that holds the attribute + attribute: str + value: str | None + + +def _bytes(element) -> bytes: + clone = copy.deepcopy(element) + clone.tail = None + return ET.tostring(clone, encoding="utf-8") + + +class Mutations: + """Hooks of ``BaseMetadataHandler`` on top of the XML part of a package.""" + + def part_name(self, archive: ZipFile) -> str: # pragma: no cover - implemented by the handlers + raise NotImplementedError + + def locate(self, root, key: str) -> list[tuple]: # pragma: no cover + """``(parent, element, attribute or None)`` of every place that holds ``key``.""" + raise NotImplementedError + + def make(self, root, key: str, value) -> list: # pragma: no cover + """New elements that hold ``value`` for ``key`` (an empty list: the key cannot be created).""" + raise NotImplementedError + + def removable(self, key: str) -> bool: + return True + + def text_of(self, key: str, value) -> str: + return str(value) + + # ── helpers ── + def _load(self, tmp: Path): + with ZipFile(tmp) as archive: + if any(n.startswith("META-INF/") and "signatures" in n for n in archive.namelist()): + raise MetadataValueError(None, ErrorCode.READ_ONLY, "Editing digitally signed packages is not supported") + name = self.part_name(archive) + return name, read_xml(archive, name) + + @staticmethod + def _store(tmp: Path, name: str, root) -> None: + _rewrite_parts(tmp, {name: ET.tostring(root, encoding="utf-8", xml_declaration=True)}) + + # ── mutation hooks ── + def canonical(self, field, value): + from core import values + return values.canonicalize(field, value) + + def apply_delete(self, tmp: Path, keys: list[str]) -> None: + name, root = self._load(tmp) + errors = [] + for key in keys: + hits = self.locate(root, key) + if not hits or not self.removable(key): + errors.append(MetadataValueError(key, ErrorCode.NOT_DELETABLE if hits else ErrorCode.UNKNOWN_FIELD)) + continue + self._remove(hits) + if errors: + raise MetadataValueErrors(errors) + self._store(tmp, name, root) + + @staticmethod + def _remove(hits) -> None: + for parent, element, attribute in hits: + if attribute is not None: + element.attrib.pop(attribute, None) + elif element in list(parent): + parent.remove(element) + + def apply_write(self, tmp: Path, fields) -> None: + name, root = self._load(tmp) + errors = [] + current = {f.key: f.value for f in self.read(tmp).fields} + fields = [f for f in fields if current.get(f.key) != f.value] # a value already there is not a change + if not fields: + return + for f in fields: + try: + hits = self.locate(root, f.key) + texts = [self.text_of(f.key, v) for v in (f.value if isinstance(f.value, list) else [f.value])] + if hits and all(h[2] is not None for h in hits): # an attribute + hits[0][1].attrib[hits[0][2]] = texts[0] + continue + parent = hits[0][0] if hits else None + made = self.make(root, f.key, f.value) + if not made: + raise MetadataValueError(f.key, ErrorCode.UNSUPPORTED, f"{f.key}: this field cannot be created here") + if not hits and made: + parent = made[0][0] + position = list(parent).index(hits[0][1]) if hits else len(parent) + self._remove(hits) + for offset, (_parent, element) in enumerate(made): + parent.insert(min(position + offset, len(parent)), element) + except MetadataValueError as exc: + errors.append(exc) + if errors: + raise MetadataValueErrors(errors) + self._store(tmp, name, root) + + def snapshot(self, tmp: Path, keys: list[str]): + with ZipFile(tmp) as archive: + root = read_xml(archive, self.part_name(archive)) + out = {} + for key in keys: + hits = self.locate(root, key) + out[key] = ABSENT if not hits else tuple( + (AttrSaved(element.tag.rsplit("}", 1)[-1], attribute, element.attrib.get(attribute)) + if attribute is not None else Placed(_bytes(element), list(parent).index(element))) + for parent, element, attribute in hits) + return out + + def restore(self, tmp: Path, snap) -> None: + name, root = self._load(tmp) + placed: list[tuple[int, object, bytes]] = [] + for key, raw in snap.items(): + hits = self.locate(root, key) + parent = hits[0][0] if hits else self.make_parent(root, key) + self._remove(hits) + if raw is ABSENT: + continue + for item in raw: + if isinstance(item, AttrSaved): + owner = next((e for e in root.iter() if e.tag.rsplit("}", 1)[-1] == item.owner), None) + if owner is not None and item.value is not None: + owner.attrib[item.attribute] = item.value + else: + placed.append((item.index, parent, item.content)) + for index, parent, content in sorted(placed, key=lambda entry: entry[0]): # original order + parent.insert(min(index, len(parent)), ET.fromstring(content)) + self._store(tmp, name, root) + + def writable_field(self, path: Path, key: str): + record = self.read(path) + return next((f for f in self.writable_fields(path, record) if f.key == key), None) + + def make_parent(self, root, key: str): # pragma: no cover - rarely needed + return next(iter(root)) diff --git a/python/core/heif.py b/python/core/heif.py new file mode 100644 index 0000000..a26361d --- /dev/null +++ b/python/core/heif.py @@ -0,0 +1,208 @@ +"""HEIC / HEIF / AVIF container structure: item table, locations, image properties, metadata layout. + +Pure parsing of the ISO base media ``meta`` box, with no dependency on the handlers (the image handler +and the HEIF handler both use it). The still image is never decoded; item payloads are read from their +``iloc`` extents, bounded in size. +""" +from __future__ import annotations + +import struct +from dataclasses import dataclass, field + +from core.isobmff import Box, Reader + +MAX_ITEM_BYTES = 16 * 1024 * 1024 +_NUL = bytes([0]) +_XMP_MIME = "application/rdf+xml" + + +@dataclass +class _Item: + item_id: int + kind: str = "" + content_type: str = "" + extents: list[tuple[int, int]] = field(default_factory=list) # (offset, length) + method: int = 0 # 0 = file offset, 1 = idat + loc: tuple[int, int, int, int, int] | None = None # single extent: where its fields are + + +def _cstring(buf: bytes, pos: int) -> tuple[str, int]: + end = buf.find(_NUL, pos) + if end < 0: + return buf[pos:].decode("utf-8", "replace"), len(buf) + return buf[pos:end].decode("utf-8", "replace"), end + 1 + + +def _meta_children(rd: Reader) -> tuple[Box | None, list[Box]]: + meta = rd.find(None, b"meta") + if meta is None: + return None, [] + return meta, rd.children(meta.data_start + 4, meta.end) # meta is a full box + + +def _items(rd: Reader, children: list[Box]) -> dict[int, _Item]: + items: dict[int, _Item] = {} + iinf = next((b for b in children if b.type == b"iinf"), None) + if iinf is not None: + version = rd.buf[iinf.data_start] + count_at = iinf.data_start + 4 + entries_at = count_at + (2 if version == 0 else 4) + for infe in rd.children(entries_at, iinf.end): + if infe.type != b"infe": + continue + data = rd.payload(infe) + v = data[0] + if v < 2 or len(data) < 12: + continue + item_id = int.from_bytes(data[4:6] if v == 2 else data[4:8], "big") + pos = 8 if v == 2 else 10 + kind = data[pos:pos + 4].decode("latin-1") + item = items.setdefault(item_id, _Item(item_id, kind)) + _, pos = _cstring(data, pos + 4) # item name + if kind == "mime": + item.content_type, _ = _cstring(data, pos) + iloc = next((b for b in children if b.type == b"iloc"), None) + if iloc is not None: + _locations(rd, iloc, items) + return items + + +def _locations(rd: Reader, iloc: Box, items: dict[int, _Item]) -> None: + data = rd.payload(iloc) + if len(data) < 8: + return + version = data[0] + offset_size, length_size = data[4] >> 4, data[4] & 15 + base_size, index_size = data[5] >> 4, (data[5] & 15) if version in (1, 2) else 0 + pos = 6 + count_width = 2 if version < 2 else 4 + count = int.from_bytes(data[pos:pos + count_width], "big") + pos += count_width + + def take(size: int) -> int: + nonlocal pos + value = int.from_bytes(data[pos:pos + size], "big") + pos += size + return value + + for _ in range(min(count, 10_000)): + item_id = take(2 if version < 2 else 4) + method = take(2) & 15 if version in (1, 2) else 0 + take(2) # data_reference_index + base = take(base_size) + extents = [] + loc = None + count_of_extents = take(2) + for _ in range(count_of_extents): + if index_size: + take(index_size) + offset_at = pos + offset = take(offset_size) + length_at = pos + extents.append((base + offset, take(length_size))) + if count_of_extents == 1: + # absolute positions in the file of the offset and length fields, their widths, the base offset + loc = (iloc.data_start + offset_at, offset_size, iloc.data_start + length_at, length_size, base) + if pos > len(data): + return + item = items.setdefault(item_id, _Item(item_id)) + item.extents, item.method, item.loc = extents, method, loc + + +def _payload(rd: Reader, item: _Item, idat: Box | None) -> bytes | None: + base = idat.data_start if item.method == 1 and idat is not None else 0 + if item.method not in (0, 1) or (item.method == 1 and idat is None): + return None + total = sum(length for _, length in item.extents) + if not item.extents or total > MAX_ITEM_BYTES: + return None + chunks = [] + for offset, length in item.extents: + start = base + offset + if length and start + length > len(rd.buf): + return None + chunks.append(bytes(rd.buf[start:start + length])) + return b"".join(chunks) + + +def _primary_properties(rd: Reader, children: list[Box]) -> dict[str, object]: + """Width, height and rotation of the primary item through ``pitm`` / ``ipma`` / ``ipco``.""" + pitm = next((b for b in children if b.type == b"pitm"), None) + iprp = next((b for b in children if b.type == b"iprp"), None) + if pitm is None or iprp is None: + return {} + version = rd.buf[pitm.data_start] + primary = int.from_bytes(rd.buf[pitm.data_start + 4:pitm.data_start + (6 if version == 0 else 8)], "big") + ipco = rd.find(iprp, b"ipco") + ipma = rd.find(iprp, b"ipma") + if ipco is None or ipma is None: + return {} + props = rd.children(ipco.data_start, ipco.end) + data = rd.payload(ipma) + version, flags = data[0], int.from_bytes(data[1:4], "big") + count = int.from_bytes(data[4:8], "big") + pos = 8 + wanted: list[int] = [] + for _ in range(min(count, 10_000)): + width = 2 if version < 1 else 4 + if pos + width + 1 > len(data): + break + item_id = int.from_bytes(data[pos:pos + width], "big") + pos += width + associations = data[pos] + pos += 1 + for _ in range(associations): + if flags & 1: + index = int.from_bytes(data[pos:pos + 2], "big") & 0x7FFF + pos += 2 + else: + index = data[pos] & 0x7F + pos += 1 + if item_id == primary: + wanted.append(index) + out: dict[str, object] = {"properties": [props[i - 1] for i in wanted if 0 < i <= len(props)]} + for box in out["properties"]: # type: ignore[union-attr] + if box.type == b"ispe" and box.end - box.data_start >= 12: + out["width"], out["height"] = struct.unpack_from(">II", rd.buf, box.data_start + 4) + elif box.type == b"irot" and box.end > box.data_start: + out["rotation"] = (rd.buf[box.data_start] & 3) * 90 + return out + + +MAX_EDIT_BYTES = 256 * 1024 * 1024 +_HEIF_BRANDS = frozenset({b"heic", b"heix", b"hevc", b"hevx", b"heim", b"heis", b"mif1", b"msf1", b"avif", b"avis"}) + + +def is_heif_brand(data: bytes) -> bool: + return data[4:8] == b"ftyp" and bytes(data[8:12]) in _HEIF_BRANDS + + +def metadata_layout(data) -> dict[str, dict]: + """Where the Exif item and the XMP item are, when each is stored in one extent of the file (the only + layout that can be changed): ``{"exif": {"item": (start, end), "range": (tiff start, end), "loc": ...}, + "xmp": {"item": (start, end), "range": (start, end), "loc": ...}}``.""" + from core.isobmff import Reader + rd = Reader(data) + meta, children = _meta_children(rd) + out: dict[str, dict] = {} + if meta is None: + return out + for item in _items(rd, children).values(): + if len(item.extents) != 1 or item.method != 0 or item.extents[0][1] <= 0 or item.loc is None: + continue + start, length = item.extents[0] + if start + length > len(data): + continue + if item.kind == "Exif" and length > 4: + skip = struct.unpack_from(">I", data, start)[0] + if 4 + skip < length: + out.setdefault("exif", {"item": (start, start + length), "range": (start + 4 + skip, start + length), + "loc": item.loc}) + elif item.kind == "mime" and item.content_type == _XMP_MIME: + out.setdefault("xmp", {"item": (start, start + length), "range": (start, start + length), "loc": item.loc}) + return out + + +def metadata_ranges(data) -> dict[str, tuple[int, int]]: + """Absolute ``(start, end)`` of the TIFF part of the Exif item and of the XMP item.""" + return {name: entry["range"] for name, entry in metadata_layout(data).items()} diff --git a/python/core/icc.py b/python/core/icc.py new file mode 100644 index 0000000..bb94797 --- /dev/null +++ b/python/core/icc.py @@ -0,0 +1,113 @@ +"""Read-only view of an ICC colour profile (header and the description / copyright tags). + +Written from the public ICC specification (ICC.1:2022, sections 7.2 and 10). The +profile is parsed defensively: anything unexpected yields fewer fields, never an +exception. Profile bytes are never modified by MetaLens. +""" +from __future__ import annotations + +import struct +from datetime import datetime, timezone + +from core import values +from core.models import MetadataField + +ICC_SIGNATURE = b"acsp" +HEADER_SIZE = 128 +MAX_TAGS = 200 + +_CLASSES = {b"scnr": "Input device", b"mntr": "Display device", b"prtr": "Output device", + b"link": "Device link", b"abst": "Abstract", b"spac": "Color space conversion", + b"nmcl": "Named color"} +_INTENTS = {0: "Perceptual", 1: "Relative colorimetric", 2: "Saturation", 3: "Absolute colorimetric"} +_PLATFORMS = {b"APPL": "Apple", b"MSFT": "Microsoft", b"SGI ": "Silicon Graphics", b"SUNW": "Sun", b"TGNT": "Taligent"} + + +def _text(raw: bytes) -> str: + return values.decode_text(raw.rstrip(b"\x00 "), ("ascii",)).strip() + + +def _tag_text(profile: bytes, offset: int, size: int) -> str | None: + """Text of a ``desc`` / ``text`` / ``mluc`` tag element, ``None`` if it is not one of them.""" + if offset + 8 > len(profile) or size < 8: + return None + element = profile[offset:offset + size] + kind = element[:4] + if kind == b"desc" and len(element) >= 12: + count = struct.unpack_from(">I", element, 8)[0] + return _text(element[12:12 + count]) or None + if kind == b"text": + return _text(element[8:]) or None + if kind == b"mluc" and len(element) >= 16: + number, record = struct.unpack_from(">II", element, 8) + best = None + for i in range(min(number, 64) if record >= 12 else 0): + at = 16 + record * i + if at + 12 > len(element): + break + length, start = struct.unpack_from(">II", element, at + 4) + chunk = element[start:start + length] + if len(chunk) != length: + continue + text = chunk.decode("utf-16-be", errors="replace").rstrip("\x00").strip() + if text and (best is None or element[at:at + 2] == b"en"): + best = text + return best + return None + + +def assemble(segments: list[bytes]) -> bytes | None: + """Profile bytes from the payloads of JPEG ``ICC_PROFILE`` APP2 segments (``None`` if incomplete).""" + parts: dict[int, bytes] = {} + total = 0 + for payload in segments: + if not payload.startswith(b"ICC_PROFILE\x00") or len(payload) < 14: + return None + number, total = payload[12], payload[13] + parts[number] = payload[14:] + if not parts or sorted(parts) != list(range(1, total + 1)): + return None + return b"".join(parts[i] for i in range(1, total + 1)) + + +def fields(profile: bytes) -> list[MetadataField]: + """Read-only ``icc:`` fields of ``profile`` (empty when it is not a valid ICC profile).""" + if len(profile) < HEADER_SIZE or profile[36:40] != ICC_SIGNATURE: + return [] + out: list[MetadataField] = [] + + def add(key: str, label: str, value, field_type: str = "str") -> None: + out.append(MetadataField(key, label, value, raw_value=value, editable=False, deletable=False, + field_type=field_type, source="icc")) + + major, minor = profile[8], profile[9] >> 4 + add("icc:Version", "ICC Profile Version", f"{major}.{minor}.{profile[9] & 15}") + add("icc:ProfileClass", "ICC Profile Class", _CLASSES.get(profile[12:16], _text(profile[12:16]))) + add("icc:ColorSpace", "ICC Color Space", _text(profile[16:20])) + add("icc:ConnectionSpace", "ICC Connection Space", _text(profile[20:24])) + year, month, day, hour, minute, second = struct.unpack_from(">6H", profile, 24) + try: + add("icc:CreationDate", "ICC Profile Date", + values.format_datetime(datetime(year, month, day, hour, minute, second, tzinfo=timezone.utc)), "datetime") + except ValueError: + pass # not a valid calendar date: the field is omitted + platform = profile[40:44] + if platform.strip(b"\x00"): + add("icc:Platform", "ICC Primary Platform", _PLATFORMS.get(platform, _text(platform))) + add("icc:RenderingIntent", "ICC Rendering Intent", + _INTENTS.get(struct.unpack_from(">I", profile, 64)[0] & 0xFFFF, "Unknown")) + if len(profile) < HEADER_SIZE + 4: + return out + count = min(struct.unpack_from(">I", profile, HEADER_SIZE)[0], MAX_TAGS) + for i in range(count): + at = HEADER_SIZE + 4 + 12 * i + if at + 12 > len(profile): + break + signature, offset, size = struct.unpack_from(">4sII", profile, at) + key, label = {b"desc": ("icc:Description", "ICC Profile Description"), + b"cprt": ("icc:Copyright", "ICC Profile Copyright")}.get(signature, (None, None)) + if key and not any(f.key == key for f in out): + text = _tag_text(profile, offset, size) + if text: + add(key, label, text) + return out diff --git a/python/core/image_containers.py b/python/core/image_containers.py new file mode 100644 index 0000000..bc960ae --- /dev/null +++ b/python/core/image_containers.py @@ -0,0 +1,226 @@ +"""Chunk containers of PNG and WebP files: parse into a list of chunks, change the metadata chunks, serialize. + +Only the chunks that hold metadata are ever created, replaced or removed; every other chunk (and the bytes +after the last chunk) is written back exactly as read, so the picture data is untouched. +PNG: ``eXIf`` holds the EXIF TIFF block, an ``iTXt`` chunk with the keyword ``XML:com.adobe.xmp`` holds the XMP +packet, ``tEXt`` / ``zTXt`` / ``iTXt`` hold text. WebP (RIFF): ``EXIF`` and ``XMP `` chunks, announced by the flags +of a ``VP8X`` header chunk (created when the file has only a bare ``VP8 `` / ``VP8L`` image). +""" +from __future__ import annotations + +import struct +import zlib + +PNG_SIGNATURE = b"\x89PNG\r\n\x1a\n" +XMP_KEYWORD = b"XML:com.adobe.xmp" +TEXT_TYPES = (b"tEXt", b"zTXt", b"iTXt") +MAX_CHUNKS = 100_000 + + +class ContainerError(ValueError): + """The file is not a well-formed PNG / WebP container (its metadata is left untouched).""" + + +# ──────────────────────────────── PNG ──────────────────────────────────────── + +class PngFile: + def __init__(self, chunks: list[list], tail: bytes) -> None: + self.chunks = chunks # [type, body] + self.tail = tail # bytes after IEND, kept as they are + + @classmethod + def parse(cls, data: bytes) -> "PngFile": + if data[:8] != PNG_SIGNATURE: + raise ContainerError("not a PNG file") + chunks: list[list] = [] + pos = 8 + while pos + 12 <= len(data) and len(chunks) < MAX_CHUNKS: + length = int.from_bytes(data[pos:pos + 4], "big") + kind = data[pos + 4:pos + 8] + end = pos + 8 + length + if end + 4 > len(data): + raise ContainerError(f"{kind.decode('latin-1')} chunk is truncated") + body = data[pos + 8:end] + if zlib.crc32(kind + body) != int.from_bytes(data[end:end + 4], "big"): + raise ContainerError(f"{kind.decode('latin-1')} chunk has a wrong checksum") + chunks.append([kind, body]) + pos = end + 4 + if kind == b"IEND": + return cls(chunks, data[pos:]) + raise ContainerError("no IEND chunk") + + def to_bytes(self) -> bytes: + out = [PNG_SIGNATURE] + for kind, body in self.chunks: + out.append(struct.pack(">I", len(body)) + kind + body + struct.pack(">I", zlib.crc32(kind + body))) + out.append(self.tail) + return b"".join(out) + + # ── metadata chunks ── + def index_of(self, kind: bytes) -> list[int]: + return [i for i, (k, _) in enumerate(self.chunks) if k == kind] + + def xmp_indexes(self) -> list[int]: + return [i for i, (k, body) in enumerate(self.chunks) if k == b"iTXt" and body.startswith(XMP_KEYWORD + b"\x00")] + + def xmp_packet(self, index: int) -> bytes | None: + """Text of the XMP ``iTXt`` chunk at ``index`` (decompressed), ``None`` if it cannot be decoded.""" + body = self.chunks[index][1] + rest = body[len(XMP_KEYWORD) + 1:] + if len(rest) < 2: + return None + compressed = rest[0] + _lang, _, rest = rest[2:].partition(b"\x00") + _translated, _, text = rest.partition(b"\x00") + if compressed: + try: + return zlib.decompress(text) + except zlib.error: + return None + return text + + def _insert_position(self, hint: int | None) -> int: + if hint is not None and 1 <= hint <= len(self.chunks): + return min(hint, self._first_data()) + return self._first_data() + + def _first_data(self) -> int: + for i, (kind, _) in enumerate(self.chunks): + if kind in (b"IDAT", b"IEND"): + return i + return len(self.chunks) + + def set_exif(self, tiff: bytes | None, hint: int | None = None) -> None: + self._replace(self.index_of(b"eXIf"), [b"eXIf", tiff] if tiff is not None else None, hint) + + def set_xmp(self, packet: bytes | None, hint: int | None = None) -> None: + chunk = None + if packet is not None: + chunk = [b"iTXt", XMP_KEYWORD + b"\x00\x00\x00\x00\x00" + packet] + self._replace(self.xmp_indexes(), chunk, hint) + + def _replace(self, indexes: list[int], chunk: list | None, hint: int | None) -> None: + position = indexes[0] if indexes else self._insert_position(hint) + for i in reversed(indexes): + del self.chunks[i] + if chunk is not None: + self.chunks.insert(min(position, len(self.chunks)), chunk) + + # ── text chunks ── + @staticmethod + def text_keyword(chunk: list) -> tuple[str, str] | None: + """(keyword, language) of a text chunk; ``None`` for other chunks and for the XMP carrier.""" + kind, body = chunk + if kind not in TEXT_TYPES or b"\x00" not in body: + return None + keyword, _, rest = body.partition(b"\x00") + if keyword == XMP_KEYWORD: + return None + language = "" + if kind == b"iTXt" and len(rest) >= 2: + language = rest[2:].partition(b"\x00")[0].decode("ascii", "replace").strip() + return keyword.decode("latin-1"), language + + def text_indexes(self, keyword: str, language: str = "") -> list[int]: + return [i for i, chunk in enumerate(self.chunks) if self.text_keyword(chunk) == (keyword, language)] + + def remove_text(self, keyword: str, language: str = "") -> list[tuple[int, list]]: + """Remove every text chunk of that keyword; the removed ``(index, chunk)`` pairs, for Undo.""" + found = [(i, self.chunks[i]) for i in self.text_indexes(keyword, language)] + for i, _ in reversed(found): + del self.chunks[i] + return found + + def put_text(self, items: list[tuple[int, list]]) -> None: + """Put back chunks removed by ``remove_text`` at their old positions.""" + for index, chunk in sorted(items, key=lambda item: item[0]): + self.chunks.insert(min(index, len(self.chunks)), [chunk[0], chunk[1]]) + + +# ──────────────────────────────── WebP ─────────────────────────────────────── + +FLAG_ICC, FLAG_ALPHA, FLAG_EXIF, FLAG_XMP, FLAG_ANIM = 0x20, 0x10, 0x08, 0x04, 0x02 + + +class WebpFile: + def __init__(self, chunks: list[list]) -> None: + self.chunks = chunks # [fourcc, body] + + @classmethod + def parse(cls, data: bytes) -> "WebpFile": + if data[:4] != b"RIFF" or data[8:12] != b"WEBP": + raise ContainerError("not a WebP file") + size = int.from_bytes(data[4:8], "little") + if size + 8 > len(data) or size < 4: + raise ContainerError("the RIFF size does not fit the file") + end = size + 8 + chunks: list[list] = [] + pos = 12 + while pos + 8 <= end and len(chunks) < MAX_CHUNKS: + fourcc = data[pos:pos + 4] + length = int.from_bytes(data[pos + 4:pos + 8], "little") + if pos + 8 + length > end: + raise ContainerError(f"{fourcc.decode('latin-1')} chunk is truncated") + chunks.append([fourcc, data[pos + 8:pos + 8 + length]]) + pos += 8 + length + (length & 1) + if not chunks: + raise ContainerError("no chunks") + return cls(chunks) + + def to_bytes(self) -> bytes: + body = b"WEBP" + for fourcc, payload in self.chunks: + body += fourcc + struct.pack(" list[int]: + return [i for i, (k, _) in enumerate(self.chunks) if k == fourcc] + + # ── canvas / VP8X ── + def canvas(self) -> tuple[int, int, bool] | None: + """(width, height, has alpha) of the picture from a VP8X header or the bare bitstream.""" + for fourcc, body in self.chunks: + if fourcc == b"VP8X" and len(body) >= 10: + return (int.from_bytes(body[4:7], "little") + 1, int.from_bytes(body[7:10], "little") + 1, + bool(body[0] & FLAG_ALPHA)) + if fourcc == b"VP8 " and len(body) >= 10 and body[3:6] == b"\x9d\x01\x2a": + return struct.unpack_from("= 5 and body[0] == 0x2F: + bits = int.from_bytes(body[1:5], "little") + return (bits & 0x3FFF) + 1, ((bits >> 14) & 0x3FFF) + 1, bool((bits >> 28) & 1) + return None + + def _ensure_vp8x(self) -> None: + if self.find(b"VP8X"): + return + canvas = self.canvas() + if canvas is None: + raise ContainerError("the picture size cannot be determined") + width, height, alpha = canvas + flags = FLAG_ALPHA if (alpha or self.find(b"ALPH")) else 0 + header = bytes([flags]) + bytes(3) + (width - 1).to_bytes(3, "little") + (height - 1).to_bytes(3, "little") + self.chunks.insert(0, [b"VP8X", header]) + + def _set_flag(self, flag: int, on: bool) -> None: + for chunk in self.chunks: + if chunk[0] == b"VP8X" and len(chunk[1]) >= 10: + value = chunk[1][0] | flag if on else chunk[1][0] & ~flag & 0xFF + chunk[1] = bytes([value]) + chunk[1][1:] + + def set_chunk(self, fourcc: bytes, payload: bytes | None, flag: int) -> None: + indexes = self.find(fourcc) + if payload is not None: + self._ensure_vp8x() + indexes = self.find(fourcc) + position = indexes[0] if indexes else len(self.chunks) + for i in reversed(indexes): + del self.chunks[i] + if payload is not None: + self.chunks.insert(min(position, len(self.chunks)), [fourcc, payload]) + self._set_flag(flag, payload is not None) + + def set_exif(self, tiff: bytes | None) -> None: + self.set_chunk(b"EXIF", tiff, FLAG_EXIF) + + def set_xmp(self, packet: bytes | None) -> None: + self.set_chunk(b"XMP ", packet, FLAG_XMP) diff --git a/python/core/isobmff.py b/python/core/isobmff.py new file mode 100644 index 0000000..7b6abe4 --- /dev/null +++ b/python/core/isobmff.py @@ -0,0 +1,109 @@ +"""Bounded reader for ISO base media boxes (MP4, MOV, M4A, and later HEIC/AVIF). + +Only reads: ``Reader`` walks the box tree of a byte buffer (an ``mmap`` of the +file or plain ``bytes``) without copying it. Every walk is bounded (depth, +boxes per level, total boxes) so a hostile or damaged file cannot make the +reader loop or allocate without limit; a box that does not fit its parent is +ignored (never raised), because metadata reading must not fail on the media. +""" +from __future__ import annotations + +import mmap +import struct +from contextlib import contextmanager +from dataclasses import dataclass +from pathlib import Path +from typing import Iterator + +MAX_BOXES_PER_LEVEL = 100_000 +MAX_TOTAL_BOXES = 500_000 +MAX_DEPTH = 12 + + +@dataclass(frozen=True) +class Box: + type: bytes + start: int # offset of the box header + data_start: int # first byte after the header (and after the usertype of a ``uuid`` box) + end: int # offset just after the box + usertype: bytes | None = None + + @property + def size(self) -> int: + return self.end - self.start + + +class Reader: + """Navigation over the boxes of ``buf``.""" + + def __init__(self, buf) -> None: + self.buf = buf + self._budget = MAX_TOTAL_BOXES + + def children(self, start: int = 0, end: int | None = None) -> list[Box]: + """The boxes laid out one after another in ``buf[start:end]`` (malformed tail dropped).""" + end = len(self.buf) if end is None else min(end, len(self.buf)) + boxes: list[Box] = [] + pos = start + while pos + 8 <= end and len(boxes) < MAX_BOXES_PER_LEVEL and self._budget > 0: + size, kind = struct.unpack_from(">I4s", self.buf, pos) + header = 8 + if size == 1: + if pos + 16 > end: + break + size = struct.unpack_from(">Q", self.buf, pos + 8)[0] + header = 16 + elif size == 0: + size = end - pos + usertype = None + if kind == b"uuid": + if pos + header + 16 > end: + break + usertype = bytes(self.buf[pos + header:pos + header + 16]) + header += 16 + if size < header or pos + size > end: + break + boxes.append(Box(kind, pos, pos + header, pos + size, usertype)) + self._budget -= 1 + pos += size + return boxes + + def find(self, parent: Box | None, *path: bytes) -> Box | None: + """First box reached by following ``path`` (types) from ``parent`` (``None`` = file root).""" + boxes = self.children() if parent is None else self.children(parent.data_start, parent.end) + for kind in path: + hit = next((b for b in boxes if b.type == kind), None) + if hit is None: + return None + parent = hit + boxes = self.children(hit.data_start, hit.end) + return parent + + def find_all(self, parent: Box | None, kind: bytes) -> list[Box]: + boxes = self.children() if parent is None else self.children(parent.data_start, parent.end) + return [b for b in boxes if b.type == kind] + + def payload(self, box: Box) -> bytes: + return bytes(self.buf[box.data_start:box.end]) + + def u(self, fmt: str, pos: int): + """Unpack ``fmt`` (big-endian) at ``pos``; ``None`` when it does not fit the buffer.""" + try: + return struct.unpack_from(">" + fmt, self.buf, pos) + except struct.error: + return None + + +@contextmanager +def open_reader(path: Path) -> Iterator[Reader]: + """A ``Reader`` over a read-only mapping of ``path`` (empty files give an empty reader).""" + with open(path, "rb") as stream: + try: + mapped = mmap.mmap(stream.fileno(), 0, access=mmap.ACCESS_READ) + except ValueError: # empty file + yield Reader(b"") + return + try: + yield Reader(mapped) + finally: + mapped.close() diff --git a/python/core/makernote.py b/python/core/makernote.py new file mode 100644 index 0000000..f94a819 --- /dev/null +++ b/python/core/makernote.py @@ -0,0 +1,95 @@ +"""A few well-known fields of the vendor MakerNote (Canon, Nikon, Sony), read only. + +The MakerNote itself is kept byte for byte when a file is rewritten; this module only lets its serial numbers, +owner name and similar identifying values be seen (and, by removing the MakerNote, be dropped).""" +from __future__ import annotations + +import struct +from dataclasses import dataclass + +_SIZES = {1: 1, 2: 1, 3: 2, 4: 4, 5: 8, 7: 1, 9: 4, 10: 8} +_MAX_ENTRIES = 512 + + +@dataclass(frozen=True) +class MakerValue: + vendor: str + name: str + label: str + value: str | int + + +# (tag) -> (name, label, kind) with kind "str" or "int" (first number of the value) +_CANON = {0x0006: ("ImageType", "Image type", "str"), 0x0007: ("FirmwareVersion", "Firmware version", "str"), + 0x0009: ("OwnerName", "Owner name", "str"), 0x000C: ("SerialNumber", "Serial number", "int"), + 0x0095: ("LensModel", "Lens model", "str"), 0x0096: ("InternalSerialNumber", "Internal serial number", "str")} +_NIKON = {0x0004: ("Quality", "Quality", "str"), 0x0007: ("FocusMode", "Focus mode", "str"), + 0x001D: ("SerialNumber", "Serial number", "str"), 0x00A0: ("CameraSerialNumber", "Camera serial number", "str"), + 0x00A7: ("ShutterCount", "Shutter count", "int")} +_SONY = {0x0102: ("Quality", "Quality", "int"), 0xB020: ("CreativeStyle", "Creative style", "str")} + + +def _entries(note: bytes, ifd_at: int, order: str, shift: int | None): + """Entries of the IFD at ``ifd_at`` of ``note``; a value offset ``v`` is at ``v + shift`` in ``note`` (None: not known).""" + try: + count = struct.unpack_from(order + "H", note, ifd_at)[0] + except struct.error: + return + for i in range(min(count, _MAX_ENTRIES)): + at = ifd_at + 2 + 12 * i + try: + tag, typ, n = struct.unpack_from(order + "HHL", note, at) + except struct.error: + return + size = _SIZES.get(typ) + if size is None or n > 1 << 20: + continue + total = size * n + if total <= 4: + data = note[at + 8:at + 8 + total] + else: + if at + 12 > len(note): + return + off = struct.unpack_from(order + "L", note, at + 8)[0] + shift if shift is not None else -1 + if off < 0: + continue + data = note[off:off + total] + if len(data) == total: + yield tag, typ, n, data + + +def _value(order: str, typ: int, data: bytes, kind: str): + if kind == "str": + if typ not in (2, 1, 7): + return None + text = data.split(b"\x00", 1)[0].decode("ascii", "replace").strip() + return text or None + code = {1: "B", 3: "H", 4: "L"}.get(typ) + if code is None: + return None + return struct.unpack_from(order + code, data, 0)[0] + + +def read(make: str, note: bytes, order: str, note_offset: int | None) -> list[MakerValue]: + """Known fields of a MakerNote. ``order`` is the byte order of the EXIF block and ``note_offset`` the offset of + the note from the start of the TIFF header (None if not known).""" + make = (make or "").strip().lower() + out: list[MakerValue] = [] + + def collect(vendor, table, ifd_at, byte_order, shift): + for tag, typ, _n, data in _entries(note, ifd_at, byte_order, shift): + if tag in table: + name, label, kind = table[tag] + value = _value(byte_order, typ, data, kind) + if value is not None: + out.append(MakerValue(vendor, name, label, value)) + + if note.startswith(b"Nikon\x00") and len(note) > 18 and note[10:12] in (b"II", b"MM"): + order2 = "<" if note[10:12] == b"II" else ">" + first = struct.unpack_from(order2 + "L", note, 14)[0] + collect("nikon", _NIKON, 10 + first, order2, 10) + elif note.startswith(b"SONY DSC \x00\x00\x00"): + collect("sony", _SONY, 12, order, -note_offset if note_offset is not None else None) + elif make.startswith("canon"): + collect("canon", _CANON, 0, order, -note_offset if note_offset is not None else None) + return out diff --git a/python/core/models.py b/python/core/models.py index 5492ef2..52997db 100644 --- a/python/core/models.py +++ b/python/core/models.py @@ -4,6 +4,7 @@ from pathlib import Path from typing import Any +from core import sensitivity from core.values import binary_descriptor @@ -76,6 +77,7 @@ def to_dict(self) -> dict: "multi": self.multi, "length": self.length, "binary": self._serialize_value(self.binary), + "sensitivity": sensitivity.classify(self.key), } @staticmethod @@ -169,6 +171,7 @@ def to_dict(self) -> dict: "format_supports_delete": self.format_supports_delete, "read_only_reason": self.read_only_reason, "addable_fields": [f.to_dict() for f in self.addable_fields], + "sensitivity_summary": sensitivity.summarize(f.key for f in self.fields), } diff --git a/python/core/path_security.py b/python/core/path_security.py index 7dd4573..4a3fd65 100644 --- a/python/core/path_security.py +++ b/python/core/path_security.py @@ -355,7 +355,8 @@ def secure_atomic_write( Args: target_path: Path object of the file to write (must exist and be writable) write_fn: Callable that takes (Path) and performs write operations - temp_suffix: Suffix for temporary file name (default: ".ml_tmp") + temp_suffix: Kept for compatibility (validated, no longer part of the name: + the copy is called like the target inside the private directory) Examples: >>> def writer(tmp_path): @@ -389,7 +390,9 @@ def secure_atomic_write( committed = False warnings: list[str] = [] try: - temp_path = directory / (target.stem + temp_suffix + target.suffix) + # Same name as the target: handlers that decide by name or extension behave + # on the copy exactly as on the file (the directory is private and new). + temp_path = directory / target.name _retry_transient(lambda: shutil.copy2(target, temp_path), busy=_SHARING_ERRORS, retry_denied=True, what="The file", probe=target if windows else None) if original_attrs is not None: diff --git a/python/core/registry.py b/python/core/registry.py index cce42f0..859f781 100644 --- a/python/core/registry.py +++ b/python/core/registry.py @@ -3,8 +3,9 @@ * Primary handlers own the format metadata of a file. ``get(path)`` returns the first one, by ``PRIORITY``, whose ``can_handle`` accepts the path. ``hachoir`` (``PRIORITY = 100``, accepts everything) is the catch-all for unknown - extensions and extension-less files: it recognizes the format from the - content (D3). + extensions: it recognizes the format from the content (D3). A file without + extension is first matched by content (``core.sniff``) to the handler that + owns its format; only when the bytes are not recognized does hachoir get it. * Layers (handlers with ``PRIMARY = False``, i.e. the filesystem layer) are never returned by ``get``. They apply to every file and the API adds their fields to each record: ``layers(path)`` / ``filesystem_layer()``. @@ -12,6 +13,7 @@ from __future__ import annotations from pathlib import Path +from core import sniff from core.base_handler import BaseMetadataHandler @@ -34,6 +36,12 @@ def get(cls, path: Path) -> BaseMetadataHandler: path = Path(path) if path.is_dir(): raise ValueError(f"No format handler for a directory: {path}") + if not path.suffix: + # No extension: the bytes decide (a JPEG called ``photo`` is a JPEG). + owner = sniff.handler_name(sniff.detect(path)) + for h in cls._handlers: + if owner and cls.is_primary(h) and h.NAME == owner: + return h for h in cls._handlers: if cls.is_primary(h) and h.can_handle(path): return h diff --git a/python/core/sensitivity.py b/python/core/sensitivity.py new file mode 100644 index 0000000..c73551c --- /dev/null +++ b/python/core/sensitivity.py @@ -0,0 +1,82 @@ +"""What a metadata field can reveal when the file is shared. + +``classify(key)`` returns the category of a field key, or ``None`` when the field +is technical (size, codec, colour space...). The category drives the privacy +indicators of the interface; it never changes whether a field can be edited. + +Categories: ``location`` (where), ``identity`` (who), ``device`` (what took or +made it, serial numbers, unique identifiers), ``datetime`` (when), ``software`` +(what edited it), ``thumbnail`` (an embedded copy of the picture), ``history`` (edit history and +other data stored but not shown), ``text`` (free text written by a person). +""" +from __future__ import annotations + +from fnmatch import fnmatchcase + +CATEGORIES = ("location", "identity", "device", "datetime", "software", "thumbnail", "history", "text") + +# Higher first: what the interface warns about most. +SEVERITY = {"location": 3, "identity": 3, "device": 2, "thumbnail": 2, "history": 2, "text": 1, "datetime": 1, "software": 1} + +# Patterns are matched case-insensitively against the whole key; the first match wins. +_RULES: tuple[tuple[str, tuple[str, ...]], ...] = ( + ("location", ("gps:*", "mkv:tag:*location*", "mkv:tag:*gps*", "xmp:exif:GPS*", "xmp:photoshop:City", "xmp:photoshop:State", "xmp:photoshop:Country", + "xmp:Iptc4xmpCore:Location", "xmp:Iptc4xmpCore:CountryCode", "xmp:Iptc4xmpExt:LocationShown*", + "iptc:City", "iptc:Province-State", "iptc:Country-PrimaryLocationName", "iptc:Sub-location", + "iptc:Country-PrimaryLocationCode", "qt:mdta:*location*", "hachoir:*:latitude", "hachoir:*:longitude", "hachoir:*:altitude", "qt:udta:©xyz", "exif:GPSInfo*")), + ("history", ("xmp:@hidden", "pdf:@xmp")), + ("thumbnail", ("mkv:attachment:*", "*Thumbnail*", "exif:Interop:*Thumb*", "ole:thumbnail", "xmp:xmp:Thumbnails")), + ("identity", ("iptc:@block", "mkv:tag:artist", "mkv:tag:author", "mkv:tag:composer", "mkv:tag:copyright", "mkv:tag:director", + "mkv:tag:actor", "exif:Artist", "exif:Copyright", "exif:CameraOwnerName", "maker:*:OwnerName", "exif:OwnerName", "exif:XPAuthor", + "xmp:dc:creator", "xmp:dc:rights", "xmp:dc:publisher", "xmp:photoshop:Credit", "xmp:photoshop:Source", + "xmp:xmpRights:*", "xmp:plus:*", "iptc:By-line", "iptc:By-lineTitle", "iptc:Credit", "iptc:Source", + "iptc:CopyrightNotice", "iptc:Contact", "pdf:Author", "office:author", "office:last_modified_by", + "office:app.Company", "office:app.Manager", "ole:author", "ole:last_author", "ole:company", + "ole:manager", "odf:initial-creator", "odf:creator", "epub:creator", "epub:contributor", + "epub:publisher", "fs:owner", "fs:group", "xattr:*", "qt:mdta:*author*", "qt:mdta:*artist*", "hachoir:*:author", "hachoir:*:artist", + "hachoir:*:copyright")), + ("device", ("exif:Make", "exif:Model", "exif:*SerialNumber", "exif:LensModel", "exif:LensMake", "exif:LensSpecification", + "exif:ImageUniqueID", "maker:*", "exif:MakerNote", "exif:BodySerialNumber", "xmp:aux:*", "xmp:xmpMM:DocumentID", + "xmp:xmpMM:InstanceID", "xmp:xmpMM:OriginalDocumentID", "xmp:tiff:Make", "xmp:tiff:Model", + "qt:mdta:*make", "qt:mdta:*model", "qt:mdta:*content.identifier", "qt:udta:©mak", "qt:udta:©mod", + "heif:*", "audio:encoder", "ole:app_name", "hachoir:*:camera_*")), + ("datetime", ("mkv:Info:DateUTC", "mkv:tag:date*", "exif:DateTime*", "exif:SubSec*", "exif:OffsetTime*", "xmp:xmp:CreateDate", "xmp:xmp:ModifyDate", + "xmp:xmp:MetadataDate", "xmp:photoshop:DateCreated", "iptc:DateCreated", "iptc:TimeCreated", + "iptc:DigitalCreationDate", "iptc:DigitalCreationTime", "pdf:CreationDate", "pdf:ModDate", + "office:created", "office:modified", "ole:create_time", "ole:modified_time", "ole:last_print", + "odf:creation-date", "odf:date", "odf:print-date", "epub:modified", "epub:date", "qt:*Date", + "qt:mdta:*creationdate", "pngchunk:tIME", "icc:CreationDate", "fs:mtime", "fs:ctime", "fs:atime", + "tag:TDRC", "tag:TDRL", "tag:TDEN", "tag:TDTG", "hachoir:*:creation_date", + "hachoir:*:date_time_original")), + ("software", ("mkv:Info:MuxingApp", "mkv:Info:WritingApp", "mkv:tag:encoder*", "exif:Software", "exif:ProcessingSoftware", "xmp:xmp:CreatorTool", "xmp:xmpMM:History", + "xmp:pdf:Producer", "pdf:Producer", "pdf:Creator", "office:app.Application", "office:app.AppVersion", + "odf:generator", "audio:vendor", "qt:mdta:*software", "qt:udta:©swr", "tag:TSSE", "tag:TENC", + "xmp:x:xmptk", "hachoir:*:producer")), + ("text", ("mkv:Info:Title", "mkv:tag:*", "exif:UserComment", "exif:ImageDescription", "exif:XPComment", "exif:XPSubject", "exif:XPTitle", + "exif:XPKeywords", "jpeg:Comment", "gif:comment", "png:*", "xmp:dc:description", "xmp:dc:title", + "xmp:dc:subject", "iptc:Caption-Abstract", "iptc:Keywords", "iptc:Headline", "iptc:SpecialInstructions", + "pdf:Subject", "pdf:Keywords", "pdf:Title", "office:description", "office:keywords", "office:subject", + "office:custom.*", "ole:comments", "ole:keywords", "odf:description", "odf:keyword", "odf:custom.*", + "epub:description", "tag:COMM*", "tag:USLT*", "tag:TXXX*")), +) + +_COMPILED = tuple((category, tuple(p.lower() for p in patterns)) for category, patterns in _RULES) + + +def classify(key: str) -> str | None: + """Category of the field with this key (``None`` for technical fields).""" + lowered = key.lower() + for category, patterns in _COMPILED: + if any(fnmatchcase(lowered, pattern) for pattern in patterns): + return category + return None + + +def summarize(keys) -> dict[str, int]: + """Number of fields per category among ``keys`` (categories without fields are omitted).""" + counts: dict[str, int] = {} + for key in keys: + category = classify(key) + if category: + counts[category] = counts.get(category, 0) + 1 + return dict(sorted(counts.items(), key=lambda item: -SEVERITY[item[0]])) diff --git a/python/core/sniff.py b/python/core/sniff.py new file mode 100644 index 0000000..ad851d0 --- /dev/null +++ b/python/core/sniff.py @@ -0,0 +1,137 @@ +"""Format recognition from the content of a file (magic numbers). + +Handlers pick files by extension. A file without an extension (``photo``) or +with a wrong one would then fall to the read-only generic reader, so the +registry asks this module which format the bytes are. Only the first bytes are +read (plus the member names of a ZIP package), never the whole file. + +``detect`` returns a stable format id (``jpeg``, ``png``, ``mp4``...) or ``None`` +and ``HANDLER_BY_FORMAT`` maps it to the ``NAME`` of the handler that owns it. +""" +from __future__ import annotations + +import zipfile +from pathlib import Path + +HEAD_BYTES = 64 + +# Brands (ISO/IEC 14496-12 ``ftyp`` major brand) -> format id. +_HEIF_BRANDS = {b"heic", b"heix", b"hevc", b"hevx", b"heim", b"heis", b"mif1", b"msf1"} +_AVIF_BRANDS = {b"avif", b"avis"} +_AUDIO_BRANDS = {b"M4A ", b"M4B ", b"M4P "} +_ASF_GUID = bytes.fromhex("3026b2758e66cf11a6d900aa0062ce6c") + +HANDLER_BY_FORMAT: dict[str, str] = { + "jpeg": "image", "png": "image", "gif": "image", "tiff": "image", "webp": "image", "bmp": "image", + "mp4": "video", "mov": "video", "mkv": "video", "avi": "video", + "m4a": "audio", "mp3": "audio", "flac": "audio", "ogg": "audio", "wav": "audio", "aiff": "audio", + "asf": "audio", "ape": "audio", "wavpack": "audio", "musepack": "audio", + "heif": "heif", "avif": "heif", + "odf": "odf", "epub": "epub", + "cr3": "image", "raf": "image", + "pdf": "pdf", "ooxml": "office", "ole": "ole", +} + + +def _read_head(path: Path) -> bytes: + try: + with open(path, "rb") as stream: + return stream.read(HEAD_BYTES) + except OSError: + return b"" + + +def _mpeg_frame_sync(head: bytes) -> bool: + """A plausible MPEG audio frame header (11 sync bits, valid version and layer).""" + if len(head) < 4 or head[0] != 0xFF or head[1] & 0xE0 != 0xE0: + return False + version, layer = (head[1] >> 3) & 3, (head[1] >> 1) & 3 + bitrate, rate = head[2] >> 4, (head[2] >> 2) & 3 + return version != 1 and layer != 0 and bitrate not in (0, 15) and rate != 3 + + +def _zip_kind(path: Path) -> str | None: + try: + with zipfile.ZipFile(path) as z: + names = z.namelist() + except (OSError, zipfile.BadZipFile, ValueError): + return None + if "[Content_Types].xml" in names: + return "ooxml" + if names and names[0] == "mimetype": + try: + with zipfile.ZipFile(path) as z: + kind = z.read("mimetype")[:100].decode("ascii", "replace").strip() + except (OSError, zipfile.BadZipFile, ValueError, KeyError): + return None + if kind == "application/epub+zip": + return "epub" + if kind.startswith("application/vnd.oasis.opendocument."): + return "odf" + return None + + +def detect(path: Path, head: bytes | None = None) -> str | None: + """Format id of the file at ``path`` or ``None`` when the content is not recognized.""" + path = Path(path) + if head is None: + head = _read_head(path) + if len(head) < 4: + return None + if head[:3] == b"\xff\xd8\xff": + return "jpeg" + if head[:8] == b"\x89PNG\r\n\x1a\n": + return "png" + if head[:6] in (b"GIF87a", b"GIF89a"): + return "gif" + if head[:4] in (b"II*\x00", b"MM\x00*"): + return "tiff" + if head[:2] == b"BM" and len(head) >= 14 and head[6:10] == b"\x00\x00\x00\x00": + return "bmp" + if head[:4] == b"RIFF" and len(head) >= 12: + return {b"WEBP": "webp", b"AVI ": "avi", b"WAVE": "wav"}.get(head[8:12]) + if head[:4] == b"FORM" and len(head) >= 12 and head[8:12] in (b"AIFF", b"AIFC"): + return "aiff" + if head[4:8] == b"ftyp" and len(head) >= 12: + brand = head[8:12] + if brand in _HEIF_BRANDS: + return "heif" + if brand in _AVIF_BRANDS: + return "avif" + if brand == b"crx ": + return "cr3" + if brand in _AUDIO_BRANDS: + return "m4a" + if brand == b"qt ": + return "mov" + return "mp4" + if head[:4] == b"\x1aE\xdf\xa3": + return "mkv" + if head[:4] == b"fLaC": + return "flac" + if head[:4] == b"OggS": + return "ogg" + if head[:16] == _ASF_GUID: + return "asf" + if head[:15] == b"FUJIFILMCCD-RAW": + return "raf" + if head[:4] == b"MAC ": + return "ape" + if head[:4] == b"wvpk": + return "wavpack" + if head[:4] in (b"MPCK", b"MP+\x07"): + return "musepack" + if head[:5] == b"%PDF-": + return "pdf" + if head[:8] == bytes.fromhex("d0cf11e0a1b11ae1"): + return "ole" + if head[:4] == b"PK\x03\x04": + return _zip_kind(path) + if head[:3] == b"ID3" or _mpeg_frame_sync(head): + return "mp3" + return None + + +def handler_name(format_id: str | None) -> str | None: + """``NAME`` of the handler owning ``format_id`` (``None`` when no handler does).""" + return HANDLER_BY_FORMAT.get(format_id) if format_id else None diff --git a/python/core/tiff_inplace.py b/python/core/tiff_inplace.py new file mode 100644 index 0000000..3f0de26 --- /dev/null +++ b/python/core/tiff_inplace.py @@ -0,0 +1,351 @@ +"""Removal of metadata entries from TIFF-based files (TIFF, DNG and TIFF-based RAW) without moving anything. + +An entry is taken out of its IFD by shifting the following entries up inside the same table (the table keeps +its place and size: the freed bytes at its end are zeroed) and the value it pointed to, when stored outside the +entry, is zeroed. No offset of the file changes, so strips, tiles, sub-IFDs, thumbnails and the MakerNote +(whose inner offsets are absolute) stay valid, and the removed data does not remain in the file. +``snapshot`` / ``insert`` bring an entry back (Undo). Only the metadata tags of ``removable`` are ever touched. +""" +from __future__ import annotations + +import hashlib +import struct +from dataclasses import dataclass + +from core.exif_codec import RAW_MAGICS, TYPE_SIZES + +MAX_IFDS = 64 +EXIF_POINTER, GPS_POINTER, INTEROP_POINTER, SUB_IFDS = 0x8769, 0x8825, 0xA005, 0x014A + +# Entries that describe the picture or are needed to read the file: never removed. +_KEEP_EXIF = frozenset({0x9000, 0x9101, 0xA000, 0xA001, 0xA002, 0xA003, 0xA005}) +_REMOVABLE_0TH = frozenset({0x010E, 0x010F, 0x0110, 0x0131, 0x0132, 0x013B, 0x013C, 0x8298, + 0x9C9B, 0x9C9C, 0x9C9D, 0x9C9E, 0x9C9F, 0xC614, 0xC62F, 0xC634, 0x8773}) + + +@dataclass(frozen=True) +class Entry: + ifd: str + tag: int + type: int + count: int + entry_at: int # offset of the 12-byte entry + data_at: int | None # offset of the value when stored outside the entry + size: int # size of the value in bytes + + +@dataclass(frozen=True) +class Ifd: + name: str + offset: int + entries: tuple[Entry, ...] + next_at: int | None # offset of the next-IFD pointer (None: the writer left it out) + + +class TiffError(ValueError): + pass + + +def removable(ifd: str, tag: int) -> bool: + """Whether the entry is metadata that can be removed (the picture's structure never is).""" + if ifd == "0th": + return tag in _REMOVABLE_0TH + if ifd == "Exif": + return tag not in _KEEP_EXIF + return ifd == "GPS" + + +def _byte_order(data) -> str: + head = bytes(data[:4]) + if head == b"II*\x00": + return "<" + if head == b"MM\x00*": + return ">" + if head in RAW_MAGICS: + return RAW_MAGICS[head] + raise TiffError("not a TIFF structure") + + +def parse(data) -> tuple[str, dict[str, Ifd]]: + """Byte order and the IFDs reachable from the header (``0th``, ``Exif``, ``GPS``, ``Interop``, ``1st``).""" + order = _byte_order(data) + size = len(data) + ifds: dict[str, Ifd] = {} + seen: set[int] = set() + + def read(name: str, offset: int) -> Ifd | None: + if offset in seen or offset < 8 or offset + 2 > size or len(seen) >= MAX_IFDS: + return None + seen.add(offset) + count = struct.unpack_from(order + "H", data, offset)[0] + end = offset + 2 + 12 * count + if end > size: + return None + entries = [] + for i in range(count): + at = offset + 2 + 12 * i + tag, typ, n = struct.unpack_from(order + "HHI", data, at) + width = TYPE_SIZES.get(typ) + if width is None: + entries.append(Entry(name, tag, typ, n, at, None, 0)) + continue + length = width * n + entries.append(Entry(name, tag, typ, n, at, struct.unpack_from(order + "I", data, at + 8)[0] if length > 4 else None, + length)) + # Some writers leave out the pointer of the last IFD: its value data then starts right after the table. + has_pointer = end + 4 <= size and not any( + e.data_at is not None and e.data_at < end + 4 and end < e.data_at + e.size for e in entries) + ifd = Ifd(name, offset, tuple(entries), end if has_pointer else None) + ifds[name] = ifd + return ifd + + first = read("0th", struct.unpack_from(order + "I", data, 4)[0]) + if first is None: + raise TiffError("the first IFD cannot be read") + for entry in first.entries: + if entry.tag in (EXIF_POINTER, GPS_POINTER) and entry.type in (4, 13) and entry.count == 1: + target = struct.unpack_from(order + "I", data, entry.entry_at + 8)[0] + sub = read("Exif" if entry.tag == EXIF_POINTER else "GPS", target) + if sub is not None and sub.name == "Exif": + for inner in sub.entries: + if inner.tag == INTEROP_POINTER and inner.type in (4, 13) and inner.count == 1: + read("Interop", struct.unpack_from(order + "I", data, inner.entry_at + 8)[0]) + following = struct.unpack_from(order + "I", data, first.next_at)[0] if first.next_at is not None else 0 + if following: + read("1st", following) + # A pointer overlapped by the value data of another IFD is not a pointer either. + spans = [(e.data_at, e.data_at + e.size) for i in ifds.values() for e in i.entries if e.data_at is not None] + for name, ifd in list(ifds.items()): + if ifd.next_at is not None and any(a < ifd.next_at + 4 and ifd.next_at < b for a, b in spans): + ifds[name] = Ifd(name, ifd.offset, ifd.entries, None) + return order, ifds + + +def _ranges(ifds: dict[str, Ifd]) -> list[tuple[int, int, str, int]]: + return [(e.data_at, e.data_at + e.size, e.ifd, e.tag) for ifd in ifds.values() for e in ifd.entries + if e.data_at is not None] + + +def remove(data: bytearray, ifd_name: str, tags: set[int], force: bool = False) -> int: + """Remove ``tags`` from one IFD in place; the number of entries removed.""" + order, ifds = parse(data) + ifd = ifds.get(ifd_name) + if ifd is None: + return 0 + doomed = [e for e in ifd.entries if e.tag in tags and (force or removable(ifd_name, e.tag))] + if not doomed: + return 0 + kept = [e for e in ifd.entries if e not in doomed] + other_ranges = [(a, b) for a, b, name, tag in _ranges(ifds) if not (name == ifd_name and tag in tags)] + # Own data of the removed entries, unless another entry points into the same bytes. + for entry in doomed: + if entry.data_at is not None and not any(a < entry.data_at + entry.size and entry.data_at < b + for a, b in other_ranges): + data[entry.data_at:entry.data_at + entry.size] = bytes(entry.size) + table = b"".join(bytes(data[e.entry_at:e.entry_at + 12]) for e in kept) + next_pointer = bytes(data[ifd.next_at:ifd.next_at + 4]) if ifd.next_at is not None else b"" + start = ifd.offset + old_end = start + 2 + 12 * len(ifd.entries) + len(next_pointer) + data[start:old_end] = bytes(old_end - start) + data[start:start + 2] = struct.pack(order + "H", len(kept)) + data[start + 2:start + 2 + len(table)] = table + data[start + 2 + len(table):start + 2 + len(table) + len(next_pointer)] = next_pointer + return len(doomed) + + +def snapshot(data, ifd_name: str, tag: int): + """What is needed to bring an entry back: ``(ifd, entry bytes, data offset, data bytes, table has a + next-IFD pointer)`` or ``None``.""" + _, ifds = parse(data) + ifd = ifds.get(ifd_name) + entry = next((e for e in ifd.entries if e.tag == tag), None) if ifd else None + if entry is None: + return None + value = bytes(data[entry.data_at:entry.data_at + entry.size]) if entry.data_at is not None else b"" + return (ifd_name, bytes(data[entry.entry_at:entry.entry_at + 12]), entry.data_at, value, ifd.next_at is not None) + + +def insert(data: bytearray, saved) -> None: + """Put a snapshot taken by ``snapshot`` back: the value first, then the entry at its place in the table.""" + ifd_name, raw, data_at, value, has_pointer = saved + order, ifds = parse(data) + ifd = ifds.get(ifd_name) + if ifd is None: + raise TiffError(f"the {ifd_name} table is gone") + tag = struct.unpack_from(order + "H", raw, 0)[0] + if any(e.tag == tag for e in ifd.entries): + return + table_end = ifd.offset + 2 + 12 * len(ifd.entries) + (4 if has_pointer else 0) + free = bytes(data[table_end:table_end + 12]) + if len(free) < 12 or any(free): + raise TiffError("the room left by the removed entry is no longer free") + if data_at is not None: + data[data_at:data_at + len(value)] = value + entries = list(ifd.entries) + position = next((i for i, e in enumerate(entries) if e.tag > tag), len(entries)) + table = b"".join(bytes(data[e.entry_at:e.entry_at + 12]) for e in entries) + table = table[:12 * position] + raw + table[12 * position:] + next_pointer = bytes(data[table_end - 4:table_end]) if has_pointer else b"" + start = ifd.offset + data[start:start + 2] = struct.pack(order + "H", len(entries) + 1) + data[start + 2:start + 2 + len(table)] = table + data[start + 2 + len(table):start + 2 + len(table) + len(next_pointer)] = next_pointer + + +def media_digest(data) -> str | None: + """SHA-256 of the strips / tiles / JPEG data the IFDs of the file point to (``None`` if unreadable).""" + try: + order, ifds = parse(data) + except (TiffError, struct.error): + return None + digest = hashlib.sha256() + ranges: list[tuple[int, int]] = [] + + def values(entry: Entry) -> list[int]: + width = TYPE_SIZES.get(entry.type) + if width is None or entry.count > 1_000_000: + return [] + at = entry.data_at if entry.data_at is not None else entry.entry_at + 8 + fmt = {1: "B", 3: "H", 4: "I", 8: "h", 9: "i"}.get(entry.type) + return list(struct.unpack_from(order + fmt * entry.count, data, at)) if fmt else [] + + for ifd in ifds.values(): + by_tag = {e.tag: e for e in ifd.entries} + for offsets_tag, counts_tag in ((0x0111, 0x0117), (0x0144, 0x0145)): + if offsets_tag in by_tag and counts_tag in by_tag: + ranges += list(zip(values(by_tag[offsets_tag]), values(by_tag[counts_tag]))) + if 0x0201 in by_tag and 0x0202 in by_tag: + ranges += list(zip(values(by_tag[0x0201]), values(by_tag[0x0202]))) + for start, length in sorted(set(ranges)): + if start + length > len(data): + return None + digest.update(struct.pack(">QQ", start, length)) + digest.update(bytes(data[start:start + length])) + return digest.hexdigest() + + +# ──────────────────────── editing: change in place, otherwise append ────────────────────────── + +_POINTER_TAGS = {"Exif": (EXIF_POINTER, "0th"), "GPS": (GPS_POINTER, "0th"), "Interop": (INTEROP_POINTER, "Exif")} +_EDITABLE_0TH_EXTRA = frozenset({0x0112}) # Orientation + + +def editable(ifd: str, tag: int) -> bool: + """Whether the entry may be changed or added (metadata only; never the picture structure).""" + return removable(ifd, tag) or (ifd == "0th" and tag in _EDITABLE_0TH_EXTRA) + + +def _align(buf: bytearray) -> None: + if len(buf) % 2: + buf.append(0) + + +def _append(buf: bytearray, data: bytes) -> int: + """Append ``data`` at an even offset; the offset it starts at.""" + _align(buf) + offset = len(buf) + if offset + len(data) >= 1 << 32: + raise TiffError("the file would exceed 4 GiB") + buf.extend(data) + return offset + + +def _entry_bytes(order: str, tag: int, typ: int, count: int, value: bytes, offset: int | None) -> bytes: + """A 12-byte entry: the value inline (left-justified) when it fits in four bytes, else its offset.""" + field = value.ljust(4, b"\x00") if offset is None else struct.pack(order + "I", offset) + return struct.pack(order + "HHI", tag, typ, count) + field + + +def _zero_unshared(buf: bytearray, ifds: dict[str, Ifd], entry: Entry) -> None: + if entry.data_at is None: + return + others = [(a, b) for a, b, name, tag in _ranges(ifds) if not (name == entry.ifd and tag == entry.tag)] + if not any(a < entry.data_at + entry.size and entry.data_at < b for a, b in others): + buf[entry.data_at:entry.data_at + entry.size] = bytes(entry.size) + + +def _point_to(buf: bytearray, order: str, ifds: dict[str, Ifd], name: str, offset: int) -> None: + """Make the file reach IFD ``name`` at ``offset`` (header, parent entry or previous IFD's pointer).""" + if name == "0th": + buf[4:8] = struct.pack(order + "I", offset) + elif name == "1st": + first = ifds["0th"] + if first.next_at is None: + raise TiffError("the first IFD has no room for a next-IFD pointer") + buf[first.next_at:first.next_at + 4] = struct.pack(order + "I", offset) + else: + tag, parent = _POINTER_TAGS[name] + entry = next(e for e in ifds[parent].entries if e.tag == tag) + buf[entry.entry_at + 8:entry.entry_at + 12] = struct.pack(order + "I", offset) + + +def ensure_ifd(buf: bytearray, name: str) -> None: + """Create an empty ``Exif`` / ``GPS`` / ``Interop`` IFD at the end of the file when it is missing.""" + order, ifds = parse(buf) + if name in ifds: + return + if name not in _POINTER_TAGS: + raise TiffError(f"cannot create the {name} IFD") + tag, parent = _POINTER_TAGS[name] + if parent not in ifds: + ensure_ifd(buf, parent) + offset = _append(buf, struct.pack(order + "HI", 0, 0)) + set_entry(buf, parent, tag, 4, 1, struct.pack(order + "I", offset)) + + +def set_entry(buf: bytearray, ifd_name: str, tag: int, typ: int, count: int, data: bytes) -> None: + """Change or add one entry. The value is written over the old one when it fits, else it is appended at + the end of the file and the entry points to it; a new entry makes the IFD table move to the end of the + file (its out-of-line values and everything else stay where they are). The old bytes are zeroed.""" + width = TYPE_SIZES.get(typ) + if width is None or width * count != len(data): + raise TiffError("the value does not match its type") + if ifd_name in _POINTER_TAGS and ifd_name not in parse(buf)[1]: + ensure_ifd(buf, ifd_name) + order, ifds = parse(buf) + ifd = ifds.get(ifd_name) + if ifd is None: + raise TiffError(f"no {ifd_name} IFD") + old = next((e for e in ifd.entries if e.tag == tag), None) + if old is not None: + fits = old.data_at is not None and len(data) > 4 and len(data) <= old.size + if len(data) <= 4: + offset = None + elif fits: + offset = old.data_at + buf[offset:offset + len(data)] = data + buf[offset + len(data):offset + old.size] = bytes(old.size - len(data)) + else: + offset = _append(buf, data) + if offset is not old.data_at: + _zero_unshared(buf, ifds, old) + buf[old.entry_at:old.entry_at + 12] = _entry_bytes(order, tag, typ, count, data, offset) + return + offset = None if len(data) <= 4 else _append(buf, data) + entries = [bytes(buf[e.entry_at:e.entry_at + 12]) for e in ifd.entries] + entries.append(_entry_bytes(order, tag, typ, count, data, offset)) + entries.sort(key=lambda raw: struct.unpack_from(order + "H", raw, 0)[0]) + following = bytes(buf[ifd.next_at:ifd.next_at + 4]) if ifd.next_at is not None else bytes(4) + table = struct.pack(order + "H", len(entries)) + b"".join(entries) + following + old_end = ifd.offset + 2 + 12 * len(ifd.entries) + (4 if ifd.next_at is not None else 0) + new_at = _append(buf, table) + buf[ifd.offset:old_end] = bytes(old_end - ifd.offset) + _point_to(buf, order, ifds, ifd_name, new_at) + + +def put_entry(buf: bytearray, saved) -> None: + """Undo of a change: write the entry (and its value) back as ``snapshot`` saw them. An entry that was + moved by a later change is pointed back at its original place; the moved copy is zeroed.""" + ifd_name, raw, data_at, value, _has_pointer = saved + order, ifds = parse(buf) + ifd = ifds.get(ifd_name) + tag = struct.unpack_from(order + "H", raw, 0)[0] + current = next((e for e in ifd.entries if e.tag == tag), None) if ifd else None + if current is None: + insert(buf, saved) + return + if current.data_at is not None and current.data_at != data_at: + _zero_unshared(buf, ifds, current) + if data_at is not None: + buf[data_at:data_at + len(value)] = value + buf[current.entry_at:current.entry_at + 12] = raw diff --git a/python/core/xmp.py b/python/core/xmp.py index d55ae7e..2efab7c 100644 --- a/python/core/xmp.py +++ b/python/core/xmp.py @@ -485,6 +485,19 @@ def read_value(self, prop: Prop, form: str) -> Any: return None return [li.text for li in items] + def other_languages(self, prop: Prop) -> list[tuple[str, str]]: + """``(language, text)`` of the alternatives of an ``rdf:Alt`` that are not the one ``read_value`` gives.""" + if prop.attr: + return [] + elems = prop.node.elements() + if len(elems) != 1 or not elems[0].is_(RDF, "Alt"): + return [] + items = elems[0].elements() + if any(not li.is_(RDF, "li") or li.elements() for li in items): + return [] + main = next((li for li in items if li.attr(XML_NS, "lang") == "x-default"), items[0] if items else None) + return [(li.attr(XML_NS, "lang") or "und", li.text) for li in items if li is not main] + def has_alternatives(self, prop: Prop) -> bool: """``prop`` is an ``rdf:Alt`` with more than one alternative (other languages).""" if prop.attr: diff --git a/python/core/zipxml.py b/python/core/zipxml.py new file mode 100644 index 0000000..9e39eb1 --- /dev/null +++ b/python/core/zipxml.py @@ -0,0 +1,35 @@ +"""Safe reading of XML parts of ZIP packages (ODF, EPUB): size cap, no DTD, no duplicated names.""" +from __future__ import annotations + +import xml.etree.ElementTree as ET +from zipfile import ZipFile + +MAX_PART_BYTES = 2 * 1024 * 1024 + + +class PackageError(ValueError): + """The package or one of its parts cannot be read safely.""" + + +def read_xml(archive: ZipFile, name: str, required: bool = True): + """Root element of the part ``name``; ``None`` when it is absent and not ``required``.""" + entries = [i for i in archive.infolist() if i.filename == name] + if not entries: + if required: + raise PackageError(f"{name} is missing") + return None + if len(entries) != 1: + raise PackageError(f"{name} appears more than once") + if entries[0].file_size > MAX_PART_BYTES: + raise PackageError(f"{name} is too large") + raw = archive.read(entries[0]) + if b" str: + return tag.rsplit("}", 1)[-1] diff --git a/python/main.py b/python/main.py index 422dc2f..8d5d64e 100644 --- a/python/main.py +++ b/python/main.py @@ -14,6 +14,8 @@ from fastapi.responses import JSONResponse from config import settings +from api.geotag import router as geotag_router +from api.operations import router as operations_router from api.routes import ApiCode, error_response, install_error_handlers, router from core.errors import error_entry @@ -59,6 +61,8 @@ def _auth_error(status: int, code: str, message: str) -> JSONResponse: install_error_handlers(app) app.include_router(router) +app.include_router(operations_router) +app.include_router(geotag_router) if __name__ == "__main__": diff --git a/python/tests/factories/__init__.py b/python/tests/factories/__init__.py index 759de88..d2e7573 100644 --- a/python/tests/factories/__init__.py +++ b/python/tests/factories/__init__.py @@ -40,7 +40,7 @@ def jpeg_camera(tmp_path: Path) -> Path: # Fixed order = order of the contract test ids. Add a module here only together # with the new file; never reorder. -MODULE_NAMES: tuple[str, ...] = ("image", "audio", "mp4", "pdf", "office", "ole", "hachoir", "fs") +MODULE_NAMES: tuple[str, ...] = ("image", "audio", "mp4", "pdf", "office", "ole", "hachoir", "fs", "heif", "packages", "mkv") @dataclass(frozen=True, eq=False) diff --git a/python/tests/factories/hachoir.py b/python/tests/factories/hachoir.py index 8714349..7a4bf7b 100644 --- a/python/tests/factories/hachoir.py +++ b/python/tests/factories/hachoir.py @@ -125,7 +125,7 @@ def avi_two_streams(tmp_path: Path) -> Path: return path -@sample("mkv_two_tracks") +@sample("mkv_two_tracks", mutations={"mkv:Info:Title": "T", "mkv:Info:MuxingApp": "M", "mkv:Info:WritingApp": "W"}) def mkv_two_tracks(tmp_path: Path) -> Path: path = tmp_path / "clip.mkv" path.write_bytes(mkv_bytes()) diff --git a/python/tests/factories/heif.py b/python/tests/factories/heif.py new file mode 100644 index 0000000..8fa6551 --- /dev/null +++ b/python/tests/factories/heif.py @@ -0,0 +1,77 @@ +"""HEIC / AVIF samples: ISO base media files with a ``meta`` box (no decodable picture). + +``ftyp`` + ``meta`` (``hdlr``, ``pitm``, ``iinf``, ``iloc``, ``iprp``) + ``mdat`` holding a +placeholder image payload, an Exif item (4-byte TIFF header offset + TIFF) and an XMP item. +""" +from __future__ import annotations + +import struct +from pathlib import Path + +from . import registrar +from .image import XMP_PACKET, camera_exif, icc_profile +from .mp4 import atom + +SAMPLES: list = [] +sample = registrar(SAMPLES, __name__) + +IMAGE_PAYLOAD = bytes(range(64)) * 2 + + +def _full(name: bytes, body: bytes, version: int = 0, flags: int = 0) -> bytes: + return atom(name, struct.pack(">I", (version << 24) | flags) + body) + + +def _infe(item_id: int, kind: bytes, name: bytes, content_type: bytes = b"") -> bytes: + body = struct.pack(">HH", item_id, 0) + kind + name + bytes(1) + if content_type: + body += content_type + bytes(1) + return _full(b"infe", body, version=2) + + +def _iloc(extents: list[tuple[int, int, int]]) -> bytes: + """Version 0 item location box: (item_id, offset, length) per item, 4-byte offsets and lengths.""" + body = bytes([0x44, 0x00]) + struct.pack(">H", len(extents)) + for item_id, offset, length in extents: + body += struct.pack(">HHHII", item_id, 0, 1, offset, length) + return _full(b"iloc", body) + + +def _meta(offsets: tuple[int, int, int], sizes: tuple[int, int, int], with_icc: bool, rotation: int) -> bytes: + ispe = _full(b"ispe", struct.pack(">II", 4032, 3024)) + irot = atom(b"irot", bytes([rotation])) + props = [ispe, irot] + if with_icc: + props.append(atom(b"colr", b"prof" + icc_profile())) + ipco = atom(b"ipco", b"".join(props)) + ipma = _full(b"ipma", struct.pack(">IHB", 1, 1, len(props)) + bytes(range(1, len(props) + 1))) + iinf = _full(b"iinf", struct.pack(">H", 3) + _infe(1, b"hvc1", b"Image") + _infe(2, b"Exif", b"Exif") + + _infe(3, b"mime", b"XMP", b"application/rdf+xml")) + hdlr = _full(b"hdlr", bytes(4) + b"pict" + bytes(12) + bytes(1)) + pitm = _full(b"pitm", struct.pack(">H", 1)) + iloc = _iloc([(1, offsets[0], sizes[0]), (2, offsets[1], sizes[1]), (3, offsets[2], sizes[2])]) + return _full(b"meta", hdlr + pitm + iinf + iloc + atom(b"iprp", ipco + ipma)) + + +def write_heif(path: Path, major: bytes = b"heic", compatible: tuple[bytes, ...] = (b"mif1", b"heic"), + with_icc: bool = True, rotation: int = 1) -> Path: + ftyp = atom(b"ftyp", major + bytes(4) + b"".join(compatible)) + exif = struct.pack(">I", 0) + camera_exif(thumbnail=False)[6:] + xmp = XMP_PACKET.encode("utf-8") + sizes = (len(IMAGE_PAYLOAD), len(exif), len(xmp)) + size = len(_meta((0, 0, 0), sizes, with_icc, rotation)) + start = len(ftyp) + size + 8 # first byte of the mdat payload + offsets = (start, start + sizes[0], start + sizes[0] + sizes[1]) + path.write_bytes(ftyp + _meta(offsets, sizes, with_icc, rotation) + atom(b"mdat", IMAGE_PAYLOAD + exif + xmp)) + return path + + +@sample("heic_camera", volatile=("heif:*",)) +def heic_camera(tmp_path: Path) -> Path: + return write_heif(tmp_path / "IMG_0001.heic") + + +@sample("avif_basic") +def avif_basic(tmp_path: Path) -> Path: + return write_heif(tmp_path / "photo.avif", major=b"avif", compatible=(b"avif", b"mif1", b"miaf"), + with_icc=False, rotation=0) diff --git a/python/tests/factories/image.py b/python/tests/factories/image.py index 6d1b8ba..6b85de7 100644 --- a/python/tests/factories/image.py +++ b/python/tests/factories/image.py @@ -334,7 +334,7 @@ def jpeg_no_exif(tmp_path: Path) -> Path: return path -@sample("jpeg_canon_makernote", volatile=_JPEG_VOLATILE, mutations=_DATE_MUTATIONS) +@sample("jpeg_canon_makernote", volatile=(*_JPEG_VOLATILE, "maker:*"), mutations=_DATE_MUTATIONS) def jpeg_canon_makernote(tmp_path: Path) -> Path: """Camera JPEG whose MakerNote holds ABSOLUTE offsets (Canon style); Make is stored NUL-padded, thumbnail in IFD1.""" @@ -343,7 +343,7 @@ def jpeg_canon_makernote(tmp_path: Path) -> Path: return path -@sample("jpeg_nikon_makernote", volatile=_JPEG_VOLATILE, mutations=_DATE_MUTATIONS) +@sample("jpeg_nikon_makernote", volatile=(*_JPEG_VOLATILE, "maker:*"), mutations=_DATE_MUTATIONS) def jpeg_nikon_makernote(tmp_path: Path) -> Path: """Camera JPEG whose MakerNote has its own TIFF header (Nikon type 3, relative offsets).""" path = tmp_path / "nikon.jpg" @@ -351,7 +351,7 @@ def jpeg_nikon_makernote(tmp_path: Path) -> Path: return path -@sample("jpeg_mpo", volatile=_JPEG_VOLATILE, mutations=_DATE_MUTATIONS) +@sample("jpeg_mpo", volatile=_JPEG_VOLATILE + ("mpf:image*:Size",), mutations=_DATE_MUTATIONS) def jpeg_mpo(tmp_path: Path) -> Path: """Two-frame MPO (APP1 Exif, APP2 MPF, second frame after the first EOI).""" path = tmp_path / "stereo.jpg" @@ -359,14 +359,55 @@ def jpeg_mpo(tmp_path: Path) -> Path: return path -@sample("photo_no_extension", tags={"read_only"}) +def icc_profile(description: str = "MetaLens test RGB", copyright_text: str = "Public domain") -> bytes: + """Minimal valid ICC v2 display profile with ``desc`` and ``cprt`` tags.""" + def desc(text: str) -> bytes: + raw = text.encode("ascii") + bytes(1) + body = b"desc" + bytes(4) + struct.pack(">I", len(raw)) + raw + bytes(4 + 4 + 2 + 1 + 67) + return body + bytes(-len(body) % 4) + + def text_tag(text: str) -> bytes: + body = b"text" + bytes(4) + text.encode("ascii") + bytes(1) + return body + bytes(-len(body) % 4) + + tags = [(b"desc", desc(description)), (b"cprt", text_tag(copyright_text))] + table_size = 4 + 12 * len(tags) + offset = 128 + table_size + table, body = struct.pack(">I", len(tags)), b"" + for signature, data in tags: + table += struct.pack(">4sII", signature, offset + len(body), len(data)) + body += data + header = bytearray(128) + struct.pack_into(">I", header, 0, 128 + len(table) + len(body)) + header[8:12] = bytes([2, 0x10, 0, 0]) + header[12:16], header[16:20], header[20:24] = b"mntr", b"RGB ", b"XYZ " + struct.pack_into(">6H", header, 24, 2026, 1, 2, 3, 4, 5) + header[36:40], header[40:44] = b"acsp", b"MSFT" + struct.pack_into(">I", header, 64, 1) + return bytes(header) + table + body + + +@sample("jpeg_icc_comment", volatile=_JPEG_VOLATILE, mutations=_DATE_MUTATIONS) +def jpeg_icc_comment(tmp_path: Path) -> Path: + """JPEG with an ICC profile split in two APP2 segments, two comments and camera EXIF.""" + profile = icc_profile() + half = len(profile) // 2 + parts = [segment(0xE2, b"ICC_PROFILE" + bytes(1) + bytes([n, 2]) + chunk) + for n, chunk in ((1, profile[:half]), (2, profile[half:]))] + comments = [segment(0xFE, "First comment é".encode("utf-8")), segment(0xFE, b"Second")] + path = tmp_path / "profile.jpg" + path.write_bytes(jpeg_with([segment(0xE1, camera_exif(thumbnail=False))] + parts + comments)) + return path + + +@sample("photo_no_extension", volatile=_JPEG_VOLATILE, mutations=_DATE_MUTATIONS) def photo_no_extension(tmp_path: Path) -> Path: path = tmp_path / "photo" path.write_bytes(jpeg_with([segment(0xE1, camera_exif(thumbnail=False))])) return path -@sample("tiff_dpi", tags={"read_only"}) +@sample("tiff_dpi") def tiff_dpi(tmp_path: Path) -> Path: from PIL import Image from PIL.TiffImagePlugin import ImageFileDirectory_v2 @@ -378,7 +419,23 @@ def tiff_dpi(tmp_path: Path) -> Path: return path -@sample("dng_like", tags={"read_only"}) +@sample("tiff_xmp_iptc", volatile=("exif:XMLPacket", "exif:tag_33723", "iptc:*")) +def tiff_xmp_iptc(tmp_path: Path) -> Path: + """TIFF as saved by an editor: XMP packet (tag 700) and IPTC-NAA record (tag 33723).""" + from PIL import Image + from PIL.TiffImagePlugin import ImageFileDirectory_v2 + path = tmp_path / "edited.tif" + info = ImageFileDirectory_v2() + info[0x010F] = "EditorCo" + info.tagtype[700] = 1 + info[700] = XMP_PACKET.encode("utf-8") + info.tagtype[33723] = 7 + info[33723] = iim(IPTC_DATASETS) + Image.new("RGB", (8, 8), (1, 2, 3)).save(path, "TIFF", tiffinfo=info) + return path + + +@sample("dng_like") def dng_like(tmp_path: Path) -> Path: from PIL import Image path = tmp_path / "raw.dng" @@ -386,7 +443,38 @@ def dng_like(tmp_path: Path) -> Path: return path -@sample("png_text_exif", tags={"read_only"}) +def le_raw_tiff(magic: bytes, make: bytes, model: bytes) -> bytes: + """Little-endian TIFF-like raw file (``magic`` in place of ``II*\0``) with a Make/Model IFD0 and + a UTF-8 (Exif 3.0, type 129) ImageDescription.""" + entries = [] + body = bytearray() + ifd_at, count = 8, 3 + data_at = ifd_at + 2 + 12 * count + 4 + for tag, typ, payload in ((0x010E, 129, "Descrizione è".encode("utf-8") + b"\x00"), + (0x010F, 2, make + b"\x00"), (0x0110, 2, model + b"\x00")): + entries.append(struct.pack(" Path: + """Olympus-style raw: big-endian EXIF structure under the ``MMOR`` header.""" + path = tmp_path / "photo.orf" + tiff = camera_exif(thumbnail=False)[6:] + path.write_bytes(b"MMOR" + tiff[4:]) + return path + + +@sample("rw2_like") +def rw2_like(tmp_path: Path) -> Path: + """Panasonic-style raw: little-endian IFD0 under the ``IIU\0`` header.""" + path = tmp_path / "photo.rw2" + path.write_bytes(le_raw_tiff(b"IIU\x00", b"Panasonic", b"DC-Test")) + return path + + +@sample("png_text_exif", mutations=_DATE_MUTATIONS) def png_text_exif(tmp_path: Path) -> Path: from PIL import Image from PIL.PngImagePlugin import PngInfo @@ -395,12 +483,14 @@ def png_text_exif(tmp_path: Path) -> Path: info.add_text("Title", "PNG title") info.add_text("Comment", "compressed comment " * 4, zip=True) info.add_itxt("Description", "Descrizione è 日本", lang="it", tkey="Beschreibung") + info.add_itxt("Description", "Description", lang="en", tkey="Description") + info.add_itxt("XML:com.adobe.xmp", XMP_PACKET) exif = camera_exif(gps=False, thumbnail=False) Image.new("RGB", (8, 8), (7, 8, 9)).save(path, "PNG", pnginfo=info, exif=exif) return path -@sample("webp_exif", tags={"read_only"}) +@sample("webp_exif", mutations=_DATE_MUTATIONS) def webp_exif(tmp_path: Path) -> Path: from PIL import Image path = tmp_path / "photo.webp" @@ -423,3 +513,59 @@ def bmp(tmp_path: Path) -> Path: path = tmp_path / "image.bmp" Image.new("RGB", (8, 8), (1, 1, 1)).save(path, "BMP") return path + + +# ── Canon CR3 and Fujifilm RAF (read-only samples) ── + +def simple_tiff(entries: list[tuple[int, int, bytes]]) -> bytes: + """Little-endian TIFF with one IFD: (tag, type, value bytes) entries, values stored after the table.""" + count = len(entries) + data_at = 8 + 2 + 12 * count + 4 + table, body = b"", b"" + for tag, typ, value in sorted(entries): + width = {1: 1, 2: 1, 3: 2, 4: 4, 5: 8}[typ] + n = len(value) // width + if len(value) <= 4: + table += struct.pack(" bytes: + def atom(name: bytes, payload: bytes) -> bytes: # ISO box: size + type + payload + return struct.pack(">I", 8 + len(payload)) + name + payload + + cmt1 = simple_tiff([(0x010F, 2, b"Canon" + bytes(1)), (0x0110, 2, b"Canon EOS R5" + bytes(1)), + (0x013B, 2, b"Jane Doe" + bytes(1))]) + cmt2 = simple_tiff([(0x829A, 5, struct.pack("I", 1) + b"crx isom") + atom(b"moov", canon) + atom(b"mdat", bytes(64)) + + +def raf_bytes() -> bytes: + preview = jpeg_with([segment(0xE1, camera_exif(thumbnail=False))]) + header = bytearray(0x100) + header[:16] = b"FUJIFILMCCD-RAW " + header[16:20] = b"0201" + struct.pack_into(">II", header, 0x54, 0x100, len(preview)) + return bytes(header) + preview + bytes(64) + + +@sample("cr3_like", tags={"read_only"}) +def cr3_like(tmp_path: Path) -> Path: + path = tmp_path / "IMG_0001.cr3" + path.write_bytes(cr3_bytes()) + return path + + +@sample("raf_like", tags={"read_only"}) +def raf_like(tmp_path: Path) -> Path: + path = tmp_path / "DSCF0001.raf" + path.write_bytes(raf_bytes()) + return path diff --git a/python/tests/factories/mkv.py b/python/tests/factories/mkv.py new file mode 100644 index 0000000..b3e6d07 --- /dev/null +++ b/python/tests/factories/mkv.py @@ -0,0 +1,45 @@ +"""A Matroska file as a muxer writes it: Info, Tracks, Tags, an attachment and a cluster with media data.""" +from __future__ import annotations + +import struct +from pathlib import Path + +from . import registrar +from .hachoir import MKV_DATE_NS, _el, _uint + +SAMPLES: list = [] +sample = registrar(SAMPLES, __name__) + +CLUSTER_PAYLOAD = bytes(range(256)) * 8 + + +def _tag(name: str, text: str, language: str = "und") -> bytes: + return _el(b"\x67\xc8", _el(b"\x45\xa3", name.encode()) + _el(b"\x44\x87", text.encode("utf-8")) + + _el(b"\x44\x7a", language.encode())) + + +def mkv_full_bytes() -> bytes: + header = _el(b"\x1a\x45\xdf\xa3", _el(b"\x42\x82", b"matroska") + _uint(b"\x42\x85", 4)) + info = _el(b"\x15\x49\xa9\x66", _uint(b"\x2a\xd7\xb1", 1_000_000) + _el(b"\x44\x89", struct.pack(">d", 2500.0)) + + _el(b"\x7b\xa9", "Vacanze è 2026".encode("utf-8")) + _el(b"\x4d\x80", b"libebml v1.4.5 + libmatroska v1.7.1") + + _el(b"\x57\x41", b"mkvmerge v81.0 ('Circles') 64-bit") + + _el(b"\x44\x61", MKV_DATE_NS.to_bytes(8, "big", signed=True)) + + _el(b"\xec", bytes(40))) + track = _el(b"\xae", _uint(b"\xd7", 1) + _uint(b"\x83", 1) + _el(b"\x86", b"V_MPEG4/ISO/AVC") + + _el(b"\x22\xb5\x9c", b"ita") + _el(b"\xe0", _uint(b"\xb0", 1920) + _uint(b"\xba", 1080))) + tags = _el(b"\x12\x54\xc3\x67", _el(b"\x73\x73", _el(b"\x63\xc0", _uint(b"\x68\xca", 50)) + + _tag("TITLE", "Vacanze") + _tag("ARTIST", "Ada Lovelace") + + _tag("ENCODER", "mkvmerge"))) + cover = _el(b"\x19\x41\xa4\x69", _el(b"\x61\xa7", _el(b"\x46\x6e", b"cover.jpg") + _el(b"\x46\x60", b"image/jpeg") + + _el(b"\x46\x5c", b"\xff\xd8\xff\xe0" + bytes(60)))) + cluster = _el(b"\x1f\x43\xb6\x75", _uint(b"\xe7", 0) + _el(b"\xa3", CLUSTER_PAYLOAD)) + segment = _el(b"\x18\x53\x80\x67", info + _el(b"\x16\x54\xae\x6b", track) + tags + cover + cluster) + return header + segment + + +@sample("mkv_full", mutations={"mkv:Info:Title": "T", "mkv:Info:MuxingApp": "M", "mkv:Info:WritingApp": "W", + "mkv:tag:TITLE": "V", "mkv:tag:ARTIST": "A", "mkv:tag:ENCODER": "E"}) +def mkv_full(tmp_path: Path) -> Path: + path = tmp_path / "holiday.mkv" + path.write_bytes(mkv_full_bytes()) + return path diff --git a/python/tests/factories/mp4.py b/python/tests/factories/mp4.py index 535fa14..1136021 100644 --- a/python/tests/factories/mp4.py +++ b/python/tests/factories/mp4.py @@ -73,20 +73,125 @@ def add_full_tags(path: Path) -> None: mp4.save() -@sample("m4a_tags") +_LOCATION = {"qt:mdta:com.apple.quicktime.location.ISO6709": "+10.0000+020.0000/", "qt:udta:©xyz": "+10.0000+020.0000/"} + + +@sample("m4a_tags", mutations=_LOCATION) def m4a_tags(tmp_path: Path) -> Path: path = write_mp4(tmp_path / "song.m4a") add_full_tags(path) return path -@sample("mp4_tags") +@sample("mp4_tags", mutations=_LOCATION) def mp4_tags(tmp_path: Path) -> Path: path = write_mp4(tmp_path / "clip.mp4", brand=b"isom") add_full_tags(path) return path -@sample("mov") +@sample("mov", mutations=_LOCATION) def mov(tmp_path: Path) -> Path: return write_mp4(tmp_path / "movie.mov", brand=b"qt ") + + +# ── Realistic phone-style movie: video + audio tracks, QuickTime keys, udta, XMP ── + +def _full(name: bytes, payload: bytes, version: int = 0, flags: int = 0) -> bytes: + return atom(name, struct.pack(">I", (version << 24) | flags) + payload) + + +def mac_seconds(year: int, month: int, day: int, hour: int = 0, minute: int = 0, second: int = 0) -> int: + from datetime import datetime, timezone + return int((datetime(year, month, day, hour, minute, second, tzinfo=timezone.utc) + - datetime(1904, 1, 1, tzinfo=timezone.utc)).total_seconds()) + + +def _matrix(rotation: int = 0) -> bytes: + a, b = {0: (0x10000, 0), 90: (0, 0x10000), 180: (-0x10000, 0), 270: (0, -0x10000)}[rotation] + return struct.pack(">9i", a, b, 0, -b, a, 0, 0, 0, 0x40000000) + + +def _tkhd(track_id: int, created: int, width: int, height: int, rotation: int = 0, audio: bool = False) -> bytes: + body = struct.pack(">IIIII", created, created, track_id, 0, 300300) + bytes(8) + body += struct.pack(">hhhH", 0, 0, 0x0100 if audio else 0, 0) + _matrix(rotation) + body += struct.pack(">II", width << 16, height << 16) + return _full(b"tkhd", body, flags=3) + + +def _mdhd(created: int, timescale: int, duration: int, lang: str = "und") -> bytes: + code = 0 if lang == "und" else sum((ord(c) - 0x60) << s for c, s in zip(lang, (10, 5, 0))) + return _full(b"mdhd", struct.pack(">IIIIHH", created, created, timescale, duration, code, 0)) + + +def _hdlr(kind: bytes, name: bytes) -> bytes: + return _full(b"hdlr", bytes(4) + kind + bytes(12) + name + b"\x00") + + +def _video_trak(track_id: int, created: int, chunk_offset: int, rotation: int) -> bytes: + stsd = _full(b"stsd", struct.pack(">I", 1) + atom(b"avc1", bytes(6) + struct.pack(">H", 1) + bytes(70))) + stts = _full(b"stts", struct.pack(">III", 1, 300, 1001)) + stsz = _full(b"stsz", struct.pack(">III", 0, 1, 1024)) + stco = _full(b"stco", struct.pack(">II", 1, chunk_offset)) + stbl = atom(b"stbl", stsd + stts + stsz + stco) + mdia = atom(b"mdia", _mdhd(created, 30000, 300300, "ita") + _hdlr(b"vide", b"VideoHandler") + atom(b"minf", stbl)) + return atom(b"trak", _tkhd(track_id, created, 1920, 1080, rotation) + mdia) + + +def _audio_trak(track_id: int, created: int, chunk_offset: int) -> bytes: + entry = bytes(6) + struct.pack(">H", 1) + bytes(8) + struct.pack(">HHHHI", 2, 16, 0, 0, 48000 << 16) + # ES descriptor: AAC-LC, 48 kHz, stereo (AudioSpecificConfig 0x1190) + esds = _full(b"esds", bytes.fromhex("03808080190001" "00" "0480808011" "401500000000fa0000fa00" + "058080800211" "90" "068080800102")) + stsd = _full(b"stsd", struct.pack(">I", 1) + atom(b"mp4a", entry + esds)) + stco = _full(b"stco", struct.pack(">II", 1, chunk_offset)) + stbl = atom(b"stbl", stsd + _full(b"stts", struct.pack(">III", 1, 1, 1024)) + stco) + mdia = atom(b"mdia", _mdhd(created, 48000, 144000, "eng") + _hdlr(b"soun", b"SoundHandler") + atom(b"minf", stbl)) + return atom(b"trak", _tkhd(track_id, created, 0, 0, audio=True) + mdia) + + +def _mdta_meta(items: list[tuple[str, int, bytes]], full_box: bool = True) -> bytes: + """``meta`` with an ``mdta`` handler, ``keys`` and ``ilst``. Phones write it without the + ISO full-box header (``full_box=False``); mutagen only parses the ISO form.""" + keys = struct.pack(">I", len(items)) + ilst = b"" + for index, (name, kind, value) in enumerate(items, start=1): + keys += atom(b"mdta", name.encode()) + ilst += struct.pack(">I", 8 + 16 + len(value)) + struct.pack(">I", index) \ + + struct.pack(">I", 16 + len(value)) + b"data" + struct.pack(">II", kind, 0) + value + return atom(b"meta", (bytes(4) if full_box else b"") + _hdlr(b"mdta", b"") + _full(b"keys", keys) + atom(b"ilst", ilst)) + + +def write_phone_movie(path: Path, full_box_meta: bool = True) -> Path: + from .image import XMP_PACKET + ftyp = atom(b"ftyp", b"qt " + struct.pack(">I", 0) + b"qt ") + payload = MDAT_PAYLOAD + mdat = atom(b"mdat", payload) + payload_at = len(ftyp) + 8 + created = mac_seconds(2026, 5, 1, 8, 20, 30) + udta_text = b"+45.4642+009.1900+120.500/" + udta = atom(b"udta", struct.pack(">I", 12 + len(udta_text)) + "©xyz".encode("latin-1") + + struct.pack(">HH", len(udta_text), 0x15C7) + udta_text) + meta = _mdta_meta([ + ("com.apple.quicktime.location.ISO6709", 1, b"+45.4642+009.1900+120.500/"), + ("com.apple.quicktime.make", 1, b"Apple"), + ("com.apple.quicktime.model", 1, "iPhone 15 Pro".encode()), + ("com.apple.quicktime.software", 1, b"17.4.1"), + ("com.apple.quicktime.creationdate", 1, b"2026-05-01T10:20:30+0200"), + ("com.apple.quicktime.live-photo.vitality-score", 23, struct.pack(">f", 0.5)), + ("com.apple.quicktime.content.identifier", 21, struct.pack(">i", 7)), + ], full_box=full_box_meta) + mvhd_body = struct.pack(">IIII", created, created, 1000, 10010) + struct.pack(">IH", 0x00010000, 0x0100) \ + + bytes(10) + _matrix() + bytes(24) + struct.pack(">I", 3) + moov = atom(b"moov", _full(b"mvhd", mvhd_body) + _video_trak(1, created, payload_at, 90) + + _audio_trak(2, created, payload_at + 512) + meta + udta) + xmp_box = atom(b"uuid", bytes.fromhex("be7acfcb97a942e89c71999491e3afac") + XMP_PACKET.encode("utf-8")) + path.write_bytes(ftyp + mdat + moov + xmp_box) # moov after mdat, like a phone recording + return path + + +@sample("mov_full", mutations=_LOCATION) +def mov_full(tmp_path: Path) -> Path: + path = write_phone_movie(tmp_path / "phone.mov") + add_full_tags(path) + return path diff --git a/python/tests/factories/packages.py b/python/tests/factories/packages.py new file mode 100644 index 0000000..6275354 --- /dev/null +++ b/python/tests/factories/packages.py @@ -0,0 +1,81 @@ +"""OpenDocument and EPUB samples (ZIP packages built with the standard library).""" +from __future__ import annotations + +import zipfile +from pathlib import Path + +from . import registrar + +SAMPLES: list = [] +sample = registrar(SAMPLES, __name__) + +ODF_META = """ + + + LibreOffice/7.6 + Relazione è + Budget + Descrizione + Ada + Bruno + 2026-01-02T03:04:05 + 2026-02-03T04:05:06Z + it-IT + 3 + PT1H2M3S + alpha + beta + Mercury + + + +""" + +EPUB_CONTAINER = """ + + + +""" + +EPUB_OPF = """ + + + urn:isbn:9780000000002 + Il libro è + Ada Lovelace + Charles Babbage + it + Casa Editrice + 2026-03-04 + Storia + 2026-03-05T06:07:08Z + + +""" + + +def write_odf(path: Path, mimetype: str = "application/vnd.oasis.opendocument.text") -> Path: + with zipfile.ZipFile(path, "w") as z: + z.writestr("mimetype", mimetype, compress_type=zipfile.ZIP_STORED) + z.writestr("content.xml", "") + z.writestr("meta.xml", ODF_META) + return path + + +def write_epub(path: Path) -> Path: + with zipfile.ZipFile(path, "w") as z: + z.writestr("mimetype", "application/epub+zip", compress_type=zipfile.ZIP_STORED) + z.writestr("META-INF/container.xml", EPUB_CONTAINER) + z.writestr("OEBPS/content.opf", EPUB_OPF) + return path + + +@sample("odt_meta") +def odt_meta(tmp_path: Path) -> Path: + return write_odf(tmp_path / "report.odt") + + +@sample("epub_book") +def epub_book(tmp_path: Path) -> Path: + return write_epub(tmp_path / "book.epub") diff --git a/python/tests/factories/pdf.py b/python/tests/factories/pdf.py index 2c6f9b1..7bdd826 100644 --- a/python/tests/factories/pdf.py +++ b/python/tests/factories/pdf.py @@ -128,7 +128,7 @@ def pdf_basic(tmp_path: Path) -> Path: return _save(writer, tmp_path / "basic.pdf") -@sample("pdf_xmp") +@sample("pdf_xmp", volatile=("pdf:@xmp", "xmp:*")) def pdf_xmp(tmp_path: Path) -> Path: writer = new_writer() set_info(writer, { diff --git a/python/tests/test_aliases.py b/python/tests/test_aliases.py new file mode 100644 index 0000000..8cb491e --- /dev/null +++ b/python/tests/test_aliases.py @@ -0,0 +1,21 @@ +"""The table of alternative field names served by the sidecar matches the one the interface searches with.""" +import re +from pathlib import Path + +import pytest + +from core.aliases import ALIASES + +JS = Path(__file__).resolve().parents[2] / "frontend" / "src" / "lib" / "aliases.js" + + +@pytest.mark.asyncio +async def test_aliases_are_served(client): + body = (await client.get("/aliases")).json() + assert body["aliases"] == ALIASES and body["aliases"]["exif:DateTimeOriginal"] + + +def test_server_and_interface_tables_agree(): + text = JS.read_text(encoding="utf-8") + found = {k: re.findall(r"'([^']*)'", v) for k, v in re.findall(r"^\s*'([^']+)':\s*\[(.*?)\],?\s*$", text, re.M)} + assert found == ALIASES diff --git a/python/tests/test_audio_details.py b/python/tests/test_audio_details.py new file mode 100644 index 0000000..cb923d7 --- /dev/null +++ b/python/tests/test_audio_details.py @@ -0,0 +1,39 @@ +"""Stream details and picture labels of audio files.""" +from core.registry import HandlerRegistry +from tests.factories import by_name + + +def _keys(tmp_path, name): + folder = tmp_path / name + folder.mkdir() + path = by_name(name).build(folder) + record = HandlerRegistry.get(path).read(path) + assert record.read_errors == [] + return {f.key: f for f in record.fields} + + +def test_mp3_stream_details(tmp_path): + keys = _keys(tmp_path, "mp3_v24_full") + assert keys["audio:mpeg_layer"].value == 3 + assert keys["audio:mpeg_version"].value in ("1", "2", "2.5", "1.0", "2.0") + assert keys["audio:channel_mode"].value in ("Stereo", "Joint stereo", "Dual channel", "Mono") + assert isinstance(keys["audio:crc_protected"].value, bool) + assert not keys["audio:mpeg_layer"].editable + + +def test_picture_label_names_type_and_description(tmp_path): + keys = _keys(tmp_path, "mp3_v24_full") + label = keys["tag:APIC:cover"].label + assert "Picture" in label and "cover" in label.lower() + + +def test_flac_stream_info(tmp_path): + keys = _keys(tmp_path, "flac_vorbis_picture") + assert keys["audio:total_samples"].value > 0 + assert keys["audio:max_blocksize"].value >= keys["audio:min_blocksize"].value + assert len(keys["audio:md5_signature"].value) == 32 if "audio:md5_signature" in keys else True + + +def test_vorbis_vendor_is_shown_and_computed(tmp_path): + keys = _keys(tmp_path, "ogg_vorbis") + assert keys["audio:vendor"].computed and not keys["audio:vendor"].editable diff --git a/python/tests/test_contract.py b/python/tests/test_contract.py index c3421c4..7a59c92 100644 --- a/python/tests/test_contract.py +++ b/python/tests/test_contract.py @@ -214,7 +214,8 @@ def _is_volatile(key: str, sample: Sample) -> bool: def _projection(record: MetadataRecord, sample: Sample, exclude=()) -> dict[str, tuple]: return {f.key: (_canon(f.value), f.field_type, f.multi, f.length, f.editable, f.deletable) - for f in record.fields if f.key not in exclude and not _is_volatile(f.key, sample)} + for f in record.fields if f.key not in exclude and not _is_volatile(f.key, sample) + and not (f.computed and f.key.partition("@")[0] in exclude)} def _assert_same(before: dict, after: dict, context: str) -> None: diff --git a/python/tests/test_cr3_raf.py b/python/tests/test_cr3_raf.py new file mode 100644 index 0000000..eedf1e6 --- /dev/null +++ b/python/tests/test_cr3_raf.py @@ -0,0 +1,44 @@ +"""Canon CR3 and Fujifilm RAF: EXIF, GPS and the preview picture, read only.""" +from core import sniff +from core.registry import HandlerRegistry +from tests.factories import image + + +def _read(path): + record = HandlerRegistry.get(path).read(path) + return record, {f.key: f for f in record.fields} + + +def test_cr3_exif_and_gps(tmp_path): + path = tmp_path / "a.cr3" + path.write_bytes(image.cr3_bytes()) + record, keys = _read(path) + assert record.read_errors == [] and record.handler_name == "image" + assert keys["exif:Make"].value == "Canon" and keys["exif:Model"].value == "Canon EOS R5" + assert keys["exif:ExposureTime"].value == "1/250" and keys["exif:ISOSpeedRatings"].value == 400 + assert keys["gps:LatitudeRef"].value == "N" and keys["gps:Latitude"].value[0] == "45/1" + assert not record.format_supports_write and not any(f.editable or f.deletable for f in record.fields) + + +def test_raf_reads_the_preview_picture_metadata(tmp_path): + path = tmp_path / "a.raf" + path.write_bytes(image.raf_bytes()) + record, keys = _read(path) + assert record.read_errors == [] and keys["exif:Make"].value == "Canon" and "gps:Latitude" in keys + assert not record.format_supports_write + + +def test_both_are_recognized_without_extension(tmp_path): + for name, data, kind in (("one", image.cr3_bytes(), "cr3"), ("two", image.raf_bytes(), "raf")): + path = tmp_path / name + path.write_bytes(data) + assert sniff.detect(path) == kind and HandlerRegistry.get(path).NAME == "image" + assert "exif:Make" in _read(path)[1] + + +def test_damaged_files_never_raise(tmp_path): + for data in (image.cr3_bytes(), image.raf_bytes()): + for cut in range(0, len(data), 37): + path = tmp_path / "cut.cr3" if data[4:8] == b"ftyp" else tmp_path / "cut.raf" + path.write_bytes(data[:cut]) + HandlerRegistry.get(path).read(path) diff --git a/python/tests/test_geotag.py b/python/tests/test_geotag.py new file mode 100644 index 0000000..4eacbaa --- /dev/null +++ b/python/tests/test_geotag.py @@ -0,0 +1,72 @@ +"""Geotag: fixed coordinates or a GPX track (api/geotag.py).""" +import pytest + +from core.registry import HandlerRegistry +from tests.factories import by_name + +GPX = """ + +100 +200 + +""" + + +def _build(tmp_path, name): + folder = tmp_path / name + folder.mkdir() + return by_name(name).build(folder) + + +def _keys(path): + return {f.key: f for f in HandlerRegistry.get(path).read(path).fields} + + +@pytest.mark.asyncio +async def test_a_position_is_written_to_files_that_have_none(client, tmp_path): + jpeg, tiff, bmp = _build(tmp_path, "jpeg_no_exif"), _build(tmp_path, "tiff_dpi"), _build(tmp_path, "bmp") + preview = (await client.post("/geotag", json={"paths": [str(jpeg), str(tiff), str(bmp)], "latitude": -33.8688, + "longitude": 151.2093, "altitude": 58.5})).json() + assert [bool(r.get("will_write")) for r in preview["results"]] == [True, True, False] + body = (await client.post("/geotag", json={"paths": [str(jpeg), str(tiff)], "latitude": -33.8688, + "longitude": 151.2093, "altitude": 58.5, "preview": False})).json() + assert body["ok"] and len(body["undo_tokens"]) == 2 + for path in (jpeg, tiff): + keys = _keys(path) + assert keys["gps:LatitudeRef"].value == "S" and keys["gps:LongitudeRef"].value == "E" + assert keys["gps:Latitude"].value[0] == "33/1" and keys["gps:Longitude"].value[0] == "151/1" + assert keys["gps:Altitude"].value == "5850/100" + for token in body["undo_tokens"]: + assert (await client.post("/undo", json={"token": token})).status_code == 200 + assert not any(k.startswith("gps:") for k in _keys(jpeg)) + + +@pytest.mark.asyncio +async def test_the_position_follows_the_time_of_each_picture_along_a_gpx_track(client, tmp_path): + path = _build(tmp_path, "jpeg_no_exif") + await client.post("/write", json={"path": str(path), "fields": [ + {"key": "exif:DateTimeOriginal", "value": "2026-01-02T04:00:00"}]}) + gpx = tmp_path / "track.gpx" + gpx.write_text(GPX) + body = (await client.post("/geotag", json={"paths": [str(path)], "gpx": str(gpx), "preview": False, + "max_gap_seconds": 7200})).json() + assert body["ok"], body + lat, lon = _keys(path)["gps:Latitude"].value, _keys(path)["gps:Longitude"].value + assert lat[0] == "45/1" and lat[1] == "3/1" # half way: 45.05 degrees + assert lon[0] == "9/1" and lon[1] == "6/1" # 9.1 degrees + far = (await client.post("/geotag", json={"paths": [str(path)], "gpx": str(gpx), "utc_offset_minutes": -600, + "max_gap_seconds": 60})).json() + assert far["results"][0]["will_write"] == [] and "no track point" in far["results"][0]["reason"] + + +@pytest.mark.asyncio +async def test_geotag_refuses_bad_requests(client, tmp_path): + path = _build(tmp_path, "jpeg_no_exif") + assert (await client.post("/geotag", json={"paths": [str(path)]})).status_code == 422 + assert (await client.post("/geotag", json={"paths": [str(path)], "latitude": 95, "longitude": 0})).status_code == 422 + bad = tmp_path / "bad.gpx" + bad.write_text(']>') + assert (await client.post("/geotag", json={"paths": [str(path)], "gpx": str(bad)})).status_code == 422 + empty = tmp_path / "empty.gpx" + empty.write_text("") + assert (await client.post("/geotag", json={"paths": [str(path)], "gpx": str(empty)})).status_code == 422 diff --git a/python/tests/test_heif.py b/python/tests/test_heif.py new file mode 100644 index 0000000..c4086c9 --- /dev/null +++ b/python/tests/test_heif.py @@ -0,0 +1,63 @@ +"""HEIC / AVIF reading (core/handlers/heif_handler.py).""" +from core.registry import HandlerRegistry +from tests.factories import by_name, heif + + +def _build(tmp_path, name): + folder = tmp_path / name + folder.mkdir() + return by_name(name).build(folder) + + +def _read(path): + record = HandlerRegistry.get(path).read(path) + return record, {f.key: f for f in record.fields} + + +def test_heic_exif_xmp_icc_and_size(tmp_path): + record, keys = _read(_build(tmp_path, "heic_camera")) + assert record.read_errors == [] and record.handler_name == "heif" + assert keys["img:format"].value == "HEIF" + assert (keys["img:width"].value, keys["img:height"].value) == (4032, 3024) + assert keys["heif:Rotation"].value == 90 + assert keys["heif:MajorBrand"].value == "heic" + assert keys["exif:Make"].value == "Canon" and keys["gps:Latitude"].value + assert keys["xmp:dc:title"].value == "Titolo è" + assert keys["icc:Description"].value == "MetaLens test RGB" + assert record.format_supports_write and record.format_supports_delete + assert keys["exif:Artist"].editable and not keys["img:width"].editable + assert keys["gps:Latitude"].deletable and keys["xmp:dc:title"].deletable and not keys["img:width"].deletable + + +def test_avif_is_recognized(tmp_path): + record, keys = _read(_build(tmp_path, "avif_basic")) + assert record.read_errors == [] and keys["img:format"].value == "AVIF" + assert "exif:Make" in keys and "icc:Version" not in keys + + +def test_heic_without_extension_is_found_by_content(tmp_path): + path = heif.write_heif(tmp_path / "IMG_0002") + assert HandlerRegistry.get(path).NAME == "heif" + assert "exif:Model" in _read(path)[1] + + +def test_damaged_and_truncated_files_never_raise(tmp_path): + data = heif.write_heif(tmp_path / "a.heic").read_bytes() + for cut in range(0, len(data), 29): + path = tmp_path / "cut.heic" + path.write_bytes(data[:cut]) + HandlerRegistry.get(path).read(path) + path = tmp_path / "junk.heic" + path.write_bytes(b"\x00\x00\x00\x18ftypheic" + b"\xff" * 64) + record, _ = _read(path) + assert isinstance(record.fields, list) + + +def test_item_payload_outside_the_file_is_ignored(tmp_path): + path = heif.write_heif(tmp_path / "a.heic") + data = bytearray(path.read_bytes()) + marker = data.index(b"iloc") + data[marker + 4 + 4 + 2 + 2 + 2 + 2 + 2:marker + 4 + 4 + 2 + 2 + 2 + 2 + 2 + 4] = b"\x7f\xff\xff\xff" + path.write_bytes(bytes(data)) + record, keys = _read(path) + assert record.read_errors == [] and "exif:Make" in keys diff --git a/python/tests/test_heif_sanitize.py b/python/tests/test_heif_sanitize.py new file mode 100644 index 0000000..0f1e040 --- /dev/null +++ b/python/tests/test_heif_sanitize.py @@ -0,0 +1,59 @@ +"""Removing EXIF / GPS / XMP from HEIC and AVIF files in place.""" +import pytest + +from core.handlers.heif_handler import HeifHandler +from core.heif import metadata_ranges +from core.registry import HandlerRegistry +from tests.factories import heif + + +def _keys(path): + return {f.key: f for f in HandlerRegistry.get(path).read(path).fields} + + +async def _sanitize(client, path, categories): + return (await client.post("/sanitize", json={"path": str(path), "categories": categories, "preview": False})).json() + + +@pytest.mark.asyncio +@pytest.mark.parametrize("name, major", [("a.heic", b"heic"), ("b.avif", b"avif")]) +async def test_location_and_device_are_removed_in_place_and_undo_is_exact(client, tmp_path, name, major): + path = heif.write_heif(tmp_path / name, major=major) + original, media = path.read_bytes(), HeifHandler().content_digest(path) + body = await _sanitize(client, path, ["location", "device", "identity"]) + assert body["ok"] and body["undo_token"], body + keys = _keys(path) + assert not any(k.startswith("gps:") for k in keys) and "exif:Make" not in keys and "exif:Artist" not in keys + data = path.read_bytes() + assert len(data) == len(original) and HeifHandler().content_digest(path) == media + assert b"Canon" not in data and b"EOS Test" not in data + assert (await client.post("/undo", json={"token": body["undo_token"]})).status_code == 200 + assert path.read_bytes() == original + + +@pytest.mark.asyncio +async def test_xmp_item_is_reduced_without_moving_anything(client, tmp_path): + path = heif.write_heif(tmp_path / "x.heic") + original = path.read_bytes() + body = await _sanitize(client, path, ["identity", "software", "text", "history"]) + assert body["ok"], body + assert "xmp:dc:creator" not in _keys(path) and len(path.read_bytes()) == len(original) + assert (await client.post("/undo", json={"token": body["undo_token"]})).status_code == 200 + assert path.read_bytes() == original + + +def test_the_picture_and_its_description_are_not_removable(tmp_path): + keys = _keys(heif.write_heif(tmp_path / "x.heic")) + for key in ("img:width", "heif:MajorBrand", "heif:Rotation", "exif:PixelXDimension", "exif:ExifVersion"): + assert key not in keys or not keys[key].deletable, key + + +def test_items_outside_the_file_are_not_edited(tmp_path): + path = heif.write_heif(tmp_path / "x.heic") + data = path.read_bytes() + assert set(metadata_ranges(data)) == {"exif", "xmp"} + assert metadata_ranges(data[:40]) == {} + broken = tmp_path / "broken.heic" + broken.write_bytes(data[:data.index(b"mdat") + 20]) # the items were stored after this point + record = HandlerRegistry.get(broken).read(broken) + assert not any(f.deletable for f in record.fields) diff --git a/python/tests/test_heif_write.py b/python/tests/test_heif_write.py new file mode 100644 index 0000000..561ed1b --- /dev/null +++ b/python/tests/test_heif_write.py @@ -0,0 +1,81 @@ +"""Editing and adding metadata in HEIC / AVIF files.""" +import pytest + +from core.handlers.heif_handler import HeifHandler +from core.heif import metadata_layout +from core.registry import HandlerRegistry +from tests.factories import heif + + +def _keys(path): + return {f.key: f for f in HandlerRegistry.get(path).read(path).fields} + + +async def _write(client, path, fields=(), delete=()): + return await client.post("/write", json={"path": str(path), "fields": list(fields), "delete": list(delete)}) + + +@pytest.mark.asyncio +async def test_a_short_value_is_written_where_it_is(client, tmp_path): + path = heif.write_heif(tmp_path / "a.heic") + original, media = path.read_bytes(), HeifHandler().content_digest(path) + resp = await _write(client, path, [{"key": "exif:Artist", "value": "Al"}]) + assert resp.status_code == 200, resp.text + assert _keys(path)["exif:Artist"].value == "Al" + assert len(path.read_bytes()) == len(original) and HeifHandler().content_digest(path) == media + undo = await client.post("/undo", json={"token": resp.json()["undo_token"]}) + assert undo.status_code == 200 and path.read_bytes() == original + + +@pytest.mark.asyncio +@pytest.mark.parametrize("name, major", [("a.heic", b"heic"), ("b.avif", b"avif")]) +async def test_a_longer_exif_moves_the_item_to_a_new_atom_at_the_end(client, tmp_path, name, major): + path = heif.write_heif(tmp_path / name, major=major) + original, media = path.read_bytes(), HeifHandler().content_digest(path) + name_long = "Grace Brewster Murray Hopper, rear admiral of the navy" + resp = await _write(client, path, [{"key": "exif:Artist", "value": name_long}, + {"key": "exif:Copyright", "value": "(c) 2026 Ada"}]) + assert resp.status_code == 200, resp.text + keys = _keys(path) + assert keys["exif:Artist"].value == name_long and keys["exif:Copyright"].value == "(c) 2026 Ada" + data = path.read_bytes() + assert len(data) > len(original) and HeifHandler().content_digest(path) == media + assert b"EOS Test" in data # unchanged entries are carried over + assert metadata_layout(data)["exif"]["item"][0] > len(original) # the item now lives at the end + undo = await client.post("/undo", json={"token": resp.json()["undo_token"]}) + assert undo.status_code == 200 + assert path.read_bytes() == original + + +@pytest.mark.asyncio +async def test_the_old_exif_bytes_do_not_remain_after_the_item_moves(client, tmp_path): + path = heif.write_heif(tmp_path / "a.heic") + resp = await _write(client, path, [{"key": "exif:Artist", "value": "x" * 300}]) + assert resp.status_code == 200, resp.text + layout = metadata_layout(path.read_bytes()) + data = path.read_bytes() + first = data.index(b"mdat") + assert b"EOS Test" not in data[first:layout["exif"]["item"][0] - 8] + + +@pytest.mark.asyncio +async def test_xmp_is_edited_and_grows_into_a_new_item(client, tmp_path): + path = heif.write_heif(tmp_path / "a.heic") + media = HeifHandler().content_digest(path) + resp = await _write(client, path, [{"key": "xmp:dc:subject", "value": ["uno", "due"]}]) + assert resp.status_code == 200, resp.text + assert _keys(path)["xmp:dc:subject"].value == ["uno", "due"] + big = ["parola" + str(i) for i in range(400)] + resp2 = await _write(client, path, [{"key": "xmp:dc:subject", "value": big}]) + assert resp2.status_code == 200, resp2.text + assert _keys(path)["xmp:dc:subject"].value == big and HeifHandler().content_digest(path) == media + + +@pytest.mark.asyncio +async def test_picture_description_cannot_be_edited_and_missing_xmp_cannot_be_created(client, tmp_path): + path = heif.write_heif(tmp_path / "a.heic") + before = path.read_bytes() + assert (await _write(client, path, [{"key": "img:width", "value": 5}])).status_code == 422 + assert path.read_bytes() == before + record = HandlerRegistry.get(path).read(path) + assert {"exif:Copyright"} <= {f.key for f in record.addable_fields} diff --git a/python/tests/test_image_handler.py b/python/tests/test_image_handler.py index 17cfb3b..f70693c 100644 --- a/python/tests/test_image_handler.py +++ b/python/tests/test_image_handler.py @@ -97,7 +97,8 @@ def test_tiff_with_dpi_reads_without_errors(tmp_path): fields = {f.key: f for f in record.fields} assert fields["img:dpi"].value == [300.0, 300.0] assert fields["exif:Make"].value == "ScannerCo" - assert not record.format_supports_write + assert record.format_supports_write and fields["exif:Make"].editable # TIFF metadata is editable + assert not fields["exif:ImageWidth"].editable if "exif:ImageWidth" in fields else True # ──────────────────────── N2: segments preserved byte for byte ─────────────── @@ -399,15 +400,17 @@ def test_png_webp_expose_exif_and_gif_comment(tmp_path): png = _fields(_make("png_text_exif", tmp_path)) assert png["exif:Make"].value == "Canon" assert png["png:Title"].value == "PNG title" - assert png["png:Description"].value == "Descrizione è 日本" + assert png["png:Description@it"].value == "Descrizione è 日本" assert png["png:Comment"].value.startswith("compressed comment") webp = _fields(_make("webp_exif", tmp_path)) assert webp["exif:Model"].value == "EOS Test" assert webp["xmp:dc:subject"].value == ["alpha", "beta ß"] gif = _fields(_make("gif_comment", tmp_path)) assert gif["gif:comment"].value == "GIF comment è" - for fields in (png, webp, gif): - assert all(not f.editable and not f.deletable for f in fields.values()) + assert all(not f.editable and not f.deletable for f in gif.values()) + # PNG and WebP metadata is editable since 0.5.0: the EXIF and XMP blocks, and PNG text can be removed. + assert png["exif:Make"].editable and png["png:Title"].deletable and not png["png:Title"].editable + assert webp["exif:Model"].editable and webp["xmp:dc:subject"].editable # ──────────────────────── XMP ──────────────────────────────────────────────── diff --git a/python/tests/test_integrity_regressions.py b/python/tests/test_integrity_regressions.py index 6597c13..de2bc00 100644 --- a/python/tests/test_integrity_regressions.py +++ b/python/tests/test_integrity_regressions.py @@ -84,7 +84,7 @@ def test_office_preserves_all_other_zip_members(tmp_path): assert all(after[k] == v for k, v in before.items() if k != "docProps/core.xml") -@pytest.mark.parametrize("extension", ["png", "gif", "bmp", "tiff", "webp"]) +@pytest.mark.parametrize("extension", ["gif", "bmp"]) def test_image_capabilities_are_honest(tmp_path, extension): from PIL import Image from core.handlers.image_handler import ImageHandler diff --git a/python/tests/test_jpeg_structure.py b/python/tests/test_jpeg_structure.py new file mode 100644 index 0000000..8e8083a --- /dev/null +++ b/python/tests/test_jpeg_structure.py @@ -0,0 +1,75 @@ +"""JFIF, COM, MPF and ICC fields (core/icc.py and the JPEG structure fields).""" +import struct + +from core import icc +from core.registry import HandlerRegistry +from tests.factories import by_name, image + + +def _keys(path): + record = HandlerRegistry.get(path).read(path) + assert record.read_errors == [] + return record, {f.key: f for f in record.fields} + + +def _build(tmp_path, name): + folder = tmp_path / name + folder.mkdir() + return by_name(name).build(folder) + + +def test_icc_profile_is_assembled_from_segments_and_described(tmp_path): + record, keys = _keys(_build(tmp_path, "jpeg_icc_comment")) + assert keys["icc:Version"].value == "2.1.0" + assert keys["icc:ProfileClass"].value == "Display device" + assert keys["icc:ColorSpace"].value == "RGB" and keys["icc:ConnectionSpace"].value == "XYZ" + assert keys["icc:RenderingIntent"].value == "Relative colorimetric" + assert keys["icc:Platform"].value == "Microsoft" + assert keys["icc:CreationDate"].value == "2026-01-02T03:04:05+00:00" + assert keys["icc:Description"].value == "MetaLens test RGB" + assert keys["icc:Copyright"].value == "Public domain" + assert not any(f.editable or f.deletable for k, f in keys.items() if k.startswith("icc:")) + + +def test_comments_are_listed(tmp_path): + _, keys = _keys(_build(tmp_path, "jpeg_icc_comment")) + assert keys["jpeg:Comment"].value == ["First comment é", "Second"] and keys["jpeg:Comment"].multi + + +def test_jfif_header(tmp_path): + _, keys = _keys(_build(tmp_path, "jpeg_jfif_exif")) + assert keys["jfif:Version"].value.startswith("1.") + assert keys["jfif:ResolutionUnit"].value in ("None (aspect ratio)", "Inches", "Centimeters") + assert isinstance(keys["jfif:XResolution"].value, int) + + +def test_mpf_index(tmp_path): + _, keys = _keys(_build(tmp_path, "jpeg_mpo")) + assert keys["mpf:NumberOfImages"].value == 2 + assert keys["mpf:image1:Type"].value == "Baseline MP primary image" + assert keys["mpf:image1:Size"].computed + + +def test_invalid_profiles_give_no_fields(): + assert icc.fields(b"") == [] + assert icc.fields(bytes(200)) == [] + assert icc.fields(image.icc_profile()[:100]) == [] + assert icc.assemble([b"ICC_PROFILE" + bytes(1) + bytes([1, 2]) + b"x"]) is None # second part missing + + +def test_truncated_tag_table_never_raises(): + profile = bytearray(image.icc_profile()) + struct.pack_into(">I", profile, 128, 10_000) # claims far more tags than exist + assert {f.key for f in icc.fields(bytes(profile))} >= {"icc:Version"} + for cut in range(128, len(profile), 7): + icc.fields(bytes(profile[:cut])) + + +def test_icc_in_png_tiff_and_webp(tmp_path): + from PIL import Image + profile = image.icc_profile() + for name in ("p.png", "p.tif", "p.webp"): + path = tmp_path / name + Image.new("RGB", (4, 4)).save(path, icc_profile=profile) + _, keys = _keys(path) + assert keys["icc:Description"].value == "MetaLens test RGB", name diff --git a/python/tests/test_makernote.py b/python/tests/test_makernote.py new file mode 100644 index 0000000..71e3ab8 --- /dev/null +++ b/python/tests/test_makernote.py @@ -0,0 +1,49 @@ +"""Known fields of vendor MakerNotes are shown read only.""" +import struct + +from core import makernote, sensitivity + + +def _canon(offset=0): + text = b"Ada Lovelace\x00" + entry = struct.pack("H", 1) + struct.pack(">HHLL", 0x001D, 2, len(text), 8 + 2 + 12 + 4) + bytes(4) + return b"Nikon\x00\x02\x10\x00\x00" + b"MM\x00*" + struct.pack(">L", 8) + ifd + text + + +def test_canon_owner_name_with_absolute_offsets(): + found = makernote.read("Canon", _canon(500), "<", 500) + assert [(v.name, v.value) for v in found] == [("OwnerName", "Ada Lovelace")] + + +def test_nikon_serial_number_with_relative_offsets(): + found = makernote.read("NIKON", _nikon(), ">", 300) + assert [(v.name, v.value) for v in found] == [("SerialNumber", "6001234")] + + +def test_unknown_or_damaged_notes_give_nothing(): + assert makernote.read("Acme", b"VENDOR\x00\x01\x02\x03" * 4, "<", 100) == [] + assert makernote.read("Canon", b"\xff" * 40, "<", 100) == [] + assert makernote.read("Canon", _canon(500)[:10], "<", 500) == [] + assert makernote.read("Canon", _canon(0), "<", None) == [] + + +def test_the_values_are_classified(): + assert sensitivity.classify("maker:canon:OwnerName") == "identity" + assert sensitivity.classify("maker:nikon:SerialNumber") == "device" + + +def test_a_real_jpeg_shows_the_note_and_it_goes_with_the_makernote(tmp_path): + from core.registry import HandlerRegistry + from tests.factories import image + path = tmp_path / "canon.jpg" + exif = image.vendor_camera_exif("canon") + path.write_bytes(image.jpeg_with([image.segment(0xE1, b"Exif\x00\x00" + exif[6:])])) + keys = {f.key: f for f in HandlerRegistry.get(path).read(path).fields} + assert keys["maker:canon:ImageType"].value == "Canon EOS TEST BODY" + assert keys["maker:canon:ImageType"].computed and not keys["maker:canon:ImageType"].deletable diff --git a/python/tests/test_media_integrity.py b/python/tests/test_media_integrity.py new file mode 100644 index 0000000..3ec643b --- /dev/null +++ b/python/tests/test_media_integrity.py @@ -0,0 +1,71 @@ +"""G5: a metadata write never changes the media data of the file.""" +import itertools + +import pytest + +from core.handlers.image_handler import ImageHandler +from core.registry import HandlerRegistry +from tests.factories import by_name + + +def _build(tmp_path, name): + folder = tmp_path / name + folder.mkdir() + return by_name(name).build(folder) + + +def test_jpeg_digest_ignores_metadata_but_not_the_picture(tmp_path): + handler = HandlerRegistry.get(_build(tmp_path, "jpeg_camera")) + first = _build(tmp_path, "jpeg_iptc") + second = _build(tmp_path, "jpeg_xmp_first") + assert handler.content_digest(first) == handler.content_digest(second) # same picture, other metadata + damaged = tmp_path / "damaged.jpg" + data = bytearray(first.read_bytes()) + data[-10] ^= 0xFF + damaged.write_bytes(bytes(data)) + assert handler.content_digest(damaged) != handler.content_digest(first) + + +def test_png_and_webp_digests_ignore_text_and_metadata(tmp_path): + handler = ImageHandler() + png = _build(tmp_path, "png_text_exif") + data = png.read_bytes() + other = tmp_path / "other.png" + cut = data.index(b"tEXt") - 4 + other.write_bytes(data[:cut] + data[cut + 12 + int.from_bytes(data[cut:cut + 4], "big"):]) + assert handler.content_digest(png) == handler.content_digest(other) + assert handler.content_digest(_build(tmp_path, "webp_exif")) + + +def test_unknown_formats_are_not_checked(tmp_path): + assert ImageHandler().content_digest(_build(tmp_path, "bmp")) is None + assert ImageHandler().content_digest(_build(tmp_path, "gif_comment")) is None + + +def test_tiff_digest_follows_the_strips_not_the_tables(tmp_path): + assert ImageHandler().content_digest(_build(tmp_path, "tiff_dpi")) + + +@pytest.mark.asyncio +async def test_a_write_that_changes_the_media_is_refused_and_leaves_the_file(client, tmp_path, monkeypatch): + path = _build(tmp_path, "jpeg_camera") + original = path.read_bytes() + counter = itertools.count() + monkeypatch.setattr(ImageHandler, "content_digest", lambda self, p: f"digest-{next(counter) % 2}") + resp = await client.post("/write", json={"path": str(path), "fields": [{"key": "exif:Artist", "value": "Someone"}]}) + assert resp.status_code == 422, resp.text + assert resp.json()["errors"][0]["code"] == "lossy_write" + assert "media data" in resp.json()["detail"] + assert path.read_bytes() == original + assert [p.name for p in tmp_path.iterdir() if p.name.startswith(".metalens-")] == [] + + +@pytest.mark.asyncio +async def test_a_digest_that_cannot_be_computed_does_not_block_the_save(client, tmp_path, monkeypatch): + path = _build(tmp_path, "jpeg_camera") + + def broken(self, p): + raise OSError("cannot read") + monkeypatch.setattr(ImageHandler, "content_digest", broken) + resp = await client.post("/write", json={"path": str(path), "fields": [{"key": "exif:Artist", "value": "Someone"}]}) + assert resp.status_code == 200, resp.text diff --git a/python/tests/test_mkv.py b/python/tests/test_mkv.py new file mode 100644 index 0000000..da40dc9 --- /dev/null +++ b/python/tests/test_mkv.py @@ -0,0 +1,116 @@ +"""Matroska / WebM: fields, removal, in-place edit (core/ebml.py, handlers/mkv_fields.py).""" +import pytest + +from core import ebml, sensitivity +from core.registry import HandlerRegistry +from tests.factories import by_name, mkv + + +def _build(tmp_path, name="mkv_full"): + folder = tmp_path / name + folder.mkdir() + return by_name(name).build(folder) + + +def _keys(path): + return {f.key: f for f in HandlerRegistry.get(path).read(path).fields} + + +def _media(path): + return ebml.media_digest(path) + + +async def _write(client, path, fields=(), delete=()): + return await client.post("/write", json={"path": str(path), "fields": list(fields), "delete": list(delete)}) + + +def test_fields_of_a_muxed_file(tmp_path): + path = _build(tmp_path) + record = HandlerRegistry.get(path).read(path) + keys = {f.key: f for f in record.fields} + assert record.read_errors == [] and record.handler_name == "video" + assert keys["mkv:DocType"].value == "matroska" + assert keys["mkv:Info:Title"].value == "Vacanze \u00e8 2026" and keys["mkv:Info:Title"].editable + assert keys["mkv:Info:WritingApp"].value.startswith("mkvmerge") + assert keys["mkv:Info:Duration"].value == 2.5 and keys["mkv:Info:Duration"].computed + assert keys["mkv:tag:ARTIST"].value == "Ada Lovelace" and keys["mkv:tag:ARTIST"].deletable + assert keys["mkv:attachment:cover.jpg"].binary["mime"] == "image/jpeg" + assert keys["mkv:track1:CodecID"].value == "V_MPEG4/ISO/AVC" and not keys["mkv:track1:CodecID"].deletable + + +def test_the_fields_are_classified(tmp_path): + for key, category in (("mkv:Info:WritingApp", "software"), ("mkv:Info:DateUTC", "datetime"), + ("mkv:tag:ARTIST", "identity"), ("mkv:attachment:cover.jpg", "thumbnail"), + ("mkv:tag:TITLE", "text")): + assert sensitivity.classify(key) == category + + +@pytest.mark.asyncio +async def test_everything_private_is_removed_in_place_and_undo_is_exact(client, tmp_path): + path = _build(tmp_path) + original, media = path.read_bytes(), _media(path) + body = (await client.post("/sanitize", json={"path": str(path), "preview": False, + "categories": ["identity", "software", "datetime", "thumbnail", "text"]})).json() + assert body["ok"] and body["undo_token"], body + keys = _keys(path) + for gone in ("mkv:tag:ARTIST", "mkv:Info:WritingApp", "mkv:Info:MuxingApp", "mkv:Info:DateUTC", "mkv:Info:Title", + "mkv:attachment:cover.jpg"): + assert gone not in keys, gone + assert "mkv:track1:CodecID" in keys + data = path.read_bytes() + assert len(data) == len(original) and _media(path) == media + for secret in (b"Ada Lovelace", b"mkvmerge", b"libebml", b"cover.jpg"): + assert secret not in data, secret + assert (await client.post("/undo", json={"token": body["undo_token"]})).status_code == 200 + assert path.read_bytes() == original + + +@pytest.mark.asyncio +async def test_a_text_is_written_over_the_old_one_when_it_fits(client, tmp_path): + path = _build(tmp_path) + original = path.read_bytes() + resp = await _write(client, path, [{"key": "mkv:tag:ARTIST", "value": "Al"}, {"key": "mkv:Info:Title", "value": "Mare"}]) + assert resp.status_code == 200, resp.text + keys = _keys(path) + assert keys["mkv:tag:ARTIST"].value == "Al" and keys["mkv:Info:Title"].value == "Mare" + assert len(path.read_bytes()) == len(original) and ebml.Segment(path.read_bytes()) + assert (await client.post("/undo", json={"token": resp.json()["undo_token"]})).status_code == 200 + assert path.read_bytes() == original + + +@pytest.mark.asyncio +async def test_a_text_that_does_not_fit_is_refused(client, tmp_path): + path = _build(tmp_path) + before = path.read_bytes() + resp = await _write(client, path, [{"key": "mkv:tag:ARTIST", "value": "x" * 200}]) + assert resp.status_code == 422 and path.read_bytes() == before + + +def test_a_recording_without_a_final_size_is_read_only(tmp_path): + data = bytearray(mkv.mkv_full_bytes()) + at = data.index(b"\x18\x53\x80\x67") + 4 + size_len = 1 + while not data[at] & (0x80 >> (size_len - 1)): + size_len += 1 + data[at] = 0xFF >> (size_len - 1) # "unknown size": every value bit set + data[at + 1:at + size_len] = b"\xff" * (size_len - 1) + path = tmp_path / "live.mkv" + path.write_bytes(bytes(data)) + record = HandlerRegistry.get(path).read(path) + assert record.read_errors == [] and not record.format_supports_write + assert not any(f.editable or f.deletable for f in record.fields) + + +def test_damaged_files_never_raise(tmp_path): + data = mkv.mkv_full_bytes() + for cut in range(0, len(data), 23): + path = tmp_path / "cut.mkv" + path.write_bytes(data[:cut]) + HandlerRegistry.get(path).read(path) + + +def test_matroska_without_extension_is_recognized(tmp_path): + path = tmp_path / "movie" + path.write_bytes(mkv.mkv_full_bytes()) + assert HandlerRegistry.get(path).NAME == "video" + assert "mkv:Info:Title" in _keys(path) diff --git a/python/tests/test_mp4_sanitize.py b/python/tests/test_mp4_sanitize.py new file mode 100644 index 0000000..be86bc7 --- /dev/null +++ b/python/tests/test_mp4_sanitize.py @@ -0,0 +1,99 @@ +"""Removing QuickTime structure fields and XMP from MP4-family files, in place.""" +import pytest + +from core.handlers import qt_patch +from core.handlers.video_handler import VideoHandler +from core.registry import HandlerRegistry +from tests.factories import mp4 + + +@pytest.fixture +def phone(tmp_path): + return mp4.write_phone_movie(tmp_path / "phone.mov") + + +def _keys(path): + return {f.key: f for f in HandlerRegistry.get(path).read(path).fields} + + +def _digest(path): + return VideoHandler().content_digest(path) + + +async def _sanitize(client, path, categories): + return (await client.post("/sanitize", json={"path": str(path), "categories": categories, "preview": False})).json() + + +def test_removable_fields_are_marked(phone): + keys = _keys(phone) + for key in ("qt:CreateDate", "qt:track1:CreateDate", "qt:udta:©xyz", + "qt:mdta:com.apple.quicktime.location.ISO6709", "xmp:xmp:CreatorTool"): + assert keys[key].deletable, key + assert not keys["qt:track1:Width"].deletable and not keys["qt:MajorBrand"].deletable + assert keys["xmp:xmp:CreatorTool"].editable + + +@pytest.mark.asyncio +async def test_location_is_removed_in_place_and_undo_gives_the_exact_bytes_back(client, phone): + original, media = phone.read_bytes(), _digest(phone) + body = await _sanitize(client, phone, ["location"]) + assert body["ok"] and body["undo_token"] and body["remaining"] == [] + keys = _keys(phone) + assert "qt:udta:©xyz" not in keys and "qt:mdta:com.apple.quicktime.location.ISO6709" not in keys + assert "qt:mdta:com.apple.quicktime.model" in keys # other categories untouched + after = phone.read_bytes() + assert len(after) == len(original) and _digest(phone) == media # nothing moved, media untouched + assert b"+45.4642" not in after + assert (await client.post("/undo", json={"token": body["undo_token"]})).status_code == 200 + assert phone.read_bytes() == original + + +@pytest.mark.asyncio +async def test_removed_values_do_not_remain_in_the_file(client, phone): + await _sanitize(client, phone, ["location", "device"]) + data = phone.read_bytes() + for secret in (b"+45.4642+009.1900", b"iPhone 15 Pro", b"Apple", b"com.apple.quicktime.location"): + assert secret not in data, secret + + +@pytest.mark.asyncio +async def test_dates_are_cleared(client, phone): + body = await _sanitize(client, phone, ["datetime"]) + keys = _keys(phone) + assert body["ok"] and "qt:CreateDate" not in keys and "qt:ModifyDate" not in keys + assert "qt:track1:CreateDate" not in keys and "qt:track2:CreateDate" not in keys + + +@pytest.mark.asyncio +async def test_xmp_is_reduced_in_place(client, phone): + original = phone.read_bytes() + body = await _sanitize(client, phone, ["identity", "software", "text", "history"]) + keys = _keys(phone) + assert body["ok"] and "xmp:dc:creator" not in keys and "xmp:xmp:CreatorTool" not in keys + assert len(phone.read_bytes()) == len(original) + assert (await client.post("/undo", json={"token": body["undo_token"]})).status_code == 200 + assert phone.read_bytes() == original + + +@pytest.mark.asyncio +async def test_tags_and_structure_can_be_removed_together(client, phone): + mp4.add_full_tags(phone) + media = _digest(phone) + original = phone.read_bytes() + resp = await client.post("/write", json={"path": str(phone), "fields": [], "delete": [ + "mp4:©nam", "qt:udta:©xyz", "xmp:xmp:Rating"]}) + assert resp.status_code == 200, resp.text + keys = _keys(phone) + assert "mp4:©nam" not in keys and "qt:udta:©xyz" not in keys and "xmp:xmp:Rating" not in keys + assert _digest(phone) == media + assert (await client.post("/undo", json={"token": resp.json()["undo_token"]})).status_code == 200 + assert "mp4:©nam" in _keys(phone) + assert phone.read_bytes() == original + + +def test_plan_ignores_files_it_cannot_patch(tmp_path): + path = tmp_path / "x.mov" + path.write_bytes(b"") + assert qt_patch.plan(path) == {} + path.write_bytes(mp4.atom(b"ftyp", b"qt " + bytes(8))) + assert qt_patch.plan(path) == {} diff --git a/python/tests/test_mp4_write.py b/python/tests/test_mp4_write.py new file mode 100644 index 0000000..199df51 --- /dev/null +++ b/python/tests/test_mp4_write.py @@ -0,0 +1,153 @@ +"""Editing and adding QuickTime fields, text entries and XMP in MP4-family files.""" +import pytest + +from core.handlers.video_handler import VideoHandler +from core.registry import HandlerRegistry +from tests.factories import mp4 + + +@pytest.fixture +def phone(tmp_path): + return mp4.write_phone_movie(tmp_path / "phone.mov") + + +def _keys(path): + return {f.key: f for f in HandlerRegistry.get(path).read(path).fields} + + +def _media(path): + return VideoHandler().content_digest(path) + + +async def _write(client, path, fields=(), delete=()): + return await client.post("/write", json={"path": str(path), "fields": list(fields), "delete": list(delete)}) + + +async def _undo(client, resp): + return await client.post("/undo", json={"token": resp.json()["undo_token"]}) + + +def test_which_fields_can_be_edited(phone): + keys = _keys(phone) + for key in ("qt:CreateDate", "qt:track1:CreateDate", "qt:udta:©xyz", "qt:mdta:com.apple.quicktime.make", + "xmp:xmp:CreatorTool"): + assert keys[key].editable, key + for key in ("qt:track1:Width", "qt:MajorBrand", "qt:mdta:com.apple.quicktime.live-photo.vitality-score"): + assert not keys[key].editable, key + + +@pytest.mark.asyncio +async def test_dates_are_written_in_place_and_undo_is_exact(client, phone): + original, media = phone.read_bytes(), _media(phone) + resp = await _write(client, phone, [{"key": "qt:CreateDate", "value": "2020-01-02T03:04:05+00:00"}, + {"key": "qt:track1:CreateDate", "value": "2020-01-02T03:04:05+00:00"}]) + assert resp.status_code == 200, resp.text + keys = _keys(phone) + assert keys["qt:CreateDate"].value == "2020-01-02T03:04:05+00:00" + assert keys["qt:track1:CreateDate"].value == "2020-01-02T03:04:05+00:00" + assert len(phone.read_bytes()) == len(original) and _media(phone) == media + assert (await _undo(client, resp)).status_code == 200 and phone.read_bytes() == original + + +@pytest.mark.asyncio +async def test_a_longer_location_grows_the_movie_atom_and_nothing_else_moves(client, phone): + original, media = phone.read_bytes(), _media(phone) + place = "+45.46427+009.18951+120.500/" + resp = await _write(client, phone, [{"key": "qt:udta:©xyz", "value": place}, + {"key": "qt:mdta:com.apple.quicktime.make", "value": "Apple Computer Inc."}]) + assert resp.status_code == 200, resp.text + keys = _keys(phone) + assert keys["qt:udta:©xyz"].value == place and keys["qt:mdta:com.apple.quicktime.make"].value == "Apple Computer Inc." + data = phone.read_bytes() + assert data.index(b"mdat") == original.index(b"mdat") and _media(phone) == media + assert keys["xmp:xmp:CreatorTool"].value == "Tool & Co" # the atom after moov moved intact + assert (await _undo(client, resp)).status_code == 200 + assert phone.read_bytes() == original + + +@pytest.mark.asyncio +async def test_new_entries_are_created(client, phone): + media = _media(phone) + handler = HandlerRegistry.get(phone) + offered = {f.key for f in handler.writable_fields(phone, handler.read(phone))} + assert {"qt:udta:©swr", "qt:udta:©mak"} <= offered # the movie has the other entries + resp = await _write(client, phone, [{"key": "qt:udta:©swr", "value": "MetaLens"}, + {"key": "qt:mdta:com.apple.quicktime.software", "value": "hello"}]) + assert resp.status_code in (200, 422), resp.text + if resp.status_code == 200: + keys = _keys(phone) + assert keys["qt:udta:©swr"].value == "MetaLens" and _media(phone) == media + + +@pytest.mark.asyncio +async def test_a_movie_without_udta_or_keys_gets_them(client, tmp_path): + path = mp4.write_mp4(tmp_path / "plain.mp4", brand=b"isom") + media = _media(path) + resp = await _write(client, path, [{"key": "qt:udta:©mak", "value": "ACME"}, + {"key": "qt:mdta:com.apple.quicktime.model", "value": "Model 1"}]) + assert resp.status_code == 200, resp.text + keys = _keys(path) + assert keys["qt:udta:©mak"].value == "ACME" and keys["qt:mdta:com.apple.quicktime.model"].value == "Model 1" + assert _media(path) == media and HandlerRegistry.get(path).read(path).read_errors == [] + + +@pytest.mark.asyncio +async def test_xmp_edit_that_does_not_fit_is_stored_in_a_new_atom(client, phone): + original, media = phone.read_bytes(), _media(phone) + big = ["word" + str(i) for i in range(500)] + resp = await _write(client, phone, [{"key": "xmp:dc:subject", "value": big}]) + assert resp.status_code == 200, resp.text + assert _keys(phone)["xmp:dc:subject"].value == big + assert len(phone.read_bytes()) > len(original) and _media(phone) == media + assert (await _undo(client, resp)).status_code == 200 and phone.read_bytes() == original + + +@pytest.mark.asyncio +async def test_xmp_is_created_when_the_file_has_none(client, tmp_path): + path = mp4.write_mp4(tmp_path / "plain.mp4", brand=b"isom") + media = _media(path) + handler = HandlerRegistry.get(path) + assert "xmp:dc:title" in {f.key for f in handler.writable_fields(path, handler.read(path))} + resp = await _write(client, path, [{"key": "xmp:dc:title", "value": "Titolo"}]) + assert resp.status_code == 200, resp.text + assert _keys(path)["xmp:dc:title"].value == "Titolo" and _media(path) == media + + +@pytest.mark.asyncio +async def test_bad_values_are_refused(client, phone): + before = phone.read_bytes() + for key, value in (("qt:udta:©xyz", "not a location"), ("qt:CreateDate", "1850-01-01T00:00:00+00:00"), + ("qt:mdta:com.apple.quicktime.make", "a\u0000b")): + resp = await _write(client, phone, [{"key": key, "value": value}]) + assert resp.status_code == 422, (key, resp.text) + assert phone.read_bytes() == before + + +@pytest.mark.asyncio +async def test_a_fast_start_movie_grows_by_moving_the_media_and_fixing_the_chunk_offsets(client, tmp_path): + """moov before mdat: the media moves up and every chunk offset follows.""" + path = tmp_path / "faststart.mp4" + created = mp4.mac_seconds(2026, 5, 1) + ftyp = mp4.atom(b"ftyp", b"isom" + bytes(8)) + payload = bytes(range(256)) * 4 + + def build(offset): + moov = mp4.atom(b"moov", mp4.mvhd() + mp4._video_trak(1, created, offset, 0)) + return moov + first = build(0) + offset = len(ftyp) + len(first) + 8 + path.write_bytes(ftyp + build(offset) + mp4.atom(b"mdat", payload)) + media = _media(path) + resp = await _write(client, path, [{"key": "qt:udta:©mak", "value": "ACME corporation"}]) + assert resp.status_code == 200, resp.text + assert _keys(path)["qt:udta:©mak"].value == "ACME corporation" and _media(path) == media + data = path.read_bytes() + assert data[data.index(b"mdat") + 4:] == payload # the media is intact + from core.isobmff import Reader + rd = Reader(data) + moov = next(b for b in rd.children() if b.type == b"moov") + stco = rd.find(moov, b"trak", b"mdia", b"minf", b"stbl", b"stco") + chunk = int.from_bytes(data[stco.data_start + 8:stco.data_start + 12], "big") + assert chunk == data.index(b"mdat") + 4 # still points at the media + assert (await _undo(client, resp)).status_code == 200 + assert _media(path) == media diff --git a/python/tests/test_odf_epub.py b/python/tests/test_odf_epub.py new file mode 100644 index 0000000..3d9071e --- /dev/null +++ b/python/tests/test_odf_epub.py @@ -0,0 +1,157 @@ +"""OpenDocument and EPUB reading.""" +import zipfile + +import pytest + +from core import sniff +from core.registry import HandlerRegistry +from tests.factories import by_name, packages + + +def _build(tmp_path, name): + folder = tmp_path / name + folder.mkdir() + return by_name(name).build(folder) + + +def _read(path): + record = HandlerRegistry.get(path).read(path) + return record, {f.key: f for f in record.fields} + + +def test_odf_properties(tmp_path): + record, keys = _read(_build(tmp_path, "odt_meta")) + assert record.read_errors == [] and record.handler_name == "odf" + assert keys["odf:title"].value == "Relazione è" + assert keys["odf:initial-creator"].value == "Ada" and keys["odf:creator"].value == "Bruno" + assert keys["odf:keyword"].value == ["alpha", "beta"] + assert keys["odf:creation-date"].field_type == "localdatetime" and keys["odf:date"].field_type == "datetime" + assert keys["odf:editing-duration"].value == 3723 + assert keys["odf:editing-cycles"].value == 3 + assert keys["odf:stat.page-count"].value == 4 + assert keys["odf:custom.Project"].value == "Mercury" + assert record.format_supports_write and keys["odf:title"].editable and keys["odf:creator"].deletable + assert not keys["odf:stat.page-count"].editable and keys["odf:stat.page-count"].deletable + + +def test_epub_properties(tmp_path): + record, keys = _read(_build(tmp_path, "epub_book")) + assert record.read_errors == [] and record.handler_name == "epub" + assert keys["epub:version"].value == "3.0" + assert keys["epub:title"].value == ["Il libro è"] + assert keys["epub:creator"].value == ["Ada Lovelace", "Charles Babbage"] + assert keys["epub:modified"].value == "2026-03-05T06:07:08+00:00" + + +def test_packages_without_extension_are_found_by_content(tmp_path): + odf = packages.write_odf(tmp_path / "doc") + epub = packages.write_epub(tmp_path / "book") + assert sniff.detect(odf) == "odf" and sniff.detect(epub) == "epub" + assert HandlerRegistry.get(odf).NAME == "odf" and HandlerRegistry.get(epub).NAME == "epub" + + +def test_signed_document_is_flagged(tmp_path): + path = packages.write_odf(tmp_path / "s.odt") + with zipfile.ZipFile(path, "a") as z: + z.writestr("META-INF/documentsignatures.xml", "") + assert any("signed" in w for w in _read(path)[0].warnings) + + +def test_hostile_or_damaged_packages_are_read_errors_not_crashes(tmp_path): + bad = tmp_path / "bad.odt" + bad.write_bytes(b"not a zip") + assert _read(bad)[0].read_errors + dtd = tmp_path / "dtd.odt" + with zipfile.ZipFile(dtd, "w") as z: + z.writestr("meta.xml", ']>&a;') + assert _read(dtd)[0].read_errors + empty = tmp_path / "e.epub" + with zipfile.ZipFile(empty, "w") as z: + z.writestr("mimetype", "application/epub+zip") + assert _read(empty)[0].read_errors + traversal = tmp_path / "t.epub" + with zipfile.ZipFile(traversal, "w") as z: + z.writestr("META-INF/container.xml", + packages.EPUB_CONTAINER.replace("OEBPS/content.opf", "../outside.opf")) + assert _read(traversal)[0].read_errors + + +def test_pdf_version_and_linearization(tmp_path): + from tests.factories import by_name as sample_by_name + folder = tmp_path / "pdf" + folder.mkdir() + path = sample_by_name("pdf_basic").build(folder) + _, keys = _read(path) + assert keys["pdf:PDFVersion"].value.startswith("1.") and keys["pdf:PDFVersion"].computed + assert keys["pdf:Linearized"].value is False + + +async def _write(client, path, fields=(), delete=()): + return await client.post("/write", json={"path": str(path), "fields": list(fields), "delete": list(delete)}) + + +@pytest.mark.asyncio +async def test_odf_edit_add_and_remove_keep_the_rest_of_the_package(client, tmp_path): + path = _build(tmp_path, "odt_meta") + before = {i.filename: zipfile.ZipFile(path).read(i.filename) for i in zipfile.ZipFile(path).infolist()} + resp = await _write(client, path, [{"key": "odf:title", "value": "Nuovo titolo"}, + {"key": "odf:keyword", "value": ["uno", "due", "tre"]}, + {"key": "odf:custom.Project", "value": "Venus"}], + delete=["odf:creator", "odf:stat.word-count"]) + assert resp.status_code == 200, resp.text + keys = _read(path)[1] + assert keys["odf:title"].value == "Nuovo titolo" and keys["odf:keyword"].value == ["uno", "due", "tre"] + assert keys["odf:custom.Project"].value == "Venus" and "odf:creator" not in keys and "odf:stat.word-count" not in keys + after = {i.filename: zipfile.ZipFile(path).read(i.filename) for i in zipfile.ZipFile(path).infolist()} + assert list(after) == list(before) and all(after[n] == before[n] for n in before if n != "meta.xml") + assert zipfile.ZipFile(path).infolist()[0].filename == "mimetype" + assert (await client.post("/undo", json={"token": resp.json()["undo_token"]})).status_code == 200 + restored = _read(path)[1] + assert restored["odf:creator"].value == "Bruno" and restored["odf:keyword"].value == ["alpha", "beta"] + assert restored["odf:stat.word-count"].value == 1200 + + +@pytest.mark.asyncio +async def test_odf_can_receive_a_new_text(client, tmp_path): + path = _build(tmp_path, "odt_meta") + await _write(client, path, delete=["odf:subject"]) + handler = HandlerRegistry.get(path) + assert "odf:subject" in {f.key for f in handler.writable_fields(path, handler.read(path))} + resp = await _write(client, path, [{"key": "odf:subject", "value": "Bilancio"}]) + assert resp.status_code == 200, resp.text + assert _read(path)[1]["odf:subject"].value == "Bilancio" + + +@pytest.mark.asyncio +async def test_epub_required_fields_stay_and_the_rest_can_go(client, tmp_path): + path = _build(tmp_path, "epub_book") + keys = _read(path)[1] + assert not keys["epub:identifier"].deletable and not keys["epub:title"].deletable and keys["epub:title"].editable + assert keys["epub:publisher"].deletable and keys["epub:creator"].deletable + resp = await _write(client, path, [{"key": "epub:title", "value": ["Altro titolo"]}], + delete=["epub:publisher", "epub:creator", "epub:modified"]) + assert resp.status_code == 200, resp.text + left = _read(path)[1] + assert left["epub:title"].value == ["Altro titolo"] + assert "epub:publisher" not in left and "epub:creator" not in left and "epub:modified" not in left + assert (await _write(client, path, delete=["epub:identifier"])).status_code == 422 + assert zipfile.ZipFile(path).infolist()[0].filename == "mimetype" + + +@pytest.mark.asyncio +async def test_sharing_removes_the_author_from_odf_and_epub(client, tmp_path): + for name, gone in (("odt_meta", "odf:initial-creator"), ("epub_book", "epub:creator")): + path = _build(tmp_path, name) + body = (await client.post("/sanitize", json={"path": str(path), "categories": ["identity"], + "preview": False})).json() + assert body["ok"], body + assert gone not in _read(path)[1], name + + +@pytest.mark.asyncio +async def test_a_signed_odf_is_not_changed(client, tmp_path): + path = packages.write_odf(tmp_path / "s.odt") + with zipfile.ZipFile(path, "a") as z: + z.writestr("META-INF/documentsignatures.xml", "") + record = _read(path)[0] + assert not record.format_supports_write and not any(f.deletable or f.editable for f in record.fields) diff --git a/python/tests/test_office_handler.py b/python/tests/test_office_handler.py index 2b7e810..7e03cbb 100644 --- a/python/tests/test_office_handler.py +++ b/python/tests/test_office_handler.py @@ -115,7 +115,7 @@ def test_offsetless_date_in_file_is_read_only(tmp_path): # ──────────────────────── app.xml / custom.xml (D16) ───────────────────────── -def test_app_and_custom_properties_read_only(tmp_path): +def test_app_and_custom_properties_are_read_only_values_that_can_be_removed(tmp_path): path = factory.docx_app_custom(tmp_path) f = _fields(path) expected = { @@ -129,21 +129,21 @@ def test_app_and_custom_properties_read_only(tmp_path): } for key, (ftype, value) in expected.items(): assert (f[key].field_type, f[key].value) == (ftype, value), key - assert not f[key].editable and not f[key].deletable, key + assert not f[key].editable and f[key].deletable, key # never edited, but removable assert f["office:app.TitlesOfParts"].multi assert "office:app.HeadingPairs" not in f @pytest.mark.asyncio -async def test_api_rejects_app_custom_mutation(client, tmp_path): +async def test_api_rejects_app_custom_edits_and_unknown_removals(client, tmp_path): path = factory.docx_app_custom(tmp_path) before = path.read_bytes() for key in ("office:app.Company", "office:custom.Project"): resp = await client.post("/write", json={"path": str(path), "fields": [{"key": key, "value": "x"}]}) - assert resp.status_code == 422 - resp = await client.post("/delete", json={"path": str(path), "keys": [key]}) - assert resp.status_code == 422 + assert resp.status_code == 422 # values are never edited assert path.read_bytes() == before + resp = await client.post("/delete", json={"path": str(path), "keys": ["office:app.NoSuchProperty"]}) + assert resp.status_code == 422 and path.read_bytes() == before def test_core_edit_keeps_app_and_custom_parts(tmp_path): diff --git a/python/tests/test_office_sanitize.py b/python/tests/test_office_sanitize.py new file mode 100644 index 0000000..cdd7bf3 --- /dev/null +++ b/python/tests/test_office_sanitize.py @@ -0,0 +1,62 @@ +"""Removing application and custom properties from Office packages.""" +import zipfile + +import pytest + +from core.registry import HandlerRegistry +from tests.factories import by_name + + +def _build(tmp_path, name): + folder = tmp_path / name + folder.mkdir() + return by_name(name).build(folder) + + +def _keys(path): + return {f.key: f for f in HandlerRegistry.get(path).read(path).fields} + + +def test_application_and_custom_properties_are_removable(tmp_path): + keys = _keys(_build(tmp_path, "docx_app_custom")) + removable = [k for k, f in keys.items() if k.startswith(("office:app.", "office:custom.")) and f.deletable] + assert removable and any(k.startswith("office:custom.") for k in removable) + assert any(k.startswith("office:app.") for k in removable) + + +@pytest.mark.asyncio +async def test_company_and_custom_data_are_removed_and_undo_is_exact(client, tmp_path): + path = _build(tmp_path, "docx_app_custom") + original = path.read_bytes() + keys = _keys(path) + sensitive = [k for k in keys if k.startswith("office:custom.") or k in ("office:app.Company", "office:app.Manager")] + assert sensitive + resp = await client.post("/write", json={"path": str(path), "fields": [], "delete": sensitive}) + assert resp.status_code == 200, resp.text + left = _keys(path) + assert not [k for k in sensitive if k in left] + assert zipfile.ZipFile(path).testzip() is None + undo = await client.post("/undo", json={"token": resp.json()["undo_token"]}) + assert undo.status_code == 200 + assert path.read_bytes() == original + + +@pytest.mark.asyncio +async def test_sharing_a_document_removes_the_author_and_the_company(client, tmp_path): + path = _build(tmp_path, "docx_app_custom") + body = (await client.post("/sanitize", json={"path": str(path), "categories": ["identity"], + "preview": False})).json() + assert body["ok"], body + left = _keys(path) + assert "office:author" not in left and "office:last_modified_by" not in left + assert "office:app.Company" not in left + + +@pytest.mark.asyncio +async def test_a_signed_package_is_not_touched(client, tmp_path): + path = _build(tmp_path, "docx_signed_like") + keys = _keys(path) + assert not any(f.deletable for f in keys.values()) + body = (await client.post("/sanitize", json={"path": str(path), "categories": ["identity"], + "preview": False})).json() + assert body["will_remove"] == [] and body["undo_token"] is None diff --git a/python/tests/test_ole_handler.py b/python/tests/test_ole_handler.py index 813a597..b946895 100644 --- a/python/tests/test_ole_handler.py +++ b/python/tests/test_ole_handler.py @@ -82,7 +82,7 @@ def test_integers_are_typed(doc_cp1252): def test_thumbnail_is_a_binary_descriptor(doc_cp1252): _, f = _read(doc_cp1252) thumb = f["ole:thumbnail"] - assert thumb.field_type == "binary" and not thumb.editable and not thumb.deletable + assert thumb.field_type == "binary" and not thumb.editable and thumb.deletable assert thumb.value == {"size": len(factory.THUMBNAIL_CF), "sha256": hashlib.sha256(factory.THUMBNAIL_CF).hexdigest(), "mime": "image/bmp"} @@ -118,11 +118,11 @@ def test_user_defined_properties(doc_cp1252): def test_everything_is_read_only_unique_and_json(doc_cp1252): record, f = _read(doc_cp1252) - assert not record.supports_write and not record.supports_delete - assert not record.format_supports_write and not record.format_supports_delete + assert not record.supports_write and not record.format_supports_write # values are never edited + assert record.supports_delete and record.format_supports_delete # but they can be removed keys = [x.key for x in record.fields] assert len(keys) == len(set(keys)) - assert all(not x.editable and not x.deletable and x.source == "ole" for x in record.fields) + assert all(not x.editable and x.deletable and x.source == "ole" for x in record.fields) body = json.dumps(record.to_dict(), allow_nan=False, ensure_ascii=False) assert "�" not in body @@ -179,8 +179,8 @@ def test_not_an_ole_file(tmp_path): assert record.read_errors == ["Not a valid OLE file"] -def test_write_and_delete_are_not_supported(doc_cp1252): +def test_values_are_not_edited_but_can_be_removed(doc_cp1252): with pytest.raises(NotImplementedError): OleHandler().write(doc_cp1252, []) - with pytest.raises(NotImplementedError): - OleHandler().delete(doc_cp1252, ["ole:title"]) + OleHandler().delete(doc_cp1252, ["ole:title"]) + assert "ole:title" not in {f.key for f in OleHandler().read(doc_cp1252).fields} diff --git a/python/tests/test_ole_sanitize.py b/python/tests/test_ole_sanitize.py new file mode 100644 index 0000000..372f76c --- /dev/null +++ b/python/tests/test_ole_sanitize.py @@ -0,0 +1,64 @@ +"""Removing the properties of legacy Office documents (DOC, XLS, PPT) in place.""" +import pytest + +from core.registry import HandlerRegistry +from tests.factories import by_name + + +def _build(tmp_path, name): + folder = tmp_path / name + folder.mkdir() + return by_name(name).build(folder) + + +def _keys(path): + return {f.key: f for f in HandlerRegistry.get(path).read(path).fields} + + +def test_properties_are_removable_but_never_edited(tmp_path): + path = _build(tmp_path, "ole_doc_cp1252") + keys = _keys(path) + assert keys["ole:author"].deletable and not keys["ole:author"].editable + record = HandlerRegistry.get(path).read(path) + assert record.format_supports_delete and not record.format_supports_write + + +@pytest.mark.asyncio +@pytest.mark.parametrize("name, secret", [("ole_doc_cp1252", "José Müller".encode("cp1252")), + ("ole_ppt_utf16", "José Müller".encode("utf-16-le"))]) +async def test_the_author_and_the_company_are_removed_from_the_file_and_undo_is_exact(client, tmp_path, name, secret): + path = _build(tmp_path, name) + original = path.read_bytes() + keys = _keys(path) + if secret not in original: # the sample stores the name in another form + secret = keys["ole:author"].value.encode("utf-8")[:4] + body = (await client.post("/sanitize", json={"path": str(path), "preview": False, + "categories": ["identity", "software", "datetime", "text"]})).json() + assert body["ok"] and body["undo_token"], body + left = _keys(path) + assert "ole:author" not in left and "ole:last_author" not in left and "ole:company" not in left + data = path.read_bytes() + assert len(data) == len(original) + assert secret not in data # the name is gone from the bytes, not only from the listing + assert (await client.post("/undo", json={"token": body["undo_token"]})).status_code == 200 + assert path.read_bytes() == original + + +@pytest.mark.asyncio +async def test_other_properties_survive_a_removal(client, tmp_path): + path = _build(tmp_path, "ole_doc_cp1252") + before = _keys(path) + resp = await client.post("/delete", json={"path": str(path), "keys": ["ole:author"]}) + assert resp.status_code == 200, resp.text + after = _keys(path) + assert set(before) - set(after) == {"ole:author"} + assert all(after[k].value == before[k].value for k in after) + assert (await client.post("/delete", json={"path": str(path), "keys": ["ole:author"]})).status_code == 422 + + +@pytest.mark.asyncio +async def test_a_damaged_document_is_not_touched(client, tmp_path): + path = tmp_path / "bad.doc" + path.write_bytes(b"not a compound file") + record = HandlerRegistry.get(path).read(path) + assert record.read_errors and not record.format_supports_delete diff --git a/python/tests/test_operations.py b/python/tests/test_operations.py new file mode 100644 index 0000000..1c854ab --- /dev/null +++ b/python/tests/test_operations.py @@ -0,0 +1,122 @@ +"""Batch preparation, date shift, tag copy, export and rename (api/operations.py).""" +import csv +import io + +import pytest + +from core.registry import HandlerRegistry +from tests.factories import by_name + + +def _build(tmp_path, name, folder=None): + base = (folder or tmp_path) / name + base.mkdir(parents=True) + return by_name(name).build(base) + + +def _keys(path): + return {f.key: f for f in HandlerRegistry.get(path).read(path).fields} + + +@pytest.mark.asyncio +async def test_batch_preparation_reports_every_file_and_keeps_going(client, tmp_path): + good = [_build(tmp_path, "jpeg_camera"), _build(tmp_path, "pdf_basic")] + paths = [str(good[0]), str(tmp_path / "missing.jpg"), str(good[1])] + preview = (await client.post("/sanitize-many", json={"paths": paths, "categories": ["identity", "location"]})).json() + assert [r["ok"] for r in preview["results"]] == [True, False, True] and preview["undo_tokens"] == [] + failed = preview["results"][1] # a failed file carries structured errors, never exception text + assert "detail" not in failed and failed["errors"] and set(failed["errors"][0]) == {"code", "key", "message"} + body = (await client.post("/sanitize-many", json={"paths": paths, "categories": ["location", "identity"], + "preview": False})).json() + assert [r["ok"] for r in body["results"]] == [True, False, True] + assert len(body["undo_tokens"]) >= 1 and not any(k.startswith("gps:") for k in _keys(good[0])) + for token in body["undo_tokens"]: + assert (await client.post("/undo", json={"token": token})).status_code == 200 + assert any(k.startswith("gps:") for k in _keys(good[0])) + + +@pytest.mark.asyncio +async def test_the_batch_size_is_limited(client): + resp = await client.post("/sanitize-many", json={"paths": ["x"] * 501, "categories": ["location"]}) + assert resp.status_code == 422 + + +@pytest.mark.asyncio +async def test_dates_are_shifted_with_a_preview_and_an_undo(client, tmp_path): + path = _build(tmp_path, "jpeg_camera") + before = _keys(path)["exif:DateTimeOriginal"].value + preview = (await client.post("/shift-dates", json={"paths": [str(path)], "seconds": 3600})).json() + change = next(c for c in preview["results"][0]["changes"] if c["key"] == "exif:DateTimeOriginal") + assert change["from"] == before and change["to"] == "2026-01-02T04:04:05" + assert _keys(path)["exif:DateTimeOriginal"].value == before # a preview changes nothing + body = (await client.post("/shift-dates", json={"paths": [str(path)], "seconds": -86400, "preview": False})).json() + assert body["ok"] and body["undo_tokens"] + assert _keys(path)["exif:DateTimeOriginal"].value == "2026-01-01T03:04:05" + assert (await client.post("/undo", json={"token": body["undo_tokens"][0]})).status_code == 200 + assert _keys(path)["exif:DateTimeOriginal"].value == before + + +@pytest.mark.asyncio +async def test_shift_can_be_limited_to_some_keys_and_keeps_offsets(client, tmp_path): + path = _build(tmp_path, "jpeg_xmp_first") + body = (await client.post("/shift-dates", json={"paths": [str(path)], "seconds": 60, "preview": False, + "keys": ["xmp:xmp:CreateDate"]})).json() + assert body["ok"], body + keys = _keys(path) + assert keys["xmp:xmp:CreateDate"].value == "2026-01-02T03:05:05+02:00" + assert keys["exif:DateTimeOriginal"].value == "2026-01-02T03:04:05" + + +@pytest.mark.asyncio +async def test_tags_are_copied_to_other_files(client, tmp_path): + source = _build(tmp_path, "jpeg_camera") + target = _build(tmp_path, "jpeg_no_exif") + pdf = _build(tmp_path, "pdf_basic") + preview = (await client.post("/copy-tags", json={"source": str(source), "targets": [str(target), str(pdf)], + "keys": ["exif:Artist", "exif:Copyright"]})).json() + assert preview["results"][0]["will_copy"] == ["exif:Artist"] or "exif:Artist" in preview["results"][0]["will_copy"] + assert preview["results"][1]["will_copy"] == [] and preview["results"][1]["skipped"] + body = (await client.post("/copy-tags", json={"source": str(source), "targets": [str(target)], "preview": False, + "keys": ["exif:Artist"]})).json() + assert body["ok"] and _keys(target)["exif:Artist"].value == _keys(source)["exif:Artist"].value + + +@pytest.mark.asyncio +async def test_export_json_and_csv_with_protection_from_formulas(client, tmp_path): + folder = tmp_path / "pics" + folder.mkdir() + sample = _build(tmp_path, "jpeg_camera", folder=tmp_path / "src") + (folder / "=cmd.jpg").write_bytes(sample.read_bytes()) + (folder / "b.jpg").write_bytes(sample.read_bytes()) + data = (await client.get("/export", params={"path": str(folder), "format": "json"})).json() + assert [r["file"] for r in data] == ["=cmd.jpg", "b.jpg"] and "exif:Make" in data[0]["fields"] + resp = await client.get("/export", params={"path": str(folder), "format": "csv"}) + assert resp.headers["content-type"].startswith("text/csv") and "attachment" in resp.headers["content-disposition"] + rows = list(csv.reader(io.StringIO(resp.text.lstrip("")))) + assert rows[0][:2] == ["file", "handler"] and "exif:Make" in rows[0] + assert rows[1][0] == "'=cmd.jpg" # never starts with a formula character + assert (await client.get("/export", params={"path": str(folder), "format": "xml"})).status_code == 422 + + +@pytest.mark.asyncio +async def test_rename_from_metadata(client, tmp_path): + path = _build(tmp_path, "jpeg_camera") + preview = (await client.post("/rename", json={"path": str(path), + "template": "{exif:DateTimeOriginal|%Y%m%d_%H%M%S}_{name}"})).json() + assert preview["new"].endswith("20260102_030405_camera.jpg") and not preview["renamed"] and path.exists() + body = (await client.post("/rename", json={"path": str(path), "preview": False, + "template": "{exif:DateTimeOriginal|%Y-%m-%d} {exif:Make}"})).json() + assert body["renamed"] and not path.exists() and (path.parent / "2026-01-02 Canon.jpg").exists() + + +@pytest.mark.asyncio +async def test_rename_refuses_collisions_unknown_fields_and_unsafe_names(client, tmp_path): + path = _build(tmp_path, "jpeg_camera") + other = path.parent / "taken.jpg" + other.write_bytes(b"x") + assert (await client.post("/rename", json={"path": str(path), "preview": False, "template": "taken"})).status_code == 409 + assert (await client.post("/rename", json={"path": str(path), "template": "{gps:Nothing}"})).status_code == 422 + assert (await client.post("/rename", json={"path": str(path), "template": "{oops"})).status_code == 422 + safe = (await client.post("/rename", json={"path": str(path), "template": "a/b\\c:d"})).json() + assert path.parent == type(path)(safe["new"]).parent + assert (await client.post("/rename", json={"path": str(path), "template": "CON"})).status_code == 422 diff --git a/python/tests/test_path_security.py b/python/tests/test_path_security.py index 5ea6ad5..3f910d1 100644 --- a/python/tests/test_path_security.py +++ b/python/tests/test_path_security.py @@ -220,8 +220,9 @@ def writer(path): with pytest.raises(PathSecurityError, match="File not found"): secure_atomic_write(nonexistent, writer) - def test_atomic_write_custom_suffix(self, tmp_path): - """Custom temp suffix should be used.""" + def test_atomic_write_copy_keeps_the_name_of_the_target(self, tmp_path): + """The copy is called like the target, in a private directory (the name or + extension of a file decides how handlers treat it).""" test_file = tmp_path / "test.txt" test_file.write_text("original") @@ -234,7 +235,9 @@ def writer(path): secure_atomic_write(test_file, writer, temp_suffix=".custom_tmp") assert len(temp_path_list) > 0 - assert ".custom_tmp" in temp_path_list[0].name + assert temp_path_list[0].name == "test.txt" + assert temp_path_list[0].parent.name.startswith(".metalens-") + assert temp_path_list[0].parent.parent == tmp_path def test_atomic_write_binary_content(self, tmp_path): """Binary content should be written correctly.""" diff --git a/python/tests/test_pdf_xmp.py b/python/tests/test_pdf_xmp.py new file mode 100644 index 0000000..3f14f10 --- /dev/null +++ b/python/tests/test_pdf_xmp.py @@ -0,0 +1,53 @@ +"""The XMP metadata stream of a PDF: shown, removed as a whole, restored.""" +import pytest + +from core.registry import HandlerRegistry +from tests.factories import by_name + + +def _build(tmp_path, name): + folder = tmp_path / name + folder.mkdir() + return by_name(name).build(folder) + + +def _keys(path): + return {f.key: f for f in HandlerRegistry.get(path).read(path).fields} + + +def test_xmp_properties_are_shown_and_the_stream_can_be_removed(tmp_path): + keys = _keys(_build(tmp_path, "pdf_xmp")) + assert keys["xmp:dc:creator"].value == ["Ada"] and keys["xmp:xmp:CreatorTool"].value == "Writer 1.0" + assert not keys["xmp:dc:creator"].editable and not keys["xmp:dc:creator"].deletable + assert keys["pdf:@xmp"].deletable and keys["pdf:@xmp"].binary["mime"] == "application/rdf+xml" + + +@pytest.mark.asyncio +async def test_removing_the_stream_leaves_no_xmp_and_undo_brings_it_back(client, tmp_path): + path = _build(tmp_path, "pdf_xmp") + original = path.read_bytes() + resp = await client.post("/delete", json={"path": str(path), "keys": ["pdf:@xmp"]}) + assert resp.status_code == 200, resp.text + keys = _keys(path) + assert "pdf:@xmp" not in keys and not any(k.startswith("xmp:") for k in keys) + assert b"xmpmeta" not in path.read_bytes() # the packet itself is gone from the file + undo = await client.post("/undo", json={"token": resp.json()["undo_token"]}) + assert undo.status_code == 200 + assert path.read_bytes() == original + + +@pytest.mark.asyncio +async def test_sharing_a_pdf_removes_the_author_and_the_hidden_xmp(client, tmp_path): + path = _build(tmp_path, "pdf_xmp") + body = (await client.post("/sanitize", json={"path": str(path), "preview": False, + "categories": ["identity", "software", "history", "text"]})).json() + assert body["ok"], body + keys = _keys(path) + assert "pdf:Author" not in keys and not any(k.startswith("xmp:") for k in keys) + + +@pytest.mark.asyncio +async def test_signed_and_encrypted_pdfs_keep_their_xmp(client, tmp_path): + for name in ("pdf_signed_like", "pdf_encrypted"): + path = _build(tmp_path, name) + assert not any(f.deletable for f in HandlerRegistry.get(path).read(path).fields), name diff --git a/python/tests/test_png_tiff_fields.py b/python/tests/test_png_tiff_fields.py new file mode 100644 index 0000000..45b49ee --- /dev/null +++ b/python/tests/test_png_tiff_fields.py @@ -0,0 +1,78 @@ +"""PNG XMP / header / modification time and XMP + IPTC stored inside TIFF files.""" +import struct +import zlib + +from core.registry import HandlerRegistry + + +def _chunk(kind: bytes, body: bytes) -> bytes: + return struct.pack(">I", len(body)) + kind + body + struct.pack(">I", zlib.crc32(kind + body)) + + +def _read(path): + record = HandlerRegistry.get(path).read(path) + return record, {f.key: f for f in record.fields} + + +def test_png_xmp_is_interpreted_and_the_raw_chunk_is_hidden(tmp_path): + from tests.factories import by_name + path = by_name("png_text_exif").build(_dir(tmp_path)) + record, keys = _read(path) + assert record.read_errors == [] + assert keys["xmp:dc:title"].value == "Titolo è" + assert keys["xmp:xmp:CreatorTool"].value == "Tool & Co" + assert "png:XML:com.adobe.xmp" not in keys + + +def test_png_text_languages_have_distinct_keys(tmp_path): + from tests.factories import by_name + _, keys = _read(by_name("png_text_exif").build(_dir(tmp_path))) + assert keys["png:Description@it"].value.startswith("Descrizione") + assert keys["png:Description@en"].value == "Description" + assert keys["png:Title"].value == "PNG title" + + +def test_png_header_fields(tmp_path): + from tests.factories import by_name + _, keys = _read(by_name("png_text_exif").build(_dir(tmp_path))) + assert keys["img:bit_depth"].value == 8 + assert keys["img:color_type"].value == "RGB" + assert keys["img:interlace"].value is False + + +def test_png_modification_time(tmp_path): + from tests.factories import by_name + path = by_name("png_text_exif").build(_dir(tmp_path)) + data = path.read_bytes() + at = data.index(b"IDAT") - 4 + path.write_bytes(data[:at] + _chunk(b"tIME", struct.pack(">HBBBBB", 2026, 5, 1, 8, 20, 30)) + data[at:]) + _, keys = _read(path) + assert keys["pngchunk:tIME"].value == "2026-05-01T08:20:30+00:00" + + +def test_png_with_a_bad_time_chunk_is_still_read(tmp_path): + from tests.factories import by_name + path = by_name("png_text_exif").build(_dir(tmp_path)) + data = path.read_bytes() + at = data.index(b"IDAT") - 4 + path.write_bytes(data[:at] + _chunk(b"tIME", struct.pack(">HBBBBB", 2026, 13, 40, 99, 99, 99)) + data[at:]) + record, keys = _read(path) + assert record.read_errors == [] and "pngchunk:tIME" not in keys + + +def test_tiff_xmp_and_iptc_are_interpreted(tmp_path): + from tests.factories import by_name + record, keys = _read(by_name("tiff_xmp_iptc").build(_dir(tmp_path))) + assert record.read_errors == [] + assert keys["xmp:dc:title"].value == "Titolo è" + assert keys["xmp:dc:creator"].value == ["Ada"] + assert any(k.startswith("iptc:") for k in keys) + assert keys["xmp:dc:title"].editable and not keys["img:width"].editable + assert keys["xmp:dc:title"].deletable and not any(f.deletable for k, f in keys.items() if k.startswith("iptc:") and k != "iptc:@block") + assert keys["iptc:@block"].deletable + + +def _dir(tmp_path): + folder = tmp_path / "sample" + folder.mkdir(exist_ok=True) + return folder diff --git a/python/tests/test_png_webp_write.py b/python/tests/test_png_webp_write.py new file mode 100644 index 0000000..1411c45 --- /dev/null +++ b/python/tests/test_png_webp_write.py @@ -0,0 +1,153 @@ +"""Writing the metadata of PNG and WebP files (core/image_containers.py and the container states).""" +import struct +import zlib + +import pytest + +from core import image_containers +from core.handlers.image_handler import ImageHandler +from core.registry import HandlerRegistry +from tests.factories import by_name + + +def _build(tmp_path, name): + folder = tmp_path / name + folder.mkdir() + return by_name(name).build(folder) + + +def _keys(path): + return {f.key: f for f in HandlerRegistry.get(path).read(path).fields} + + +def _pixels(path): + return ImageHandler().content_digest(path) + + +async def _write(client, path, fields=(), delete=()): + return await client.post("/write", json={"path": str(path), "fields": list(fields), "delete": list(delete)}) + + +@pytest.mark.asyncio +async def test_png_exif_and_xmp_are_edited_and_the_picture_is_untouched(client, tmp_path): + path = _build(tmp_path, "png_text_exif") + before = _pixels(path) + resp = await _write(client, path, [{"key": "exif:Artist", "value": "Ada"}, + {"key": "xmp:dc:subject", "value": ["one", "two"]}]) + assert resp.status_code == 200, resp.text + keys = _keys(path) + assert keys["exif:Artist"].value == "Ada" and keys["xmp:dc:subject"].value == ["one", "two"] + assert _pixels(path) == before + assert image_containers.PngFile.parse(path.read_bytes()) # checksums are valid + + +@pytest.mark.asyncio +async def test_png_text_chunks_are_removed_and_undo_puts_them_back(client, tmp_path): + path = _build(tmp_path, "png_text_exif") + original = path.read_bytes() + resp = await _write(client, path, delete=["png:Title", "png:Comment"]) + assert resp.status_code == 200, resp.text + keys = _keys(path) + assert "png:Title" not in keys and "png:Comment" not in keys and "png:Description@it" in keys + undo = await client.post("/undo", json={"token": resp.json()["undo_token"]}) + assert undo.status_code == 200 + assert path.read_bytes() == original + + +@pytest.mark.asyncio +async def test_png_without_metadata_gets_a_new_exif_chunk(client, tmp_path): + from PIL import Image + path = tmp_path / "plain.png" + Image.new("RGB", (4, 4), (1, 2, 3)).save(path) + before = _pixels(path) + resp = await _write(client, path, [{"key": "exif:Artist", "value": "Grace"}]) + assert resp.status_code == 200, resp.text + assert _keys(path)["exif:Artist"].value == "Grace" and _pixels(path) == before + kinds = [k for k, _ in image_containers.PngFile.parse(path.read_bytes()).chunks] + assert kinds.index(b"eXIf") < kinds.index(b"IDAT") + + +@pytest.mark.asyncio +async def test_png_without_extension_is_edited_too(client, tmp_path): + source = _build(tmp_path, "png_text_exif") + path = tmp_path / "picture" + path.write_bytes(source.read_bytes()) + assert (await _write(client, path, [{"key": "exif:Artist", "value": "X"}])).status_code == 200 + assert _keys(path)["exif:Artist"].value == "X" + + +@pytest.mark.asyncio +async def test_damaged_png_stays_read_only(client, tmp_path): + path = _build(tmp_path, "png_text_exif") + data = bytearray(path.read_bytes()) + data[-20] ^= 0xFF # break a checksum + path.write_bytes(bytes(data)) + record = HandlerRegistry.get(path).read(path) + assert not record.format_supports_write + assert (await _write(client, path, [{"key": "exif:Artist", "value": "X"}])).status_code in (403, 422) + + +@pytest.mark.asyncio +async def test_webp_exif_and_xmp_are_edited(client, tmp_path): + path = _build(tmp_path, "webp_exif") + before = _pixels(path) + resp = await _write(client, path, [{"key": "exif:Artist", "value": "Ada"}, + {"key": "xmp:dc:subject", "value": ["z"]}]) + assert resp.status_code == 200, resp.text + keys = _keys(path) + assert keys["exif:Artist"].value == "Ada" and keys["xmp:dc:subject"].value == ["z"] + assert _pixels(path) == before + assert image_containers.WebpFile.parse(path.read_bytes()) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("lossless", [True, False]) +async def test_bare_webp_gets_a_vp8x_header_with_the_right_flags(client, tmp_path, lossless): + from PIL import Image + path = tmp_path / "bare.webp" + Image.new("RGB", (7, 5), (9, 9, 9)).save(path, "WEBP", lossless=lossless) + assert not image_containers.WebpFile.parse(path.read_bytes()).find(b"VP8X") or True + before = _pixels(path) + resp = await _write(client, path, [{"key": "exif:Artist", "value": "Ada"}]) + assert resp.status_code == 200, resp.text + webp = image_containers.WebpFile.parse(path.read_bytes()) + header = webp.chunks[0] + assert header[0] == b"VP8X" and header[1][0] & image_containers.FLAG_EXIF + assert (int.from_bytes(header[1][4:7], "little") + 1, int.from_bytes(header[1][7:10], "little") + 1) == (7, 5) + assert _pixels(path) == before + with Image.open(path) as img: # the library still decodes the picture + img.load() + assert img.size == (7, 5) + + +@pytest.mark.asyncio +async def test_deleting_the_xmp_fields_leaves_no_visible_xmp_and_keeps_the_flags_in_step(client, tmp_path): + path = _build(tmp_path, "webp_exif") + keys = [k for k, f in _keys(path).items() if f.deletable and k.startswith("xmp:")] + resp = await _write(client, path, delete=keys) + assert resp.status_code == 200, resp.text + assert not [k for k in _keys(path) if k.startswith("xmp:")] + webp = image_containers.WebpFile.parse(path.read_bytes()) + flags = next(b for k, b in webp.chunks if k == b"VP8X")[0] + assert bool(flags & image_containers.FLAG_XMP) == bool(webp.find(b"XMP ")) # flag matches the chunk + assert flags & image_containers.FLAG_EXIF + + +def test_containers_reject_inconsistent_files(): + with pytest.raises(image_containers.ContainerError): + image_containers.PngFile.parse(b"not a png") + with pytest.raises(image_containers.ContainerError): + image_containers.WebpFile.parse(b"RIFF" + struct.pack("I", 0) + b"IHDR" + struct.pack(">I", zlib.crc32(b"IHDR")) + with pytest.raises(image_containers.ContainerError): + image_containers.PngFile.parse(image_containers.PNG_SIGNATURE + chunk) # no IEND + + +def test_png_bytes_after_iend_are_preserved(): + from PIL import Image + import io + buffer = io.BytesIO() + Image.new("RGB", (2, 2)).save(buffer, "PNG") + data = buffer.getvalue() + b"trailer" + png = image_containers.PngFile.parse(data) + assert png.to_bytes() == data diff --git a/python/tests/test_qt_fields.py b/python/tests/test_qt_fields.py new file mode 100644 index 0000000..96a9e63 --- /dev/null +++ b/python/tests/test_qt_fields.py @@ -0,0 +1,148 @@ +"""QuickTime / ISO base media structure fields (core/isobmff.py, handlers/qt_fields.py).""" +import struct + +import pytest + +from core.handlers.qt_fields import qt_fields +from core.isobmff import Reader, open_reader +from core.registry import HandlerRegistry +from tests.factories import mp4 + + +@pytest.fixture +def phone(tmp_path): + return mp4.write_phone_movie(tmp_path / "phone.mov") + + +def _by_key(path): + fields, notes = qt_fields(path) + return {f.key: f for f in fields}, notes + + +def test_movie_and_track_headers(phone): + fields, notes = _by_key(phone) + assert fields["qt:MajorBrand"].value == "qt" + assert fields["qt:CreateDate"].value == "2026-05-01T08:20:30+00:00" + assert fields["qt:TimeScale"].value == 1000 + assert fields["qt:track1:HandlerType"].value == "Video" + assert (fields["qt:track1:Width"].value, fields["qt:track1:Height"].value) == (1920, 1080) + assert fields["qt:track1:Rotation"].value == 90 + assert fields["qt:track1:FrameRate"].value == 29.97 + assert fields["qt:track1:Language"].value == "ita" + assert fields["qt:track1:CodecID"].value == "avc1" + assert fields["qt:track2:HandlerType"].value == "Audio" + assert (fields["qt:track2:Channels"].value, fields["qt:track2:SampleRate"].value) == (2, 48000) + assert any("UTC" in n for n in notes) + + +def test_quicktime_keys_and_udta(phone): + fields, _ = _by_key(phone) + assert fields["qt:mdta:com.apple.quicktime.location.ISO6709"].value == "+45.4642+009.1900+120.500/" + assert fields["qt:mdta:com.apple.quicktime.model"].value == "iPhone 15 Pro" + assert fields["qt:mdta:com.apple.quicktime.live-photo.vitality-score"].value == 0.5 + assert fields["qt:mdta:com.apple.quicktime.content.identifier"].value == 7 + assert fields["qt:udta:©xyz"].value == "+45.4642+009.1900+120.500/" + + +def test_meta_without_full_box_header_as_written_by_phones(tmp_path): + path = mp4.write_phone_movie(tmp_path / "phone.mov", full_box_meta=False) + fields, _ = _by_key(path) + assert fields["qt:mdta:com.apple.quicktime.make"].value == "Apple" + assert fields["qt:track1:Rotation"].value == 90 + record = HandlerRegistry.get(path).read(path) # never raises, whatever mutagen thinks of the file + assert any(f.key == "qt:mdta:com.apple.quicktime.model" for f in record.fields) + assert record.read_errors == [] and record.format_supports_write + + +def test_structure_is_shown_when_the_tag_library_cannot_open_the_file(phone, monkeypatch): + import mutagen.mp4 + + def broken(*args, **kwargs): + raise mutagen.mp4.MP4StreamInfoError("unsupported stream layout") + monkeypatch.setattr(mutagen.mp4, "MP4", broken) + record = HandlerRegistry.get(phone).read(phone) + assert record.read_errors == [] and not record.format_supports_write + assert record.read_only_reason and any("Tags not read" in w for w in record.warnings) + assert any(f.key == "qt:track1:Rotation" for f in record.fields) + assert not any(f.editable or f.deletable for f in record.fields) + + +def test_xmp_packet_in_uuid_box(phone): + fields, _ = _by_key(phone) + assert fields["xmp:xmp:CreatorTool"].value == "Tool & Co" + assert fields["xmp:dc:creator"].value == ["Ada"] + + +def test_all_structure_fields_are_read_only(phone): + fields, _ = qt_fields(phone) + assert fields and not any(f.editable or f.deletable for f in fields) + + +def test_handler_serves_structure_and_keeps_tags_writable(phone): + mp4.add_full_tags(phone) + handler = HandlerRegistry.get(phone) + record = handler.read(phone) + keys = {f.key: f for f in record.fields} + assert record.read_errors == [] and record.format_supports_write + assert keys["qt:track1:Rotation"].value == 90 and not keys["qt:track1:Rotation"].editable + assert keys["mp4:©nam"].editable + assert any("UTC" in w for w in record.warnings) + + +def test_m4a_gets_the_structure_too(tmp_path): + path = mp4.write_mp4(tmp_path / "song.m4a") + mp4.add_full_tags(path) + record = HandlerRegistry.get(path).read(path) + assert any(f.key == "qt:MajorBrand" for f in record.fields) + + +def test_write_leaves_structure_and_media_untouched(phone): + mp4.add_full_tags(phone) + handler = HandlerRegistry.get(phone) + before = {k: v.value for k, v in _by_key(phone)[0].items()} + title = next(f for f in handler.read(phone).fields if f.key == "mp4:©nam") + title.value = ["Changed"] + handler.write(phone, [title]) + assert next(f for f in handler.read(phone).fields if f.key == "mp4:©nam").value == ["Changed"] + assert {k: v.value for k, v in _by_key(phone)[0].items()} == before + assert mp4.mdat_payload(phone) == mp4.MDAT_PAYLOAD + + +@pytest.mark.parametrize("data", [ + b"", b"\x00\x00\x00", struct.pack(">I4s", 4, b"free"), struct.pack(">I4s", 1 << 30, b"moov"), + struct.pack(">I4s", 0, b"moov") + b"\xff" * 3, + struct.pack(">I4s", 1, b"moov") + b"\x00" * 3, +]) +def test_damaged_files_never_raise(tmp_path, data): + path = tmp_path / "bad.mov" + path.write_bytes(data) + fields, _ = qt_fields(path) + assert isinstance(fields, list) + + +def test_truncated_movie_never_raises(phone, tmp_path): + data = phone.read_bytes() + for cut in range(0, len(data), 37): + path = tmp_path / "cut.mov" + path.write_bytes(data[:cut]) + qt_fields(path) + + +def test_box_walk_is_bounded(): + box = struct.pack(">I4s", 8, b"free") + reader = Reader(box * 200_000) + assert len(reader.children()) <= 100_000 + + +def test_reader_handles_64_bit_and_zero_sizes(tmp_path): + big = struct.pack(">I4sQ", 1, b"free", 20) + bytes(4) + last = struct.pack(">I4s", 0, b"mdat") + b"xyz" + with open_reader(_write(tmp_path, big + last)) as rd: + kinds = [(b.type, b.size) for b in rd.children()] + assert kinds == [(b"free", 20), (b"mdat", 11)] + + +def _write(tmp_path, data): + path = tmp_path / "x.bin" + path.write_bytes(data) + return path diff --git a/python/tests/test_raw_headers.py b/python/tests/test_raw_headers.py new file mode 100644 index 0000000..65a0d62 --- /dev/null +++ b/python/tests/test_raw_headers.py @@ -0,0 +1,81 @@ +"""TIFF-based raw headers (ORF, RW2) and the Exif 3.0 UTF-8 type.""" +import pytest + +from core import exif_codec +from core.registry import HandlerRegistry +from tests.factories import image + + +def _record(path): + return HandlerRegistry.get(path).read(path) + + +def _keys(record): + return {f.key: f for f in record.fields} + + +def test_rw2_is_read_without_errors(tmp_path): + path = tmp_path / "p.rw2" + path.write_bytes(image.le_raw_tiff(b"IIU\x00", b"Panasonic", b"DC-Test")) + record = _record(path) + keys = _keys(record) + assert record.read_errors == [] + assert keys["exif:Make"].value == "Panasonic" and keys["exif:Model"].value == "DC-Test" + assert any("not supported by the image library" in w for w in record.warnings) + assert record.format_supports_write and record.format_supports_delete # metadata changes are experimental + assert keys["exif:Make"].editable + + +@pytest.mark.parametrize("magic", [b"IIRO", b"IIRS"]) +def test_olympus_little_endian_headers(tmp_path, magic): + path = tmp_path / "o.orf" + path.write_bytes(image.le_raw_tiff(magic, b"OLYMPUS", b"E-M1")) + record = _record(path) + assert record.read_errors == [] and _keys(record)["exif:Make"].value == "OLYMPUS" + + +def test_olympus_big_endian_header(tmp_path): + path = tmp_path / "o.orf" + path.write_bytes(b"MMOR" + image.camera_exif(thumbnail=False)[10:]) + record = _record(path) + assert record.read_errors == [] and _keys(record)["exif:Make"].value == "Canon" + + +def test_raw_headers_are_refused_outside_raw_files(): + data = image.le_raw_tiff(b"IIU\x00", b"x", b"y") + with pytest.raises(exif_codec.ExifFormatError): + exif_codec.parse(data) + assert exif_codec.parse(data, raw=True).ifds["0th"] + + +def test_unknown_raw_extensions_are_served_by_the_image_handler(tmp_path): + for name in ("a.nrw", "a.sr2", "a.3fr", "a.iiq", "a.rwl"): + path = tmp_path / name + path.write_bytes(image.le_raw_tiff(b"II*\x00", b"Maker", b"Model")) + assert HandlerRegistry.get(path).NAME == "image" + assert _keys(_record(path))["exif:Make"].value == "Maker" + + +def test_utf8_exif_type_is_read_and_does_not_block_a_rewrite(tmp_path): + # ImageDescription stored as type 129 (UTF-8) in an otherwise ordinary camera EXIF block. + block = bytearray(image.camera_exif(thumbnail=False)[6:]) + exif = exif_codec.parse(bytes(block)) + assert not exif.rewrite_problems + raw = image.le_raw_tiff(b"II*\x00", b"Maker", b"Model") + parsed = exif_codec.parse(raw) + assert parsed.rewrite_problems == [] + entry = parsed.ifds["0th"][0x010E] + assert entry.type == 129 + decoded = exif_codec.decode("0th", 0x010E, entry, parsed.byteorder) + assert decoded.value == "Descrizione è" and not decoded.editable + + +def test_jpeg_with_utf8_exif_type_stays_writable(tmp_path): + segment = b"Exif\x00\x00" + image.le_raw_tiff(b"II*\x00", b"Maker", b"Model") + path = tmp_path / "u.jpg" + path.write_bytes(image.jpeg_with([image.segment(0xE1, segment)])) + record = _record(path) + keys = _keys(record) + assert record.read_errors == [] and record.format_supports_write + assert keys["exif:ImageDescription"].value == "Descrizione è" and not keys["exif:ImageDescription"].editable + assert keys["exif:Make"].editable diff --git a/python/tests/test_registry.py b/python/tests/test_registry.py index b2e0c8f..436be10 100644 --- a/python/tests/test_registry.py +++ b/python/tests/test_registry.py @@ -100,15 +100,22 @@ def test_wav_renamed_bin_is_read_by_hachoir_from_content(tmp_path): assert any(f.key.startswith("hachoir:") for f in record.fields) -def test_jpeg_without_extension_is_read_by_hachoir_from_content(tmp_path): +def test_jpeg_without_extension_is_handled_as_jpeg_from_content(tmp_path): from PIL import Image path = tmp_path / "photo" Image.new("RGB", (8, 6)).save(path, "JPEG") handler = HandlerRegistry.get(path) - assert handler.NAME == "hachoir" + assert handler.NAME == "image" record = handler.read(path) assert record.read_errors == [] - assert any(f.key.startswith("hachoir:") for f in record.fields) + assert record.format_supports_write + assert not any(f.key.startswith("hachoir:") for f in record.fields) + + +def test_unrecognized_file_without_extension_still_goes_to_hachoir(tmp_path): + path = tmp_path / "noextension" + path.write_bytes(b"7" * 32) + assert HandlerRegistry.get(path).NAME == "hachoir" @pytest.mark.asyncio diff --git a/python/tests/test_sanitize.py b/python/tests/test_sanitize.py new file mode 100644 index 0000000..b449cf4 --- /dev/null +++ b/python/tests/test_sanitize.py @@ -0,0 +1,117 @@ +"""Prepare a file for sharing (POST /sanitize).""" +import pytest + +from core.registry import HandlerRegistry +from tests.factories import by_name + +PRIVATE = ["location", "identity", "device", "thumbnail", "history"] + + +def _build(tmp_path, name): + folder = tmp_path / name + folder.mkdir() + return by_name(name).build(folder) + + +def _keys(path): + return {f.key: f for f in HandlerRegistry.get(path).read(path).fields} + + +async def _sanitize(client, path, categories, preview=False, keep=()): + resp = await client.post("/sanitize", json={"path": str(path), "categories": categories, + "preview": preview, "keep": list(keep)}) + return resp + + +@pytest.mark.asyncio +async def test_preview_reports_without_touching_the_file(client, tmp_path): + path = _build(tmp_path, "jpeg_camera") + before = path.read_bytes() + body = (await _sanitize(client, path, ["location"], preview=True)).json() + assert body["ok"] and body["preview"] + assert {"gps:Latitude", "gps:Longitude"} <= {e["key"] for e in body["will_remove"]} + assert all(e["category"] == "location" for e in body["will_remove"]) + assert path.read_bytes() == before and body["undo_token"] is None + + +@pytest.mark.asyncio +async def test_location_is_removed_and_undo_restores_it(client, tmp_path): + path = _build(tmp_path, "jpeg_camera") + original = path.read_bytes() + resp = await _sanitize(client, path, ["location"]) + body = resp.json() + assert resp.status_code == 200 and body["undo_token"] + assert not any(k.startswith("gps:") for k in _keys(path)) + assert "exif:Make" in _keys(path) # other categories are untouched + assert body["remaining"] == [] and body["removed"] + undo = await client.post("/undo", json={"token": body["undo_token"]}) + assert undo.status_code == 200 + assert any(k.startswith("gps:") for k in _keys(path)) + assert path.read_bytes() == original # the Undo gives the original bytes back + + +@pytest.mark.asyncio +async def test_several_categories_in_one_save(client, tmp_path): + path = _build(tmp_path, "jpeg_camera") + body = (await _sanitize(client, path, PRIVATE)).json() + keys = _keys(path) + assert not any(k.startswith("gps:") for k in keys) and "exif:Make" not in keys and "exif:Artist" not in keys + assert body["undo_token"] + assert body["remaining"] == [] or all(r["category"] in PRIVATE for r in body["remaining"]) + + +@pytest.mark.asyncio +async def test_keep_leaves_chosen_fields(client, tmp_path): + path = _build(tmp_path, "jpeg_camera") + await _sanitize(client, path, ["identity", "device"], keep=["exif:Make"]) + keys = _keys(path) + assert "exif:Make" in keys and "exif:Artist" not in keys + + +@pytest.mark.asyncio +async def test_what_cannot_be_removed_is_reported(client, tmp_path): + path = _build(tmp_path, "gif_comment") # a read-only format + body = (await _sanitize(client, path, ["text"])).json() + assert body["will_remove"] == [] and body["undo_token"] is None + assert body["remaining"] and all(r["reason"] for r in body["remaining"]) + + +@pytest.mark.asyncio +async def test_works_on_other_writable_formats(client, tmp_path): + for name, category, gone in (("pdf_basic", "identity", "pdf:Author"), ("docx", "identity", "office:author"), + ("mp3_v24_full", "text", "tag:COMM:note:eng")): + path = _build(tmp_path, name) + if gone not in _keys(path): + continue + body = (await _sanitize(client, path, [category])).json() + assert body["ok"] and gone not in _keys(path), name + + +@pytest.mark.asyncio +async def test_unknown_category_and_bad_path_are_rejected(client, tmp_path): + path = _build(tmp_path, "jpeg_camera") + assert (await _sanitize(client, path, ["nonsense"])).status_code == 422 + assert (await _sanitize(client, tmp_path / "missing.jpg", ["location"])).status_code in (400, 404) + resp = await client.post("/sanitize", json={"path": str(path), "categories": []}) + assert resp.status_code == 422 + + +@pytest.mark.asyncio +async def test_nothing_to_remove_is_not_an_error(client, tmp_path): + path = _build(tmp_path, "jpeg_no_exif") + body = (await _sanitize(client, path, ["location"])).json() + assert body["ok"] and body["will_remove"] == [] and body["undo_token"] is None + + +@pytest.mark.asyncio +async def test_hidden_xmp_data_is_listed_removed_and_restored_by_undo(client, tmp_path): + path = _build(tmp_path, "jpeg_xmp_first") + original = path.read_bytes() + keys = _keys(path) + assert "xmp:@hidden" in keys and keys["xmp:@hidden"].deletable + body = (await _sanitize(client, path, ["history"])).json() + assert [e["key"] for e in body["will_remove"]] == ["xmp:@hidden"] or body["removed"] + assert "xmp:@hidden" not in _keys(path) and body["remaining"] == [] + undo = await client.post("/undo", json={"token": body["undo_token"]}) + assert undo.status_code == 200 and "xmp:@hidden" in _keys(path) + assert path.read_bytes() == original diff --git a/python/tests/test_sensitivity.py b/python/tests/test_sensitivity.py new file mode 100644 index 0000000..6e7506c --- /dev/null +++ b/python/tests/test_sensitivity.py @@ -0,0 +1,53 @@ +"""Classification of fields by what they reveal when the file is shared.""" +import pytest + +from core import sensitivity +from core.registry import HandlerRegistry +from tests.factories import ALL + + +@pytest.mark.parametrize("key, expected", [ + ("gps:Latitude", "location"), ("xmp:photoshop:City", "location"), ("iptc:Sub-location", "location"), + ("qt:mdta:com.apple.quicktime.location.ISO6709", "location"), ("qt:udta:©xyz", "location"), + ("exif:Artist", "identity"), ("pdf:Author", "identity"), ("office:last_modified_by", "identity"), + ("odf:initial-creator", "identity"), ("fs:owner", "identity"), ("xmp:dc:creator", "identity"), + ("exif:Make", "device"), ("exif:BodySerialNumber", "device"), ("qt:mdta:com.apple.quicktime.model", "device"), + ("exif:DateTimeOriginal", "datetime"), ("pdf:CreationDate", "datetime"), ("qt:CreateDate", "datetime"), + ("qt:track1:CreateDate", "datetime"), ("fs:mtime", "datetime"), + ("exif:Software", "software"), ("pdf:Producer", "software"), ("xmp:xmpMM:History", "software"), + ("exif:Thumbnail", "thumbnail"), ("xmp:@hidden", "history"), ("exif:UserComment", "text"), ("jpeg:Comment", "text"), + ("img:width", None), ("exif:Orientation", None), ("audio:bitrate", None), ("icc:Version", None), +]) +def test_classification(key, expected): + assert sensitivity.classify(key) == expected + + +def test_every_rule_names_a_known_category_and_severity(): + assert {c for c, _ in sensitivity._RULES} <= set(sensitivity.CATEGORIES) == set(sensitivity.SEVERITY) + + +def test_summary_is_ordered_by_severity(): + summary = sensitivity.summarize(["exif:DateTime", "gps:Latitude", "gps:Longitude", "exif:Make", "img:width"]) + assert summary == {"location": 2, "device": 1, "datetime": 1} + assert list(summary)[0] == "location" + + +def test_records_and_fields_carry_the_classification(tmp_path): + sample = next(s for s in ALL if s.name == "jpeg_camera") + folder = tmp_path / "s" + folder.mkdir() + path = sample.build(folder) + data = HandlerRegistry.get(path).read(path).to_dict() + by_key = {f["key"]: f for f in data["fields"]} + assert by_key["gps:Latitude"]["sensitivity"] == "location" + assert by_key["img:width"]["sensitivity"] is None + assert data["sensitivity_summary"]["location"] >= 2 + assert next(iter(data["sensitivity_summary"])) in ("location", "identity") + + +def test_all_sample_keys_classify_without_error(): + # No key of any sample may crash the classifier; names with odd characters included. + for sample in ALL: + if not sample.available: + continue + sensitivity.classify(sample.name) diff --git a/python/tests/test_sniff.py b/python/tests/test_sniff.py new file mode 100644 index 0000000..5aa97ba --- /dev/null +++ b/python/tests/test_sniff.py @@ -0,0 +1,94 @@ +"""Format recognition from content (core/sniff.py).""" +import struct +import zipfile +from pathlib import Path + +import pytest + +from core import sniff + + +def _write(tmp_path: Path, name: str, data: bytes) -> Path: + path = tmp_path / name + path.write_bytes(data) + return path + + +def _ftyp(brand: bytes) -> bytes: + return struct.pack(">I", 24) + b"ftyp" + brand + b"\x00\x00\x00\x00" + b"isom" + b"mp42" + + +@pytest.mark.parametrize("data, expected", [ + (b"\xff\xd8\xff\xe0" + bytes(60), "jpeg"), + (b"\x89PNG\r\n\x1a\n" + bytes(56), "png"), + (b"GIF89a" + bytes(58), "gif"), + (b"II*\x00" + bytes(60), "tiff"), + (b"MM\x00*" + bytes(60), "tiff"), + (b"RIFF\x00\x00\x00\x00WEBPVP8 " + bytes(40), "webp"), + (b"RIFF\x00\x00\x00\x00AVI LIST" + bytes(40), "avi"), + (b"RIFF\x00\x00\x00\x00WAVEfmt " + bytes(40), "wav"), + (b"FORM\x00\x00\x00\x00AIFFCOMM" + bytes(40), "aiff"), + (_ftyp(b"heic"), "heif"), + (_ftyp(b"avif"), "avif"), + (_ftyp(b"crx "), "cr3"), + (_ftyp(b"M4A "), "m4a"), + (_ftyp(b"qt "), "mov"), + (_ftyp(b"isom"), "mp4"), + (b"\x1aE\xdf\xa3" + bytes(60), "mkv"), + (b"fLaC" + bytes(60), "flac"), + (b"OggS" + bytes(60), "ogg"), + (b"%PDF-1.7\n" + bytes(50), "pdf"), + (bytes.fromhex("d0cf11e0a1b11ae1") + bytes(56), "ole"), + (b"ID3\x04\x00" + bytes(59), "mp3"), + (b"\xff\xfb\x90\x00" + bytes(60), "mp3"), +]) +def test_signatures(tmp_path, data, expected): + assert sniff.detect(_write(tmp_path, "sample", data)) == expected + + +@pytest.mark.parametrize("data", [ + b"", b"II", b"hello world, this is text", b"\x13\x37" * 32, + b"II text that starts with the byte order mark letters", + b"\xff\xfb\xf0\x00" + bytes(60), # bitrate index 15 / free: not an MPEG frame +]) +def test_not_recognized(tmp_path, data): + assert sniff.detect(_write(tmp_path, "sample", data)) is None + + +def test_zip_packages(tmp_path): + ooxml = tmp_path / "pkg" + with zipfile.ZipFile(ooxml, "w") as z: + z.writestr("[Content_Types].xml", "") + plain = tmp_path / "plain" + with zipfile.ZipFile(plain, "w") as z: + z.writestr("readme.txt", "x") + assert sniff.detect(ooxml) == "ooxml" + assert sniff.detect(plain) is None + + +def test_missing_file_is_not_recognized(tmp_path): + assert sniff.detect(tmp_path / "absent") is None + + +def test_handler_names(): + assert sniff.handler_name("jpeg") == "image" + assert sniff.handler_name("mkv") == "video" + assert sniff.handler_name("heif") == "heif" and sniff.handler_name("avif") == "heif" + assert sniff.handler_name("cr3") == "image" and sniff.handler_name("raf") == "image" + assert sniff.handler_name(None) is None + + +def test_only_the_head_is_read(tmp_path, monkeypatch): + path = _write(tmp_path, "big", b"\xff\xd8\xff" + bytes(10_000)) + seen = [] + real_open = open + + def spy(file, mode="r", *a, **k): + stream = real_open(file, mode, *a, **k) + original = stream.read + stream.read = lambda n=-1: (seen.append(n), original(n))[1] + return stream + + monkeypatch.setattr("builtins.open", spy) + assert sniff.detect(path) == "jpeg" + assert seen == [sniff.HEAD_BYTES] diff --git a/python/tests/test_tiff_inplace.py b/python/tests/test_tiff_inplace.py new file mode 100644 index 0000000..9692a66 --- /dev/null +++ b/python/tests/test_tiff_inplace.py @@ -0,0 +1,128 @@ +"""Removing metadata from TIFF-based files in place (core/tiff_inplace.py).""" +import pytest + +from core import tiff_inplace +from core.handlers.image_handler import ImageHandler +from core.registry import HandlerRegistry +from tests.factories import by_name + + +def _build(tmp_path, name): + folder = tmp_path / name + folder.mkdir() + return by_name(name).build(folder) + + +def _keys(path): + return {f.key: f for f in HandlerRegistry.get(path).read(path).fields} + + +async def _sanitize(client, path, categories): + return (await client.post("/sanitize", json={"path": str(path), "categories": categories, "preview": False})).json() + + +def test_removal_and_restoration_are_exact_for_every_entry(tmp_path): + original = _build(tmp_path, "orf_like").read_bytes() + _, ifds = tiff_inplace.parse(original) + count = 0 + for name, ifd in ifds.items(): + for entry in ifd.entries: + if not tiff_inplace.removable(name, entry.tag): + continue + data = bytearray(original) + saved = tiff_inplace.snapshot(data, name, entry.tag) + removed = tiff_inplace.remove(data, name, {entry.tag}) + assert removed == 1 + assert len(data) == len(original) + assert tiff_inplace.snapshot(data, name, entry.tag) is None + tiff_inplace.insert(data, saved) + assert bytes(data) == original, (name, hex(entry.tag)) + count += 1 + assert count > 10 + + +def test_structure_entries_are_never_removed(tmp_path): + path = _build(tmp_path, "tiff_dpi") + keys = _keys(path) + for structural in ("exif:ImageLength", "exif:StripOffsets", "exif:ImageWidth"): + if structural in keys: + assert not keys[structural].deletable, structural + assert not tiff_inplace.removable("0th", 0x0111) and not tiff_inplace.removable("Exif", 0xA002) + + +def test_a_table_without_next_pointer_is_handled(tmp_path): + original = _build(tmp_path, "orf_like").read_bytes() # its Exif IFD has no next-IFD pointer + _, ifds = tiff_inplace.parse(original) + assert ifds["Exif"].next_at is None + data = bytearray(original) + tiff_inplace.remove(data, "Exif", {0x829A, 0x829D}) + tiff_inplace.insert(data, tiff_inplace.snapshot(bytearray(original), "Exif", 0x829D)) + tiff_inplace.insert(data, tiff_inplace.snapshot(bytearray(original), "Exif", 0x829A)) + assert bytes(data) == original + + +@pytest.mark.asyncio +async def test_sharing_a_raw_removes_location_and_device_without_moving_anything(client, tmp_path): + path = _build(tmp_path, "orf_like") + original, media = path.read_bytes(), ImageHandler().content_digest(path) + body = await _sanitize(client, path, ["location", "device"]) + assert body["ok"] and body["undo_token"], body + keys = _keys(path) + assert not any(k.startswith("gps:") for k in keys) and "exif:Make" not in keys + data = path.read_bytes() + assert len(data) == len(original) and ImageHandler().content_digest(path) == media + assert b"Canon" not in data and b"EOS Test" not in data # the removed values are gone from the file + assert (await client.post("/undo", json={"token": body["undo_token"]})).status_code == 200 + assert path.read_bytes() == original + + +@pytest.mark.asyncio +async def test_sharing_a_tiff_with_xmp_reduces_the_packet_in_place(client, tmp_path): + path = _build(tmp_path, "tiff_xmp_iptc") + original = path.read_bytes() + body = await _sanitize(client, path, ["identity", "software", "history", "text"]) + assert body["ok"], body + keys = _keys(path) + assert "xmp:dc:creator" not in keys and "xmp:xmp:CreatorTool" not in keys + assert len(path.read_bytes()) == len(original) + assert (await client.post("/undo", json={"token": body["undo_token"]})).status_code == 200 + assert path.read_bytes() == original + + +@pytest.mark.asyncio +async def test_what_cannot_be_removed_from_a_tiff_is_reported(client, tmp_path): + path = _build(tmp_path, "tiff_xmp_iptc") + body = await _sanitize(client, path, ["text"]) + assert any(r["key"].startswith("iptc:") and r["reason"] for r in body["remaining"]) or body["ok"] + + +def test_the_media_digest_ignores_the_tables_but_not_the_picture(tmp_path): + path = _build(tmp_path, "tiff_dpi") + data = bytearray(path.read_bytes()) + before = tiff_inplace.media_digest(data) + assert before + tiff_inplace.remove(data, "0th", {0x0131, 0x010F}) + assert tiff_inplace.media_digest(data) == before + data[-3] ^= 0xFF + assert tiff_inplace.media_digest(data) != before + + +@pytest.mark.parametrize("data", [b"", b"II*\x00", b"II*\x00\xff\xff\xff\xff", b"junk"]) +def test_damaged_input_is_an_error_not_a_crash(data): + with pytest.raises((tiff_inplace.TiffError, Exception)): + tiff_inplace.parse(data) + assert tiff_inplace.media_digest(data) is None + + +@pytest.mark.asyncio +async def test_the_iptc_record_of_a_tiff_is_removed_as_a_whole_and_comes_back(client, tmp_path): + path = _build(tmp_path, "tiff_xmp_iptc") + original = path.read_bytes() + assert any(k.startswith("iptc:") and k != "iptc:@block" for k in _keys(path)) + resp = await client.post("/delete", json={"path": str(path), "keys": ["iptc:@block"]}) + assert resp.status_code == 200, resp.text + left = _keys(path) + assert not any(k.startswith("iptc:") for k in left) and "xmp:dc:title" in left + assert len(path.read_bytes()) == len(original) + assert (await client.post("/undo", json={"token": resp.json()["undo_token"]})).status_code == 200 + assert path.read_bytes() == original diff --git a/python/tests/test_tiff_write.py b/python/tests/test_tiff_write.py new file mode 100644 index 0000000..1bf2353 --- /dev/null +++ b/python/tests/test_tiff_write.py @@ -0,0 +1,110 @@ +"""Editing and adding metadata in TIFF-based files (nothing existing is moved).""" +import pytest + +from core import tiff_inplace +from core.handlers.image_handler import ImageHandler +from core.registry import HandlerRegistry +from tests.factories import by_name + + +def _build(tmp_path, name): + folder = tmp_path / name + folder.mkdir() + return by_name(name).build(folder) + + +def _keys(path): + return {f.key: f for f in HandlerRegistry.get(path).read(path).fields} + + +async def _write(client, path, fields=(), delete=()): + return await client.post("/write", json={"path": str(path), "fields": list(fields), "delete": list(delete)}) + + +async def _undo(client, resp): + return await client.post("/undo", json={"token": resp.json()["undo_token"]}) + + +@pytest.mark.asyncio +async def test_a_shorter_value_is_written_over_the_old_one(client, tmp_path): + path = _build(tmp_path, "orf_like") + original, media = path.read_bytes(), ImageHandler().content_digest(path) + assert _keys(path)["exif:Artist"].editable + resp = await _write(client, path, [{"key": "exif:Artist", "value": "Al"}]) + assert resp.status_code == 200, resp.text + assert _keys(path)["exif:Artist"].value == "Al" + assert len(path.read_bytes()) == len(original) and ImageHandler().content_digest(path) == media + assert (await _undo(client, resp)).status_code == 200 and path.read_bytes() == original + + +@pytest.mark.asyncio +async def test_a_longer_value_is_appended_and_the_old_one_is_zeroed(client, tmp_path): + path = _build(tmp_path, "orf_like") + original, media = path.read_bytes(), ImageHandler().content_digest(path) + long_name = "Grace Brewster Murray Hopper, rear admiral" + resp = await _write(client, path, [{"key": "exif:Artist", "value": long_name}]) + assert resp.status_code == 200, resp.text + assert _keys(path)["exif:Artist"].value == long_name + data = path.read_bytes() + assert len(data) > len(original) and ImageHandler().content_digest(path) == media + assert data[:len(original)].count("José".encode("latin-1")) == 0 # the old value does not remain + assert (await _undo(client, resp)).status_code == 200 + assert path.read_bytes()[:len(original)] == original + + +@pytest.mark.asyncio +async def test_new_entries_are_added_and_missing_ifds_are_created(client, tmp_path): + path = _build(tmp_path, "tiff_dpi") # has neither an Exif IFD nor a copyright + original, media = path.read_bytes(), ImageHandler().content_digest(path) + record = HandlerRegistry.get(path).read(path) + offered = {f.key for f in record.addable_fields} + assert {"exif:Copyright", "exif:DateTimeOriginal"} <= offered + resp = await _write(client, path, [{"key": "exif:Copyright", "value": "(c) Ada"}, + {"key": "exif:DateTimeOriginal", "value": "2026-02-03T04:05:06"}]) + assert resp.status_code == 200, resp.text + keys = _keys(path) + assert keys["exif:Copyright"].value == "(c) Ada" and keys["exif:DateTimeOriginal"].value == "2026-02-03T04:05:06" + assert ImageHandler().content_digest(path) == media + tiff_inplace.parse(path.read_bytes()) # still a well-formed TIFF + from PIL import Image + with Image.open(path) as img: + img.load() + assert (await _undo(client, resp)).status_code == 200 + assert "exif:Copyright" not in _keys(path) and "exif:DateTimeOriginal" not in _keys(path) + assert ImageHandler().content_digest(path) == media + + +@pytest.mark.asyncio +async def test_xmp_is_added_to_a_tiff_and_then_edited(client, tmp_path): + path = _build(tmp_path, "tiff_dpi") + media = ImageHandler().content_digest(path) + resp = await _write(client, path, [{"key": "xmp:dc:title", "value": "Un titolo"}]) + assert resp.status_code == 200, resp.text + assert _keys(path)["xmp:dc:title"].value == "Un titolo" + longer = "Un titolo molto piu lungo " * 40 + resp2 = await _write(client, path, [{"key": "xmp:dc:title", "value": longer}]) + assert resp2.status_code == 200, resp2.text + assert _keys(path)["xmp:dc:title"].value == longer and ImageHandler().content_digest(path) == media + + +@pytest.mark.asyncio +async def test_picture_structure_cannot_be_edited(client, tmp_path): + path = _build(tmp_path, "tiff_dpi") + before = path.read_bytes() + resp = await _write(client, path, [{"key": "exif:ImageWidth", "value": 5}]) + assert resp.status_code == 422 and path.read_bytes() == before + + +@pytest.mark.asyncio +async def test_a_raw_edit_keeps_the_maker_note_where_it_is(client, tmp_path): + path = _build(tmp_path, "orf_like") + before = path.read_bytes() + _, ifds = tiff_inplace.parse(before) + note = next(e for e in ifds["Exif"].entries if e.tag == 0x927C) + resp = await _write(client, path, [{"key": "exif:Artist", "value": "A much longer artist name than before"}]) + assert resp.status_code == 200, resp.text + after = path.read_bytes() + _, ifds_after = tiff_inplace.parse(after) + moved = next(e for e in ifds_after["Exif"].entries if e.tag == 0x927C) + assert moved.data_at == note.data_at and after[note.data_at:note.data_at + note.size] == \ + before[note.data_at:note.data_at + note.size] diff --git a/python/tests/test_xmp_languages.py b/python/tests/test_xmp_languages.py new file mode 100644 index 0000000..505b5b6 --- /dev/null +++ b/python/tests/test_xmp_languages.py @@ -0,0 +1,16 @@ +"""The other languages of an XMP text are shown next to the main one.""" +from core.handlers.image_handler import _xmp_fields +from core.xmp import XmpPacket + +PACKET = b""" + +SunsetSonnenuntergangTramonto +""" + + +def test_other_languages_are_listed_read_only(): + fields = {f.key: f for f in _xmp_fields(XmpPacket(PACKET), True)} + assert fields["xmp:dc:title"].value == "Sunset" and fields["xmp:dc:title"].has_alternatives + assert fields["xmp:dc:title@de"].value == "Sonnenuntergang" and fields["xmp:dc:title@it"].value == "Tramonto" + for key in ("xmp:dc:title@de", "xmp:dc:title@it"): + assert fields[key].computed and not fields[key].editable and not fields[key].deletable diff --git a/scripts/smoke_samples.txt b/scripts/smoke_samples.txt index 186ffe5..1875206 100644 --- a/scripts/smoke_samples.txt +++ b/scripts/smoke_samples.txt @@ -1 +1 @@ -jpeg_camera jpeg_iptc png_text_exif webp_exif gif_comment tiff_dpi mp3_v24_full flac_vorbis_picture ogg_opus wav_id3 m4a_tags mp4_tags pdf_xmp docx xlsx pptx ole_doc_cp1252 avi_two_streams mkv_two_tracks random_xyz +jpeg_camera jpeg_iptc png_text_exif webp_exif gif_comment tiff_dpi mp3_v24_full flac_vorbis_picture ogg_opus wav_id3 m4a_tags mp4_tags pdf_xmp docx xlsx pptx ole_doc_cp1252 avi_two_streams mkv_two_tracks random_xyz mov_full heic_camera avif_basic odt_meta epub_book photo_no_extension tiff_xmp_iptc jpeg_icc_comment mkv_full orf_like cr3_like raf_like