Fix leaks, round-trip failures and config bugs from the second bug hunt

CHANGELOG.md

@@ -29,6 +29,22 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
   (`Рапорт Білого Ярослава Миколайовича`).
 - `Шановна Любове Андріївно!`: the vocative `Любове` was not recognised, so
   the name got a surname mask and the patronymic a male name.
+- A name in brackets, slashes or asterisks (`(Петренко Іван Петрович)`,
+  `/Петренко І.П./`) kept the surname open, masked the name as a surname and
+  dropped the closing bracket from the masked text. The rank in
+  `(сержант Петренко)` was not masked either.
+- An infinitive opening an order item was taken for a surname when no
+  patronymic followed: `Видати Петренку Івану` became
+  `Вильчи Павлу Хомович`, with the surname masked as a first name.
+
+### Fixed — numbers left unmasked
+- A mask could equal the original, and the number stayed open. Masks of
+  military IDs change two digits out of six, so about one ID in a hundred
+  (`АА 585285`) was not masked. A one-digit BR number (`№5дск`) was not
+  masked one time in ten. Passport numbers, tax numbers (ІПН) and military
+  units were also affected, more rarely. Each of these masks is now
+  regenerated until it differs from the original and from the open text.
+  Masks that already differed stay the same as before.
 
 ### Fixed — unmasking did not restore the original
 - A rank `… медичної служби` masked as `генерал-майор медичної служби` was
@@ -49,6 +65,109 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
   are now recorded in text order.
 - A brigade number broken across lines lost the rest of its name in the
   masked text (`60 окремої\nмеханізованої бригади` → `30 окремої`).
+- JSON files: unmasking counted the occurrences of each mask anew in every
+  string, while masking numbers them across the whole document. When two
+  different values in different strings got the same mask (`№ 2` and `№ 6`
+  both masked as `№ 4`), the second one came back as the first. Masking also
+  checked a new mask only against the open words of its own string. Both now
+  treat all the strings of a file as one text.
+- An order number mask could equal part of an open number (`№ 2` masked as
+  `1` next to `п. 1.2`), and unmasking gave its value to the wrong place.
+- A naval rank after `*` or `/` (`*старший матрос Кравченко …*`) was masked
+  without its first word, and the result (`*старший моряк`) was not restored.
+- A text date with a capitalised month came back in lower case
+  (`06 Жовтня 2025` → `06 жовтня 2025`). A month in capitals is now masked in
+  capitals and restored in capitals too. The end of a longer number was taken
+  for a day (`123 травня 2025` masked as `125 серпня 2026`) and was not
+  restored.
+- A value listed in `exclusions.always_mask` or matched by a custom pattern
+  with `action: mask` was masked once more as a surname (`сержант Сокіл` →
+  `Цуліж` → `Цулій`), and unmasking returned the first mask instead of
+  `Сокіл`. A custom pattern such as `[A-Z]{3,}` also matched the placeholder
+  of a protected phrase, which then disappeared from both texts.
+
+### Fixed — configuration
+- `exclusions` (words, phrases, always_mask, remove) now match every
+  apostrophe: `Дем'ян` in the list also covers `Дем’ян` and `Демʼян`. Before,
+  `always_mask` left those spellings open, and `phrases`/`words` did not
+  protect them.
+- `config_local.example.yaml` set every `exclusions` list to `[]`. A copy of
+  the template therefore erased the lists from the shared `config.yaml`,
+  `always_mask` included. The template now uses `null`, which keeps the
+  shared value.
+- `config.yaml` and `config_example.yaml` set `validation.max_name_length: 50`,
+  so with them a longer double surname stayed open, while without a
+  configuration it was masked. The value is now `null` (no limit), and masks
+  with the shared file are again the same as without it.
+- An environment variable with an unknown boolean word turned the option off:
+  `DATA_MASKING_ENCRYPT_OUTPUT=y` wrote a plaintext mapping even with
+  `encrypt_output: true` in the file. Such a value is now skipped with a
+  warning, like an invalid number.
+- `DATA_MASKING_PRESERVE_CASE` lost to `masking_rules.preserve_case` from the
+  file; the environment now wins, as for every other key.
+- `router_rules.skip_types` / `only_types` in a file made the same file's
+  `masking_rules.enable_*` switches ignored. Both now apply: a type is masked
+  only when neither turns it off. `--only` / `--exclude` on the command line
+  still override `enable_*`.
+- `security.password_generation: "false"` (a quoted string) was ignored and
+  replaced the whole section. It is now read as a boolean, and any other
+  string is an error.
+- A fractional value of an integer key (`hash_digest_size: 2.9`) was cut down
+  silently; it is now an error.
+- With `--re-mask N`, the count of matches left open by a custom pattern with
+  `action: warn` was multiplied by N.
+- An empty section (`exclusions:` with only comments, `masking_rules: {}`)
+  was reported as a key with no effect, and `strict_mode` stopped the run.
+- Configuration errors that ended in a traceback now give a message and exit
+  code 1: `system.hash_algorithm` in capitals (`SHA256`), `backup_suffix`
+  with a path separator, `logging.level` that is not a string, and a section
+  key that is not a string (YAML `on:`, `yes:`, a number). An unknown
+  `logging.level` (`verbose`) now gives a warning instead of silently
+  becoming `INFO`.
+- `python -m datamasking.diagnose` returned 0 for an unreadable or encrypted
+  file and for a JSON array, and stopped with a traceback on
+  `{"mappings": null}` or on an entry without `masked_as`. It now reports the
+  problem and returns 1.
+
+### Fixed — command line and files
+- With `system.encoding: auto`, a UTF-16 file was read as cp1251, so nothing
+  was masked and the run still exited with 0. A byte order mark (UTF-8,
+  UTF-16, UTF-32) now decides the encoding, and the output and the restored
+  file keep it byte for byte. Text that decodes only with NUL characters
+  (UTF-16 without a BOM) is now an error instead of being guessed as cp1251.
+- `--only` / `--exclude` did not control the numbers of other documents
+  (`довідку № 4567`): `--only surname` still masked them, and
+  `--exclude order_number` did not stop it. They now follow `order_number`,
+  as with `enable_orders` in the configuration.
+- Unmasking chose between JSON and text by the masked file's extension, while
+  masking chose by the input's extension. Masked text saved as `out.json`
+  could not be unmasked, and JSON saved as `.txt` came back only in part. The
+  mapping now records the input format (`input_format`); older mappings
+  still use the extension.
+- A JSON input with a UTF-8 BOM was rejected; it is now read like a text
+  file with a BOM.
+- `unmask` could not read files over 100 MB even when
+  `validation.max_input_size_mb` allowed them, while `mask` did read them.
+  Both now apply the same limit.
+- `--password` (or `--password-env`) with `security.encrypt_output: true` in
+  the configuration was rejected as if `--encrypt` were missing.
+- A password that is not valid UTF-8 (from a terminal in another encoding)
+  failed only when the mapping was being encrypted. By then the masked file
+  was already written, and its mapping was not. Such a password is now
+  rejected before any file is written.
+- `-o` pointing to a hard link of the input, with `--force`, overwrote the
+  original document; it is now refused like `-o` equal to the input.
+- A mapping entry with an empty mask made unmasking insert the original
+  value between all characters of the text; such a mapping is now rejected.
+- With `--re-mask 50`, the report said 50 passes, although the run was capped
+  at `remask.max_passes` (10). It now shows the passes actually made.
+- Tracebacks replaced with error messages (exit code 1): a damaged header of a
+  format-2 `.enc` file, and `--init-config -c` with a missing folder or a
+  folder as the target.
+- `unmask` now exits with 2 (wrong usage), as `mask` does, when
+  `--password-env` names an unset variable, when no `--map` is given for a
+  file that is not named `output_*`, and when `--to-version` is out of range
+  or used with a mapping that is not a chain. Before, it exited with 1.
 
 ### Changed — masks
 - Naval ranks get naval masks: `капітан 1-го рангу` and `капітан-лейтенант`

config.yaml

@@ -244,8 +244,10 @@ validation:
   # Дозволити скорочене по батькові (І.І.)
   allow_abbreviated_patronymic: true
 
-  # Обмеження довжини імен
-  max_name_length: 50
+  # Обмеження довжини імен (max_name_length: null — без обмеження, як без
+  # конфігурації; до 3.1.21 тут стояло 50, і довше подвійне прізвище з
+  # цим файлом лишалось відкритим)
+  max_name_length: null
   min_name_length: 3
 
 # --------------------------------------------------------------------------

config_example.py

@@ -107,7 +107,7 @@ class ValidationConfig:
     validate_rank_dictionary: bool = True  # лише True
     strict_pib_format: bool = False
     allow_abbreviated_patronymic: bool = True
-    max_name_length: int = 50
+    max_name_length: Optional[int] = None  # None — без обмеження
     min_name_length: int = 3
 
 

config_example.yaml

@@ -240,8 +240,10 @@ validation:
   # Дозволити скорочене по батькові (І.І.)
   allow_abbreviated_patronymic: true
 
-  # Обмеження довжини імен
-  max_name_length: 50
+  # Обмеження довжини імен (max_name_length: null — без обмеження, як без
+  # конфігурації; до 3.1.21 тут стояло 50, і довше подвійне прізвище з
+  # цим файлом лишалось відкритим)
+  max_name_length: null
   min_name_length: 3
 
 # --------------------------------------------------------------------------

config_local.example.yaml

@@ -16,14 +16,17 @@
 # ==========================================================================
 
 # Приклад: власні позивні маскуються завжди, місцеві абревіатури й назви —
-# ніколи. Списки exclusions ДОДАЮТЬСЯ до переліків dictionaries з config.yaml.
+# ніколи. Списки exclusions ДОДАЮТЬСЯ до переліків dictionaries з config.yaml,
+# але ЗАМІНЮЮТЬ однойменні списки exclusions з config.yaml (повторіть тут
+# спільні записи). null — лишити список із config.yaml (до 3.1.21 тут стояло
+# [], і скопійований шаблон стирав спільні always_mask, phrases …).
 exclusions:
-  abbreviations: []          # напр. [ТРО, ОК]
-  words: []                  # напр. [Рада, Раді]
-  phrases: []                # напр. ["Верховн* Рад*"]
-  legal_acts: []             # напр. ["розпорядженн* президента"]
-  always_mask: []            # напр. ["Грім", "Сокіл"]
-  remove: []                 # прибрати запис із dictionaries
+  abbreviations: null        # напр. [ТРО, ОК]
+  words: null                # напр. [Рада, Раді]
+  phrases: null              # напр. ["Верховн* Рад*"]
+  legal_acts: null           # напр. ["розпорядженн* президента"]
+  always_mask: null          # напр. ["Грім", "Сокіл"]
+  remove: null               # прибрати запис із dictionaries
 
 # Приклад: інші значення спільних ключів (розкоментуйте потрібне)
 # system:

datamasking/_jsontext.py

@@ -0,0 +1,57 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+"""
+Рядкові значення JSON як один текст (v3.1.21).
+
+Номери входжень масок (instance tracking) при маскуванні JSON спільні для
+всього документа, а unmask до 3.1.21 обробляв кожен рядок окремо, з
+лічильниками від нуля: якщо два різні оригінали в різних рядках діставали
+одну маску («№ 2» і «№ 6» → «№ 4»), другий відновлювався першим. Перевірка
+колізій маски з відкритим текстом теж бачила лише свій рядок. Тому і mask,
+і unmask обробляють усі рядки разом — через роздільник, якого немає в
+жодному з них (окремий рядок із символу приватної зони Unicode: не літера,
+не цифра, не пробіл, тож жоден шаблон його не перетинає).
+
+Ключі словників не маскуються й сюди не входять. Лише stdlib — модуль
+імпортують і mask, і unmask.
+"""
+from typing import Any, Iterator, List, Optional
+
+
+def collect_strings(data: Any) -> List[str]:
+    """Рядкові значення (не ключі) у порядку обходу dict/list."""
+    out: List[str] = []
+
+    def walk(node: Any) -> None:
+        if isinstance(node, dict):
+            for value in node.values():
+                walk(value)
+        elif isinstance(node, list):
+            for item in node:
+                walk(item)
+        elif isinstance(node, str):
+            out.append(node)
+
+    walk(data)
+    return out
+
+
+def replace_strings(data: Any, values: Iterator[str]) -> Any:
+    """Копія *data*, де рядкові значення по черзі замінено на *values*."""
+    if isinstance(data, dict):
+        return {key: replace_strings(value, values) for key, value in data.items()}
+    if isinstance(data, list):
+        return [replace_strings(item, values) for item in data]
+    if isinstance(data, str):
+        return next(values)
+    return data
+
+
+def separator(strings: List[str]) -> Optional[str]:
+    """Роздільник, якого немає в жодному рядку (None — якщо всі символи
+    приватної зони вже зайняті)."""
+    for code in range(0xE000, 0xF900):
+        ch = chr(code)
+        if not any(ch in s for s in strings):
+            return "\n" + ch + "\n"
+    return None

datamasking/_textio.py

@@ -15,6 +15,9 @@
 from typing import Iterable, List, Optional, Tuple
 
 DEFAULT_ENCODING = "utf-8"
+# UTF-32 — перед UTF-16: BOM UTF-32-LE починається з BOM UTF-16-LE
+_BOMS = ((codecs.BOM_UTF32_LE, "utf-32-le"), (codecs.BOM_UTF32_BE, "utf-32-be"),
+         (codecs.BOM_UTF16_LE, "utf-16-le"), (codecs.BOM_UTF16_BE, "utf-16-be"))
 DEFAULT_ALLOWED = ("utf-8", "cp1251", "latin-1")
 
 
@@ -56,15 +59,32 @@ def read_text(path: Path, encoding: str = DEFAULT_ENCODING,
     Переноси рядків не перетворюються (як open(..., newline='')).
     """
     raw = Path(path).read_bytes()
+    if encoding == "auto":
+        # BOM однозначно називає кодування. До 3.1.21 UTF-16 «декодувався»
+        # як cp1251 (однобайтові кодування приймають будь-які байти), і
+        # документ виходив незамаскованим з кодом 0. Кодек із явним порядком
+        # байтів лишає BOM у тексті символом U+FEFF — і вихід пишеться так само
+        for bom, enc in _BOMS:
+            if raw.startswith(bom):
+                return raw.decode(enc), enc
     candidates = list(allowed) if encoding == "auto" else [encoding]
     last_error: Exception = UnicodeDecodeError(DEFAULT_ENCODING, b"", 0, 1, "no encodings to try")
     for enc in candidates:
         try:
-            return raw.decode(enc), normalize_encoding(enc)
+            text = raw.decode(enc)
         except UnicodeDecodeError as e:
             last_error = e
+            continue
+        if encoding == "auto" and "\x00" in text:
+            # Символи NUL у «тексті» — це UTF-16/32 без BOM, а не cp1251
+            pos = text.index("\x00")
+            last_error = UnicodeDecodeError(enc, raw, pos, pos + 1,
+                                            "NUL character (UTF-16 or UTF-32 without a BOM?)")
+            continue
+        return text, normalize_encoding(enc)
     if encoding == "auto":
         raise UnicodeDecodeError(
             getattr(last_error, "encoding", "auto"), raw, 0, 1,
-            f"none of the allowed encodings ({', '.join(candidates)}) could decode the file")
+            f"none of the allowed encodings ({', '.join(candidates)}) could decode the file "
+            f"({getattr(last_error, 'reason', last_error)})")
     raise last_error