Fix bugs from the third bug hunt: CSV, look-alike letters, ranks, old mappings, API

CHANGELOG.md

@@ -4,6 +4,102 @@ All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 
+## [Unreleased]
+
+### Fixed — data left unmasked
+- Tables exported from Excel as CSV (`Петренко Іван Петрович;сержант;1234567890`,
+  `Петренко;Іван;Петрович;сержант`): a `;` with no space after it now
+  separates cells like `|` does, so the name, the rank and the patronymic
+  in the last cell are masked. Before, the patronymic glued to the next cell
+  stayed open. A `;` followed by a space still ends a name, as in prose. A
+  comma glued to the next word (`Петрович,сержант`) is read as `, `.
+- A Latin letter that looks like a Cyrillic one inside a name (`Ярoслав`
+  with a Latin `o`, typical of text pasted from elsewhere) made the name
+  unrecognised. Its mask could then be the real name (`Ярослав`), and a
+  surname in an oblique case next to it (`Литвином`) stayed open. Names and
+  patronymics are now analysed with such letters read as Cyrillic; the
+  mapping keeps the original spelling.
+- Ranks followed by `запасу` or `резерву` (`майор запасу Петренко …`) and
+  ranks in the dative ending in `-ові` (`майорові`, `солдатові`,
+  `старшому лейтенантові`) were not recognised and stayed open next to a
+  masked name. The `-ові` form gets a mask that has the same form.
+- After a rank at the end of a line, a rank at the start of the next line
+  (`… капітаном` / `СТАРШИЙ ЛЕЙТЕНАНТ ПИЛИП'ЮК …`) had its first word masked
+  as a surname, also inside the rank's mask (`СТЕМИЙ СТЕМИЙ ПРАПОРЩИК`). A
+  rank of several words right after another rank is now the person's rank.
+
+### Fixed — unmasking did not restore the original
+- When two values got masks that differ only in letter case (a patronymic
+  first seen in a line in capitals, `ГОРДІЇВНИ`, and another one,
+  `Гордіївни`), one occurrence was not restored: masking numbered the
+  occurrences by the mask's case at its first use, while the text shows the
+  mask in the case of each occurrence. Occurrences are now numbered
+  regardless of case, and new mappings say so (`"instance_case":
+  "insensitive"`); mappings from earlier versions are unmasked as before.
+- A double surname that also appears in a line in capitals came back in
+  capitals in an ordinary line (`ДОРОШЕНКУ-ҐУДЗЮ` instead of
+  `Дорошенку-Ґудзю`); the case of each part is now taken from the text.
+  A double first name came back with a lower-case second part
+  (`Анна-марія`).
+- In a long order that mentions many ranks without names, a rank's mask could
+  equal one of those open ranks: only the neighbours two steps up or down
+  were tried. Unmasking then replaced the open rank (`доповіла капітану роти`
+  came back as `майору`). The nearest free rank of the hierarchy is now taken.
+  When every rank of the hierarchy is open in the document, the rank is left
+  as it is rather than masked with a rank that unmasking would confuse.
+
+### Fixed — older mapping files
+- A v2.0 mapping (entries without `instances`) restored nothing, with exit
+  code 0 and no warning. Such an entry now restores every occurrence of its
+  mask.
+- Ranks in v2.0 mappings were never restored; they now are, as for 2.1+.
+- A mapping without a `version` key was read as v1 and restored nothing; its
+  format is now taken from its structure.
+- v1 mappings (plain `"original": "mask"` pairs) were restored by replacing
+  one mask after another. A restored value that was itself another mask was
+  replaced again (`Сидоренко` → `Петренко` → …), and a mask was also found
+  inside longer words (`Петро` inside `Петрович`). They are now restored in
+  one pass, whole words only.
+
+### Fixed — masks
+- The case forms of one surname in a document now share one synthetic stem
+  (`Коваль`/`Коваля`/`Ковалем` → `Ковар`/`Коваря`/`Коварем`), as the
+  documentation promises. Surnames declined like nouns ending in a consonant
+  (Коваль, Ткач, Бондар, Мороз, Ґудзь) got a different stem in each case:
+  their case endings match those of surnames ending in `-а`/`-я`. A form now
+  takes the stem of another form already masked in the document. The `-ин`
+  and `-ін` families also cover `Литвином`, `Литвинові` and `Кузьміним`.
+  A bare surname ending in a consonant gets a mask ending in a consonant
+  where possible.
+- A surname mask could contain the apostrophe `ʼ` (U+02BC) from the name
+  dictionary. A document in cp1251 with such a mask then failed to save.
+  Surname masks now use the ASCII apostrophe, as name masks already did.
+
+### Fixed — Python API
+- `data_masking` (compatibility module): `MASK_NAMES = False` left surnames
+  and patronymics masked, and `MASK_DATES = False` left text dates masked;
+  they now switch the related flags too, as the configuration does.
+  `MASK_SURNAMES`, `MASK_PATRONYMICS`, `MASK_DATE_TEXT` and
+  `MASK_DOCUMENT_NUMBERS` can be read and set.
+  `apply_filter_to_globals(f, data_masking.__dict__)`, as its docstring
+  suggested, had no effect; it now sets the engine's flags.
+- `tools.mask_name_direct`, `mask_patronymic_direct` and `mask_value` for
+  names and patronymics assumed a male name unless told otherwise
+  (`Марія` → a male name). The gender now comes from the value, as in
+  `data-mask`. `mask_value("", "name")` no longer fails.
+- `generate_password(ensure_variety=True)` could leave out a character class
+  (about one short password in ten). A length below 1 is now an error instead
+  of an empty password, and the generator's `-l 0` / `-n 0` are rejected.
+- `MappingChain.get_chain_mapping(a, b)` forward was off by one pass
+  (`(1, 2)` mapped the original text, not pass 1, to pass 2).
+  `ChainUnmasker` restored nothing for a chain whose passes have no `"pass"`
+  number, or after `chain.passes = [...]`. `MappingChain.from_dict` no longer
+  changes the dictionary it was given.
+
+### Fixed — command line
+- `--only ,` (an empty list) enabled every type; an empty `--only` or
+  `--exclude` list is now an error (exit code 2).
+
 ## [3.1.21] - 2026-11
 
 ### Fixed — names left unmasked

data_masking.py

@@ -120,9 +120,19 @@
     "MASK_NAMES", "MASK_IPN", "MASK_PASSPORT", "MASK_MILITARY_ID",
     "MASK_RANKS", "MASK_BRIGADES", "MASK_UNITS", "MASK_ORDERS",
     "MASK_BR_NUMBERS", "MASK_DATES",
+    "MASK_SURNAMES", "MASK_PATRONYMICS", "MASK_DATE_TEXT", "MASK_DOCUMENT_NUMBERS",
     "DEBUG_MODE", "PRESERVE_CASE", "HASH_ALGORITHM",
 })
 
+# Загальний прапорець вмикає/вимикає й похідні — як follow() у CLI
+# (до 3.1.22 MASK_NAMES = False лишав прізвища й по батькові маскованими,
+# MASK_DATES = False — текстові дати, MASK_ORDERS = False — номери довідок)
+_FOLLOWERS = {
+    "MASK_NAMES": ("MASK_SURNAMES", "MASK_PATRONYMICS"),
+    "MASK_DATES": ("MASK_DATE_TEXT",),
+    "MASK_ORDERS": ("MASK_DOCUMENT_NUMBERS",),
+}
+
 
 def __getattr__(name):
     if name in _LIVE_FLAGS:
@@ -134,6 +144,8 @@ class _LiveFlagsModule(_ModuleType):
     def __setattr__(self, name, value):
         if name in _LIVE_FLAGS:
             setattr(_cfg, name, value)
+            for follower in _FOLLOWERS.get(name, ()):
+                setattr(_cfg, follower, value)
         else:
             super().__setattr__(name, value)
 

datamasking/extras/password_generator.py

@@ -158,6 +158,10 @@ def generate_password(
         pwd = generate_password(length=8, include_ascii_upper=False,
                                 include_ascii_lower=False, include_special=False)
     """
+    if isinstance(length, bool) or not isinstance(length, int) or length < 1:
+        # До 3.1.22 довжина 0 чи від'ємна мовчки давала порожній пароль
+        raise ValueError(f"password length must be a positive integer, got {length!r}")
+
     # Формуємо набір символів та список обов'язкових наборів
     charset = ""
     required_charsets: List[str] = []
@@ -200,19 +204,13 @@ def generate_password(
 
     # Гарантуємо наявність символів кожного типу (якщо довжина дозволяє)
     if ensure_variety:
-        positions_used: set = set()
-        for req_charset in required_charsets:
-            if len(positions_used) >= length:
-                break
-            # Знаходимо позицію, яка ще не використана
-            attempts = 0
-            while attempts < length * 2:
-                pos = secrets.randbelow(length)
-                if pos not in positions_used:
-                    password_list[pos] = secrets.choice(req_charset)
-                    positions_used.add(pos)
-                    break
-                attempts += 1
+        # Різні позиції для кожного обов'язкового набору — одразу, без
+        # повторних спроб (до 3.1.22 випадкові спроби з лімітом інколи не
+        # знаходили вільної позиції: ~9 % коротких паролів без одного класу)
+        k = min(len(required_charsets), length)
+        positions = secrets.SystemRandom().sample(range(length), k)
+        for pos, req_charset in zip(positions, required_charsets):
+            password_list[pos] = secrets.choice(req_charset)
 
     return ''.join(password_list)
 def generate_password_from_config(config: PasswordConfig) -> Optional[str]:
@@ -482,6 +480,11 @@ def main():
                         help='Тільки пароль, без додаткової інформації')
 
     args = parser.parse_args()
+    # Довжина й кількість — додатні (до 3.1.22 «-l 0» друкував порожній пароль)
+    if args.length < 1:
+        parser.error(f"--length must be a positive integer, got {args.length}")
+    if args.count < 1:
+        parser.error(f"--count must be a positive integer, got {args.count}")
 
     # Аналіз існуючого пароля
     if args.analyze:

datamasking/extras/re_mask.py

@@ -128,6 +128,8 @@ def passes(self) -> List[Dict]:
     @passes.setter
     def passes(self, value: List[Dict]) -> None:
         self._passes = value
+        # Інакше ChainUnmasker після chain.passes = [...] не бачив проходів
+        self._current_pass = len(value)
 
     @property
     def current_pass(self) -> int:
@@ -174,6 +176,11 @@ def get_pass(self, pass_number: int) -> Optional[Dict]:
         for p in self._passes:
             if p.get("pass") == pass_number:
                 return p
+        # Проходи без поля "pass" (ланцюг, зібраний вручну; його приймають і
+        # validate_mapping_schema, і unmask_chain) — за позицією. До 3.1.22
+        # ChainUnmasker для такого ланцюга мовчки нічого не відновлював
+        if 1 <= pass_number <= len(self._passes) and "pass" not in self._passes[pass_number - 1]:
+            return self._passes[pass_number - 1]
         return None
 
     def get_chain_mapping(self, from_pass: int, to_pass: int) -> Dict:
@@ -197,16 +204,12 @@ def get_chain_mapping(self, from_pass: int, to_pass: int) -> Dict:
         return self._build_reverse_chain(from_pass, to_pass)
 
     def _build_forward_chain(self, from_pass: int, to_pass: int) -> Dict:
-        """Compose forward mappings from from_pass to to_pass.
-        Chains original->masked relationships through intermediate passes."""
-        # Seed with the from_pass mapping
+        """Compose forward mappings from the text after pass *from_pass*
+        (0 = original) to the text after pass *to_pass*: passes
+        from_pass+1 .. to_pass, as the reverse chain counts them. Before
+        3.1.22 the from_pass mapping itself was included, so (1, 2) gave
+        original -> pass 2 instead of pass 1 -> pass 2."""
         combined: Dict[str, str] = {}
-        seed_data = self.get_pass(from_pass)
-        if seed_data is not None:
-            for _cat, pairs in seed_data.get("mappings", {}).items():
-                if isinstance(pairs, dict):
-                    combined.update(_flat_pairs(pairs))
-        # Chain through subsequent passes
         for step in range(from_pass + 1, to_pass + 1):
             step_data = self.get_pass(step)
             if step_data is None:
@@ -276,7 +279,8 @@ def from_dict(cls, data: Dict[str, Any]) -> "MappingChain":
         chain_id = data.get("chain_id", _generate_chain_id())
         chain = cls(chain_id=chain_id)
         chain._created_at = data.get("created_at", chain._created_at)
-        chain._passes = data.get("passes", [])
+        # Копія: add_pass не має змінювати словник, переданий викликачем
+        chain._passes = copy.deepcopy(data.get("passes", []))
         chain._current_pass = len(chain._passes)
         return chain
 

datamasking/extras/selective.py

@@ -306,6 +306,10 @@ def create_filter(
     """
     only_set = parse_type_list(only) if only else None
     exclude_set = parse_type_list(exclude) if exclude else None
+    # «--only ,» — порожній перелік; до 3.1.22 він означав «усі типи»
+    for label, raw, parsed in (("only", only, only_set), ("exclude", exclude, exclude_set)):
+        if raw is not None and str(raw).strip() and not parsed:
+            raise ValueError(f"--{label}: no masking type given in {raw!r}")
     return SelectiveFilter(only_types=only_set, exclude_types=exclude_set)
 
 
@@ -320,9 +324,16 @@ def apply_filter_to_globals(
 
     Args:
         selective_filter: The filter to apply.
-        globals_dict: The module globals dict (e.g. ``data_masking.__dict__``
-                      or the result of ``globals()``).
+        globals_dict: Where the flags live: ``vars(datamasking.masking.
+                      constants)`` (what the engine reads) or another dict.
+                      ``data_masking.__dict__`` (the compatibility module)
+                      is redirected to the engine's flags: writing there
+                      directly had no effect before 3.1.22 and left stale
+                      values that hid the live ones.
     """
+    if globals_dict.get("__name__") == "data_masking":
+        from datamasking.masking import constants as _cfg
+        globals_dict = vars(_cfg)
     # Collect which MASK_* keys should be True
     enabled_globals: Set[str] = set()
     for type_name in selective_filter.enabled_types:

datamasking/extras/tools.py

@@ -277,9 +277,9 @@ def get_next_instance(masked_value: str, instance_counters: Dict[str, int]) -> i
     Returns:
         The incremented instance number (starting from 1).
     """
-    instance_counters.setdefault(masked_value, 0)
-    instance_counters[masked_value] += 1
-    return instance_counters[masked_value]
+    # Без урахування регістру маски — як у рушії (masking/helpers.py)
+    from datamasking.masking.helpers import get_next_instance as _engine_next
+    return _engine_next(masked_value, instance_counters)
 
 
 def add_to_mapping(
@@ -327,6 +327,8 @@ def add_to_mapping(
     else:
         masked = masking_dict["mappings"][entity_type][original]["masked_as"]
 
+    from datamasking.masking.helpers import INSTANCE_CASE_INSENSITIVE, INSTANCE_CASE_KEY
+    masking_dict[INSTANCE_CASE_KEY] = INSTANCE_CASE_INSENSITIVE
     instance_num = get_next_instance(masked, instance_counters)
     masking_dict["mappings"][entity_type][original]["instances"].append(
         instance_num
@@ -520,7 +522,7 @@ def mask_name_direct(
     value: str,
     masking_dict: Dict,
     instance_counters: Dict,
-    gender: str = 'male',
+    gender: Optional[str] = None,
 ) -> str:
     """Mask a first name, preserving case and respecting gender.
 
@@ -532,7 +534,9 @@ def mask_name_direct(
         value: First name string.
         masking_dict: Shared masking dictionary (mutated).
         instance_counters: Instance tracking counters (mutated).
-        gender: ``'male'`` (default) or ``'female'``.
+        gender: ``'male'``, ``'female'`` or ``None`` (default: from the name
+                itself, as ``data-mask`` does; before 3.1.22 the default
+                was ``'male'``, and «Марія» got a male mask).
 
     Returns:
         Masked first name with case preserved.
@@ -546,7 +550,7 @@ def mask_patronymic_direct(
     value: str,
     masking_dict: Dict,
     instance_counters: Dict,
-    gender: str = 'male',
+    gender: Optional[str] = None,
 ) -> str:
     """Mask a patronymic, preserving case and respecting gender.
 
@@ -557,7 +561,8 @@ def mask_patronymic_direct(
         value: Patronymic string.
         masking_dict: Shared masking dictionary (mutated).
         instance_counters: Instance tracking counters (mutated).
-        gender: ``'male'`` (default) or ``'female'``.
+        gender: ``'male'``, ``'female'`` or ``None`` (default: from the
+                patronymic itself — «Іванівна» is female).
 
     Returns:
         Masked patronymic with case preserved.

datamasking/masking/constants.py

@@ -11,6 +11,7 @@
 from faker import Faker
 
 from datamasking.rank_data import (
+    RANK_DATIVE_OVI,
     RANK_DECLENSIONS,
     RANK_FEMININE_MAP,
     RANK_DECLENSIONS_FEMALE,
@@ -165,6 +166,9 @@ def set_faker_locale(locale: str) -> None:
 # жіноча форма маскується чоловічою і unmask повертає «солдата» замість
 # «солдатки» (v3.1.14). Задається рушієм на час обробки документа.
 FEMININE_RANK_BASES: frozenset = frozenset()
+# Звання, що в документі стоять у давальному на «-ові» (див. rank_data.
+# RANK_DATIVE_OVI): маска має теж мати цю форму. Задається рушієм
+OVI_RANK_BASES: frozenset = frozenset()
 # Базові звання, які в поточному документі записані з нерозривним пробілом,
 # табом чи кількома пробілами між словами: маска має ту саму кількість слів,
 # щоб роздільники перенеслись у неї й назад (після 3.1.20). Задається рушієм.