gen_bittings and check_bittings follow and note the key shape rules (D64, step 3)

README.md

@@ -19,8 +19,8 @@ Everything else runs without them. Python 3.11 or newer.
 
 | Command (from a checkout) | Installed as | What it does |
 | --- | --- | --- |
-| `./gen_bittings.py` | `sfic-gen-bittings` | Random bittings from a parity pattern, or with any parity if you give none, optionally staying away from existing bittings. |
-| `./check_bittings.py` | `sfic-check-bittings` | Quick check of a few `NAME=BITTING` values: format, parity, adjacent-cut limit, pairwise closeness. |
+| `./gen_bittings.py` | `sfic-gen-bittings` | Random bittings from a parity pattern, or with any parity if you give none, optionally staying away from existing bittings, and following the [key shape rules](#key-shape-rules). |
+| `./check_bittings.py` | `sfic-check-bittings` | Quick check of a few `NAME=BITTING` values: format, parity, adjacent-cut limit, pairwise closeness, with advice on the [key shape rules](#key-shape-rules). |
 | `./check_system.py` | `sfic-check-system` | Whole-scheme check of a system file: per-key rules, duplicates, closeness, operating-set sizes, cross-operation, residual risk. |
 | `./solve_system.py` | `sfic-solve-system` | Fills in the `null` bittings of a system file by random search plus hill climbing, then runs the full check. |
 | `./pin_system.py` | `sfic-pin-system` | Prints the pinning chart of every core of a system file that sets `pinning`, in the layout `check_charts` reads: the pins to put in each chamber, with `--draw` a drawing of the stacks and with `--pdf` one PDF to print. |
@@ -89,6 +89,42 @@ only estimate the chance that they cross-operate. The estimate assumes every
 undecoded unit key is a random valid bitting, uniformly chosen. If your real
 keys were chosen differently, the numbers do not apply.
 
+## Key shape rules
+
+Beside the rules a key must follow to cut and pin (cut depths, the adjacent-cut limit, parity),
+there are rules about what a key should look like. They keep the tools from handing you a key
+with equal neighbouring cuts, one depth over and over, or a master that is shallow from end
+to end. Without a parity pattern more than half of all valid bittings have a pair of equal
+adjacent cuts, so a generator that ignored this would give you one more often than not.
+
+| Rule | Flag | Default | A key breaks it when |
+| --- | --- | --- | --- |
+| `max_run` | `--max-run N` | 1 | more than `N` equal cuts in a row (1 means no equal neighbours) |
+| `max_same_depth` | `--max-same-depth N` | 3 | any depth appears more than `N` times |
+| `forbid_monotone` | `--allow-monotone` (turns it off) | on | the cuts never go down, or never go up, along a key of three or more cuts |
+| `master_min_span` | `--master-min-span N` | 6 | (masters only) the deepest cut minus the shallowest is under `N` |
+| `min_total_variation` | `--min-variation N` | off | the sum of the differences between adjacent cuts is under `N` |
+
+Give `off` instead of a number to turn a rule off. The span rule's default is lower if no key
+of the system could span 6 (few pins, or a small `max_step`). The rules are ours: the
+industry names rules of this kind, but the thresholds are not public.
+The reasons, the numbers behind the defaults and what each costs in key space are in
+[docs/designs/key-shape-rules.md](docs/designs/key-shape-rules.md).
+
+- `gen_bittings.py` only prints bittings that follow them. Add `--master` when you are
+  drawing master keys, so the span rule applies. If the search cannot find enough
+  bittings it says which shape rules turned draws down; it never relaxes one on its own.
+- `check_bittings.py` prints a `SHAPE` line for each rule a key breaks, with pin positions and
+  never the cuts. They are advice: they do not count as problems or change the exit status.
+  Name the masters with `--master NAME` (repeat the flag for more than one).
+- The solver and `check_system` do not use the rules yet.
+
+```bash
+./gen_bittings.py --pins 7 -n 3 --master          # three masters, no parity pattern
+./gen_bittings.py OOEOEOE -n 8 --max-run 2        # allow a pair of equal neighbours
+./check_bittings.py --master top top=0453037 area=6130254
+```
+
 ## System file format
 
 A JSON file. Fields (only `keys` and `cores` are needed for the basics):
@@ -327,7 +363,8 @@ printed, and the printed pins are the ones the checker's own pinner computed.
    ```
 
    Output is in generation order (not sorted), so taking the first one is as
-   random as any other. Spot-check candidates with
+   random as any other. Add `--master` for master keys (see
+   [Key shape rules](#key-shape-rules)). Spot-check candidates with
    `./check_bittings.py --pattern OOEOEOE name=1234567 other=7654321`.
 
 2. **Write the system file.** List keys, retired keys and the `cores` hierarchy.

docs/design.md

@@ -113,7 +113,8 @@ path (D1).
 The shared modules are `model.py` (the pure maths, built around `KeySpace`), `config.py`
 (loading and validating a system file), `pinning.py` (pinning system records and the
 pinner), `population.py` (who the undecoded unit keys might be, D55), `joint.py` (the
-exact construction of pinnable bittings, D60), `lock.py` (the simulated lock), `charts.py`
+exact construction of pinnable bittings, D60), `shape_args.py` (the command-line flags for
+the shape rules, D64), `lock.py` (the simulated lock), `charts.py`
 and `chartwriter.py` (reading and writing pinning charts) and `pdfwriter.py` (writing
 pages of text as a PDF). The tools are `gen_bittings`, `check_bittings`, `check_system`,
 `solve_system`, `pin_system`, `check_charts` and `scan_charts`, whose scanning stages live

docs/designs/key-shape-rules.md

@@ -150,8 +150,13 @@ what the tools count alone. The risk figures are unchanged.
   bound, and a set small enough to list is filtered whole. If the bound is reached with
   none, the group is reported as above.
 - **`gen_bittings`.** Applies the defaults, with `--max-run`, `--max-same-depth`,
-  `--min-variation` and `--master` to change them, and `--allow-monotone`.
-- **`check_bittings`, `check_system`.** The advisory `SHAPE` lines.
+  `--min-variation`, `--master-min-span` and `--allow-monotone` to change them and `--master`
+  to say the bittings are masters. A number, or `off`, turns a rule on or off; a flag that no
+  key of the key space could meet is a usage error. The flags are shared with
+  `check_bittings` (`shape_args.py`). When the search gives up, the message says which rules
+  turned draws down.
+- **`check_bittings`, `check_system`.** The advisory `SHAPE` lines. `check_bittings` takes
+  the same flags and names its masters with `--master NAME`, repeated.
 - **System file.** `shape` is optional; a file without it gets the defaults, so seeded
   solver output changes for existing files. That is deliberate and the log entry says so (D6
   asks for a stated reason). An unknown field inside `shape` is an error, not a warning,
@@ -205,6 +210,12 @@ shows seeded output unchanged for files that turn every rule off.
 
 ## What cannot be known yet
 
+- Whether the rules can be met together. The reachability bounds look at each rule alone and
+  ignore the parity pattern, so a combination can pass them and still have no key: 3 pins
+  with `--master` and the default `max_step` of 5 need a span of 6, which no key of three
+  cuts has unless it is monotone, which the default rules forbid. The tool then runs to its
+  attempt limit and says so, naming the rule that turned down most of the draws, and does
+  not relax anything. Three pins is not a real key, so the bounds were left simple.
 - How many pinnable candidates survive the rules in a system with many known keys: a unit
   master that must also take a decoded unit's neighbours has a thin pinnable set already,
   and the first run on such a file will say whether `master_min_span` = 6 is too strict.

sfic_solver/check_bittings.py

@@ -2,18 +2,22 @@
 """Sanity-check a set of bittings (7 pins unless --pins or --pattern says otherwise).
 
 Checks each bitting for format, parity (if --pattern is given) and the
-adjacent-cut limit, then lists every pair closer than --min-diff positions.
+adjacent-cut limit, then lists every pair closer than --min-diff positions. It also
+notes any bitting that breaks a shape rule (docs/designs/key-shape-rules.md): SHAPE
+lines are advice, positions only, and do not count as problems or change the exit status.
 
 Examples:
     ./check_bittings.py --pattern OOEOEOE key_a=5961634 key_b=7305496 key_c=5721276
     ./check_bittings.py --min-diff 5 key_a=5961634 key_b=7305496 new_master=7587672
+    ./check_bittings.py --master new_master a=1234567 new_master=7587672
 
 Exits with status 1 if anything is flagged.
 """
 import argparse
 import itertools
 
 from . import model
+from .shape_args import add_shape_arguments, shape_rules
 
 
 def parse_item(text, space):
@@ -37,6 +41,10 @@ def main(argv=None):
     ap.add_argument("--min-diff", type=int,
                     help="flag pairs differing in fewer positions than this "
                          "(default 5, or the pin count if smaller)")
+    ap.add_argument("--master", action="append", default=[], metavar="NAME",
+                    help="the name of a key that is a master, so the master span rule "
+                         "applies to it; repeat the flag for more than one")
+    add_shape_arguments(ap)
     args = ap.parse_args(argv)
 
     if args.pins is not None and args.pins < 1:
@@ -66,7 +74,14 @@ def main(argv=None):
     except ValueError as err:
         ap.error(str(err))
 
+    names = [name for name, _ in keys]
+    for master in args.master:
+        if master not in names:
+            ap.error(f"--master {master!r} is not one of the keys given")
+    rules = shape_rules(ap, args, space)
+
     problems = 0
+    notes = 0
     for name, cuts in keys:
         if pattern:
             bad = space.parity_bad(cuts)
@@ -77,6 +92,9 @@ def main(argv=None):
         if big:
             print(f"MACS    {name}: adjacent cuts too far apart after pin(s) {big}")
             problems += 1
+        for rule, pins_ in rules.violations(cuts, name in args.master):
+            print(f"SHAPE   {name}: {rules.describe(rule, pins_)}")
+            notes += 1
 
     pairs = sorted(((model.distance(a[1], b[1]), a[0], b[0])
                     for a, b in itertools.combinations(keys, 2)))
@@ -88,6 +106,8 @@ def main(argv=None):
     closest = pairs[0] if pairs else None
     if closest:
         print(f"closest pair: {closest[1]} vs {closest[2]} ({closest[0]} positions)")
+    if notes:
+        print(f"{notes} shape note(s): advice only, not counted as problems")
     print("OK" if not problems else f"{problems} problem(s) flagged")
     return 1 if problems else 0
 

sfic_solver/config.py

@@ -94,15 +94,9 @@ def _shape_rules(raw, space):
         rules = model.ShapeRules.for_space(space, **fields)
     except ValueError as err:
         raise ConfigError(f"shape: {err}") from None
-    if rules.master_min_span is not None and rules.master_min_span > space.widest_span:
-        raise ConfigError(f"shape: master_min_span is {rules.master_min_span} but no key can "
-                          f"span more than {space.widest_span} ({space.pins} pins, cuts 0 to "
-                          f"{space.depths - 1}, max_step is {space.max_step})")
-    if (rules.min_total_variation is not None
-            and rules.min_total_variation > space.most_variation):
-        raise ConfigError(f"shape: min_total_variation is {rules.min_total_variation} but no "
-                          f"key of {space.pins} pins can vary by more than "
-                          f"{space.most_variation} (max_step is {space.max_step})")
+    problem = rules.unmeetable(space)
+    if problem:
+        raise ConfigError(f"shape: {problem}")
     return rules
 
 

sfic_solver/gen_bittings.py

@@ -9,16 +9,25 @@
 keys can be pinned together depends on the core, so check_system and
 solve_system are where that is tested, not this command.
 
+Keys also follow the shape rules (docs/designs/key-shape-rules.md): by default no equal
+adjacent cuts, no depth used more than 3 times, no key whose cuts only go one way, and, with
+--master, a span of at least 6 between the deepest and shallowest cut. The flags under "key
+shape rules" change them; a number, or "off", for each.
+
 Examples:
     ./gen_bittings.py OOEOEOE
     ./gen_bittings.py OOEOEOE -n 20
     ./gen_bittings.py OOEOEOE -n 5 --avoid 5961634 7305496 --min-diff 4
     ./gen_bittings.py --pins 7 -n 5             # any parity
+    ./gen_bittings.py --pins 7 -n 3 --master    # masters must also span the depths
+    ./gen_bittings.py OOEOEOE --max-run 2       # allow a pair of equal neighbours
 """
 import argparse
+import collections
 import secrets
 
 from . import model
+from .shape_args import FLAGS, add_shape_arguments, shape_rules
 
 MAX_ATTEMPTS = 100_000
 
@@ -50,6 +59,25 @@ def generate(space):
     raise RuntimeError("no valid bitting found; is --max-step too small?")
 
 
+MOST_DRAWS = 0.8
+
+
+def give_up_message(rejected, draws):
+    """Why the search ended: the old advice, and which shape rules turned draws down, by the
+    flag that governs each. A rule that turned down most of the draws is named, since that
+    usually means it cannot be met together with the others."""
+    message = "could not find enough bittings; loosen the constraints"
+    if rejected:
+        counts = ", ".join(f"{FLAGS[name]} {count:,}" for name, count in rejected.most_common())
+        message += (f" (draws that broke a shape rule, by the flag that governs it: {counts}; "
+                    f"see the key shape rules flags in --help)")
+        name, count = rejected.most_common(1)[0]
+        if draws and count >= MOST_DRAWS * draws:
+            message += (f". {FLAGS[name]} turned down {count / draws:.0%} of the draws, so it "
+                        f"may not be possible with the other rules")
+    return message
+
+
 def differs_enough(cuts, avoid, min_diff):
     return all(sum(a != b for a, b in zip(cuts, old)) >= min_diff for old in avoid)
 
@@ -71,6 +99,9 @@ def main(argv=None):
     ap.add_argument("--min-diff", type=int,
                     help="min positions that must differ from each --avoid bitting "
                          "and from each other (default 3, or the pin count if smaller)")
+    ap.add_argument("--master", action="store_true",
+                    help="the bittings are master keys: the master span rule applies to them")
+    add_shape_arguments(ap)
     args = ap.parse_args(argv)
 
     if args.max_step < 1:
@@ -86,18 +117,24 @@ def main(argv=None):
         avoid = [parse_bitting(b, space) for b in args.avoid]
     except ValueError as err:
         ap.error(str(err))
+    rules = shape_rules(ap, args, space)
     if args.min_diff is None:
         args.min_diff = min(3, pins)
     elif args.min_diff > pins:
         ap.error(f"--min-diff {args.min_diff} is more than the {pins} pins")
 
     accepted = []
+    rejected = collections.Counter()
     attempts = 0
     while len(accepted) < args.count:
         attempts += 1
         if attempts > MAX_ATTEMPTS:
-            raise SystemExit("could not find enough bittings; loosen the constraints")
+            raise SystemExit(give_up_message(rejected, attempts - 1))
         cuts = generate(space)
+        broken = rules.violations(cuts, args.master)
+        if broken:
+            rejected.update(name for name, _ in broken)
+            continue
         # New bittings must also stay --min-diff away from each other, not just
         # from the --avoid list. Also rejects duplicates when min-diff >= 1.
         if cuts not in accepted and differs_enough(cuts, avoid + accepted, args.min_diff):

sfic_solver/model.py

@@ -264,6 +264,40 @@ def for_space(cls, space, **fields):
         fields.setdefault("master_min_span", span if span >= 1 else None)
         return cls(**fields)
 
+    def describe(self, rule, pins):
+        """A phrase for a broken rule and its pins (from `violations`), saying where and
+        never what the cuts are."""
+        where = f" at pin(s) {list(pins)}" if pins else ""
+        if rule == "max_run":
+            if self.max_run == 1:
+                return f"equal adjacent cuts{where}"
+            return f"more than {self.max_run} equal cuts in a row{where}"
+        if rule == "max_same_depth":
+            return f"a depth used more than {self.max_same_depth} times{where}"
+        if rule == "forbid_monotone":
+            return "the cuts only go one way along the key (never down, or never up)"
+        if rule == "master_min_span":
+            return (f"a master whose deepest and shallowest cuts differ by less than "
+                    f"{self.master_min_span}")
+        if rule == "min_total_variation":
+            return f"total variation under {self.min_total_variation}"
+        raise ValueError(f"unknown shape rule {rule!r}")
+
+    def unmeetable(self, space):
+        """A message saying why no key of the space could meet these rules, or None. The
+        bounds ignore the parity pattern, so a rule that some key could meet is never
+        reported, but one that none can may pass."""
+        if self.master_min_span is not None and self.master_min_span > space.widest_span:
+            return (f"master_min_span is {self.master_min_span} but no key can span more than "
+                    f"{space.widest_span} ({space.pins} pins, cuts 0 to {space.depths - 1}, "
+                    f"max_step is {space.max_step})")
+        if (self.min_total_variation is not None
+                and self.min_total_variation > space.most_variation):
+            return (f"min_total_variation is {self.min_total_variation} but no key of "
+                    f"{space.pins} pins can vary by more than {space.most_variation} "
+                    f"(max_step is {space.max_step})")
+        return None
+
     def violations(self, cuts, master=False):
         """The rules this bitting breaks, as (rule name, 1-based pins) in the order of
         SHAPE_RULES. The pins are those in an over-long run or of an over-used depth; the