A record has a place in the archive, and the place can only ever narrow it (#202) by ChelseaKR · Pull Request #224 · ChelseaKR/ledger

ChelseaKR · 2026-09-11T03:30:04Z

Closes #202.

ledger described items and nothing above them. It now has an arrangement: a
collection, a series under it, and a record's placement in exactly one of them —
with a scope-and-content note, extent, dates, and two access policies. Browse,
search, EAD, OAI-PMH, the exports and the CLI all learn the hierarchy.

This is a draft on purpose. It is a schema change to a published contract and
it changes what several surfaces publish. The sweep merges any non-draft green PR
in this repo within minutes, so marking it ready is the merge decision.

It is also independent of #222 (physical holdings, also a draft). #222 adds no
container and no placement; this adds no holding kind. ADR 0020 rather than
0019, so the numbers do not collide on merge. The two will conflict textually in
three places — the trailing field lists of Record and DisclosedRecord, and
disclose's constructor call — and each is an additive one-line resolution.

The two decisions that had to be made together

Adding arrangement to an archive for at-risk contributors raises two questions,
and answering either one alone produces something worse than no arrangement.

1. How does a container's policy combine with a record's? As a logical AND
over the root-to-parent chain, not as a comparison between policies:

visible(record) == is_visible(record's own policy)
                   AND is_visible(ancestor 1's records_policy)
                   AND is_visible(ancestor 2's records_policy)

An AND has no ordering over AccessPolicy to get wrong. There is no lattice, no
table of which level dominates which, and no cell in which a broad container
makes a narrow record more visible — the property holds because of the shape of
the expression. All 144 cells of (6 record policies × 8 container ceilings ×
3 viewers) are asserted against a hand-written truth table, including the case
#202 names by name: a container broader than the record it holds. The table is
itself checked against is_visible, so it cannot drift into comparing two wrong
answers.

2. Is a container's existence disclosable? Yes, and separately from what it
holds. A collection titled "2019 raid testimony, deposited by Casa Abierta"
outs its depositor by aggregation even when every record inside it is sealed. So
a container carries policy over its own description and records_policy as
the ceiling over what it holds, and they are independent. A public flyer
filed in a steward-only collection is disclosed with no placement at all —
byte-identical, on every anonymous surface, to the same flyer filed nowhere.

Everything fails closed, and says so

A dangling placement, a corrupt parent link, a chain deeper than the two-level
vocabulary, and a read path that was never given the arrangement all deny the
record — to stewards too. A ceiling that cannot be applied is a ceiling of
unknown height. The last of those is what makes it impossible for this change to
be half-shipped: a surface that has not been taught about arrangement serves
nothing rather than serving it unclamped.

Failing closed is silent by construction, so ledger arrange check names every
record the resolver is hiding for that reason and exits non-zero.

Nothing is migrated

#202's third "decide first" item offered an implicit "unarranged" container for
every existing record, and called the alternative "every read path carries a null
branch". Measured: the null branch is in one function (arrangement_permits
short-circuits on placement is None), because no read path reads placement
for itself. Placement stays optional, and serialize_record omits the property
entirely when there is none — so every manifest written before this change
serialises to exactly the bytes it always did, and no committed manifest, bag
manifest or tag-file digest moves. The real objection to the other answer: giving
every record a parent gives every record a new way to be clamped.

What it adds

Surface	Change
`models.py` / `arrangement.py`	`ArchivalContainer`, `ContainerLevel`, `PlacementStep`, `DisclosedContainer`, `Record.placement`; the graph, its shape rules, its store
`access/policy.py`	`arrangement_permits`, `container_is_visible`, `visible_placement`, `disclose_container`; `is_listable`/`disclose` take an arrangement
CLI	`ledger arrange describe\|place\|list\|check`, `ledger ingest --collection`
Server	`/collections`, `/collection/{id}`, `/collection/{id}/ead.xml`; a `collection` facet composing with search and the Dublin Core facets; a "Part of" breadcrumb
EAD	real `<c01 level="collection">` / `<c02 level="series">` / item nesting and an `archdesc` scope note
OAI-PMH	`ListSets`, a `<setSpec>` per visible level, `&set=` filtering
METS / DC	a `logical` structMap; DCMI `isPartOf` as `dc:relation`, inside the canonical `relation` slot
CSV / print / drive	a `collection` column (appended, so position-indexed consumers are unaffected); a plain-text "Part of" line in both offline formats
i18n	15 strings in en/es/fr/ar
Schema	`container.schema.json` (new); `placement` added to `record.schema.json`

EAD had no caller from any read path before this — the module was exported and
unreachable, emitting the flat <c01 level="item"> list its own docstring
described as a hierarchy. /collection/{id}/ead.xml is its first.

Evidence

make verify green on the pushed head (fbaf927), run in this checkout:
2028 passed, verify: all gates green, exit 0. The log was grepped for
FAILED/Traceback/make: ***/would be reformatted/is stale — zero hits —
rather than read off the exit code.
27 of 27 negative controls fired, re-measured on that same head. Table below.
access/policy.py and arrangement.py are at 100% statement and branch;
all 8 per-module coverage floors pass (tools/check_coverage_floors.py, exit 0).
2028 tests, up from 1929 on main.

The anonymous differentials

Three of them, each comparing two live servers byte for byte across 22
anonymous routes (normalized only for the ephemeral port and for timestamps,
exactly as tests/test_sealed_existence_leaks.py normalizes):

two archives that differ only in…	routes that differ
the same public flyer filed in a hidden collection vs. filed nowhere	0
a public collection holding three community-only records vs. holding none	0
a hidden container page vs. an absent one	same 404, same body

Each has a floor so it cannot pass vacuously: a steward does see the container,
the placement and the records, so an archive where describe_container silently
did nothing would fail.

Negative controls

Sabotage, assert the mutation landed (git hash-object moved and the
signature is inside the region under test), run a named subset, restore, assert
the blob is back. The harness refuses a dirty tree at the start rather than
restoring one, clears __pycache__ around every mutation, and refuses to report
a red set at all if pytest printed "no tests ran".

#	sabotage	tests red
C1	`arrangement_permits` always returns True	14
C2	the AND becomes an OR	1
C3	an unresolvable chain permits	3
C4	a caller with no arrangement is served the record unclamped	1
C5	the record's own policy stops being consulted	3
C6	container visibility stops narrowing down the chain	2
C7	a hidden container's name is disclosed on the record	5
C8	`disclose_container` stops refusing	8
C9	an inherited scope note is presented as the container's own	2
C9b	a container with no note inherits nothing	2
C10	`chain` walks past a corrupt collection-with-a-parent	1
C11	`chain` drops its depth/cycle bound	1
C12	`chain` returns `()` instead of `None` for an unknown id	5
C13	an unknown policy is defaulted instead of refused	1
C14	`validate_container` accepts a series under a series	1
C15	a traversing container id is accepted	8
C16	`Archive.browse` stops passing the arrangement	14
C17	ingest stops refusing a missing container	1
C18	`serialize_record` writes `placement: null` instead of omitting it	1
C19	the collection facet stops filtering	1
C20	the container page lists records disclosed to a steward	1
C21	an empty placement still prints a "Part of:" line	1
C22	`/collection/{id}` 404s only for an absent container, not a hidden one	3
C23	OAI `ListSets` is built from every container, not the disclosed set	1
C24	EAD nests by the raw placement rather than the rendered containers	2
C25	`isPartOf` is emitted outside the canonical `relation` slot	1
C26	the CSV names the outermost container instead of the deepest	1

C2 reddening exactly one test is the interesting row. The 144-cell table
uses one container, where any and all agree; only
test_two_levels_of_container_both_apply_and_the_narrower_one_wins distinguishes
them. An exhaustive table over one level is not a test of composition.

Two controls did not fire on the first run, and both were real gaps — the
container-page comparison was f(x) vs f(x) (the renderer is never handed the
withheld records, which is the actual safety property, so it is asserted
directly now), and nothing asserted that a record with no visible placement
renders no breadcrumb rather than an empty "Part of:" label. Both are tests in
this PR; the table above is the re-run.

Three things found on the way, fixed here

tests/test_sealed_existence_leaks.py's own completeness check had stopped
checking. test_the_route_list_covers_every_anonymous_get recovered routes
with a regex over do_GET's source text; Remove code-quality waivers and raise published-library coverage floor #83 moved them into route tables as
data, so it has matched zero routes ever since — measured — and its set
difference was against an empty set. Green, in the one file whose inventory is
a disclosure control: any route added since Remove code-quality waivers and raise published-library coverage floor #83 could have escaped the
differential sweep. It reads the tables now, under a floor.
Two unreachable guards in the new code, found by the 100% floor on
access/policy.py. An unreachable guard reads like protection and proves
nothing; both are removed and the property each implied is asserted by a test.
/collections and /collection/{id} arrive already covered by the static
accessibility gate rather than as two new documented gaps, and the sample
archive it renders now carries a real collection and series so both pages are
checked populated — a list/table equivalence check over two empty views
examines nothing.

What this deliberately does not do

Give records an arrangement: collections and series with inherited description and narrowing policy #202's own example is not expressible as written. "This whole box is
community-only until 2030, except the three flyers already public" needs an
override, and an override is exactly the hole the AND closes. Those flyers sit
outside the container, or the container is public and the other 397 records are
narrowed individually. Recorded in ADR 0020 rather than worked around.
/contribute offers no container picker. Which collections exist is
policy-gated; a public form cannot list them without answering the question the
container policy exists to refuse. A steward files a submission after review.
Accession is deferred, which is Give records an arrangement: collections and series with inherited description and narrowing policy #202's fourth "decide first" item. An
accession carries a depositor, and that is the one thing arrangement must not.
Nothing here forecloses it: a future entity can reference a collection by id
without either schema moving.
/status's pre-existing emptiness oracle is not touched. Its anonymous
headline comes from a sweep over every bag, so it tells an empty archive apart
from one whose records are all invisible. That arrived with An empty archive is not a verified one, and /status now says so in the reader's language #218, is recorded
as the owner's call, and Give records an arrangement: collections and series with inherited description and narrowing policy #202 does not widen what it distinguishes — it adds
one more way for a record to be invisible while still counting as "not empty".
It is pinned by a named test rather than hidden behind a silent exclusion in
the differentials, so it goes red when the owner closes it.

Prepared with AI assistance; reviewed before submission.

#202 says the policy resolver "should land first and alone, because every other surface depends on the answer being right". This is that layer: the entities, the graph over them, and the one decision they add to `ledger.access.policy`. No read path consults it yet. A record gets `placement`: one container id, or none. A container carries two policies, because it is two things at once — `policy` governs its own description (a collection titled "2019 raid testimony, deposited by Casa Abierta" outs by aggregation even when every record in it is sealed), and `records_policy` is the ceiling over what it holds. The resolver is a logical AND over the root-to-parent chain, not a comparison between policies. There is no ordering over AccessPolicy to get wrong, and no cell where a broad container widens a narrow record — 144 of them are asserted, including the sentinel the issue names. Everything fails closed. A dangling placement, a corrupt parent link, a chain deeper than the two-level vocabulary, and a read path that was never given the arrangement at all: each denies the record, to stewards too. A ceiling that cannot be applied is not assumed away, and a surface that has not been taught about arrangement serves nothing rather than serving it unclamped. Nothing is migrated. An unarranged record has an empty chain, an empty AND permits, and `serialize_record` omits `placement` entirely when there is none — so every manifest written before this change serialises to exactly the bytes it always did.

…202) `containers/` beside `records/`, one manifest per collection or series, with a published schema of its own. `Archive.browse` and `Archive.disclose` now load the arrangement and pass it to the resolver, so what a viewer is shown is judged against the ceiling that is on disk right now — re-describe a container and 400 records narrow with it, and no record's own manifest is rewritten. The write path refuses what the read path would otherwise hide silently. An ingest or a `place` into a container that is absent, or whose parent chain does not resolve, is refused at the door: a record nobody could ever see is not stored while reporting success. `ledger arrange check` is the operable half — it names every record the resolver is denying to everyone and exits non-zero. `ledger arrange describe|place|list|check` and `ledger ingest --collection`. Both container policies default to the narrowest thing that still lets the container exist, because a container's title can out a depositor. Placement is an accountable change: filing a record writes a PREMIS `arrangement` event to its own log, describing a container writes one to `logs/arrangement.premis.json`. Neither copies a title or a scope note into a log — those are the disclosable material the container's own policy governs.

The read surfaces. A `collection` facet beside subject/type/language that composes with them and with search; `/collections` and `/collection/{id}`, each with the list and table equivalents the accessibility gate requires; a "Part of" breadcrumb on the record page. EAD finally emits the hierarchy its own docstring has described since it was written — `<c01 level="collection">`, `<c02 level="series">`, items under whichever they are filed in, and an `archdesc` scope note. Pass no arrangement and the output is byte-identical to the flat form, which is what keeps every existing caller and every committed expectation intact. `/collection/{id}/ead.xml` is the module's first caller from a read path; before this it was exported and unreachable. OAI-PMH gains `ListSets`, a `<setSpec>` per visible level on every header, and `&set=` filtering. An archive with no arrangement and an archive whose every collection is sealed from the harvester both answer `noSetHierarchy`, and an unknown set and an empty one are both `noRecordsMatch` — a harvester cannot establish that a collection exists by probing. Nothing new is visible to an outsider. Every surface reads the *disclosed* placement chain, so a record filed in a container this viewer may not describe renders, facets, harvests and breadcrumbs exactly like an unarranged one. The container page's empty state is one sentence for "holds nothing" and "everything inside is withheld from you", deliberately. Both new pages are covered by the static accessibility gate from this commit rather than added to the documented gap list, and the sample archive it renders now carries a real collection and series so the two views are checked populated.

METS gains a `logical` structMap naming the units an item sits in, and DCMI `isPartOf` as `dc:relation` values inside the `relation` slot so the canonical element order still holds. OAI keeps expressing membership through `setSpec`, its own native mechanism, rather than repeating it in `oai_dc` — the same fact in each format's own vocabulary. The CSV gains a `collection` column, appended rather than inserted so a consumer reading by position is unaffected. The print booklet and the courier drive state it as plain text, because both are read where there is no network to follow a link. Every one of them reads the *disclosed* placement chain, so a record filed in a container the viewer may not describe exports exactly like an unarranged one: an empty CSV cell, no `dc:relation`, no logical structMap, no "Part of" line. 15 new interface strings in en/es/fr/ar. The catalogs are machine-drafted and unreviewed like every other string in them, and every page serving them already carries the banner that says so. Three French forms are byte-identical to the English and are recorded in `identical_by_design.json` with a reason, beside the `Date`/`Type`/`Types` entries they sit next to. Deliberately not done: `/contribute` offers no container picker. Which collections exist is itself policy-gated, so a public form cannot list them without answering the question the container policy exists to refuse. A steward files a submission with `ledger arrange place` after review.

…king (#202) `test_the_route_list_covers_every_anonymous_get` recovered routes with `path == "(/[^"]*)"` over `do_GET`'s source text. #83 moved the routes into `_GET_PAGES` / `_GET_QUERY_PAGES` as data, so that regex has matched **zero** routes since — measured today: 0 — and the set difference it computes was against an empty set. Green, and checking nothing, in the one file whose inventory is a disclosure control: a route added at any point since #83 could have escaped the differential sweep entirely. It reads the tables as data now, with a floor that fails if it recovers implausibly few. `/collections` and the `/collection/` family are added to the sweep, which is what surfaced it.

Eighteen tests over browse, the container pages, EAD, OAI-PMH and the exports. Half of them are about the feature working; the other half are #188's rule applied to this axis — for every surface a non-steward is shown, is it computed over a larger set than they can list? The load-bearing ones compare two live archives byte for byte across 22 anonymous routes: - the same public flyer, once filed in a collection whose title would out its depositor and once filed nowhere: no route differs; - a public collection holding three community-only records against the same collection holding none: no route differs; - a hidden container page against an absent one: the same 404, same body once the id the caller typed is substituted back. And a floor under all three: a steward *does* see the difference, so an archive where `describe_container` silently did nothing could not pass them. One route needed explaining rather than excluding. `/status` tells an empty archive apart from one whose records are all invisible, because its headline is derived from a sweep over every bag rather than over what the caller may list. That is the pre-existing oracle recorded as OWNER-DECISIONS §H item 4 — it arrived with #218, #202 does not widen what it distinguishes, and closing it is a published-contract call. It is pinned by a named test rather than hidden behind a silent exclusion, so it goes red when the owner closes it.

ADR 0020 records the two decisions that had to be made together, because either one alone produces something worse than no arrangement: the resolution is a logical AND over the chain (so there is no ordering over AccessPolicy to get wrong, and no cell where a broad container widens a narrow record), and a container's existence is sealable separately from what it holds. It also records what #202 asked and this does not do. Its example — "this whole box is community-only until 2030, except the three flyers already public" — is not expressible as written, because making the exception expressible means an override and an override is the hole the AND closes. Its third "decide first" item is answered the other way: placement stays optional, and the premise that this "means every read path carries a null branch" is wrong — the branch is in one function, because no read path reads `placement` for itself. Its fourth is deferred with a reason: an accession carries a depositor, which is the one thing arrangement must not. `DATA-GOVERNANCE.md` classifies `containers/` as its own store and names the reason a container's title is sensitive even when it identifies nobody. `ARCHITECTURE.md` gains the module and the three new functions in the safety package. `ADR 0020` rather than 0019, so the number does not collide with the in-flight #188 branch. Numbering note: ADR 0019 is claimed by #222 (physical holdings); the two changes are independent and neither pre-empts the other.

#202) `access/policy.py` has a 100% per-module coverage floor and this change put it at 98.37%. Both misses were branches nothing can reach, which is the useful thing the floor found: an unreachable guard reads like protection and proves nothing. - The scope-note inheritance re-checked whether an ancestor was visible. It cannot be invisible: `container_is_visible` ANDs down the chain, so reaching that line means every ancestor already passed. It also walked the chain, which carries a second iteration a two-level vocabulary cannot produce. Both gone; the walk is an index, and two tests carry what they implied — a hidden collection hides its series (over two sealed shapes and two viewers), and the index is valid only while `MAX_CHAIN_DEPTH == 2`, which fails if the vocabulary grows. - `Arrangement.chain` checked that the top of a resolved chain is a collection. The two shape checks above it already guarantee that: the loop exits only on `current is None`, and only a parentless collection sets it. Three tests added for branches that *are* reachable and had no fixture: a dangling placement names no container, a parentless series on disk resolves to no chain, and a series with no note under a collection with no note inherits nothing. `access/policy.py` and `arrangement.py` are both at 100% statement and branch; all 8 module floors pass.

The first 27 negative controls found two tests that could not fail. `test_a_container_page_says_the_same_thing_when_empty_and_when_withheld` compared `f(x)` with `f(x)`: the renderer is never handed the withheld records, so its inputs were identical in both archives and the page comparison was a tautology. That absence is the actual safety property, so it is asserted directly now — the disclosed inputs are equal, and the withheld side's record list is empty — with a pointer to the end-to-end version over a live server. Nothing asserted that a record with no visible placement renders no breadcrumb. An empty "Part of:" line would tell a reader the record is filed somewhere they may not see, which is the sentence the gating exists to avoid; a control that removed the empty-chain guard stayed green. It is a test now, over both the hidden-container and unarranged cases, with a steward floor so it is about gating rather than about a breadcrumb nobody renders.

Rendering the pages against a real archive rather than a fixture: - `?collection=casa-abierta` headed the browse page "Collection: casa-abierta". Every Dublin Core facet's value is its own label; this one's is an identifier. It resolves through the same facet list the sidebar renders, so the heading can only ever name a container this viewer may already describe — a lookup straight into the arrangement could name one they may not. - A collection's series list repeated the collection's own scope note under every child that had inherited it. On the series' own page that note sits beside the sentence saying it is inherited; in a list of siblings it reads as each one's own description, which is noise and a small dishonesty. Both have tests. `make verify` green: 2028 passed, exit 0.

ChelseaKR added 11 commits September 10, 2026 19:19

style: ruff format the repaired route-coverage check (#202)

0f5acdd

Navigation Menu

A record has a place in the archive, and the place can only ever narrow it (#202) - #224

A record has a place in the archive, and the place can only ever narrow it (#202)#224
ChelseaKR wants to merge 11 commits into
mainfrom
feat/arrangement-collections-series-202

ChelseaKR commented Sep 11, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

ChelseaKR commented Sep 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

The two decisions that had to be made together

Everything fails closed, and says so

Nothing is migrated

What it adds

Evidence

The anonymous differentials

Negative controls

Three things found on the way, fixed here

What this deliberately does not do

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

ChelseaKR commented Sep 11, 2026 •

edited

Loading