Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions docs/reporting-release-notes.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,18 @@ Reads handle both forms. `migrate_inline_rows` moves legacy rows with verificati
rows do not reproduce its binding, and `retire_legacy_inline_rows` then renames the legacy table so older binaries fail
closed. See [row storage](reporting-row-storage.md).

## The per-object lock is kept only where it pairs

Retirement, the intent sweep and a commit that adopts an object its own intent recorded no longer take the per-object
advisory lock, and no longer hold a transaction or pooled connection across the provider delete. The lock stays in the
two places that pair with each other: the orphan collector's delete, which locks every binding's key for the object, and
a commit that adopts an object its intent did not record. The lock text is unchanged, so a Python deleter and a
JavaScript adopter still agree, and rolling upgrades between a version that locks everywhere and this one are safe (every
pair that needs exclusion keeps the lock on both sides). A commit now closes its intent with `RETURNING created_objects`
to know which adopted objects it recorded. No schema or wire change. See
[the object lock contract](reporting-row-storage.md#the-object-lock-contract)
([#1492](https://github.com/adcontextprotocol/adcp-client-python/issues/1492), adcp-client#3606).

## Opt-in orphaned-object collector, fenced against clones

`PgReportingLedgerStore(orphan_collection=ReportingOrphanCollectionOptionsV1(...))` and
Expand Down
32 changes: 31 additions & 1 deletion docs/reporting-row-storage.md
Original file line number Diff line number Diff line change
Expand Up @@ -327,6 +327,36 @@ stays as a small marker that makes reads answer as expired; the records are alre
whose horizon excludes periods past retention. Versioned buckets, soft delete and retention locks are allowed as
backstops, but a `pruned` period is not "erased" while they keep the old bytes.

## The object lock contract

The per-object advisory lock is kept only where nothing else orders the two sides, in the contract the JavaScript SDK
shares ([adcp-client#3606](https://github.com/adcontextprotocol/adcp-client/pull/3606)). The lock text is
`adcp-reporting-object:` + binding ID + `:` + object ref, passed to `pg_advisory_xact_lock(hashtextextended(text, 0))`,
so a Python deleter and a JavaScript adopter agree in one database.

1. **No lock:** retirement, the intent sweep and a commit that adopts an object its own intent recorded. Per object, one
statement checks that no `live` or `unavailable` row set outside the operation names the same binding, `object_ref`
and `native_version_ref`, and no `open` intent in the binding lists it in `created_objects` (both the current and the
pre-alignment `objectKey` / `nativeVersion` spellings are read); then the exact version is deleted. No transaction or
pooled connection is held across the provider delete.
2. **A commit** takes the account lock and closes its intent with `DELETE ... WHERE owner_token = ... AND state =
'open' RETURNING created_objects` (no row fails `STATE_UNAVAILABLE`). For each adopted object whose
`(object_ref, native_version_ref)` is **not** in the returned list it takes its binding's object lock, in object-ref
order. It then proves every adopted object with a version-pinned `stat` (a full read when the provider has none)
before inserting the chunk rows.
3. **The orphan collector** locks every binding's key for the object (configured plus registered, sorted by binding
ID), repeats the reference check across all bindings under those locks, then deletes. It is the only locked delete.
4. Lock order: the account lock, then object locks. Native versions must be write-unique (the conformance suite's
`write-unique-version`).

Why this is safe without the lock: the account lock orders commits, retirement phase 1, intent takeover and the sweep
claim; `pruning` is committed under it before any delete, and `pruning` row sets neither count as references nor can be
shared; an intent moves `open -> sweeping` against the commit's `DELETE ... WHERE state = 'open'`, in the same transaction
as the chunk insert, so a reader sees the open intent or the chunk rows, never neither; deletes name an exact native
version; and every writable key includes `{revision_key}`, so one row set alone can name it. **Precondition (documented,
not enforced):** two writable bindings must not share one bucket and prefix, because the reference check is per binding
outside the collector.

## Orphaned objects: the opt-in listing collector

The intent sweep deletes only what an intent lists, so it cannot see three cases: an upload whose `PUT` succeeded before
Expand Down Expand Up @@ -363,7 +393,7 @@ these hold:
old or new JSON keys) naming the key. Two bindings can share a bucket and prefix (a retired `read_only` binding kept
next to its successor), so the check is not scoped to the collecting binding.
4. Under the per-object advisory lock of **every** binding, taken in a fixed order so the delete is serialized with an
adoption through any of them, no `live` or `unavailable` row set references that exact version and no open intent
adoption through any of them (see [the object lock contract](#the-object-lock-contract)), no `live` or `unavailable` row set references that exact version and no open intent
created it. The provider then deletes that exact version.

`read_only` bindings are skipped unless `include_read_only`. `dry_run` deletes nothing and reports up to 1000
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,8 @@
* no row set that is not already ``pruned`` has a chunk at its key, and no write intent lists it,
at any version and under any binding (a pre-check that errs towards leaking, because two bindings
can share a bucket and prefix);
* under the per-object advisory lock of **every** binding, no ``live`` or ``unavailable`` row set
* under the per-object advisory lock of **every** binding (taken only here and by a commit that
adopts an object its own intent did not record), no ``live`` or ``unavailable`` row set
under any binding references that exact version and no open intent created it (the check every
other delete makes, widened to every binding), and the provider then deletes that exact version.

Expand Down Expand Up @@ -541,15 +542,13 @@ async def gate(binding_id: str, mode: Literal["claim", "check"]) -> _Gate:
since_fence_check = 0
since_fence_check += 1
try:
# The per-object lock and exact-version reference check every other
# delete makes, widened to every binding.
removed = await storage.delete_unreferenced(
store._pool,
summary.id,
[(entry.object_ref, entry.native_version_ref)],
any_binding=True,
# The only delete that takes the per-object lock: no record names this
# object, so a commit may be adopting it unrecorded. The reference check
# across every binding runs under the lock of every binding.
removed = await storage.delete_orphaned_object(
store._pool, summary.id, entry.object_ref, entry.native_version_ref
)
if removed > 0:
if removed:
outcome.deleted += 1
else:
outcome.referenced += 1
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -378,8 +378,9 @@ async def delete_period_objects(

An object that a live chunk of another obligation still references is kept: the default
key template cannot collide across obligations, but a custom template can, and a
delete must never take bytes another revision still needs. See
``ReportingRowStorageV1.delete_unreferenced`` for the locking contract.
delete must never take bytes another revision still needs. No object lock is taken: see the
contract in ``ReportingRowStorageV1`` (``# -- object lifetime``) and
``ReportingRowStorageV1.delete_unreferenced``.
"""
storage = store._row_storage
assert storage is not None
Expand All @@ -397,8 +398,10 @@ async def delete_period_objects(
for binding_id, object_ref, native_version_ref in objects:
by_binding.setdefault(binding_id, []).append((object_ref, native_version_ref))
for binding_id, pairs in by_binding.items():
# Objects a chunk of any other row set still references are kept, checked
# atomically with the delete so a concurrent commit cannot adopt a doomed object.
# Objects a chunk of any other row set still references are kept. The check needs no lock
# against a committing writer: the period's row sets are already ``pruning`` (committed
# under the account lock in phase 1), which orders this delete against every other
# deleter.
await storage.delete_unreferenced(
store._pool, binding_id, pairs, obligation_id=obligation_id
)
Expand Down
Loading
Loading