diff --git a/docs/reporting-release-notes.md b/docs/reporting-release-notes.md index 7b656ce232..94f82b3ef4 100644 --- a/docs/reporting-release-notes.md +++ b/docs/reporting-release-notes.md @@ -53,6 +53,29 @@ Reads handle both forms. `migrate_inline_rows` moves legacy rows with verificati rows do not reproduce its binding, and `retire_legacy_inline_rows` then renames the legacy table so older binaries fail closed. See [row storage](reporting-row-storage.md). +## Opt-in orphaned-object collector, fenced against clones + +`PgReportingLedgerStore(orphan_collection=ReportingOrphanCollectionOptionsV1(...))` and +`collect_orphaned_row_objects` list each object binding under its owned prefix and delete the objects no ledger record +or open upload names, which the write-intent sweep cannot see (an upload that succeeded before its intent recorded it, +intents for a binding no host configures, intents lost to a restore). It mirrors the JavaScript SDK +([adcp-client#3291](https://github.com/adcontextprotocol/adcp-client/pull/3291) and the clone fence in +[#3593](https://github.com/adcontextprotocol/adcp-client/pull/3593)). Nothing changes unless you opt in. An object is +deleted only when it is a well-formed row key under `{prefix}/{namespace_key}/`, older than `min_age_seconds` (default +7 days, minimum 1 day), named by no record or upload at any version under any binding, and unreferenced under the +per-object lock of every binding. A real pass first claims the namespace in a create-only fence in the bucket and +refuses to delete while another database holds a newer claim, so a clone or restore of the ledger database that shares +its namespace key and its opt-in cannot delete the live database's newer objects; `take_over_orphan_collector_fence` +is the deliberate step after a restore. + +Providers gain an optional `list`. The provider conformance suite checks it when present, so its result now ends with a +`list` entry for providers that implement it; compare accordingly. A provider without `list` keeps working and is +reported `unsupported`. `create_schema` adds the shared `adcp_reporting_orphan_collector_fences` table, additively and +outside the required-schema manifest. The production worker reports `fenced` bindings as stalled work and `unsupported` +bindings as pending work. **Roll out:** start with `dry_run=True`, claim from the live database first, and never enable +the collector on a copy of the deployment. See [the listing collector](reporting-row-storage.md#orphaned-objects-the-opt-in-listing-collector) +([#1829](https://github.com/adcontextprotocol/adcp-client-python/issues/1829)). + ## Row binding IDs and credential references accept 255 characters `row_binding_id` and a binding's `credential_ref` were capped at 128 characters. Every comparable AdCP identifier diff --git a/docs/reporting-row-storage.md b/docs/reporting-row-storage.md index e51ff68a2a..76c30bb088 100644 --- a/docs/reporting-row-storage.md +++ b/docs/reporting-row-storage.md @@ -327,6 +327,111 @@ stays as a small marker that makes reads answer as expired; the records are alre whose horizon excludes periods past retention. Versioned buckets, soft delete and retention locks are allowed as backstops, but a `pruned` period is not "erased" while they keep the old bytes. +## Orphaned objects: the opt-in listing collector + +The intent sweep deletes only what an intent lists, so it cannot see three cases: an upload whose `PUT` succeeded before +the intent recorded it, intents for a binding no running host configures, and intents lost to a database restore. +`collect_orphaned_row_objects` covers them by listing, and it is **off unless you ask for it**. It mirrors the +JavaScript SDK's collector and fence ([adcp-client#3291](https://github.com/adcontextprotocol/adcp-client/pull/3291), +[#3593](https://github.com/adcontextprotocol/adcp-client/pull/3593)); the design record is +`specs/shared-sdk-persistence-proposal.md` section 4.2. + +```python +from adcp_reporting_storage import ReportingOrphanCollectionOptionsV1 + +store = PgReportingLedgerStore( + pool=pool, + row_storage=options, + # Start with dry_run=True and read the candidates in the task result. A real pass needs the + # acknowledgement below. + orphan_collection=ReportingOrphanCollectionOptionsV1(acknowledge_exclusive_namespace=True), +) +# or, from your own scheduler: +result = await store.collect_orphaned_row_objects(dry_run=True) +``` + +It lists each configured object binding under `{prefix}/{namespace_key}/` and deletes an object only when **all** of +these hold: + +1. Its key is a well-formed row object (`.jsonl` or `.jsonl.gz`, the grammar `render_reporting_row_object_key_v1` + produces, no `.` or `..` segment) under that prefix. Anything else, or anything a provider returns outside the + prefix, is dropped and counted as `foreign`; `delete_objects` independently refuses keys outside the owned prefix. +2. It was written more than `min_age_seconds` ago (default 7 days, **minimum 1 day**, against a write deadline of at + most 10 minutes). An unreadable or non-RFC 3339 write time counts as young. Providers report the later of creation + and modification time. +3. A version- and binding-blind pre-check finds no chunk of a not-yet-`pruned` row set and no write intent (any state, + old or new JSON keys) naming the key. Two bindings can share a bucket and prefix (a retired `read_only` binding kept + next to its successor), so the check is not scoped to the collecting binding. +4. Under the per-object advisory lock of **every** binding, taken in a fixed order so the delete is serialized with an + adoption through any of them, no `live` or `unavailable` row set references that exact version and no open intent + created it. The provider then deletes that exact version. + +`read_only` bindings are skipped unless `include_read_only`. `dry_run` deletes nothing and reports up to 1000 +`candidates`. Deleting requires `acknowledge_exclusive_namespace=True` (a dry run does not). `deleted` counts delete +requests that found nothing referenced; an object another host removed first still counts. A delete that fails is +counted and the pass moves on, and is retried on the next full listing; a listing failure retries the same page. + +**Providers.** A provider gains an optional `list` (`ReportingRowObjectListInputV1` in, a page of +`ReportingRowObjectListEntryV1` out: `object_ref`, `native_version_ref`, `size_bytes`, `created_at`), implemented by the +filesystem, GCS, S3 and Azure Blob providers (on versioned stores it lists noncurrent versions too and never delete +markers). `run_reporting_row_object_provider_conformance_v1` checks it when present and then returns a `list` entry, so +code that compares the whole list must expect it. A provider without `list` keeps working; its binding is reported +`unsupported`. `list` must be read-after-write consistent for keys `put_if_absent` returned: the fence below reads the +newest claim by listing. The filesystem provider's `list` reads and sorts each directory in full and neither lists nor follows symlinks, so it is +for tests and single hosts. + +**Budget and resumability.** `limit` bounds the objects examined per binding per pass; a page cap bounds empty pages; a +repeated page token or an oversized page fails the binding instead of looping; a failure on the first page of a resumed +cursor drops the cursor so it cannot wedge. Pass `result.cursors` back as `cursors` and `result.completed` as +`skip_bindings`. As a store option, `interval_seconds` (default one day) spaces full listings, a listing that does not +finish within `limit` continues on every pass, and the position lives in the process, so a restart begins a new +listing. + +### The clone fence + +A clone or point-in-time restore of the ledger database shares the live database's installation identity, namespace +key and configuration, **including `acknowledge_exclusive_namespace`**, but sees an older set of references, so its +collector would delete the live database's newer objects once they pass the age threshold. A real pass is therefore +fenced: + +- **Fence.** `{prefix}/{namespace_key}/_collector-fence/{epoch:016d}.json`: one create-only object per epoch, + `{v, namespace_key, epoch, token, previous_token, claimed_at}`, the bytes the JavaScript SDK writes. The highest epoch + is the current claim. The database records its last `(epoch, token, previous_token)` in + `adcp_reporting_orphan_collector_fences`, keyed by a hash of provider, location, prefix and namespace key, so + bindings that reach the same bucket and prefix share one fence. +- **Claim.** A real pass takes a session advisory lock, proceeds only if the bucket's highest epoch is exactly the + recorded claim, commits the intent for `epoch + 1` under a fresh token, writes that object create-only, verifies the + token at the key and prunes all but the predecessor. It holds the lock through the write, so a sibling host of the + same database cannot interleave, and it does this before the first delete. A claim holds one pool connection across + the object-store reads and write. +- **Check.** The fence is read again before the first delete of every page and every 25 deletes within one. Any + mismatch stops the pass; `deleted` keeps what went before. +- **Refusal.** A refused binding has `status == "fenced"` and `fence` (`cause`, `detail`, `fence_epoch`, + `recorded_epoch`); the result lists it in `fenced`. Causes: `claimed_elsewhere`, `fence_advanced`, `token_mismatch`, + `fence_missing`, `fence_regressed`, `fence_unreadable`, `claim_lost`, `migration_required`. Everything unexpected + fails closed, including a fence-directory listing in which the provider left out an entry it could not address (it could be + the newest claim). A provider or database error is `failed`, not `fenced`. `dry_run` reads the fence without claiming it + and reports fenced bindings too. +- **Crash safety.** The intent commits before the object is written. A crash between the two leaves the database one + epoch ahead of the bucket with the matching previous token; the next pass finishes the claim under a **fresh** token, + so a copy taken in that window cannot adopt the same token as the original. +- **Takeover.** After a restore, or when the previous owner is gone, `await store.take_over_orphan_collector_fence( + acknowledge_previous_owner_gone=True)` claims `max(epoch) + 1` and supersedes every older object and version, + including a damaged chain. It refuses without the acknowledgement. Stop the previous owner's collector first: a pass + already running notices the new epoch at its next check, at most 25 deletes later. + +Known limits. **First claim wins:** nothing in a database distinguishes a copy from the original until one of them +claims, so a clone that collects before the original has run a pass since the copy claims first and the original is +fenced (and reported). The clone still cannot delete an object the original wrote after the copy until it is +`min_age_seconds` old. **Claim from the live database first** when you roll this out: one real pass, or a takeover. +The fence needs write access to each collected binding (a `read_only` binding collected with `include_read_only` +included), and a lifecycle rule must not expire `_collector-fence/`. + +**Rollout.** The fence table is created by `reporting_row_storage.sql` (`create_schema`), outside the required-schema +manifest, so a deployment that never enables the collector needs nothing from it. Until it exists a pass reports every +binding `fenced` with `migration_required`, a dry run included, and deletes nothing. Python supports one writer SDK per +ledger database; if a JavaScript deployment shares the database, claim from one SDK only. + ## Stored names follow AdCP Stored values use the names and bounds AdCP already defines (`reporting-file-manifest.json`, @@ -556,6 +661,7 @@ Tombstones live in `adcp_reporting_period_tombstones`, created with row storage | `row_storage` | `prune_expired_row_sets` | 20 periods | | `row_storage` or `change_feed` | `retire_expired_periods` | 20 periods | | `change_feed` | `prune_changes` | 10,000 rows | +| `orphan_collection` | `collect_orphaned_row_objects` | `limit` objects per binding per pass (default 1000), full listings spaced by `interval_seconds` | Each operation is bounded and resumable. A failing operation never stops the worker: the next interval retries it. Only the closed operation name and integer counters are logged, never an exception, provider message or identifier: @@ -563,11 +669,13 @@ Only the closed operation name and integer counters are logged, never an excepti - `REPORTING_MAINTENANCE_FAILED` (warning): the operation raised or reported failed items (a binding that cannot delete, for example). Read `causes` from the result for the stable per-item code. - `REPORTING_MAINTENANCE_PENDING` (info): work was left in place on purpose, with the counters `protected`, `held`, - `deferred` and `skipped_unconfigured`. A retirement that does nothing because every period is held shows up here + `deferred`, `skipped_unconfigured` and `unsupported` (object bindings whose provider cannot list). A retirement that does nothing because every period is held shows up here instead of looking like a healthy idle worker. - `REPORTING_MAINTENANCE_STALLED` (warning): work nobody is reaching, with `overdue` (intents expired for more than `overdue_after_seconds`, default 24 hours, and still unswept: a binding no host configures any more, or one that - keeps failing, so objects leak) and `stuck` (periods with five or more failed retirement attempts). + keeps failing, so objects leak), `stuck` (periods with five or more failed retirement attempts) and `fenced` + (object bindings the orphan collector refused to delete from because another database holds the namespace's claim; + see [the clone fence](#the-clone-fence)). The last pass's counters are also on `ReportingProductionSupport.maintenance_report`, per operation. `prune_expired_row_sets` is scheduled so row bytes are reclaimed even while a period's records stay; its only hold is diff --git a/packages/adcp-reporting-storage/src/adcp_reporting_storage/__init__.py b/packages/adcp-reporting-storage/src/adcp_reporting_storage/__init__.py index 8979c0afc8..0eef67eb4c 100644 --- a/packages/adcp-reporting-storage/src/adcp_reporting_storage/__init__.py +++ b/packages/adcp-reporting-storage/src/adcp_reporting_storage/__init__.py @@ -57,6 +57,27 @@ from adcp_reporting_storage.ledger.delivery_pg import ( PgReportingReconciliationStore as PgReportingReconciliationStore, ) + from adcp_reporting_storage.ledger.orphan_collector import ( + ReportingOrphanBindingResultV1 as ReportingOrphanBindingResultV1, + ) + from adcp_reporting_storage.ledger.orphan_collector import ( + ReportingOrphanCandidateV1 as ReportingOrphanCandidateV1, + ) + from adcp_reporting_storage.ledger.orphan_collector import ( + ReportingOrphanCollectionOptionsV1 as ReportingOrphanCollectionOptionsV1, + ) + from adcp_reporting_storage.ledger.orphan_collector import ( + ReportingOrphanCollectionResultV1 as ReportingOrphanCollectionResultV1, + ) + from adcp_reporting_storage.ledger.orphan_collector import ( + ReportingOrphanCollectionStateV1 as ReportingOrphanCollectionStateV1, + ) + from adcp_reporting_storage.ledger.orphan_collector import ( + ReportingOrphanFenceTakeoverV1 as ReportingOrphanFenceTakeoverV1, + ) + from adcp_reporting_storage.ledger.orphan_fence import ( + ReportingOrphanFenceRefusalV1 as ReportingOrphanFenceRefusalV1, + ) from adcp_reporting_storage.ledger.period_retirement import ( ReportingPeriodRetirementV1 as ReportingPeriodRetirementV1, ) @@ -257,6 +278,22 @@ "ReportingRowBindingSelectionInputV1", ), "ReportingRowIntentSweepV1": ("ledger.row_retention", "ReportingRowIntentSweepV1"), + "ReportingOrphanBindingResultV1": ("ledger.orphan_collector", "ReportingOrphanBindingResultV1"), + "ReportingOrphanCandidateV1": ("ledger.orphan_collector", "ReportingOrphanCandidateV1"), + "ReportingOrphanCollectionOptionsV1": ( + "ledger.orphan_collector", + "ReportingOrphanCollectionOptionsV1", + ), + "ReportingOrphanCollectionResultV1": ( + "ledger.orphan_collector", + "ReportingOrphanCollectionResultV1", + ), + "ReportingOrphanCollectionStateV1": ( + "ledger.orphan_collector", + "ReportingOrphanCollectionStateV1", + ), + "ReportingOrphanFenceTakeoverV1": ("ledger.orphan_collector", "ReportingOrphanFenceTakeoverV1"), + "ReportingOrphanFenceRefusalV1": ("ledger.orphan_fence", "ReportingOrphanFenceRefusalV1"), "ReportingRowPruneV1": ("ledger.row_retention", "ReportingRowPruneV1"), "ReportingPeriodRetirementV1": ("ledger.period_retirement", "ReportingPeriodRetirementV1"), "ReportingRowMigrationBatchV1": ("ledger.row_migration", "ReportingRowMigrationBatchV1"), diff --git a/packages/adcp-reporting-storage/src/adcp_reporting_storage/ledger/orphan_collector.py b/packages/adcp-reporting-storage/src/adcp_reporting_storage/ledger/orphan_collector.py new file mode 100644 index 0000000000..3f16d92f2c --- /dev/null +++ b/packages/adcp-reporting-storage/src/adcp_reporting_storage/ledger/orphan_collector.py @@ -0,0 +1,987 @@ +"""Opt-in collector for orphaned row objects, fenced against a cloned or restored database. + +Shared SDK persistence spec, adcontextprotocol/adcp#7996 section 4.2; mirrors the JavaScript SDK's +``collectOrphanedRowObjects`` (adcp-client#3291) and its ownership fence (adcp-client#3593). + +The write-intent sweep (:mod:`adcp_reporting_storage.ledger.row_retention`) deletes only what an +intent lists. It cannot see an object whose ``PUT`` succeeded before the intent recorded it, an +intent for a binding no running host configures, or an intent lost to a database restore. This +collector lists each configured object binding under its owned prefix, +``{prefix}/{namespace_key}/``, and deletes an object only when all of these hold: + +* its key is a well-formed row object (``.jsonl`` or ``.jsonl.gz``) under that prefix; +* it was written more than ``min_age_seconds`` ago (default 7 days, at least 1 day), so no writer + that still holds a lease can be mid-upload; +* no row set that is not already ``pruned`` has a chunk at its key, and no write intent lists it, + at any version and under any binding (a pre-check that errs towards leaking, because two bindings + can share a bucket and prefix); +* under the per-object advisory lock of **every** binding, no ``live`` or ``unavailable`` row set + under any binding references that exact version and no open intent created it (the check every + other delete makes, widened to every binding), and the provider then deletes that exact version. + +A clone or point-in-time restore of the ledger database shares the live database's installation +identity, namespace key and configuration, so it would also share the opt-in. A real pass therefore +first claims the namespace in an ownership fence +(:mod:`adcp_reporting_storage.ledger.orphan_fence`) and refuses to delete while another database +holds a newer claim. See ``docs/reporting-row-storage.md``. +""" + +from __future__ import annotations + +import asyncio +import math +import time +import uuid +from collections.abc import Awaitable, Callable, Mapping, Sequence +from dataclasses import dataclass, field +from datetime import datetime, timedelta, timezone +from typing import TYPE_CHECKING, Any, Literal + +from adcp_reporting_storage.ledger.orphan_fence import ( + OrphanFenceDecision, + OrphanFenceEntry, + OrphanFenceRecord, + OrphanFenceRow, + ReportingOrphanFenceRefusalV1, + decide_orphan_fence, +) +from adcp_reporting_storage.ledger.row_storage import ReportingRowStoreError +from adcp_reporting_storage.ledger.row_storage_object import ( + format_row_instant, + parse_row_instant, +) + +if TYPE_CHECKING: + from adcp_reporting_storage.ledger.pg import PgReportingLedgerStore + +__all__ = [ + "ReportingOrphanBindingResultV1", + "ReportingOrphanCandidateV1", + "ReportingOrphanCollectionOptionsV1", + "ReportingOrphanCollectionResultV1", + "ReportingOrphanCollectionStateV1", + "ReportingOrphanFenceTakeoverV1", + "collect_orphaned_row_objects", + "run_orphan_collection_pass", + "take_over_orphan_collector_fence", +] + +DEFAULT_ORPHAN_MIN_AGE_SECONDS = 7 * 24 * 3600 +MIN_ORPHAN_MIN_AGE_SECONDS = 24 * 3600 +DEFAULT_ORPHAN_INTERVAL_SECONDS = 86_400 +#: A fenced namespace looks again within the hour, so applying the migration or taking the +#: namespace over does not leave collection idle for the rest of a day. +FENCED_RECHECK_SECONDS = 3_600 +MAX_ORPHAN_CANDIDATES = 1_000 +#: Deletes between reads of the collector fence within one listing page. +ORPHAN_FENCE_RECHECK_DELETES = 25 +_FENCE_LOCK_TIMEOUT = "120s" +#: The suffixes ``render_reporting_row_object_key_v1`` gives every key: nothing else under the +#: prefix is a row object. +_ROW_OBJECT_SUFFIXES = (".jsonl", ".jsonl.gz") + +_BindingStatus = Literal["complete", "partial", "failed", "unsupported", "read_only", "fenced"] + + +@dataclass(frozen=True, slots=True) +class ReportingOrphanCandidateV1: + """An object a dry run would delete.""" + + binding_id: str + object_ref: str + native_version_ref: str + size_bytes: int + created_at: str + + +@dataclass(slots=True) +class ReportingOrphanBindingResultV1: + """What one binding's listing did. + + ``status``: ``complete`` (the listing reached its end), ``partial`` (stopped at ``limit``; + resume from ``cursors``), ``failed`` (the listing or its provider failed; ``error`` is the + stable code), ``unsupported`` (the provider has no ``list``), ``read_only`` (skipped because + the binding is ``read_only``) or ``fenced`` (another database holds the namespace's collector + fence, or this database cannot prove it holds it, so no further object was deleted; ``deleted`` + counts any that went before a mid-pass refusal and ``fence`` says why). + """ + + binding_id: str + status: _BindingStatus = "complete" + #: Objects the provider listed, including ones dropped as foreign. + listed: int = 0 + #: Old enough, but named by a ledger record or open upload; left alone. + referenced: int = 0 + #: Younger than ``min_age_seconds``, or with no readable write time. + too_young: int = 0 + #: Outside the owned prefix or not shaped like a row object; never touched. + foreign: int = 0 + deleted: int = 0 + #: Dry run only: orphans that a real pass would delete. + would_delete: int = 0 + failed: int = 0 + error: str | None = None + fence: ReportingOrphanFenceRefusalV1 | None = None + + +@dataclass(slots=True) +class ReportingOrphanCollectionResultV1: + """A collection pass. ``deleted`` counts delete requests that found nothing referenced; an + object another host removed first still counts.""" + + dry_run: bool + listed: int = 0 + referenced: int = 0 + too_young: int = 0 + foreign: int = 0 + deleted: int = 0 + would_delete: int = 0 + #: Delete failures plus bindings whose listing failed. + failed: int = 0 + #: Bindings whose provider cannot list: not covered by this collector. + unsupported: tuple[str, ...] = () + #: ``read_only`` bindings skipped. + skipped_read_only: tuple[str, ...] = () + #: Bindings whose collector fence another database holds: no further object was deleted there. + #: See each binding's ``fence``. + fenced: tuple[str, ...] = () + #: Dry run only: the orphans a real pass would delete, at most 1000. + candidates: tuple[ReportingOrphanCandidateV1, ...] = () + #: Resume tokens for bindings whose listing is not finished. Pass back as ``cursors``. + cursors: Mapping[str, str] = field(default_factory=dict) + #: Bindings whose listing reached its end this pass. Pass them back as ``skip_bindings`` until + #: every binding has. + completed: tuple[str, ...] = () + bindings: tuple[ReportingOrphanBindingResultV1, ...] = () + + +@dataclass(frozen=True, slots=True) +class ReportingOrphanFenceTakeoverV1: + """One fence a takeover claimed.""" + + #: The bindings that share the fence. + binding_ids: tuple[str, ...] + #: The epoch now held. + epoch: int + #: Highest epoch found in the object store before the takeover; ``None`` when the namespace + #: was never claimed. + previous_epoch: int | None + + +@dataclass(frozen=True, slots=True) +class ReportingOrphanCollectionOptionsV1: + """Opt-in scheduling of :func:`collect_orphaned_row_objects` (``PgReportingLedgerStore( + orphan_collection=...)``). + + Enable it only on the ledger database that owns the namespace. A clone or restore of that + database pointed at the same bucket shares its namespace, so a real pass is fenced: it claims + the namespace and refuses, reporting the binding ``fenced``, when the object store holds a newer + claim than this database recorded. After a restore, or when the original is gone, call + ``take_over_orphan_collector_fence``. Start with ``dry_run=True`` and read the candidates. + """ + + #: Report what would be deleted and delete nothing. + dry_run: bool = False + #: Required unless ``dry_run``: your statement that this ledger database is the only one that + #: owns the namespace, and not a clone or restore pointed at the same bucket. A copy of the + #: configuration carries this flag too; the ownership fence is what stops such a copy. + acknowledge_exclusive_namespace: bool = False + #: Minimum time between full listings, counted from when the previous one finished. Passes in + #: between do nothing; a listing that has not finished (it stops at ``limit``) continues on + #: every pass. + interval_seconds: int = DEFAULT_ORPHAN_INTERVAL_SECONDS + #: Objects younger than this are never deleted. At least 1 day; keep it well above the longest + #: write deadline plus intent expiry. + min_age_seconds: int = DEFAULT_ORPHAN_MIN_AGE_SECONDS + #: Objects examined per binding per pass. + limit: int = 1_000 + #: Also collect from ``read_only`` bindings, which are skipped by default. + include_read_only: bool = False + + def __post_init__(self) -> None: + for flag in ("dry_run", "acknowledge_exclusive_namespace", "include_read_only"): + if type(getattr(self, flag)) is not bool: + raise ValueError(f"orphan_collection.{flag} must be a boolean") + if not self.dry_run and not self.acknowledge_exclusive_namespace: + raise ValueError( + "orphan_collection deletes objects: set acknowledge_exclusive_namespace=True to " + "confirm this ledger database is the only one that owns the namespace (not a " + "clone or restore pointed at the same bucket), or set dry_run=True" + ) + for name in ("interval_seconds", "limit"): + value = getattr(self, name) + if type(value) is not int or value < 1: + raise ValueError(f"orphan_collection.{name} must be a positive integer") + if ( + type(self.min_age_seconds) is not int + or self.min_age_seconds < MIN_ORPHAN_MIN_AGE_SECONDS + ): + raise ValueError( + "orphan_collection.min_age_seconds must be an integer of at least 86400 (1 day): " + "a younger object may belong to an upload still in flight" + ) + + +@dataclass(slots=True) +class ReportingOrphanCollectionStateV1: + """Progress of the collector between passes, kept in the process. + + It is not persisted, so a restart begins a new listing. + """ + + cursors: dict[str, str] = field(default_factory=dict) + #: Bindings whose listing finished in the current cycle; not listed again until every binding + #: has. + completed: list[str] = field(default_factory=list) + #: ``time.monotonic()`` seconds before which no new full listing starts. + next_cycle_at: float | None = None + + +@dataclass(frozen=True, slots=True) +class _Gate: + refusal: ReportingOrphanFenceRefusalV1 | None = None + + @property + def ok(self) -> bool: + return self.refusal is None + + +# -- the collector --------------------------------------------------------------------------- + + +def _positive(value: int, name: str, maximum: int = 1_000_000) -> None: + if type(value) is not int or not 1 <= value <= maximum: + raise ValueError(f"{name} must be between 1 and {maximum}") + + +def _names(value: Sequence[str] | None, name: str) -> tuple[str, ...] | None: + """Binding IDs as a tuple. A bare string would otherwise act as a substring test.""" + if value is None: + return None + if isinstance(value, (str, bytes)) or not all(isinstance(item, str) for item in value): + raise ValueError(f"{name} must be a sequence of binding IDs") + return tuple(value) + + +def _code(error: Exception) -> str: + return error.code if isinstance(error, ReportingRowStoreError) else "INTERNAL_ERROR" + + +async def _present_tables(store: PgReportingLedgerStore) -> set[str]: + async with store._pool.connection() as connection: + rows = await ( + await connection.execute( + "SELECT table_name FROM information_schema.tables" + " WHERE table_schema = current_schema() AND table_name = ANY(%s::text[])", + ( + [ + "adcp_reporting_row_sets", + "adcp_reporting_row_write_intents", + "adcp_reporting_orphan_collector_fences", + ], + ), + ) + ).fetchall() + return {row[0] for row in rows} + + +async def _keys_named_by_records(store: PgReportingLedgerStore, keys: Sequence[str]) -> set[str]: + """Of these object keys, the ones a ledger record names at any version, under any binding. + + A chunk of a row set that is not yet ``pruned``, or an object any write intent lists. Version- + and binding-blind on purpose: two bindings can share a bucket and prefix (a retired binding kept + ``read_only`` next to its successor), and the orphan collector would rather leak than delete + near a record. + """ + if not keys: + return set() + async with store._pool.connection() as connection: + rows = await ( + await connection.execute( + "SELECT chunk.object_ref FROM adcp_reporting_row_chunks chunk" + " JOIN adcp_reporting_row_sets row_set ON row_set.row_set_id = chunk.row_set_id" + " WHERE chunk.object_ref = ANY(%(keys)s::text[]) AND row_set.rows_state <> 'pruned'" + " UNION" + " SELECT COALESCE(item ->> 'object_ref', item ->> 'objectKey')" + " FROM adcp_reporting_row_write_intents intent," + " jsonb_array_elements(intent.created_objects) item" + " WHERE COALESCE(item ->> 'object_ref', item ->> 'objectKey')" + " = ANY(%(keys)s::text[])", + {"keys": list(keys)}, + ) + ).fetchall() + return {row[0] for row in rows} + + +async def collect_orphaned_row_objects( + store: PgReportingLedgerStore, + *, + min_age_seconds: int | None = None, + limit: int = 1_000, + page_size: int | None = None, + dry_run: bool = False, + include_read_only: bool = False, + bindings: Sequence[str] | None = None, + cursors: Mapping[str, str] | None = None, + skip_bindings: Sequence[str] | None = None, + acknowledge_exclusive_namespace: bool = False, + now: datetime | None = None, +) -> ReportingOrphanCollectionResultV1: + """Delete row objects that no ledger record or open upload names. + + Opt-in, listing-based and read-only unless it deletes. See the module docstring for the full + list of conditions an object must meet. ``read_only`` bindings are skipped unless + ``include_read_only``. A binding whose provider has no ``list`` is reported in + ``unsupported``. ``dry_run`` deletes nothing and reports what would go in ``candidates``. + ``limit`` bounds the objects examined per binding per pass; pass ``result.cursors`` back as + ``cursors`` to resume where the last pass stopped (a binding absent from ``cursors`` starts from + the beginning, which is also where a binding that finished its listing starts next time), and + ``result.completed`` as ``skip_bindings`` so a cycle that spans several passes does not rescan + bindings that already finished. + + Deleting needs ``acknowledge_exclusive_namespace=True``: enable it only on the ledger database + that owns the namespace. A clone or restore of that database that still points at the same + bucket shares its ``namespace_key``, and would delete the objects of every row set the original + wrote after the clone was taken. That flag is copied with the configuration, so it is backed by + a fence: a real pass first claims the namespace (a create-only object under + ``{prefix}/{namespace_key}/_collector-fence/`` and a row in + ``adcp_reporting_orphan_collector_fences``) and deletes only while the object store's highest + claim is exactly the one this database recorded. A database copied before the original's last + pass sees a newer claim and refuses: its binding is reported ``fenced`` with a ``cause``, and + nothing more is deleted (a fence that advances mid-pass stops the pass at the next check, within + 25 deletes, and ``deleted`` keeps what went before). A ``dry_run`` checks the fence without + claiming it and reports fenced bindings the same way. After a restore, or when the original is + gone, take the namespace over deliberately with ``take_over_orphan_collector_fence``. Everything + the listing finds under the owned prefix with a row-object key is treated as written by this + SDK, so do not store other ``.jsonl`` files there. + """ + storage = store._row_storage + if storage is None: + raise ReportingRowStoreError( + "INVALID_INPUT", "collect_orphaned_row_objects requires row storage" + ) + bindings = _names(bindings, "bindings") + skip_bindings = _names(skip_bindings, "skip_bindings") + if min_age_seconds is None: + min_age_seconds = DEFAULT_ORPHAN_MIN_AGE_SECONDS + if type(min_age_seconds) is not int or min_age_seconds < MIN_ORPHAN_MIN_AGE_SECONDS: + raise ValueError( + f"min_age_seconds must be an integer of at least {MIN_ORPHAN_MIN_AGE_SECONDS} " + "(1 day): an object younger than the longest write deadline plus intent expiry may " + "belong to an upload still in flight" + ) + _positive(limit, "limit") + page_size = min(limit, 500) if page_size is None else page_size + _positive(page_size, "page_size") + if dry_run is not True: + dry_run = False + if not dry_run and acknowledge_exclusive_namespace is not True: + raise ValueError( + "collect_orphaned_row_objects deletes objects: pass " + "acknowledge_exclusive_namespace=True to confirm this ledger database is the only one " + "that owns the namespace (not a clone or restore pointed at the same bucket), or pass " + "dry_run=True" + ) + clock = now if now is not None else datetime.now(timezone.utc) + if not isinstance(clock, datetime) or clock.tzinfo is None: + raise ValueError("now must be a timezone-aware datetime") + await store.ready_row_storage() + summaries = storage.object_binding_summaries() + for binding_id in bindings or (): + if not any(summary.id == binding_id for summary in summaries): + raise ValueError( + f"bindings names {binding_id}, which is not a configured object binding" + ) + + result = ReportingOrphanCollectionResultV1(dry_run=dry_run) + outcomes: list[ReportingOrphanBindingResultV1] = [] + candidates: list[ReportingOrphanCandidateV1] = [] + completed: list[str] = [] + unsupported: list[str] = [] + skipped_read_only: list[str] = [] + fenced: list[str] = [] + next_cursors: dict[str, str] = {} + + def finish() -> ReportingOrphanCollectionResultV1: + for outcome in outcomes: + result.listed += outcome.listed + result.referenced += outcome.referenced + result.too_young += outcome.too_young + result.foreign += outcome.foreign + result.deleted += outcome.deleted + result.would_delete += outcome.would_delete + result.failed += outcome.failed + result.unsupported = tuple(unsupported) + result.skipped_read_only = tuple(skipped_read_only) + result.fenced = tuple(fenced) + result.candidates = tuple(candidates) + result.cursors = next_cursors + result.completed = tuple(completed) + result.bindings = tuple(outcomes) + return result + + tables = await _present_tables(store) + if "adcp_reporting_row_sets" not in tables or "adcp_reporting_row_write_intents" not in tables: + return finish() + has_fence_table = "adcp_reporting_orphan_collector_fences" in tables + # One fence check or claim per physical fence per pass: bindings that share a bucket and prefix + # share it. + gates: dict[str, _Gate] = {} + + async def gate(binding_id: str, mode: Literal["claim", "check"]) -> _Gate: + scope, _, _ = await storage.collector_fence_location(binding_id) + if mode == "claim" and scope in gates: + return gates[scope] + outcome = await _orphan_fence_gate( + store, + binding_id, + scope, + "check" if mode == "claim" and dry_run else mode, + has_fence_table, + ) + if mode == "claim": + gates[scope] = outcome + return outcome + + cutoff = clock - timedelta(seconds=min_age_seconds) + for summary in summaries: + if bindings is not None and summary.id not in bindings: + continue + if skip_bindings is not None and summary.id in skip_bindings: + continue + outcome = ReportingOrphanBindingResultV1(summary.id) + outcomes.append(outcome) + if summary.read_only and not include_read_only: + outcome.status = "read_only" + skipped_read_only.append(summary.id) + continue + if not summary.listable: + outcome.status = "unsupported" + unsupported.append(summary.id) + continue + token = cursors.get(summary.id) if cursors is not None else None + # The ownership fence comes before any listing: a clone or restore must not even enumerate + # for deletion. + try: + claim = await gate(summary.id, "claim") + except Exception as error: + outcome.status = "failed" + outcome.error = _code(error) + outcome.failed += 1 + # The listing never started, so the position it had stays where it was. + if token is not None: + next_cursors[summary.id] = token + continue + if not claim.ok: + outcome.status = "fenced" + outcome.fence = claim.refusal + fenced.append(summary.id) + continue + # A failure on the first page of a resumed listing drops the cursor: a stale or invalid + # token must not wedge the binding, and restarting from the beginning is always safe. + resumed = token is not None + drop_cursor = False + examined = 0 + pages = 0 + # Pages that carry no objects (S3 delete markers, unversioned entries) do not move + # ``examined``, so the pass also stops after a bounded number of requests. + max_pages = math.ceil(limit / page_size) * 4 + try: + while True: + page = await storage.list_objects( + summary.id, page_size=min(page_size, limit - examined), page_token=token + ) + resumed = False + pages += 1 + examined += len(page.objects) + page.foreign + outcome.listed += len(page.objects) + page.foreign + outcome.foreign += page.foreign + old = [] + for entry in page.objects: + if not entry.object_ref.endswith(_ROW_OBJECT_SUFFIXES): + outcome.foreign += 1 + continue + written = parse_row_instant(entry.created_at) + # An unreadable write time is treated as young: never delete on a guess. + if written is None or written > cutoff: + outcome.too_young += 1 + continue + old.append(entry) + named = await _keys_named_by_records(store, [entry.object_ref for entry in old]) + # Another database may claim the namespace while this pass runs: read the fence + # again before the first delete of every page and then every + # ORPHAN_FENCE_RECHECK_DELETES deletes after it. + since_fence_check = ORPHAN_FENCE_RECHECK_DELETES + fenced_mid_pass = False + for entry in old: + if entry.object_ref in named: + outcome.referenced += 1 + continue + if dry_run: + outcome.would_delete += 1 + if len(candidates) < MAX_ORPHAN_CANDIDATES: + candidates.append( + ReportingOrphanCandidateV1( + summary.id, + entry.object_ref, + entry.native_version_ref, + entry.size_bytes, + entry.created_at, + ) + ) + continue + if since_fence_check >= ORPHAN_FENCE_RECHECK_DELETES: + again = await gate(summary.id, "check") + if not again.ok: + outcome.status = "fenced" + outcome.fence = again.refusal + fenced.append(summary.id) + fenced_mid_pass = True + break + since_fence_check = 0 + since_fence_check += 1 + try: + # The per-object lock and exact-version reference check every other + # delete makes, widened to every binding. + removed = await storage.delete_unreferenced( + store._pool, + summary.id, + [(entry.object_ref, entry.native_version_ref)], + any_binding=True, + ) + if removed > 0: + outcome.deleted += 1 + else: + outcome.referenced += 1 + except Exception: + outcome.failed += 1 + if fenced_mid_pass: + break + if page.next_page_token is not None and page.next_page_token == token: + drop_cursor = True + raise ReportingRowStoreError( + "PROVIDER_UNAVAILABLE", "provider returned the same page token twice" + ) + token = page.next_page_token + if token is None: + completed.append(summary.id) + break + if examined >= limit or pages >= max_pages: + outcome.status = "partial" + next_cursors[summary.id] = token + break + except Exception as error: + outcome.status = "failed" + outcome.error = _code(error) + outcome.failed += 1 + # A listing failure retries the same page next pass. (A delete that fails is counted + # and the pass moves on, so one stuck object cannot hold up the rest; it is retried on + # the next full listing.) + if token is not None and not resumed and not drop_cursor: + next_cursors[summary.id] = token + return finish() + + +# -- the fence: claim, check, take over ----------------------------------------------------- + + +async def _fence_row(connection: Any, scope: str) -> tuple[OrphanFenceRow, str] | None: + row = await ( + await connection.execute( + "SELECT epoch, token, previous_token, claimed_at" + " FROM adcp_reporting_orphan_collector_fences WHERE fence_scope = %s", + (scope,), + ) + ).fetchone() + if row is None: + return None + return OrphanFenceRow(int(row[0]), row[1], row[2]), format_row_instant(row[3]) + + +async def _latest_orphan_fence( + store: PgReportingLedgerStore, binding_id: str +) -> tuple[list[OrphanFenceEntry], tuple[int, str] | None]: + """The object store's current claim: the highest epoch listed, read and verified.""" + storage = store._row_storage + assert storage is not None + entries = await storage.list_collector_fence(binding_id) + if not entries: + return entries, None + record = await storage.read_collector_fence(binding_id, entries[-1]) + return entries, (record.epoch, record.token) + + +def _unreadable() -> _Gate: + return _Gate( + ReportingOrphanFenceRefusalV1( + "fence_unreadable", + "the collector fence objects under this namespace are malformed, oversized, " + "duplicated or name another namespace. Inspect them; once the namespace is known to " + "be owned by this database alone, call " + "take_over_orphan_collector_fence(acknowledge_previous_owner_gone=True).", + ) + ) + + +async def _orphan_fence_gate( + store: PgReportingLedgerStore, + binding_id: str, + scope: str, + mode: Literal["claim", "check"], + fences_table: bool, +) -> _Gate: + """Whether this database may collect orphans under a binding's namespace. + + For a real pass this is also the claim that makes it so. ``check`` only reads: it is what + ``dry_run`` and the per-page recheck use. ``claim`` runs under a per-fence advisory lock: it + verifies the object store's highest epoch is exactly the claim this database recorded, records + the next epoch and a fresh token in the database, and only then writes that epoch's object + create-only. The intent is committed before the object is written, so a crash between the two + is finished by the next pass instead of looking like another database's claim. + + Each real pass claims a new epoch, so a cycle spread over many partial passes writes and prunes + one tiny object per pass. The claim holds a pool connection and the advisory lock across the + object-store listing and read, so a slow bucket pins one connection for up to the read deadline. + """ + if not fences_table: + return _Gate( + ReportingOrphanFenceRefusalV1( + "migration_required", + "the orphan collector fence table does not exist: apply " + "reporting_row_storage.sql (create_schema does), then orphan collection can " + "claim its namespace", + ) + ) + storage = store._row_storage + assert storage is not None + try: + if mode == "check": + # The bucket first, then the row. A claim commits its intent to the row before it + # writes the object, so a row read after the bucket is never behind a claim of this + # database that the bucket already shows (a sibling host claiming between the two + # reads would otherwise look like another database's claim). + _, latest = await _latest_orphan_fence(store, binding_id) + async with store._pool.connection() as connection: + stored = await _fence_row(connection, scope) + decision = decide_orphan_fence(stored[0] if stored else None, latest) + if decision.action == "fenced": + return _Gate(decision.refusal) + return _Gate() + + async def plan(stored: tuple[OrphanFenceRow, str] | None) -> _Plan: + entries, latest = await _latest_orphan_fence(store, binding_id) + decision = decide_orphan_fence(stored[0] if stored else None, latest) + if decision.action == "fenced": + return _Plan(refusal=decision.refusal) + # An unfinished claim is finished under a fresh token, never the recorded one: a copy + # of this database taken while the claim was unfinished carries the same token, and + # both could then "adopt" the same object. With distinct tokens, create-only leaves + # exactly one of them the owner. + completing = decision.action == "complete" + assert stored is not None or not completing + return _Plan( + record=OrphanFenceRecord( + stored[0].epoch if completing and stored else _epoch(decision), + str(uuid.uuid4()), + stored[0].previous_token if completing and stored else decision.previous_token, + format_row_instant(datetime.now(timezone.utc)), + ), + entries=entries, + previous_epoch=latest[0] if latest else None, + ) + + claimed = await _claim_orphan_fence(store, binding_id, scope, True, plan) + if claimed.kind == "fenced": + return _Gate(claimed.refusal) + if claimed.kind == "lost": + assert claimed.record is not None and claimed.found is not None + return _Gate( + ReportingOrphanFenceRefusalV1( + "claim_lost", + f"another database claimed epoch {claimed.record.epoch} first. If this " + "database is the only one that owns this namespace, call " + "take_over_orphan_collector_fence(acknowledge_previous_owner_gone=True); " + "otherwise leave orphan collection off on this copy.", + claimed.found.epoch, + claimed.record.epoch, + ) + ) + return _Gate() + except ReportingRowStoreError as error: + # Fence objects that are not exactly what this collector writes fail closed. + if error.code == "ROWS_INTEGRITY_FAILED": + return _unreadable() + raise + + +def _epoch(decision: OrphanFenceDecision) -> int: + assert decision.epoch is not None + return decision.epoch + + +@dataclass(slots=True) +class _Plan: + refusal: ReportingOrphanFenceRefusalV1 | None = None + record: OrphanFenceRecord | None = None + entries: list[OrphanFenceEntry] = field(default_factory=list) + previous_epoch: int | None = None + + +@dataclass(slots=True) +class _Claim: + kind: Literal["fenced", "lost", "claimed"] + refusal: ReportingOrphanFenceRefusalV1 | None = None + record: OrphanFenceRecord | None = None + found: OrphanFenceRecord | None = None + previous_epoch: int | None = None + + +async def _claim_orphan_fence( + store: PgReportingLedgerStore, + binding_id: str, + scope: str, + keep_predecessor: bool, + plan: Callable[[tuple[OrphanFenceRow, str] | None], Awaitable[_Plan]], +) -> _Claim: + """Claim one fence epoch for this database, under a session-level advisory lock held from the + decision to the object write. + + ``plan`` runs inside the transaction with the stored row, reads the bucket and returns either a + refusal or the claim to make (and the entries it supersedes). The claim's intent (epoch, token) + is committed before the object is written, so a crash between the two is finished by the next + pass instead of looking like another database's claim; holding the lock through the write keeps + a sibling host of this database from finishing that claim under another token while the first + write is still in flight. A clone has its own database and its own lock, which is exactly why + the object store's create-only write decides between them. + """ + storage = store._row_storage + assert storage is not None + lock = f"adcp.orphan_fence:{scope}" + async with store._pool.connection() as connection: + original_autocommit = connection.autocommit + await connection.set_autocommit(True) + locked = False + try: + # A sibling holds the lock across its object-store write, which has its own deadline. + await connection.execute(f"SET lock_timeout = '{_FENCE_LOCK_TIMEOUT}'") + # Marked before the call: a cancellation after the server grants the lock but before + # the client reads the reply must still unlock (unlocking a lock never taken only + # returns false), or the connection would go back to the pool holding it. + locked = True + await connection.execute("SELECT pg_advisory_lock(hashtextextended(%s, 0))", (lock,)) + await connection.execute("RESET lock_timeout") + async with connection.transaction(): + planned = await plan(await _fence_row(connection, scope)) + if planned.record is not None: + await connection.execute( + "INSERT INTO adcp_reporting_orphan_collector_fences" + " (fence_scope, epoch, token, previous_token, claimed_at)" + " VALUES (%(scope)s, %(epoch)s, %(token)s, %(previous)s, %(claimed)s)" + " ON CONFLICT (fence_scope) DO UPDATE" + " SET epoch = EXCLUDED.epoch, token = EXCLUDED.token," + " previous_token = EXCLUDED.previous_token," + " claimed_at = EXCLUDED.claimed_at", + { + "scope": scope, + "epoch": planned.record.epoch, + "token": planned.record.token, + "previous": planned.record.previous_token, + "claimed": parse_row_instant(planned.record.claimed_at), + }, + ) + if planned.refusal is not None: + return _Claim("fenced", refusal=planned.refusal) + assert planned.record is not None + _, written = await storage.put_collector_fence(binding_id, planned.record) + if written.token != planned.record.token: + return _Claim("lost", record=planned.record, found=written) + # A claim keeps its predecessor (an unfinished claim is recognised by it) and prunes + # everything older. A takeover has just written and confirmed its object, so it + # supersedes every object and version it saw. + await storage.delete_collector_fence( + binding_id, + [ + entry + for entry in planned.entries + if entry.epoch < planned.record.epoch - (1 if keep_predecessor else 0) + ], + ) + return _Claim("claimed", record=planned.record, previous_epoch=planned.previous_epoch) + finally: + try: + await connection.execute("RESET lock_timeout") + if locked: + await connection.execute( + "SELECT pg_advisory_unlock(hashtextextended(%s, 0))", (lock,) + ) + await connection.set_autocommit(original_autocommit) + except BaseException: + # A connection that may still hold the session lock must not go back to the pool. + await asyncio.shield(connection.close()) + raise + + +async def take_over_orphan_collector_fence( + store: PgReportingLedgerStore, + *, + acknowledge_previous_owner_gone: bool, + bindings: Sequence[str] | None = None, + include_read_only: bool = False, +) -> list[ReportingOrphanFenceTakeoverV1]: + """Take over the orphan collector's namespace fence. + + The deliberate step after a restore, or when the database that held the fence is gone for + good. It claims the next epoch for this database, so the previous holder (and every other copy + of its state) is fenced out, and this database's collector can delete again. + + Do this only when this database is the one that should own the namespace. If the original is + still running, taking over from a clone fences the original and lets the clone delete its + objects: the acknowledgement exists because nothing else can tell them apart. + ``acknowledge_previous_owner_gone=True`` is required. + + A takeover does not stop a collector that is already mid-pass: that pass notices the new epoch + at its next fence check, which is at most 25 deletes later. Stop the previous owner's + collector before you take over. + + Takes over every writable (or, with ``include_read_only``, every) object binding whose provider + can list, or only the ``bindings`` named; bindings that share a bucket, prefix and installation + share one fence and are reported together. Requires the fence table + (``reporting_row_storage.sql``). + """ + if acknowledge_previous_owner_gone is not True: + raise ValueError( + "take_over_orphan_collector_fence fences out the namespace's current owner: pass " + "acknowledge_previous_owner_gone=True to confirm that database is gone or must never " + "collect again" + ) + storage = store._row_storage + if storage is None: + raise ReportingRowStoreError( + "INVALID_INPUT", "take_over_orphan_collector_fence requires row storage" + ) + bindings = _names(bindings, "bindings") + await store.ready_row_storage() + summaries = storage.object_binding_summaries() + for binding_id in bindings or (): + summary = next((item for item in summaries if item.id == binding_id), None) + if summary is None: + raise ValueError( + f"bindings names {binding_id}, which is not a configured object binding" + ) + if not summary.listable: + raise ValueError(f"bindings names {binding_id}, whose provider cannot list objects") + if "adcp_reporting_orphan_collector_fences" not in await _present_tables(store): + raise ReportingRowStoreError( + "STATE_UNAVAILABLE", + "the orphan collector fence table does not exist: apply reporting_row_storage.sql", + ) + selected = [ + summary + for summary in summaries + if summary.listable + and ( + summary.id in bindings + if bindings is not None + else include_read_only or not summary.read_only + ) + ] + by_scope: dict[str, list[str]] = {} + for summary in selected: + scope, _, _ = await storage.collector_fence_location(summary.id) + by_scope.setdefault(scope, []).append(summary.id) + taken: list[ReportingOrphanFenceTakeoverV1] = [] + for scope, binding_ids in by_scope.items(): + binding_id = binding_ids[0] + + async def plan( + stored: tuple[OrphanFenceRow, str] | None, binding_id: str = binding_id + ) -> _Plan: + entries = await storage.list_collector_fence(binding_id, tolerant=True) + newest = entries[-1] if entries else None + # A malformed newest object is exactly what a takeover is for: its key still gives the + # epoch to supersede. Several versions of the newest epoch are ambiguous, so none of + # them names the previous token. + previous: OrphanFenceRecord | None = None + if newest is not None and sum(1 for item in entries if item.epoch == newest.epoch) == 1: + try: + previous = await storage.read_collector_fence(binding_id, newest) + except ReportingRowStoreError as error: + if error.code != "ROWS_INTEGRITY_FAILED": + raise + return _Plan( + record=OrphanFenceRecord( + max(newest.epoch if newest else 0, stored[0].epoch if stored else 0) + 1, + str(uuid.uuid4()), + previous.token if previous else None, + format_row_instant(datetime.now(timezone.utc)), + ), + entries=entries, + previous_epoch=newest.epoch if newest else None, + ) + + claimed = await _claim_orphan_fence(store, binding_id, scope, False, plan) + if claimed.kind != "claimed" or claimed.record is None: + lost = claimed.kind == "lost" and claimed.record is not None + raise ReportingRowStoreError( + "STATE_UNAVAILABLE", + ( + f"another database claimed epoch {claimed.record.epoch} while this takeover " + "ran; run it again" + if lost and claimed.record is not None + else "the takeover was refused" + ), + ) + taken.append( + ReportingOrphanFenceTakeoverV1( + tuple(binding_ids), claimed.record.epoch, claimed.previous_epoch + ) + ) + return taken + + +# -- scheduling ------------------------------------------------------------------------------ + + +def _idle_result(options: ReportingOrphanCollectionOptionsV1) -> ReportingOrphanCollectionResultV1: + return ReportingOrphanCollectionResultV1(dry_run=options.dry_run) + + +async def run_orphan_collection_pass( + store: PgReportingLedgerStore, + options: ReportingOrphanCollectionOptionsV1, + state: ReportingOrphanCollectionStateV1, +) -> ReportingOrphanCollectionResultV1: + """One scheduled pass: resumes the listing where the last one stopped and honours the interval. + + ``state`` is owned by the caller and updated in place, so a large listing resumes where the + last pass stopped and ``interval_seconds`` is honoured. + """ + if state.next_cycle_at is not None and time.monotonic() < state.next_cycle_at: + # Between full listings: nothing to do this pass. + return _idle_result(options) + outcome = await collect_orphaned_row_objects( + store, + dry_run=options.dry_run, + acknowledge_exclusive_namespace=options.acknowledge_exclusive_namespace, + min_age_seconds=options.min_age_seconds, + limit=options.limit, + include_read_only=options.include_read_only, + cursors=dict(state.cursors), + skip_bindings=list(state.completed), + ) + state.cursors = dict(outcome.cursors) + # A finished, clean cycle rests for the interval; anything else carries on next pass, without + # listing again the bindings that already finished. (A delete that keeps failing does not stop + # the rest: it is tried again on the next listing.) + if not outcome.cursors and not any(item.status == "failed" for item in outcome.bindings): + state.completed = [] + # A fenced binding looks again within the hour, so applying the migration or taking the + # namespace over does not leave collection idle for the rest of a day. + rest = options.interval_seconds + if any(item.status == "fenced" for item in outcome.bindings): + rest = min(rest, FENCED_RECHECK_SECONDS) + state.next_cycle_at = time.monotonic() + rest + else: + state.completed = list(dict.fromkeys([*state.completed, *outcome.completed])) + state.next_cycle_at = None + return outcome diff --git a/packages/adcp-reporting-storage/src/adcp_reporting_storage/ledger/orphan_fence.py b/packages/adcp-reporting-storage/src/adcp_reporting_storage/ledger/orphan_fence.py new file mode 100644 index 0000000000..db2c984a94 --- /dev/null +++ b/packages/adcp-reporting-storage/src/adcp_reporting_storage/ledger/orphan_fence.py @@ -0,0 +1,323 @@ +"""The orphan collector's ownership fence (adcp-client#3593, adcp-client-python#1829). + +The collector deletes objects under ``{prefix}/{namespace_key}/`` that no ledger record names. +A clone or point-in-time restore of the ledger database carries the same configuration, +installation identity and namespace key, but an older set of references, so its collector would +delete the live database's newer objects. Nothing inside the database can tell the two apart; the +object store they share can. + +The fence is a chain of create-only objects, one per epoch, under the binding's owned prefix:: + + {prefix}/{namespace_key}/_collector-fence/0000000000000007.json + +Each holds ``{v, namespace_key, epoch, token, previous_token, claimed_at}``. The highest epoch is +the current claim. The database records the ``(epoch, token)`` it last claimed in +``adcp_reporting_orphan_collector_fences``. A real pass deletes only when the highest epoch in the +store is exactly the claim this database recorded, then claims the next epoch (create-only, so +exactly one claimant wins) before it deletes anything. Every other state refuses with a cause. A +database that was copied before the original's last claim sees a higher epoch and refuses; so does +a restore, until an operator calls ``take_over_orphan_collector_fence``. + +This module is the pure part: the decision table, the key and record formats and the scope. The +I/O and the transaction live in :mod:`adcp_reporting_storage.ledger.orphan_collector`. The bytes +and the scope hash are the JavaScript SDK's, so either SDK reads the other's fence. +""" + +from __future__ import annotations + +import hashlib +import json +import re +from collections.abc import Mapping +from dataclasses import dataclass +from typing import Literal + +import rfc8785 + +from adcp_reporting_storage.ledger.row_storage_object import is_row_instant + +__all__ = [ + "ORPHAN_FENCE_DIRECTORY", + "ORPHAN_FENCE_MAX_BYTES", + "OrphanFenceDecision", + "OrphanFenceEntry", + "OrphanFenceRecord", + "OrphanFenceRow", + "ReportingOrphanFenceCauseV1", + "ReportingOrphanFenceRefusalV1", + "decide_orphan_fence", + "encode_orphan_fence_record", + "is_orphan_fence_key", + "orphan_fence_directory", + "orphan_fence_epoch_of_key", + "orphan_fence_key", + "orphan_fence_scope", + "parse_orphan_fence_record", +] + +#: Object-store directory (under the owned prefix) that holds the fence chain. +ORPHAN_FENCE_DIRECTORY = "_collector-fence" +#: Fence objects are tiny; anything larger is not one. +ORPHAN_FENCE_MAX_BYTES = 4096 + +_EPOCH_DIGITS = 16 +_UUID = re.compile(r"[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}") +_KEY_NAME = re.compile(rf"([0-9]{{{_EPOCH_DIGITS}}})\.json") +_FIELDS = frozenset({"claimed_at", "epoch", "namespace_key", "previous_token", "token", "v"}) + +#: Why a binding's collector refused to delete. +#: +#: * ``claimed_elsewhere``: this database never claimed the namespace, but the object store holds +#: a claim. A clone that has not collected before, a database whose fence table was lost, or +#: another installation. +#: * ``fence_advanced``: the object store holds a newer claim than this database recorded. Another +#: database (usually the live one this was cloned from) has collected since this database's +#: state was taken. +#: * ``token_mismatch``: the object store holds this database's epoch under a different token. Two +#: databases claimed the same epoch; the other one won. +#: * ``fence_missing``: this database recorded a claim and the object store holds none: the bucket +#: or prefix was reset or is a different one. +#: * ``fence_regressed``: the object store holds an older claim than this database recorded: a +#: database restored from a newer state, or a bucket restored from an older one. +#: * ``fence_unreadable``: the fence objects are malformed, oversized, duplicated or name another +#: namespace. +#: * ``claim_lost``: this pass tried to claim the next epoch and another database got there first. +#: * ``migration_required``: the fence table does not exist; apply ``reporting_row_storage.sql``. +ReportingOrphanFenceCauseV1 = Literal[ + "claimed_elsewhere", + "fence_advanced", + "token_mismatch", + "fence_missing", + "fence_regressed", + "fence_unreadable", + "claim_lost", + "migration_required", +] + + +@dataclass(frozen=True, slots=True) +class ReportingOrphanFenceRefusalV1: + """Why a binding was fenced, reported on its collection outcome.""" + + cause: ReportingOrphanFenceCauseV1 + #: What to check, and how to take the namespace over if this database really is its only owner. + detail: str + #: Highest epoch found in the object store, when readable. + fence_epoch: int | None = None + #: Epoch this database last recorded, when it has one. + recorded_epoch: int | None = None + + +@dataclass(frozen=True, slots=True) +class OrphanFenceEntry: + """One fence object found by a listing.""" + + epoch: int + object_ref: str + native_version_ref: str + + +@dataclass(frozen=True, slots=True) +class OrphanFenceRecord: + """The contents of one fence object.""" + + epoch: int + token: str + previous_token: str | None + claimed_at: str + + +@dataclass(frozen=True, slots=True) +class OrphanFenceRow: + """What this database recorded for a fence scope.""" + + epoch: int + token: str + previous_token: str | None + + +@dataclass(frozen=True, slots=True) +class OrphanFenceDecision: + """``claim``: this database may collect. A real pass claims ``epoch`` first (the next epoch + after its own claim, or 1 when the namespace was never claimed); a read-only check just + proceeds. ``complete``: this database recorded a claim and crashed or failed before its object + was written: write it now. ``fenced``: refuse, with ``refusal``.""" + + action: Literal["claim", "complete", "fenced"] + epoch: int | None = None + previous_token: str | None = None + refusal: ReportingOrphanFenceRefusalV1 | None = None + + +_TAKEOVER_HINT = ( + "If this database is the only one that owns this namespace (for example after restoring it " + "from a backup, and the original is gone), call " + "take_over_orphan_collector_fence(acknowledge_previous_owner_gone=True). Otherwise leave " + "orphan collection off on this copy." +) + + +def decide_orphan_fence( + row: OrphanFenceRow | None, latest: tuple[int, str] | None +) -> OrphanFenceDecision: + """Decide, from what this database recorded and what the object store holds, whether it may + collect. Pure: the caller does the I/O and the transaction. + + ``latest`` is the highest ``(epoch, token)`` in the object store, or ``None`` when it holds no + fence. + """ + + def refuse(cause: ReportingOrphanFenceCauseV1, detail: str) -> OrphanFenceDecision: + return OrphanFenceDecision( + "fenced", + refusal=ReportingOrphanFenceRefusalV1( + cause, + f"{detail}. {_TAKEOVER_HINT}", + latest[0] if latest else None, + row.epoch if row else None, + ), + ) + + if row is None: + if latest is None: + return OrphanFenceDecision("claim", epoch=1) + return refuse( + "claimed_elsewhere", + f"the namespace is already claimed (epoch {latest[0]}) by a database this one has no " + "record of: it is a clone or restore of that database, or another installation that " + "shares the namespace", + ) + if latest is None: + # An unconfirmed first claim: this database recorded epoch 1 and did not write the object. + if row.epoch == 1 and row.previous_token is None: + return OrphanFenceDecision("complete") + return refuse( + "fence_missing", + f"this database recorded epoch {row.epoch} and the object store holds no fence: the " + "bucket or prefix was reset or is a different one", + ) + latest_epoch, latest_token = latest + if latest_epoch == row.epoch: + if latest_token == row.token: + return OrphanFenceDecision("claim", epoch=row.epoch + 1, previous_token=row.token) + return refuse( + "token_mismatch", + f"epoch {row.epoch} in the object store belongs to another claim: two databases " + "claimed the same epoch and the other one won", + ) + if latest_epoch > row.epoch: + return refuse( + "fence_advanced", + f"the object store holds epoch {latest_epoch} and this database recorded epoch " + f"{row.epoch}: another database that shares this namespace has collected since this " + "database was copied or restored", + ) + # latest_epoch < row.epoch. One behind with the recorded previous token is an unconfirmed + # claim of this database (it wrote the intent and not the object); anything else is a + # regression. + if ( + latest_epoch == row.epoch - 1 + and row.previous_token is not None + and latest_token == row.previous_token + ): + return OrphanFenceDecision("complete") + return refuse( + "fence_regressed", + f"the object store holds epoch {latest_epoch}, older than the epoch {row.epoch} this " + "database recorded: the database was restored from a newer state than the bucket, or " + "the bucket from an older one", + ) + + +def orphan_fence_directory(owned_prefix: str) -> str: + """The listing prefix of the fence chain under a binding's owned prefix.""" + return f"{owned_prefix}{ORPHAN_FENCE_DIRECTORY}/" + + +def orphan_fence_key(owned_prefix: str, epoch: int) -> str: + """Object key of one epoch's fence object under ``{prefix}/{namespace_key}/``.""" + return f"{orphan_fence_directory(owned_prefix)}{epoch:0{_EPOCH_DIGITS}d}.json" + + +def is_orphan_fence_key(owned_prefix: str, key: str) -> bool: + """Whether a listed key is part of the fence chain.""" + return key.startswith(orphan_fence_directory(owned_prefix)) + + +def orphan_fence_epoch_of_key(owned_prefix: str, key: str) -> int | None: + """The epoch a fence key names, or ``None`` for any other key under the fence directory.""" + directory = orphan_fence_directory(owned_prefix) + if not key.startswith(directory): + return None + match = _KEY_NAME.fullmatch(key[len(directory) :]) + if match is None: + return None + epoch = int(match.group(1)) + return epoch if epoch >= 1 else None + + +def encode_orphan_fence_record(namespace_key: str, record: OrphanFenceRecord) -> bytes: + """The fence object's bytes: the JSON the JavaScript SDK writes, one line.""" + return ( + json.dumps( + { + "v": 1, + "namespace_key": namespace_key, + "epoch": record.epoch, + "token": record.token, + "previous_token": record.previous_token, + "claimed_at": record.claimed_at, + }, + separators=(",", ":"), + ensure_ascii=False, + ) + + "\n" + ).encode("utf-8") + + +def parse_orphan_fence_record( + data: bytes, *, namespace_key: str, epoch: int +) -> OrphanFenceRecord | None: + """Parse a fence object strictly. + + ``None`` for anything that is not exactly a fence of this namespace and epoch. + """ + if len(data) > ORPHAN_FENCE_MAX_BYTES: + return None + try: + value = json.loads(data.decode("utf-8")) + except (ValueError, UnicodeError): + return None + if not isinstance(value, dict) or set(value) != _FIELDS: + return None + token = value["token"] + previous = value["previous_token"] + if ( + value["v"] != 1 + or type(value["v"]) is not int + or value["namespace_key"] != namespace_key + or type(value["epoch"]) is not int + or value["epoch"] != epoch + or not isinstance(token, str) + or _UUID.fullmatch(token) is None + or not ( + previous is None + or (isinstance(previous, str) and _UUID.fullmatch(previous) is not None) + ) + or not is_row_instant(value["claimed_at"]) + ): + return None + return OrphanFenceRecord(epoch, token, previous, value["claimed_at"]) + + +def orphan_fence_scope( + *, provider: str, location: Mapping[str, str], prefix: str, namespace_key: str +) -> str: + """Identity of one physical fence: the provider, its location, the prefix and the namespace. + + Bindings that share all four share a fence, because a collector listing through either one + reaches the same objects. + """ + return hashlib.sha256( + rfc8785.dumps(["adcp.orphan_fence.v1", provider, dict(location), prefix, namespace_key]) + ).hexdigest() diff --git a/packages/adcp-reporting-storage/src/adcp_reporting_storage/ledger/pg.py b/packages/adcp-reporting-storage/src/adcp_reporting_storage/ledger/pg.py index fc5c442d40..c84109c437 100644 --- a/packages/adcp-reporting-storage/src/adcp_reporting_storage/ledger/pg.py +++ b/packages/adcp-reporting-storage/src/adcp_reporting_storage/ledger/pg.py @@ -76,7 +76,7 @@ import base64 import hashlib import json -from collections.abc import AsyncIterator, Awaitable, Callable, Sequence +from collections.abc import AsyncIterator, Awaitable, Callable, Mapping, Sequence from contextlib import asynccontextmanager from contextvars import ContextVar from copy import deepcopy @@ -170,6 +170,11 @@ ReportingChangeKindV1, ReportingChangesPageV1, ) + from adcp_reporting_storage.ledger.orphan_collector import ( + ReportingOrphanCollectionOptionsV1, + ReportingOrphanCollectionResultV1, + ReportingOrphanFenceTakeoverV1, + ) from adcp_reporting_storage.ledger.period_retirement import ReportingPeriodRetirementV1 from adcp_reporting_storage.ledger.row_migration import ReportingRowMigrationBatchV1 from adcp_reporting_storage.ledger.row_retention import ( @@ -251,6 +256,7 @@ def __init__( notifications: bool = False, row_storage: bool | ReportingRowStorageOptionsV1 | None = None, change_feed: bool = False, + orphan_collection: ReportingOrphanCollectionOptionsV1 | None = None, ) -> None: if not PG_AVAILABLE: raise ImportError(_INSTALL_HINT) @@ -264,6 +270,20 @@ def __init__( row_storage if isinstance(row_storage, ReportingRowStorageOptionsV1) else None ) ) + # Opt-in listing collector for row objects the write-intent sweep cannot see, scheduled by + # the production worker. It needs row storage; its position lives in this process. + if orphan_collection is not None: + from adcp_reporting_storage.ledger.orphan_collector import ( + ReportingOrphanCollectionOptionsV1, + ReportingOrphanCollectionStateV1, + ) + + if not isinstance(orphan_collection, ReportingOrphanCollectionOptionsV1): + raise TypeError("orphan_collection must be ReportingOrphanCollectionOptionsV1") + if self._row_storage is None: + raise ValueError("orphan_collection requires row_storage") + self._orphan_state = ReportingOrphanCollectionStateV1() + self._orphan_collection = orphan_collection self._row_sets_known = False self._tombstones_known = False # Opt-in shared host change feed (adcp_reporting_changes), PostgreSQL 13+. @@ -425,8 +445,16 @@ def maintenance_operations(self) -> tuple[tuple[str, Callable[[], Awaitable[Any] operations.append(("period_retirement", lambda: self.retire_expired_periods(limit=20))) if self._change_feed: operations.append(("change_pruning", lambda: self.prune_changes())) + if self._orphan_collection is not None: + operations.append(("orphan_collection", self._run_orphan_collection)) return tuple(operations) + async def _run_orphan_collection(self) -> ReportingOrphanCollectionResultV1: + from adcp_reporting_storage.ledger.orphan_collector import run_orphan_collection_pass + + assert self._orphan_collection is not None + return await run_orphan_collection_pass(self, self._orphan_collection, self._orphan_state) + @asynccontextmanager async def source_publication( self, account_id: str, *, seals: ReportingSealStore | None = None @@ -539,6 +567,72 @@ async def sweep_row_write_intents( self, limit=limit, account_id=account_id, overdue_after_seconds=overdue_after_seconds ) + async def collect_orphaned_row_objects( + self, + *, + min_age_seconds: int | None = None, + limit: int = 1_000, + page_size: int | None = None, + dry_run: bool = False, + include_read_only: bool = False, + bindings: Sequence[str] | None = None, + cursors: Mapping[str, str] | None = None, + skip_bindings: Sequence[str] | None = None, + acknowledge_exclusive_namespace: bool = False, + now: datetime | None = None, + ) -> ReportingOrphanCollectionResultV1: + """Delete row objects that no ledger record or open upload names. + + Opt-in and listing-based: it covers what the write-intent sweep cannot see (an upload + whose ``PUT`` succeeded before its intent recorded it, intents for a binding no running + host configures, intents lost to a database restore). It lists each configured object + binding under ``{prefix}/{namespace_key}/`` and deletes an object only when it is a + well-formed row object, older than ``min_age_seconds`` (default 7 days, at least 1 day), + and no ledger record or open upload names it, under the per-object lock of every binding. + Deleting needs ``acknowledge_exclusive_namespace=True`` and is fenced against a clone or + restore of this database; ``dry_run`` deletes nothing. See + :func:`adcp_reporting_storage.ledger.orphan_collector.collect_orphaned_row_objects`. + """ + from adcp_reporting_storage.ledger.orphan_collector import collect_orphaned_row_objects + + return await collect_orphaned_row_objects( + self, + min_age_seconds=min_age_seconds, + limit=limit, + page_size=page_size, + dry_run=dry_run, + include_read_only=include_read_only, + bindings=bindings, + cursors=cursors, + skip_bindings=skip_bindings, + acknowledge_exclusive_namespace=acknowledge_exclusive_namespace, + now=now, + ) + + async def take_over_orphan_collector_fence( + self, + *, + acknowledge_previous_owner_gone: bool, + bindings: Sequence[str] | None = None, + include_read_only: bool = False, + ) -> list[ReportingOrphanFenceTakeoverV1]: + """Claim the orphan collector's namespace for this database, fencing every other copy out. + + The deliberate step after a restore, or when the database that held the fence is gone for + good. Stop the previous owner's collector first. See + :func:`adcp_reporting_storage.ledger.orphan_collector.take_over_orphan_collector_fence`. + """ + from adcp_reporting_storage.ledger.orphan_collector import ( + take_over_orphan_collector_fence, + ) + + return await take_over_orphan_collector_fence( + self, + acknowledge_previous_owner_gone=acknowledge_previous_owner_gone, + bindings=bindings, + include_read_only=include_read_only, + ) + async def prune_expired_row_sets( self, *, diff --git a/packages/adcp-reporting-storage/src/adcp_reporting_storage/ledger/reporting_row_storage.sql b/packages/adcp-reporting-storage/src/adcp_reporting_storage/ledger/reporting_row_storage.sql index 1181ce05f3..1fe2a715d4 100644 --- a/packages/adcp-reporting-storage/src/adcp_reporting_storage/ledger/reporting_row_storage.sql +++ b/packages/adcp-reporting-storage/src/adcp_reporting_storage/ledger/reporting_row_storage.sql @@ -369,3 +369,24 @@ UPDATE adcp_reporting_row_write_intents SELECT 1 FROM jsonb_array_elements(created_objects) AS element(created) WHERE jsonb_typeof(created) = 'object' AND created ? 'objectKey' ); + +-- The orphan collector's ownership fence (adcp-client#3593, adcp-client-python#1829): one row per +-- fence, recording the (epoch, token) this database last claimed in the object store under +-- {prefix}/{namespace_key}/_collector-fence/. collect_orphaned_row_objects deletes only while the +-- object store's highest fence epoch is exactly the claim recorded here, so a clone or restore of +-- this database that the original has since out-claimed refuses to delete. The same table and +-- columns the JavaScript SDK ships, so either SDK reads the other's claim. +-- +-- Deliberately outside the required-schema manifest and REPORTING_ROW_STORAGE_TABLES: the collector +-- is opt-in, so a database that never enables it needs nothing from this table. Until it exists a +-- collector pass refuses every binding as fenced (migration_required) and deletes nothing. +CREATE TABLE IF NOT EXISTS adcp_reporting_orphan_collector_fences ( + fence_scope TEXT COLLATE "C" PRIMARY KEY CHECK (fence_scope ~ '^[0-9a-f]{64}$'), + epoch BIGINT NOT NULL CHECK (epoch >= 1), + token TEXT COLLATE "C" NOT NULL + CHECK (token ~ '^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$'), + previous_token TEXT COLLATE "C" + CHECK (previous_token IS NULL + OR previous_token ~ '^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$'), + claimed_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp() +); diff --git a/packages/adcp-reporting-storage/src/adcp_reporting_storage/ledger/row_storage.py b/packages/adcp-reporting-storage/src/adcp_reporting_storage/ledger/row_storage.py index a7415ddb2d..af9f32f31d 100644 --- a/packages/adcp-reporting-storage/src/adcp_reporting_storage/ledger/row_storage.py +++ b/packages/adcp-reporting-storage/src/adcp_reporting_storage/ledger/row_storage.py @@ -53,18 +53,35 @@ ReportingRowStoreError, ReportingRowStoreErrorCode, ) +from adcp_reporting_storage.ledger.orphan_fence import ( + ORPHAN_FENCE_MAX_BYTES, + OrphanFenceEntry, + OrphanFenceRecord, + encode_orphan_fence_record, + is_orphan_fence_key, + orphan_fence_directory, + orphan_fence_epoch_of_key, + orphan_fence_key, + orphan_fence_scope, + parse_orphan_fence_record, +) from adcp_reporting_storage.ledger.row_storage_object import ( REPORTING_ROW_BINDING_ID_MAX_LENGTH, REPORTING_ROW_CREDENTIAL_REF_MAX_LENGTH, REPORTING_ROW_DEFAULT_KEY_TEMPLATE, + REPORTING_ROW_OBJECT_REF_MAX_LENGTH, ReportingRowObjectContextV1, ReportingRowObjectDeleteInputV1, ReportingRowObjectGetInputV1, + ReportingRowObjectListEntryV1, + ReportingRowObjectListInputV1, ReportingRowObjectProviderV1, ReportingRowObjectPutInputV1, ReportingRowObjectStatInputV1, assert_reporting_row_key_template_v1, assert_reporting_row_prefix_v1, + is_row_instant, + is_row_object_key_shape, render_reporting_row_object_key_v1, ) from adcp_reporting_storage.ledger.row_storage_provider_io import is_row_identifier @@ -373,6 +390,28 @@ def _inflate(physical: bytes, compression: Compression, byte_count: int) -> byte return None +@dataclass(frozen=True, slots=True) +class ReportingObjectBindingSummaryV1: + """An object binding this host configures, as the orphan collector sees it.""" + + id: str + read_only: bool + #: Whether the binding's provider implements ``list``. + listable: bool + + +@dataclass(frozen=True, slots=True) +class ReportingRowObjectListingV1: + """One page of the objects under a binding's owned prefix, and nothing else.""" + + owned_prefix: str + objects: tuple[ReportingRowObjectListEntryV1, ...] + #: Entries dropped: outside the owned prefix, malformed, or left out by the provider as + #: unaddressable. + foreign: int + next_page_token: str | None = None + + class ReportingRowStorageV1: """Reads and writes revision row sets over a caller-owned connection. @@ -1052,6 +1091,321 @@ async def delete_objects(self, binding_id: str, objects: Sequence[tuple[str, str self._options.write_deadline_seconds, ) + # -- listing and the collector fence ---------------------------------- + + def object_binding_ids(self) -> list[str]: + """The object bindings this host configures, read-only ones included.""" + return [ + binding_id + for binding_id, binding in self._bindings.items() + if isinstance(binding, ReportingObjectRowBindingV1) + ] + + def object_binding_summaries(self) -> list[ReportingObjectBindingSummaryV1]: + """Each object binding: whether it is ``read_only`` and whether its provider can list.""" + return [ + ReportingObjectBindingSummaryV1( + binding_id, + binding.read_only, + callable(getattr(self._providers.get(binding.provider), "list", None)), + ) + for binding_id, binding in self._bindings.items() + if isinstance(binding, ReportingObjectRowBindingV1) + ] + + def _lister_binding( + self, binding_id: str + ) -> tuple[ReportingObjectRowBindingV1, ReportingRowObjectProviderV1, Any]: + """The binding, its provider and the provider's ``list``; refuses one that cannot list.""" + if self._installation is None: + raise ReportingRowStoreError("STATE_UNAVAILABLE", "row storage is not ready") + binding = self._bindings.get(binding_id) + if not isinstance(binding, ReportingObjectRowBindingV1): + raise ReportingRowStoreError( + "STATE_UNAVAILABLE", f"row binding {binding_id} is not configured for objects" + ) + provider = self._providers[binding.provider] + lister = getattr(provider, "list", None) + if not callable(lister): + raise ReportingRowStoreError( + "INVALID_INPUT", f"row binding {binding_id}'s provider cannot list objects" + ) + return binding, provider, lister + + async def list_objects( + self, binding_id: str, *, page_size: int, page_token: str | None = None + ) -> ReportingRowObjectListingV1: + """One page of the objects under a binding's owned prefix, ``{prefix}/{namespace_key}/``. + + An entry the provider returns outside it, or one that is not a well-formed object + reference, is dropped and counted in ``foreign`` instead of being returned. The collector's + own fence chain lives under the owned prefix; it is not a row object and not foreign. + Raises ``INVALID_INPUT`` when the binding's provider has no ``list``. + """ + binding, _, lister = self._lister_binding(binding_id) + assert self._installation is not None + owned_prefix = f"{binding.prefix}/{self._installation[1]}/" + page = await self._provider_call( + lister( + ReportingRowObjectListInputV1( + binding.location, owned_prefix, page_size, page_token, binding.credential_ref + ), + ReportingRowObjectContextV1(self._options.read_deadline_seconds), + ), + self._options.read_deadline_seconds, + ) + if page is None or not isinstance(getattr(page, "objects", None), (tuple, list)): + raise ReportingRowStoreError( + "PROVIDER_UNAVAILABLE", "provider returned an invalid listing" + ) + token = getattr(page, "next_page_token", None) + if token is not None and (not isinstance(token, str) or not token): + raise ReportingRowStoreError( + "PROVIDER_UNAVAILABLE", "provider returned an invalid page token" + ) + if len(page.objects) > page_size: + raise ReportingRowStoreError( + "PROVIDER_UNAVAILABLE", "provider returned more objects than the page size" + ) + skipped = getattr(page, "skipped", 0) + foreign = skipped if type(skipped) is int and skipped > 0 else 0 + objects: list[ReportingRowObjectListEntryV1] = [] + for entry in page.objects: + object_ref = getattr(entry, "object_ref", None) + if isinstance(object_ref, str) and is_orphan_fence_key(owned_prefix, object_ref): + continue + version = getattr(entry, "native_version_ref", None) + size = getattr(entry, "size_bytes", None) + if ( + not is_row_object_key_shape(object_ref, REPORTING_ROW_OBJECT_REF_MAX_LENGTH) + or not isinstance(object_ref, str) + or not object_ref.startswith(owned_prefix) + or not isinstance(version, str) + or not 1 <= len(version) <= _NATIVE_VERSION_REF_MAX_LENGTH + or type(size) is not int + or size < 0 + or not is_row_instant(getattr(entry, "created_at", None)) + ): + foreign += 1 + continue + objects.append(entry) + return ReportingRowObjectListingV1(owned_prefix, tuple(objects), foreign, token) + + async def collector_fence_location(self, binding_id: str) -> tuple[str, str, str]: + """``(scope, directory, namespace_key)`` of an object binding's collector fence. + + ``scope`` keys the database row; ``directory`` holds the chain in the object store. + Bindings that share provider, location, prefix and installation share a fence. Raises + ``STATE_UNAVAILABLE`` for a binding that is not an object binding and ``INVALID_INPUT`` + when its provider cannot list (the fence is read by listing). + """ + binding, provider, _ = self._lister_binding(binding_id) + assert self._installation is not None + namespace_key = self._installation[1] + return ( + orphan_fence_scope( + # The provider's own name, not this host's registry key: two registry entries of + # one kind that reach the same bucket and prefix are one physical fence. + provider=provider.name, + location=binding.location, + prefix=binding.prefix, + namespace_key=namespace_key, + ), + orphan_fence_directory(f"{binding.prefix}/{namespace_key}/"), + namespace_key, + ) + + async def list_collector_fence( + self, binding_id: str, *, tolerant: bool = False + ) -> list[OrphanFenceEntry]: + """Every fence object of a binding's namespace, ordered by epoch. + + Keys under the fence directory that are not an epoch name are ignored. Two live versions + of one epoch's key (a versioned store after a manual delete and re-create) are ambiguous + and fail closed with ``ROWS_INTEGRITY_FAILED``, as does a chain too long to be one this + collector wrote (it prunes to two objects). ``tolerant`` is for a takeover, which must be + able to supersede a damaged chain: every version of every epoch is returned, so it can all + be removed, and the page bound is wider. + """ + _, directory, _ = await self.collector_fence_location(binding_id) + binding, _, lister = self._lister_binding(binding_id) + assert self._installation is not None + owned_prefix = f"{binding.prefix}/{self._installation[1]}/" + entries: dict[int, OrphanFenceEntry] = {} + versions: list[OrphanFenceEntry] = [] + page_size = 50 + token: str | None = None + page_number = 0 + while True: + if page_number >= (200 if tolerant else 20): + raise ReportingRowStoreError( + "ROWS_INTEGRITY_FAILED", "the collector fence directory holds too many objects" + ) + page_number += 1 + listed = await self._provider_call( + lister( + ReportingRowObjectListInputV1( + binding.location, directory, page_size, token, binding.credential_ref + ), + ReportingRowObjectContextV1(self._options.read_deadline_seconds), + ), + self._options.read_deadline_seconds, + ) + if ( + listed is None + or not isinstance(getattr(listed, "objects", None), (tuple, list)) + or len(listed.objects) > page_size + ): + raise ReportingRowStoreError( + "PROVIDER_UNAVAILABLE", "provider returned an invalid listing" + ) + skipped = getattr(listed, "skipped", 0) + if not tolerant and type(skipped) is int and skipped > 0: + # The provider left out an entry it could not address or age. In a directory that + # holds only this collector's own objects that could be the newest claim, and a + # check cannot tell "no claim" from "a claim I cannot see": fail closed. + raise ReportingRowStoreError( + "ROWS_INTEGRITY_FAILED", "the collector fence listing skipped an object" + ) + for item in listed.objects: + object_ref = getattr(item, "object_ref", None) + version = getattr(item, "native_version_ref", None) + if not isinstance(object_ref, str) or not isinstance(version, str): + continue + if not 1 <= len(version) <= _NATIVE_VERSION_REF_MAX_LENGTH: + continue + epoch = orphan_fence_epoch_of_key(owned_prefix, object_ref) + if epoch is None or object_ref != orphan_fence_key(owned_prefix, epoch): + continue + entry = OrphanFenceEntry(epoch, object_ref, version) + if tolerant: + versions.append(entry) + if epoch in entries: + if tolerant: + continue + raise ReportingRowStoreError( + "ROWS_INTEGRITY_FAILED", + "the collector fence holds two versions of one epoch", + ) + entries[epoch] = entry + following = getattr(listed, "next_page_token", None) + if following is None: + break + if not isinstance(following, str) or not following or following == token: + raise ReportingRowStoreError( + "PROVIDER_UNAVAILABLE", "provider returned an invalid page token" + ) + token = following + return sorted(versions if tolerant else entries.values(), key=lambda item: item.epoch) + + async def read_collector_fence( + self, binding_id: str, entry: OrphanFenceEntry + ) -> OrphanFenceRecord: + """Read and verify one fence object. + + A missing version is ``PROVIDER_UNAVAILABLE`` (it was listed a moment ago, so this is a + race to retry); content that is not exactly a fence of this namespace and epoch is + ``ROWS_INTEGRITY_FAILED``. + """ + binding, provider = self._object_binding(binding_id) + assert self._installation is not None + try: + data = await self._provider_call( + provider.get( + ReportingRowObjectGetInputV1( + binding.location, + entry.object_ref, + entry.native_version_ref, + ORPHAN_FENCE_MAX_BYTES, + None, + binding.credential_ref, + ), + ReportingRowObjectContextV1(self._options.read_deadline_seconds), + ), + self._options.read_deadline_seconds, + ) + except ReportingRowStoreError as error: + # An object over the read cap is the same finding as bad content. + if error.code == "ROWS_INTEGRITY_FAILED": + raise ReportingRowStoreError( + "ROWS_INTEGRITY_FAILED", "the collector fence object is malformed" + ) from None + raise + if data is None: + raise ReportingRowStoreError( + "PROVIDER_UNAVAILABLE", "a collector fence object disappeared while it was read" + ) + record = parse_orphan_fence_record( + data, namespace_key=self._installation[1], epoch=entry.epoch + ) + if record is None: + raise ReportingRowStoreError( + "ROWS_INTEGRITY_FAILED", "the collector fence object is malformed" + ) + return record + + async def put_collector_fence( + self, binding_id: str, record: OrphanFenceRecord + ) -> tuple[bool, OrphanFenceRecord]: + """Create-only write of one epoch's fence object. + + When the key is taken, returns the record already there (verified), so the caller can tell + its own earlier write from another database's claim: ``(created, record)``. + """ + binding, provider = self._object_binding(binding_id) + assert self._installation is not None + installation_id, namespace_key = self._installation + object_ref = orphan_fence_key(f"{binding.prefix}/{namespace_key}/", record.epoch) + put = await self._provider_call( + provider.put_if_absent( + ReportingRowObjectPutInputV1( + binding.location, + object_ref, + encode_orphan_fence_record(namespace_key, record), + "application/x-ndjson", + { + "adcp-installation": installation_id, + "adcp-collector-fence": str(record.epoch), + }, + binding.credential_ref, + ), + ReportingRowObjectContextV1(self._options.write_deadline_seconds), + ), + self._options.write_deadline_seconds, + ) + _assert_native_version_ref(put.native_version_ref) + if put.created: + return True, record + return False, await self.read_collector_fence( + binding_id, OrphanFenceEntry(record.epoch, object_ref, put.native_version_ref) + ) + + async def delete_collector_fence( + self, binding_id: str, entries: Sequence[OrphanFenceEntry] + ) -> None: + """Delete superseded fence objects. Best effort: a failure leaves a stale object that the + next claim prunes.""" + if not isinstance(self._bindings.get(binding_id), ReportingObjectRowBindingV1): + return + binding, provider = self._object_binding(binding_id) + context = ReportingRowObjectContextV1(self._options.write_deadline_seconds) + for entry in entries: + try: + await self._provider_call( + provider.delete( + ReportingRowObjectDeleteInputV1( + binding.location, + entry.object_ref, + entry.native_version_ref, + binding.credential_ref, + ), + context, + ), + self._options.write_deadline_seconds, + ) + except ReportingRowStoreError: + continue + # -- object lifetime -------------------------------------------------- # # Chunks are adopted create-only, so a custom key template that omits an identity @@ -1142,6 +1496,7 @@ async def delete_unreferenced( *, obligation_id: str | None = None, row_set_id: str | None = None, + any_binding: bool = False, ) -> int: """Delete each object nothing outside the operation's scope still needs. @@ -1152,20 +1507,37 @@ async def delete_unreferenced( the reference check and the provider delete happen together, so a concurrent commit adopting the object either already references it (it is kept) or finds it gone and fails closed. Returns the number of objects deleted. + + ``any_binding`` widens both checks to every binding and takes every binding's object lock, + in a fixed order: two bindings can share a bucket and prefix, and a writer adopting an + object takes the lock of its own binding only, so the delete must be serialized with an + adoption through any of them. Writers hold one such lock, so no cycle forms. The orphan + collector deletes this way. """ deleted = 0 for object_ref, native_version_ref in objects: async with pool.connection() as connection, connection.transaction(): - await connection.execute( - "SELECT pg_advisory_xact_lock(hashtextextended(%s, 0))", - (self._object_lock_text(binding_id, object_ref),), - ) + lock_bindings = [binding_id] + if any_binding: + known = await ( + await connection.execute( + "SELECT row_binding_id FROM adcp_reporting_row_bindings" + ) + ).fetchall() + lock_bindings = sorted( + {binding_id, *self.object_binding_ids(), *(row[0] for row in known)} + ) + for lock_binding in lock_bindings: + await connection.execute( + "SELECT pg_advisory_xact_lock(hashtextextended(%s, 0))", + (self._object_lock_text(lock_binding, object_ref),), + ) referenced = await ( await connection.execute( "(SELECT 1 FROM adcp_reporting_row_chunks chunk" " JOIN adcp_reporting_row_sets row_set" " ON row_set.row_set_id = chunk.row_set_id" - " WHERE row_set.row_binding_id = %(binding)s" + " WHERE (%(any_binding)s::boolean OR row_set.row_binding_id = %(binding)s)" " AND chunk.object_ref = %(key)s" " AND chunk.native_version_ref = %(version)s" " AND row_set.rows_state IN ('live', 'unavailable')" @@ -1175,12 +1547,14 @@ async def delete_unreferenced( " LIMIT 1)" " UNION ALL" " (SELECT 1 FROM adcp_reporting_row_write_intents intent" - " WHERE intent.row_binding_id = %(binding)s AND intent.state = 'open'" + " WHERE (%(any_binding)s::boolean OR intent.row_binding_id = %(binding)s)" + " AND intent.state = 'open'" " AND (intent.created_objects @> %(created)s::jsonb" " OR intent.created_objects @> %(created_legacy)s::jsonb)" " LIMIT 1)", { "binding": binding_id, + "any_binding": any_binding, "key": object_ref, "version": native_version_ref, "obligation": obligation_id, diff --git a/packages/adcp-reporting-storage/src/adcp_reporting_storage/ledger/row_storage_azure.py b/packages/adcp-reporting-storage/src/adcp_reporting_storage/ledger/row_storage_azure.py index a2d12e811c..f585a7f2c2 100644 --- a/packages/adcp-reporting-storage/src/adcp_reporting_storage/ledger/row_storage_azure.py +++ b/packages/adcp-reporting-storage/src/adcp_reporting_storage/ledger/row_storage_azure.py @@ -24,6 +24,7 @@ import re import uuid from collections.abc import Mapping +from datetime import datetime from typing import Any, Literal from adcp.reporting.ledger.row_storage_errors import ReportingRowStoreError @@ -31,11 +32,15 @@ ReportingRowObjectContextV1, ReportingRowObjectDeleteInputV1, ReportingRowObjectGetInputV1, + ReportingRowObjectListEntryV1, + ReportingRowObjectListInputV1, + ReportingRowObjectListPageV1, ReportingRowObjectProviderV1, ReportingRowObjectPutInputV1, ReportingRowObjectPutResultV1, ReportingRowObjectStatInputV1, ReportingRowObjectStatResultV1, + format_row_instant, ) from adcp_reporting_storage.ledger.row_storage_provider_io import ( assert_location_keys, @@ -366,6 +371,69 @@ def _stat(blob: Any, version: str) -> ReportingRowObjectStatResultV1 | None: raise ReportingRowStoreError("PROVIDER_UNAVAILABLE", "blob metadata is malformed") return ReportingRowObjectStatResultV1(version, size) + async def list( + self, request: ReportingRowObjectListInputV1, context: ReportingRowObjectContextV1 + ) -> ReportingRowObjectListPageV1: + container = _Budgeted( + self._container(request.location, request.credential_ref), context.timeout_seconds + ) + if not isinstance(request.prefix, str) or len(request.prefix) > 1024: + raise ReportingRowStoreError("INVALID_INPUT", "list prefix is invalid") + if type(request.page_size) is not int or request.page_size < 1: + raise ReportingRowStoreError("INVALID_INPUT", "page_size must be a positive integer") + return await asyncio.to_thread(self._list, container, request) + + @staticmethod + def _list( + container: Any, request: ReportingRowObjectListInputV1 + ) -> ReportingRowObjectListPageV1: + # Versioned listing: every (name, version_id) a delete could name, the current version + # included. + pages = container.list_blobs( + name_starts_with=request.prefix, + include=["versions"], + results_per_page=min(request.page_size, 5000), + ).by_page(continuation_token=request.page_token) + page = next(pages, None) + objects: list[ReportingRowObjectListEntryV1] = [] + skipped = 0 + for item in list(page) if page is not None else []: + name = getattr(item, "name", None) + version = getattr(item, "version_id", None) + size = getattr(item, "size", None) + # The later of creation and last modification: an overwrite can move the second + # without the first. + times = [ + moment + for moment in ( + getattr(item, "creation_time", None), + getattr(item, "last_modified", None), + ) + if isinstance(moment, datetime) and moment.tzinfo is not None + ] + # An entry that cannot be addressed or aged is left out: it is never collected, and + # one bad entry must not stop the listing of the rest. A blob written before + # versioning was enabled has no version ID and is never addressable by a recorded + # one. + if ( + not isinstance(name, str) + or not name.startswith(request.prefix) + or not isinstance(version, str) + or not _VERSION_ID.fullmatch(version) + or type(size) is not int + or size < 0 + or not times + ): + skipped += 1 + continue + objects.append( + ReportingRowObjectListEntryV1(name, version, size, format_row_instant(max(times))) + ) + token = getattr(pages, "continuation_token", None) + return ReportingRowObjectListPageV1( + tuple(objects), token if isinstance(token, str) and token else None, skipped + ) + async def delete( self, delete: ReportingRowObjectDeleteInputV1, context: ReportingRowObjectContextV1 ) -> Literal["deleted", "absent"]: diff --git a/packages/adcp-reporting-storage/src/adcp_reporting_storage/ledger/row_storage_gcs.py b/packages/adcp-reporting-storage/src/adcp_reporting_storage/ledger/row_storage_gcs.py index 782465687d..2c1bff8e87 100644 --- a/packages/adcp-reporting-storage/src/adcp_reporting_storage/ledger/row_storage_gcs.py +++ b/packages/adcp-reporting-storage/src/adcp_reporting_storage/ledger/row_storage_gcs.py @@ -17,6 +17,7 @@ import asyncio import re from collections.abc import Mapping +from datetime import datetime from typing import Any, Literal from adcp.reporting.ledger.row_storage_errors import ReportingRowStoreError @@ -24,11 +25,15 @@ ReportingRowObjectContextV1, ReportingRowObjectDeleteInputV1, ReportingRowObjectGetInputV1, + ReportingRowObjectListEntryV1, + ReportingRowObjectListInputV1, + ReportingRowObjectListPageV1, ReportingRowObjectProviderV1, ReportingRowObjectPutInputV1, ReportingRowObjectPutResultV1, ReportingRowObjectStatInputV1, ReportingRowObjectStatResultV1, + format_row_instant, ) from adcp_reporting_storage.ledger.row_storage_provider_io import ( assert_key, @@ -294,6 +299,68 @@ def _stat( raise ReportingRowStoreError("PROVIDER_UNAVAILABLE", "object metadata is malformed") return ReportingRowObjectStatResultV1(str(generation), size) + async def list( + self, request: ReportingRowObjectListInputV1, context: ReportingRowObjectContextV1 + ) -> ReportingRowObjectListPageV1: + bucket = self._bucket(request.location, request.credential_ref) + if not isinstance(request.prefix, str) or len(request.prefix.encode()) > 1024: + raise ReportingRowStoreError("INVALID_INPUT", "list prefix is invalid") + if type(request.page_size) is not int or request.page_size < 1: + raise ReportingRowStoreError("INVALID_INPUT", "page_size must be a positive integer") + return await asyncio.to_thread(self._list, bucket, request, context.timeout_seconds) + + @staticmethod + def _list( + bucket: Any, request: ReportingRowObjectListInputV1, timeout: float + ) -> ReportingRowObjectListPageV1: + # ``versions=True`` lists every generation, noncurrent ones included, so each + # (key, generation) a delete could name is visible. On a bucket without versioning it + # lists the live objects. + iterator = bucket.list_blobs( + prefix=request.prefix, + versions=True, + max_results=min(request.page_size, 1000), + page_token=request.page_token, + timeout=timeout, + ) + page = next(iter(iterator.pages), None) + objects: list[ReportingRowObjectListEntryV1] = [] + skipped = 0 + for blob in list(page) if page is not None else []: + name = getattr(blob, "name", None) + size = getattr(blob, "size", None) + generation = str(getattr(blob, "generation", "") or "") + created = getattr(blob, "time_created", None) + updated = getattr(blob, "updated", None) + # The later of the generation's creation and last metadata update: an object is + # never reported older than it is. + times = [ + moment + for moment in (created, updated) + if isinstance(moment, datetime) and moment.tzinfo is not None + ] + # An entry that cannot be addressed or aged is left out: it is never collected, and + # one bad entry must not stop the listing of the rest. + if ( + not isinstance(name, str) + or not name.startswith(request.prefix) + or type(size) is not int + or size < 0 + or not times + or not _GENERATION.fullmatch(generation) + ): + skipped += 1 + continue + objects.append( + ReportingRowObjectListEntryV1( + name, generation, size, format_row_instant(max(times)) + ) + ) + token = getattr(iterator, "next_page_token", None) + return ReportingRowObjectListPageV1( + tuple(objects), token if isinstance(token, str) and token else None, skipped + ) + async def delete( self, delete: ReportingRowObjectDeleteInputV1, context: ReportingRowObjectContextV1 ) -> Literal["deleted", "absent"]: diff --git a/packages/adcp-reporting-storage/src/adcp_reporting_storage/ledger/row_storage_object.py b/packages/adcp-reporting-storage/src/adcp_reporting_storage/ledger/row_storage_object.py index cfa1876cf2..704d726967 100644 --- a/packages/adcp-reporting-storage/src/adcp_reporting_storage/ledger/row_storage_object.py +++ b/packages/adcp-reporting-storage/src/adcp_reporting_storage/ledger/row_storage_object.py @@ -22,14 +22,18 @@ import uuid from collections.abc import Mapping from dataclasses import dataclass +from datetime import datetime, timezone from pathlib import Path -from typing import Literal, Protocol, runtime_checkable +from typing import Any, Literal, Protocol, runtime_checkable import rfc8785 from adcp.reporting.ledger.row_storage_errors import ReportingRowStoreError __all__ = [ + "ReportingRowObjectListEntryV1", + "ReportingRowObjectListInputV1", + "ReportingRowObjectListPageV1", "REPORTING_ROW_BINDING_ID_MAX_LENGTH", "REPORTING_ROW_CREDENTIAL_REF_MAX_LENGTH", "REPORTING_ROW_DEFAULT_KEY_TEMPLATE", @@ -133,6 +137,43 @@ class ReportingRowObjectStatResultV1: size_bytes: int +@dataclass(frozen=True, slots=True) +class ReportingRowObjectListInputV1: + location: Mapping[str, str] + #: Key prefix to list under. The SDK passes the binding's owned prefix, + #: ``{prefix}/{namespace_key}/``. + prefix: str + #: Upper bound on objects in the page. A provider may return fewer and still return a token. + page_size: int + #: Opaque continuation from the previous page; ``None`` for the first page. + page_token: str | None = None + credential_ref: str | None = None + + +@dataclass(frozen=True, slots=True) +class ReportingRowObjectListEntryV1: + """One stored object version found by a listing.""" + + #: AdCP ``object_ref``: the key, as it would be passed to ``get`` and ``delete``. + object_ref: str + #: AdCP ``native_version_ref``: the exact version, as ``put_if_absent`` would have recorded it. + native_version_ref: str + size_bytes: int + #: When this version was written, as an RFC 3339 UTC instant (``...Z``). Never earlier than + #: the write. + created_at: str + + +@dataclass(frozen=True, slots=True) +class ReportingRowObjectListPageV1: + objects: tuple[ReportingRowObjectListEntryV1, ...] + #: Absent on the last page. + next_page_token: str | None = None + #: Entries the provider left out because they cannot be addressed or aged (no usable version, + #: size or write time). They are never collected; one bad entry must not stop the listing. + skipped: int = 0 + + @runtime_checkable class ReportingRowObjectProviderV1(Protocol): """The port a row object store implements. @@ -192,6 +233,21 @@ async def delete( # versions are write-unique, so the same version at the recorded size is the same bytes) # without downloading it under the account lock. A provider without ``stat`` falls back to # a full read there. Never return a size for a different version. + # + # async def list(self, request: ReportingRowObjectListInputV1, context) + # -> ReportingRowObjectListPageV1 + # + # Every stored version whose key starts with ``prefix``, in a stable order, one page at a + # time. The orphan collector (``collect_orphaned_row_objects``) and its ownership fence use + # it; a binding whose provider has no ``list`` is reported ``unsupported`` there. Return each + # version once as ``(object_ref, native_version_ref)`` exactly as ``put_if_absent`` recorded + # it (on a versioned store, noncurrent versions too, never delete markers), the stored size + # and the time the version was written. Return only keys under ``prefix``. A page may be + # short; ``next_page_token`` is absent on the last page. A token is only valid for the same + # prefix and location. Leave out an entry that cannot be addressed or aged and count it in + # ``skipped`` rather than failing the page. The listing must be strongly consistent for keys + # ``put_if_absent`` has already returned (read-after-write): the collector's fence reads the + # newest claim by listing. #: Placeholders allowed in object key templates. @@ -213,6 +269,54 @@ async def delete( _KEY = re.compile(r"^[A-Za-z0-9_.=-]+(?:/[A-Za-z0-9_.=-]+)*$") _PLACEHOLDER = re.compile(r"\{([a-z0-9_]+)\}") +_INSTANT = re.compile( + r"([0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2})(?:\.([0-9]{1,9}))?Z" +) + + +def is_row_object_key_shape( + key: object, max_length: int = REPORTING_ROW_OBJECT_REF_MAX_LENGTH +) -> bool: + """Whether a key has the shape ``render_reporting_row_object_key_v1`` gives every row object. + + The same character set, no empty, ``.`` or ``..`` segment, and within AdCP's ``object_ref`` + bound. ``fullmatch``: ``$`` would also accept a trailing newline. + """ + return ( + isinstance(key, str) + and len(key) <= max_length + and _KEY.fullmatch(key) is not None + and not any(segment in (".", "..") for segment in key.split("/")) + ) + + +def parse_row_instant(value: object) -> datetime | None: + """A strict RFC 3339 UTC instant (``2026-10-01T00:00:00.123Z``) as an aware datetime. + + Anything else, including an offset other than ``Z``, a date-only value or an impossible + date, is ``None``: an object whose write time cannot be read is treated as young, never + deleted on a guess. + """ + if not isinstance(value, str): + return None + match = _INSTANT.fullmatch(value) + if match is None: + return None + fraction = (match.group(2) or "").ljust(6, "0")[:6] + try: + return datetime.fromisoformat(f"{match.group(1)}.{fraction}+00:00") + except ValueError: + return None + + +def is_row_instant(value: object) -> bool: + return parse_row_instant(value) is not None + + +def format_row_instant(moment: datetime) -> str: + """An aware datetime as the millisecond RFC 3339 UTC instant providers report.""" + utc = moment.astimezone(timezone.utc) + return utc.strftime("%Y-%m-%dT%H:%M:%S.") + f"{utc.microsecond // 1000:03d}Z" _REVISION_KEY_REQUIRED = ( @@ -629,6 +733,101 @@ def _stat(target: str, native_version_ref: str) -> ReportingRowObjectStatResultV return None return ReportingRowObjectStatResultV1(version, int(version.rsplit(":", 1)[1])) + async def list( + self, request: ReportingRowObjectListInputV1, context: ReportingRowObjectContextV1 + ) -> ReportingRowObjectListPageV1: + """List objects under ``prefix`` in key order, one page at a time. + + Reads and sorts each directory it walks in full, so keep directories small: it is for + tests and single hosts, not millions of objects per prefix. + """ + if type(request.page_size) is not int or request.page_size < 1: + raise ReportingRowStoreError("INVALID_INPUT", "page_size must be a positive integer") + root = self.roots.get(request.location.get("root", "")) + if root is None: + raise ReportingRowStoreError("UNSAFE_BINDING", "filesystem root is not configured") + if not isinstance(request.prefix, str) or ".." in request.prefix.split("/"): + raise ReportingRowStoreError("INVALID_INPUT", "list prefix is invalid") + return await asyncio.to_thread(self._list, root, request) + + @staticmethod + def _list(root: str, request: ReportingRowObjectListInputV1) -> ReportingRowObjectListPageV1: + after = None if request.page_token is None else tuple(request.page_token.split("/")) + objects: list[ReportingRowObjectListEntryV1] = [] + more = False + + def walk(directory: tuple[str, ...]) -> bool: + nonlocal more + try: + with os.scandir(os.path.join(root, *directory)) as scanner: + # A symlink is neither listed nor followed: a link inside the root must not + # make a key resolve outside it. + entries = sorted( + (entry for entry in scanner if not entry.is_symlink()), + key=lambda entry: entry.name, + ) + except (FileNotFoundError, NotADirectoryError): + return True + except OSError: + raise ReportingRowStoreError( + "PROVIDER_UNAVAILABLE", "filesystem list failed" + ) from None + for entry in entries: + name = entry.name + segments = (*directory, name) + path = entry.path + is_directory = entry.is_dir(follow_symlinks=False) + if after is not None: + head = after[: len(segments)] + if segments < head: + continue + if segments == head and not is_directory and len(segments) == len(after): + continue + if is_directory: + # Keep only subtrees that can hold a key under the prefix. + rendered = "/".join(segments) + "/" + if not rendered.startswith(request.prefix) and not request.prefix.startswith( + rendered + ): + continue + if not walk(segments): + return False + continue + if name.endswith(".tmp") or not entry.is_file(follow_symlinks=False): + continue + key = "/".join(segments) + if not key.startswith(request.prefix): + continue + if len(objects) >= request.page_size: + more = True + return False + try: + stat = os.stat(path) + version = _filesystem_version(path) + birth = _birth_time_ns(path, stat) + except FileNotFoundError: + continue + except OSError: + raise ReportingRowStoreError( + "PROVIDER_UNAVAILABLE", "filesystem list failed" + ) from None + written_ns = max(birth, stat.st_mtime_ns) + objects.append( + ReportingRowObjectListEntryV1( + key, + version, + stat.st_size, + format_row_instant( + datetime.fromtimestamp(written_ns / 1_000_000_000, tz=timezone.utc) + ), + ) + ) + return True + + walk(()) + token = objects[-1].object_ref if more and objects else None + return ReportingRowObjectListPageV1(tuple(objects), token) + async def delete( self, delete: ReportingRowObjectDeleteInputV1, context: ReportingRowObjectContextV1 ) -> Literal["deleted", "absent"]: @@ -785,4 +984,111 @@ def stat_input(version: str) -> ReportingRowObjectStatInputV1: outcome = await provider.delete(delete(recreated.native_version_ref), context) assert outcome == "deleted", "delete did not remove the re-created version" passed.append("write-unique-version") + lister = getattr(provider, "list", None) + if lister is not None: + await _check_listing(lister, location, prefix, credential_ref, provider, context) + passed.append("list") return passed + + +async def _check_listing( + lister: Any, + location: Mapping[str, str], + prefix: str, + credential_ref: str | None, + provider: ReportingRowObjectProviderV1, + context: ReportingRowObjectContextV1, +) -> None: + """The listing contract the orphan collector depends on. + + Every object under the prefix exactly once across pages, each with the version, size and + write time ``put_if_absent`` reports, nothing outside the prefix, and deleted objects gone. + """ + folder = f"{prefix}/conformance/{uuid.uuid4()}/" + bodies = { + f"{folder}a.jsonl": b'{"a":1}\n', + f"{folder}b.jsonl.gz": b'{"b":2}\n{"b":3}\n', + f"{folder}nested/c.jsonl": b'{"c":4}\n', + } + metadata = {"adcp-installation": "conformance", "adcp-intent": "conformance"} + created: dict[str, str] = {} + started = datetime.now(timezone.utc) + + async def list_all(under: str, page_size: int) -> list[ReportingRowObjectListEntryV1]: + found: list[ReportingRowObjectListEntryV1] = [] + token: str | None = None + for _ in range(100): + page = await lister( + ReportingRowObjectListInputV1(location, under, page_size, token, credential_ref), + context, + ) + assert ( + isinstance(page.objects, tuple) and len(page.objects) <= page_size + ), "list returned more objects than page_size" + found.extend(page.objects) + previous, token = token, page.next_page_token + if token is None: + return found + assert ( + isinstance(token, str) and token and token != previous + ), "list returned an invalid or repeated token" + raise AssertionError("list did not finish within 100 pages") + + try: + for key, data in bodies.items(): + put = await provider.put_if_absent( + ReportingRowObjectPutInputV1( + location, + key, + data, + "application/gzip" if key.endswith(".gz") else "application/x-ndjson", + metadata, + credential_ref, + ), + context, + ) + assert put.created, "list: put_if_absent did not create a fresh object" + created[key] = put.native_version_ref + for page_size in (1, 2, 100): + found = await list_all(folder, page_size) + keys = [entry.object_ref for entry in found] + assert len(set(keys)) == len( + keys + ), f"list returned an object twice (page_size {page_size})" + assert set(keys) == set( + bodies + ), f"list did not return exactly the objects under the prefix (page_size {page_size})" + for entry in found: + assert ( + entry.native_version_ref == created[entry.object_ref] + ), "list reported a native version that put_if_absent did not" + assert entry.size_bytes == len( + bodies[entry.object_ref] + ), "list reported a wrong size" + written = parse_row_instant(entry.created_at) + # Object-store clocks differ from this host's; a wide margin still catches a + # missing or zero time. + assert ( + written is not None and abs((written - started).total_seconds()) <= 600 + ), "list did not report when the version was written" + assert len(await list_all(f"{folder}nested/", 10)) == 1, "list ignored the prefix" + assert ( + len(await list_all(f"{folder}zz", 10)) == 0 + ), "list returned objects for an unused prefix" + first = next(iter(created)) + outcome = await provider.delete( + ReportingRowObjectDeleteInputV1(location, first, created[first], credential_ref), + context, + ) + assert outcome == "deleted", "list: delete did not remove the object" + del created[first] + remaining = sorted(entry.object_ref for entry in await list_all(folder, 2)) + assert remaining == sorted(created), "list returned a deleted object, or lost a live one" + finally: + for key, version in created.items(): + try: + await provider.delete( + ReportingRowObjectDeleteInputV1(location, key, version, credential_ref), context + ) + except Exception: # noqa: S110 # nosec B110 + pass diff --git a/packages/adcp-reporting-storage/src/adcp_reporting_storage/ledger/row_storage_s3.py b/packages/adcp-reporting-storage/src/adcp_reporting_storage/ledger/row_storage_s3.py index d3d96e4232..9484172ab2 100644 --- a/packages/adcp-reporting-storage/src/adcp_reporting_storage/ledger/row_storage_s3.py +++ b/packages/adcp-reporting-storage/src/adcp_reporting_storage/ledger/row_storage_s3.py @@ -20,9 +20,13 @@ from __future__ import annotations import asyncio +import base64 +import binascii +import json import re import uuid from collections.abc import Mapping +from datetime import datetime from typing import Any, Literal from adcp.reporting.ledger.row_storage_errors import ReportingRowStoreError @@ -30,11 +34,15 @@ ReportingRowObjectContextV1, ReportingRowObjectDeleteInputV1, ReportingRowObjectGetInputV1, + ReportingRowObjectListEntryV1, + ReportingRowObjectListInputV1, + ReportingRowObjectListPageV1, ReportingRowObjectProviderV1, ReportingRowObjectPutInputV1, ReportingRowObjectPutResultV1, ReportingRowObjectStatInputV1, ReportingRowObjectStatResultV1, + format_row_instant, ) from adcp_reporting_storage.ledger.row_storage_provider_io import ( assert_key, @@ -458,6 +466,92 @@ def _stat( raise ReportingRowStoreError("PROVIDER_UNAVAILABLE", "object metadata is malformed") return ReportingRowObjectStatResultV1(version, size) + async def list( + self, request: ReportingRowObjectListInputV1, context: ReportingRowObjectContextV1 + ) -> ReportingRowObjectListPageV1: + client, bucket = self._target(request.location, request.credential_ref) + if not isinstance(request.prefix, str) or len(request.prefix.encode()) > 1024: + raise ReportingRowStoreError("INVALID_INPUT", "list prefix is invalid") + if type(request.page_size) is not int or request.page_size < 1: + raise ReportingRowStoreError("INVALID_INPUT", "page_size must be a positive integer") + markers: tuple[str, str] | None = None + if request.page_token is not None: + try: + decoded = json.loads( + base64.urlsafe_b64decode( + request.page_token + "=" * (-len(request.page_token) % 4) + ) + ) + if not isinstance(decoded.get("k"), str) or not isinstance(decoded.get("v"), str): + raise ValueError("invalid") + markers = (decoded["k"], decoded["v"]) + except (ValueError, binascii.Error, UnicodeError, AttributeError): + raise ReportingRowStoreError( + "INVALID_INPUT", "list page token is invalid" + ) from None + return await asyncio.to_thread(self._list, client, bucket, request, markers) + + @staticmethod + def _list( + client: Any, + bucket: str, + request: ReportingRowObjectListInputV1, + markers: tuple[str, str] | None, + ) -> ReportingRowObjectListPageV1: + # Row buckets are versioned and the native version is the VersionId, so list versions: + # each (Key, VersionId) is what a delete would name. Delete markers are not objects. + parameters: dict[str, Any] = { + "Bucket": bucket, + "Prefix": request.prefix, + "MaxKeys": min(request.page_size, 1000), + } + if markers is not None: + parameters["KeyMarker"], parameters["VersionIdMarker"] = markers + output = client.list_object_versions(**parameters) + objects: list[ReportingRowObjectListEntryV1] = [] + skipped = 0 + for version in output.get("Versions") or []: + key = version.get("Key") + version_id = version.get("VersionId") + size = version.get("Size") + modified = version.get("LastModified") + # An entry that cannot be addressed or aged is left out: it is never collected, and + # one bad entry must not stop the listing of the rest. A version written before + # versioning was enabled is ``null`` and is never addressable by a recorded version. + if ( + not isinstance(key, str) + or not key.startswith(request.prefix) + or not isinstance(version_id, str) + or version_id == "null" + or not _VERSION_ID.fullmatch(version_id) + or type(size) is not int + or size < 0 + or not isinstance(modified, datetime) + or modified.tzinfo is None + ): + skipped += 1 + continue + objects.append( + ReportingRowObjectListEntryV1(key, version_id, size, format_row_instant(modified)) + ) + next_key = output.get("NextKeyMarker") + next_version = output.get("NextVersionIdMarker") + token: str | None = None + if output.get("IsTruncated"): + if not isinstance(next_key, str) or not isinstance(next_version, str): + # Ending here would report a partial listing as complete. + raise ReportingRowStoreError( + "PROVIDER_UNAVAILABLE", "object listing is truncated without a marker" + ) + token = ( + base64.urlsafe_b64encode( + json.dumps({"k": next_key, "v": next_version}, separators=(",", ":")).encode() + ) + .rstrip(b"=") + .decode("ascii") + ) + return ReportingRowObjectListPageV1(tuple(objects), token, skipped) + async def delete( self, delete: ReportingRowObjectDeleteInputV1, context: ReportingRowObjectContextV1 ) -> Literal["deleted", "absent"]: diff --git a/packages/adcp-reporting-storage/src/adcp_reporting_storage/migration.py b/packages/adcp-reporting-storage/src/adcp_reporting_storage/migration.py index 75cca712ad..f9753f3e63 100644 --- a/packages/adcp-reporting-storage/src/adcp_reporting_storage/migration.py +++ b/packages/adcp-reporting-storage/src/adcp_reporting_storage/migration.py @@ -123,6 +123,9 @@ class _Statement: _STARTUP_ROWS = { "adcp_reporting_change_horizon": "pruned_through_seq = 0 AND pruned_through_xid = '0'", "adcp_reporting_row_bindings": "TRUE", + # An opt-in orphan collector's claim on its bucket namespace. A replacement also drops the + # installation identity, so the namespace it fenced is not this ledger's any more. + "adcp_reporting_orphan_collector_fences": "TRUE", } _DEPENDENT_OBJECTS_STILL_EXIST = "2BP01" _SAVEPOINT = "adcp_reporting_replacement" diff --git a/src/adcp/reporting/production/_diagnostics.py b/src/adcp/reporting/production/_diagnostics.py index 366574dd8e..5bf91843ba 100644 --- a/src/adcp/reporting/production/_diagnostics.py +++ b/src/adcp/reporting/production/_diagnostics.py @@ -44,7 +44,7 @@ def _worker_stopped(*, boundary: _WorkerBoundary) -> None: _MAINTENANCE_OPERATIONS = frozenset( - {"intent_sweep", "row_pruning", "period_retirement", "change_pruning"} + {"intent_sweep", "row_pruning", "period_retirement", "change_pruning", "orphan_collection"} ) #: The only counters a maintenance event may carry: integers, never identifiers. _MAINTENANCE_COUNTERS = frozenset( @@ -58,6 +58,8 @@ def _worker_stopped(*, boundary: _WorkerBoundary) -> None: "stuck", "skipped_unconfigured", "overdue", + "unsupported", + "fenced", } ) diff --git a/src/adcp/reporting/production/service.py b/src/adcp/reporting/production/service.py index d3600c997e..75630545cf 100644 --- a/src/adcp/reporting/production/service.py +++ b/src/adcp/reporting/production/service.py @@ -714,14 +714,17 @@ async def _maintain(self) -> None: # is visible instead of looking like a healthy idle worker. pending = { key: counts[key] - for key in ("protected", "held", "deferred", "skipped_unconfigured") + for key in ("protected", "held", "deferred", "skipped_unconfigured", "unsupported") if counts.get(key) } if pending: _maintenance_pending(operation=name, counts=pending) - # Work nobody is reaching: intents expired past the overdue threshold, or periods that - # keep failing (five or more attempts). The same signals the JavaScript SDK reports. - stalled = {key: counts[key] for key in ("overdue", "stuck") if counts.get(key)} + # Work nobody is reaching: intents expired past the overdue threshold, periods that + # keep failing (five or more attempts), or orphan collection refused by another + # database's claim on its namespace. The same signals the JavaScript SDK reports. + stalled = { + key: counts[key] for key in ("overdue", "stuck", "fenced") if counts.get(key) + } if stalled: _maintenance_stalled(operation=name, counts=stalled) @@ -837,6 +840,8 @@ def _counts(result: Any) -> dict[str, int]: "stuck", "skipped_unconfigured", "overdue", + "unsupported", + "fenced", ): value = getattr(result, name, None) if isinstance(value, (tuple, list)): diff --git a/tests/conformance/reporting/_orphan_support.py b/tests/conformance/reporting/_orphan_support.py new file mode 100644 index 0000000000..92a9c73120 --- /dev/null +++ b/tests/conformance/reporting/_orphan_support.py @@ -0,0 +1,429 @@ +"""Shared fixture for the orphan collector and its ownership fence. + +A real PostgreSQL schema and the filesystem provider, with hooks on the provider: fail a delete, +fail or replace a listing page, make every object look older, or count fence reads. A "clone" is a +point-in-time copy of every table in the live schema, which is how a database clone or restore looks +to the collector: the same installation identity, namespace key, bindings and fence row. +""" + +from __future__ import annotations + +import os +import secrets +from collections.abc import AsyncIterator, Awaitable, Callable +from contextlib import asynccontextmanager +from dataclasses import dataclass, field, replace +from datetime import datetime, timedelta, timezone +from pathlib import Path +from typing import Any + +import pytest +from adcp_reporting_storage import PgReportingLedgerStore, ReportingRowStorageOptionsV1 +from adcp_reporting_storage.ledger.orphan_collector import ( + ReportingOrphanBindingResultV1, + ReportingOrphanCollectionResultV1, +) +from adcp_reporting_storage.ledger.row_storage import ( + ReportingObjectRowBindingV1, + ReportingPostgresRowBindingV1, +) +from adcp_reporting_storage.ledger.row_storage_object import ( + ReportingRowObjectContextV1, + ReportingRowObjectDeleteInputV1, + ReportingRowObjectListEntryV1, + ReportingRowObjectListInputV1, + ReportingRowObjectListPageV1, + ReportingRowObjectPutInputV1, + create_filesystem_reporting_row_object_provider_v1, + format_row_instant, + parse_row_instant, +) + +from ._generation_support import isolated_reporting_pool +from .test_reporting_row_storage_pg import fetch_all, many_rows, seeded, typed_revision + +DAY = timedelta(days=1) +CONTEXT = ReportingRowObjectContextV1(timeout_seconds=10.0) +LOCATION = {"root": "rows"} +# In foreign-key order. +OWNED_TABLES = ( + "adcp_persistence_installation", + "adcp_reporting_row_bindings", + "adcp_reporting_row_sets", + "adcp_reporting_row_chunks", + "adcp_reporting_chunk_bodies", + "adcp_reporting_row_write_intents", + "adcp_reporting_orphan_collector_fences", +) + + +def later() -> datetime: + """A clock past the age threshold, so every object written by the test is old enough.""" + return datetime.now(timezone.utc) + 8 * DAY + + +class HookedProvider: + """The filesystem provider with test hooks on ``delete`` and ``list``.""" + + name = "filesystem" + ranged_reads = True + + def __init__(self, inner: Any) -> None: + self.inner = inner + self.failing_keys: set[str] = set() + self.list_failures: list[BaseException] = [] + self.delete_calls: list[str] = [] + self.list_calls: list[str] = [] + self.list_override: ( + Callable[ + [ReportingRowObjectListInputV1, Callable[..., Awaitable[Any]]], + Awaitable[ReportingRowObjectListPageV1], + ] + | None + ) = None + self.age_shift: timedelta = timedelta(0) + self.put_failures: list[BaseException] = [] + self.on_put: Callable[[ReportingRowObjectPutInputV1], Awaitable[None]] | None = None + self.on_delete: Callable[[str], Awaitable[None]] | None = None + self.fence_list_failures: list[BaseException] = [] + self.fence_list_override: ReportingRowObjectListPageV1 | None = None + + def validate_location(self, location: Any) -> None: + self.inner.validate_location(location) + + async def probe(self, *args: Any, **kwargs: Any) -> None: + await self.inner.probe(*args, **kwargs) + + async def put_if_absent(self, put: ReportingRowObjectPutInputV1, context: Any) -> Any: + if self.put_failures: + raise self.put_failures.pop(0) + if self.on_put is not None: + await self.on_put(put) + return await self.inner.put_if_absent(put, context) + + async def get(self, *args: Any, **kwargs: Any) -> Any: + return await self.inner.get(*args, **kwargs) + + async def stat(self, *args: Any, **kwargs: Any) -> Any: + return await self.inner.stat(*args, **kwargs) + + async def delete(self, delete: ReportingRowObjectDeleteInputV1, context: Any) -> Any: + if self.on_delete is not None: + await self.on_delete(delete.object_ref) + if delete.object_ref in self.failing_keys: + raise RuntimeError("provider unavailable") + self.delete_calls.append(f"{delete.object_ref}@{delete.native_version_ref}") + return await self.inner.delete(delete, context) + + async def list( + self, request: ReportingRowObjectListInputV1, context: Any + ) -> ReportingRowObjectListPageV1: + self.list_calls.append(request.prefix) + # The hooks aim at the collector's object listing, not at the fence chain it also lists. + collecting = "_collector-fence" not in request.prefix + if not collecting and self.fence_list_failures: + raise self.fence_list_failures.pop(0) + if not collecting and self.fence_list_override is not None: + return self.fence_list_override + if collecting and self.list_failures: + raise self.list_failures.pop(0) + + async def real(page: ReportingRowObjectListInputV1) -> ReportingRowObjectListPageV1: + listed = await self.inner.list(page, context) + if not self.age_shift: + return listed # type: ignore[no-any-return] + shifted = [] + for entry in listed.objects: + moment = parse_row_instant(entry.created_at) + assert moment is not None + shifted.append( + replace(entry, created_at=format_row_instant(moment - self.age_shift)) + ) + return replace(listed, objects=tuple(shifted)) + + if collecting and self.list_override is not None: + return await self.list_override(request, real) + return await real(request) + + +class NoListProvider: + """The same provider without ``list``: a binding that cannot be collected.""" + + name = "filesystem" + ranged_reads = True + + def __init__(self, inner: Any) -> None: + for method in ("validate_location", "probe", "put_if_absent", "get", "stat", "delete"): + setattr(self, method, getattr(inner, method)) + + +@dataclass +class Fixture: + pool: Any + store: PgReportingLedgerStore + root: Path + provider: HookedProvider + namespace_key: str + options: ReportingRowStorageOptionsV1 + obligation: Any = None + revisions: int = 0 + extras: dict[str, Any] = field(default_factory=dict) + + def key(self, name: str, prefix: str = "orph") -> str: + return f"{prefix}/{self.namespace_key}/{name}" + + def on_disk(self, object_ref: str) -> bool: + return (self.root / "rows" / object_ref).exists() + + async def put_object(self, object_ref: str, body: bytes = b'{"a":1}\n') -> str: + created = await self.provider.inner.put_if_absent( + ReportingRowObjectPutInputV1( + LOCATION, + object_ref, + body, + "application/x-ndjson", + {"adcp-installation": "x", "adcp-intent": "y"}, + ), + CONTEXT, + ) + return str(created.native_version_ref) + + async def add_intent( + self, + row_set_id: str, + object_ref: str | None, + native_version_ref: str | None, + *, + binding: str = "objects", + state: str = "open", + legacy_names: bool = False, + ) -> None: + import json + + created = [] + if object_ref is not None: + created = [ + ( + {"objectKey": object_ref, "nativeVersion": native_version_ref} + if legacy_names + else {"object_ref": object_ref, "native_version_ref": native_version_ref} + ) + ] + async with self.pool.connection() as connection: + await connection.execute( + "INSERT INTO adcp_reporting_row_write_intents (row_binding_id, account_id," + " row_set_id, content_sha256, state, owner_token, created_objects, expires_at)" + " VALUES (%s, 'acct_orphans', %s, %s, %s, 'owner', %s::jsonb," + " clock_timestamp() + INTERVAL '1 hour')", + (binding, row_set_id, "e" * 64, state, json.dumps(created)), + ) + + async def commit_revision(self, suffix: str) -> tuple[str, str]: + """Commit an object-bound revision; returns the chunk's ``(object_ref, native_version)``.""" + if self.obligation is None: + self.obligation = await seeded(self.store) + async with self.pool.connection() as connection: + await connection.execute( + "UPDATE reporting_configurations SET lease_worker_id = 'worker-1'," + " lease_expires_at = clock_timestamp() + INTERVAL '10 minutes'" + ) + rows = many_rows(4) + revision = typed_revision(self.obligation, rows, suffix=suffix) + assert await self.store.stage_revision_rows(revision, rows) is True + await self.store.commit_revision(revision, rows) + found = await fetch_all( + self.pool, + "SELECT object_ref, native_version_ref FROM adcp_reporting_row_chunks" + " WHERE row_set_id = %s", + (revision.reporting_revision_id,), + ) + return str(found[0][0]), str(found[0][1]) + + def fence_dir(self) -> Path: + return self.root / "rows" / "orph" / self.namespace_key / "_collector-fence" + + def fence_files(self) -> list[str]: + directory = self.fence_dir() + return sorted(item.name for item in directory.iterdir()) if directory.exists() else [] + + def write_fence( + self, + epoch: int, + *, + token: str | None = None, + previous: str | None = None, + namespace: str | None = None, + raw: bytes | None = None, + ) -> str: + """Put a fence object in the bucket by hand, as another database would have.""" + import uuid + + from adcp_reporting_storage.ledger.orphan_fence import ( + OrphanFenceRecord, + encode_orphan_fence_record, + orphan_fence_key, + ) + + key = orphan_fence_key(f"orph/{self.namespace_key}/", epoch) + path = self.root / "rows" / key + path.parent.mkdir(parents=True, exist_ok=True) + chosen = token or str(uuid.uuid4()) + path.write_bytes( + raw + if raw is not None + else encode_orphan_fence_record( + namespace or self.namespace_key, + OrphanFenceRecord(epoch, chosen, previous, "2026-01-01T00:00:00.000Z"), + ) + ) + return chosen + + async def fence_row(self, pool: Any = None) -> tuple[Any, ...] | None: + rows = await fetch_all( + pool or self.pool, + "SELECT epoch, token, previous_token FROM adcp_reporting_orphan_collector_fences", + ) + return rows[0] if rows else None + + async def collect(self, **overrides: Any) -> ReportingOrphanCollectionResultV1: + arguments: dict[str, Any] = {"now": later(), "acknowledge_exclusive_namespace": True} + arguments.update(overrides) + return await self.store.collect_orphaned_row_objects(**arguments) + + +def binding_result( + result: ReportingOrphanCollectionResultV1, binding_id: str +) -> ReportingOrphanBindingResultV1: + return next(item for item in result.bindings if item.binding_id == binding_id) + + +def row_storage_options(tmp_path: Path, provider: HookedProvider) -> ReportingRowStorageOptionsV1: + return ReportingRowStorageOptionsV1( + deployment_namespace="orphans", + bindings={ + "inline": ReportingPostgresRowBindingV1(), + "objects": ReportingObjectRowBindingV1( + provider="filesystem", location=LOCATION, prefix="orph" + ), + "legacy": ReportingObjectRowBindingV1( + provider="filesystem", location=LOCATION, prefix="legacy", read_only=True + ), + "nolist": ReportingObjectRowBindingV1( + provider="nolist", location=LOCATION, prefix="nolist" + ), + # Shares ``objects``' root and prefix, as a renamed or successor binding does. + "twin": ReportingObjectRowBindingV1( + provider="filesystem", + location=LOCATION, + prefix="orph", + key_template="{prefix}/{namespace_key}/twin/{revision_key}/{content_sha256}/{chunk_index}", + read_only=True, + ), + }, + providers={ + "filesystem": provider, # type: ignore[dict-item] + "nolist": NoListProvider(provider.inner), # type: ignore[dict-item] + }, + select_row_binding=lambda selection: "objects", + ) + + +@asynccontextmanager +async def orphan_fixture(tmp_path: Path, **store_options: Any) -> AsyncIterator[Fixture]: + async with isolated_reporting_pool() as pool: + provider = HookedProvider( + create_filesystem_reporting_row_object_provider_v1(roots={"rows": tmp_path / "rows"}) + ) + options = row_storage_options(tmp_path, provider) + store = PgReportingLedgerStore(pool=pool, row_storage=options, **store_options) + await store.create_schema() + assert store._row_storage is not None + namespace_key = store._row_storage.installation_identity[1] + yield Fixture(pool, store, tmp_path, provider, namespace_key, options) + + +async def clone_schema(pool: Any) -> str: + """Copy every row-storage table into a new schema: a point-in-time clone of the ledger. + + Returns the schema name; ``cloned_store`` opens a store on it. The copy keeps the installation + identity, the namespace key, the bindings and the fence row the live database had when it was + taken. The row-storage DDL is applied first so the clone carries the same triggers and + constraints the catalog check requires. + """ + from adcp_reporting_storage.ledger.row_storage import REPORTING_ROW_STORAGE_DDL_PATH + from psycopg import AsyncConnection, sql + + url = os.environ["ADCP_PG_TEST_URL"] + async with pool.connection() as live: + source = (await (await live.execute("SELECT current_schema()")).fetchone())[0] + target = f"adcp_reporting_clone_{secrets.token_hex(6)}" + async with await AsyncConnection.connect(url, autocommit=True) as admin: + await admin.execute(sql.SQL("CREATE SCHEMA {}").format(sql.Identifier(target))) + await admin.execute(sql.SQL("SET search_path TO {}").format(sql.Identifier(target))) + await admin.execute(REPORTING_ROW_STORAGE_DDL_PATH.read_text()) + await admin.execute("DELETE FROM adcp_persistence_installation") + for table in OWNED_TABLES: + await admin.execute( + sql.SQL("INSERT INTO {}.{} SELECT * FROM {}.{}").format( + sql.Identifier(target), + sql.Identifier(table), + sql.Identifier(source), + sql.Identifier(table), + ) + ) + return target + + +@asynccontextmanager +async def cloned_store(fixture: Fixture, schema: str) -> AsyncIterator[PgReportingLedgerStore]: + """A second store on the clone: same configuration, its own database state.""" + url = os.environ["ADCP_PG_TEST_URL"] + from psycopg import AsyncConnection, sql + from psycopg_pool import AsyncConnectionPool + + async with AsyncConnectionPool( + url, + kwargs={ + "options": f"-csearch_path={schema} -cstatement_timeout=15000", + "autocommit": False, + }, + min_size=1, + max_size=4, + open=False, + ) as pool: + await pool.wait(timeout=10) + store = PgReportingLedgerStore(pool=pool, row_storage=fixture.options) + await store.ready_row_storage() + try: + yield store + finally: + async with await AsyncConnection.connect(url, autocommit=True) as admin: + await admin.execute( + sql.SQL("DROP SCHEMA IF EXISTS {} CASCADE").format(sql.Identifier(schema)) + ) + + +def require_database() -> None: + if not os.environ.get("ADCP_PG_TEST_URL"): + pytest.skip("ADCP_PG_TEST_URL not set — requires real PostgreSQL") + + +async def count_changes(fixture: Fixture) -> int: + rows = await fetch_all(fixture.pool, "SELECT count(*) FROM adcp_reporting_row_write_intents") + return int(rows[0][0]) + + +__all__ = [ + "CONTEXT", + "DAY", + "LOCATION", + "Fixture", + "HookedProvider", + "NoListProvider", + "binding_result", + "clone_schema", + "cloned_store", + "later", + "orphan_fixture", + "ReportingRowObjectListEntryV1", +] diff --git a/tests/conformance/reporting/test_reporting_change_feed_pg.py b/tests/conformance/reporting/test_reporting_change_feed_pg.py index 9d41968d37..5250559552 100644 --- a/tests/conformance/reporting/test_reporting_change_feed_pg.py +++ b/tests/conformance/reporting/test_reporting_change_feed_pg.py @@ -812,13 +812,14 @@ async def apply() -> None: async def test_reapplying_the_changes_migration_upgrades_an_older_install_in_place() -> None: from adcp_reporting_storage.ledger.change_feed import REPORTING_CHANGES_DDL_PATH - from psycopg.errors import CheckViolation + + psycopg = pytest.importorskip("psycopg") async with isolated_reporting_pool() as pool: await PgReportingLedgerStore(pool=pool).create_schema() async with pool.connection() as connection: await connection.execute(PREVIOUS_CHANGES_DDL) - with pytest.raises(CheckViolation): + with pytest.raises(psycopg.errors.CheckViolation): async with pool.connection() as connection: await connection.execute( "INSERT INTO adcp_reporting_feed_consumers" @@ -841,7 +842,7 @@ async def test_reapplying_the_changes_migration_upgrades_an_older_install_in_pla " adcp_reporting_feed_consumers", ) == [("old-account-sink", None, "5")], "a seq-only account consumer is kept" # The deployment feed still needs an xid; the old check is gone, the new one stays. - with pytest.raises(CheckViolation): + with pytest.raises(psycopg.errors.CheckViolation): async with pool.connection() as connection: await connection.execute( "INSERT INTO adcp_reporting_feed_consumers" diff --git a/tests/conformance/reporting/test_reporting_orphan_collector_pg.py b/tests/conformance/reporting/test_reporting_orphan_collector_pg.py new file mode 100644 index 0000000000..497a161f70 --- /dev/null +++ b/tests/conformance/reporting/test_reporting_orphan_collector_pg.py @@ -0,0 +1,469 @@ +"""The opt-in listing collector for orphaned row objects, on PostgreSQL and the filesystem provider. + +Ported from the JavaScript SDK (adcp-client #3291, ``reporting-row-orphan-collector-pg``). +""" + +from __future__ import annotations + +import asyncio +from dataclasses import replace +from datetime import datetime, timezone +from pathlib import Path +from typing import Any + +import pytest +from adcp_reporting_storage.ledger.row_storage import ReportingRowStoreError +from adcp_reporting_storage.ledger.row_storage_object import ( + ReportingRowObjectListEntryV1, + ReportingRowObjectListInputV1, + ReportingRowObjectListPageV1, +) + +from ._orphan_support import binding_result, orphan_fixture +from .test_reporting_row_storage_pg import fetch_all + + +async def test_deletes_an_orphan_and_keeps_committed_intent_named_and_young_objects( + tmp_path: Path, +) -> None: + async with orphan_fixture(tmp_path) as fx: + committed, _ = await fx.commit_revision("orphans_commit") + # An upload that succeeded and was never recorded anywhere. + orphan = fx.key("acct/orphan.jsonl.gz") + await fx.put_object(orphan) + # Named by an open upload that has not committed yet. + in_flight = fx.key("acct/in-flight.jsonl.gz") + await fx.add_intent("rrev_in_flight", in_flight, await fx.put_object(in_flight)) + young = fx.key("acct/young.jsonl") + await fx.put_object(young) + + everything = [orphan, committed, in_flight, young] + dry = await fx.collect(dry_run=True, bindings=["objects"]) + assert dry.dry_run is True + assert sorted(c.object_ref for c in dry.candidates) == sorted([orphan, young]) + assert {c.binding_id for c in dry.candidates} == {"objects"} + assert dry.would_delete == 2 and dry.deleted == 0 + assert all(fx.on_disk(ref) for ref in everything), "a dry run deletes nothing" + + # At the real clock nothing is old enough. + early = await fx.store.collect_orphaned_row_objects(acknowledge_exclusive_namespace=True) + assert early.deleted == 0 + assert binding_result(early, "objects").too_young == 4 + assert all(fx.on_disk(ref) for ref in everything) + + real = await fx.collect(bindings=["objects"]) + assert real.deleted == 2 + assert not fx.on_disk(orphan) and not fx.on_disk(young) + assert fx.on_disk(committed) and fx.on_disk(in_flight) + assert binding_result(real, "objects").referenced == 2 + assert binding_result(real, "objects").status == "complete" + assert real.completed == ("objects",) + + +async def test_never_touches_keys_outside_the_owned_prefix_or_objects_that_are_not_rows( + tmp_path: Path, +) -> None: + async with orphan_fixture(tmp_path) as fx: + for name in ("a.txt", "b.json", "nested/c.bin"): + await fx.put_object(fx.key(name)) + elsewhere = "orph/not-the-namespace/x.jsonl" + sibling = f"orphan-extra/{fx.namespace_key}/y.jsonl" + for ref in (elsewhere, sibling): + await fx.put_object(ref) + result = await fx.collect(bindings=["objects"]) + assert result.deleted == 0 and result.foreign >= 3 + assert fx.on_disk(elsewhere) and fx.on_disk(sibling) + assert all(fx.on_disk(fx.key(name)) for name in ("a.txt", "b.json", "nested/c.bin")) + + +async def test_skips_read_only_bindings_unless_asked_and_reports_a_provider_that_cannot_list( + tmp_path: Path, +) -> None: + async with orphan_fixture(tmp_path) as fx: + legacy = fx.key("acct/legacy.jsonl", "legacy") + await fx.put_object(legacy) + skipped = await fx.collect() + assert "legacy" in skipped.skipped_read_only + assert binding_result(skipped, "legacy").status == "read_only" + assert fx.on_disk(legacy) + assert skipped.unsupported == ("nolist",) + assert binding_result(skipped, "nolist").status == "unsupported" + assert skipped.fenced == () + + included = await fx.collect(include_read_only=True, bindings=["legacy"]) + assert binding_result(included, "legacy").deleted == 1 + assert not fx.on_disk(legacy) + + +async def test_pages_with_a_per_pass_budget_and_resumes_from_the_returned_cursors( + tmp_path: Path, +) -> None: + async with orphan_fixture(tmp_path) as fx: + refs = [fx.key(f"acct/o{index}.jsonl") for index in range(7)] + for ref in refs: + await fx.put_object(ref) + cursors: dict[str, str] = {} + deleted = 0 + for _ in range(10): + result = await fx.collect(bindings=["objects"], limit=3, page_size=2, cursors=cursors) + deleted += result.deleted + cursors = dict(result.cursors) + if not cursors: + break + assert deleted == 7 and not any(fx.on_disk(ref) for ref in refs) + + +async def test_reports_the_bindings_whose_listing_finished_and_skips_the_ones_it_is_told_to( + tmp_path: Path, +) -> None: + async with orphan_fixture(tmp_path) as fx: + await fx.put_object(fx.key("acct/a.jsonl")) + first = await fx.collect(bindings=["objects", "legacy"], include_read_only=True) + assert set(first.completed) == {"objects", "legacy"} + fx.provider.list_calls.clear() + again = await fx.collect( + bindings=["objects", "legacy"], include_read_only=True, skip_bindings=["objects"] + ) + assert again.completed == ("legacy",) + assert [b.binding_id for b in again.bindings] == ["legacy"] + assert all( + "/legacy/" in call or call.startswith("legacy/") for call in fx.provider.list_calls + ) + + +async def test_a_delete_that_fails_is_counted_and_does_not_stop_the_rest(tmp_path: Path) -> None: + async with orphan_fixture(tmp_path) as fx: + stuck, fine = fx.key("acct/a-stuck.jsonl"), fx.key("acct/b-fine.jsonl") + await fx.put_object(stuck) + await fx.put_object(fine) + fx.provider.failing_keys.add(stuck) + result = await fx.collect(bindings=["objects"]) + assert result.deleted == 1 and result.failed == 1 + assert fx.on_disk(stuck) and not fx.on_disk(fine) + assert binding_result(result, "objects").status == "complete" + # It is retried on the next full listing. + fx.provider.failing_keys.clear() + retried = await fx.collect(bindings=["objects"]) + assert retried.deleted == 1 and not fx.on_disk(stuck) + + +async def test_a_listing_failure_keeps_its_cursor_and_names_the_cause(tmp_path: Path) -> None: + async with orphan_fixture(tmp_path) as fx: + for index in range(4): + await fx.put_object(fx.key(f"acct/o{index}.jsonl")) + partial = await fx.collect(bindings=["objects"], limit=2, page_size=2) + assert binding_result(partial, "objects").status == "partial" + cursor = partial.cursors["objects"] + fx.provider.list_failures.append(ReportingRowStoreError("PROVIDER_UNAVAILABLE")) + # The first page of a resumed cursor fails: the stale cursor is dropped. + failed = await fx.collect(bindings=["objects"], cursors={"objects": cursor}) + outcome = binding_result(failed, "objects") + assert outcome.status == "failed" and outcome.error == "PROVIDER_UNAVAILABLE" + assert "objects" not in failed.cursors + # A failure after the first page keeps the position it reached. + calls = 0 + real_list = fx.provider.list + + async def second_page_fails( + request: ReportingRowObjectListInputV1, real: Any + ) -> ReportingRowObjectListPageV1: + nonlocal calls + calls += 1 + if calls == 2: + raise ReportingRowStoreError("DEADLINE_EXCEEDED") + return await real(request) # type: ignore[no-any-return] + + fx.provider.list_override = second_page_fails + mid = await fx.collect(bindings=["objects"], limit=100, page_size=2) + assert binding_result(mid, "objects").error == "DEADLINE_EXCEEDED" + assert "objects" in mid.cursors + assert real_list is not None + + +async def test_a_record_naming_the_key_at_another_version_or_an_unpruned_row_set_keeps_it( + tmp_path: Path, +) -> None: + async with orphan_fixture(tmp_path) as fx: + ref, version = await fx.commit_revision("named_other_version") + # The same key, now at a different version, is still named by a record at the old one. + await fx.provider.inner.delete( + __import__( + "adcp_reporting_storage.ledger.row_storage_object", fromlist=["x"] + ).ReportingRowObjectDeleteInputV1({"root": "rows"}, ref, version), + __import__( + "adcp_reporting_storage.ledger.row_storage_object", fromlist=["x"] + ).ReportingRowObjectContextV1(), + ) + await fx.put_object(ref) # a new version at the recorded key + result = await fx.collect(bindings=["objects"]) + assert result.deleted == 0 + assert binding_result(result, "objects").referenced == 1 + assert fx.on_disk(ref) + + +async def test_a_row_set_that_is_pruning_or_unavailable_keeps_its_object_and_a_pruned_one_does_not( + tmp_path: Path, +) -> None: + async with orphan_fixture(tmp_path) as fx: + ref, _ = await fx.commit_revision("states") + for state in ("pruning", "unavailable"): + async with fx.pool.connection() as connection: + await connection.execute( + "UPDATE adcp_reporting_row_sets SET rows_state = %s", (state,) + ) + kept = await fx.collect(bindings=["objects"]) + assert kept.deleted == 0 and fx.on_disk(ref), state + async with fx.pool.connection() as connection: + await connection.execute("UPDATE adcp_reporting_row_sets SET rows_state = 'pruned'") + freed = await fx.collect(bindings=["objects"]) + assert freed.deleted == 1 and not fx.on_disk(ref) + + +async def test_an_intent_written_by_an_older_sdk_keeps_its_object(tmp_path: Path) -> None: + async with orphan_fixture(tmp_path) as fx: + ref = fx.key("acct/legacy-intent.jsonl.gz") + await fx.add_intent("rrev_old", ref, await fx.put_object(ref), legacy_names=True) + result = await fx.collect(bindings=["objects"]) + assert result.deleted == 0 and fx.on_disk(ref) + for state in ("sweeping",): + async with fx.pool.connection() as connection: + await connection.execute( + "UPDATE adcp_reporting_row_write_intents SET state = %s", (state,) + ) + # The pre-check is blind to the intent state: an intent in any state names its key. + assert (await fx.collect(bindings=["objects"])).deleted == 0 and fx.on_disk(ref) + + +async def test_a_record_under_another_binding_sharing_the_bucket_and_prefix_keeps_the_object( + tmp_path: Path, +) -> None: + async with orphan_fixture(tmp_path) as fx: + ref = fx.key("acct/shared.jsonl.gz") + version = await fx.put_object(ref) + # An open upload under the twin binding (same bucket and prefix) names the object. + await fx.add_intent("rrev_twin", ref, version, binding="twin") + result = await fx.collect(bindings=["objects"]) + assert result.deleted == 0 and fx.on_disk(ref) + + +async def test_a_hostile_provider_cannot_make_it_delete_outside_the_owned_prefix( + tmp_path: Path, +) -> None: + async with orphan_fixture(tmp_path) as fx: + outside = "other-tenant/data.jsonl" + await fx.put_object(outside) + owned = fx.key("acct/owned.jsonl") + await fx.put_object(owned) + old = datetime(2020, 1, 1, tzinfo=timezone.utc) + stamp = old.strftime("%Y-%m-%dT%H:%M:%S.000Z") + + def entry(ref: Any, **changes: Any) -> Any: + base = ReportingRowObjectListEntryV1(ref, "1:2:3:4", 5, stamp) + return replace(base, **changes) + + hostile = [ + entry(outside), + entry(f"{fx.key('x')}/../../{outside}"), + entry(fx.key("a/./b.jsonl")), + entry(fx.key("a//b.jsonl")), + entry(fx.key("bad key.jsonl")), + entry(fx.key("acct/newline.jsonl\n")), + entry(fx.key("acct/no-version.jsonl"), native_version_ref=""), + entry(fx.key("acct/long-version.jsonl"), native_version_ref="v" * 1025), + entry(fx.key("acct/negative.jsonl"), size_bytes=-1), + entry(fx.key("acct/bad-time.jsonl"), created_at="yesterday"), + entry(fx.key("acct/offset-time.jsonl"), created_at="2020-01-01T00:00:00+02:00"), + entry("x" * 1500 + ".jsonl"), + entry(owned), + ] + + async def hostile_page( + request: ReportingRowObjectListInputV1, real: Any + ) -> ReportingRowObjectListPageV1: + return ReportingRowObjectListPageV1(tuple(hostile), None) + + fx.provider.list_override = hostile_page + result = await fx.collect(bindings=["objects"]) + outcome = binding_result(result, "objects") + assert outcome.status == "complete" + assert outcome.foreign == len(hostile) - 1, "every malformed or foreign entry is dropped" + assert fx.on_disk(outside), "nothing outside the owned prefix is ever touched" + assert all(call.startswith(fx.key("")) for call in fx.provider.delete_calls) + assert len(fx.provider.delete_calls) == 1 + # An entry whose version is not the stored one deletes nothing: the provider pins versions. + assert fx.on_disk(owned) + + +async def test_a_provider_that_repeats_a_token_or_returns_too_many_objects_fails_the_binding( + tmp_path: Path, +) -> None: + async with orphan_fixture(tmp_path) as fx: + await fx.put_object(fx.key("acct/a.jsonl")) + + async def repeat( + request: ReportingRowObjectListInputV1, real: Any + ) -> ReportingRowObjectListPageV1: + return ReportingRowObjectListPageV1((), "same-token") + + fx.provider.list_override = repeat + looped = await fx.collect(bindings=["objects"], limit=100, page_size=10) + outcome = binding_result(looped, "objects") + assert outcome.status == "failed" and outcome.error == "PROVIDER_UNAVAILABLE" + assert "objects" not in looped.cursors, "a repeated token is not a cursor to resume" + + async def too_many( + request: ReportingRowObjectListInputV1, real: Any + ) -> ReportingRowObjectListPageV1: + entry = ReportingRowObjectListEntryV1( + fx.key("a/x.jsonl"), "v", 1, "2020-01-01T00:00:00.000Z" + ) + return ReportingRowObjectListPageV1((entry,) * (request.page_size + 1), None) + + fx.provider.list_override = too_many + many = await fx.collect(bindings=["objects"], page_size=3) + assert binding_result(many, "objects").status == "failed" + assert fx.provider.delete_calls == [] + + +async def test_it_validates_its_inputs_and_requires_the_acknowledgement_to_delete( + tmp_path: Path, +) -> None: + async with orphan_fixture(tmp_path) as fx: + store = fx.store + with pytest.raises(ValueError, match="acknowledge_exclusive_namespace"): + await store.collect_orphaned_row_objects() + for bad in (0, 3600, 86_399, 1.5, "7"): + with pytest.raises(ValueError, match="min_age_seconds"): + await store.collect_orphaned_row_objects( + min_age_seconds=bad, acknowledge_exclusive_namespace=True # type: ignore[arg-type] + ) + for name in ("limit", "page_size"): + for bad in (0, -1, 1.5): + with pytest.raises(ValueError, match=name): + await store.collect_orphaned_row_objects( + dry_run=True, **{name: bad} # type: ignore[arg-type] + ) + with pytest.raises(ValueError, match="not a configured object binding"): + await store.collect_orphaned_row_objects(dry_run=True, bindings=["inline"]) + with pytest.raises(ValueError, match="timezone-aware"): + await store.collect_orphaned_row_objects(dry_run=True, now=datetime(2030, 1, 1)) + # A dry run needs no acknowledgement. + assert (await store.collect_orphaned_row_objects(dry_run=True)).dry_run is True + + +async def test_a_store_without_row_storage_refuses_the_collector() -> None: + from ._generation_support import isolated_reporting_pool + + async with isolated_reporting_pool() as pool: + from adcp_reporting_storage import PgReportingLedgerStore + + store = PgReportingLedgerStore(pool=pool) + with pytest.raises(ReportingRowStoreError) as failure: + await store.collect_orphaned_row_objects(dry_run=True) + assert failure.value.code == "INVALID_INPUT" + + +async def test_the_collector_takes_the_per_object_lock_and_rechecks_under_it( + tmp_path: Path, +) -> None: + """An intent committed while the collector waits on the lock keeps the object.""" + async with orphan_fixture(tmp_path) as fx: + ref = fx.key("acct/raced.jsonl.gz") + version = await fx.put_object(ref) + storage = fx.store._row_storage + assert storage is not None + lock_text = storage._object_lock_text("objects", ref) + holder = await fx.pool.getconn() + try: + await holder.execute( + "SELECT pg_advisory_xact_lock(hashtextextended(%s, 0))", (lock_text,) + ) + task = asyncio.create_task(fx.collect(bindings=["objects"])) + await _until_blocked(fx.pool, task) + assert not task.done(), "the collector waits on the object's lock" + # While it waits, an upload that names the object commits. + await holder.execute( + "INSERT INTO adcp_reporting_row_write_intents (row_binding_id, account_id," + " row_set_id, content_sha256, state, owner_token, created_objects, expires_at)" + " VALUES ('objects', 'acct', 'rrev_raced', %s, 'open', 'o', %s::jsonb," + " clock_timestamp() + INTERVAL '1 hour')", + ( + "e" * 64, + f'[{{"object_ref": "{ref}", "native_version_ref": "{version}"}}]', + ), + ) + await holder.commit() + finally: + await fx.pool.putconn(holder) + result = await asyncio.wait_for(task, 20) + # The pre-check ran before the intent existed; the recheck under the lock saw it. + assert result.deleted == 0 and fx.on_disk(ref) + assert binding_result(result, "objects").referenced == 1 + + +async def test_it_serializes_with_an_adoption_through_another_binding_sharing_the_prefix( + tmp_path: Path, +) -> None: + async with orphan_fixture(tmp_path) as fx: + ref = fx.key("acct/adopted-by-twin.jsonl.gz") + version = await fx.put_object(ref) + storage = fx.store._row_storage + assert storage is not None + # A writer adopting through the twin binding holds the twin's lock for this object. + holder = await fx.pool.getconn() + try: + await holder.execute( + "SELECT pg_advisory_xact_lock(hashtextextended(%s, 0))", + (storage._object_lock_text("twin", ref),), + ) + task = asyncio.create_task(fx.collect(bindings=["objects"])) + await _until_blocked(fx.pool, task) + assert not task.done(), "the delete waits for an adoption through any binding" + await holder.execute( + "INSERT INTO adcp_reporting_row_write_intents (row_binding_id, account_id," + " row_set_id, content_sha256, state, owner_token, created_objects, expires_at)" + " VALUES ('twin', 'acct', 'rrev_twin', %s, 'open', 'o', %s::jsonb," + " clock_timestamp() + INTERVAL '1 hour')", + ( + "e" * 64, + f'[{{"object_ref": "{ref}", "native_version_ref": "{version}"}}]', + ), + ) + await holder.commit() + finally: + await fx.pool.putconn(holder) + result = await asyncio.wait_for(task, 20) + assert result.deleted == 0 and fx.on_disk(ref) + + +async def _until_blocked(pool: Any, task: asyncio.Task[Any]) -> None: + """Wait until some backend is blocked on an advisory lock, proven with ``pg_locks``.""" + for _ in range(400): + if task.done(): + return + rows = await fetch_all( + pool, + "SELECT count(*) FROM pg_locks WHERE locktype = 'advisory' AND NOT granted", + ) + if rows[0][0]: + return + await asyncio.sleep(0.025) + pytest.fail("the collector never blocked on the object lock") + + +async def test_dry_run_through_the_store_option_reports_and_deletes_nothing( + tmp_path: Path, +) -> None: + from adcp_reporting_storage.ledger.orphan_collector import ReportingOrphanCollectionOptionsV1 + + async with orphan_fixture( + tmp_path, orphan_collection=ReportingOrphanCollectionOptionsV1(dry_run=True) + ) as fx: + ref = fx.key("acct/maintained.jsonl") + await fx.put_object(ref) + operations = dict(fx.store.maintenance_operations()) + assert "orphan_collection" in operations + # Too young at the real clock: nothing is a candidate yet. + result = await operations["orphan_collection"]() + assert result.dry_run is True and result.would_delete == 0 + assert fx.on_disk(ref) diff --git a/tests/conformance/reporting/test_reporting_orphan_fence_pg.py b/tests/conformance/reporting/test_reporting_orphan_fence_pg.py new file mode 100644 index 0000000000..4159d9d746 --- /dev/null +++ b/tests/conformance/reporting/test_reporting_orphan_fence_pg.py @@ -0,0 +1,650 @@ +"""The orphan collector's ownership fence against a cloned or restored ledger database. + +Ported from the JavaScript SDK (adcp-client #3593, ``reporting-orphan-collector-fence-pg``). A +clone is a point-in-time copy of every row-storage table in the live schema: the same installation +identity, namespace key, bindings and fence row, sharing the live database's bucket. +""" + +from __future__ import annotations + +import asyncio +import json +import time +import uuid +from pathlib import Path +from typing import Any + +import pytest +from adcp_reporting_storage.ledger.orphan_collector import ( + ReportingOrphanCollectionOptionsV1, + ReportingOrphanCollectionStateV1, + run_orphan_collection_pass, +) +from adcp_reporting_storage.ledger.row_storage import ReportingRowStoreError + +from ._orphan_support import ( + HookedProvider, + binding_result, + clone_schema, + cloned_store, + orphan_fixture, +) +from .test_reporting_row_storage_pg import fetch_all + + +def fenced_cause(result: Any, binding_id: str = "objects") -> str | None: + outcome = binding_result(result, binding_id) + return outcome.fence.cause if outcome.fence else None + + +async def test_the_first_real_pass_claims_epoch_one_and_each_later_pass_the_next( + tmp_path: Path, +) -> None: + async with orphan_fixture(tmp_path) as fx: + assert await fx.fence_row() is None + first = await fx.collect(bindings=["objects"]) + assert first.fenced == () + assert fx.fence_files() == ["0000000000000001.json"] + assert (await fx.fence_row())[:1] == (1,) # type: ignore[index] + await fx.collect(bindings=["objects"]) + # A claim keeps its predecessor (an unfinished claim is recognised by it) and prunes the + # rest. + assert fx.fence_files() == ["0000000000000001.json", "0000000000000002.json"] + await fx.collect(bindings=["objects"]) + assert fx.fence_files() == ["0000000000000002.json", "0000000000000003.json"] + row = await fx.fence_row() + assert row is not None and row[0] == 3 + assert row[2] is not None, "a later claim records the token it superseded" + + +async def test_the_fence_chain_is_not_a_row_object_it_is_neither_listed_foreign_nor_deleted( + tmp_path: Path, +) -> None: + async with orphan_fixture(tmp_path) as fx: + await fx.collect(bindings=["objects"]) + assert fx.fence_files() + storage = fx.store._row_storage + assert storage is not None + listing = await storage.list_objects("objects", page_size=100) + assert listing.objects == () and listing.foreign == 0 + result = await fx.collect(bindings=["objects"]) + assert result.deleted == 0 and result.foreign == 0 + assert fx.fence_files(), "the collector never deletes its own fence" + + +async def test_a_dry_run_checks_the_fence_without_claiming_it(tmp_path: Path) -> None: + async with orphan_fixture(tmp_path) as fx: + orphan = fx.key("acct/orphan.jsonl") + await fx.put_object(orphan) + dry = await fx.collect(dry_run=True, bindings=["objects"]) + assert dry.would_delete == 1 and dry.fenced == () + assert fx.fence_files() == [] and await fx.fence_row() is None, "no claim, no object" + # It reports a fenced namespace the same way a real pass does. + fx.write_fence(4) + refused = await fx.collect(dry_run=True, bindings=["objects"]) + assert refused.fenced == ("objects",) + assert fenced_cause(refused) == "claimed_elsewhere" + assert refused.would_delete == 0 + + +async def test_a_clone_taken_before_the_original_last_collected_is_fenced_and_deletes_nothing( + tmp_path: Path, +) -> None: + async with orphan_fixture(tmp_path) as fx: + await fx.collect(bindings=["objects"]) # epoch 1 + schema = await clone_schema(fx.pool) + committed, _ = await fx.commit_revision("clone_live") # written after the copy + await fx.collect(bindings=["objects"]) # the original claims epoch 2 + + async with cloned_store(fx, schema) as clone: + result = await clone.collect_orphaned_row_objects( + now=_later(), acknowledge_exclusive_namespace=True, bindings=["objects"] + ) + assert result.fenced == ("objects",) and result.deleted == 0 + assert fenced_cause(result) == "fence_advanced" + outcome = binding_result(result, "objects") + assert outcome.fence is not None + assert (outcome.fence.fence_epoch, outcome.fence.recorded_epoch) == (2, 1) + assert "take_over_orphan_collector_fence" in outcome.fence.detail + assert fx.on_disk(committed), "the clone cannot delete an object the original wrote" + assert result.cursors == {} + + # Control: the same clone, once it has deliberately taken over, deletes it. The fence + # is the only thing standing between the clone and the live object. + await clone.take_over_orphan_collector_fence(acknowledge_previous_owner_gone=True) + deleted = await clone.collect_orphaned_row_objects( + now=_later(), acknowledge_exclusive_namespace=True, bindings=["objects"] + ) + assert deleted.deleted >= 1 and not fx.on_disk(committed) + + +async def test_a_clone_that_collects_first_claims_the_namespace_and_fences_the_original( + tmp_path: Path, +) -> None: + async with orphan_fixture(tmp_path) as fx: + schema = await clone_schema(fx.pool) # neither side has claimed yet + async with cloned_store(fx, schema) as clone: + first = await clone.collect_orphaned_row_objects( + now=_later(), acknowledge_exclusive_namespace=True, bindings=["objects"] + ) + assert first.fenced == () + original = await fx.collect(bindings=["objects"]) + assert original.fenced == ("objects",) + assert fenced_cause(original) == "claimed_elsewhere", "first claim wins" + + +async def test_a_restore_to_an_older_database_state_is_fenced_until_an_operator_takes_over( + tmp_path: Path, +) -> None: + async with orphan_fixture(tmp_path) as fx: + await fx.collect(bindings=["objects"]) + await fx.collect(bindings=["objects"]) # bucket epoch 2 + async with fx.pool.connection() as connection: + # The restore: the database's own record goes back to epoch 1. + await connection.execute( + "UPDATE adcp_reporting_orphan_collector_fences SET epoch = 1," + " token = %s, previous_token = NULL", + (str(uuid.uuid4()),), + ) + restored = await fx.collect(bindings=["objects"]) + assert fenced_cause(restored) == "fence_advanced" + taken = await fx.store.take_over_orphan_collector_fence( + acknowledge_previous_owner_gone=True + ) + assert [(t.binding_ids, t.epoch, t.previous_epoch) for t in taken] == [(("objects",), 3, 2)] + after = await fx.collect(bindings=["objects"]) + assert after.fenced == () + + +async def test_a_takeover_fences_out_the_previous_owner(tmp_path: Path) -> None: + async with orphan_fixture(tmp_path) as fx: + await fx.collect(bindings=["objects"]) + schema = await clone_schema(fx.pool) + async with cloned_store(fx, schema) as clone: + await clone.take_over_orphan_collector_fence(acknowledge_previous_owner_gone=True) + out = await fx.collect(bindings=["objects"]) + assert fenced_cause(out) == "fence_advanced" + with pytest.raises(ValueError, match="acknowledge_previous_owner_gone"): + await clone.take_over_orphan_collector_fence(acknowledge_previous_owner_gone=False) + + +async def test_a_takeover_validates_its_bindings_and_covers_a_shared_fence_once( + tmp_path: Path, +) -> None: + async with orphan_fixture(tmp_path) as fx: + store = fx.store + for bad, message in ( + (["inline"], "not a configured object binding"), + (["nolist"], "cannot list"), + (["nope"], "not a configured object binding"), + ): + with pytest.raises(ValueError, match=message): + await store.take_over_orphan_collector_fence( + acknowledge_previous_owner_gone=True, bindings=bad + ) + # ``objects`` and its read-only ``twin`` share one bucket and prefix, so one fence. + taken = await store.take_over_orphan_collector_fence( + acknowledge_previous_owner_gone=True, include_read_only=True + ) + shared = [t for t in taken if "objects" in t.binding_ids] + assert len(shared) == 1 and set(shared[0].binding_ids) == {"objects", "twin"} + # Read-only bindings are skipped unless asked for. + default = await store.take_over_orphan_collector_fence(acknowledge_previous_owner_gone=True) + assert all("legacy" not in t.binding_ids for t in default) + only = await store.take_over_orphan_collector_fence( + acknowledge_previous_owner_gone=True, bindings=["legacy"] + ) + assert [t.binding_ids for t in only] == [("legacy",)] + + +async def test_bindings_that_share_a_bucket_and_prefix_share_one_fence_and_one_claim_per_pass( + tmp_path: Path, +) -> None: + async with orphan_fixture(tmp_path) as fx: + result = await fx.collect(bindings=["objects", "twin"], include_read_only=True) + assert result.fenced == () + assert fx.fence_files() == ["0000000000000001.json"], "one claim for both bindings" + rows = await fetch_all( + fx.pool, "SELECT count(*) FROM adcp_reporting_orphan_collector_fences" + ) + assert rows == [(1,)] + + +async def test_a_database_with_no_recorded_claim_refuses_a_namespace_that_already_has_one( + tmp_path: Path, +) -> None: + async with orphan_fixture(tmp_path) as fx: + fx.write_fence(3) + result = await fx.collect(bindings=["objects"]) + assert fenced_cause(result) == "claimed_elsewhere" + outcome = binding_result(result, "objects") + assert outcome.fence is not None and outcome.fence.fence_epoch == 3 + assert outcome.fence.recorded_epoch is None + assert await fx.fence_row() is None, "a refusal never records a claim" + + +async def test_a_recorded_claim_with_no_fence_in_the_bucket_is_refused_as_missing( + tmp_path: Path, +) -> None: + async with orphan_fixture(tmp_path) as fx: + await fx.collect(bindings=["objects"]) + await fx.collect(bindings=["objects"]) + for name in fx.fence_files(): + (fx.fence_dir() / name).unlink() + result = await fx.collect(bindings=["objects"]) + assert fenced_cause(result) == "fence_missing" + assert fx.fence_files() == [], "never silently re-claimed" + + +async def test_a_database_newer_than_its_bucket_is_refused_as_regressed(tmp_path: Path) -> None: + async with orphan_fixture(tmp_path) as fx: + await fx.collect(bindings=["objects"]) + async with fx.pool.connection() as connection: + await connection.execute( + "UPDATE adcp_reporting_orphan_collector_fences SET epoch = 5, token = %s," + " previous_token = %s", + (str(uuid.uuid4()), str(uuid.uuid4())), + ) + result = await fx.collect(bindings=["objects"]) + assert fenced_cause(result) == "fence_regressed" + outcome = binding_result(result, "objects") + assert outcome.fence is not None + assert (outcome.fence.fence_epoch, outcome.fence.recorded_epoch) == (1, 5) + + +async def test_two_claims_of_one_epoch_the_loser_is_fenced_as_a_token_mismatch( + tmp_path: Path, +) -> None: + async with orphan_fixture(tmp_path) as fx: + await fx.collect(bindings=["objects"]) + (fx.fence_dir() / "0000000000000001.json").unlink() + fx.write_fence(1) # another claim of epoch 1, under another token + result = await fx.collect(bindings=["objects"]) + assert fenced_cause(result) == "token_mismatch" + + +async def test_a_lost_race_for_the_next_epoch_is_a_lost_claim(tmp_path: Path) -> None: + async with orphan_fixture(tmp_path) as fx: + await fx.collect(bindings=["objects"]) + raced = False + + async def other_database_claims_first(put: Any) -> None: + nonlocal raced + if "_collector-fence/0000000000000002" in put.object_ref and not raced: + raced = True + fx.write_fence(2) # another database wins the create-only write + + fx.provider.on_put = other_database_claims_first + result = await fx.collect(bindings=["objects"]) + assert fenced_cause(result) == "claim_lost" + outcome = binding_result(result, "objects") + assert outcome.fence is not None and outcome.fence.fence_epoch == 2 + + +@pytest.mark.parametrize( + "damage", + ["garbage", "other_namespace", "oversized", "extra_field", "bad_token", "wrong_epoch"], +) +async def test_malformed_foreign_namespace_or_oversized_fence_objects_fail_closed( + tmp_path: Path, damage: str +) -> None: + async with orphan_fixture(tmp_path) as fx: + await fx.put_object(fx.key("acct/orphan.jsonl")) + good = str(uuid.uuid4()) + if damage == "garbage": + fx.write_fence(1, raw=b"not json\n") + elif damage == "other_namespace": + fx.write_fence(1, namespace="f" * 64) + elif damage == "oversized": + fx.write_fence(1, raw=b"x" * 5000) + elif damage == "extra_field": + fx.write_fence( + 1, + raw=json.dumps( + { + "v": 1, + "namespace_key": fx.namespace_key, + "epoch": 1, + "token": good, + "previous_token": None, + "claimed_at": "2026-01-01T00:00:00.000Z", + "extra": 1, + } + ).encode(), + ) + elif damage == "bad_token": + fx.write_fence(1, token="not-a-uuid") + else: + fx.write_fence(1) + (fx.fence_dir() / "0000000000000001.json").rename( + fx.fence_dir() / "0000000000000002.json" + ) + result = await fx.collect(bindings=["objects"]) + assert fenced_cause(result) == "fence_unreadable", damage + assert result.deleted == 0 and fx.on_disk(fx.key("acct/orphan.jsonl")) + + +async def test_two_versions_of_one_epoch_are_ambiguous_and_fail_closed(tmp_path: Path) -> None: + from adcp_reporting_storage.ledger.row_storage_object import ( + ReportingRowObjectListEntryV1, + ReportingRowObjectListPageV1, + ) + + async with orphan_fixture(tmp_path) as fx: + await fx.collect(bindings=["objects"]) + key = f"orph/{fx.namespace_key}/_collector-fence/0000000000000001.json" + duplicate = ReportingRowObjectListPageV1( + tuple( + ReportingRowObjectListEntryV1(key, f"{index}:1:1:1", 10, "2026-01-01T00:00:00.000Z") + for index in (1, 2) + ), + None, + ) + fx.provider.fence_list_override = duplicate + result = await fx.collect(bindings=["objects"]) + assert fenced_cause(result) == "fence_unreadable" + + +async def test_an_unfinished_claim_is_completed_by_the_next_pass(tmp_path: Path) -> None: + async with orphan_fixture(tmp_path) as fx: + fx.provider.put_failures.append(RuntimeError("bucket unavailable")) + crashed = await fx.collect(bindings=["objects"]) + outcome = binding_result(crashed, "objects") + assert outcome.status == "failed" and outcome.fence is None, "a failure is not a fence" + assert fx.fence_files() == [], "the object was never written" + row = await fx.fence_row() + assert row is not None and row[0] == 1, "the intent was committed first" + recovered = await fx.collect(bindings=["objects"]) + assert recovered.fenced == () and binding_result(recovered, "objects").status == "complete" + assert fx.fence_files() == ["0000000000000001.json"] + done = await fx.fence_row() + assert done is not None and done[1] != row[1], "a finished claim mints a fresh token" + + +async def test_a_copy_taken_while_a_claim_was_unfinished_cannot_adopt_the_same_token( + tmp_path: Path, +) -> None: + async with orphan_fixture(tmp_path) as fx: + fx.provider.put_failures.append(RuntimeError("bucket unavailable")) + await fx.collect(bindings=["objects"]) # intent committed, object never written + schema = await clone_schema(fx.pool) # the copy carries the same unfinished intent + original = await fx.collect(bindings=["objects"]) + assert original.fenced == () + async with cloned_store(fx, schema) as clone: + result = await clone.collect_orphaned_row_objects( + now=_later(), acknowledge_exclusive_namespace=True, bindings=["objects"] + ) + assert fenced_cause(result) == "token_mismatch" + assert result.deleted == 0 + + +async def test_two_hosts_of_the_same_database_collecting_at_once_do_not_fence_each_other( + tmp_path: Path, +) -> None: + async with orphan_fixture(tmp_path) as fx: + for index in range(6): + await fx.put_object(fx.key(f"acct/o{index}.jsonl")) + first, second = await asyncio.gather( + fx.collect(bindings=["objects"]), fx.collect(bindings=["objects"]) + ) + assert first.fenced == () and second.fenced == () + assert first.deleted + second.deleted >= 6 + row = await fx.fence_row() + assert row is not None and row[0] == 2 + + +async def test_another_database_claiming_mid_pass_stops_the_deletes_at_the_next_check( + tmp_path: Path, +) -> None: + async with orphan_fixture(tmp_path) as fx: + refs = [fx.key(f"acct/o{index:03d}.jsonl") for index in range(60)] + for ref in refs: + await fx.put_object(ref) + deleted = 0 + + async def other_database_claims(object_ref: str) -> None: + nonlocal deleted + deleted += 1 + if deleted == 10: + fx.write_fence(99) # a newer claim appears while this pass is running + + fx.provider.on_delete = other_database_claims + result = await fx.collect(bindings=["objects"]) + outcome = binding_result(result, "objects") + assert outcome.status == "fenced" and fenced_cause(result) == "fence_advanced" + assert outcome.deleted == 25, "a check before the first delete and every 25 after it" + assert sum(fx.on_disk(ref) for ref in refs) == 35 + assert result.deleted == 25, "a mid-pass refusal keeps what went before" + + +async def test_the_fence_is_claimed_before_the_first_delete_of_a_pass(tmp_path: Path) -> None: + async with orphan_fixture(tmp_path) as fx: + await fx.put_object(fx.key("acct/o.jsonl")) + seen: list[list[str]] = [] + + async def look(object_ref: str) -> None: + seen.append(fx.fence_files()) + + fx.provider.on_delete = look + await fx.collect(bindings=["objects"]) + assert seen == [["0000000000000001.json"]] + + +async def test_without_the_fence_table_a_real_pass_and_a_dry_run_refuse_and_delete_nothing( + tmp_path: Path, +) -> None: + async with orphan_fixture(tmp_path) as fx: + orphan = fx.key("acct/o.jsonl") + await fx.put_object(orphan) + async with fx.pool.connection() as connection: + await connection.execute("DROP TABLE adcp_reporting_orphan_collector_fences") + for dry_run in (False, True): + result = await fx.collect(bindings=["objects"], dry_run=dry_run) + assert fenced_cause(result) == "migration_required" + assert result.deleted == 0 and result.would_delete == 0 and fx.on_disk(orphan) + with pytest.raises(ReportingRowStoreError, match="fence table does not exist"): + await fx.store.take_over_orphan_collector_fence(acknowledge_previous_owner_gone=True) + + +async def test_the_fence_table_is_part_of_the_row_storage_ddl_and_reapplying_it_changes_nothing( + tmp_path: Path, +) -> None: + from adcp_reporting_storage.ledger.row_storage import REPORTING_ROW_STORAGE_DDL_PATH + + async with orphan_fixture(tmp_path) as fx: + await fx.collect(bindings=["objects"]) + before = await fx.fence_row() + async with fx.pool.connection() as connection: + await connection.execute(REPORTING_ROW_STORAGE_DDL_PATH.read_text()) + assert await fx.fence_row() == before + columns = await fetch_all( + fx.pool, + "SELECT column_name FROM information_schema.columns WHERE table_schema =" + " current_schema() AND table_name = 'adcp_reporting_orphan_collector_fences'" + " ORDER BY column_name", + ) + assert [row[0] for row in columns] == [ + "claimed_at", + "epoch", + "fence_scope", + "previous_token", + "token", + ] + + +async def test_a_provider_failure_while_reading_or_writing_the_fence_is_failed_not_fenced( + tmp_path: Path, +) -> None: + async with orphan_fixture(tmp_path) as fx: + await fx.put_object(fx.key("acct/o.jsonl")) + fx.provider.fence_list_failures.append(ReportingRowStoreError("PROVIDER_UNAVAILABLE")) + read = await fx.collect(bindings=["objects"]) + outcome = binding_result(read, "objects") + assert outcome.status == "failed" and outcome.error == "PROVIDER_UNAVAILABLE" + assert read.fenced == () and read.deleted == 0 + fx.provider.put_failures.append(RuntimeError("down")) + write = await fx.collect(bindings=["objects"]) + assert binding_result(write, "objects").status == "failed" and write.fenced == () + ok = await fx.collect(bindings=["objects"]) + assert ok.deleted == 1 + + +async def test_a_failed_claim_keeps_the_resume_position_of_a_listing(tmp_path: Path) -> None: + async with orphan_fixture(tmp_path) as fx: + for index in range(4): + await fx.put_object(fx.key(f"acct/o{index}.jsonl")) + partial = await fx.collect(bindings=["objects"], limit=2, page_size=2) + cursor = partial.cursors["objects"] + fx.provider.fence_list_failures.append(ReportingRowStoreError("PROVIDER_UNAVAILABLE")) + again = await fx.collect( + bindings=["objects"], limit=2, page_size=2, cursors={"objects": cursor} + ) + assert binding_result(again, "objects").status == "failed" + assert again.cursors == {"objects": cursor}, "the listing never started" + + +async def test_a_takeover_recovers_every_refusal_and_moves_past_a_database_ahead_of_its_bucket( + tmp_path: Path, +) -> None: + async with orphan_fixture(tmp_path) as fx: + # claimed_elsewhere + fx.write_fence(3) + assert fenced_cause(await fx.collect(bindings=["objects"])) == "claimed_elsewhere" + await fx.store.take_over_orphan_collector_fence(acknowledge_previous_owner_gone=True) + assert (await fx.collect(bindings=["objects"])).fenced == () + row = await fx.fence_row() + assert row is not None and row[0] >= 4 + # fence_regressed: the database is ahead of its bucket + async with fx.pool.connection() as connection: + await connection.execute( + "UPDATE adcp_reporting_orphan_collector_fences SET epoch = 50, token = %s", + (str(uuid.uuid4()),), + ) + assert fenced_cause(await fx.collect(bindings=["objects"])) == "fence_regressed" + await fx.store.take_over_orphan_collector_fence(acknowledge_previous_owner_gone=True) + assert (await fx.collect(bindings=["objects"])).fenced == () + row = await fx.fence_row() + assert row is not None and row[0] > 50, "a takeover moves past a database that is ahead" + # fence_unreadable + fx.write_fence(row[0] + 1, raw=b"damaged") + assert fenced_cause(await fx.collect(bindings=["objects"])) == "fence_unreadable" + await fx.store.take_over_orphan_collector_fence(acknowledge_previous_owner_gone=True) + assert (await fx.collect(bindings=["objects"])).fenced == () + assert len(fx.fence_files()) == 2 or len(fx.fence_files()) == 1 + + +async def test_a_fenced_clone_refuses_every_binding_that_shares_the_fence_with_one_check( + tmp_path: Path, +) -> None: + async with orphan_fixture(tmp_path) as fx: + fx.write_fence(7) + fx.provider.list_calls.clear() + result = await fx.collect(bindings=["objects", "twin"], include_read_only=True) + assert set(result.fenced) == {"objects", "twin"} + assert result.cursors == {} + # The fence chain was read once for the shared scope, and no object listing began. + listings = fx.provider.list_calls + assert all("_collector-fence" in prefix for prefix in listings), listings + + +async def test_a_fenced_namespace_never_rests_a_whole_interval_in_the_maintenance_cycle( + tmp_path: Path, +) -> None: + async with orphan_fixture(tmp_path) as fx: + options = ReportingOrphanCollectionOptionsV1( + acknowledge_exclusive_namespace=True, interval_seconds=86_400 + ) + state = ReportingOrphanCollectionStateV1() + fx.write_fence(4) + first = await run_orphan_collection_pass(fx.store, options, state) + assert first.fenced == ("objects",) + assert state.next_cycle_at is not None + assert state.next_cycle_at - time.monotonic() <= 3_600 + 1, "looks again within the hour" + # The operator fixes it; the next cycle (the hour elapsing) resumes collection. + await fx.store.take_over_orphan_collector_fence(acknowledge_previous_owner_gone=True) + state.next_cycle_at = time.monotonic() - 1 + resumed = await run_orphan_collection_pass(fx.store, options, state) + assert resumed.fenced == () + assert state.next_cycle_at is not None + assert state.next_cycle_at - time.monotonic() > 3_600, "a clean cycle rests the interval" + + +def _later() -> Any: + from ._orphan_support import later + + return later() + + +def test_the_hook_provider_is_a_real_provider_shape() -> None: + assert HookedProvider.name == "filesystem" + + +async def test_a_listing_that_skipped_a_fence_entry_fails_closed_rather_than_reading_no_claim( + tmp_path: Path, +) -> None: + from adcp_reporting_storage.ledger.row_storage_object import ReportingRowObjectListPageV1 + + async with orphan_fixture(tmp_path) as fx: + await fx.put_object(fx.key("acct/orphan.jsonl")) + # The provider could not address an entry in the fence directory: it might be the newest + # claim, and an empty-looking chain would otherwise read as "never claimed". + fx.provider.fence_list_override = ReportingRowObjectListPageV1((), None, skipped=1) + result = await fx.collect(bindings=["objects"]) + assert fenced_cause(result) == "fence_unreadable" + assert result.deleted == 0 and fx.on_disk(fx.key("acct/orphan.jsonl")) + + +async def test_a_cancelled_claim_leaves_no_advisory_lock_on_a_pooled_connection( + tmp_path: Path, +) -> None: + async with orphan_fixture(tmp_path) as fx: + await fx.collect(bindings=["objects"]) + storage = fx.store._row_storage + assert storage is not None + scope, _, _ = await storage.collector_fence_location("objects") + holder = await fx.pool.getconn() + try: + await holder.execute( + "SELECT pg_advisory_lock(hashtextextended(%s, 0))", (f"adcp.orphan_fence:{scope}",) + ) + task = asyncio.create_task(fx.collect(bindings=["objects"])) + for _ in range(200): + rows = await fetch_all( + fx.pool, + "SELECT count(*) FROM pg_locks WHERE locktype = 'advisory' AND NOT granted", + ) + if rows[0][0]: + break + await asyncio.sleep(0.025) + task.cancel() + (outcome,) = await asyncio.gather(task, return_exceptions=True) + assert isinstance(outcome, asyncio.CancelledError) + await holder.execute( + "SELECT pg_advisory_unlock(hashtextextended(%s, 0))", + (f"adcp.orphan_fence:{scope}",), + ) + await holder.commit() + finally: + await fx.pool.putconn(holder) + held = await fetch_all( + fx.pool, + "SELECT count(*) FROM pg_locks WHERE locktype = 'advisory' AND granted" + " AND pid <> pg_backend_pid()", + ) + assert held == [(0,)], "no pooled connection kept a session lock" + again = await asyncio.wait_for(fx.collect(bindings=["objects"]), 20) + assert again.fenced == () + + +async def test_binding_name_arguments_must_be_sequences_of_ids_not_strings( + tmp_path: Path, +) -> None: + async with orphan_fixture(tmp_path) as fx: + for name in ("bindings", "skip_bindings"): + for bad in ("objects", "", b"objects", [1], ["objects", None]): + with pytest.raises(ValueError, match=name): + await fx.collect(dry_run=True, **{name: bad}) + with pytest.raises(ValueError, match="bindings"): + await fx.store.take_over_orphan_collector_fence( + acknowledge_previous_owner_gone=True, bindings="objects" # type: ignore[arg-type] + ) diff --git a/tests/fixtures/public_api_snapshot.json b/tests/fixtures/public_api_snapshot.json index c176447468..cc4edb3fe2 100644 --- a/tests/fixtures/public_api_snapshot.json +++ b/tests/fixtures/public_api_snapshot.json @@ -8199,6 +8199,13 @@ "ReportingChangeRecordV1": "adcp_reporting_storage.ledger.change_feed", "ReportingChangesPageV1": "adcp_reporting_storage.ledger.change_feed", "ReportingObjectRowBindingV1": "adcp_reporting_storage.ledger.row_storage", + "ReportingOrphanBindingResultV1": "adcp_reporting_storage.ledger.orphan_collector", + "ReportingOrphanCandidateV1": "adcp_reporting_storage.ledger.orphan_collector", + "ReportingOrphanCollectionOptionsV1": "adcp_reporting_storage.ledger.orphan_collector", + "ReportingOrphanCollectionResultV1": "adcp_reporting_storage.ledger.orphan_collector", + "ReportingOrphanCollectionStateV1": "adcp_reporting_storage.ledger.orphan_collector", + "ReportingOrphanFenceRefusalV1": "adcp_reporting_storage.ledger.orphan_fence", + "ReportingOrphanFenceTakeoverV1": "adcp_reporting_storage.ledger.orphan_collector", "ReportingOwnershipBackfill": "adcp_reporting_storage.migration", "ReportingOwnershipMigrationError": "adcp_reporting_storage.migration", "ReportingPeriodRetirementV1": "adcp_reporting_storage.ledger.period_retirement", diff --git a/tests/test_reporting_orphan_fence.py b/tests/test_reporting_orphan_fence.py new file mode 100644 index 0000000000..6f818b456d --- /dev/null +++ b/tests/test_reporting_orphan_fence.py @@ -0,0 +1,207 @@ +"""The orphan collector's fence decision table, formats and scope, without a database. + +Ported from the JavaScript SDK (adcp-client #3593, ``reporting-orphan-fence``). +""" + +from __future__ import annotations + +import json +import uuid + +import pytest +from adcp_reporting_storage.ledger.orphan_fence import ( + ORPHAN_FENCE_MAX_BYTES, + OrphanFenceRecord, + OrphanFenceRow, + decide_orphan_fence, + encode_orphan_fence_record, + is_orphan_fence_key, + orphan_fence_directory, + orphan_fence_epoch_of_key, + orphan_fence_key, + orphan_fence_scope, + parse_orphan_fence_record, +) + +NAMESPACE = "a" * 64 +OWNED = f"orph/{NAMESPACE}/" +T1, T2, T3 = (str(uuid.uuid4()) for _ in range(3)) + + +def row(epoch: int, token: str = T1, previous: str | None = None) -> OrphanFenceRow: + return OrphanFenceRow(epoch, token, previous) + + +def test_it_claims_epoch_one_when_neither_side_has_a_claim_and_the_next_when_they_match() -> None: + first = decide_orphan_fence(None, None) + assert (first.action, first.epoch, first.previous_token) == ("claim", 1, None) + nxt = decide_orphan_fence(row(4, T1), (4, T1)) + assert (nxt.action, nxt.epoch, nxt.previous_token) == ("claim", 5, T1) + + +def test_it_refuses_a_namespace_claimed_by_a_database_it_has_no_record_of() -> None: + decision = decide_orphan_fence(None, (3, T1)) + assert decision.action == "fenced" and decision.refusal is not None + assert decision.refusal.cause == "claimed_elsewhere" + assert (decision.refusal.fence_epoch, decision.refusal.recorded_epoch) == (3, None) + assert "take_over_orphan_collector_fence" in decision.refusal.detail + + +def test_it_finishes_its_own_unfinished_claim() -> None: + # The object is missing: the first claim was recorded and never written. + assert decide_orphan_fence(row(1, T1, None), None).action == "complete" + # One epoch behind with the recorded previous token: the intent committed, the write did not. + assert decide_orphan_fence(row(5, T2, T1), (4, T1)).action == "complete" + + +@pytest.mark.parametrize( + ("recorded", "latest", "cause"), + [ + (row(3, T1, T2), None, "fence_missing"), + (row(1, T1, T2), None, "fence_missing"), + (row(4, T1), (4, T2), "token_mismatch"), + (row(4, T1), (6, T1), "fence_advanced"), + (row(4, T1), (2, T1), "fence_regressed"), + (row(5, T2, T1), (4, T3), "fence_regressed"), + (row(5, T2, None), (4, T1), "fence_regressed"), + ], +) +def test_it_refuses_every_other_state_with_the_epochs_in_the_refusal( + recorded: OrphanFenceRow, latest: tuple[int, str] | None, cause: str +) -> None: + decision = decide_orphan_fence(recorded, latest) + assert decision.action == "fenced" and decision.refusal is not None + assert decision.refusal.cause == cause + assert decision.refusal.recorded_epoch == recorded.epoch + assert decision.refusal.fence_epoch == (latest[0] if latest else None) + + +def test_no_input_makes_it_delete_when_the_bucket_is_ahead_of_the_record() -> None: + for recorded in (None, row(1), row(7, T1, T2)): + for epoch in range(1, 12): + floor = recorded.epoch if recorded else 0 + if epoch <= floor: + continue + for token in (T1, T2, T3): + assert decide_orphan_fence(recorded, (epoch, token)).action == "fenced" + + +def test_a_record_round_trips_and_a_first_claim_carries_a_null_previous_token() -> None: + record = OrphanFenceRecord(1, T1, None, "2026-10-01T00:00:00.000Z") + data = encode_orphan_fence_record(NAMESPACE, record) + assert data.endswith(b"\n") and json.loads(data)["previous_token"] is None + assert list(json.loads(data)) == [ + "v", + "namespace_key", + "epoch", + "token", + "previous_token", + "claimed_at", + ], "the key order the JavaScript SDK writes" + assert parse_orphan_fence_record(data, namespace_key=NAMESPACE, epoch=1) == record + chained = OrphanFenceRecord(2, T2, T1, "2026-10-02T00:00:00.000Z") + assert ( + parse_orphan_fence_record( + encode_orphan_fence_record(NAMESPACE, chained), namespace_key=NAMESPACE, epoch=2 + ) + == chained + ) + + +def test_it_parses_strictly_and_rejects_anything_that_is_not_exactly_a_fence_of_this_epoch() -> ( + None +): + good = { + "v": 1, + "namespace_key": NAMESPACE, + "epoch": 3, + "token": T1, + "previous_token": T2, + "claimed_at": "2026-10-01T00:00:00.000Z", + } + + def parse(value: object, epoch: int = 3, namespace: str = NAMESPACE) -> object: + data = value if isinstance(value, bytes) else json.dumps(value).encode() + return parse_orphan_fence_record(data, namespace_key=namespace, epoch=epoch) + + assert parse(good) is not None + assert parse(good, epoch=4) is None, "another epoch" + assert parse(good, namespace="b" * 64) is None, "another namespace" + for change in ( + {"v": 2}, + {"v": True}, + {"epoch": 4}, + {"epoch": "3"}, + {"epoch": True}, + {"token": "nope"}, + {"token": T1.upper()}, + {"previous_token": "nope"}, + {"previous_token": 5}, + {"claimed_at": "yesterday"}, + {"claimed_at": "2026-10-01"}, + {"claimed_at": 5}, + {"extra": 1}, + ): + assert parse({**good, **change}) is None, change + missing = dict(good) + del missing["previous_token"] + assert parse(missing) is None + for bad in (b"", b"not json", b"[]", b"null", b"3", b"\xff\xfe"): + assert parse(bad) is None + assert parse(b" " * (ORPHAN_FENCE_MAX_BYTES + 1)) is None + + +def test_keys_sort_by_epoch_and_only_exact_epoch_names_under_the_fence_directory_count() -> None: + assert orphan_fence_directory(OWNED) == f"{OWNED}_collector-fence/" + keys = [orphan_fence_key(OWNED, epoch) for epoch in (2, 10, 1, 9)] + assert sorted(keys) == [orphan_fence_key(OWNED, epoch) for epoch in (1, 2, 9, 10)] + assert orphan_fence_key(OWNED, 7).endswith("/0000000000000007.json") + assert orphan_fence_epoch_of_key(OWNED, orphan_fence_key(OWNED, 12)) == 12 + directory = orphan_fence_directory(OWNED) + for other in ( + f"{directory}7.json", + f"{directory}00000000000000007.json", + f"{directory}0000000000000000.json", + f"{directory}0000000000000007.json.tmp", + f"{directory}0000000000000007.JSON", + f"{directory}٣000000000000007.json", + f"{directory}sub/0000000000000007.json", + f"{OWNED}0000000000000007.json", + ): + assert orphan_fence_epoch_of_key(OWNED, other) is None, other + assert is_orphan_fence_key(OWNED, f"{directory}anything") + assert not is_orphan_fence_key(OWNED, f"{OWNED}acct/row.jsonl") + + +def test_a_fence_is_one_physical_place_provider_location_prefix_and_namespace() -> None: + base = { + "provider": "s3", + "location": {"bucket": "rows", "region": "us-east-1"}, + "prefix": "adcp-rows", + "namespace_key": NAMESPACE, + } + scope = orphan_fence_scope(**base) # type: ignore[arg-type] + assert len(scope) == 64 and int(scope, 16) >= 0 + assert orphan_fence_scope(**{**base, "location": {"region": "us-east-1", "bucket": "rows"}}) == scope # type: ignore[arg-type] + for change in ( + {"provider": "gcs"}, + {"location": {"bucket": "other", "region": "us-east-1"}}, + {"prefix": "other"}, + {"namespace_key": "b" * 64}, + ): + assert orphan_fence_scope(**{**base, **change}) != scope, change # type: ignore[arg-type] + + +def test_the_scope_hash_is_the_one_the_javascript_sdk_computes() -> None: + """sha256 of the RFC 8785 form of ["adcp.orphan_fence.v1", provider, location, prefix, ns].""" + import hashlib + + canonical = ( + '["adcp.orphan_fence.v1","filesystem",{"root":"rows"},"orph","' + NAMESPACE + '"]' + ).encode() + assert ( + orphan_fence_scope( + provider="filesystem", location={"root": "rows"}, prefix="orph", namespace_key=NAMESPACE + ) + == hashlib.sha256(canonical).hexdigest() + ) diff --git a/tests/test_reporting_orphan_maintenance.py b/tests/test_reporting_orphan_maintenance.py new file mode 100644 index 0000000000..6e8034254a --- /dev/null +++ b/tests/test_reporting_orphan_maintenance.py @@ -0,0 +1,201 @@ +"""Scheduling the orphan collector: options, the store option and the cycle state. + +Ported from the JavaScript SDK (adcp-client #3291 and #3593, ``reporting-maintenance-warnings`` and +``reliable-reporting-service``). No database: the collector itself is replaced. +""" + +from __future__ import annotations + +import time +from dataclasses import replace +from typing import Any + +import pytest +from adcp_reporting_storage import PgReportingLedgerStore +from adcp_reporting_storage.ledger import orphan_collector +from adcp_reporting_storage.ledger.orphan_collector import ( + ReportingOrphanBindingResultV1, + ReportingOrphanCollectionOptionsV1, + ReportingOrphanCollectionResultV1, + ReportingOrphanCollectionStateV1, + run_orphan_collection_pass, +) +from adcp_reporting_storage.ledger.orphan_fence import ReportingOrphanFenceRefusalV1 + + +def options(**overrides: Any) -> ReportingOrphanCollectionOptionsV1: + return ReportingOrphanCollectionOptionsV1( + **{"acknowledge_exclusive_namespace": True, **overrides} + ) + + +def test_deleting_needs_the_acknowledgement_and_every_bound_is_checked() -> None: + with pytest.raises(ValueError, match="acknowledge_exclusive_namespace"): + ReportingOrphanCollectionOptionsV1() + assert ReportingOrphanCollectionOptionsV1(dry_run=True).dry_run is True + assert options().interval_seconds == 86_400 and options().min_age_seconds == 7 * 86_400 + assert options(min_age_seconds=86_400).min_age_seconds == 86_400 + for bad in ( + {"min_age_seconds": 86_399}, + {"min_age_seconds": 0}, + {"min_age_seconds": 1.5}, + {"limit": 0}, + {"limit": -1}, + {"limit": 1.5}, + {"interval_seconds": 0}, + {"interval_seconds": True}, + {"dry_run": "yes"}, + {"include_read_only": 1}, + {"acknowledge_exclusive_namespace": "true"}, + ): + with pytest.raises(ValueError): + options(**bad) + + +def test_the_store_option_needs_row_storage_and_adds_the_maintenance_operation() -> None: + pytest.importorskip("psycopg_pool") + pool = object() + with pytest.raises(ValueError, match="requires row_storage"): + PgReportingLedgerStore(pool=pool, orphan_collection=options()) # type: ignore[arg-type] + with pytest.raises(TypeError): + PgReportingLedgerStore( + pool=pool, # type: ignore[arg-type] + row_storage=True, + orphan_collection=True, # type: ignore[arg-type] + ) + plain = PgReportingLedgerStore(pool=pool, row_storage=True) # type: ignore[arg-type] + assert "orphan_collection" not in [name for name, _ in plain.maintenance_operations()] + scheduled = PgReportingLedgerStore( + pool=pool, # type: ignore[arg-type] + row_storage=True, + orphan_collection=options(), + ) + names = [name for name, _ in scheduled.maintenance_operations()] + assert names[-1] == "orphan_collection" and "intent_sweep" in names + + +def result(**changes: Any) -> ReportingOrphanCollectionResultV1: + return replace(ReportingOrphanCollectionResultV1(dry_run=False), **changes) + + +def binding(binding_id: str, status: Any = "complete", **changes: Any) -> Any: + return ReportingOrphanBindingResultV1(binding_id, status, **changes) + + +class Recorder: + def __init__(self, *outcomes: ReportingOrphanCollectionResultV1) -> None: + self.outcomes = list(outcomes) + self.calls: list[dict[str, Any]] = [] + + async def __call__(self, store: Any, **kwargs: Any) -> ReportingOrphanCollectionResultV1: + self.calls.append(kwargs) + return self.outcomes.pop(0) + + +async def test_a_pass_forwards_its_options_replaces_the_saved_state_and_rests_after_a_clean_listing( + monkeypatch: pytest.MonkeyPatch, +) -> None: + recorder = Recorder( + result( + cursors={"objects": "page-2"}, + completed=("legacy",), + bindings=(binding("objects", "partial"),), + ), + result(cursors={}, completed=("objects",), bindings=(binding("objects"),)), + ) + monkeypatch.setattr(orphan_collector, "collect_orphaned_row_objects", recorder) + configured = options( + dry_run=False, + limit=50, + min_age_seconds=3 * 86_400, + include_read_only=True, + interval_seconds=600, + ) + state = ReportingOrphanCollectionStateV1(cursors={"stale": "x"}, completed=["done"]) + await run_orphan_collection_pass(object(), configured, state) # type: ignore[arg-type] + assert recorder.calls[0] == { + "dry_run": False, + "acknowledge_exclusive_namespace": True, + "min_age_seconds": 3 * 86_400, + "limit": 50, + "include_read_only": True, + "cursors": {"stale": "x"}, + "skip_bindings": ["done"], + } + # An unfinished listing carries on next pass, remembering what already finished. + assert state.cursors == {"objects": "page-2"} + assert state.completed == ["done", "legacy"] and state.next_cycle_at is None + await run_orphan_collection_pass(object(), configured, state) # type: ignore[arg-type] + assert recorder.calls[1]["cursors"] == {"objects": "page-2"} + assert state.cursors == {} and state.completed == [] + assert state.next_cycle_at is not None + assert 590 < state.next_cycle_at - time.monotonic() <= 600, "a clean cycle rests the interval" + # Between full listings the pass does nothing, and does not call the collector. + idle = await run_orphan_collection_pass(object(), configured, state) # type: ignore[arg-type] + assert idle.listed == 0 and idle.dry_run is False and len(recorder.calls) == 2 + + +async def test_a_binding_that_finished_is_not_listed_again_while_another_needs_more_passes( + monkeypatch: pytest.MonkeyPatch, +) -> None: + recorder = Recorder( + result( + cursors={"big": "t1"}, + completed=("small",), + bindings=(binding("small"), binding("big", "partial")), + ), + result(cursors={"big": "t2"}, completed=(), bindings=(binding("big", "partial"),)), + result(cursors={}, completed=("big",), bindings=(binding("big"),)), + ) + monkeypatch.setattr(orphan_collector, "collect_orphaned_row_objects", recorder) + state = ReportingOrphanCollectionStateV1() + for _ in range(3): + await run_orphan_collection_pass(object(), options(), state) # type: ignore[arg-type] + assert [call["skip_bindings"] for call in recorder.calls] == [[], ["small"], ["small"]] + assert state.completed == [] and state.cursors == {} and state.next_cycle_at is not None + + +async def test_a_failed_binding_keeps_the_task_retrying_every_pass( + monkeypatch: pytest.MonkeyPatch, +) -> None: + failing = result( + cursors={}, failed=1, bindings=(binding("objects", "failed", error="PROVIDER_UNAVAILABLE"),) + ) + recorder = Recorder(failing, failing) + monkeypatch.setattr(orphan_collector, "collect_orphaned_row_objects", recorder) + state = ReportingOrphanCollectionStateV1() + for _ in range(2): + await run_orphan_collection_pass(object(), options(), state) # type: ignore[arg-type] + assert state.next_cycle_at is None, "a failure does not rest the interval" + assert len(recorder.calls) == 2 + + +async def test_a_fenced_binding_rests_at_most_an_hour_whatever_the_interval( + monkeypatch: pytest.MonkeyPatch, +) -> None: + refusal = ReportingOrphanFenceRefusalV1("fence_advanced", "take over", 5, 3) + fenced = result(fenced=("objects",), bindings=(binding("objects", "fenced", fence=refusal),)) + recorder = Recorder(fenced, fenced) + monkeypatch.setattr(orphan_collector, "collect_orphaned_row_objects", recorder) + state = ReportingOrphanCollectionStateV1() + await run_orphan_collection_pass(object(), options(interval_seconds=86_400), state) # type: ignore[arg-type] + assert state.next_cycle_at is not None + assert 3_590 < state.next_cycle_at - time.monotonic() <= 3_600 + # A short interval is kept as it is. + short = ReportingOrphanCollectionStateV1() + await run_orphan_collection_pass(object(), options(interval_seconds=120), short) # type: ignore[arg-type] + assert short.next_cycle_at is not None + assert 110 < short.next_cycle_at - time.monotonic() <= 120 + + +async def test_a_rejected_collector_propagates_so_the_worker_counts_it_as_a_failed_task( + monkeypatch: pytest.MonkeyPatch, +) -> None: + async def explode(store: Any, **kwargs: Any) -> Any: + raise RuntimeError("boom") + + monkeypatch.setattr(orphan_collector, "collect_orphaned_row_objects", explode) + state = ReportingOrphanCollectionStateV1(cursors={"a": "b"}) + with pytest.raises(RuntimeError): + await run_orphan_collection_pass(object(), options(), state) # type: ignore[arg-type] + assert state.cursors == {"a": "b"} and state.next_cycle_at is None diff --git a/tests/test_reporting_production_maintenance.py b/tests/test_reporting_production_maintenance.py index d814590a6b..4341c957fa 100644 --- a/tests/test_reporting_production_maintenance.py +++ b/tests/test_reporting_production_maintenance.py @@ -220,3 +220,51 @@ async def test_a_quiet_pass_logs_nothing_and_still_records_its_counters( "protected": 0, "held": 0, } + + +async def test_orphan_collection_reports_fenced_and_unsupported_bindings_as_integer_counters( + caplog: pytest.LogCaptureFixture, +) -> None: + """A fenced namespace is stalled work; a binding that cannot list is pending work. + + Only the closed operation name and counters are logged, never a binding or bucket name. + """ + from types import SimpleNamespace + + class OrphanStore: + def maintenance_operations(self) -> tuple[tuple[str, Any], ...]: + async def run() -> Any: + return SimpleNamespace( + failed=2, + fenced=("secret-binding", "other-binding"), + unsupported=("nolist-binding",), + deleted=9, + ) + + return (("orphan_collection", run),) + + worker = service(OrphanStore()) + with caplog.at_level(logging.INFO, logger="adcp.reporting.production"): + await worker._maintain() + by_code = { + code: [r for r in caplog.records if getattr(r, "code", "") == code] + for code in ( + "REPORTING_MAINTENANCE_FAILED", + "REPORTING_MAINTENANCE_PENDING", + "REPORTING_MAINTENANCE_STALLED", + ) + } + assert [r.operation for r in by_code["REPORTING_MAINTENANCE_FAILED"]] == ["orphan_collection"] + (pending,) = by_code["REPORTING_MAINTENANCE_PENDING"] + assert pending.operation == "orphan_collection" and pending.unsupported == 1 + (stalled,) = by_code["REPORTING_MAINTENANCE_STALLED"] + assert stalled.operation == "orphan_collection" and stalled.fenced == 2 + assert stalled.levelno == logging.WARNING + assert all( + "binding" not in r.getMessage() and "secret" not in r.getMessage() for r in caplog.records + ) + assert worker.maintenance_report["orphan_collection"] == { + "failed": 2, + "fenced": 2, + "unsupported": 1, + } diff --git a/tests/test_reporting_row_storage_azure.py b/tests/test_reporting_row_storage_azure.py index c42c2b5af2..2bf22c0685 100644 --- a/tests/test_reporting_row_storage_azure.py +++ b/tests/test_reporting_row_storage_azure.py @@ -3,6 +3,7 @@ from __future__ import annotations from dataclasses import dataclass, field +from datetime import datetime, timezone from types import SimpleNamespace from typing import Any @@ -18,6 +19,7 @@ ReportingRowObjectContextV1, ReportingRowObjectDeleteInputV1, ReportingRowObjectGetInputV1, + ReportingRowObjectListInputV1, ReportingRowObjectPutInputV1, run_reporting_row_object_provider_conformance_v1, ) @@ -43,6 +45,7 @@ class _Version: content_type: str | None cache_control: str | None current: bool = True + created: datetime = field(default_factory=lambda: datetime.now(timezone.utc)) @property def etag(self) -> str: @@ -73,6 +76,55 @@ def get_container_properties(self, **transport: Any) -> Any: def get_blob_client(self, key: str) -> Any: return Blob(name, key) + def list_blobs( + self, + name_starts_with: str | None = None, + include: list[str] | None = None, + results_per_page: int | None = None, + **transport: Any, + ) -> Any: + service.transport.append(transport) + entries = [ + SimpleNamespace( + name=key, + version_id=version.version_id, + size=len(version.data), + creation_time=version.created, + last_modified=version.created, + ) + for (container, key), history in sorted(service.blobs.items()) + if container == name and key.startswith(name_starts_with or "") + for version in history + ] + + class Pages: + continuation_token: str | None = None + + def __init__(self, start: int) -> None: + self.start = start + self.served = False + + def __iter__(self) -> Pages: + return self + + def __next__(self) -> Any: + if self.served: + raise StopIteration + self.served = True + size = results_per_page or len(entries) + chunk = entries[self.start : self.start + size] + following = self.start + len(chunk) + self.continuation_token = ( + str(following) if following < len(entries) else None + ) + return iter(chunk) + + class Paged: + def by_page(self, continuation_token: str | None = None) -> Pages: + return Pages(int(continuation_token) if continuation_token else 0) + + return Paged() + class Blob: def __init__(self, container: str, key: str) -> None: self.container, self.key = container, key @@ -216,7 +268,7 @@ async def test_passes_the_shared_provider_conformance_suite() -> None: passed = await run_reporting_row_object_provider_conformance_v1( provider, location=LOCATION, prefix="adcp-rows" ) - assert passed[-1] == "write-unique-version" and "ranged-read" in passed + assert "write-unique-version" in passed and "ranged-read" in passed and passed[-1] == "list" async def test_create_only_upload_sets_headers_and_identifier_safe_metadata() -> None: @@ -395,3 +447,60 @@ def test_the_official_client_exposes_everything_the_provider_calls() -> None: from azure.core import MatchConditions assert hasattr(MatchConditions, "IfNotModified") + + +# -- list --------------------------------------------------------------------------------------- + + +async def test_list_pages_every_version_and_carries_the_call_budget() -> None: + service = FakeBlobService() + provider = create_azure_blob_reporting_row_object_provider_v1(client=service) + keys = [f"adcp-rows/ns/o{index}.jsonl" for index in range(5)] + versions = {} + for key in keys: + versions[key] = (await provider.put_if_absent(put(key, b"abc"), CONTEXT)).native_version_ref + found = [] + token = None + context = ReportingRowObjectContextV1(7.5) + for _ in range(20): + page = await provider.list( + ReportingRowObjectListInputV1(LOCATION, "adcp-rows/ns/", 2, token), context + ) + assert len(page.objects) <= 2 + found.extend(page.objects) + token = page.next_page_token + if token is None: + break + assert {(e.object_ref, e.native_version_ref, e.size_bytes) for e in found} == { + (key, versions[key], 3) for key in keys + } + assert service.transport[-1]["timeout"] == 8 and service.transport[-1]["read_timeout"] == 7.5 + + +async def test_list_skips_a_blob_without_a_version_id_size_or_time_and_counts_it() -> None: + service = FakeBlobService() + provider = create_azure_blob_reporting_row_object_provider_v1(client=service) + await provider.put_if_absent(put("adcp-rows/ns/good.jsonl"), CONTEXT) + # A blob written before versioning was enabled has no version ID. + service.blobs[("rows", "adcp-rows/ns/before-versioning.jsonl")] = [ + type(service.blobs[("rows", "adcp-rows/ns/good.jsonl")][0])( + "not a version id!", b"x", {}, None, None + ) + ] + page = await provider.list( + ReportingRowObjectListInputV1(LOCATION, "adcp-rows/ns/", 10), CONTEXT + ) + assert [e.object_ref for e in page.objects] == ["adcp-rows/ns/good.jsonl"] + assert page.skipped == 1 + + +async def test_list_refuses_a_bad_page_size_and_an_oversized_prefix() -> None: + provider = create_azure_blob_reporting_row_object_provider_v1(client=FakeBlobService()) + for size in (0, -1, 1.5): + with pytest.raises(ReportingRowStoreError) as failure: + await provider.list( + ReportingRowObjectListInputV1(LOCATION, "adcp-rows/", size), CONTEXT # type: ignore[arg-type] + ) + assert failure.value.code == "INVALID_INPUT" + with pytest.raises(ReportingRowStoreError): + await provider.list(ReportingRowObjectListInputV1(LOCATION, "p" * 1025, 5), CONTEXT) diff --git a/tests/test_reporting_row_storage_filesystem.py b/tests/test_reporting_row_storage_filesystem.py index 099f0e0828..cfb4d2a622 100644 --- a/tests/test_reporting_row_storage_filesystem.py +++ b/tests/test_reporting_row_storage_filesystem.py @@ -3,7 +3,10 @@ from __future__ import annotations import os +from dataclasses import replace +from datetime import datetime, timezone from pathlib import Path +from typing import Any import pytest from adcp_reporting_storage.ledger.row_storage_object import ( @@ -11,10 +14,14 @@ ReportingRowObjectContextV1, ReportingRowObjectDeleteInputV1, ReportingRowObjectGetInputV1, + ReportingRowObjectListInputV1, ReportingRowObjectPutInputV1, + ReportingRowObjectStatInputV1, assert_reporting_row_key_template_v1, assert_reporting_row_prefix_v1, create_filesystem_reporting_row_object_provider_v1, + is_row_object_key_shape, + parse_row_instant, render_reporting_row_object_key_v1, reporting_row_account_key_v1, run_reporting_row_object_provider_conformance_v1, @@ -43,6 +50,7 @@ async def test_the_provider_passes_the_shared_conformance_suite(tmp_path: Path) "version-pinned-stat", "exact-version-delete", "write-unique-version", + "list", ] @@ -321,3 +329,252 @@ def key(row_set_id: str) -> str: ) assert key("rrev_1") != key("rrev_2") + + +# -- list (the orphan collector's capability) ------------------------------------------------- + +LIST_NAMESPACE = f"adcp-rows/{NAMESPACE}" + + +async def _put_all(provider: Any, keys: list[str]) -> dict[str, str]: + versions = {} + for key in keys: + created = await provider.put_if_absent( + ReportingRowObjectPutInputV1( + {"root": "rows"}, key, f"{key}\n".encode(), "application/x-ndjson", {} + ), + CONTEXT, + ) + versions[key] = created.native_version_ref + return versions + + +async def _list_all(provider: Any, prefix: str, page_size: int) -> list[Any]: + found: list[Any] = [] + token = None + for _ in range(200): + page = await provider.list( + ReportingRowObjectListInputV1({"root": "rows"}, prefix, page_size, token), CONTEXT + ) + assert len(page.objects) <= page_size + found.extend(page.objects) + token = page.next_page_token + if token is None: + return found + raise AssertionError("listing did not finish") + + +async def test_list_returns_every_object_under_the_prefix_once_in_a_stable_order( + tmp_path: Path, +) -> None: + provider = provider_in(tmp_path) + keys = [ + f"{LIST_NAMESPACE}/a/1.jsonl", + f"{LIST_NAMESPACE}/a/2.jsonl.gz", + f"{LIST_NAMESPACE}/a/b/3.jsonl", + f"{LIST_NAMESPACE}/a.jsonl", + f"{LIST_NAMESPACE}/b.jsonl", + f"{LIST_NAMESPACE}/z/9.jsonl", + ] + await _put_all(provider, keys) + expected = [entry.object_ref for entry in await _list_all(provider, f"{LIST_NAMESPACE}/", 100)] + assert sorted(expected) == sorted(keys) and len(set(expected)) == len(keys) + for page_size in (1, 2, 3, 5, 100): + seen = [e.object_ref for e in await _list_all(provider, f"{LIST_NAMESPACE}/", page_size)] + assert seen == expected, page_size + assert [e.object_ref for e in await _list_all(provider, f"{LIST_NAMESPACE}/a/", 2)] == [ + f"{LIST_NAMESPACE}/a/1.jsonl", + f"{LIST_NAMESPACE}/a/2.jsonl.gz", + f"{LIST_NAMESPACE}/a/b/3.jsonl", + ] + assert await _list_all(provider, f"{LIST_NAMESPACE}/zz", 5) == [] + + +async def test_list_reports_the_version_size_and_write_time_the_other_operations_use( + tmp_path: Path, +) -> None: + provider = provider_in(tmp_path) + key = f"{LIST_NAMESPACE}/acct/x.jsonl" + created = await provider.put_if_absent( + ReportingRowObjectPutInputV1({"root": "rows"}, key, b"0123456789", "application/gzip", {}), + CONTEXT, + ) + (entry,) = await _list_all(provider, f"{LIST_NAMESPACE}/", 10) + assert entry.native_version_ref == created.native_version_ref + stat = await provider.stat( + ReportingRowObjectStatInputV1({"root": "rows"}, key, created.native_version_ref), CONTEXT + ) + assert stat is not None and entry.size_bytes == stat.size_bytes == 10 + written = parse_row_instant(entry.created_at) + assert written is not None and abs((written - datetime.now(timezone.utc)).total_seconds()) < 60 + assert entry.created_at.endswith("Z") + + +async def test_list_never_returns_a_temporary_file_or_a_key_outside_the_prefix( + tmp_path: Path, +) -> None: + provider = provider_in(tmp_path) + await _put_all(provider, [f"{LIST_NAMESPACE}/ok.jsonl", "other/prefix.jsonl"]) + leftover = tmp_path / "rows" / LIST_NAMESPACE / "half-written.jsonl.123.tmp" + leftover.write_bytes(b"x") + keys = [e.object_ref for e in await _list_all(provider, f"{LIST_NAMESPACE}/", 10)] + assert keys == [f"{LIST_NAMESPACE}/ok.jsonl"] + + +async def test_list_resumes_after_the_object_a_previous_page_ended_on_was_deleted( + tmp_path: Path, +) -> None: + provider = provider_in(tmp_path) + keys = [f"{LIST_NAMESPACE}/o{index}.jsonl" for index in range(5)] + versions = await _put_all(provider, keys) + first = await provider.list( + ReportingRowObjectListInputV1({"root": "rows"}, f"{LIST_NAMESPACE}/", 2), CONTEXT + ) + assert first.next_page_token == keys[1] + for key in keys[:2]: + await provider.delete( + ReportingRowObjectDeleteInputV1({"root": "rows"}, key, versions[key]), CONTEXT + ) + rest = await _list_all_from(provider, f"{LIST_NAMESPACE}/", first.next_page_token) + assert rest == keys[2:] + + +async def _list_all_from(provider: Any, prefix: str, token: str | None) -> list[str]: + found: list[str] = [] + while True: + page = await provider.list( + ReportingRowObjectListInputV1({"root": "rows"}, prefix, 2, token), CONTEXT + ) + found.extend(entry.object_ref for entry in page.objects) + token = page.next_page_token + if token is None: + return found + + +async def test_list_refuses_a_bad_page_size_an_unknown_root_and_a_prefix_that_climbs_out( + tmp_path: Path, +) -> None: + provider = provider_in(tmp_path) + for bad in (0, -1, 1.5, "5"): + with pytest.raises(ReportingRowStoreError) as failure: + await provider.list( + ReportingRowObjectListInputV1({"root": "rows"}, "adcp-rows/", bad), CONTEXT # type: ignore[arg-type] + ) + assert failure.value.code == "INVALID_INPUT" + with pytest.raises(ReportingRowStoreError) as unknown: + await provider.list( + ReportingRowObjectListInputV1({"root": "nope"}, "adcp-rows/", 5), CONTEXT + ) + assert unknown.value.code == "UNSAFE_BINDING" + with pytest.raises(ReportingRowStoreError) as climbing: + await provider.list( + ReportingRowObjectListInputV1({"root": "rows"}, "adcp-rows/../x/", 5), CONTEXT + ) + assert climbing.value.code == "INVALID_INPUT" + + +class _BrokenLister: + """A filesystem provider whose ``list`` breaks one rule the collector relies on.""" + + name = "filesystem" + ranged_reads = True + + def __init__(self, inner: Any, defect: str) -> None: + self._inner, self._defect = inner, defect + for method in ("validate_location", "probe", "put_if_absent", "get", "stat", "delete"): + setattr(self, method, getattr(inner, method)) + + async def list(self, request: Any, context: Any) -> Any: + page = await self._inner.list(request, context) + objects = list(page.objects) + if self._defect == "duplicate" and objects: + objects.append(objects[0]) + elif self._defect == "wrong_version" and objects: + objects[0] = replace(objects[0], native_version_ref="9:9:9:9") + elif self._defect == "wrong_size" and objects: + objects[0] = replace(objects[0], size_bytes=objects[0].size_bytes + 1) + elif self._defect == "no_time" and objects: + objects[0] = replace(objects[0], created_at="2000-01-01T00:00:00.000Z") + elif self._defect == "ignores_prefix": + return await self._inner.list(replace(request, prefix=request.prefix[:-1]), context) + elif self._defect == "too_many" and objects: + objects = objects * (request.page_size + 1) + elif self._defect == "repeat_token": + return replace(page, next_page_token="same") + elif self._defect == "drops_last" and len(objects) > 1: + objects.pop() + return replace(page, objects=tuple(objects)) + + +@pytest.mark.parametrize( + "defect", + [ + "duplicate", + "wrong_version", + "wrong_size", + "no_time", + "too_many", + "repeat_token", + "drops_last", + ], +) +async def test_the_conformance_runner_rejects_a_list_that_breaks_the_collectors_contract( + tmp_path: Path, defect: str +) -> None: + provider = _BrokenLister(provider_in(tmp_path), defect) + with pytest.raises(AssertionError): + await run_reporting_row_object_provider_conformance_v1( + provider, location={"root": "rows"}, prefix="adcp-rows" # type: ignore[arg-type] + ) + + +async def test_a_provider_without_list_still_passes_conformance_and_reports_no_list_check( + tmp_path: Path, +) -> None: + inner = provider_in(tmp_path) + + class WithoutList: + name = "filesystem" + ranged_reads = True + + def __init__(self) -> None: + for method in ("validate_location", "probe", "put_if_absent", "get", "stat", "delete"): + setattr(self, method, getattr(inner, method)) + + passed = await run_reporting_row_object_provider_conformance_v1( + WithoutList(), location={"root": "rows"}, prefix="adcp-rows" # type: ignore[arg-type] + ) + assert "list" not in passed and passed[-1] == "write-unique-version" + + +def test_strict_instants_and_key_shapes() -> None: + assert parse_row_instant("2026-10-01T00:00:00Z") is not None + assert parse_row_instant("2026-10-01T00:00:00.123456789Z") is not None + for bad in ( + "2026-10-01", + "2026-10-01T00:00:00", + "2026-10-01T00:00:00+00:00", + "2026-10-01 00:00:00Z", + "2026-13-01T00:00:00Z", + "2026-02-30T00:00:00Z", + "", + None, + 7, + ): + assert parse_row_instant(bad) is None, bad + assert is_row_object_key_shape("adcp-rows/ns/acct/0001.jsonl.gz") + for bad_key in ("", "a//b", "a/./b", "a/../b", "a b", "a\n", "/a", "a/", "x" * 1025): + assert not is_row_object_key_shape(bad_key), bad_key + + +async def test_list_does_not_follow_symlinks_out_of_the_root(tmp_path: Path) -> None: + provider = provider_in(tmp_path) + await _put_all(provider, [f"{LIST_NAMESPACE}/real.jsonl"]) + outside = tmp_path / "outside" + outside.mkdir() + (outside / "secret.jsonl").write_bytes(b"x") + owned = tmp_path / "rows" / LIST_NAMESPACE + (owned / "linked-dir").symlink_to(outside, target_is_directory=True) + (owned / "linked-file.jsonl").symlink_to(outside / "secret.jsonl") + keys = [e.object_ref for e in await _list_all(provider, f"{LIST_NAMESPACE}/", 10)] + assert keys == [f"{LIST_NAMESPACE}/real.jsonl"] diff --git a/tests/test_reporting_row_storage_gcs.py b/tests/test_reporting_row_storage_gcs.py index 59ecb68f4c..05c3502e89 100644 --- a/tests/test_reporting_row_storage_gcs.py +++ b/tests/test_reporting_row_storage_gcs.py @@ -3,6 +3,7 @@ from __future__ import annotations from dataclasses import dataclass, field +from datetime import datetime, timezone from types import SimpleNamespace from typing import Any @@ -18,6 +19,7 @@ ReportingRowObjectContextV1, ReportingRowObjectDeleteInputV1, ReportingRowObjectGetInputV1, + ReportingRowObjectListInputV1, ReportingRowObjectPutInputV1, run_reporting_row_object_provider_conformance_v1, ) @@ -35,6 +37,7 @@ class _Version: metadata: dict[str, str] content_type: str cache_control: str | None + created: datetime = field(default_factory=lambda: datetime.now(timezone.utc)) @dataclass @@ -82,6 +85,39 @@ def lifecycle_rules(self) -> Any: def blob(self, key: str, generation: int | None = None) -> Any: return Blob(name, key, generation) + def list_blobs( + self, + prefix: str | None = None, + versions: bool | None = None, + max_results: int | None = None, + page_token: str | None = None, + timeout: Any = None, + ) -> Any: + storage.calls.append(("list", {"prefix": prefix, "versions": versions})) + entries = [ + (key, version) + for (bucket, key), history in sorted(storage.objects.items()) + if bucket == name and key.startswith(prefix or "") + for version in (history if versions else history[-1:]) + ] + start = int(page_token) if page_token else 0 + chunk = entries[start : start + (max_results or len(entries))] + following = start + len(chunk) + blobs = [ + SimpleNamespace( + name=key, + generation=version.generation, + size=len(version.data), + time_created=version.created, + updated=version.created, + ) + for key, version in chunk + ] + return SimpleNamespace( + pages=iter([blobs]), + next_page_token=str(following) if following < len(entries) else None, + ) + def get_blob(self, key: str, timeout: Any = None) -> Any: live = storage.objects.get((name, key)) if not live: @@ -175,6 +211,7 @@ async def test_passes_the_shared_provider_conformance_suite() -> None: "version-pinned-stat", "exact-version-delete", "write-unique-version", + "list", ] @@ -454,3 +491,113 @@ def bucket(self, name: str) -> Any: ReportingRowObjectDeleteInputV1(LOCATION, "adcp-rows/ns/x", "123"), context ) assert seen == [("reload", 4.5), ("delete", 4.5)] + + +# -- list --------------------------------------------------------------------------------------- + + +async def test_list_lists_every_generation_in_pages_and_leaves_out_what_it_cannot_address() -> None: + storage = FakeStorage().add_bucket() + provider = create_gcs_reporting_row_object_provider_v1(storage=storage) + keys = [f"adcp-rows/ns/o{index}.jsonl" for index in range(5)] + versions = {} + for key in keys: + versions[key] = (await provider.put_if_absent(put(key, b"abc"), CONTEXT)).native_version_ref + # A noncurrent generation of the first key is listed too: it is what a delete could name. + storage.objects[("rows-bucket", keys[0])].append( + type(storage.objects[("rows-bucket", keys[0])][0])( + storage.next_generation, b"later", {}, "application/gzip", None + ) + ) + found = [] + token = None + for _ in range(20): + page = await provider.list( + ReportingRowObjectListInputV1(LOCATION, "adcp-rows/ns/", 2, token), CONTEXT + ) + assert len(page.objects) <= 2 + found.extend(page.objects) + token = page.next_page_token + if token is None: + break + assert [(e.object_ref, e.size_bytes) for e in found].count((keys[0], 3)) == 1 + assert len(found) == 6 and {e.object_ref for e in found} == set(keys) + call = next(options for name, options in storage.calls if name == "list") + assert call["versions"] is True and call["prefix"] == "adcp-rows/ns/" + + +async def test_list_skips_an_entry_without_a_usable_generation_size_or_time_and_counts_it() -> None: + from types import SimpleNamespace + + storage = FakeStorage().add_bucket() + await create_gcs_reporting_row_object_provider_v1(storage=storage).put_if_absent( + put("adcp-rows/ns/good.jsonl"), CONTEXT + ) + + class Wrapped: + def bucket(self, name: str) -> Any: + real = storage.bucket(name) + + class Bucket: + def __getattr__(self, attribute: str) -> Any: + return getattr(real, attribute) + + def list_blobs(self, **options: Any) -> Any: + iterator = real.list_blobs(**options) + blobs = list(next(iter(iterator.pages))) + now = datetime.now(timezone.utc) + prefix = options["prefix"] + blobs += [ + SimpleNamespace( + name=f"{prefix}no-generation.jsonl", + generation=None, + size=1, + time_created=now, + updated=now, + ), + SimpleNamespace( + name=f"{prefix}bad-size.jsonl", + generation=5, + size=-1, + time_created=now, + updated=now, + ), + SimpleNamespace( + name=f"{prefix}no-time.jsonl", + generation=6, + size=1, + time_created=None, + updated=None, + ), + SimpleNamespace( + name="elsewhere/x.jsonl", + generation=7, + size=1, + time_created=now, + updated=now, + ), + ] + return SimpleNamespace( + pages=iter([blobs]), next_page_token=iterator.next_page_token + ) + + return Bucket() + + provider = create_gcs_reporting_row_object_provider_v1(storage=Wrapped()) + page = await provider.list( + ReportingRowObjectListInputV1(LOCATION, "adcp-rows/ns/", 50), CONTEXT + ) + assert [e.object_ref for e in page.objects] == ["adcp-rows/ns/good.jsonl"] + assert page.skipped == 4 + + +async def test_list_refuses_a_bad_page_size_and_an_oversized_prefix() -> None: + provider = create_gcs_reporting_row_object_provider_v1(storage=FakeStorage().add_bucket()) + for size in (0, -1, 1.5): + with pytest.raises(ReportingRowStoreError) as failure: + await provider.list( + ReportingRowObjectListInputV1(LOCATION, "adcp-rows/", size), CONTEXT # type: ignore[arg-type] + ) + assert failure.value.code == "INVALID_INPUT" + with pytest.raises(ReportingRowStoreError): + await provider.list(ReportingRowObjectListInputV1(LOCATION, "p" * 1025, 5), CONTEXT) diff --git a/tests/test_reporting_row_storage_s3.py b/tests/test_reporting_row_storage_s3.py index 0a0ff48abe..913ca84673 100644 --- a/tests/test_reporting_row_storage_s3.py +++ b/tests/test_reporting_row_storage_s3.py @@ -14,6 +14,7 @@ ReportingRowObjectContextV1, ReportingRowObjectDeleteInputV1, ReportingRowObjectGetInputV1, + ReportingRowObjectListInputV1, ReportingRowObjectPutInputV1, run_reporting_row_object_provider_conformance_v1, ) @@ -51,7 +52,7 @@ async def test_passes_the_shared_provider_conformance_suite(s3: Any) -> None: passed = await run_reporting_row_object_provider_conformance_v1( provider, location=LOCATION, prefix="adcp-rows" ) - assert passed[-1] == "write-unique-version" and "ranged-read" in passed + assert "write-unique-version" in passed and "ranged-read" in passed and passed[-1] == "list" async def test_create_only_writes_record_the_version_id_and_adopt_on_conflict(s3: Any) -> None: @@ -278,3 +279,91 @@ async def test_a_client_without_finite_socket_timeouts_is_refused_at_the_probe() with pytest.raises(ReportingRowStoreError) as refused: await provider.probe(LOCATION, "adcp-rows", CONTEXT) assert refused.value.code == "UNSAFE_BINDING" and "read_timeout" in str(refused.value) + + +# -- list --------------------------------------------------------------------------------------- + + +async def _list_pages(provider: Any, prefix: str, page_size: int) -> tuple[list[Any], int]: + found: list[Any] = [] + skipped = 0 + token = None + for _ in range(100): + page = await provider.list( + ReportingRowObjectListInputV1(LOCATION, prefix, page_size, token), CONTEXT + ) + found.extend(page.objects) + skipped += page.skipped + token = page.next_page_token + if token is None: + return found, skipped + raise AssertionError("listing did not finish") + + +async def test_list_returns_noncurrent_versions_and_never_a_delete_marker(s3: Any) -> None: + provider = create_s3_reporting_row_object_provider_v1(client=s3) + key = "adcp-rows/ns/versions.jsonl" + first = s3.put_object(Bucket="rows-bucket", Key=key, Body=b"one\n")["VersionId"] + second = s3.put_object(Bucket="rows-bucket", Key=key, Body=b"three\n")["VersionId"] + s3.delete_object(Bucket="rows-bucket", Key=key) # a delete marker, not an object + other = ( + await provider.put_if_absent(put("adcp-rows/ns/other.jsonl"), CONTEXT) + ).native_version_ref + for page_size in (1, 2, 10): + found, skipped = await _list_pages(provider, "adcp-rows/ns/", page_size) + assert skipped == 0 + assert sorted((e.object_ref, e.native_version_ref) for e in found) == sorted( + [(key, first), (key, second), ("adcp-rows/ns/other.jsonl", other)] + ), page_size + sizes = { + e.native_version_ref: e.size_bytes + for e in (await _list_pages(provider, "adcp-rows/", 5))[0] + } + assert sizes[first] == 4 and sizes[second] == 6 + + +async def test_list_leaves_out_a_version_written_before_versioning_and_counts_it(s3: Any) -> None: + provider = create_s3_reporting_row_object_provider_v1(client=s3) + s3.create_bucket(Bucket="young-bucket") + s3.put_object(Bucket="young-bucket", Key="adcp-rows/ns/before.jsonl", Body=b"x\n") + s3.put_bucket_versioning(Bucket="young-bucket", VersioningConfiguration={"Status": "Enabled"}) + s3.put_object(Bucket="young-bucket", Key="adcp-rows/ns/after.jsonl", Body=b"y\n") + found, skipped = await _list_in(provider, "young-bucket") + assert [e.object_ref for e in found] == ["adcp-rows/ns/after.jsonl"] + assert skipped == 1, "a null-version object is never addressable by a recorded version" + + +async def _list_in(provider: Any, bucket: str) -> tuple[list[Any], int]: + page = await provider.list( + ReportingRowObjectListInputV1({"bucket": bucket}, "adcp-rows/ns/", 10), CONTEXT + ) + return list(page.objects), page.skipped + + +async def test_list_refuses_a_bad_token_a_bad_page_size_and_a_truncation_without_markers( + s3: Any, +) -> None: + from types import SimpleNamespace + + provider = create_s3_reporting_row_object_provider_v1(client=s3) + for token in ("not base64!", "e30", "bm90LWpzb24"): + with pytest.raises(ReportingRowStoreError) as failure: + await provider.list( + ReportingRowObjectListInputV1(LOCATION, "adcp-rows/", 5, token), CONTEXT + ) + assert failure.value.code == "INVALID_INPUT", token + for size in (0, -3, 1.5): + with pytest.raises(ReportingRowStoreError): + await provider.list( + ReportingRowObjectListInputV1(LOCATION, "adcp-rows/", size), CONTEXT # type: ignore[arg-type] + ) + truncated = SimpleNamespace( + put_object=lambda **_: {}, + list_object_versions=lambda **_: {"Versions": [], "IsTruncated": True}, + ) + broken = create_s3_reporting_row_object_provider_v1(client=truncated) + with pytest.raises(ReportingRowStoreError) as partial: + await broken.list(ReportingRowObjectListInputV1(LOCATION, "adcp-rows/", 5), CONTEXT) + assert ( + partial.value.code == "PROVIDER_UNAVAILABLE" + ), "a partial listing is never reported complete"