Dogfood scraping against historical corpus and real sources

[sfloess]

Objective

Validate FlossWare/scraping end-to-end using the historical scraping corpus plus a small set of real web sources, without prematurely expanding the feature set.

Existing corpus

Historical acquisition data lives on aio-01 under:

/mnt/aio-01/claude-orchestrator/scraped-data/raw/{category}/{md5(url)}.json

The historical corpus is the compatibility reference. Do not mutate it in place.

Dogfood phases

1. Inventory the historical corpus: categories, record counts, schema variants, URL/hash relationships, media types, sizes, timestamps, and malformed/outlier records.
2. Define compatibility invariants from the actual corpus rather than assuming a new schema.
3. Replay representative historical URLs/resources through the new acquisition boundary where safely possible.
4. Run real-source acquisition using explicit URI, URI catalog, sitemap, and bounded link discovery paths.
5. Validate local corpus output: raw preservation, SHA-256 content identity, manifest provenance, deduplication, deterministic URI normalization, and failure accounting.
6. Produce a machine-readable dogfood report suitable for CI/local regression runs.
7. Only after acquisition is proven, hand the resulting normalized content to downstream extraction/chunking work.

Important boundary

The current repository intentionally stops acquisition before parsing, normalization, chunking, embeddings, indexing, and retrieval. Dogfooding should validate that boundary first; it should not turn this repository into the whole knowledge pipeline.

Current implementation reference

main currently exposes scrape with HTTP(S), FTP, filesystem, URI catalogs, XML sitemaps, and bounded same-host/domain discovery. The local corpus uses SHA-256 content identity and JSONL provenance manifests.

First deliverable

Add a dogfood/validation path that can point at a historical corpus root and an output corpus, report compatibility/invariant failures without modifying the source corpus, and exercise representative live sources separately from the legacy replay run.

Acceptance criteria

• Historical source corpus remains untouched.
• Legacy schema is measured and documented from real data.
• New corpus artifacts satisfy the current acquisition contract.
• Duplicate bytes are content-addressed once while URI provenance remains distinct.
• Live-source run is bounded and respects robots/rate/size/network safety defaults.
• A single command produces a useful pass/fail summary and machine-readable report.
• No chunking/embedding implementation is introduced as part of this acquisition dogfood milestone.