Website Clone

website-clone captures a rendered HTTP(S) page and its transitive resources, then replays the captured browser traffic from a local archive. Replay runs in Chromium and does not fall back to the source network.

Installation

With uv:

uv sync --all-groups
uv run playwright install chromium

With pip:

python -m pip install .
python -m playwright install chromium

The supported Python versions are 3.11 and newer. Chromium is required for capture and replay.

Commands

Capture a page into an archive:

uv run website-clone capture https://example.com/ --output captures/example

Capture breadth is selected only for capture with --profile:

• rendered records natural browser traffic, the final DOM, screenshot, Blob, and data-URL evidence. It does not issue supplemental HTTP requests.
• offline adds the top-document DOM and transitive CSS-closure discovery.
• exhaustive is the default. It progressively scrolls, discovers references in live frames and open shadow roots, inspects dormant lazy and <noscript> content, and repeats closure until two settled passes converge.

Profile selection does not filter observed cross-origin, loopback, private, or link-local HTTP(S) traffic. Retaining a body still depends on collection success and the configured limits. The overall deadline and byte ceilings apply to every profile; the supplemental-reference ceiling applies only to offline and exhaustive.

For a persistent capture in PowerShell:

uv run website-clone capture `
  "https://asurascans.com/comics/surviving-as-a-genius-on-borrowed-time-b57aa235/chapter/74" `
  --profile exhaustive `
  --output ".\asurascans\chapter-74"

Replay an archive after the source site is unavailable:

uv run website-clone replay captures/example

Keep the offline archive open for interactive browsing:

uv run website-clone replay captures/example --keep-open

Inspect an archive without launching a browser:

uv run website-clone inspect captures/example

Export readable request, response, and provenance evidence:

uv run website-clone export captures/example --output exports/example

Analyze retained bodies and metadata for traceable candidates:

uv run website-clone analyze captures/example --output reports/example.json

Use --headed with capture or replay to show Chromium. Capture also supports --overwrite, --timeout, --idle, --max-total-bytes, --max-resource-bytes, --max-references, --storage-state, --headers, --recipe, --max-actions, --action-timeout, --max-scroll-steps, and --max-resources. Replay supports --timeout, --idle, --no-screenshot, and --keep-open. Export supports --overwrite; analyze supports --output, --overwrite, --max-scan-bytes, and --max-leads. Every command supports --show-secrets.

Replay accepts --max-scroll-steps (default 10,000). Recipe scrolling and exhaustive scrolling share this budget; reaching it makes replay partial. --keep-open implies headed Chromium and waits until you close its window. --timeout still limits the automated replay, but does not limit interactive viewing. Archive interception stays active while the window is open. Requests without an archived match remain blocked and are reported as replay misses; replay never falls through to the live network.

Capture defaults are a 300-second overall deadline, a 2-second quiet period, a 10 GiB total retained-payload limit, a 2 GiB per-payload limit, and 10,000 supplemental references for offline and exhaustive. The total and per-payload byte ceilings cover retained request bodies, response bodies, and inline Blob/data payloads. A zero byte or reference limit disables that ceiling; the overall deadline always applies. max_references does not apply to rendered, which performs no supplemental fetches. Action defaults are 1,000 expanded operations, 30 seconds per action, 10,000 combined recipe and exhaustive-scroll steps, and 100,000 browser resources; a zero resource limit is unlimited. These defaults favor breadth.

Exit status is 0 for a complete capture, replay, export, or analysis, 1 for persisted partial/failed capture, replay misses, or partial export/analysis, and 2 for invalid command input, invalid archives, refused outputs, or safe overwrite failures.

Archive behavior and limitations

Archives contain a schema-versioned manifest, final HTML, a capture screenshot, content-addressed response bodies, and replay evidence. New captures use schema version 3 with a strict provenance sidecar; readers accept schema v1, v2, and v3. Schema-v1/v2 archives have no provenance and remain compatible. The manifest retains exact URLs, request metadata, and response headers for fidelity. Treat archives as sensitive data. By default, console output and errors recursively redact URL credentials, query values, fragments, and credential-shaped headers. Pass --show-secrets to any command to print nested values exactly. This flag does not change archive, export, or analysis files, but it can expose credentials in terminal scrollback, redirected output, and CI logs. Archives, readable exports, and analyze --output reports contain exact sensitive evidence and must be protected accordingly.

Capture session inputs are direct JSON files. --storage-state uses Playwright's storage-state shape ({"cookies": [...], "origins": [...]}), --headers uses a string-to-string JSON object, and --recipe uses schema version 1 with explicit wait, navigate, scroll, click, and input actions. Waits use seconds or a selector; navigation uses an absolute HTTP(S) URL; scroll and click repeat counts are positive; input uses a selector and a string value. Actions are required by default and can set required: false and settle: quiet|none. No implicit submission or arbitrary script runs. Action, scroll, resource, and time ceilings produce explicit partial status.

Export and analysis use the caller's named repository/work directory. Analysis staging is a sibling of the named output, never the system temporary directory. analyze items are candidates with confidence and exact provenance, not validated vulnerabilities. Analysis validates every retained body. It scans valid UTF-8 content even when its MIME type is misleading, but does not treat recognized binary response media (images, audio/video, fonts, PDF/WASM, archives, and legacy font MIME types) as invalid text. Analysis limits and invalid UTF-8 in textual or unknown media produce a truthful partial report. The offline replay path serves only exact archived matches and never falls through to live networking. Recipe replay starts after the configured initial quiet period. If transient page timing produces a different pre-action DOM, replay records the exact mismatch and returns partial instead of normalizing or hiding it.

For schema v2, request bodies are stored byte-for-byte as content-addressed objects when retained. Each request records the exact method, URL, body size, and post_data_sha256. That SHA-256 is the body input to the stable method+URL+body request identity. post_data_object_digest is separate: it is an optional content-addressed object reference for the retained body. A missing or truncated request body is recorded as collector evidence rather than silently replaced.

Capture and inspect summaries include capture_report, or null when no report is available. The report contains:

• profile: the selected capture breadth;
• discovery: total discovery-loop pass count, progressive scroll-step count, convergence, and the unique normalized reference count;
• images: unique candidate count, fetched and embedded counts, retained byte count, recognized-valid, invalid, and unvalidated signature counts, failed count, and exact failed URLs;
• capture_report.images.missing_numbered: conservative internal gaps only when at least three observed numeric basenames share origin, directory, extension, and zero-padding. The collector does not extrapolate outside the observed range.

Recognized image signatures include PNG, JPEG, GIF, WebP, AVIF, SVG, BMP, and ICO. Unknown formats remain unvalidated. Report URLs follow the same recursive redaction rules as all other summary values, with exact output available only through --show-secrets.

Capture follows resource-bearing references from the rendered DOM, inline CSS, and captured stylesheets. It does not recursively crawl navigation links or automate authentication. Observed cross-origin, loopback, private, and link-local HTTP(S) traffic is not filtered by profile, but body retention can still fail because collection failed or a configured limit was reached.

A source request that fails in the live browser remains faithful evidence and does not alone make capture partial. Collector loss, unresolved work, missing request or response evidence, supplemental-reference failures, configured limits, and cancellation produce a persisted partial archive with structured issues. A partial archive is useful evidence but is not a completeness claim. WebSocket and service-worker behavior is inventoried as unsupported when it cannot be replayed. Browser-local blob: and data: payloads are recorded as evidence; they are not network requests.

Replay serves only exact archived method, origin, URL, and request-body matches. Unmatched HTTP(S) requests are aborted and written to replay/last-run.json. A successful replay therefore proves that no source network fallback was used. Replay blocks WebSockets and service workers. It also records and aborts archived redirects for methods other than GET or HEAD instead of risking changed raw-body redirect semantics. A static directory served by http.server is not the canonical replay surface because it cannot preserve request methods, origins, redirects, cookies, or request ordering. GET/HEAD navigation redirects also replay in secondary tabs. Redirected subresources in secondary pages do not share the primary page's CDP session; replay blocks them, records cdp-redirect-unhandled, and returns partial.

Rerunning capture refuses an existing destination unless --overwrite is provided. Overwrite is accepted only for a valid archive directory.

Development

uv lock --check
uv sync --all-groups
uv run ruff check .
uv run mypy src/website_clone
uv run pytest -q
uv run website-clone --help

The test suite uses local fixture servers and Chromium. It verifies capture, offline replay, request identity, archive containment, and visual equivalence.