Zero-Fail Scrape Fleet

A small, provider-agnostic Python engine for DC-first proxy lease failover.

“Zero-Fail” is the project name, not an uptime guarantee. Networks, providers, destinations, credentials, and caller-supplied validation can all fail. The library makes failure handling explicit, ordered, cache-safe, and finite.

Why this exists

Long-running data collection often needs more than random proxy rotation. A useful fallback mechanism should:

1. try inexpensive datacenter capacity first;
2. reuse a proven sticky exit while it remains valid;
3. discard that cached exit immediately after a failed validation;
4. count only consecutive fresh failures toward rung advancement;
5. exhaust all DC rungs before entering a fallback tier; and
6. stop at explicit attempt and cycle budgets rather than retry forever.

This repository isolates that state machine from HTTP clients, proxy vendors, credentials, and destination-specific logic.

What is included

• ordered DC and FALLBACK candidate tiers;
• capability-gated sticky-token caching;
• generation-safe invalidation for concurrent callers;
• bounded fresh-failure, cycle, and total-attempt budgets;
• opaque/redacted lease and cache representations;
• caller-provided adapter and validation interfaces;
• deterministic, network-free standard-library tests.

What is deliberately not included

• an HTTP or browser client;
• proxy-provider endpoints or credential formats;
• environment-file loading;
• destination-specific checks or bypass logic;
• fingerprint impersonation or anti-detection behavior;
• persistent or cross-process cache coordination;
• claims of anonymity, universal compatibility, or uninterrupted operation.

State machine

For every cycle:

all DC rungs, in order
    cached proven token (only when sticky + reusable are declared)
    fresh leases until the consecutive-failure threshold
then all fallback rungs, in order
stop when acquired, cancelled, or a finite budget is exhausted

A failed cached lease is invalidated by its exact observed generation and does not count as a fresh failure. A successful fresh lease is cached only when the adapter declares both STICKY_LEASE and REUSABLE_TOKEN and returns a token with a future expiry.

Minimal integration

Implement three adapter methods and one validator. Configuration and secrets stay in your adapter; the fleet never inspects them.

from zero_fail_scrape_fleet import (
    Candidate,
    Capability,
    CapabilityDeclaration,
    Fleet,
    Lease,
    Policy,
    Tier,
)


class MyAdapter:
    def declare_capabilities(self, candidate):
        return CapabilityDeclaration(
            frozenset({Capability.STICKY_LEASE, Capability.REUSABLE_TOKEN}),
            version="adapter-v1",
        )

    def open_fresh(self, candidate):
        # Build provider-specific material outside this library.
        return Lease(
            material=my_fresh_client(candidate),
            reusable_token=my_reusable_token(),
            expires_at=my_monotonic_expiry(),
        )

    def open_reusable(self, candidate, reusable_token):
        return Lease(material=my_client_for_token(candidate, reusable_token))


def validator(candidate, material):
    # Use a lawful, destination-appropriate success check owned by your application.
    return my_bounded_check(material)


fleet = Fleet(
    [
        Candidate("dc-primary", Tier.DC),
        Candidate("dc-secondary", Tier.DC),
        Candidate("last-resort", Tier.FALLBACK),
    ],
    MyAdapter(),
    validator,
    policy=Policy(fresh_failures_per_candidate=3, cycles=2, max_attempts=30),
)

result = fleet.acquire("catalog-refresh")
if result.acquired:
    try:
        use_material(result.lease.material)
    finally:
        result.lease.close()

The adapter and validator must be bounded and non-blocking. Budgets are checked between callbacks; Python cannot preempt a callback that never returns.

Offline tests

No provider, destination, DNS, or network call is needed:

PYTHONDONTWRITEBYTECODE=1 PYTHONPATH=src python -m unittest discover -s tests -v

The tests use known inputs and fake adapters. They validate only the local state-machine mechanics; they do not establish real-world provider or destination performance.

Runnable offline simulation

examples/simulated_fleet.py prints a safe transition trace for three controlled scenarios:

• one DC rung fails and a second succeeds, then its sticky token is reused;
• all DC rungs fail and the fallback tier activates;
• an unhealthy fleet stops at its finite attempt budget.

Run it directly from a checkout:

PYTHONDONTWRITEBYTECODE=1 PYTHONPATH=src python examples/simulated_fleet.py

The simulation contains no HTTP client, DNS lookup, credentials, provider integration, or external destination. Its outputs are CONTROLLED known-input results.

Optional profile selection and sticky binding

ProfileSelector supports weighted selection over caller-supplied, opaque client profiles. ProfiledAdapter binds the selected profile ID to the provider's reusable token, so the same profile is restored when a sticky lease is reused. A new fresh lease selects again.

This is useful for compatibility testing, controlled configuration diversity, and keeping one coherent configuration for a session:

selector = ProfileSelector(
    [
        ClientProfile("compatibility-a", opaque_config_a, weight=3),
        ClientProfile("compatibility-b", opaque_config_b, weight=1),
    ],
    version="profiles-v1",
)
profiled_adapter = ProfiledAdapter(my_profile_aware_adapter, selector)
fleet = Fleet(candidates, profiled_adapter, validator)

Profile payloads and upstream reusable tokens are redacted from representations. Profile IDs, revisions, and weights form a non-secret selector fingerprint used in the cache namespace; changing them prevents incompatible cached reuse.

Run the offline walkthrough:

PYTHONDONTWRITEBYTECODE=1 PYTHONPATH=src python examples/profile_binding.py

The project intentionally ships no User-Agent catalog, browser impersonation preset, TLS fingerprint, header recipe, or anti-detection logic. Applications remain responsible for lawful configuration and destination compatibility.

Safety and lawful use

Use this library only for systems and data you are authorized to access. Respect applicable law, destination policies, rate limits, provider terms, privacy requirements, and costs. A proxy is not a substitute for authorization and does not guarantee anonymity.

Research status

The public implementation is newly authored from general reliability-engineering principles. No private endpoints, prompts, conversations, operational logs, credentials, or historical measurements are included. See studies/README.md for evidence labels and limitations.

License

MIT — see LICENSE.