Please report security vulnerabilities privately through GitHub's security-advisory flow rather than opening a public issue.

SyndCrawler treats outbound URL handling as a security boundary. Production defaults reject non-HTTP(S) schemes, localhost, loopback, link-local, private, reserved, and otherwise non-global IP destinations. Hostnames are resolved and returned addresses are checked before crawl work is accepted.

Private-network crawling is available only through an explicit opt-in intended for local development and controlled intranet use. Do not enable it on an internet-facing multi-tenant crawler unless access to those networks is intentional.

Safe raw/discovery HTTP fetching disables implicit redirect traversal and revalidates each redirect destination before the next request. Conditional validators are origin-scoped: If-None-Match and If-Modified-Since are removed when a redirect crosses origins.

The current Python HTTP engines resolve hostnames themselves when opening network connections. SyndCrawler validates DNS results before handing a URL to those engines, but it does not yet pin the validated IP address through the subsequent TLS/HTTP connection. A hostile hostname that deliberately changes DNS answers between validation and connection therefore represents a residual DNS-rebinding/TOCTOU risk.

For an internet-facing service that accepts arbitrary untrusted crawl targets, application-level checks should be combined with network-layer egress controls that block private, link-local, loopback, cloud-metadata, and other sensitive destination ranges from crawler containers. Transport-level validated-IP pinning is a post-v0.1 hardening item.

Crawlee is pinned to the security-fixed 1.9+ line. Transport-level validation and SyndCrawler's own egress checks are intentionally defense in depth rather than substitutes for one another.

Browser rendering is an extraction tier, not an anti-bot or access-control bypass mechanism. Chromium sandboxing is enabled by default and disabling it requires explicit configuration.

Browser final navigation URLs are checked through the central egress policy. Full subresource-level browser egress enforcement is not yet implemented. Consequently, the current browser worker should not be treated as a fully isolated hostile multi-tenant browser sandbox. Operators accepting arbitrary untrusted crawl targets should isolate browser workers at the network/container layer and keep access to internal/cloud-metadata networks blocked.

The FastAPI control plane requires bearer authentication by default. The supplied Docker Compose profile:

• requires an API bearer token and PostgreSQL password,
• binds the API to loopback by default,
• does not publish Redis or PostgreSQL ports,
• runs the crawler application image as a non-root user,
• persists PostgreSQL and Redis data in named volumes.

Terminate TLS at a trusted reverse proxy before exposing the API beyond localhost. Use long random secrets and do not commit .env files containing credentials.

Redis owns transient distributed frontier/leasing state. PostgreSQL owns durable crawl metadata, results, change history, lifecycle, and distributed recrawl claims. Recrawl claims use expiring tokens so a dead worker cannot permanently strand scheduled work.

Cancellation changes the durable lifecycle before Redis cleanup. If workers already hold leases, their frontier namespace is retained so those leases can ACK safely; the namespace is purged when leases drain and is also assigned a bounded TTL as crash cleanup.

This project is for normal web crawling, indexing, archival, research, monitoring, and structured-data extraction. It does not include CAPTCHA solving, anti-bot challenge bypass, credential abuse, access-control circumvention, persistence on third-party systems, stealth/fingerprint evasion, or automatic expansion into targets outside configured crawl scope.