Skip to content

fix: allow single-label hostnames in deep crawl URL validation - #2175

Closed
Battleplus wants to merge 2 commits into
unclecode:mainfrom
Battleplus:fix/2079-single-label-hostnames
Closed

fix: allow single-label hostnames in deep crawl URL validation#2175
Battleplus wants to merge 2 commits into
unclecode:mainfrom
Battleplus:fix/2079-single-label-hostnames

Conversation

@Battleplus

Copy link
Copy Markdown

What

The previous implementation rejected URLs with single-label hostnames (e.g. https://name/xyz/) by checking for a dot in netloc. This is overly strict and prevents crawling internal DNS hosts or local development servers.

Why

Fixes #2079 - BFSDeepCrawlStrategy.can_process_url() rejects valid single-label hostnames.

Changes

  • Removed the dot check in both BFS and BFF deep crawl strategies
  • Single-label hostnames are now accepted when the URL has a valid scheme and non-empty netloc

Testing

  • Verified that single-label hostnames are now accepted
  • Verified that invalid URLs (missing scheme, invalid scheme) are still rejected

Fixes #2079

The previous implementation rejected URLs with single-label hostnames
(e.g. 'https://name/xyz/') by checking for a dot in netloc. This is
overly strict and prevents crawling internal DNS hosts or local
development servers.

Fixes unclecode#2079

Signed-off-by: Battleplus <3559424769@qq.com>
Signed-off-by: Battleplus <3559424769@qq.com>
@Battleplus

Copy link
Copy Markdown
Author

Follow-up audit: tightened the relaxed validation to require a parsed hostname (rather than merely a non-empty netloc), so valid single-label/internal hosts and IP literals are accepted while malformed URLs such as http://:8080/path remain rejected. Added parameterized coverage for BFS and Best-First strategies across single-label hosts, IPv4/IPv6, missing schemes, invalid schemes, and missing hostnames. 16 passed. Commit: a515dc1.

@Battleplus

Copy link
Copy Markdown
Author

Closing as a duplicate after the full audit. Earlier PR #2143 already covers both BFS and Best-First validation and includes broader targeted coverage for single-label/Docker hosts, localhost, IP literals, malformed netlocs, invalid schemes, and FilterChain execution. I added hostname-hardening and tests here during the audit, but keeping a third PR for the same issue would only split review. Please use #2143.

@Battleplus Battleplus closed this Aug 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: BFSDeepCrawlStrategy.can_process_url() rejects valid single-label hostnames (netloc without a dot)

1 participant