fix: allow single-label hostnames in deep crawl URL validation - #2175
fix: allow single-label hostnames in deep crawl URL validation#2175Battleplus wants to merge 2 commits into
Conversation
The previous implementation rejected URLs with single-label hostnames (e.g. 'https://name/xyz/') by checking for a dot in netloc. This is overly strict and prevents crawling internal DNS hosts or local development servers. Fixes unclecode#2079 Signed-off-by: Battleplus <3559424769@qq.com>
Signed-off-by: Battleplus <3559424769@qq.com>
|
Follow-up audit: tightened the relaxed validation to require a parsed hostname (rather than merely a non-empty netloc), so valid single-label/internal hosts and IP literals are accepted while malformed URLs such as http://:8080/path remain rejected. Added parameterized coverage for BFS and Best-First strategies across single-label hosts, IPv4/IPv6, missing schemes, invalid schemes, and missing hostnames. 16 passed. Commit: a515dc1. |
|
Closing as a duplicate after the full audit. Earlier PR #2143 already covers both BFS and Best-First validation and includes broader targeted coverage for single-label/Docker hosts, localhost, IP literals, malformed netlocs, invalid schemes, and FilterChain execution. I added hostname-hardening and tests here during the audit, but keeping a third PR for the same issue would only split review. Please use #2143. |
What
The previous implementation rejected URLs with single-label hostnames (e.g.
https://name/xyz/) by checking for a dot in netloc. This is overly strict and prevents crawling internal DNS hosts or local development servers.Why
Fixes #2079 - BFSDeepCrawlStrategy.can_process_url() rejects valid single-label hostnames.
Changes
Testing
Fixes #2079