Issues

• Open Web Crawler cannot extract Swiftype-style meta tags via extraction_rulesets (meta class filtering limitation)

  enhancementteam:extract-and-transform
  elastic/crawler#413 · Filter by author lcrane777 opened on Jan 28, 2026

• Get url from image tag

  community-drivencomplexity:mediumenhancementgood first issuepriority:lowsdh-driventeam:extract-and-transform
  elastic/crawler#361 · Filter by author uit2712 opened on Jul 23, 2025
  2

• Validate domains before running crawl

  enhancementteam:extract-and-transform
  elastic/crawler#345 · Filter by author navarone-feekery opened on Jul 15, 2025

• Validate command should output a warning if seed URL domains don't match main domain

  enhancementteam:extract-and-transform
  elastic/crawler#342 · Filter by author navarone-feekery opened on Jul 14, 2025

• Some styles of nested YAML fields are not parsed correctly.

  bugsdh-driventeam:extract-and-transform
  Bug

  elastic/crawler#327 · Filter by author navarone-feekery opened on Jun 24, 2025

• Add warning to log if SSL certificate is about to expire

  enhancement
  elastic/crawler#305 · Filter by author navarone-feekery opened on May 27, 2025

• crawl_rules.policy needs additional option(s)

  enhancement
  elastic/crawler#231 · Filter by author that-gp opened on Mar 12, 2025

• Support 304 and non-modified content better

  enhancement
  elastic/crawler#221 · Filter by author navarone-feekery opened on Mar 4, 2025

• Add HTML to Markdown functionality

  complexity:mediumenhancementpriority:medium
  elastic/crawler#197 · Filter by author navarone-feekery opened on Feb 4, 2025
  4

• Allow optional ingestion of sitemap data

  complexity:lowenhancementpriority:low
  elastic/crawler#181 · Filter by author navarone-feekery opened on Jan 22, 2025

• Scheduled purge of "hasn't been crawled in N days"

  complexity:highenhancementpriority:medium
  elastic/crawler#168 · Filter by author seanstory opened on Nov 23, 2024

• join_as option that only grabs the first match

  complexity:lowenhancementpriority:low
  elastic/crawler#167 · Filter by author seanstory opened on Nov 21, 2024

• Support custom HTTP headers

  complexity:mediumenhancementpriority:medium
  elastic/crawler#161 · Filter by author vidok opened on Nov 4, 2024
  4

• Expand extraction rules to deny content based on selectors

  complexity:highenhancementpriority:medium
  elastic/crawler#151 · Filter by author navarone-feekery opened on Oct 4, 2024

• HTML Content Extraction

  complexity:highenhancementpriority:medium
  elastic/crawler#144 · Filter by author that-gp opened on Sep 18, 2024
  9

• Crawl ID remains the same across scheduled crawls

  bugcomplexity:lowpriority:lowv0.2.0
  Bug

  elastic/crawler#131 · Filter by author navarone-feekery opened on Sep 4, 2024
  1

• Add option to not crawl URLs already crawled in an index

  community-drivencomplexity:highenhancementpriority:low
  elastic/crawler#105 · Filter by author jtele2 opened on Aug 22, 2024
  2

• Make purge crawl pagination lazy

  complexity:mediumenhancementpriority:low
  elastic/crawler#102 · Filter by author navarone-feekery opened on Aug 21, 2024

• Dependency Dashboard

  team:extract-and-transform
  elastic/crawler#78 · Filter by author elastic-renovate-prod opened on Aug 9, 2024

• Add tool to re-attempt failed bulk index payloads

  complexity:mediumenhancementpriority:low
  elastic/crawler#66 · Filter by author navarone-feekery opened on Jul 26, 2024
  3

• ML inference to ingested document is not working

  bugcomplexity:lowpriority:low
  elastic/crawler#56 · Filter by author mikecalizo opened on Jul 4, 2024
  2

• Add "depth" field

  community-drivencomplexity:mediumenhancementpriority:medium
  elastic/crawler#52 · Filter by author YazdanJahedi opened on Jun 18, 2024
  4

• Add a general delay between crawl requests

  complexity:lowenhancementgood first issuepriority:medium
  elastic/crawler#49 · Filter by author navarone-feekery opened on Jun 17, 2024
  1

• Separating of Heading HTML tags

  community-drivencomplexity:lowenhancementpriority:medium
  elastic/crawler#47 · Filter by author YazdanJahedi opened on Jun 13, 2024

• Abide by crawl delays found in robots.txt

  complexity:mediumenhancementpriority:low
  elastic/crawler#46 · Filter by author navarone-feekery opened on Jun 12, 2024