fix: remove_overlay_elements can remove body if it has a global popup…

[Nalhin]

Summary

Fixing: #2161 (created by me).

Issue summary:

remove_overlay_elements can remove body (leaving empty page) when class name contains "popup".

This can happen for some popular wp themes e.g. https://themeforest.net/item/bridge-creative-multipurpose-wordpress-theme/7315054 with class .qode_popup_menu_push_text_right (more details here: https://www.google.com/search?q=qode_popup_menu_push_text_right).

List of files changed and why

crawl4ai/js_snippet/remove_overlay_elements.js - preventing html/body/head from being removed.
tests/test_issue_2161_overlay_html_body.py - tests targeting the specific issue

How Has This Been Tested?

Added unit tests. Tested on examples in the issue.

Checklist:

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added/updated unit tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes

[SohamKukreti]

Thanks for finding the issue and raising the PR! Tested this locally against develop (37ee60a). Fix works as intended, adding a guard to ensure elements with html, head or body tag are not removed. However second repro URL from #2161 is still broken

https://charlesproper.com/avilene/ is still emptied after this patch:

• remove_overlay_elements=False → markdown 6,769 chars
• remove_overlay_elements=True + patch → markdown 14 chars

Dumped the live DOM to find the culprit - it's a different element:

{ "tag": "DIV", "cls": "wrapper", "pos": "relative", "z": "1000", "textlen": 1962 }

The entire page sits inside <div class="wrapper"> with z-index: 1000. The zIndex > 999 heuristic matches it, and since it isn't html/head/body the new guard correctly doesn't apply.

Two ways to go, your call:

1. Keep [Bug]: remove_overlay_elements can remove body if it has a global popup class #2161 open, and do the wrapper case in a follow-up PR. This change is self-contained and already fixes one of the two URLs, so there's real value in landing it now.
2. Address it here, if you'd rather close the issue in one go. You can address the other URL in this PR and close it.

[Nalhin]

Hey, @SohamKukreti these look a bit like distinct issues to me. I think it makes sense to land this PR and address the z-index 1000 + position: relative issue in a follow up PR. WDYT?

[SohamKukreti]

Hey @Nalhin, sure, that makes sense. Let’s go with a follow up PR for this. Thanks for taking the time to work on it!