You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Stop noise matching from eating real article content - #192
Closes#186. The noise regex matched substrings inside class tokens, so names like gradient, download-guide or headline were treated as ads and real article text silently disappeared. Instead of anchoring the pattern to the whole attribute value (which silently stops matching multi-class elements such as 'content ad-banner'), the improved fix matches per word: class/id values are split on whitespace and then on '-'/'_' and only whole words are compared against the noise word set. That keeps compound noise names (ad-banner, social-share, comment-list, sidebar-widget, author-bio) and longer noise names (navigation, advert, newsletter, breadcrumb, cookie-banner) being removed, including when they appear on multi-class elements, while leaving look-alike content classes (gradient, download-guide, headline) untouched. Removed elements are guarded so a nested element already decomposed by its parent is not decomposed twice. Added regression tests covering preserved classes, still-removed noise, multi-class elements, longer noise names and noise ids.
Two soundness gaps in the whole-word noise matching, both reported on the
pull request review:
- the expanded word set contained neutral layout/content labels (profile,
gallery, review, accordion, search, ...), so a single such class deleted a
whole subtree of real text; the set now holds only names that are
boilerplate on their own,
- "_" was normalised to "-" only after the exact-word lookup, so
"author_bio" survived while "author-bio" was removed; the normalisation
now happens before the lookup and both separators behave the same way.
Regression tests cover book-review, profile, gallery, accordion-content and
search-results being preserved, and underscore class/id values being removed.
Read _decomposed directly to avoid quadratic descendant searches
src/pyscrappy/generic/extractors.py:159
On intact BeautifulSoup tags, _decomposed is absent, so getattr falls through to Tag.__getattr__, which searches descendants for that tag name. Doing this for every element can make noise removal quadratic on deeply nested pages. Read the instance dictionary directly in both decomposition guards to avoid these subtree searches.
Two gaps in the noise word set, both reported on the pull request review:
- the bare words "cookie"/"cookies" deleted content that merely mentions
cookies ("post tag-cookies", "cookie-recipe"); the cookie interface block
is named explicitly instead ("cookie-banner", "cookie-consent"), so the
bare word is gone from the set,
- the word "breadcrumb" was promised by this pull request but missing from
the set, so breadcrumb blocks outside already-removed containers stayed in
the extracted text; the exact word is now listed.
Explicit compound names match as a whole token only: prefix matching is what
used to delete real text, so a wrapper such as "cookie-banner-wrapper" is
deliberately left alone.
Regression tests cover cookie content labels being preserved, the explicit
cookie interface names still being removed, and breadcrumb blocks being
removed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #186. The noise regex matched substrings inside class tokens, so names like gradient, download-guide or headline were treated as ads and real article text silently disappeared. Instead of anchoring the pattern to the whole attribute value (which silently stops matching multi-class elements such as 'content ad-banner'), the improved fix matches per word: class/id values are split on whitespace and then on '-'/'_' and only whole words are compared against the noise word set. That keeps compound noise names (ad-banner, social-share, comment-list, sidebar-widget, author-bio) and longer noise names (navigation, advert, newsletter, breadcrumb, cookie-banner) being removed, including when they appear on multi-class elements, while leaving look-alike content classes (gradient, download-guide, headline) untouched. Removed elements are guarded so a nested element already decomposed by its parent is not decomposed twice. Added regression tests covering preserved classes, still-removed noise, multi-class elements, longer noise names and noise ids.