Add CustomFeedWeb plugin to build feeds from HTML index pages - #151
Merged
Conversation
A site that publishes no feed but keeps a list page is the case SubscriptionLink was written for, and it makes an item of every anchor on the page: the navigation, the footer, the categories and the articles alike. There was no way to say which of them an article is. CustomFeedWeb fetches the pages a Recipe names and builds one feed per page from the article links it lists. CSS selectors say where an article is and what belongs to it: item_selector for the article node, link_selector for the permalink, and title, description and date selectors read inside that node. With no selectors at all it reads every a[href], which is where a new Recipe starts. Candidate URLs are resolved against the page they were found on, so a relative, a ../ and a scheme-relative link all become the URL a reader would follow. The fragment is removed, the query string is kept, and no canonical form is guessed. What survives that is judged in a fixed order -- HTTP or HTTPS, not the page itself, same_host, include, exclude, already seen -- and cut to fetch_items, in the order the page lists them. The plugin keeps no state. What the page lists now is what it returns, and whether an item has been published before is StorePermalink's record, which is what the documented Recipe puts after it. A page that could not be fetched is retried, logged and skipped, and the other sites still produce their feeds. Settings that cannot be carried out -- a site that is not a mapping, a missing or unfetchable url, a pattern that is not a regular expression, a selector combination that names no article -- are refused before anything is fetched, because a second attempt would fail identically. It reads HTML with nokogiri through Automatic.require_optional and fetches through Automatic::Http, giving the parser the stream so that the document's own meta charset is what decides its encoding. nokogiri stays an optional plugin dependency: the framework's runtime dependencies are unchanged, and the spec is guarded like the other specs that need it. SubscriptionLink and FeedParser.parse_html are untouched. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QFKfnkUtPRHSvNWyVrfQUn
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Introduces
CustomFeedWeb, a new plugin that fetches HTML index pages and builds RSS feeds from the article links they contain. This enables feed generation from websites that don't publish feeds but have structured list pages.Key Changes
New plugin:
plugins/custom_feed/web.rb— Fetches HTML pages and extracts article links using CSS selectors, with support for:Comprehensive test suite:
spec/plugins/custom_feed/web_spec.rb— 529 lines covering:Documentation updates — Added detailed section to
doc/PLUGINS.mdexplaining:StorePermalinkfor deduplication across runsnokogiri(optional plugin gem)Implementation Details
Nokogiri::HTMLto parse pages, reading from the stream to respect document charset@seenset within a single run to deduplicate URLs (cross-run deduplication is delegated toStorePermalink)FilterSort)nokogirithroughAutomatic.require_optionalrather than a hard dependencyhttps://claude.ai/code/session_01QFKfnkUtPRHSvNWyVrfQUn