Skip to content

Move the grid download loop into an importable scrape package - #20

Merged
daniel-widrick merged 2 commits into
feature/lineup-endpointfrom
feature/scrape-package
Sep 13, 2026
Merged

daniel-widrick merged 2 commits into
feature/lineup-endpointfrom
feature/scrape-package

Conversation

@daniel-widrick

Copy link
Copy Markdown
Owner

PR 4 of #15. Stacked on #19.

Changes

  • New scrape package with Fetch(ctx, prefs, opts) (*guide.TVGuide, error). It downloads every six-hour slot for a lineup and assembles the guide: Channels deduplicated by station in first-seen order, Lineup with every position, Programs deduplicated by station and time, and Source. No enrichment, no file I/O, no env reads.
  • Options covers days, inter-slot delay, an injectable GridFetcher (which *web.Client satisfies), a clock, a per-slot Progress callback, and a logger. The zero value is production configuration.
  • Context cancellation is honored before each slot, inside the request, and during the inter-slot delay.
  • main.runScrape calls scrape.Fetch, maps progress onto the setup status page, and cancels the context when the active lineup changes mid-download (polled once per second, with a live re-check after the fetch).
  • README gains a "Using as a Library" section; CLAUDE.md and the project structure are updated.

Behavior changes to be aware of

  • When every grid slot fails, the scrape now returns scrape.ErrNoData instead of writing an empty guide. The previous guide stays live and the existing 15-minute retry applies.
  • XMLTV channel order is now first-seen order instead of Go map iteration order, so it is stable between runs.

Tests

  • scrape: slot alignment and count; full assembly with a station at two numbers; distinct events across slots kept while repeats are deduplicated; a failed slot skipped and reported with progress counts monotonic; ErrNoData when all fail; cancellation during the delay, during a request, and before the first request; option defaults. Runs clean under -race.
  • main: runScrape end to end with a fake fetcher (lineup assembled, proxy rewrite applied, files written, progress stages reported); a lineup change is reported as errScrapeSourceChanged and the persister never runs; total failure surfaces ErrNoData and the persister never runs.

scrape.Fetch(ctx, prefs, opts) downloads every six-hour slot for a
lineup and assembles a guide.TVGuide: stations deduplicated in
first-seen order, every lineup position retained, programs deduplicated.
It does no enrichment, file I/O, or environment reads.

main.runScrape now calls it, mapping progress to the setup status and
cancelling the context when the active lineup changes mid-download.

Behavior change: when every slot fails, the scrape is now an error
(scrape.ErrNoData) instead of writing an empty guide, so the previous
guide stays live and the 15-minute retry applies. Channel order in the
XMLTV output is now first-seen rather than map order.
@daniel-widrick
daniel-widrick merged commit 528f9df into feature/lineup-endpoint Sep 13, 2026
1 check passed
@daniel-widrick
daniel-widrick deleted the feature/scrape-package branch September 13, 2026 04:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant