From 5b579149d16b8ea5c73e1a55046c373e9273deea Mon Sep 17 00:00:00 2001 From: Daniel Widrick Date: Sat, 12 Sep 2026 23:37:57 -0400 Subject: [PATCH 1/2] Move the grid download loop into an importable scrape package scrape.Fetch(ctx, prefs, opts) downloads every six-hour slot for a lineup and assembles a guide.TVGuide: stations deduplicated in first-seen order, every lineup position retained, programs deduplicated. It does no enrichment, file I/O, or environment reads. main.runScrape now calls it, mapping progress to the setup status and cancelling the context when the active lineup changes mid-download. Behavior change: when every slot fails, the scrape is now an error (scrape.ErrNoData) instead of writing an empty guide, so the previous guide stays live and the 15-minute retry applies. Channel order in the XMLTV output is now first-seen rather than map order. --- CLAUDE.md | 4 +- README.md | 28 ++++- main.go | 128 +++++++++++--------- run_scrape_test.go | 139 ++++++++++++++++++++++ scrape/scrape.go | 188 +++++++++++++++++++++++++++++ scrape/scrape_test.go | 271 ++++++++++++++++++++++++++++++++++++++++++ 6 files changed, 696 insertions(+), 62 deletions(-) create mode 100644 run_scrape_test.go create mode 100644 scrape/scrape.go create mode 100644 scrape/scrape_test.go diff --git a/CLAUDE.md b/CLAUDE.md index 1a9735e..0198e9a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -27,12 +27,12 @@ Run `go test ./...` for the setup, configuration, and provider-client tests. ## Architecture -The binary is a single Go process that scrapes GraceNote/TMS for 14 days of TV listings and serves the data as XMLTV over HTTP. Runtime orchestration lives in `main.go`; setup handlers live in `setup.go` and persisted configuration lives in `appconfig/`. +The binary is a single Go process that scrapes GraceNote/TMS for 14 days of TV listings and serves the data as XMLTV over HTTP. The grid download loop and guide assembly live in `scrape/` (`scrape.Fetch`), which is importable by other programs and does no enrichment, file I/O, or env reads. Enrichment, persistence, and runtime orchestration live in `main.go`; setup handlers live in `setup.go` and persisted configuration lives in `appconfig/`. **Data flow:** 1. `/setup` uses `web.ProviderClient` to discover Gracenote lineups by country and postal code. `appconfig.Store` persists the selected non-secret source in `config.json`; complete legacy `GN_*` settings can bootstrap it. -2. `web.Client.GetDataByTime` fetches 6-hour grid slices from the GraceNote API (`tvlistings.gracenote.com/api/grid`) — 56 slots for 14 days. A 5-second sleep separates requests. Raw JSON types live in `web/web.go`. +2. `scrape.Fetch` drives `web.Client.GetDataByTimeContext` over 6-hour grid slices from the GraceNote API (`tvlistings.gracenote.com/api/grid`) — 56 slots for 14 days. A 5-second sleep separates requests. Failed slots are skipped; if every slot fails the scrape returns `scrape.ErrNoData` and the previous guide stays live. Raw JSON types live in `web/web.go`. 3. `guide.ConvertChannel` / `guide.ConvertEvent` translate the raw JSON into `guide.TVGuide` (internal canonical types). `TVGuide.Channels` is deduplicated by station for XMLTV; `TVGuide.Lineup` retains every provider position (number plus station) and is served by `/api/lineup.json`. `TVGuide.Source` records the lineup the guide came from. The `guide.tmpl` template renders these to XMLTV. `index.html`, `setup.html`, and `guide.tmpl` are embedded at build time via `//go:embed`. 4. `tmdb.Client.Lookup` enriches programs (poster images, ratings, overview, year) via TMDB search API. Deduplicates by `(title, isMovie)` before hitting the API. Rate-limited to ~4 req/sec. 5. `tvlogo.Client.Resolve` replaces Gracenote channel icons with verified PNGs from `github.com/tv-logo/tv-logos`. Generates candidate URL slugs from callsign/affiliate name and HEAD-checks each (rate-limited to ~5 req/sec). diff --git a/README.md b/README.md index cb7664c..7242a99 100644 --- a/README.md +++ b/README.md @@ -167,11 +167,37 @@ The server includes a built-in retro-styled TV guide web UI at the root URL. If ![The Grid](https://gist.githubusercontent.com/daniel-widrick/2c52c4d023ffe75d163b4eff58263c77/raw/demo.gif) +## Using as a Library + +The grid download and conversion live in the `scrape` package, so another Go program can fetch a lineup's listings without running the server: + +```go +import ( + "context" + "fmt" + + "github.com/daniel-widrick/GraceNoteScraper/scrape" + "github.com/daniel-widrick/GraceNoteScraper/web" +) + +func main() { + prefs := web.Preferences{Country: "USA", ZipCode: "13490", Headend: "lineupId", LineupId: "USA-lineupId-DEFAULT", Device: "-", Language: "en-us"} + g, err := scrape.Fetch(context.Background(), prefs, scrape.Options{Days: 1}) + if err != nil { + panic(err) + } + fmt.Printf("%d stations, %d lineup positions, %d programs\n", len(g.Channels), len(g.Lineup), len(g.Programs)) +} +``` + +`Fetch` returns a `guide.TVGuide`: `Channels` deduplicated by station, `Lineup` with every position, `Programs`, and `Source`. It pauses five seconds between grid requests by default and skips slots that fail; it returns `scrape.ErrNoData` only when every slot failed. Logo and TMDB enrichment are not part of the package. Lineup discovery by postal code is available through `web.NewProviderClient().FindProviders`. + ## Project Structure ``` appconfig/ Persisted non-secret provider configuration -main.go Entry point, HTTP server, scraper, image proxy +main.go Entry point, HTTP server, enrichment, image proxy +scrape/ Grid download loop and guide assembly (importable) guide/ GraceNote data types and XMLTV conversion web/ HTTP client for GraceNote API tmdb/ TMDB client and cache diff --git a/main.go b/main.go index 0197102..d0643ad 100644 --- a/main.go +++ b/main.go @@ -21,12 +21,14 @@ import ( "sort" "strings" "sync" + "sync/atomic" "syscall" "text/template" "time" "github.com/daniel-widrick/GraceNoteScraper/appconfig" "github.com/daniel-widrick/GraceNoteScraper/guide" + "github.com/daniel-widrick/GraceNoteScraper/scrape" "github.com/daniel-widrick/GraceNoteScraper/tmdb" "github.com/daniel-widrick/GraceNoteScraper/tvlogo" "github.com/daniel-widrick/GraceNoteScraper/util" @@ -276,70 +278,32 @@ func runScrape(pref web.Preferences, tmdbClient *tmdb.Client, baseURL string, ch } } } - client := web.NewClient(pref) - now := time.Now().UTC() - midnight := time.Date(now.Year(), now.Month(), now.Day(), 0, 0, 0, 0, time.UTC) - endTime := midnight.Add(14 * 24 * time.Hour) - - channelMap := make(map[string]guide.Channel) - lineupMap := make(map[string]guide.LineupPosition) - eventMap := make(map[string]bool) - var programs []guide.Program - - totalSlots := int(endTime.Sub(midnight) / (6 * time.Hour)) - slot := 0 - for t := midnight; t.Before(endTime); t = t.Add(6 * time.Hour) { - if sourceCurrent != nil && !sourceCurrent() { - return nil, errScrapeSourceChanged - } - slot++ - ts := t.Unix() - report(scrapeProgressUpdate{Stage: "gracenote", Message: fmt.Sprintf("Downloading guide data (%d of %d)", slot, totalSlots), Completed: slot - 1, Total: totalSlots, Channels: len(channelMap), Programs: len(programs)}) - log.Printf("Fetching grid %d/%d for time=%d (%s)", slot, totalSlots, ts, t.Format(time.RFC3339)) - - grid, err := client.GetDataByTime(ts) - if err != nil { - log.Printf("Error fetching grid at %d: %v", ts, err) - continue - } - - for _, ch := range grid.Channels { - if _, exists := channelMap[ch.ChannelID]; !exists { - channelMap[ch.ChannelID] = guide.ConvertChannel(ch) - } - position := guide.ConvertLineupPosition(ch) - if _, exists := lineupMap[position.Key()]; !exists { - lineupMap[position.Key()] = position - } - - for _, ev := range ch.Events { - dedupKey := ch.ChannelID + "|" + ev.StartTime + "|" + ev.EndTime - if eventMap[dedupKey] { - continue - } - eventMap[dedupKey] = true - programs = append(programs, guide.ConvertEvent(ev, ch.ChannelID, pref.Language, pref.Country)) - } - } - - log.Printf("Channels so far: %d, Events so far: %d", len(channelMap), len(programs)) - report(scrapeProgressUpdate{Stage: "gracenote", Message: fmt.Sprintf("Downloaded guide data (%d of %d)", slot, totalSlots), Completed: slot, Total: totalSlots, Channels: len(channelMap), Programs: len(programs)}) + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + sourceChanged := watchSourceChange(ctx, cancel, sourceCurrent) - if t.Add(6 * time.Hour).Before(endTime) { - time.Sleep(5 * time.Second) + opts := newScrapeOptions(pref) + opts.Progress = func(p scrape.Progress) { + message := fmt.Sprintf("Downloading guide data (%d of %d)", p.Slot, p.TotalSlots) + if p.Phase == scrape.PhaseFetched { + message = fmt.Sprintf("Downloaded guide data (%d of %d)", p.Slot, p.TotalSlots) } + report(scrapeProgressUpdate{Stage: "gracenote", Message: message, Completed: p.Completed, Total: p.TotalSlots, Channels: p.Channels, Programs: p.Programs}) } - - var channels []guide.Channel - for _, ch := range channelMap { - channels = append(channels, ch) + fetched, err := scrape.Fetch(ctx, pref, opts) + if err != nil { + if sourceChanged() { + return nil, errScrapeSourceChanged + } + return nil, err } - lineup := make([]guide.LineupPosition, 0, len(lineupMap)) - for _, position := range lineupMap { - lineup = append(lineup, position) + if sourceChanged() { + return nil, errScrapeSourceChanged } - guide.SortLineup(lineup) + channels := fetched.Channels + lineup := fetched.Lineup + programs := fetched.Programs logoClient := tvlogo.NewClient(pref.Country, "tvlogo_cache.json") if logoClient != nil { @@ -363,7 +327,7 @@ func runScrape(pref web.Preferences, tmdbClient *tmdb.Client, baseURL string, ch Channels: channels, Programs: programs, Lineup: lineup, - Source: guide.SourceFromPreferences(pref, time.Now().UTC()), + Source: fetched.Source, } if channelFilter != nil { @@ -399,6 +363,52 @@ func renderXMLTV(w io.Writer, tvGuide *guide.TVGuide) error { return nil } +// newScrapeOptions builds the production scrape configuration. Tests replace +// it to inject a fake grid fetcher and remove the inter-slot delay. +var newScrapeOptions = func(pref web.Preferences) scrape.Options { + return scrape.Options{Fetcher: web.NewClient(pref)} +} + +// watchSourceChange cancels ctx as soon as sourceCurrent reports false, so a +// lineup change during the grid download stops the scrape promptly. The +// returned func reports whether a change has been observed, re-checking live +// so a change between the last tick and the call is not missed. +func watchSourceChange(ctx context.Context, cancel context.CancelFunc, sourceCurrent func() bool) func() bool { + if sourceCurrent == nil { + return func() bool { return false } + } + var changed atomic.Bool + check := func() bool { + if changed.Load() { + return true + } + if !sourceCurrent() { + changed.Store(true) + cancel() + return true + } + return false + } + if check() { + return check + } + go func() { + ticker := time.NewTicker(time.Second) + defer ticker.Stop() + for { + select { + case <-ctx.Done(): + return + case <-ticker.C: + if check() { + return + } + } + } + }() + return check +} + func persistGuideFiles(tvGuide *guide.TVGuide, sourceFingerprint string) error { log.Printf("Rendering XMLTV: %d channels, %d programs", len(tvGuide.Channels), len(tvGuide.Programs)) diff --git a/run_scrape_test.go b/run_scrape_test.go new file mode 100644 index 0000000..75c1a8e --- /dev/null +++ b/run_scrape_test.go @@ -0,0 +1,139 @@ +package main + +import ( + "context" + "errors" + "io" + "log" + "os" + "sync/atomic" + "testing" + + "github.com/daniel-widrick/GraceNoteScraper/guide" + "github.com/daniel-widrick/GraceNoteScraper/scrape" + "github.com/daniel-widrick/GraceNoteScraper/web" +) + +type countingFetcher struct { + calls atomic.Int32 + grid *web.GridResponse + err error +} + +func (f *countingFetcher) GetDataByTimeContext(ctx context.Context, t int64) (*web.GridResponse, error) { + f.calls.Add(1) + if err := ctx.Err(); err != nil { + return nil, err + } + if f.err != nil { + return nil, f.err + } + return f.grid, nil +} + +// useFakeScrape swaps the production scrape options for a fake fetcher with +// one day of slots and no delay. The preferences use a country tvlogo does not +// support so no logo lookups reach the network. +func useFakeScrape(t *testing.T, f *countingFetcher) web.Preferences { + t.Helper() + t.Chdir(t.TempDir()) + original := newScrapeOptions + newScrapeOptions = func(web.Preferences) scrape.Options { + return scrape.Options{Fetcher: f, Days: 1, SlotDelay: scrape.NoDelay, Logger: log.New(io.Discard, "", 0)} + } + t.Cleanup(func() { newScrapeOptions = original }) + return web.Preferences{Country: "ZZZ", ZipCode: "00000", Headend: "h", LineupId: "L", Device: "-", Language: "en-us"} +} + +func TestRunScrapeBuildsLineupAndPersists(t *testing.T) { + f := &countingFetcher{grid: &web.GridResponse{Channels: []web.JSONChannel{ + {ChannelID: "s1", ID: "s10", ChannelNo: "2.1", CallSign: "AAA", AffiliateName: "A Net", StationFilters: []string{"filter-news"}, + Events: []web.JSONEvent{{StartTime: "2026-09-13T00:00:00Z", EndTime: "2026-09-13T01:00:00Z", Duration: "60", SeriesID: "SH1", Program: web.JSONProgram{ID: "EP1", Title: "one"}}}}, + {ChannelID: "s1", ID: "s199", ChannelNo: "1002", CallSign: "AAA", AffiliateName: "A Net"}, + }}} + pref := useFakeScrape(t, f) + + var persisted *guide.TVGuide + persister := func(g *guide.TVGuide) (bool, error) { + persisted = g + return true, persistGuideFiles(g, "fp") + } + var updates []scrapeProgressUpdate + got, err := runScrape(pref, nil, "http://base:8080", nil, "fp", func() bool { return true }, persister, func(u scrapeProgressUpdate) { updates = append(updates, u) }) + if err != nil { + t.Fatalf("runScrape: %v", err) + } + if f.calls.Load() != 4 { + t.Errorf("grid fetches = %d, want 4 for one day", f.calls.Load()) + } + if persisted != got { + t.Error("persister should receive the returned guide") + } + if len(got.Channels) != 1 || len(got.Lineup) != 2 || len(got.Programs) != 1 { + t.Fatalf("channels/lineup/programs = %d/%d/%d", len(got.Channels), len(got.Lineup), len(got.Programs)) + } + if got.Lineup[0].LogoURL != "" { + t.Errorf("no logo source available, got %q", got.Lineup[0].LogoURL) + } + if got.Source.LineupID != "L" { + t.Errorf("source = %+v", got.Source) + } + if _, err := os.Stat("xmlguide.xmltv"); err != nil { + t.Errorf("xmlguide.xmltv not written: %v", err) + } + stages := map[string]bool{} + for _, u := range updates { + stages[u.Stage] = true + } + for _, want := range []string{"gracenote", "logos", "saving"} { + if !stages[want] { + t.Errorf("missing progress stage %q (got %v)", want, stages) + } + } + if stages["tmdb"] { + t.Error("no TMDB client configured, so no tmdb stage should be reported") + } + var gracenote []scrapeProgressUpdate + for _, u := range updates { + if u.Stage == "gracenote" { + gracenote = append(gracenote, u) + } + } + if len(gracenote) != 8 || gracenote[0].Total != 4 || gracenote[7].Completed != 4 { + t.Errorf("gracenote progress = %d updates, first %+v, last %+v", len(gracenote), gracenote[0], gracenote[len(gracenote)-1]) + } +} + +func TestRunScrapeReportsSourceChange(t *testing.T) { + f := &countingFetcher{grid: &web.GridResponse{Channels: []web.JSONChannel{{ChannelID: "s1", ChannelNo: "2"}}}} + pref := useFakeScrape(t, f) + + var seen atomic.Int32 + sourceCurrent := func() bool { + // Current for the initial check, changed afterwards. + return seen.Add(1) == 1 + } + persisterCalled := false + _, err := runScrape(pref, nil, "", nil, "fp", sourceCurrent, func(*guide.TVGuide) (bool, error) { + persisterCalled = true + return true, nil + }) + if !errors.Is(err, errScrapeSourceChanged) { + t.Fatalf("err = %v, want errScrapeSourceChanged", err) + } + if persisterCalled { + t.Error("persister must not run after a source change") + } +} + +func TestRunScrapeSurfacesTotalFailure(t *testing.T) { + f := &countingFetcher{err: errors.New("gracenote down")} + pref := useFakeScrape(t, f) + _, err := runScrape(pref, nil, "", nil, "fp", nil, func(*guide.TVGuide) (bool, error) { + t.Fatal("persister must not run with no data") + return false, nil + }) + if !errors.Is(err, scrape.ErrNoData) { + t.Fatalf("err = %v, want ErrNoData", err) + } +} diff --git a/scrape/scrape.go b/scrape/scrape.go new file mode 100644 index 0000000..0ee481b --- /dev/null +++ b/scrape/scrape.go @@ -0,0 +1,188 @@ +// Package scrape fetches a Gracenote lineup's listings and converts them into +// a guide.TVGuide. It performs no enrichment, file I/O, or environment reads, +// so it can be imported by other programs. +package scrape + +import ( + "context" + "errors" + "fmt" + "log" + "time" + + "github.com/daniel-widrick/GraceNoteScraper/guide" + "github.com/daniel-widrick/GraceNoteScraper/web" +) + +const ( + // SlotDuration is the window Gracenote serves per grid request. + SlotDuration = 6 * time.Hour + // DefaultDays is how far ahead a scrape reaches when Options.Days is zero. + DefaultDays = 14 + // DefaultSlotDelay is the pause between grid requests when + // Options.SlotDelay is zero. Gracenote is an unofficial API; be polite. + DefaultSlotDelay = 5 * time.Second +) + +// ErrNoData is returned when every grid slot failed, so there is nothing to +// build a guide from. A partial failure is not an error; failed slots are +// skipped and reported through Options.Progress. +var ErrNoData = errors.New("scrape: no grid slot returned data") + +// GridFetcher retrieves one six-hour grid. *web.Client satisfies it. +type GridFetcher interface { + GetDataByTimeContext(ctx context.Context, t int64) (*web.GridResponse, error) +} + +// Phase says whether a Progress report precedes or follows a slot fetch. +type Phase int + +const ( + // PhaseFetching is reported just before a slot is requested. + PhaseFetching Phase = iota + // PhaseFetched is reported after a slot succeeded or failed. + PhaseFetched +) + +// Progress describes one step of a scrape. +type Progress struct { + Phase Phase + Slot int // 1-based index of the slot being reported + TotalSlots int // total slots for this scrape + SlotTime time.Time // start of the six-hour window + Completed int // slots finished so far, successful or not + Channels int // distinct stations collected so far + Programs int // programs collected so far + Err error // non-nil on PhaseFetched when the slot failed and was skipped +} + +// Options tunes a Fetch. The zero value is the production configuration. +type Options struct { + Days int // days of listings to fetch; default DefaultDays + SlotDelay time.Duration // pause between slots; default DefaultSlotDelay (tests use a negative value for none) + Fetcher GridFetcher // grid source; default web.NewClient(prefs) + Now func() time.Time // clock; default time.Now + Progress func(Progress) // optional per-slot callback + Logger *log.Logger // default log.Default() +} + +// NoDelay is a SlotDelay value that disables the inter-slot pause. +const NoDelay = -1 + +func (o Options) withDefaults(prefs web.Preferences) Options { + if o.Days <= 0 { + o.Days = DefaultDays + } + if o.SlotDelay == 0 { + o.SlotDelay = DefaultSlotDelay + } + if o.Fetcher == nil { + o.Fetcher = web.NewClient(prefs) + } + if o.Now == nil { + o.Now = time.Now + } + if o.Logger == nil { + o.Logger = log.Default() + } + return o +} + +// Slots returns the six-hour window start times a Fetch will request for the +// given clock and day count, beginning at UTC midnight of the current day. +func Slots(now time.Time, days int) []time.Time { + if days <= 0 { + days = DefaultDays + } + now = now.UTC() + start := time.Date(now.Year(), now.Month(), now.Day(), 0, 0, 0, 0, time.UTC) + end := start.Add(time.Duration(days) * 24 * time.Hour) + slots := make([]time.Time, 0, days*int(24*time.Hour/SlotDuration)) + for t := start; t.Before(end); t = t.Add(SlotDuration) { + slots = append(slots, t) + } + return slots +} + +// Fetch downloads every grid slot for prefs and returns the assembled guide. +// Channels are deduplicated by station in first-seen order, Lineup keeps every +// position, and Programs are deduplicated by station, start, and end. +func Fetch(ctx context.Context, prefs web.Preferences, opts Options) (*guide.TVGuide, error) { + opts = opts.withDefaults(prefs) + slots := Slots(opts.Now(), opts.Days) + + channelIndex := make(map[string]int) + lineupIndex := make(map[string]struct{}) + eventIndex := make(map[string]struct{}) + var channels []guide.Channel + var lineup []guide.LineupPosition + var programs []guide.Program + succeeded := 0 + + report := func(p Progress) { + if opts.Progress != nil { + opts.Progress(p) + } + } + + for i, slotTime := range slots { + if err := ctx.Err(); err != nil { + return nil, err + } + slot := i + 1 + report(Progress{Phase: PhaseFetching, Slot: slot, TotalSlots: len(slots), SlotTime: slotTime, Completed: i, Channels: len(channels), Programs: len(programs)}) + opts.Logger.Printf("Fetching grid %d/%d for time=%d (%s)", slot, len(slots), slotTime.Unix(), slotTime.Format(time.RFC3339)) + + grid, err := opts.Fetcher.GetDataByTimeContext(ctx, slotTime.Unix()) + if err != nil { + if ctxErr := ctx.Err(); ctxErr != nil { + return nil, ctxErr + } + opts.Logger.Printf("Error fetching grid at %d: %v", slotTime.Unix(), err) + report(Progress{Phase: PhaseFetched, Slot: slot, TotalSlots: len(slots), SlotTime: slotTime, Completed: slot, Channels: len(channels), Programs: len(programs), Err: err}) + } else { + succeeded++ + for _, ch := range grid.Channels { + if _, seen := channelIndex[ch.ChannelID]; !seen { + channelIndex[ch.ChannelID] = len(channels) + channels = append(channels, guide.ConvertChannel(ch)) + } + position := guide.ConvertLineupPosition(ch) + if _, seen := lineupIndex[position.Key()]; !seen { + lineupIndex[position.Key()] = struct{}{} + lineup = append(lineup, position) + } + for _, ev := range ch.Events { + key := ch.ChannelID + "|" + ev.StartTime + "|" + ev.EndTime + if _, seen := eventIndex[key]; seen { + continue + } + eventIndex[key] = struct{}{} + programs = append(programs, guide.ConvertEvent(ev, ch.ChannelID, prefs.Language, prefs.Country)) + } + } + opts.Logger.Printf("Channels so far: %d, Events so far: %d", len(channels), len(programs)) + report(Progress{Phase: PhaseFetched, Slot: slot, TotalSlots: len(slots), SlotTime: slotTime, Completed: slot, Channels: len(channels), Programs: len(programs)}) + } + + if slot < len(slots) && opts.SlotDelay > 0 { + select { + case <-time.After(opts.SlotDelay): + case <-ctx.Done(): + return nil, ctx.Err() + } + } + } + + if succeeded == 0 { + return nil, fmt.Errorf("%w (%d slots attempted)", ErrNoData, len(slots)) + } + + guide.SortLineup(lineup) + return &guide.TVGuide{ + Channels: channels, + Programs: programs, + Lineup: lineup, + Source: guide.SourceFromPreferences(prefs, opts.Now().UTC()), + }, nil +} diff --git a/scrape/scrape_test.go b/scrape/scrape_test.go new file mode 100644 index 0000000..ab8c207 --- /dev/null +++ b/scrape/scrape_test.go @@ -0,0 +1,271 @@ +package scrape + +import ( + "context" + "errors" + "io" + "log" + "sync" + "testing" + "time" + + "github.com/daniel-widrick/GraceNoteScraper/web" +) + +var fixedNow = time.Date(2026, 9, 13, 15, 42, 0, 0, time.UTC) + +func testPrefs() web.Preferences { + return web.Preferences{Country: "USA", ZipCode: "13490", Headend: "lineupId", LineupId: "USA-lineupId-DEFAULT", Device: "-", Language: "en-us"} +} + +// fakeFetcher serves canned grids keyed by slot time and records every call. +type fakeFetcher struct { + mu sync.Mutex + grids map[int64]*web.GridResponse + errs map[int64]error + always *web.GridResponse + calls []int64 + block chan struct{} // when set, the first call blocks until closed +} + +func (f *fakeFetcher) GetDataByTimeContext(ctx context.Context, t int64) (*web.GridResponse, error) { + f.mu.Lock() + f.calls = append(f.calls, t) + block := f.block + f.block = nil + f.mu.Unlock() + if block != nil { + select { + case <-block: + case <-ctx.Done(): + return nil, ctx.Err() + } + } + if err, ok := f.errs[t]; ok { + return nil, err + } + if g, ok := f.grids[t]; ok { + return g, nil + } + if f.always != nil { + return f.always, nil + } + return &web.GridResponse{}, nil +} + +func channel(station, number string, events ...web.JSONEvent) web.JSONChannel { + return web.JSONChannel{ChannelID: station, ID: station + "0", ChannelNo: number, CallSign: "C" + station, AffiliateName: "Net " + station, Events: events} +} + +func event(start, end, title string) web.JSONEvent { + return web.JSONEvent{StartTime: start, EndTime: end, Duration: "60", SeriesID: "SH1", Program: web.JSONProgram{ID: "EP1", Title: title}} +} + +func quietOptions(f *fakeFetcher) Options { + return Options{Days: 1, SlotDelay: NoDelay, Fetcher: f, Now: func() time.Time { return fixedNow }, Logger: log.New(io.Discard, "", 0)} +} + +func TestSlotsAreMidnightAlignedSixHourWindows(t *testing.T) { + slots := Slots(fixedNow, 2) + if len(slots) != 8 { + t.Fatalf("slots = %d, want 8", len(slots)) + } + if !slots[0].Equal(time.Date(2026, 9, 13, 0, 0, 0, 0, time.UTC)) { + t.Errorf("first slot = %s", slots[0]) + } + for i := 1; i < len(slots); i++ { + if slots[i].Sub(slots[i-1]) != SlotDuration { + t.Errorf("slot %d not 6h after previous", i) + } + } + if len(Slots(fixedNow, 0)) != DefaultDays*4 { + t.Errorf("zero days should default to %d days", DefaultDays) + } +} + +func TestFetchRequestsEverySlotAndAssemblesGuide(t *testing.T) { + f := &fakeFetcher{always: &web.GridResponse{Channels: []web.JSONChannel{ + channel("s1", "2.1", event("2026-09-13T00:00:00Z", "2026-09-13T01:00:00Z", "a")), + channel("s2", "5", event("2026-09-13T00:00:00Z", "2026-09-13T02:00:00Z", "b")), + channel("s1", "1002"), // same station at a second number + }}} + opts := quietOptions(f) + opts.Days = 2 + + g, err := Fetch(context.Background(), testPrefs(), opts) + if err != nil { + t.Fatalf("Fetch: %v", err) + } + if len(f.calls) != 8 { + t.Fatalf("calls = %d, want 8", len(f.calls)) + } + if f.calls[0] != time.Date(2026, 9, 13, 0, 0, 0, 0, time.UTC).Unix() { + t.Errorf("first request time = %d", f.calls[0]) + } + if len(g.Channels) != 2 { + t.Errorf("channels = %d, want 2 (deduplicated by station)", len(g.Channels)) + } + if g.Channels[0].ID != "s1" || g.Channels[1].ID != "s2" { + t.Errorf("channels should keep first-seen order: %v %v", g.Channels[0].ID, g.Channels[1].ID) + } + if len(g.Lineup) != 3 { + t.Fatalf("lineup = %d, want 3 (never collapsed)", len(g.Lineup)) + } + if g.Lineup[0].ChannelNo != "2.1" || g.Lineup[1].ChannelNo != "5" || g.Lineup[2].ChannelNo != "1002" { + t.Errorf("lineup order = %s %s %s", g.Lineup[0].ChannelNo, g.Lineup[1].ChannelNo, g.Lineup[2].ChannelNo) + } + if len(g.Programs) != 2 { + t.Errorf("programs = %d, want 2 (same events across 8 slots deduplicated)", len(g.Programs)) + } + if g.Source.LineupID != "USA-lineupId-DEFAULT" || g.Source.PostalCode != "13490" || !g.Source.GeneratedAt.Equal(fixedNow) { + t.Errorf("source = %+v", g.Source) + } +} + +func TestFetchKeepsDistinctEventsAcrossSlots(t *testing.T) { + slots := Slots(fixedNow, 1) + f := &fakeFetcher{grids: map[int64]*web.GridResponse{ + slots[0].Unix(): {Channels: []web.JSONChannel{channel("s1", "2", event("2026-09-13T00:00:00Z", "2026-09-13T06:00:00Z", "morning"))}}, + slots[1].Unix(): {Channels: []web.JSONChannel{channel("s1", "2", event("2026-09-13T06:00:00Z", "2026-09-13T12:00:00Z", "midday"))}}, + }} + g, err := Fetch(context.Background(), testPrefs(), quietOptions(f)) + if err != nil { + t.Fatal(err) + } + if len(g.Programs) != 2 { + t.Fatalf("programs = %d, want 2", len(g.Programs)) + } +} + +func TestFetchSkipsFailedSlotAndReportsIt(t *testing.T) { + slots := Slots(fixedNow, 1) + boom := errors.New("upstream timeout") + f := &fakeFetcher{ + always: &web.GridResponse{Channels: []web.JSONChannel{channel("s1", "2")}}, + errs: map[int64]error{slots[1].Unix(): boom}, + } + var reports []Progress + opts := quietOptions(f) + opts.Progress = func(p Progress) { reports = append(reports, p) } + + g, err := Fetch(context.Background(), testPrefs(), opts) + if err != nil { + t.Fatalf("partial failure must not be an error: %v", err) + } + if len(g.Channels) != 1 { + t.Fatalf("channels = %d", len(g.Channels)) + } + if len(reports) != 8 { + t.Fatalf("progress reports = %d, want 2 per slot", len(reports)) + } + failed := reports[3] // slot 2, PhaseFetched + if failed.Phase != PhaseFetched || failed.Slot != 2 || !errors.Is(failed.Err, boom) { + t.Errorf("failed slot report = %+v", failed) + } + for i, r := range reports { + if r.TotalSlots != 4 { + t.Errorf("report %d TotalSlots = %d", i, r.TotalSlots) + } + if i > 0 && r.Completed < reports[i-1].Completed { + t.Errorf("Completed went backwards at report %d", i) + } + } + last := reports[len(reports)-1] + if last.Completed != 4 || last.Slot != 4 || last.Phase != PhaseFetched { + t.Errorf("last report = %+v", last) + } +} + +func TestFetchReturnsErrNoDataWhenEverySlotFails(t *testing.T) { + f := &fakeFetcher{errs: map[int64]error{}} + for _, s := range Slots(fixedNow, 1) { + f.errs[s.Unix()] = errors.New("down") + } + _, err := Fetch(context.Background(), testPrefs(), quietOptions(f)) + if !errors.Is(err, ErrNoData) { + t.Fatalf("err = %v, want ErrNoData", err) + } +} + +func TestFetchStopsPromptlyWhenContextCancelledDuringDelay(t *testing.T) { + f := &fakeFetcher{always: &web.GridResponse{Channels: []web.JSONChannel{channel("s1", "2")}}} + opts := quietOptions(f) + opts.SlotDelay = time.Minute + ctx, cancel := context.WithCancel(context.Background()) + + done := make(chan error, 1) + go func() { + _, err := Fetch(ctx, testPrefs(), opts) + done <- err + }() + // Let the first slot complete and the delay begin, then cancel. + deadline := time.After(2 * time.Second) + for { + f.mu.Lock() + n := len(f.calls) + f.mu.Unlock() + if n >= 1 { + break + } + select { + case <-deadline: + t.Fatal("first slot never requested") + default: + time.Sleep(5 * time.Millisecond) + } + } + cancel() + select { + case err := <-done: + if !errors.Is(err, context.Canceled) { + t.Fatalf("err = %v, want context.Canceled", err) + } + case <-time.After(2 * time.Second): + t.Fatal("Fetch did not return after cancellation") + } + if len(f.calls) != 1 { + t.Errorf("calls after cancel = %d, want 1", len(f.calls)) + } +} + +func TestFetchStopsWhenContextCancelledDuringRequest(t *testing.T) { + f := &fakeFetcher{always: &web.GridResponse{}, block: make(chan struct{})} + ctx, cancel := context.WithCancel(context.Background()) + done := make(chan error, 1) + go func() { + _, err := Fetch(ctx, testPrefs(), quietOptions(f)) + done <- err + }() + time.Sleep(20 * time.Millisecond) + cancel() + select { + case err := <-done: + if !errors.Is(err, context.Canceled) { + t.Fatalf("err = %v", err) + } + case <-time.After(2 * time.Second): + t.Fatal("Fetch did not return") + } +} + +func TestFetchHonoursAlreadyCancelledContext(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + cancel() + f := &fakeFetcher{} + if _, err := Fetch(ctx, testPrefs(), quietOptions(f)); !errors.Is(err, context.Canceled) { + t.Fatalf("err = %v", err) + } + if len(f.calls) != 0 { + t.Errorf("no requests should be made, got %d", len(f.calls)) + } +} + +func TestOptionsDefaults(t *testing.T) { + o := Options{}.withDefaults(testPrefs()) + if o.Days != DefaultDays || o.SlotDelay != DefaultSlotDelay || o.Fetcher == nil || o.Now == nil || o.Logger == nil { + t.Fatalf("defaults not applied: %+v", o) + } + if _, ok := o.Fetcher.(*web.Client); !ok { + t.Fatalf("default fetcher should be *web.Client, got %T", o.Fetcher) + } +} From 5fc179248670bb987416e600adea52c2bc491e9e Mon Sep 17 00:00:00 2001 From: Daniel Widrick Date: Sat, 12 Sep 2026 23:39:38 -0400 Subject: [PATCH 2/2] docs: note the default grid client's on-disk cache --- CLAUDE.md | 2 +- README.md | 2 ++ scrape/scrape.go | 5 +++-- 3 files changed, 6 insertions(+), 3 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 0198e9a..69f072a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -27,7 +27,7 @@ Run `go test ./...` for the setup, configuration, and provider-client tests. ## Architecture -The binary is a single Go process that scrapes GraceNote/TMS for 14 days of TV listings and serves the data as XMLTV over HTTP. The grid download loop and guide assembly live in `scrape/` (`scrape.Fetch`), which is importable by other programs and does no enrichment, file I/O, or env reads. Enrichment, persistence, and runtime orchestration live in `main.go`; setup handlers live in `setup.go` and persisted configuration lives in `appconfig/`. +The binary is a single Go process that scrapes GraceNote/TMS for 14 days of TV listings and serves the data as XMLTV over HTTP. The grid download loop and guide assembly live in `scrape/` (`scrape.Fetch`), which is importable by other programs and does no enrichment or env reads (the default `web.Client` it uses does keep `grid_cache/` on disk). Enrichment, persistence, and runtime orchestration live in `main.go`; setup handlers live in `setup.go` and persisted configuration lives in `appconfig/`. **Data flow:** diff --git a/README.md b/README.md index 7242a99..cec7897 100644 --- a/README.md +++ b/README.md @@ -192,6 +192,8 @@ func main() { `Fetch` returns a `guide.TVGuide`: `Channels` deduplicated by station, `Lineup` with every position, `Programs`, and `Source`. It pauses five seconds between grid requests by default and skips slots that fail; it returns `scrape.ErrNoData` only when every slot failed. Logo and TMDB enrichment are not part of the package. Lineup discovery by postal code is available through `web.NewProviderClient().FindProviders`. +The default grid client is `web.NewClient`, which retries failed requests and keeps a raw-response cache in a `grid_cache/` directory under the working directory. Pass your own `Options.Fetcher` to change that. + ## Project Structure ``` diff --git a/scrape/scrape.go b/scrape/scrape.go index 0ee481b..cf863d5 100644 --- a/scrape/scrape.go +++ b/scrape/scrape.go @@ -1,6 +1,7 @@ // Package scrape fetches a Gracenote lineup's listings and converts them into -// a guide.TVGuide. It performs no enrichment, file I/O, or environment reads, -// so it can be imported by other programs. +// a guide.TVGuide. It performs no enrichment or environment reads, so it can +// be imported by other programs. The default fetcher, web.NewClient, keeps its +// own raw-response cache in ./grid_cache; supply Options.Fetcher to avoid it. package scrape import (