Skip to content

Subtitle auto-sync: whole-file reference (Matroska + MP4) - #756

Merged
ProdigyV21 merged 4 commits into
ProdigyV21:mainfrom
silentbil:subtitle_autosync_fork
Sep 27, 2026
Merged

ProdigyV21 merged 4 commits into
ProdigyV21:mainfrom
silentbil:subtitle_autosync_fork

Conversation

@silentbil

Copy link
Copy Markdown
Collaborator

Summary

Find Best Match now syncs against the embedded subtitle timeline of the whole file instead of the
few lines buffered around the playhead. It can also retime subtitles, and it refuses matches that
only look right.

Changes

  • Whole-file reference, read from the Matroska Cues index or MP4 sample tables over HTTP range
    requests. It's prefetched when the stream opens and shared per stream.
  • Matching uses the whole candidate pool instead of the top 10 and a per-third offset profile. It
    falls back to the file's other embedded tracks when the main track is mistimed.
  • Retiming handles frame-rate mismatches and large offsets (e.g. −47 s, −147 s).
  • A subtitle is refused when its retime isn't consistent across the episode, when another embedded
    track accepts it unshifted, or when an AI-measured shift doesn't fit the whole file.
  • Image-only (PGS) tracks are never used as a reference.
  • The remembered-match cache is turned off, so every playback scans.
  • Also: 4K hardware-first decoding on phones, and addon-only source lookups no longer overwrite
    the full cached source list.

@silentbil silentbil changed the title Subtitle auto-sync: whole-file reference (Matroska + MP4 Subtitle auto-sync: whole-file reference (Matroska + MP4) Sep 25, 2026
@github-actions github-actions Bot added documentation Improvements or additions to documentation area: android Changes to the Android app or Gradle build labels Sep 25, 2026
@ProdigyV21

Copy link
Copy Markdown
Owner

Thanks for the work on this! Whole-file matching and fixing subtitle drift look like useful improvements for ARVIO. It merges cleanly, both Android variants compile, and 118 focused tests passed, but please address these before we merge:

  1. In PlayerViewModel, if (bestBySignature.isEmpty()) return@let current restores the earlier scores/offsets when every retiming attempt fails. This can accept a match even after the consistency check rejected it. Keep those evaluated candidates rejected, even when none passes, and add a regression test.
  2. The normal alternate-reference path uses timeline.tracks without excluding PGS/VobSub image tracks. Please apply the text-only reference filter consistently to primary, alternate and fallback paths, with a mixed text/image-track test.
  3. loadAll() downloads and retains the entire candidate pool before deduplicating. Six concurrent downloads limit concurrency, not total memory or waiting time. Please process bounded batches with an overall time/memory budget and retain provider diversity, so large lists don't make scans too heavy on TVs.

The new 4K decoder preference also needs phone/TV playback testing separately; the automated subtitle tests don't verify that hardware behavior. I haven't physically tested this build. The feature is worth pursuing, but let's close these gaps first. Thanks!

@silentbil

Copy link
Copy Markdown
Collaborator Author

@ProdigyV21 thanks for the review! All three are addressed in fb0bc10:

  1. Retime verdicts when nothing retimes: the early return@let current is gone. The retimer's verdicts now go through applyRetimeVerdicts, so a candidate it evaluated and could not pair consistently stays rejected (below the accept bar, unshifted) even when no candidate retimed. Candidates it didn't evaluate keep their own verdict. Regression test in MatchCandidatePoolTest.
  2. Image tracks as references: one shared MatroskaSubtitleIndex.isImageTrack (PGS/VobSub/DVB) is now applied on every reference path: the preferred-language track (pickTrackForLanguage), the alternates (new alternateReferenceTracks) and the timing-shape fallback. Tests cover a mixed text/image file, including a PGS track listed before the text track.
  3. Bounded candidate pool: loadAll() now downloads in batches of 6, stops at 30 candidates or once a 20s budget has passed, and orders the pool with providers in turn (best-ranked first within each), so a long list can't crowd out a provider. Tests cover the ordering, the cap and the time budget.

4K decoder: I dropped HardwareFirstVideoRenderer. The sideload FFmpeg extension's video renderer is an unimplemented stub (supportsFormat always returns unsupported), so the wrapper was a no-op. The remaining change lets the platform decoder take a track that "exceeds capabilities" on every device. Tested on a Redmi Note 13 Pro, where 4K HEVC Main10 went from no video to playing normally. It still needs a TV-box check.

Testing: the subtitle tests pass (78, 8 new). On an Android TV emulator: Peaky Blinders S01E04 still finds its -47.8s match, and The Shards S01E04, where every retime is refused, correctly ends on no match.

@ProdigyV21
ProdigyV21 merged commit ff6528a into ProdigyV21:main Sep 27, 2026
5 checks passed
@ProdigyV21

Copy link
Copy Markdown
Owner

Thanks @silentbil for the feature and follow-up fixes. I added a small follow-up commit to enforce the candidate-download deadline, including servers that stall after sending response headers. Completed candidates are retained and reused, expired budgets start no new downloads, and cancellation aborts the HTTP request. Added five regression tests. Validation: 131 focused Play subtitle/retiming/container tests passed; GitHub's full Sideload unit-test build, 737 web tests, 112 backend tests, and web type check passed. Merged with your original commits/authorship preserved. I have not done an additional physical-TV playback test for this follow-up.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: android Changes to the Android app or Gradle build documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants