Add retry logic for transient network failures and failure tracking

README.md

@@ -80,12 +80,18 @@ ytscript run                     # first run: the latest 30 videos
 ytscript run                     # later runs: only what is new
 
 ytscript polish scripts          # re-clean scripts already written
+
+ytscript failures                # what an earlier run could not finish
+ytscript run --only-failed       # take another run at exactly those
 ```
 
 Scripts land in `output_dir` as `2024-05-01_Video-title_VIDEOID.txt`, and every finished
 video is recorded in the state file, so re-running is cheap and safe. State is written
 after each video, so an interrupted backfill resumes where it stopped. A video that
-fails is not recorded and is retried on the next run.
+fails is not recorded as done — it goes on the state file's failure list instead, so
+`ytscript failures` can show it and `--retry-failed` can pick it back up long after it
+has scrolled out of the `check_limit` window. See [When a download
+drops](#when-a-download-drops).
 
 Useful flags on `run`:
 
@@ -100,6 +106,9 @@ Useful flags on `run`:
 | `--format txt,md,json` | Write more than one rendering |
 | `--timestamps` | Prefix each paragraph with `[hh:mm:ss]` |
 | `--dry-run` | List what is missing without downloading anything |
+| `--retry-failed` | Also re-attempt videos an earlier run could not finish |
+| `--only-failed` | Do just those, skipping the channel listing and the attempt limit |
+| `--retries N` | Extra attempts a download gets when the connection drops (default 3) |
 | `--keep-audio` | Keep the downloaded audio next to the scripts |
 | `--drive` / `--no-drive` | Also copy each script into Google Drive, or not |
 | `--drive-folder ID` | The Drive folder they go into |
@@ -110,6 +119,10 @@ Useful flags on `run`:
 
 The cookie and members-only flags work on `list` too.
 
+`ytscript failures` prints the failure list — id, attempts, when, title and the error —
+and `ytscript failures --clear [ID ...]` forgets entries, which also resets their
+attempt count.
+
 `ytscript polish` runs the same clean-up a run does over scripts that already exist —
 useful after adding a term to the vocabulary, or on a backlog transcribed before it had
 one. It takes files or directories (`.txt` and `.md`), rewrites them in place, and has
@@ -128,6 +141,12 @@ language = "zh"              # main spoken language, ISO 639-1; "auto" to detect
 initial_backfill = 30        # videos transcribed on the very first run
 check_limit = 5              # videos inspected on later runs
 
+download_retries = 3         # extra attempts when the connection drops; 0 means one try
+retry_backoff = 5.0          # seconds before the second attempt, doubling after that
+socket_timeout = 30.0        # seconds a stalled connection gets before it counts as failed
+retry_failed = false         # every run also re-attempts what the failure list holds
+retry_max_attempts = 3       # times a failed video is picked up again before it is left alone
+
 backend = "faster-whisper"   # or "openai"
 whisper_model = "large-v3"   # tiny | base | small | medium | large-v3 | distil-large-v3
 whisper_device = "cuda"      # "cpu", "cuda", ...
@@ -168,6 +187,62 @@ include_members_only = false          # true also transcribes members-only video
 Every key has a matching environment variable: `YTSCRIPT_CHANNEL`,
 `YTSCRIPT_LANGUAGE`, `YTSCRIPT_BACKEND`, and so on.
 
+### When a download drops
+
+YouTube hangs up part-way through a download often enough that a backfill of thirty
+videos rarely gets through untouched:
+
+```
+[download]  63.2% of 48.19MiB at 1.02MiB/s ETA 00:17
+[download] Got error: ('Connection aborted.', ConnectionResetError(10054,
+'远程主机强迫关闭了一个现有的连接。', None, 10054, None))
+```
+
+That is the network, not the video — the same URL usually works seconds later. ytscript
+handles it in two places.
+
+**During the run.** A request that fails on something that looks like network trouble is
+made again, waiting `retry_backoff` seconds, then twice that, then twice that again, for
+`download_retries` extra attempts. yt-dlp resumes from the `.part` file it already has,
+so a drop at 63% costs the pause, not the 63%. Refusals are told apart from drops and
+are not retried: a members-only video or a deleted one fails immediately, as it should.
+`--retries 6` widens it for a bad line, `--retries 0` turns it off.
+
+**Between runs.** A video that still fails — its retries used up, the backend out of
+memory, Drive refusing the upload — is written to the state file under `failures`, with
+the error, the attempt count and enough about the video to fetch it again:
+
+```bash
+ytscript failures
+# 1 video(s) failed; 'ytscript run --retry-failed' tries them again:
+#   dQw4w9WgXcQ  2 attempt(s)  2024-05-02T09:14:31+00:00  Market wrap, May 1
+#       could not download audio for dQw4w9WgXcQ: ('Connection aborted.', ...)
+```
+
+This matters for a backfill. A plain run only looks at the newest `check_limit` videos,
+so a video that failed while thirty were being transcribed is out of the window by the
+next run and would never be seen again. `--retry-failed` puts the failure list back in
+front of the queue whatever its age, and `--only-failed` does those and nothing else,
+skipping the channel listing entirely:
+
+```bash
+ytscript run --retry-failed      # the newest few, plus everything that failed before
+ytscript run --only-failed       # just the failures, no listing
+```
+
+Set `retry_failed = true` to make every run do it. A video that keeps failing is picked
+up `retry_max_attempts` times and then left alone, so a genuinely broken one does not
+cost a download on every run:
+
+```
+left 1 failed video(s) alone after retry_max_attempts; 'ytscript run --only-failed'
+tries them anyway
+```
+
+`--only-failed` ignores that limit — it is an explicit request — and `ytscript failures
+--clear [ID ...]` forgets entries entirely, resetting their counts. A video that
+succeeds drops off the list by itself.
+
 ### Members-only videos
 
 A channel's members-only videos show up in its uploads listing, but YouTube refuses the
@@ -583,7 +658,9 @@ print(report.written)
 
 `Pipeline` takes an optional `client` and `transcriber`, so a different source or
 speech-to-text engine only has to match the small protocol in
-`ytscript/transcribers/base.py`.
+`ytscript/transcribers/base.py`. `run()` takes the retry switches too —
+`run(retry_failed=True)` and `run(only_failed=True)` — and the report it returns carries
+`failed`, `retried` and `given_up` alongside `written`.
 
 ## Development
 

src/ytscript/cli.py

@@ -12,6 +12,7 @@
 from .models import RunReport
 from .pipeline import Pipeline
 from .polish import polish_text
+from .state import State
 from .vocabulary import VocabularyError, load_vocabulary
 from .youtube import YouTubeError
 
@@ -110,6 +111,33 @@ def add_common(target: argparse.ArgumentParser) -> None:
         metavar="ID_OR_URL",
         help="Google Drive folder the scripts go into",
     )
+    retry = run.add_mutually_exclusive_group()
+    retry.add_argument(
+        "--retry-failed",
+        dest="retry_failed",
+        action="store_true",
+        default=None,
+        help="also re-attempt videos an earlier run could not finish, however old they are",
+    )
+    retry.add_argument(
+        "--no-retry-failed",
+        dest="retry_failed",
+        action="store_false",
+        default=None,
+        help="only look at the newest videos (the default)",
+    )
+    retry.add_argument(
+        "--only-failed",
+        action="store_true",
+        help="re-attempt just those, skipping the channel listing and the attempt limit",
+    )
+    run.add_argument(
+        "--retries",
+        dest="download_retries",
+        type=int,
+        metavar="N",
+        help="extra attempts a download gets when the connection drops (default 3)",
+    )
     run.add_argument("--keep-audio", dest="keep_audio", action="store_true", default=None)
     run.add_argument("--state-file", dest="state_file", type=Path)
     run.add_argument(
@@ -147,6 +175,19 @@ def add_common(target: argparse.ArgumentParser) -> None:
     add_common(listing)
     listing.add_argument("--limit", type=int, default=10)
 
+    failures = sub.add_parser(
+        "failures",
+        help="show the videos an earlier run could not finish",
+    )
+    failures.add_argument("--state-file", dest="state_file", type=Path)
+    failures.add_argument(
+        "--clear",
+        nargs="*",
+        metavar="ID",
+        default=None,
+        help="forget these failures, or all of them when given no id",
+    )
+
     sub.add_parser(
         "drive-auth",
         help="sign in to Google Drive once and cache the token for later runs",
@@ -162,6 +203,8 @@ def add_common(target: argparse.ArgumentParser) -> None:
 _OVERRIDE_FIELDS = (
     "channel",
     "language",
+    "retry_failed",
+    "download_retries",
     "backend",
     "whisper_model",
     "whisper_batch_size",
@@ -192,6 +235,13 @@ def _config_from_args(args: argparse.Namespace) -> Config:
 
 def _print_report(report: RunReport, dry_run: bool) -> None:
     print(f"checked {report.checked} video(s); {len(report.skipped)} already had a script")
+    if report.retried:
+        print(f"picked {len(report.retried)} video(s) back up from the failure list")
+    if report.given_up:
+        print(
+            f"left {len(report.given_up)} failed video(s) alone after retry_max_attempts; "
+            "'ytscript run --only-failed' tries them anyway"
+        )
     if report.members_only:
         print(
             f"skipped {len(report.members_only)} members-only video(s); "
@@ -212,6 +262,12 @@ def _print_report(report: RunReport, dry_run: bool) -> None:
             print(f"  {item}")
     for video_id, error in report.failed:
         print(f"  failed: {video_id}: {error}", file=sys.stderr)
+    if report.failed and not dry_run:
+        print(
+            f"{len(report.failed)} failure(s) recorded; "
+            "'ytscript run --retry-failed' takes another run at them",
+            file=sys.stderr,
+        )
 
 
 def cmd_run(args: argparse.Namespace) -> int:
@@ -224,6 +280,7 @@ def cmd_run(args: argparse.Namespace) -> int:
         limit=limit,
         dry_run=args.dry_run,
         on_progress=lambda label: print(label, flush=True),
+        only_failed=args.only_failed,
     )
     _print_report(report, args.dry_run)
     return 1 if report.failed else 0
@@ -293,6 +350,35 @@ def cmd_polish(args: argparse.Namespace) -> int:
     return 0
 
 
+def cmd_failures(args: argparse.Namespace) -> int:
+    # No channel is needed to read the state file.
+    overrides = {"state_file": args.state_file} if args.state_file else {}
+    config = load_config(path=args.config, overrides=overrides)
+    state = State.load(config.state_file)
+
+    if args.clear is not None:
+        dropped = state.forget_failures(args.clear or None)
+        if dropped:
+            state.save()
+        print(f"forgot {len(dropped)} failure(s)")
+        for video_id in dropped:
+            print(f"  {video_id}")
+        return 0
+
+    entries = state.failed_videos()
+    if not entries:
+        print("no failures on record")
+        return 0
+    print(f"{len(entries)} video(s) failed; 'ytscript run --retry-failed' tries them again:")
+    for entry in entries:
+        attempts = entry.get("attempts", 1)
+        when = entry.get("last_failed_at", "")
+        title = entry.get("title", "")
+        print(f"  {entry['id']}  {attempts} attempt(s)  {when}  {title}")
+        print(f"      {entry.get('error', '')}")
+    return 0
+
+
 def cmd_drive_auth(args: argparse.Namespace) -> int:
     # No channel is needed to authorise, so this skips the usual validation.
     config = load_config(path=args.config)
@@ -334,6 +420,7 @@ def main(argv: list[str] | None = None) -> int:
         "run": cmd_run,
         "list": cmd_list,
         "polish": cmd_polish,
+        "failures": cmd_failures,
         "init": cmd_init,
         "drive-auth": cmd_drive_auth,
     }

src/ytscript/config.py

@@ -39,6 +39,28 @@ class Config:
     check_limit: int = 5
     """Newest videos inspected on later runs; unseen ones get transcribed."""
 
+    # --- when something goes wrong ---------------------------------------
+    download_retries: int = 3
+    """Extra attempts a YouTube request gets when the connection drops. ``0`` means one try.
+
+    This is the fix for ``('Connection aborted.', ConnectionResetError(10054, ...))``
+    and its friends: the download starts again and yt-dlp resumes the part it has."""
+
+    retry_backoff: float = 5.0
+    """Seconds before the second attempt; each further wait doubles it (5, 10, 20...)."""
+
+    socket_timeout: float = 30.0
+    """Seconds a stalled connection is given before it counts as a failed attempt."""
+
+    retry_failed: bool = False
+    """Also re-attempt the videos recorded as failed, even when they have fallen out of
+    the ``check_limit`` window. ``--retry-failed`` turns it on for a single run."""
+
+    retry_max_attempts: int = 3
+    """How many times a failed video is picked up again before it is left alone.
+    ``ytscript run --only-failed`` retries it regardless, and ``ytscript failures --clear``
+    starts the count over."""
+
     # --- speech-to-text -------------------------------------------------
     backend: str = "faster-whisper"
     whisper_model: str = "large-v3"
@@ -176,6 +198,14 @@ def validate(self) -> None:
             raise ConfigError("check_limit must be at least 1")
         if self.whisper_batch_size < 1:
             raise ConfigError("whisper_batch_size must be at least 1 (1 turns batching off)")
+        if self.download_retries < 0:
+            raise ConfigError("download_retries cannot be negative (0 means one attempt)")
+        if self.retry_backoff < 0:
+            raise ConfigError("retry_backoff cannot be negative")
+        if self.socket_timeout <= 0:
+            raise ConfigError("socket_timeout must be greater than 0")
+        if self.retry_max_attempts < 1:
+            raise ConfigError("retry_max_attempts must be at least 1")
         # Reading the file now means a typo fails the command, not the first video.
         try:
             load_vocabulary(self.vocabulary)
@@ -304,6 +334,19 @@ def load_config(
 initial_backfill = 30
 check_limit = 5
 
+# A dropped connection mid-download ("Connection aborted", ConnectionResetError) is
+# retried this many extra times, waiting 5s, then 10s, then 20s between attempts.
+download_retries = 3
+retry_backoff = 5.0
+socket_timeout = 30.0
+
+# A video that still fails is written to the state file's "failures" list. Turn this
+# on to re-attempt those on every run, even once they are older than `check_limit`;
+# `ytscript run --retry-failed` does it for one run, and `ytscript failures` shows
+# what is on the list. Each one is picked up at most `retry_max_attempts` times.
+retry_failed = false
+retry_max_attempts = 3
+
 # "faster-whisper" runs locally, "openai" calls the hosted transcription API.
 backend = "faster-whisper"
 

src/ytscript/models.py

@@ -81,5 +81,11 @@ class RunReport:
     members_only: list[str] = field(default_factory=list)
     """Videos passed over because ``include_members_only`` is off."""
 
+    retried: list[str] = field(default_factory=list)
+    """Videos brought back from the state file's failure list, outside the usual window."""
+
+    given_up: list[str] = field(default_factory=list)
+    """Failures left alone because they have already had ``retry_max_attempts`` goes."""
+
     uploaded: list[str] = field(default_factory=list)
     """Scripts copied into Google Drive, as ``name  link``. Empty unless ``drive_upload`` is on."""