Skip to content

fix(upload,metadata): self-heal videos stuck in uploaded - #86

Merged
SlavaKuntsov merged 5 commits into
mainfrom
fix/upload-stuck-status
Oct 22, 2026
Merged

SlavaKuntsov merged 5 commits into
mainfrom
fix/upload-stuck-status

Conversation

@SlavaKuntsov

Copy link
Copy Markdown
Owner

Why

A video row is created in metadata before the bytes land (status='uploaded' by default), and the only transition out of uploaded is the transcoder consuming video.uploaded. So any upload that never reaches complete — closed tab, lost resume key, aborted e2e run, publish failure, or a queue recreated by topology self-heal — left a row hanging in uploaded forever. The player polls forever and shows an endless "Transcoding…" spinner.

What

  • Self-healing reconciler in the upload service (services/upload/internal/reconciler). Every UPLOAD_RECONCILE_INTERVAL_SECONDS it takes uploaded rows older than UPLOAD_STALE_TTL_SECONDS and:
    • re-publishes video.uploaded when raw/{id}/original.mp4 exists → recovers a lost event (transcoder is idempotent, skips ready/failed);
    • aborts the dangling multipart session and marks the row failed otherwise → resolves an abandoned upload.
  • metadata: new internal GET /internal/videos/stale?older_than_seconds=&limit= endpoint + VideoRepo.ListStaleUploaded + model.StaleUpload.
  • upload: MetadataClient.ListStaleUploaded/SetStatus, MinioClient.AbortMultipartUpload, reconciler wiring in cmd/server; new env knobs in .env.example and deploy/docker-compose.yml.
  • Feed: GET /api/v1/videos now shows only public+ready to everyone; the owner still sees their own non-ready videos (uploaded/processing/failed) to delete or re-upload. Non-ready junk no longer leaks to other users.
  • frontend: while status is uploaded the watch page says "Waiting for upload…" instead of "Transcoding…".
  • e2e: the interrupted-resume case now calls /complete, so runs no longer leave uploaded zombies.

Verification

  • go test ./... (metadata, upload) green; golangci-lint run 0 issues; npm run lint clean.
  • Live stack: 9 stuck rows auto-resolved to failed in one reconciler pass (no manual SQL).
  • Anonymous feed returned only ready; owner feed returned its ready + failed rows.
  • make e2e PASS (presign / multipart / resumable flows; resumed upload now reaches ready).

Notes

  • services/metadata/docs (swagger) not regenerated — swag isn't installed locally; the new endpoint is annotated and will appear after make swagger.
  • If a real upload runs longer than UPLOAD_STALE_TTL_SECONDS, its row may be failed — raise the TTL for slow links.

A video row is created before the bytes land, so an upload that never completes (closed tab, lost resume key, publish failure, recreated queue) stayed in 'uploaded' forever and the player spun endlessly. The upload service now reconciles stale rows: it re-publishes video.uploaded when the original exists in MinIO (lost event) and aborts the multipart session plus marks failed when it does not (abandoned upload), backed by a new internal /internal/videos/stale endpoint and UPLOAD_STALE_TTL_SECONDS / UPLOAD_RECONCILE_INTERVAL_SECONDS knobs. The feed also now hides non-ready videos from everyone but the owner.
While a video is in 'uploaded' the player said "Transcoding…", which hid that the upload itself had not finished (or was abandoned). Label the state honestly so the stuck-upload case is not mistaken for transcoding.
The interrupted-resume case never called /complete, so every e2e run left a video stuck in 'uploaded'. Finish the upload so the test does not pollute the feed with abandoned rows.
@SlavaKuntsov SlavaKuntsov self-assigned this Oct 22, 2026
main reverted the frontend redesign (PR #84) and re-merged only VideoCard (#85), so the redesign branch no longer fast-forwards. Keep main's frontend and re-apply only the stuck-upload label change on top of it.
requires-python = ">=3.14" has no upper bound and the repo had no .python-version, so uv selected the newest interpreter (cpython-3.15.1) on clean CI runners. psycopg-binary 3.3.4 ships only cp314 wheels, so installing the auth env failed and took lint-py/test-py down (main red since 2026-10-09). Pin both services to 3.14, matching the python:3.14-slim images and the py314 black/mypy/ruff targets. Locally the breakage was masked because a 3.14 interpreter was already installed.
@SlavaKuntsov
SlavaKuntsov merged commit 35f0cf7 into main Oct 22, 2026
4 checks passed
@SlavaKuntsov
SlavaKuntsov deleted the fix/upload-stuck-status branch October 22, 2026 17:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant