htrflow-batch

Not for use yet. This repository is under active development and is not ready for others to run: interfaces, chart values and the campaigns format still change without notice. The releases are pre-releases for trying it out, not for production.

Run an htrflow pipeline on whole archival volumes, across a Kubernetes cluster of GPU nodes. Your pipeline stays exactly as it is; htrflow-batch decides where each volume runs, streams its pages in, and publishes ALTO and PAGE XML page by page. What to transcribe is a file in git.

Quickstart

One page of a sample volume, transcribed on your own machine and opened in the viewer. You need git, make and Docker with Compose; no cluster, no GPU.

git clone https://github.com/AI-Riksarkivet/htrflow-batch && cd htrflow-batch
make compose-up
docker compose -f .docker/docker-compose.yml logs -f wrapper    # returns after "COMPLETE 1 pages"

Then log in at http://localhost:8080/login as htr-reader / htr-reader-pass, open http://localhost:8080/uv.html#?manifest=http://localhost:8080/results/htr-batch/demo-v1/mock-vol/iiif.json, and make compose-down when you are done. Ports taken, or anything else unexpected: Quickstart.

On a cluster

A campaign lists volumes and names a pipeline. You open a pull request, and once it is merged an apply sends it to the cluster as one Kubernetes Indexed Job, one pod per volume, queued by Kueue until its GPUs are free.

# campaigns/kyrkobocker-1.yaml
pipeline: demo-v1
volumes:
  - id: loc-mal2459400
    manifest: https://…/manifest.json
  - id: loose-scans
    images:
      - https://…/scan-0001.jpg

Source images can also be cached in a private S3 bucket, so a volume run again needs nothing from the IIIF server (Cache source images).

A web front shows every campaign and volume live, with each volume's run log and the transcription in the viewer:

• Deploy: the production install.
• Run a campaign: your first campaigns repo.
• How it works, and the slides that walk through it in pictures.

Developing

make install && make test     # uv workspace sync + the Python tests
make ci                       # every gate CI runs, through dagger

Development covers the repository layout, testing, CI, releasing and a dev cluster. The documentation site is built from docs/ (make docs-serve locally).

License

htrflow-batch is licensed under the European Union Public Licence (EUPL-1.2), the same licence as htrflow. See LICENSE. The third-party components the images ship are listed in Third-party licences.