scripts

Shared scripts

Docker runners and retail data generators used by integration-tests/ and retail-example/.

Runners

./scripts/run-postgres.sh up
./scripts/run-marquez.sh up   # OpenLineage / Marquez (API :5001, UI :3001)
./scripts/run-hop.sh <hop-project-dir> <workflow>
./scripts/run-adventureworks.sh up   # SQL Server 2025 + AdventureWorks2025.bak restore
./scripts/smoke-lineage-view.sh   # optional Marquez GET /lineage + hop_export check

${MARQUEZ_API} (http://localhost:5001/api/v1/lineage) is the Export data lineage POST URL. ${MARQUEZ_BASE_URL} (http://localhost:5001) is the Marquez host root for a lineage backend. See docs/hop-lineage-view.adoc. smoke-lineage-view.sh is not part of mvn test or run-tests-all-databases.sh.

Project folder	Example
integration-tests	./scripts/run-hop.sh integration-tests tests/run-tests.hwf
retail-example	./scripts/run-hop.sh retail-example workflows/run-retail-initial.hwf
adventureworks	./scripts/run-adventureworks.sh up then load / update / spot-check

Thin wrappers also exist in integration-tests/run-tests.sh and integration-tests/run-postgres.sh.

Retail data generators

Source CSV and ASN waves are produced by the Synthetic data transform in retail-example/pipelines/generate-retail-data.hpl (see docs/synthetic-data.adoc). Python helpers that remain:

• retail-example/scripts/activate-source-wave.py — switch catalog file masks
• retail-example/scripts/generate-catalog-sources.py — refresh committed fixtures/catalog-sources/ after source-contract edits (--write-fixtures). Initial setup copies those files from workflows/bootstrap-retail-work.hwf and does not run this script.
• scripts/end-to-end/ — shared copies of the catalog and wave-activation helpers used outside the retail project

Video tutorials

Rebuild a narrated tutorial MP4 from docs/video-tutorials/<name>/ using xAI TTS and MoviePy:

export XAI_API_KEY=your-key-here

# Quick test (first 3 scenes)
./scripts/rebuild-video-tutorial.sh your-first-data-vault-model --limit 3

# Full build
./scripts/rebuild-video-tutorial.sh your-first-data-vault-model --locale en

# Video only from cached TTS
./scripts/rebuild-video-tutorial.sh your-first-data-vault-model --video-only --locale en

The script creates scripts/video/.venv on first run and installs moviepy, pydub, requests, and PyYAML. Video builds require ffmpeg on PATH.

Each scene lasts for the spoken audio plus a 1 second pause (configurable with --pause-after-sec). See docs/video-tutorials/your-first-data-vault-model/README.md.

Lower-level entry point: scripts/video/build_moviepy_video.py.