Repository navigation
docs(infra): add pipeline runbook (TODO-5) - #86
Merged
Merged
Conversation
Document the operational procedures for monitoring, diagnosing and verifying the production pipeline. Four sections: 1. Weekly surveillance (mitigation for the D109 coverage gap on silent scheduler failures, pending TODO-4 Niveau 2 log-based metric). 2. Diagnostic of a broken run (4-step methodology validated on the D113 incident — Cloud Run executions / image deployed / structlog logs filtered by execution / BigQuery freshness). 3. Active Cloud Monitoring alerts (D105, D106, D109) + explicit list of uncovered failure modes to avoid false confidence. 4. JSON path quick reference for gcloud describe commands, including the Knative v1 nesting trap for Cloud Run Jobs v2 resources. Refs: D109, D110, D113, TODO-5
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Contexte
Ce runbook capture les procédures opérationnelles découvertes pendant l'incident D113 du 2026-05-25 (premier run Scheduler post-PR #81 cassé sur env vars manquantes). La méthodologie de diagnostic en 4 steps qui a permis d'isoler la cause racine en ~30 min mérite d'être tracée formellement — sans documentation, ce savoir-faire opérationnel disparaît à la prochaine session.
Absorbe TODO-5 (« runbook minimal
docs/runbook.md») slotté initialement Bloc 2 par la mitigation humaine D109. Scope élargi vs TODO-5 initial : 4 sections au lieu des 3 prévues, le diagnostic Phase 4 est devenu le cœur du document plutôt qu'une section annexe.Anticipe D28 (
docs/IaC-driven, livrable de soutenance final) —docs/runbook.mdrejointdocs/setup.mdexistant comme deuxième artefact du dossier.Décisions techniques
docs/runbook.md(versionné, livrable D28) plutôt que~/explain/(hors repo) — un runbook est par nature un livrable opérationnel partageablejqvspython3 -cjqpartout, idiomatique pour le manipulation JSON CLI. Plus lisible que les heredocs Python utilisés pendant le diagnostic d'aujourd'huidescribe,list,read,query). Aucune commande destructive ou modifianteChangements
docs/runbook.mdcréé (250 lignes), 4 sections :gcloud scheduler jobs describechaque lundi après-midigcloud ... --format=json | jq, incluant le piège Knative v1 sur Cloud Run Jobs v2Validation
jqsyntaxes vérifiées localement.Décisions liées
docs/IaC-driven (anticipe le livrable soutenance)Refs : TODO-5 absorbé (et étendu), TODO-4 et TODO-8 mentionnés comme extensions futures dans la section "Alertes non couvertes".