Skip to content

Once the night judges a declared list, streak and verdict are reported by nothing #930

Open

Description

@stephrobert

What changes

#926 (issue #830) stops the nightly report from judging jobs that merely happen
to run beside it. runtime-proof.yml declares the four jobs whose status decides
the verdict:

judged-jobs: |
  incus
  incus-ovn
  stacks (incus-ovn)
  stacks (incus-ovn, last release)

That is the right fix for #830. It also means streak and verdict leave the
verdict entirely. Whatever they do, the night reads green, no scheduled-red
issue opens, and an open one is closed by --apply.

Why that is not fine as it stands

These jobs break, and they break quietly. runtime-proof.yml:1034 records it in
its own comment:

This job counted the streak from the Actions API and needed nothing from the
repository, so it checked nothing out, and then #795 added a step that runs
tools/ci/streak-report.sh. Six nights of No such file or directory,
exit 127.

That specific breakage is fixed, by a checkout step held by
TestEveryJobThatRunsARepositoryScriptChecksItOut. The point is the duration:
six nights, in a job whose status was still part of the verdict. With the judged
list, the same class of failure has no path to anyone at all.

So the state after #926 is two jobs with no reporting whatsoever. Before, they
reported by accident, which was the defect of #830. Removing the accident is
right; replacing it with nothing is the untreated half.

What closes this

One of these, decided and written down, not left implicit:

  1. A second verdict for the auxiliary jobs. They do not gate the night's
    runtime proof, but a red one opens or updates an issue of its own, distinct
    from scheduled-red so the two signals stay separable.
  2. Or an explicit declaration that they are unreported, in docs/limits.md,
    naming what is therefore not watched and what a reader should check by hand.

Plus, in either case, the closure check that #926's review asks for: run jobs,
minus judged, minus a declared ignore list, must be empty. Today a job can
leave the verdict by being forgotten. It should only be able to leave it by being
named. That is what turns this from "two jobs we know about" into a property that
survives the next job somebody adds.

What this is not

Not a request to re-judge streak and verdict in the night's verdict. That is
#830, and #926 is right to take them out.

Where it came from

Review of #926. Raised there and moved here rather than left as a paragraph in a
pull request thread, which is not a tracked debt. It is not the contributor's to
fix in that pull request.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions