chore(ci): remove the firestore flake probe workflow - #795
Conversation
The probe was added in FirebaseExtended#780 as a temporary measurement tool for FirebaseExtended#776, which is now closed. It has no callers: nothing outside the file referenced it, and it was workflow_dispatch only, so removing it changes no scheduled or PR-triggered behaviour. FirebaseExtended#783's removal conditions were written when the @grpc/grpc-js override was still the candidate fix. That override was measured and did not work (5/60 vs 4/60, Fisher p = 1.0), so it never landed. The fix was FirebaseExtended#791's timeout ceilings instead, and the probe verified it: 0/60 flakes against an 11/60 baseline, with the desync rate statistically unchanged. Closes FirebaseExtended#783
armando-navarro
left a comment
There was a problem hiding this comment.
The removal checks out: nothing in the tree or on any other branch references the workflow, and before the artifacts expire I re-derived the run 32061237569 column of your closing table, and the five rescue durations under it, from the raw logs, every number matching to the millisecond.
The override figures are swapped
The body says "the override was measured at 5/60 against a 4/60 baseline". What I found:
- Your own table on #776 has it the other way: baseline (1.9.16) 5/60, override (1.14.4) 4/60.
- The conclusion holds in either direction, p = 1.0 both ways, so nothing about the argument changes.
- The commit message carries the same figures in a sentence whose subject is "That override", so they read override-first there too, and a squash carries that into history.
It may be worth rewording both, or catching the commit-message half in the squash editor at merge.
The raw evidence deletes itself on 2026-08-24
- The probe uploads artifacts with a 7-day retention, so the 08-11 baseline set is already gone (it lapsed on the 18th) and run 32061237569's two sets expire on the 24th.
- The runs' job logs keep the headline counts beyond that, but the per-iteration output behind the finer numbers (the rescue durations, the desync lines) lives only in the artifacts.
- I pulled a copy of the proof run's artifacts while re-deriving the table, so shout if you ever want them.
Attaching them to #776 or a gist before the date would make the record self-supporting, since the workflow that could remake them is the one being deleted.
Optional
- A dispatch-only workflow has to exist on the default branch to run at all (GitHub's docs), so restoring the probe someday means a PR to main, not a checkout. One line in the #783 close-out naming a commit that carries it (23da94e does) would save that future person the discovery.
- The fixed-position repro signature you said would go upstream doesn't seem to have been filed on firebase-tools#8654 yet (its only comment is from another affected user), and now is the cheapest time, while the data behind it is fresh.
Approving. The swap is the only thing I'd want cleaned up, and if I've misread which arm was which, point me at it and I'll strike the whole item.
|
You read it right, and thanks for going to the table rather than taking my sentence for it. #776 has baseline (1.9.16) at 5/60 and override (1.14.4) at 4/60; I had it override-first in both the body and the commit message. Both fixed before merge, so the swap didn't reach history. Conclusion is unmoved, p = 1.0 either way. Yes to the artifacts, please. Run 32061237569's sets expire on the 24th and the workflow that could remake them is now deleted, so your copy is the only one left. I'll attach them to #776 so the closing table stands on its own. Good catch on the dispatch-only constraint. I'll put a line on #783 naming 23da94e as a commit that still carries the workflow. On firebase-tools#8654: I'd parked it on the grounds that the evidence was already public, but you're right that nothing over there points to it, which makes that reason thinner than I'd written it down. Worth revisiting while the data is fresh. |
Why
.github/workflows/flake-probe.yamlwas added in #780 to measure the firestore flake in #776. #776 is now closed, so the probe has done its job.Verified while the file still exists, since afterwards neither claim can be checked without going back to this branch: it is
workflow_dispatchonly, and nothing outside the file references it. Removing it therefore changes no scheduled or PR-triggered behaviour, and it cannot orphan a required status check (mainrequirescla/google,Build,Test Node.js 22 (Ubuntu)andTest Node.js 24 (Ubuntu)).On #783's removal conditions
They were written when the
@grpc/grpc-jsoverride was still the candidate fix, so the checklist asks for the probe to be re-run "if the override lands". It never landed: the override (1.14.4) was measured at 4/60 against a 5/60 baseline (1.9.16), Fisher p = 1.0.The conditions are met by a different route. The fix was #791's timeout ceilings, and the probe is what verified it: 0/60 flakes against an 11/60 baseline (p = 0.0006), with the desync rate statistically unchanged at 13/60 against 19/60. Five iterations show the rescue directly, passing at ~9.8s where the old 1000ms
waitForbudget gave up. Full result on #776.Closes #783