Flaky flows and quarantine
What flaky means, why a flow is flaky, and taking it out of the way while you fix it.
A flow that passes on Monday and fails on Tuesday with nothing changed is worse than one that always fails: it trains the team to ignore red. TestedOK notices, says why, and can take the flow out of the way while you fix it.
What flaky means
A flow is flaky when its result changes with nothing else changing:
- it both passed and failed on the same commit (and environment) in the last two weeks, or
- its last ten runs flipped between pass and fail at least twice.
A flow that only fails is broken, not flaky, and stays red. The Flows page marks flaky flows; the Health page counts them; testedok status lists them.
Why it is flaky
Open a flow and TestedOK compares its failing runs with its passing ones, step by step, and says what it found:
- the step where they part — what passing runs do there, and what the failing runs did instead;
- what was on screen there that a passing run never saw: a permission dialog (grant it in
setup), a screen still loading (timing), an error state (the backend, not the flow), or leftover data from an earlier run; - whether that step took much longer to settle in the failing runs;
- whether the model was unsure there only in failing runs (name the control as the app labels it);
- whether every failure shares an environment or device the passes do not.
They are hints, not a diagnosis, but they are the questions you would ask first.
Quarantine
Quarantine keeps a flow running and reporting — it is still in every suite and schedule, still on the Flows page — but its runs stop blocking:
- the GitHub check marks its failures quarantined and does not turn red;
- nobody is notified;
- no bug is filed.
Quarantine a flow from its page (owners and admins), with a reason if you like, and release it there when it is fixed.
Automatic
Under App settings → Quarantine flaky flows, TestedOK does it itself: when a failing run makes a flow flaky by the definition above, the flow is quarantined with the hints, and the project's Slack or email hears. A flow TestedOK quarantined is released after five straight passes, with a note. A flow you quarantined by hand stays until you release it.
It is off by default: a new project should see every red until the team knows its flows. Turn it on once the suite is established and a flaky flow costs more attention than it deserves.