Running

Flaky flows and quarantine

What flaky means, why a flow is flaky, and taking it out of the way while you fix it.

A flow that passes on Monday and fails on Tuesday with nothing changed is worse than one that always fails: it trains the team to ignore red. TestedOK notices, says why, and can take the flow out of the way while you fix it.

What flaky means

A flow is flaky when its result changes with nothing else changing:

  • it both passed and failed on the same commit (and environment) in the last two weeks, or
  • its last ten runs flipped between pass and fail at least twice.

A flow that only fails is broken, not flaky, and stays red. The Flows page marks flaky flows; the Health page counts them; testedok status lists them.

Why it is flaky

Open a flow and TestedOK compares its failing runs with its passing ones, step by step, and says what it found:

  • the step where they part — what passing runs do there, and what the failing runs did instead;
  • what was on screen there that a passing run never saw: a permission dialog (grant it in setup), a screen still loading (timing), an error state (the backend, not the flow), or leftover data from an earlier run;
  • whether that step took much longer to settle in the failing runs;
  • whether the model was unsure there only in failing runs (name the control as the app labels it);
  • whether every failure shares an environment or device the passes do not.

They are hints, not a diagnosis, but they are the questions you would ask first.

Quarantine

Quarantine keeps a flow running and reporting — it is still in every suite and schedule, still on the Flows page — but its runs stop blocking:

  • the GitHub check marks its failures quarantined and does not turn red;
  • nobody is notified;
  • no bug is filed.

Quarantine a flow from its page (owners and admins), with a reason if you like, and release it there when it is fixed.

Automatic

Under App settings → Quarantine flaky flows, TestedOK does it itself: when a failing run makes a flow flaky by the definition above, the flow is quarantined with the hints, and the project's Slack or email hears. A flow TestedOK quarantined is released after five straight passes, with a note. A flow you quarantined by hand stays until you release it.

It is off by default: a new project should see every red until the team knows its flows. Turn it on once the suite is established and a flaky flow costs more attention than it deserves.