Running

Schedules, health and notifications

Suites that run by themselves, blocking or informational; the health page; Slack and email.

Suites that run by themselves, a page that says how the app is doing, and a channel that hears when something breaks. Together they are what a QA lead reads on Monday morning instead of asking.

Blocking and informational suites

A suite in testedok.yaml blocks by default: a failed flow fails the check on the commit and is announced. Mark a suite blocking: false and it becomes informational: its failures are still listed in the check and on the run page, but the check turns neutral rather than red, and nobody is paged.

testedok.yaml
suites:
  smoke: { tags: [smoke] }
  nightly: { tags: all, blocking: false }

Use it for a suite that is new, slow, or covers a part of the app that is changing every day. Promote it to blocking when it has been green for a week.

Schedules

A schedule runs a suite on a cron, against an environment, without anyone typing. Add one on the Schedules page: a name, the suite, the environment (optional), a cron expression and its timezone.

The API cannot drive a simulator, so a schedule only makes a job. A machine of yours runs it: in the repository, on a Mac that stays on (a Mac mini under a desk, a runner that is idle at night), sign in and start the agent.

testedok login
testedok agent --label mac-mini --pull

The agent waits for jobs, runs each one as testedok run --suite <suite> --env <environment> in a child process, and reports the result. --pull runs git pull --ff-only before each job, so the suite that runs is the one on the branch. --label names the agent on the Schedules page; --project picks the project when the account has several. It keeps going until stopped; a launchd plist or a screen session is enough to keep it up.

Two agents on the same project share the queue: a job goes to whichever asks first, never to both. A job nobody claims within a day expires; a job whose agent vanished mid-run is marked failed after three hours, so the schedule goes on.

Runs from a schedule show Schedule as their trigger and carry the suite and environment like any other.

Health

The Health page is the app in the numbers a lead asks for first:

  • flows passing on their last run, out of all flows;
  • pass rate and number of runs over 7, 30 or 90 days;
  • median run time;
  • flaky flows: ones that both passed and failed on the same commit in the last two weeks — a flow that only fails is broken, not flaky;
  • the flows failing most, and the last run per environment.

The same numbers are one line to paste into a channel, from the page's Copy button or the CLI:

$ testedok status
Rote: 47/48 passing, 1 flaky, 6.2s median

GET /v1/apps/<id>/health returns them as JSON for a dashboard of your own.

Notifications

Under App settings → Notifications, add a Slack incoming webhook or an email address. Each hears about:

  • every blocking flow that fails or errors, with the reason and a link to the run — one message per run, as it finishes;
  • a digest when a scheduled suite finishes, pass or fail, if the schedule asks for one.

A passing manual run tells nobody. An informational suite's failures tell nobody either; they are on the Health page and in the check.