Schedules, health and notifications
Suites that run by themselves, blocking or informational; the health page; Slack and email.
Suites that run by themselves, a page that says how the app is doing, and a channel that hears when something breaks. Together they are what a QA lead reads on Monday morning instead of asking.
Blocking and informational suites
A suite in testedok.yaml blocks by default: a failed flow fails the check on the commit and is announced. Mark a suite blocking: false and it becomes informational: its failures are still listed in the check and on the run page, but the check turns neutral rather than red, and nobody is paged.
suites:
smoke: { tags: [smoke] }
nightly: { tags: all, blocking: false }Use it for a suite that is new, slow, or covers a part of the app that is changing every day. Promote it to blocking when it has been green for a week.
Schedules
A schedule runs a suite on a cron, against an environment, without anyone typing. Add one on the Schedules page: a name, the suite, the environment (optional), a cron expression and its timezone.
The API cannot drive a simulator, so a schedule only makes a job. A machine of yours runs it: in the repository, on a Mac that stays on (a Mac mini under a desk, a runner that is idle at night), sign in and start the agent.
testedok login
testedok agent --label mac-mini --pullThe agent waits for jobs, runs each one as testedok run --suite <suite> --env <environment> in a child process, and reports the result. --pull runs git pull --ff-only before each job, so the suite that runs is the one on the branch. --label names the agent on the Schedules page; --project picks the project when the account has several. It keeps going until stopped; a launchd plist or a screen session is enough to keep it up.
Two agents on the same project share the queue: a job goes to whichever asks first, never to both. A job nobody claims within a day expires; a job whose agent vanished mid-run is marked failed after three hours, so the schedule goes on.
Runs from a schedule show Schedule as their trigger and carry the suite and environment like any other.
Health
The Health page is the app in the numbers a lead asks for first:
- flows passing on their last run, out of all flows;
- pass rate and number of runs over 7, 30 or 90 days;
- median run time;
- flaky flows: ones that both passed and failed on the same commit in the last two weeks — a flow that only fails is broken, not flaky;
- the flows failing most, and the last run per environment.
The same numbers are one line to paste into a channel, from the page's Copy button or the CLI:
$ testedok status
Rote: 47/48 passing, 1 flaky, 6.2s medianGET /v1/apps/<id>/health returns them as JSON for a dashboard of your own.
Notifications
Under App settings → Notifications, add a Slack incoming webhook or an email address. Each hears about:
- every blocking flow that fails or errors, with the reason and a link to the run — one message per run, as it finishes;
- a digest when a scheduled suite finishes, pass or fail, if the schedule asks for one.
A passing manual run tells nobody. An informational suite's failures tell nobody either; they are on the Health page and in the check.