Fixtures

Data checks

After the screen says it worked, ask the backend: HTTP and shell checks with the run's own values.

"Order placed" on the screen is what the app claims. Whether the order exists is a question for the backend. A flow's verify list asks it, after the UI part passed, and fails the run when the answer is wrong.

tests/checkout.json
{
  "name": "Shop: place an order",
  "app": { "bundleId": "com.example.shop" },
  "goal": "Add \"Espresso Beans\" to the cart, check out, and pay with the saved card. Use \"Test order {{run}}\" as the note.",
  "expectText": ["Order placed"],
  "verify": [
    {
      "name": "the order is in the API",
      "http": "GET {{ vars.api }}/orders?note=Test%20order%20{{run}}",
      "headers": { "authorization": "Bearer {{ secrets.api_key }}" },
      "expect": { "status": 200, "json": { "items[0].status": "placed", "items[0].total": 12.5 } },
      "retry": 30
    },
    {
      "name": "and in the database",
      "shell": "psql \"$STAGING_DB\" -Atc \"select count(*) from orders where note = 'Test order {{run}}'\"",
      "expect": { "equals": "1" }
    }
  ]
}

Where checks run

On the machine running the flow, after the last UI step passed — the machine that can reach your staging API or database, holding the secrets TestedOK's API never sees. Secrets are filled in only there and masked in what is reported. The results go with the run: each check's name, whether it held, what it got, and how long it took, on the run page and in the reason when one failed.

A run whose UI part failed does not run its checks; there is nothing to verify.

A check

FieldMeaning
http"[METHOD] URL", GET by default. headers and body (a string, or JSON) as needed.
shellA command for sh -c; its output is the response, its exit code the status. Runs in the flow's working directory with your environment.
expectWhat the response must satisfy (below). Nothing stated means "it answered": a 2xx/3xx, or exit 0.
retryKeep trying for this many seconds until it passes — for a backend that catches up after the app says done.
timeoutSeconds before one attempt is given up (default 20).
nameWhat the check is called in the report.

Expectations

FieldHolds when
statusthe HTTP status (or exit code) is exactly this
equalsthe whole response, trimmed, is exactly this
containsthe response contains this text
regexthe response matches this regular expression
jsoneach path into the JSON response holds its value: { "items[0].status": "placed", "count": 1 }. Numbers and their strings agree; objects and arrays compare as JSON.

Every stated expectation must hold. Placeholders are the flow's own: {{run}} (the run's tag, so leftovers from earlier runs never satisfy a check), {{ vars.x }} from the environment, {{ secrets.x }}, {{ account.role }} and {{ inbox.new }}.

Why not assert in the UI

The screen shows what the app decided to show. A check reads what the system of record holds: the row, the webhook that fired, the email that went out. It is also the only way to assert on what the UI does not show — the price that was charged, the flag that was set, the event that was logged.