Running

CI and GitHub

Runs in CI, suites as checks on commits, and comments on pull requests.

TestedOK runs wherever a simulator or emulator runs: a Mac in GitHub Actions, Xcode Cloud, a Mac mini under a desk. The result comes back as a check on the commit and a comment on the pull request.

Running in CI

.github/workflows/testedok.yml
jobs:
  flows:
    runs-on: macos-15
    steps:
      - uses: actions/checkout@v4
      - run: xcodebuild -scheme Rote -destination 'platform=iOS Simulator,name=iPhone 17 Pro' -derivedDataPath build build
      - run: curl -fsSL https://testedok.dev/install | sh
      - run: xcrun simctl boot "iPhone 17 Pro" && xcrun simctl install booted build/Build/Products/Debug-iphonesimulator/Rote.app
      - run: testedok run --suite smoke --env ci --junit report.xml
        env:
          TESTEDOK_API_URL: https://api.testedok.dev
          TESTEDOK_TOKEN: ${{ secrets.TESTEDOK_TOKEN }}
          TESTEDOK_SECRET_STAGING_PASSWORD: ${{ secrets.STAGING_PASSWORD }}

TESTEDOK_TOKEN is an API token from Account settings in the dashboard. Secrets arrive as TESTEDOK_SECRET_<NAME>. The job fails when a flow does, and --junit gives the runner something to show. An environment in testedok.yaml whose app is the built .app path installs it itself.

Where a run came from

The CLI records the commit, repository, branch and pull request with each run: from GitHub Actions, Xcode Cloud, EAS Build, Bitrise, CircleCI or GitLab CI when it runs there, from the checkout elsewhere, or from TESTEDOK_COMMIT, TESTEDOK_REPO, TESTEDOK_BRANCH and TESTEDOK_PR for pipelines that know better. The run page shows them.

The GitHub App

Connect it once per project (App settings → GitHub → Connect GitHub), choosing the repositories. From then on, every run that knows its commit posts a check run on that commit: one check named after the suite, listing every flow that ran on the commit with its result, time and a link to the run, red if any failed. On a pull request the same summary is kept as one comment, edited as runs finish, never one per run.

Checks are what branch protection reads. Make the smoke suite required and a broken flow stops the merge; leave a nightly suite unrequired and it only reports.

A suite marked blocking: false in testedok.yaml is informational: its failures appear in the check's table, marked as such, but the check's conclusion is neutral rather than failure. See Schedules, health and notifications.

Fastlane

The fastlane-plugin-testedok plugin runs a suite against the build a lane just made, installing the CLI if the machine lacks it, and fails the lane when a blocking flow does.

fastlane/Fastfile
lane :smoke do
  build_app(scheme: "Rote", destination: "generic/platform=iOS Simulator", derived_data_path: "build", skip_archive: true, skip_codesigning: true)
  sh("xcrun simctl boot 'iPhone 17 Pro' || true")
  testedok(suite: "smoke", env: "ci", app: "build/Build/Products/Debug-iphonesimulator/Rote.app", junit: "fastlane/test_output/testedok.xml")
end

Options: suite, tag, env, app (the .app or .apk to install; default the environment's), paths, sim, junit, video, project, token (or TESTEDOK_TOKEN), api_url, install_dir, fail_build. Add gem "fastlane-plugin-testedok" to the Gemfile.

Xcode Cloud

Xcode Cloud runs ci_scripts/ci_post_xcodebuild.sh after every build. Ours finds the simulator .app in DerivedData, installs the CLI, boots a simulator and runs the suite; the commit, branch and pull request come from Xcode Cloud's own CI_* variables, so the check and the comment land on GitHub.

mkdir -p ci_scripts && curl -fsSL https://testedok.dev/integrations/xcode-cloud/ci_post_xcodebuild.sh -o ci_scripts/ci_post_xcodebuild.sh && chmod +x ci_scripts/ci_post_xcodebuild.sh

In the workflow's environment set TESTEDOK_TOKEN (secret), TESTEDOK_SUITE (default smoke), TESTEDOK_ENV (optional), TESTEDOK_SIMULATOR (default "iPhone 17 Pro") and any TESTEDOK_SECRET_*. The workflow's build action needs a simulator destination; an archive has no .app to run and the script says so and exits clean.

Expo EAS

On a simulator build profile (eas.json: "ios": { "simulator": true }), an eas-build-on-success hook runs the suite on the build worker:

mkdir -p scripts && curl -fsSL https://testedok.dev/integrations/eas/testedok-eas.sh -o scripts/testedok-eas.sh && chmod +x scripts/testedok-eas.sh
package.json
{ "scripts": { "eas-build-on-success": "sh scripts/testedok-eas.sh" } }

Set TESTEDOK_TOKEN and friends as EAS secrets or in the profile's env. The commit and branch come from EAS_BUILD_GIT_*; add TESTEDOK_REPO (owner/name) and TESTEDOK_PR for the check and the comment.

Bitrise, CircleCI, GitLab

Any macOS runner works with the two lines from the top of this page: install the CLI, testedok run --suite smoke --env ci --app <built .app> --junit report.xml. The CLI reads the commit, branch and pull request from Bitrise (BITRISE_GIT_*), CircleCI (CIRCLE_*) and GitLab CI (CI_COMMIT_SHA, CI_MERGE_REQUEST_IID) on its own; --app installs the build every flow then runs against.