Device matrix
The same flows on several simulators and OS versions, results side by side.
The flow that passes on an iPhone 17 Pro may fail on an iPhone SE, where the button sits below the fold, or on iOS 18, where the sheet looks different. The device matrix runs the same flows on several simulators at once and puts the results side by side.
testedok run --suite smoke --devices "iPhone 17 Pro,iPhone SE (3rd generation)@18.5,iPad Pro 13-inch (M4)"3 flows on 3 devices: iPhone 17 Pro (iOS 26.5), iPhone SE (3rd generation) (iOS 18.5), iPad Pro 13-inch (M4) (iOS 26.5)
…
iPhone 17 Pro 26.5 iPhone SE (3rd generation) 18.5 iPad Pro 13-inch (M4) 26.5
Rote: open today pass pass pass
Rote: create pass FAIL pass
Rote: settings pass pass pass
iPhone 17 Pro 26.5: 3/3 passed
iPhone SE (3rd generation) 18.5: 2/3 passed
iPad Pro 13-inch (M4) 26.5: 3/3 passedDevices are named as Xcode names them; @18.5 picks an OS, otherwise the newest installed. A booted simulator of that name is used as is, a shut-down one is booted, and a name Xcode does not have is an error listing what there is (make one under Xcode → Settings → Platforms, or xcrun simctl create). Each device runs every flow, and the devices run at the same time, one driver each; an app missing from a simulator is copied over from one that has it.
Named sets
matrices:
phones: { devices: ["iPhone 17 Pro", "iPhone SE (3rd generation)"] }
everything: { devices: ["iPhone 17 Pro", "iPhone SE (3rd generation)@18.5", "iPad Pro 13-inch (M4)"] }testedok run --suite smoke --matrix phonesReading the grid
The Device matrix page in the dashboard lists every invocation that spanned more than one device, and each one as a grid: flows down, devices across, a run in every cell with its time and a link. A cell that fails on one device only is a layout or a device difference; a row that fails everywhere is the flow or the app. On a pull request the check's table gains a Device column when a commit's runs span devices.
--junit writes one case per flow and device (Rote: create [iPhone SE (3rd generation) 18.5]). --suite, --tag, --env and --app apply as they do to any run.
Limits
Simulators only, for now: an Android matrix needs several emulators, which the driver does not yet run side by side. Flows that lease test accounts or use an inbox are not run in a matrix yet, since each device would need its own; run those with --sim.