Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 41 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -115,7 +115,7 @@ Both channels take these:
| `await-completion` | | `true` | Wait for results. `false` starts the run and exits |
| `continue-on-failure` | | `false` | Report failing and blocked checks but always exit 0. Does not cover an incomplete result set |
| `fail-on-blocked` | | `true` | Treat blocked checks as failures |
| `timeout-seconds` | | `7200` | How long to wait. Budget for the whole suite — a 23-check web suite takes about an hour |
| `timeout-seconds` | | `7200` | How long to wait. Budget for the whole suite — a 23-check web suite takes about an hour, and a mobile suite scales down with `max-concurrency` |
| `poll-interval-seconds` | | `15` | Seconds between status polls |
| `results-settle-seconds` | | `60` | How long to re-read results after the session finishes, until the set stops changing |
| `junit-path` | | | Write a JUnit XML report here |
Expand All @@ -142,11 +142,13 @@ Both channels take these:
| `device-type` | | | Narrows auto-selection, e.g. `phone` or `tablet` |
| `os-version` | | | Narrows auto-selection to this OS version |
| `manufacturer` | | | Narrows auto-selection to this manufacturer |
| `device-location` | | | Narrows auto-selection to this datacenter, e.g. `EU` or `US` |
| `device-backend` | | `mobitru` | Device provider |
| `app-binary-id` | | | UUID of an uploaded binary to install before the run |
| `app-package` | | | Package name (Android) or Bundle ID (iOS) of an app already on the device |
| `mobile-browser` | | `false` | Test the device browser instead of an app |
| `prerequisites` | | | Free-form setup notes to run before the checks, e.g. login steps |
| `max-concurrency` | | installation default | How many checks to run at once, each on its own device |

Setting a web input on a mobile run logs a warning and changes nothing — the mobile API takes no URL, browser, viewport, workflow type or replay flag.

Expand All @@ -164,6 +166,25 @@ A mobile run needs `channel: mobile`, a `product-id` for a mobile product, and t

The action rejects zero sources and more than one. The API treats an unset source as "install a binary" and only notices the missing upload at start time, which would leave you a session that cannot run, so it is settled before anything is created.

**How many at a time.** `max-concurrency` runs that many checks in parallel, each on its own device. The installation books a pool of devices when the session starts, runs one check per device, and hands each device to the next waiting check as it frees up, so a 23-check suite does not book 23 devices.

Three things cap it, and the lowest wins:

- the **Device Quota** an admin sets per installation, under System Configuration -> Mobitru
- the number of checks in the suite
- how many matching devices the farm actually has free

Because the quota is a ceiling rather than an override, asking for more than it allows books the quota's worth instead of failing. Asking for more devices than the farm can supply *does* fail, before anything is booked, so a short pool never leaves half a session holding devices.

Leave it unset to use the installation default, which is normally one check at a time.

Two things worth knowing:

- It is ignored with a pinned `device-serial`, which is a pool of exactly one device. The action warns rather than failing.
- The number you ask for is remembered even if the run ends up narrower — a device the app will not install on is swapped out, and if no replacement is free the session simply runs on fewer. A later re-run still asks for the original number.

Concurrency also costs worker capacity on the installation, not just devices: each running check occupies one agent slot for its whole duration. Setting `max-concurrency` above the number of slots books devices that then sit idle waiting their turn, so it is worth checking with whoever runs the installation before going wide.

## Outputs

| Name | Description |
Expand Down Expand Up @@ -302,6 +323,24 @@ Run a mobile suite on an auto-selected Android phone, installing a binary:
app-binary-id: ${{ vars.AGENTIC_QA_APP_BINARY }}
```

Five checks at a time on EU phones, for a suite that would otherwise take hours:

```yaml
- uses: test-IO/agentic-qa-github-action@v1
with:
host: ${{ vars.AGENTIC_QA_HOST }}
token: ${{ secrets.AGENTIC_QA_TOKEN }}
project-id: ${{ vars.AGENTIC_QA_PROJECT }}
check-suite-id: ${{ vars.AGENTIC_QA_MOBILE_SUITE }}
channel: mobile
product-id: ${{ vars.AGENTIC_QA_MOBILE_PRODUCT }}
device-platform: android
device-type: phone
device-location: EU
max-concurrency: 5
app-binary-id: ${{ vars.AGENTIC_QA_APP_BINARY }}
```

Or on one pinned device, against an app that is already installed:

```yaml
Expand Down Expand Up @@ -410,7 +449,7 @@ Fire and forget, for a nightly run you inspect in the UI:

**Listing app binaries needs an owner token.** `GET /products/:id/mobile_binary_files` is owner-only, though `app-binary-id` works with any token that can reach the product. Look the ID up once and store it as a repository variable.

**There is no device-location filter.** The API documents `search_criteria[location]` for picking a datacenter, but does not accept it, so the action does not offer it. Pin a `device-serial` if you need a specific device.
**`device-location` only narrows auto-selection.** It is dropped with a warning alongside a pinned `device-serial`, which already names one device. Older installations may not accept it — the filter was added to the mobile API after the datacenter values themselves were exposed.

**Private installations need a reachable host.** GitHub-hosted runners must be able to open an HTTPS connection to `host`. If your installation sits behind a firewall or VPN, use a self-hosted runner.

Expand Down
10 changes: 10 additions & 0 deletions action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,10 @@ inputs:
description: 'Narrows auto-selection to this manufacturer.'
required: false
default: ''
device-location:
description: 'Narrows auto-selection to devices in this datacenter, e.g. EU or US.'
required: false
default: ''
device-backend:
description: 'Device provider. Defaults to the installation default (mobitru).'
required: false
Expand All @@ -120,6 +124,10 @@ inputs:
description: 'Free-form setup notes to run before the checks, e.g. login steps (mobile only).'
required: false
default: ''
max-concurrency:
description: 'How many checks to run at once, each on its own device (mobile only). Capped by the installation Device Quota and by the number of checks in the suite; the session refuses to start if the device farm cannot supply that many. Ignored when device-serial pins a single device. Defaults to the installation setting.'
required: false
default: ''

results-settle-seconds:
description: 'How long to keep re-reading results after the session finishes, until the set stops changing. Guards against reporting a partial run.'
Expand Down Expand Up @@ -186,8 +194,10 @@ runs:
AQ_DEVICE_TYPE: ${{ inputs.device-type }}
AQ_OS_VERSION: ${{ inputs.os-version }}
AQ_MANUFACTURER: ${{ inputs.manufacturer }}
AQ_DEVICE_LOCATION: ${{ inputs.device-location }}
AQ_DEVICE_BACKEND: ${{ inputs.device-backend }}
AQ_APP_BINARY_ID: ${{ inputs.app-binary-id }}
AQ_APP_PACKAGE: ${{ inputs.app-package }}
AQ_MOBILE_BROWSER: ${{ inputs.mobile-browser }}
AQ_PREREQUISITES: ${{ inputs.prerequisites }}
AQ_MAX_CONCURRENCY: ${{ inputs.max-concurrency }}
18 changes: 18 additions & 0 deletions scripts/run.sh
Original file line number Diff line number Diff line change
Expand Up @@ -146,6 +146,20 @@ else
die "app-package runs an app already installed on one device, so it needs device-serial."
fi

# The API rejects a non-positive value with a 400 after it has already saved the
# setup instruction, leaving an orphan behind, so settle it here first.
if [[ -n "$AQ_MAX_CONCURRENCY" ]]; then
if [[ ! "$AQ_MAX_CONCURRENCY" =~ ^[1-9][0-9]*$ ]]; then
die "max-concurrency must be a positive integer, got '$AQ_MAX_CONCURRENCY'."
fi
if [[ -n "$AQ_DEVICE_SERIAL" ]]; then
echo "::warning::max-concurrency is ignored when device-serial pins one device."
fi
fi
if [[ -n "$AQ_DEVICE_LOCATION" && -n "$AQ_DEVICE_SERIAL" ]]; then
echo "::warning::device-location narrows auto-selection and is ignored when device-serial is set."
fi

ignored_on_mobile() {
echo "::warning::$1 is a web input and is ignored when channel is mobile."
}
Expand Down Expand Up @@ -185,9 +199,11 @@ if [[ "$AQ_CHANNEL" == "mobile" ]]; then
--arg dtype "$AQ_DEVICE_TYPE" \
--arg osver "$AQ_OS_VERSION" \
--arg vendor "$AQ_MANUFACTURER" \
--arg dcloc "$AQ_DEVICE_LOCATION" \
--arg artifact "$AQ_APP_BINARY_ID" \
--arg package "$AQ_APP_PACKAGE" \
--arg prereq "$AQ_PREREQUISITES" \
--arg conc "$AQ_MAX_CONCURRENCY" \
--argjson browser "$browser_json" '
{test_session: (
{name: $name, check_suite_id: $suite, product_id: $product}
Expand All @@ -201,12 +217,14 @@ if [[ "$AQ_CHANNEL" == "mobile" ]]; then
+ (if $dtype != "" then {device_type: $dtype} else {} end)
+ (if $osver != "" then {os_version: $osver} else {} end)
+ (if $vendor != "" then {manufacturer: $vendor} else {} end)
+ (if $dcloc != "" then {location: $dcloc} else {} end)
)}
else {} end)
+ (if $artifact != "" then {selected_artifact_id: $artifact} else {} end)
+ (if $package != "" then {app_package: $package} else {} end)
+ (if $browser then {mobile_browser: true} else {} end)
+ (if $prereq != "" then {prerequisites: $prereq} else {} end)
+ (if $conc != "" then {max_concurrency: ($conc | tonumber)} else {} end)
)}')
else
if is_true "$AQ_USE_REPLAYS"; then replays_json=true; else replays_json=false; fi
Expand Down
16 changes: 16 additions & 0 deletions tests/run_tests.sh
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ run_action() {
AQ_CHANNEL="web" AQ_PRODUCT_ID="" AQ_DEVICE_SERIAL="" AQ_DEVICE_PLATFORM="" \
AQ_DEVICE_TYPE="" AQ_OS_VERSION="" AQ_MANUFACTURER="" AQ_DEVICE_BACKEND="" \
AQ_APP_BINARY_ID="" AQ_APP_PACKAGE="" AQ_MOBILE_BROWSER="false" AQ_PREREQUISITES="" \
AQ_DEVICE_LOCATION="" AQ_MAX_CONCURRENCY="" \
AQ_PROXY_CONFIG_ID="" AQ_RESULTS_SETTLE_SECONDS="30" \
"$@" bash "$ROOT/scripts/run.sh" > "$WORK/log" 2>&1
# shellcheck disable=SC2319 # $? is the run above; `local rc` would reset it
Expand Down Expand Up @@ -151,6 +152,21 @@ assert_file_has "$WORK/payload.json" '"manufacturer": "Google"' "sends the manu
assert_file_has "$WORK/payload.json" '"mobile_browser": true' "sends mobile_browser"
assert_file_has "$WORK/payload.json" '"prerequisites": "log in first"' "sends prerequisites"

run_action green 0 "runs a mobile session several checks at a time" \
"${MOBILE_BASE[@]}" AQ_DEVICE_PLATFORM=android AQ_MOBILE_BROWSER=true \
AQ_MAX_CONCURRENCY=5 AQ_DEVICE_LOCATION=EU
assert_file_has "$WORK/payload.json" '"max_concurrency": 5' "sends max_concurrency as a number"
assert_file_has "$WORK/payload.json" '"location": "EU"' "sends the device location"

run_action green 1 "rejects a non-numeric max-concurrency" \
"${MOBILE_BASE[@]}" AQ_DEVICE_PLATFORM=android AQ_MOBILE_BROWSER=true AQ_MAX_CONCURRENCY=lots
run_action green 1 "rejects a zero max-concurrency" \
"${MOBILE_BASE[@]}" AQ_DEVICE_PLATFORM=android AQ_MOBILE_BROWSER=true AQ_MAX_CONCURRENCY=0

run_action green 0 "warns that max-concurrency is ignored on a pinned device" \
"${MOBILE_BASE[@]}" AQ_DEVICE_SERIAL=ABC123 AQ_MOBILE_BROWSER=true AQ_MAX_CONCURRENCY=5
assert_file_has "$WORK/log" "::warning::max-concurrency is ignored" "warns about max-concurrency on a pinned device"

run_action green 0 "drops search criteria when a serial is pinned" \
"${MOBILE_BASE[@]}" AQ_DEVICE_SERIAL=ABC123 AQ_DEVICE_PLATFORM=android AQ_MOBILE_BROWSER=true
if grep -qF 'search_criteria' "$WORK/payload.json"; then
Expand Down
Loading