[VC-59470] Start only the matching e2e suite when its label is added

[wallrj-cyberark]

Adding the test-e2e, test-ark or test-ngts label to an open pull request does not start a run. This makes it start one — and only the suite you asked for.

Why now?

I hit this on #835. I added the test-e2e label, nothing happened, and I had to force-push to fire a synchronize before the e2e would start. It looks exactly like a broken job.

The cause is that tests.yaml uses on: pull_request: {}, which takes the default activity types.

By default, a workflow only runs when a pull_request event's activity type is opened, synchronize, or reopened.

— Events that trigger workflows

labeled is not in that list, so the label sits there and no run starts. Re-running does not help either, because a re-run replays the original event payload, which had no labels.

So the label route documented in #833 has always needed a push afterwards. #833 gave us a reliable manual route with workflow_dispatch, which covers the release case. This fixes the pull request case.

What changes

The three e2e jobs move into a new .github/workflows/e2e.yaml, which subscribes to labeled as well as the default activity types. Each one runs only if github.event.label.name — the single label that was just added — is its own label. Adding test-ngts does not start the ark or GKE suites.

tests.yaml keeps verify and test, keeps the default activity types, and needs no job-level guards.

Why a separate file, rather than adding labeled to tests.yaml? Because a skipped job still publishes a check run, and GitHub shows the most recent one per job name. A label event in a workflow containing verify and test replaces their green results with skipped. Splitting means a label event never creates a check run for them at all. Thanks to @FelixPhipps for catching this — an earlier revision of this pull request had exactly that bug, and you can see it in the review thread below.

Nothing else changes. Push, workflow_dispatch and opened/synchronize/reopened behave as they do today, so a suite whose label is already on the pull request still re-runs on every new commit. The e2e jobs never ran on push, so dropping that trigger from e2e.yaml changes no behaviour.

Action required when you use keep-e2e-cluster

Add keep-e2e-cluster before test-e2e, not after. The cleanup step reads the labels from the event payload, frozen when the run started, and test-e2e now starts the run immediately. Get the order wrong and the cluster you wanted to keep is deleted. This is now written next to the Delete GKE Cluster step, in CONTRIBUTING.md and in README.md.

Full behaviour table — what each event runs, and what it publishes as skipped

Event	Runs	Publishes skipped
push to master	verify, test	—
workflow_dispatch on tests	verify, test	—
workflow_dispatch on e2e	all three suites	—
PR opened / synchronize, no labels	verify, test	the three e2e jobs
PR opened / synchronize, has test-e2e	verify, test, test-e2e	ark-test-e2e, ngts-test-e2e
label kind/cleanup added	nothing	the three e2e jobs
label test-e2e added	test-e2e	ark-test-e2e, ngts-test-e2e
label test-ngts added, test-e2e already on	ngts-test-e2e	ark-test-e2e, test-e2e
label keep-e2e-cluster added	nothing	the three e2e jobs

verify and test never appear in the right-hand column. That is the property the split buys.

There is a residual in the last three rows: a label event still republishes the two non-matching e2e jobs as skipped, so it can overwrite an earlier green test-e2e. Eliminating that entirely would need one workflow file per suite, which is three copies of the same boilerplate. Since skipped is the normal resting state for an opt-in e2e job, that did not seem worth it. Say if you disagree.

Verified live: a label no longer touches the verify and test results

Evidence is in the comments below. The short version, all on commit 0b4c90c:

• Pushing ran tests (verify, test both green) and e2e (all three skipped).
• Adding kind/cleanup created an e2e run only. No tests run was created at all, and gh pr checks still reported verify pass and test pass.
• Adding test-ngts started ngts-test-e2e alone, with ark-test-e2e and test-e2e skipped and still no tests run.

What this does not fix

• e2e still never runs automatically on master, on merge or on a schedule. A green master therefore still does not mean the e2e suites passed. The split makes adding a schedule: trigger easier, since there is now a workflow that contains only the e2e jobs.
• ark-test-e2e still carries its TEMPORARY comment about a recurring 400 "conflicting tagging values" error. Whether that flake is still real is unanswered, and it is the thing blocking a nightly run.

Both are tracked separately.

[with Claude]

[wallrj-cyberark]

Superseded. This describes the first revision, where adding any label also re-ran verify and test. That was needless and has been changed: a label event now starts only the e2e suite named by that label. See the current evidence. Kept for history.

---

Verified: the label alone started a run

Adding a label to this pull request started a second workflow run, with no push. That is the behaviour that does not happen on master today.

	Run 1	Run 2
Trigger	pull request opened	label added
Run	34473375778	34473438702
Created	11:49:57Z	11:50:38Z
Head SHA	c83c994	c83c994
Result	success	success

The two runs share the same head SHA, so no commit was pushed between them. Run 2 was created three seconds after I added kind/cleanup at 11:50:35Z. Nothing else could have created it.

I used kind/cleanup rather than an e2e label deliberately, so all three e2e jobs still skipped and no GKE cluster was created. Both runs show verify and test succeeding and ark-test-e2e, ngts-test-e2e and test-e2e skipped, which is correct for a pull request carrying none of the e2e labels.

What this run does not prove

It shows the labeled activity type now triggers the workflow. It does not separately re-prove that the if: gates read the new label, because kind/cleanup is not one of them. That part is unchanged by this pull request: the gates already read github.event.pull_request.labels successfully today, they were just never reached on a label event.

If you would rather see it end to end before merging, say so and I will add test-e2e here, which will run the full GKE suite.

[wallrj-cyberark]

Verified on this pull request

Revised after review feedback: an unrelated label used to re-run verify and test, which was needless. Now a label event starts only the suite named by that label. All four runs below are on this pull request.

#	What I did	Run	Jobs that ran
1	Force-pushed (synchronize)	34487851234 — success	verify, test. Three e2e jobs skipped
2	Added kind/cleanup, no push	34488035695 — skipped	none. All five jobs skipped
3	Added test-ngts, no push	34488147664 — cancelled by me	ngts-test-e2e only. verify, test, ark-test-e2e, test-e2e all skipped
4	Removed a label	no run	unlabeled is not subscribed

Run 1 is the no-regression check: ordinary pushes still behave exactly as before.

Run 2 is the fix you asked for. The direct before/after is on the same pull request and the same action — adding kind/cleanup. Under the previous revision that produced 34473438702, where verify and test both ran to success. Now the whole run reports skipped and no runner is claimed.

Run 3 is the positive half, and the only real proof that github.event.label.name is populated on a labeled event — the GitHub webhook docs do not spell out the per-action payload fields, so I tested it rather than trusting it. Note test-e2e skipped even though ngts-test-e2e ran, which is the "only the matching suite" property.

I cancelled run 3 about 45 seconds in, once the job selection was visible, to avoid a full 30-minute QA run nobody asked for. It had reached the first seconds of make -j ngts-test-e2e; setup steps had completed and no test result was produced. I then removed the test-ngts label so later pushes here do not start the NGTS suite.

One behaviour change to know about

Add keep-e2e-cluster before test-e2e, not after. Adding it afterwards now starts nothing, because it matches no job. Added first, it is already in github.event.pull_request.labels when the test-e2e run starts, so the cleanup step still sees it and keeps the cluster.

Not covered by these runs

No GKE cluster was created at any point, so the test-e2e job and its cleanup step are unexercised here. Its gate is the same expression as the other two, and run 3 shows that expression correctly skipping it.

[FelixPhipps]

Review

The trigger fix itself is correct. I walked every event this workflow subscribes to through all five gates and found no hole: github.event.label is null on push and workflow_dispatch, so .name == '...' is false there rather than erroring, and the action != 'labeled' guard on the contains() clause is what stops one e2e label from starting the other two suites. The behaviour table in the description matches what the expressions actually do, and the evidence runs cover the interesting cases.

One notable finding, and one minor note.

1. A label event replaces the verify / test check runs on the head commit with skipped

Check runs are keyed by job name per commit, and GitHub surfaces the most recent one under each name. Since labeled now starts a run of tests.yaml in which verify and test are skipped, the green results from the preceding push run stop being what the pull request reports.

This is live on this pull request right now. For cb858f9:

Run	Trigger	verify	test
34487851234 14:15Z	synchronize	success	success
34488035695 14:17Z	labeled	skipped	skipped
34488147664 14:18Z	labeled	skipped	skipped

gh pr checks 837 currently reports verify skipping and test skipping. The unit tests did pass on this commit, but nothing in the checks list says so any more.

Merging should not be blocked by this — GitHub counts a job-level if: skip as satisfying a required status check. The cost is the signal, not the gate: "skipped because somebody added a label" is now indistinguishable from "never ran", both to a reviewer glancing at the checks list and to anything automated that reads conclusions off the head SHA. That is a bit more than the "line in the Actions list" the description prices it at, and it lands on every pull request in the repo, including ones where the label had nothing to do with e2e.

The clean fix is to not subscribe tests.yaml to labeled at all: move the three e2e jobs into their own workflow file taking [opened, synchronize, reopened, labeled], and leave tests.yaml on the default activity types. A label event then creates check runs only for the e2e jobs and never touches verify or test, which also makes the two if: github.event.action != 'labeled' guards unnecessary. That is a larger change than this one, so whether it belongs here or in a follow-up is your call — as it stands this is still a clear improvement on a label doing nothing at all.

2. Minor: the keep-e2e-cluster ordering rule only exists in this description

Adding test-e2e now starts the GKE run immediately, so the window to add keep-e2e-cluster first is gone the moment you apply the label — and getting the order wrong silently deletes the cluster you wanted to keep. Worth putting that sentence in a comment next to the Delete GKE Cluster step, where somebody debugging will actually encounter it, rather than leaving it in a merged pull request description.

[Generated with claude code]

[wallrj-cyberark]

@FelixPhipps thank you, finding 1 was right and I have taken your suggested fix. Both points are addressed in 0b4c90c.

1. Check runs — fixed by splitting the workflow

I confirmed your finding before acting on it. gh pr checks 837 did report verify skipping and test skipping, even though both had passed on that commit minutes earlier. My description priced this as "a line in the Actions list", which was wrong: the cost is the reported state of the pull request, on every pull request in the repo.

I have done what you suggested rather than leaving it as a follow-up. The three e2e jobs now live in .github/workflows/e2e.yaml, which takes [opened, synchronize, reopened, labeled]. tests.yaml keeps verify and test on the default activity types, and the two if: github.event.action != 'labeled' guards are gone, exactly as you predicted.

Verified live on 0b4c90c:

#	What I did	Result
1	pushed	tests 35341615649 green, e2e 35341615700 all three skipped
2	added kind/cleanup, no push	e2e 35341920797 only. No tests run created. gh pr checks still verify pass, test pass
3	added test-ngts, no push	35342017802: ngts-test-e2e alone, other two skipped, still no tests run

Step 2 is the direct before/after against the state you documented.

One residual I should name rather than let you find: a label event still republishes the two non-matching e2e jobs as skipped, so adding test-ngts can overwrite an earlier green test-e2e on the same commit. Same mechanism, confined to the e2e jobs. Removing it completely needs one workflow file per suite, which is three copies of the boilerplate. skipped is the resting state for an opt-in e2e job, so I judged that not worth it — happy to be told otherwise.

I cancelled run 3 once the job selection was visible, to avoid a 30 minute QA run, then re-ran 35341615700 so the checks list is not left showing a cancelled job.

2. keep-e2e-cluster ordering — written down in three places

Agreed, and it was worse than a documentation gap: the ordering rule only existed in a pull request description that would have been invisible after merge. It is now a comment on the Delete GKE Cluster step, and in CONTRIBUTING.md and README.md next to the existing keep-e2e-cluster text.

I considered making the cleanup step read the labels from the API at cleanup time instead of the frozen payload, which would let you add the label while the run is in flight. I left it alone. That step is the only thing between us and a leaked GKE cluster, and a failed API call there has to fail safe. Worth doing separately if the ordering trips anyone up.

Also in this push

Rebased onto master for #838, and updated RELEASE.md. Step 3 told the releaser to check the tests workflow on master; the e2e suites are no longer in it, so it now also points at the Run workflow button on the e2e workflow, which is the route #833 added.

[FelixPhipps]

Both points addressed — thanks for taking the split rather than deferring it.

Agreed on the residual: republishing non-matching e2e jobs as skipped is the same mechanism confined to opt-in jobs whose resting state is skipped, and three copies of the boilerplate is the wrong trade. Leaving the cleanup step reading the frozen payload is also right — fail-safe matters more there than convenience.

LGTM!