Skip to content

test: verify the test-merge base resolution in real GitHub Actions - #21

Open
gregberge wants to merge 10 commits into
mainfrom
test/merge-base-probe
Open

test: verify the test-merge base resolution in real GitHub Actions#21
gregberge wants to merge 10 commits into
mainfrom
test/merge-base-probe

Conversation

@gregberge

@gregberge gregberge commented Sep 4, 2026

Copy link
Copy Markdown
Member

Real-condition test for the base-commit resolution added in argos-ci/argos-javascript#371, and the repro for the "older baseline" bug reported on top of it.

Setup

commit
fork point (merge base — the pre-#371 behaviour) e5ba9c18
main tip (the commit GitHub merges in) 78fe66e7
pull request head this branch

main was advanced twice after this branch forked, so the merge base and the
commit GitHub merges in are two different commits — an unrebased branch.

What runs

Nothing here re-implements the resolver. The real source jobs check out
argos-ci/argos-javascript, bundle packages/core/src/ci-environment and
find-reference-commit.ts with esbuild, and call the real functions. The
expected value is read from the GitHub API by the workflow, so the assertions do
not depend on the code under test. The probe jobs run an instrumented port
that names the guard that fired, across every realistic checkout configuration.

#371 itself works

@argos-ci/core Found base commit from the test-merge commit 78fe66e7…
REAL_RESOLVED=78fe66e79fba612ed45dbe869253274b16f9c9a3
EXPECTED_BASE=78fe66e79fba612ed45dbe869253274b16f9c9a3
PASS

Same result on @v6/@v4 defaults, fetch-depth: 0, fetch-depth: 50 and
persist-credentials: false. The only configuration that falls back is
ref: refs/pull/N/head, which is correct: those screenshots don't contain the
base branch changes, so the merge base is the right baseline for them.

The whole client-side resolution is sound too — with /baseline stubbed to find
nothing, the CLI hands the server referenceCommit = 78fe66e7 plus all 36
ancestors, fork point included.

The actual bug: a stale head.sha sends the build back to the fork point

getTestMergeBaseCommitSha() recognises GitHub's test-merge commit by comparing
its second parent against pull_request.head.sha from the event payload.
That payload can lag behind the merge ref the runner checked out — this very run
shows base.sha reporting c0f449d3 while main was already at 78fe66e7.

When it lags, a genuine test-merge build is rejected and silently falls back to
the merge base:

@argos-ci/core 06b875ec… is not a test-merge commit
resolved with a stale payload : e5ba9c18   ← the fork point
the commit GitHub merged in   : 78fe66e7

No orphan, just an older baseline — the branch is treated as if it had never
been merged with main.

The fix

argos-ci/argos-javascript@fix/test-merge-ref identifies the test merge from
GITHUB_REF (refs/pull/<n>/merge), which the runner sets from the ref the run
was triggered on and which cannot go stale. The payload comparison stays as the
fallback when the merge ref is absent.

Both branches run side by side here:

main fix/test-merge-ref
normal payload 78fe66e7 78fe66e7
stale head.sha e5ba9c18 78fe66e7

🤖 Generated with Claude Code

gregberge and others added 7 commits September 4, 2026 14:06
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Bundles packages/core/src/ci-environment straight from the argos-javascript source with esbuild, so the probe exercises the shipped code and not a re-implementation of it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The expected base is read from the API by the workflow, so the assertion does not depend on the code under test.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@gregberge gregberge changed the title test: probe test-merge base resolution test: verify the test-merge base resolution in real GitHub Actions Sep 4, 2026
gregberge and others added 3 commits September 4, 2026 14:59
Stubs only the /baseline API so the candidate commits the CLI offers the server are visible, and asserts the fork point is among them.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant