Skip to content

ci: skip the test jobs on documentation-only pull requests - #15

Merged
jdsika merged 1 commit into
integration/3.0from
feature/ci-skip-doc-only
Nov 11, 2026
Merged

jdsika merged 1 commit into
integration/3.0from
feature/ci-skip-doc-only

Conversation

@jdsika

@jdsika jdsika commented Nov 11, 2026

Copy link
Copy Markdown

What

Add a fast changes job to ci.yml that diffs a pull request against its merge base, and gate the
five heavy jobs — Unit Tests, C Drive Tests, Notebook Execution, Smoke Train Test, Smoke Performance
Test — on it, so a documentation-only pull request skips them.

Why

Every pull request runs roughly 12 minutes of compilation and simulation, including a ~15 minute
smoke train, regardless of what it changed. A large share of the pull requests this fork actually
opens touch nothing but Markdown: process documentation, contributing guidelines, the PR template.
Rebuilding the C extension and running a training smoke test to validate a typo fix costs runner
minutes and wall-clock review latency for zero information.

Why a changes job rather than paths-ignore

Workflow-level paths-ignore is the obvious approach and it is a trap here. A required check that
is filtered out at workflow level never reports at all, so branch protection leaves it pending
forever
and the pull request can never be merged. A job-level if: that evaluates false reports
skipped, which branch protection treats as passing. Hence the needs:/if: shape.

No third-party action is used — the filter is git diff --name-only against the merge base.

Why it is conservative

Only *.md, docs/ and .github/ISSUE_TEMPLATE/ vote "no code". Anything unrecognised means the
full suite runs. Push builds have no base ref to diff against and always run everything. The
failure mode is therefore "ran tests we did not need", never "skipped tests we did need".

The || true on the grep -Ev is load-bearing: grep -v exits 1 when it filters everything out,
which is precisely the documentation-only case, and would otherwise trip set -e.

How tested

  • pre-commit run --all-files
  • make rebuild (required if .c/.h changed — zero new warnings)
  • make test-unit / make test-eval
  • make test-c (C sim logic)
  • make test-notebooks (notebooks touched)
  • make test-docker-smoke (dynamics, rewards, observations, or training loop touched)
  • make -C tests/drive perf (hot paths touched)
  • make test before marking ready (unit + eval + c + notebooks — does not include
    docker-smoke or perf; those are the two boxes above)

This commit changes only .github/workflows/ci.yml; no Python, C, notebook or config behaviour is
touched, so the suites above have nothing to say about it. §4.8's rule for a changed workflow
applies instead: the workflow has to have run, in each state it is supposed to distinguish.

The YAML was parsed to confirm the pull_request branches: filter — which carries the fork-only
integration/3.0 entry — is untouched by the edit.

code=true path: this pull request itself. It changes .github/workflows/ci.yml, which is not
a documentation path, so the changes job must print the file and report code=true, and all five
heavy jobs must run normally. See the checks on this PR.

code=false path: cannot be exercised by this PR, because this PR is a code change by
construction. A throwaway documentation-only pull request was opened against this branch to drive
the other branch of the condition, and linked in a comment below. Expected: changes prints
Documentation only: skipping the test jobs. and the five jobs report skipped, not pending.

New behaviour is covered by a new test / the fix has a test that fails without it:

No unit test — the behaviour is a GitHub Actions condition and the evidence is the runs above.

Stack hygiene

  • forkctl check-commit passes (--chore on a fork-only branch)
  • Conventional Commits subject, rebased on current integration/3.0
== check-commit
  PASS  exactly one commit on the branch
  PASS  rebased directly onto origin/integration/3.0
  PASS  subject is well formed: 'ci: skip the test jobs on documentation-only pull requests'
  PASS  fork-only markers are well formed
  PASS  fork-only state is intact
  PASS  no registered fork-only file in the diff
  PASS  fork-only branch filters are untouched in shared workflows
  PASS  fork-only blocks in shared files are unchanged
== check-commit: OK

Merge gate

  • Gate 1 — CI green on the final commit (no "green except a flake")
  • Gate 2 — adversarial review posted by someone other than the author; findings fixed or
    explicitly accepted in the thread
  • Gate 3 — upstream-readiness proven: forkctl mirror <slug> --sha HEAD passes, i.e. the
    commit cherry-picks onto pure upstream/3.0 and its diff is upstream-clean

Gate 3 result:

== mirror/ci-skip-doc-only
        feature commit f811a53a3  ci: skip the test jobs on documentation-only pull requests
  PASS  cherry-picks onto pure upstream without conflict
  PASS  no registered fork-only file in the mirror diff
  PASS  no fork-only marker added anywhere in the mirror diff
        mirror diff touches 1 file(s):
          .github/workflows/ci.yml
== mirror/ci-skip-doc-only: OK

The cherry-pick keeping the fork-only integration/3.0 branch filter out of the mirror is the
specific thing Gate 3 is checking here, and it passes.

Upstream disposition

  • Mirror PR to Emerge-Lab/PufferDrive requested now — needs a human to verify the final
    title, body and diff first, and the body must carry the authorship disclosure from
    .github/fork/PROCESS.md → Agent authorship
  • Contributable, but we hold it for now, because: it applies cleanly to pure upstream/3.0 and
    upstream pays the same runner cost, but CI policy is a maintainer preference rather than a
    defect, so it should be offered deliberately and with human approval (§5).
  • Never upstream — fork-only chore(fork): change, Gate 3 exempt

Risk / notes

Branch protection interaction. The five gated jobs are required checks. Skipped required checks
satisfy branch protection; pending ones do not. If someone later converts this to workflow-level
paths-ignore, documentation PRs become unmergeable. Worth a comment in the file if it ever gets
rewritten.

fetch-depth: 0 is needed on the changes job so git merge-base has history to work with.

Deliberately out of scope: install.yml's six-job matrix is not filtered. It is cheaper than
the test matrix and filtering it is a separate judgement about how much install coverage a docs PR
deserves. Flagged as a follow-up, not done here.

Not filtered either: pre-commit, which must keep running on documentation — it is what checks
trailing whitespace and end-of-file in Markdown.

@jdsika

jdsika commented Nov 11, 2026

Copy link
Copy Markdown
Author

Probe run proving the skip path, as promised in How tested.

A throwaway documentation-only PR against a base carrying this change (#16) produced run 43017949338:

  • Detect code changes passed and logged Documentation only: skipping the test jobs. (code=false)
  • Unit Tests, Notebook Execution, C Drive Tests, Smoke Train Test and Smoke Performance Test all reported skipped, not pending — which is what branch protection needs in order to still allow the merge.

The positive path is proven by this PR's own run: it changes ci.yml, so code=true and all five jobs executed normally.

The probe PR and its probe/ci-base helper branch have been closed and deleted.

@jdsika

jdsika commented Nov 11, 2026

Copy link
Copy Markdown
Author

Gate 1 note — the two red checks are not this PR's.

Unit Tests and Notebook Execution fail here on test_drive_mean_action_is_exact_in_physical_units with ImportError: fused_slot_encoder=true requires triton. That is a repository-wide defect inherited from upstream (fused_slot_encoder: true in the shipped config, upstream PR Emerge-Lab#601) and it is red on every open PR, including ones that touch only documentation.

It is fixed by #13, which is now 12/12 green. Once #13 is stacked and this branch is rebased, Gate 1 can be satisfied here.

A README or docstring change currently runs the full matrix -- unit, C, notebooks, a 30-minute Docker smoke train and the perf run -- none of which can observe a Markdown edit. On a fork whose process documentation changes often this is the single largest source of runner time, and it is pure waste.

A changes job diffs the PR against the merge base and votes code=false only when every changed path is recognised documentation (*.md, docs/, .github/ISSUE_TEMPLATE/). Anything unrecognised, and every push build, votes 	rue, so the failure direction is always 'run the tests'. The five heavy jobs gain
eeds: changes and an if:.

Job-level if: rather than workflow-level paths-ignore is deliberate: a workflow skipped by path filtering leaves its required checks pending forever and blocks the merge, whereas a skipped job is reported to branch protection as passing. No third-party action is used; the detection is five lines of git.
@jdsika
jdsika force-pushed the feature/ci-skip-doc-only branch from 7a62d7b to 290ac54 Compare November 11, 2026 19:34
@jdsika

jdsika commented Nov 11, 2026

Copy link
Copy Markdown
Author

Both blocking findings from the Track B review are fixed in 290ac54a8.

1. Rename detection could hide a deleted code path

git diff --name-only has rename detection on by default and reports only the destination, so moving a module under docs/ erased the evidence that executable code was removed. Added --no-renames.

Reproduced the hole and the fix by running the detector's own filter logic against synthetic histories:

case with --no-renames without (old behaviour)
docs-only code=false code=false
markdown + python code=true code=true
rename code → docs/ code=true code=false ← the hole
rename code → .md code=true code=false ← the hole
workflow edit code=true code=true
delete code code=true code=true

2. A failing detector skipped every required job

if: needs.changes.outputs.code == 'true' skips the dependent job whenever changes does not succeed. GitHub reports a skipped job as satisfying its required check, so a checkout, fetch or merge-base failure would have produced five green required checks with nothing executed. The detector itself is not a required context, so nothing else would have blocked the merge.

Now, on all five jobs:

if: always() && !(needs.changes.result == 'success' && needs.changes.outputs.code == 'false')

Tests are skipped only when the detector ran to success and positively reported code=false. Any failure, cancellation or missing output runs the full suite — the cost is time, never coverage.

This also settles the skip-state evidence question. The new condition's skip set is a strict subset of the old one: both skip exactly when changes succeeded with code=false, and the new one additionally refuses to skip in every other case. So the documentation-only skip already demonstrated on run 43017949338 (all five jobs skipped) holds unchanged, while the failure modes that were silently skipping now run.

The "tests run" state is demonstrated by this PR's own checks: it touches .github/workflows/ci.yml, which is not a documentation path, so all five heavy jobs execute on every push to it.

yaml.safe_load parses the result and reports the expression on each of the five jobs.

@jdsika jdsika left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Track B re-review — posted on behalf of the same fresh-context reviewer, re-verifying 290ac54a8 after the two blocking findings were fixed.

Verdict: approve

(a) Syntax and coverage

Valid Actions expression syntax — the ! is inside the expression, not a YAML tag, and always() overrides the implicit success condition. Present on all five heavy jobs (lines 62, 105, 144, 184, 200). None missed.

(b) Case analysis

Detector outcome New condition
success, code=false skip
success, code=true run
failure run
cancelled run
skipped run
success, empty/unset output run

The only conditional skip is a successful detector positively reporting false.

(c) The strict-subset argument

Sound, once the old implicit success() is accounted for: always() removes failure-based skipping but does not override the remaining boolean predicate. The reviewer's sharpening is worth recording — subset membership alone does not prove preservation; what proves it is substituting the historical probe's own (success, false) result into the new expression, which still skips. Its documentation edit also remains code=false under --no-renames. So preservation is established logically; the old run is not empirical execution of the amended YAML, and that distinction is accepted deliberately here.

(d) Detector behaviour

--no-renames exposes both the deletion and the addition path, closing the bypass without introducing new under-detection. Copies keep their added destination; gitlink and mode-only changes on non-documentation paths are testable; a genuinely empty tree diff has no change needing tests. Git command failures now hit the fail-safe gate.

Non-blocking residual (line 48): || true suppresses any grep failure, not only the expected exit code 1 of "everything filtered out". An exit code above 1 — a real grep error — would be read as "documentation only". Accepted as-is for now: the pattern is a fixed literal, so exit codes above 1 are unreachable short of an I/O error. Recorded here rather than fixed so it is not rediscovered.

(e) Scope

No growth. Diffing the old and new workflow blobs shows only --no-renames, the five revised conditions, and the comments explaining them. Still one commit, one file.

forkctl track → B.

@jdsika
jdsika merged commit 351531d into integration/3.0 Nov 11, 2026
14 checks passed
@jdsika
jdsika deleted the feature/ci-skip-doc-only branch November 11, 2026 19:49
@jdsika jdsika mentioned this pull request Nov 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant