Stop extension README media from triggering extension test suites #562

Part of #554, change set C. Rulings: #554 rulings 5, 6, 9, 15 and 16.

Summary

• Today. .github/filters/extensions.yaml keys each extension on its whole folder. A README screenshot or demo GIF change starts that extension's full clean-room suite.
• Measured cost. Push run 34821528307 changed two PNG files and nothing else.
  • Its 🔎 Detect Changes log (job 103904024263) prints Filter extension-hypermultimedia = true and Changes output set to ["extension-hypermultimedia"].
  • The Ext suite ran 193 s. 🏗️ Build Verification waited for it until +220 s, though the other gates had ended by +107 s.
  • That run's Build Verification failed for another reason. Use only its job seconds, not its M6.
• After. README media skips the suite: extensions/<ext>/assets/**, and *.gif, *.png, *.jpg, *.jpeg, *.webp, *.mp4 at an extension's root. README.md, CHANGELOG.md and package.json still run it, because the preflight reads them. Per the [Tracking] Faster, change-aware CI/CD pipeline #554 first shared rule, nothing else is excluded.
• Also: the filter step gets predicate-quantifier: 'some-with-excludes'. _shared gains scripts/check-quickstart-example.ts, .github/workflows/prod.docs.plus.yml and .github/filters/**. scripts/check-ci.ts mirrors all of it.
• The filter is global. It changes (build): deploy pushes too.
  • A media-only (build): front push skips its Ext job. A (build): back push runs no Ext job, as today.
  • A (build): front push that touches the three new _shared entries now waits for all six Ext jobs. Today it runs none. The slowest suite ran 222 s (job 117562677263, cron run 39089384453).
  • Before Start the extension and webapp unit tests without waiting for backend validation #558 lands, that push gains about 200–220 s. Runs 39398067267 (+394 s against +182 s), 37906965365 (+443 s against +241 s) and 37431581531 (+436 s against +219 s) show it.
  • After Start the extension and webapp unit tests without waiting for backend validation #558, 🏗️ Build Verification still waits for 🔧 Backend Validation (+138 to +160 s). The gain falls to about 130–150 s.

Depends on / Blocks

• Depends on: nothing.
• Blocks: Run only the CI suites a pull request or push can affect #563. It adds scopes to the same 🔎 Detect Changes job and relies on this predicate-quantifier line.
• Touches the same job as Start the extension and webapp unit tests without waiting for backend validation #558, which edits the extension-tests needs and if lines. Line numbers may move; Step 0 handles that.

Current behaviour

Every link is pinned to commit d0a53756f8676556358b6adc6c2397d2ed7c64c8.

1. The filter keys each extension on its whole folder, with no exclusions (.github/filters/extensions.yaml:1-43).
   _shared (L5-L22) lacks the three new entries. Each key is - *shared plus 'extensions/<ext>/**' (L24-L43). extension-hypermultimedia also watches 'extensions/extension-hyperlink/**' (L33-L34).
2. The filter step sets no quantifier (prod.docs.plus.yml:146-149):

   - id: filter
     uses: dorny/paths-filter@ceb8a2b8f2d89434be7ff52d3de7ec3738c5cc9d # v4.0.3
     with:
       filters: .github/filters/extensions.yaml

3. workflow_dispatch, schedule and a push without a diff base list all six extensions (prod.docs.plus.yml:157).
4. The matrix is the filter output (prod.docs.plus.yml:272-286). It runs only when needs.changes.outputs.changed_extensions != '[]', then runs the suite and the preflight (:298-306). build accepts a skipped extension-tests (:375).
5. The preflight reads the README (scripts/extension-preflight.sh:16-17) through scripts/check-quickstart-example.ts, which needs a ## Quickstart section (scripts/check-quickstart-example.ts:19-22).
6. No CI suite, build or preflight reads assets/.
   • Specs match cypress/e2e/**/*.cy.ts (cypress.config.ts:7). Gallery specs in cypress/docs/ run only through docs:screenshots.
   • --readme-media-dir ./assets/readme-media (package.json:41) is served only on a /readme-media/* request (launch.ts:69-84). No cypress/e2e spec makes one.
   • flattenReadmeScreenshots writes to assets/ only for readme-gallery.cy.ts (:10-17). The tarball ships dist, CHANGELOG.md and LICENSE only (package.json:53-57).
   • Every .gif, .png, .jpg, .webp and .mp4 file under extensions/ sits in extensions/<ext>/assets/.
7. The cache key (.github/actions/build-extensions/action.yml:34-52) hashes extensions/*/package.json and extensions/*/src/** only.
8. scripts/check-ci.ts mirrors the filter for the Husky pre-push run (:19-42, :98-107). Its touched() counts any file under extensions/<dir>/. AGENTS.md:222 says to change both together.
9. The pinned dorny/paths-filter v4.0.3 supports 'some-with-excludes' (action.yml:47-55). It applies per key, and an exclusion is final (src/filter.ts:133-149; some at :150-151).

Target behaviour

• Push 39283678757 changed mermaid README.md and package.json beside its GIFs. Its mermaid suite (job 118163701213, 194 s) still runs, by design.
• scripts/check-ci.ts picks the same suites for every row.

Implementation steps

Read every file in Read first before Step 0. "Comment here" means a comment on this issue.

Step 0. Check the base

1. Compare the cited files with the pin:

   git diff --stat d0a53756f8676556358b6adc6c2397d2ed7c64c8 HEAD -- .github/filters/extensions.yaml .github/workflows/prod.docs.plus.yml .github/actions/build-extensions/action.yml scripts/check-ci.ts scripts/extension-preflight.sh scripts/check-quickstart-example.ts AGENTS.md

   For each listed file, compare the cited lines with Current behaviour. A moved line is fine; use the new number. If the filter keys, the filter step, SHARED_EXACT, SHARED_PREFIXES or touched() changed in meaning, stop and comment here.
2. This must print one line, with ceb8a2b8f2d89434be7ff52d3de7ec3738c5cc9d # v4.0.3:

   grep -n "dorny/paths-filter" .github/workflows/*.yml

   If the pin changed, read action.yml and src/filter.ts at the new SHA. If some-with-excludes no longer behaves as item 9 says, stop and comment here.
3. Both commands must print nothing:

   git ls-files 'extensions/*' | grep -E '\.(gif|png|jpe?g|webp|mp4)$' | grep -v -E '^extensions/[^/]+/assets/'
   git grep -n -E "assets/|readme-media" -- 'extensions/*/src/*' 'extensions/*/test/*' 'extensions/*/cypress/e2e/*' 'extensions/*/cypress/support/*' 'extensions/*/tsup.config.ts' 'extensions/*/cypress.config.ts' 'packages/release-tooling' 'scripts/run-tests.sh' 'scripts/extension-preflight.sh' 'scripts/build-extensions.sh'

   If the second prints a line, that file reads assets/ in CI. Stop and comment here with the line. Each directory pathspec ends in /*, or git matches no file below it. Check each one: for example, git ls-files -- 'extensions/*/src/*' | wc -l must print more than 0.
4. Confirm the extension keys match scripts/publishable-extensions.ts: extension-indent, extension-hyperlink, extension-hypermultimedia, extension-inline-code, extension-placeholder, extension-mermaid. Give any seventh key the same *readme_media line in Step 1.

Step 1. The filter file

Replace .github/filters/extensions.yaml with this. Keep any key or _shared entry that main gained after the pin.

# Shared inputs for ALL extensions, appended to every key via YAML anchor.
# Mirrored in scripts/check-ci.ts; dist inputs also in build-extensions hashFiles.
# Test-only entries (playground, run-tests.sh, setup-cypress, the quickstart check,
# the workflow, this folder) run the suites but need no cache bust: dist is unchanged.
_shared: &shared
  - 'package.json'
  - 'bun.lock'
  - 'tsconfig.base.json'
  - 'tsup.base.ts'
  - 'packages/floating-popover/**'
  - 'packages/floating-tooltip/**'
  - '.github/actions/build-extensions/**'
  - '.github/actions/setup-cypress/**'
  # Resolves the Bun version and runs the frozen-lockfile install for every ext job.
  - '.github/actions/setup-bun/**'
  - 'packages/playground/**'
  - 'scripts/publishable-extensions.ts'
  - 'scripts/build-extensions.sh'
  - 'scripts/run-tests.sh'
  - 'scripts/extension-preflight.sh'
  # extension-preflight.sh runs it to compare each README Quickstart with examples/vanilla.
  - 'scripts/check-quickstart-example.ts'
  # The workflow that runs these suites, and this filter.
  - '.github/workflows/prod.docs.plus.yml'
  - '.github/filters/**'
  # Implements `release-preflight`, which every extension's prepublishOnly runs.
  - 'packages/release-tooling/**'

# README media: no suite, build or preflight reads it. The step sets
# predicate-quantifier: 'some-with-excludes', so a '!' rule is final for its key.
# README.md, CHANGELOG.md and package.json stay in, because the preflight reads them.
_readme_media: &readme_media
  - '!extensions/*/assets/**'
  - '!extensions/*/*.{gif,png,jpg,jpeg,webp,mp4}'

extension-indent:
  - *shared
  - *readme_media
  - 'extensions/extension-indent/**'
extension-hyperlink:
  - *shared
  - *readme_media
  - 'extensions/extension-hyperlink/**'
extension-hypermultimedia:
  - *shared
  - *readme_media
  - 'extensions/extension-hypermultimedia/**'
  # Its playground imports extension-hyperlink, and linked-image.cy.ts tests that pair.
  - 'extensions/extension-hyperlink/**'
extension-inline-code:
  - *shared
  - *readme_media
  - 'extensions/extension-inline-code/**'
extension-placeholder:
  - *shared
  - *readme_media
  - 'extensions/extension-placeholder/**'
extension-mermaid:
  - *shared
  - *readme_media
  - 'extensions/extension-mermaid/**'

• _shared and _readme_media are filter outputs that assemble ignores. A key of only ! rules never matches, so _readme_media is always false.
• Every key needs - *readme_media. Without it on extension-hypermultimedia, a hyperlink GIF would still start that suite.
• Check: bunx prettier --check .github/filters/extensions.yaml passes.

Step 2. The filter step

In .github/workflows/prod.docs.plus.yml, change the filter step of 🔎 Detect Changes (L146-L149 at the pin) to:

      - id: filter
        uses: dorny/paths-filter@ceb8a2b8f2d89434be7ff52d3de7ec3738c5cc9d # v4.0.3
        with:
          filters: .github/filters/extensions.yaml
          # Under the default 'some', a '!' rule matches every other file and starts every suite.
          predicate-quantifier: 'some-with-excludes'

• Change nothing else. The stale "~14 min" comment at L117 belongs to Make CI failures reach Telegram, and fix stale CI facts #557.
• Check: actionlint -config-file .github/actionlint.yaml .github/workflows/prod.docs.plus.yml exits 0.

Step 3. Leave the build-extensions cache key

Do not edit .github/actions/build-extensions/action.yml. The new entries do not change dist bytes, and adding them to hashFiles would bust the cache. Its L34-L37 comment stays true, because the Step 1 header names the test-only entries.

Step 4. The check-ci.ts mirror

In scripts/check-ci.ts:

1. Append to SHARED_EXACT (L20-L29 at the pin):

     'scripts/extension-preflight.sh',
     'scripts/check-quickstart-example.ts',
     '.github/workflows/prod.docs.plus.yml'
   ])

2. Append to SHARED_PREFIXES (L30-L38):

     '.github/actions/setup-bun/',
     '.github/filters/'
   ]

3. Below ALSO_WATCHES (L40-L42), add:

   // The filter's `_readme_media` excludes: no suite or preflight reads README media.
   const README_MEDIA = /^extensions\/[^/]+\/(assets\/|[^/]+\.(gif|png|jpe?g|webp|mp4)$)/

4. Change touched in extensionsFor (L102-L103) to:

     const touched = (dir: string) =>
       changed.some(
         (file) =>
           !README_MEDIA.test(file) &&
           (file === `extensions/${dir}` || file.startsWith(`extensions/${dir}/`))
       )

• Export nothing and add no test file. The script runs its gates at import.
• Check: bunx prettier --check scripts/check-ci.ts and bunx eslint scripts/check-ci.ts --max-warnings=0 pass. Do not run bun run check:ci.

Step 5. AGENTS.md

At AGENTS.md:222, after the sentence that ends "ALSO_WATCHES, so change both together.", add:

README media is excluded: extensions/*/assets/** and image or video files at an extension root start no suite. The step sets predicate-quantifier: 'some-with-excludes', because under the default some a ! rule matches every other file. README.md, CHANGELOG.md and package.json still start the suite, because the preflight reads them. scripts/check-ci.ts mirrors the exclusion in README_MEDIA.

• In the same paragraph, find "the build-extensions action, and the extension build scripts". Replace it with "the build-extensions action, the extension build and preflight scripts, this workflow, and .github/filters/".
• Check: bun run check:agent-docs passes.

Step 6. Prove the filter

FORCE_ALL (L157) makes a dispatch list all six suites, so use a local copy of the v4.0.3 matching rules instead.

• The script uses the repo's picomatch@2 copy in the Bun store and downloads nothing. If no picomatch@2.* folder exists under node_modules/.bun, run bun install first.
• Run the whole block as one shell command from the repo root. It lives in /tmp/ext-filter-sim, outside the repo.

T=/tmp/ext-filter-sim; mkdir -p "$T"
cat > "$T/filter-sim.ts" <<'EOF'
// bun filter-sim.ts <filters.yaml> <some|some-with-excludes> <file>... → matching keys, as dorny/paths-filter v4.0.3
import { readFileSync } from 'node:fs'
import { Glob } from 'bun'
const repo = process.env.REPO ?? process.cwd()
const pmDir = [...new Glob('node_modules/.bun/picomatch@2.*/node_modules/picomatch').scanSync({ cwd: repo, onlyFiles: false })][0]
if (!pmDir) throw new Error('no picomatch@2 under node_modules/.bun; run bun install first')
const picomatch = require(`${repo}/${pmDir}`)
const [yamlPath, quantifier, ...files] = Bun.argv.slice(2)
const doc = Bun.YAML.parse(readFileSync(yamlPath, 'utf8')) as Record<string, unknown>
const flat = (x: unknown): string[] => (Array.isArray(x) ? x.flatMap(flat) : [x as string])
const out: string[] = []
for (const [key, item] of Object.entries(doc)) {
  const ms = flat(item).map((p) => picomatch(p, { dot: true }, true))
  const inc = ms.filter((m: any) => !m.state.negated), exc = ms.filter((m: any) => m.state.negated)
  const hit = (f: string) =>
    quantifier === 'some-with-excludes'
      ? !exc.some((m: any) => !m(f)) && inc.some((m: any) => m(f))
      : ms.some((m: any) => m(f))
  if (files.some(hit)) out.push(key)
}
console.log(JSON.stringify(out.filter((k) => !k.startsWith('_'))))
EOF
git show d0a53756f8676556358b6adc6c2397d2ed7c64c8:.github/filters/extensions.yaml > "$T/before.yaml"
N=.github/filters/extensions.yaml; B="$T/before.yaml"; Q=some-with-excludes
sim() { REPO=$PWD bun "$T/filter-sim.ts" "$@"; }
sim $B some extensions/extension-hypermultimedia/assets/image-dark.png extensions/extension-hypermultimedia/assets/image-light.png
sim $N $Q extensions/extension-hypermultimedia/assets/image-dark.png extensions/extension-hypermultimedia/assets/image-light.png
sim $N $Q extensions/extension-mermaid/assets/demo-dark.gif
sim $N $Q extensions/extension-indent/preview.png
sim $N $Q extensions/extension-hyperlink/assets/demo-dark.gif
sim $N $Q extensions/extension-hypermultimedia/assets/readme-media/ATTRIBUTION.md
sim $N $Q extensions/extension-mermaid/README.md
sim $N $Q extensions/extension-indent/CHANGELOG.md
sim $N $Q extensions/extension-placeholder/package.json
sim $N $Q $(git show --name-only --format= 427c245c25a79f40f32117a760c8f2d0797eac74)
sim $N $Q extensions/extension-hypermultimedia/cypress/fixtures/x.png
sim $N $Q extensions/extension-mermaid/src/icons/a.png
sim $N $Q extensions/extension-hyperlink/src/index.ts
sim $N $Q scripts/check-quickstart-example.ts
sim $N $Q .github/workflows/prod.docs.plus.yml
sim $N $Q .github/filters/extensions.yaml
sim $N $Q bun.lock
sim $N $Q apps/webapp/src/pages/index.tsx
sim $N some apps/webapp/src/pages/index.tsx
sim $N some extensions/extension-mermaid/assets/demo-dark.gif
rm -r "$T"

Expect exactly these 20 lines. ALL stands for ["extension-indent","extension-hyperlink","extension-hypermultimedia","extension-inline-code","extension-placeholder","extension-mermaid"].

• Lines 19 and 20 are traps under some. They show why Step 2 is required.
• If any row differs, stop. Fix the filter, or comment here when the cause is not clear.
• Post the 20 output lines as a comment here, headed "Filter simulation".

Step 7. Gates for the touched files

actionlint -config-file .github/actionlint.yaml .github/workflows/prod.docs.plus.yml
bunx prettier --check .github/filters/extensions.yaml .github/workflows/prod.docs.plus.yml scripts/check-ci.ts AGENTS.md
bunx eslint scripts/check-ci.ts --max-warnings=0
bun run check:agent-docs

All four must exit 0. bash .github/scripts/parse-build-trigger.test.sh is not needed, because triage does not change.

Review checkpoint

Show git diff --stat and git diff for .github/filters/extensions.yaml, .github/workflows/prod.docs.plus.yml, scripts/check-ci.ts and AGENTS.md. Show the Step 6 comment link and the Step 7 output. Then stop. Do not run git add, git commit, git push or git stash, and create no branch or pull request.

The maintainer commits and pushes per SP2 (#554), as its own commit with a non-deploy subject such as ci(filters): skip extension suites for README media.

Acceptance criteria

• .github/filters/extensions.yaml matches Step 1. Every extension key holds - *readme_media.
• The filter step sets predicate-quantifier: 'some-with-excludes'. No other workflow line changed.
• .github/actions/build-extensions/action.yml is unchanged.
• scripts/check-ci.ts holds the two new SHARED_EXACT entries, the .github/filters/ prefix and README_MEDIA, and touched() skips README_MEDIA files.
• AGENTS.md:222 names the media exclusion, the quantifier and the README_MEDIA mirror.
• The "Filter simulation" comment exists, and all 20 lines match the Step 6 table.
• The four Step 7 commands exit 0.
• After merge, the landing push shows Changes output set to ["_shared","extension-indent","extension-hyperlink","extension-hypermultimedia","extension-inline-code","extension-placeholder","extension-mermaid"], and all six 🧪 Ext (…) jobs pass.
• After merge, a change that touches only extension README media shows Changes output set to [] and no 🧪 Ext (…) job.
• After merge, a change to an extension README.md still runs that extension's 🧪 Ext (…) job, with its ✈️ Preflight step.
• The "Measured, " comments exist, as Accountability says.

Verification

Before merge (the builder): run the Step 7 gates and the Step 6 simulation.

A dry_run_subject dispatch is optional; it only proves that the step loads the new input. Run SP1 (#554) first. Run one dispatch at a time, and wait for it with gh run watch <id> --repo docs-plus/docs.plus. Nobody pushes to main while the dispatch runs. Expect six 🧪 Ext (…) jobs and no Input parameter 'predicate-quantifier' error. Ignore its Changes output set to line.

After merge (the maintainer, or an agent the maintainer names). To read a 🔎 Detect Changes log:

gh run view <run-id> --repo docs-plus/docs.plus --log --job <detect-changes-job-id> | grep -a -E -A1 "Changes will be detected|name-status|\[(added|modified|removed|renamed)\] |Detected [0-9]+ changed files|Filter (_shared|extension-)|Changes output set to"
gh run view <run-id> --repo docs-plus/docs.plus --json jobs -q '.jobs[] | select(.name | test("Ext \\(|Build Verification")) | [.name, .conclusion, .startedAt, .completedAt] | @tsv'

Keep -a. A push diff line holds NUL bytes, and without -a GNU grep drops the needed lines.

1. The landing push. Expect the full Changes output set to list from Acceptance criteria and six passing 🧪 Ext (…) jobs. This proves that the step accepts the quantifier.
2. The first media-only change. Preferred: the maintainer opens a draft pull request that adds one copied PNG under extensions/extension-mermaid/assets/. Read the run, then close it unmerged. Or wait for the next media-only push. Expect Changes output set to [] and no 🧪 Ext (…) job. Before Run only the CI suites a pull request or push can affect #563 lands, 🏗️ Build Verification starts right after the other gates. After Run only the CI suites a pull request or push can affect #563, it skips, so record M6 only.
3. The first README change. A draft pull request with a one-line edit to extensions/extension-mermaid/README.md outside ## Quickstart, then closed. Or wait for a real README change. Expect Changes output set to ["extension-mermaid"] and a passing 🧪 Ext (extension-mermaid) job with its ✈️ Preflight (extension-mermaid) step.
4. The next weekly cron (Sunday 00:00 UTC). Expect all six 🧪 Ext (…) jobs.
5. The next (build): deploy. Expect Changes output set to to name every extension with a non-media file in that push's diff, and no other. The diff is the range in Changes will be detected between <before> and main.
   • If the diff touches the three new _shared entries, expect _shared and all six names. deploy then waits for build (:427), as Summary says.

Accountability

• The builder posts the "Filter simulation" comment before review.
• The maintainer checks item 1 within one day of the push, then items 2 and 3.
• Record per SP3 ([Tracking] Faster, change-aware CI/CD pipeline #554), one comment per checked run. Add the 🔎 Detect Changes job id, its Detected N changed files and Changes output set to lines, and the Ext jobs with seconds. For a non-deploy run, add the 🏗️ Build Verification start offset and M6. Metrics: Measure the CI/CD after benchmark for v2.3 #569.

M2g, M2i, M3, M4, M5 and W do not change. Close this issue only after the comments for items 1, 2 and 3 exist.

Rollback

Revert per SP4 (#554) when any run after merge shows one of these:

• A diff file outside assets/ and the root media globs leaves its extension out of Changes output set to. Read the diff lines with the Verification grep.
• The 🔎 Detect Changes step fails on the predicate-quantifier input.
• The weekly cron fails on a spec that an earlier filtered push changed, while that push showed no 🧪 Ext (…) job for it.

Fix, do not revert, when a media-only change still runs its suite. That costs time only. Comment here with the run id.

After the revert:

• The revert run starts no 🧪 Ext (…) job, as expected: none of the four files is an old filter input.
• Skipped suites do not rerun on their own. Run SP1 ([Tracking] Faster, change-aware CI/CD pipeline #554), then gh workflow run prod.docs.plus.yml --repo docs-plus/docs.plus --ref main with no inputs. FORCE_ALL runs all six suites, and nothing deploys.
• Run one dispatch at a time, and wait for it with gh run watch <id> --repo docs-plus/docs.plus. Nobody pushes to main while the dispatch runs.
• Expect six passing 🧪 Ext (…) jobs. Post the run id here.

Docs to update

• AGENTS.md:222: Step 5.
• Nothing else. .github/WORKFLOWS.md:67 and .cursor/docs/scripts-naming-convention.md:178 stay true after Step 4.

Out of scope

• The webapp, admin, backend and docs_only scopes, and skipping Build Verification on docs-only or media-only pushes: Run only the CI suites a pull request or push can affect #563.
• Removing back-validation from the extension-tests needs: Start the extension and webapp unit tests without waiting for backend validation #558.
• The stale "~14 min" comment at prod.docs.plus.yml:117: Make CI failures reach Telegram, and fix stale CI facts #557.
• Excluding any non-media file, such as README.md, CHANGELOG.md, package.json, examples/** or cypress/docs/**.
• Excluding media under src/** or cypress/fixtures/**. Those can be build or test inputs.
• Changing FORCE_ALL, the weekly cron, the matrix if or build's accepted results.

Read first

• AGENTS.md: §Git And Commits, §Test Policy, and line 222.
• extensions/CLAUDE.md: the extension workflow and the docs:gif and docs:screenshots notes.
• .github/filters/extensions.yaml, in full.
• .github/workflows/prod.docs.plus.yml: L124-L176 (🔎 Detect Changes), L272-L306 (extension-tests) and L375.
• scripts/check-ci.ts: L1-L110.
• scripts/extension-preflight.sh and scripts/check-quickstart-example.ts.
• .github/actions/build-extensions/action.yml.
• dorny/paths-filter at ceb8a2b8f2d89434be7ff52d3de7ec3738c5cc9d: action.yml:47-55 and src/filter.ts:122-214.