Skip to content

fix(core): scan .robot/.resource files for test markers (#1105) - #1107

Closed
avrabe wants to merge 1 commit into
mainfrom
fix/issue-1105-robot-scanner
Closed

avrabe wants to merge 1 commit into
mainfrom
fix/issue-1105-robot-scanner

Conversation

@avrabe

@avrabe avrabe commented Oct 20, 2026

Copy link
Copy Markdown
Contributor

Closes #1105

Summary

test_scanner::detect_language had no entry for Robot Framework (.robot, .resource), so # rivet: verifies REQ-NNN markers in Robot suites were silently skipped. Reported downstream: 145 inflated warnings from a project whose tests are Robot Framework suites.

Same shape as the shell gap (REQ-319), the yaml/toml gap (REQ-352) and the lean/rocq gap (#992) — the pattern itself was never the obstacle; scan_file bailed before any pattern ran.

Fix

  • Add robot as its own language in default_patterns, using # comments.
  • Map .robot and .resource to robot in detect_language.
  • Add robot to find_enclosing_function's early-return list (Robot test cases are names under *** Test Cases ***, not functions — guessing a name would be the REQ-326 misattribution shape the Lean/Rocq guard already prevents).

Acceptance criteria from the issue

  • .robot / .resource files are scanned with # comments. detect_language now maps both extensions to "robot", and default_patterns carries a matching entry.
  • A marker placed on the line before a test case name is credited. robot_framework_files_are_scanned_for_markers places the marker on the line above My Test Case (and My Keyword in a .resource file) and asserts both ids are found.
  • A test with a .robot fixture asserts the marker is credited. Added robot_framework_files_are_scanned_for_markers in rivet-core/src/test_scanner.rs.
  • A negative control drops the extension mapping and watches it fail. robot_markers_fall_back_to_file_and_line plants def helper( above the marker; removing the "robot" guard from find_enclosing_function latches the generic catch-all onto helper and the assert_ne!(test_name, "helper") fails. The two assertions together kill a mutant that drops either the extension mapping or the enclosing-function guard.

Test plan

  • cargo test -p rivet-core --lib test_scanner:: — 27 passed (both new tests included).
  • CI green.

Generated by Claude Code

Robot Framework suites are tests, but `test_scanner::detect_language`
had no entry for `.robot` or `.resource`, so `# rivet: verifies REQ-NNN`
markers in those files were silently skipped — the suite could never
be credited as verification evidence and inflated the reporter's
warning count by 145.

Same shape as the shell gap (REQ-319), the yaml/toml gap (REQ-352)
and the lean/rocq gap (#992): the pattern itself was never the
obstacle; `scan_file` bailed before any pattern ran.

Fix: add `robot` as its own language in `default_patterns` (`#`
comments, matching Robot Framework syntax), map `.robot` and
`.resource` to it in `detect_language`, and add it to
`find_enclosing_function`'s early-return list (Robot test cases are
names under `*** Test Cases ***`, not functions — a guessed name
would be an REQ-326-shape misattribution).

Verified by two new tests: one asserts both extensions are credited;
the other plants a `def helper(` above the marker to prove the
`robot` guard is load-bearing against the generic catch-all.

Fixes: REQ-389

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01E1rzxvHN7qT1UaX8GEtcYw

avrabe commented Oct 20, 2026

Copy link
Copy Markdown
Contributor Author

Security Audit (RustSec) failing — standing down on this one as it is not this PR's.

The diff is a single Rust-logic change in rivet-core/src/test_scanner.rs; no Cargo.toml / Cargo.lock / deny.toml edits. The job reports six advisories:

  • RUSTSEC-2024-0384 (instant unmaintained) — already in deny.toml ignore list
  • RUSTSEC-2026-0251 (sized-chunks unmaintained) — already in deny.toml
  • RUSTSEC-2023-0126 (im unsound, 2023-02-04)
  • RUSTSEC-2026-0308 (salsa use-after-free) — already assessed not-applicable in deny.toml
  • RUSTSEC-2026-0205 (scc double-free, 2026-07-06)
  • RUSTSEC-2026-0255 (sized-chunks panic-safety unsound, 2026-08-11)

All six predate this PR, and the same check was green on the latest scheduled main run (2026-10-19). The audit job in .github/workflows/ci.yml:706 installs cargo-audit unpinned (cargo install cargo-audit --locked), so a release landing between that run and today likely tightened its default (treating unmaintained/unsound as errors rather than warnings) — the --ignore list in the workflow covers wasmtime-era advisories but none of the six now surfacing. The three already in deny.toml are not duplicated to the workflow's --ignore list because they historically only mattered to cargo deny.

No fix PR exists yet. The right fix is a separate CI-only change: either pin cargo-audit to a version matching last run's defaults, or extend the workflow's --ignore list to mirror deny.toml. Not porting it into this PR to keep the diff focused on #1105.

Will spend my one allotted re-run to confirm it reproduces, then wait on the fix landing on main.


Generated by Claude Code

avrabe commented Oct 20, 2026

Copy link
Copy Markdown
Contributor Author

Playwright E2E failing — also not this PR's.

BEWARE: your OS is not officially supported by Playwright; installing dependencies for ubuntu26.04-x64 as a fallback.
Cannot install dependencies for ubuntu26.04-x64 with Playwright 1.58.2!
Error: ERROR: Playwright does not support chromium on ubuntu26.04-x64

GitHub's ubuntu-latest moved to ubuntu-26.04 and the pinned Playwright 1.58.2 doesn't recognize it. The diff touches no tests/playwright/** and no workflow file. Same job was green on the latest scheduled main run (2026-10-19), so the runner image bumped between then and today.

Fix for a separate CI PR: either bump the Playwright pin to a version that supports 26.04, or pin the runner to ubuntu-24.04. Not porting into this PR to keep it focused on #1105.


Generated by Claude Code

@github-actions

Copy link
Copy Markdown

📐 Rivet artifact delta

No artifact changes in this PR. Code-only changes (renderer, CLI wiring, tests) don't touch the artifact graph.

avrabe commented Oct 20, 2026

Copy link
Copy Markdown
Contributor Author

Cargo Deny failing — also not this PR's.

Two newly-published wasmtime 48.0.6 CVEs:

  • RUSTSEC-2026-0378 — panic lowering a host-created component-model stream (announced 2026-10-20)
  • RUSTSEC-2026-0379 — Cranelift aarch64 br_table jumps to incorrect target (announced 2026-10-20)

Both have the same fix: cargo update -p wasmtime to >=48.0.7, <49.0.0 OR >=49.0.4. The diff touches no Cargo.toml / Cargo.lock, and rivet-core only pins the parent wasmtime version (b24baa6 bumped it to 48.0.6, which these advisories flag). The same job was green on the latest scheduled main run (2026-10-19) because the advisories hadn't been published yet. Main is now red against the fresh advisory DB too — unlike the audit/playwright failures above this is a real vulnerability that needs a wasmtime bump on main, not a CI config change.

Not porting into this PR to keep the diff focused on #1105. A separate build(deps): wasmtime 48.0.6 → 48.0.7 PR will clear it, same shape as b24baa6.


Generated by Claude Code

@codecov

codecov Bot commented Oct 20, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 92.68293% with 3 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
rivet-core/src/test_scanner.rs 92.68% 3 Missing ⚠️

📢 Thoughts on this report? Let us know!

@avrabe

avrabe commented Oct 21, 2026

Copy link
Copy Markdown
Contributor Author

Superseded by #1108 (merged as cc3b807, ships in v0.41.1, REQ-407): .robot / .resource markers are scanned there too, and a marker is attributed to the Robot test case or keyword it precedes or sits in (falling back to file:line outside any case), with a negative control. This branch now conflicts with main and fails on advisories cleared in #1108.

@avrabe avrabe closed this Oct 21, 2026
@avrabe
avrabe deleted the fix/issue-1105-robot-scanner branch October 21, 2026 18:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Source markers in Robot Framework (.robot) files are not scanned

2 participants