Skip to content

fix: honor explicit TestQL targets and validate dry runs - #45

Merged
ifuri-validator-agent[bot] merged 2 commits into
mainfrom
ticket/030-explicit-target-and-dry-run
Oct 26, 2026
Merged

ifuri-validator-agent[bot] merged 2 commits into
mainfrom
ticket/030-explicit-target-and-dry-run

Conversation

@tom-sapletta-com

@tom-sapletta-com tom-sapletta-com commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Scenarios with a CONFIG base_url could override an explicit --url and send tests to localhost instead of the requested device. Pin the CLI-selected base URL through SET and INCLUDE while preserving scenario defaults when no override is supplied.

Dry-run now validates response assertion syntax without evaluating absent responses. JSON output distinguishes validated steps from execution and rejects malformed assertions. Existing interpreter step statuses remain compatible.

Validation: 114 focused regression tests pass with managed governance. All 19 device E2E scenarios validate 428 steps with zero execution. Two independent local HTTP servers verify that only the CLI-selected target receives requests. Physical actuator commands were not used.

Ticket: ticket-030. Parent requests: autogrammar/testql STARTER-023 and maskservice PLF-17350/17354. Exact head: 9e34c19

Seeded JSON-path regressions execute assertion evaluation without dry-run; positive and mismatch checks are preserved.

@ifuri-validator-agent ifuri-validator-agent Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Validator approval after policy checks for exact head 9e34c19ab7625c98fbaed48b8227b4f0d1f0ef68.

Ticket: ticket-030
Correlation ID: codex-testql030-9e34c19
Model: openai/cursor-auto
Reviewed diff chunks: 3
Advisory LLM verdict: REQUEST_CHANGES
Advisory summary: Reviewed all 3 diff chunk(s). Required checks pass, but the visible ASSERT_JSON change broadens dry-run skipping beyond response-dependent assertions. | The visible chunk consistently separates dry-run validation from execution, validates assertion operators and required operands, preserves live assertion behavior, and enforces the explicit CLI base URL over scenario SET values. Focused regression tests cover network avoidance, malformed assertions, live evaluation, and target-selection behavior. The sole required check, test, passed. | The visible regression tests correctly verify that an included scenario cannot override an explicit CLI target and that seeded JSON-path assertion tests run in execution mode rather than syntax-only dry-run mode. The sole protected required check, test, passed.
Advisory findings: In _cmd_assert_json, _validate_dry_response_assertion(desc) runs before determining whether the path refers to response data or an ordinary interpreter variable. Consequently, dry-run marks assertions such as ASSERT_JSON _foo ... as skipped without evaluating them, even though those values can be available during validation. Move the dry-run response skip into the response-data branch so variable-backed assertions retain their existing validation behavior.
The LLM output above is advisory and was not used as the approval trust root.
Semantic review prerequisite: not_required; policy 676cb4516bbfed2a000e40b9b1b6e4a430ecc761ec546aeb53d721a1905cfdd7.

Actual PR impact radar

Exact range: 5dc442a173b3c39684d37ae7ed5e309c04244d14...9e34c19ab7625c98fbaed48b8227b4f0d1f0ef68
Change digest: b2308cec85031171c631728b5eadc4ba086ffecdad0c0a4ebdbb1ab2eb16ca90
Score: 64/100 (L), estimated 79 min, split recommended: true
Affected services/components: repository-wide/unclassified

Machine-readable radar JSONL and SVG
{"actual_change":{"additions":382,"base_sha":"5dc442a173b3c39684d37ae7ed5e309c04244d14","binary_files":0,"categories":{"code":3,"configuration":1,"docs":2,"tests":3},"change_digest":"b2308cec85031171c631728b5eadc4ba086ffecdad0c0a4ebdbb1ab2eb16ca90","comparison":"5dc442a173b3c39684d37ae7ed5e309c04244d14...9e34c19ab7625c98fbaed48b8227b4f0d1f0ef68","deletions":28,"file_count":9,"files":["project/TICKETS.md","project/ticket-030/README.md","project/ticket-030/intent.json","testql/commands/run_cmd.py","testql/interpreter/_assertions.py","testql/interpreter/interpreter.py","tests/test_dry_run_validation.py","tests/test_explicit_target.py","tests/test_navigate_json_path.py"],"head_sha":"9e34c19ab7625c98fbaed48b8227b4f0d1f0ef68","service_count":0,"services":[]},"assessment_mode":"observed-pr","axes":{"coupling":5,"delivery":2,"scope":5,"uncertainty":3,"validation":1},"complexity":"L","confidence":0.9,"diagnostics":["RADAR-ACCEPTANCE-MISSING","RADAR-BUDGET-EXCEEDED"],"estimate":{"budget_minutes":30,"minutes":79,"within_budget":false},"impact":{"components":["PLF-17350","autogrammar","project","testql","tests"],"files":["PLF-17350/17354","autogrammar/testql","project/TICKETS.md","project/ticket-030/README.md","project/ticket-030/intent.json","testql/commands/run_cmd.py","testql/interpreter/_assertions.py","testql/interpreter/interpreter.py","tests/test_dry_run_validation.py","tests/test_explicit_target.py","tests/test_navigate_json_path.py"],"public_interfaces":[],"runtime_dependencies":0},"schema":"subactor.ticket-radar/v1","score":64,"split":{"parts":[{"estimated_minutes":14,"name":"Implement PLF-17350","scope":["PLF-17350"]},{"estimated_minutes":14,"name":"Implement autogrammar","scope":["autogrammar"]},{"estimated_minutes":14,"name":"Implement project","scope":["project"]},{"estimated_minutes":14,"name":"Implement testql","scope":["testql"]},{"estimated_minutes":14,"name":"Implement tests","scope":["tests"]},{"estimated_minutes":15,"name":"Validate and project to trackers","scope":["tests","planfile","github/gitlab/jira projections"]}],"reason":"estimated_minutes_exceed_budget","recommended":true},"standards":[{"id":"wellmanifest/dsl","revision":"6c60fc4e0dd1f1bb74f46a7745e28019908d1203","version":"0.1.0-dev"},{"id":"wellmanifest/ticket-lifecycle","revision":"5bf581907a87b46a13a73e6c033d3abe4d9a306f","version":"0.1.0-dev"},{"id":"wellmanifest/git-lifecycle","revision":"7d77d4b7af57e69bc75c3a0290b3a4805c5c4438","version":"0.2.0-dev"},{"id":"wellmanifest/logs","revision":"48c284ef7a069055c0bcb6b900147ce5e65f8b43","version":"0.3.0"}],"ticket_ref":"ticket-030"}
<svg xmlns="http://www.w3.org/2000/svg" width="128" height="128" viewBox="0 0 128 128" role="img"><title>ticket-030: fix: honor explicit TestQL targets and validate dry runs</title><rect width="128" height="128" rx="12" fill="#f8fafc"/><g stroke-width="1"><polygon points="64,55 72,61 69,71 59,71 56,61" fill="none" stroke="#d7dde5"/><polygon points="64,47 80,59 74,78 54,78 48,59" fill="none" stroke="#d7dde5"/><polygon points="64,38 89,56 79,85 49,85 39,56" fill="none" stroke="#d7dde5"/><polygon points="64,30 97,53 84,92 44,92 31,53" fill="none" stroke="#d7dde5"/><polygon points="64,21 105,51 89,99 39,99 23,51" fill="none" stroke="#d7dde5"/><line x1="64" y1="64" x2="64" y2="21" stroke="#aab4c0"/><line x1="64" y1="64" x2="105" y2="51" stroke="#aab4c0"/><line x1="64" y1="64" x2="89" y2="99" stroke="#aab4c0"/><line x1="64" y1="64" x2="39" y2="99" stroke="#aab4c0"/><line x1="64" y1="64" x2="23" y2="51" stroke="#aab4c0"/></g><polygon points="64,21 105,51 79,85 59,71 48,59" fill="#fb923c" fill-opacity="0.45" stroke="#c2410c" stroke-width="2"/><circle cx="64" cy="64" r="3" fill="#c2410c"/><g font-family="sans-serif" font-size="7" fill="#334155"><text x="64" y="11" text-anchor="middle">SCO</text><text x="114" y="48" text-anchor="middle">COU</text><text x="95" y="107" text-anchor="middle">UNC</text><text x="33" y="107" text-anchor="middle">VAL</text><text x="14" y="48" text-anchor="middle">DEL</text></g><text x="64" y="124" text-anchor="middle" font-family="sans-serif" font-size="8" fill="#0f172a">L · 79m</text></svg>
Merge will be attempted after this approval when explicitly authorized. ## Decision record (recomputable)
DECISION D-030-6153
TICKET ticket-030
HEAD_SHA 9e34c19ab7625c98fbaed48b8227b4f0d1f0ef68
CORRELATION_ID codex-testql030-9e34c19
ACTOR agent:ifuri-validator-agent[bot]
APPLIED_RULE P-CORE-015
INPUT author_login = "tom-sapletta-com"
INPUT observed_checks = ["metadata=PASS","governance / enforce=PASS","test=PASS","governance / remote lifecycle=PASS"]
INPUT required_checks = ["test"]
INPUT required_checks_source = "protected registry (env/request)"
INPUT reviewer_login = "ifuri-validator-agent[bot]"
INPUT semantic_review_assessment = {"schema":"subactor.validator/semantic-review-assessment/v1","subject":{"repository":"autogrammar/testql","pull_request":45,"head_sha":"9e34c19ab7625c98fbaed48b8227b4f0d1f0ef68","base_sha":"5dc442a173b3c39684d37ae7ed5e309c04244d14","diff_sha256":"12a64ec2ffec5a8a7c7794d2c210a2ddd95c75f93a0c4bdecbe81b270e82d2e1"},"policy":{"policy_schema":"subactor.validator/semantic-review-policy/v1","policy_version":1,"policy_sha256":"676cb4516bbfed2a000e40b9b1b6e4a430ecc761ec546aeb53d721a1905cfdd7","required":false,"critical_paths":[],"observed_paths":["project/TICKETS.md","project/ticket-030/README.md","project/ticket-030/intent.json","testql/commands/run_cmd.py","testql/interpreter/_assertions.py","testql/interpreter/interpreter.py","tests/test_dry_run_validation.py","tests/test_explicit_target.py","tests/test_navigate_json_path.py"]},"grounding":"full-diff-not-per-finding-proof","execution_authority":false,"status":"not_required","reason":null,"review_sha256":null,"unresolved":[]}
INPUT superseded_checks = []
INPUT ticket_radar_receipt = {"schema":"subactor.ticket-radar/v1","base_sha":"5dc442a173b3c39684d37ae7ed5e309c04244d14","head_sha":"9e34c19ab7625c98fbaed48b8227b4f0d1f0ef68","change_digest":"b2308cec85031171c631728b5eadc4ba086ffecdad0c0a4ebdbb1ab2eb16ca90","score":64,"complexity":"L","estimated_minutes":79,"split_recommended":true,"services":[],"authority":"ADVISORY","promotion":"FORBIDDEN"}
VERDICT APPROVE AUTHORITY DETERMINISTIC
REJECTED REQUEST_CHANGES BECAUSE NO_UNSAFE_CHANGE_REASON_FOUND
ADVISORY llm_verdict = "REQUEST_CHANGES" MODEL "openai/cursor-auto"
ASSERT VERDICT_AUTHORITY != "ADVISORY"

@ifuri-validator-agent
ifuri-validator-agent Bot merged commit 3c83a8d into main Oct 26, 2026
4 checks passed
@ifuri-validator-agent
ifuri-validator-agent Bot deleted the ticket/030-explicit-target-and-dry-run branch October 26, 2026 10:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant