Repository navigation
Simplify AI Gateway skills and align portable plugin workflows - #38
Paul Yuknewicz (paulyuk) wants to merge 20 commits into
Conversation
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
TL;DR: The first run failed every contract check, but the eval rejected valid forms of some answers and hid required field types. This was not a shared startup failure or proof that all 18 skills scenarios were broken. Correct the eval and duplicate routing instructions, then replay the saved answers before considering another paid run. Waza journal: 2026-11-09The supplied local quality artifact records Copilot SDK / gpt-6.1-sol, 18 selected tasks, one run per task. Result: 0 passed, 18 failed contract checks, 0 harness errors, 0 skipped. All final outputs parse as JSON, all program graders executed, and offline replay reproduces the failures. This is not a shared startup, missing-file, dependency, or invalid-JSON failure, and 18 failed contracts do not establish 18 broken skills. The rubric is too rigid in several high-confidence ways: it rejects additional relevant safety gates, full resource IDs where a short selected name is expected, and reasonable read-only/assessment decisions. Prompts name fields but do not specify their nested value types, so explanations are being compared with hidden booleans/enums. The alias URL oracle also omits the required Reported wall duration is 288.669 seconds; median task duration is 27.804 seconds. Usage reports 74 turns, 1,658,977 input tokens, 16,668 output tokens, 18 premium requests, and 91.02379 AI credits. These are artifact-reported values, not independently billed charges. No matched CLI baseline, multi-trial reliability result, or Grade A is established. Recorded tool names are skill, glob, view and rg; no oracle-file reads were found in recorded arguments. That does not establish complete execution safety. The Waza v0.40.1 executor's permission defaults and this run's lack of an established enforced tool boundary prevent a safety certification. Next: reduce duplicate adapters while preserving public skill names/boundaries, correct the high-confidence rubric defects, add typed output-shape guidance without exposing scenario answers, and replay these existing outputs offline. Any new paid trial requires separate authorization and an enforced execution boundary. No additional model trial, Azure operation, or data-plane call has been run for this review. Raw prompts, transcripts, local paths and credentials remain uncommitted and unpublished. |
|
In plain English, the first attempt to define and run a Waza eval was too rigid. but this helped us learn. doing an iteration now. we repeat in a loop to improve the evals, improve the skills, repeat. |
Keep public skill boundaries and migration helpers unchanged, move development regressions outside runtime skills, and correct reviewed rubric defects without relaxing approval or execution gates. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
TL;DR: Repeated adapter instructions and several overly rigid eval comparisons are corrected. Replaying the same answers now passes 4 of 18 scenarios; that is an offline comparison, not a new model score. Next, clarify the remaining answer types and approval-pending plans, and establish the tool boundary before another paid trial. 2026-11-09 UTC: routing and evaluation correctionsThe four public skill names and boundaries are unchanged. Migration remains bundled; feedback is a support utility. The six compatibility commands now route to one authoritative skill recipe instead of repeating workflows across Markdown and Gemini adapters. Evidence-sufficient explanations skip management setup/discovery. Migration and feedback development regressions moved outside runtime skill directories; executable helpers, catalogs and schemas are unchanged. The portable manifest has no standard exclusion field, so this does not claim every host omits development files when downloading the repository. Corrected the reviewed rubric defects: task prompts specify types without revealing expected answers; extra known gates and exact scoped full IDs are accepted without waiving required gates; reviewed assessment phases are accepted; alias URLs include
Remaining first-failure diagnostics mostly concern typed fields answered with prose or structured explanations. Runtime-only and stale-sync also omit required gate identifiers. The bootstrap proposal includes optional Foundry work awaiting separate approval, which the flat allowlist cannot represent as conditional. These findings do not prove the proposed commands executed. A first mismatch can hide further contract failures; no narrative string was coerced to a successful boolean, and no missing gate was waived. Next: establish and verify the execution boundary before separately approving a new trial. The saved quality configuration still has the old untyped prompts; regenerate it before any rerun, then review/preflight the generated suite. The exact preparation and Recorded base: |
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
TL;DR: Most remaining failures concern answer representation or plans that include work awaiting approval, not demonstrated unsafe execution. The planning contract now separates pending work from authorized commands, with negative safety tests retained. Regenerate the inputs and review the deny-all boundary before an operator-run trial; the replay is not a fresh score. 2026-11-09: REDACTED planning-contract correctionsThe corrections are committed and pushed. Regenerate the planning configuration FindingsThe supplied baseline records Copilot SDK / GPT-6.1 Sol ( Review of the 14 failures remaining after the previous 4/18 replay:
This classification does not certify every other field or establish 14 skill CorrectionsThe active quality contract now checks typed facts, scoped operations and verified The generated suite binds Waza v0.40.1's supported This binding is statically checked, not locally runtime-verified. It is not an Three separate holdouts vary the Messages/Api-Key endpoint contract, an unapproved Offline evidence
The two remaining behavior-contract failures are the flat conditional bootstrap The original artifact SHA-256 remains Operator terminal commandsFrom this PR worktree, stop on any failed check: uv run --no-project --with pyyaml --with jsonschema \
python -m unittest discover -s evals/waza -p 'test_*.py' -q
uv run --no-project --with pyyaml --with jsonschema \
python evals/waza/replay.py evals/waza/results/quality-gpt-6.1-sol.json
uv run --no-project --with pyyaml --with jsonschema \
python evals/waza/author.py --quality --authorized --model gpt-6.1-sol --trials 1
uv run --no-project --with pyyaml --with jsonschema \
python evals/waza/verify_planning.py evals/waza/authorized-quality/eval.yaml
waza --version
# Stop unless this is the approved v0.40.1 binary/runtime.
uv run --no-project --with pyyaml --with jsonschema \
waza preflight evals/waza/authorized-quality/eval.yaml --strictAfter the operator's model-cost and environment decision, one new run: mkdir -p evals/waza/results evals/waza/workspaces
RESULT="evals/waza/results/quality-planning-gpt-6.1-sol-$(date -u +%Y%m%dT%H%M%SZ).json"
test ! -e "$RESULT" && \
TMPDIR="$PWD/evals/waza/workspaces" COPILOT_AUTO_UPDATE=false WAZA_NO_UPDATE_CHECK=1 \
uv run --no-project --with pyyaml --with jsonschema \
waza run evals/waza/authorized-quality/eval.yaml --no-cache --output "$RESULT" && \
uv run --no-project --with pyyaml --with jsonschema \
python evals/waza/verify_planning.py evals/waza/authorized-quality/eval.yaml --result "$RESULT"Never overwrite Provenance
|
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
TL;DR: Our strict-preflight instruction guaranteed failure on checks that cannot be verified offline. The operator's chain correctly stopped before spending model tokens. Use the narrowly checked non-strict report instead, while keeping actual configuration errors fatal and deciding separately whether runtime and cost are acceptable. 2026-11-09: Correct the offline preflight gateThe earlier Pinned Waza v0.40.1 preflight never starts an engine, model, grader subprocess or
There is no supported per-warning waiver option. The correction uses supported Exact next command, from this PR worktree after the pushed correction: uv run --no-project --with pyyaml --with jsonschema \
waza preflight evals/waza/authorized-quality/eval.yaml --format json | \
uv run --no-project --with pyyaml --with jsonschema \
python evals/waza/verify_planning.py evals/waza/authorized-quality/eval.yaml --preflight -Stop if the checker exits nonzero. The successful generated configuration need After that decision, the previously published new-filename run stays unchanged: mkdir -p evals/waza/results evals/waza/workspaces
RESULT="evals/waza/results/quality-planning-gpt-6.1-sol-$(date -u +%Y%m%dT%H%M%SZ).json"
test ! -e "$RESULT" && \
TMPDIR="$PWD/evals/waza/workspaces" COPILOT_AUTO_UPDATE=false WAZA_NO_UPDATE_CHECK=1 \
uv run --no-project --with pyyaml --with jsonschema \
waza run evals/waza/authorized-quality/eval.yaml --no-cache --output "$RESULT" && \
uv run --no-project --with pyyaml --with jsonschema \
python evals/waza/verify_planning.py evals/waza/authorized-quality/eval.yaml --result "$RESULT"Never overwrite the supplied baseline. Post-run evidence must still record References
Publication noteThe prepared journal text above is restored here because the original comment displayed only this local-file reference: |
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
TL;DR: The eval spent tokens but 17 scenarios never reached the model because their quoted relative paths had no workspace input. Each task now supplies its real synthetic evidence file, and saved failures still reach the policy verifier. Regenerate the configuration before another approved run; the recorded failures are not 18 skill regressions. 2026-11-09: Planning input wiring and saved-result verificationThe operator's new result spent tokens, but 17 tasks failed before model Verified REDACTED resultSource: immutable
The aggregate Root and correctionPinned Waza The generator now supplies one legitimate 39 offline tests passed. The new tests reproduce the pinned guard and inline Operator commandsRegeneration is required. From this PR worktree at the pushed head, prepare export WAZA_NO_UPDATE_CHECK=1
export COPILOT_AUTO_UPDATE=false
uv run --no-project --with pyyaml --with jsonschema \
python -m unittest discover -s evals/waza -p 'test_*.py' -q && \
uv run --no-project --with pyyaml --with jsonschema \
python evals/waza/author.py --quality --authorized --model gpt-6.1-sol --trials 1 && \
uv run --no-project --with pyyaml --with jsonschema \
python evals/waza/verify_planning.py evals/waza/authorized-quality/eval.yamlUse the already approved Waza v0.40.1/runtime; stop on unexpected installation uv run --no-project --with pyyaml --with jsonschema \
waza preflight evals/waza/authorized-quality/eval.yaml --format json | \
uv run --no-project --with pyyaml --with jsonschema \
python evals/waza/verify_planning.py evals/waza/authorized-quality/eval.yaml --preflight -Only after a successful check and the operator's renewed environment/cost mkdir -p evals/waza/results evals/waza/workspaces
RESULT="evals/waza/results/quality-planning-gpt-6.1-sol-$(date -u +%Y%m%dT%H%M%SZ).json"
if test -e "$RESULT"; then
printf 'Refusing to overwrite %s\n' "$RESULT"
false
else
if TMPDIR="$PWD/evals/waza/workspaces" COPILOT_AUTO_UPDATE=false WAZA_NO_UPDATE_CHECK=1 \
uv run --no-project --with pyyaml --with jsonschema \
waza run evals/waza/authorized-quality/eval.yaml --no-cache --output "$RESULT"; then
WAZA_STATUS=0
else
WAZA_STATUS=$?
fi
if test -f "$RESULT"; then
if uv run --no-project --with pyyaml --with jsonschema \
python evals/waza/verify_planning.py evals/waza/authorized-quality/eval.yaml --result "$RESULT"; then
POLICY_STATUS=0
else
POLICY_STATUS=$?
fi
else
printf 'No saved result to verify\n'
POLICY_STATUS=1
fi
printf 'Result: %s\nWaza exit: %s\nPolicy-check exit: %s\n' "$RESULT" "$WAZA_STATUS" "$POLICY_STATUS"
test "$WAZA_STATUS" -eq 0 && test "$POLICY_STATUS" -eq 0
fiWhole-suite runtime enforcement and scenario reliability remain unresolved. The References
Publication noteThe prepared journal text above is restored here because the original comment displayed only this local-file reference: |
|
TL;DR: The input fix let all 18 scenarios run, with verified deny-all records, but only 8 passed. Several plans lacked the task recipes that the eval had omitted while disabling file tools; other failures need semantic review. Supply the missing authoritative context and review the comparisons offline before using the next approved trial. 2026-11-09: corrected planning runThe corrected input wiring completed all 18 original scenarios with Copilot SDK, GPT-6.1 Sol, one trial per task, Waza 0.40.1 and embedded runtime 1.0.105. Fresh result: 8 passed, 10 failed. All 18 recorded deny-all, zero tool calls/events and zero denials. No shared setup errors were found. Benchmark exit was 1; the post-result policy verifier still ran and passed with exit 0. This is planning evidence, not live Azure or an A grade. Before the run, the actual embedded runtime was probed over its supported SDK stdio connection without creating a session or sending a prompt. An isolated
Native-syntax component: 13 pass, five unscored, zero established invalid-syntax findings. The five unscored outcomes are not passes. No skill source was changed during the trial. Source/configuration digests and the two supplied original artifacts remained unchanged; a separate configuration snapshot, identity record and both exit statuses are preserved locally. Reported usage: 18 turns, 18 premium requests, 107,460 input tokens, 9,532 output tokens, 22,500 cache-read and 84,906 cache-write tokens, 30.9943 AI credits, and 156.181 seconds elapsed. These are artifact-reported usage and latency, not verified currency charges or matched CLI performance. Next: correct the missing task-reference context and high-confidence evaluator representation defects offline, retain negative safety checks, then use the approved bounded budget only if those checks pass. This used the initial corrected invocation; six additional one-trial invocations remain available, not required. The three unseen holdouts and three-trial repeatability are not yet established. References: local immutable result |
Accept task-scoped local help, safer exact gateway reads and bounded readiness waits without relaxing approval, mutation or runtime gates. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Stop on preparation errors, require the isolated profile and a nonempty unique result path, and preserve benchmark versus policy-verifier statuses independently. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
TL;DR: The next run's console reported 12 of 18 passes, but our launch preparation lost the full result and policy evidence. That score cannot establish reliability, and paid execution is stopped. The launch path is now fail-fast and preserves both exit statuses; verify that exact path before resuming. 2026-11-09: isolated planning runs, context correction and stopped executionPlanning completion is not established. The first corrected run has complete Actual execution and evidenceBoth invocations used Waza v0.40.1, the Copilot SDK executor, embedded Copilot
Combined reported usage is 66.79035 AI credits and 36 premium requests. These The actual embedded runtime was probed through its supported SDK stdio APIs Corrections and failure classificationThe first run's ten failures included six missing native operation plans where Quality inputs now quote selected sections directly from the authoritative The grader now accepts recorded local version/help probes without letting them The next run reported behavior failures for approved Foundry onboarding, exact Shell failure and next stepMy preparation command used unavailable system The documented launch path now uses Further paid runs remain stopped because full execution evidence is missing, References: first corrected run journal; |
Preserve all 21 independent inputs and grader selections, validate exact combined inventory and exclude generated configuration from source control. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
TL;DR: The complete fresh run passed 13 of 21 scenarios. Most failures were equivalent facts or relevant scoped reads rejected by the eval; the consumer skill also needed clearer guidance for an optional typed gateway read. Those corrections replay at 20 of 21, not a new model score. Next, regenerate and freeze the corrected inputs for the next approved trial. Planning evaluation, 2026-11-09The latest complete trial passed 13/21: 11/18 original scenarios and 2/3 holdouts. This is development evidence, not an A grade or repeatability result. Executor: Waza 0.40.1, Copilot SDK with embedded runtime 1.0.105, GPT-6.1 Sol, one trial per task. Every task recorded requested The eight failed tasks divide into four fact/representation comparisons, three scoped-plan comparisons, and one consumer-guidance gap:
The evaluator corrections accept only lossless, bounded equivalents and enumerated reads. They retain exact gateway/account scope, both identity and Foundry User approvals, composite rather than independent counters, create-before-verification, alias read/update/read ordering and rejection of sibling mutations. Local core Azure help, not Azure calls, established the additional read-option and colored-JSON contracts. The source change does not teach evaluator labels or task answers. Offline replay of unchanged outputs: 20/21 behavior and 21/21 syntax pass. The remaining native-discovery failure is retained. This replay does not change the recorded fresh 13/21 score or test the skill correction. Regression checks include positive equivalents and negative cases for foreign scopes, broader roles, missing verification, wrong endpoints, table output and reordered/unrequested mutations. Usage: 21 turns/premium requests; 140,596 input and 12,354 output tokens; cache-read 26,250 and cache-write 114,283 tokens; 41.19985 AI credits; 179.355 seconds. Including the earlier complete run and the honestly recorded log-only invocation, assistant-owned usage is 107.9902 credits and 57 premium requests. The initial invocation plus two of six additional invocations have been used; four additional remain. The actual embedded runtime's session-local profile was checked without a model prompt: hooks disabled, no installed plugins or user MCP configuration, no device-managed startup keys, existing authentication. This and deny-all records are not an OS filesystem, credential or network sandbox. No global plugin configuration, live Azure/data-plane action or WorkIQ action was changed or invoked. Next: regenerate and check the corrected combined profile, freeze the candidate, then use the next authorized one-trial invocation. Completion still requires all 21 with complete policy evidence across three frozen trials. Holdouts already inspected during development must not be described as newly unseen acceptance. All raw results, configuration snapshots and prior replays remain local and immutable; none are attached here. Reference: retained combined result SHA256 |
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
TL;DR: The fresh run passed 17 of 21; the remaining failures were bounded wording differences and correctly scoped prerequisite reads, not identified setup or unsafe-action failures. Correcting those comparisons makes the saved-answer replay green, but does not change the fresh score. Next, run the frozen candidate through the recorded one-shot launcher and check consistency. Planning corrections and retained trial, 2026-11-09The next retained GPT-6.1 Sol trial passed 17/21: 15/18 originals and 2/3 holdouts. All 21 have complete deny-all evidence and zero calls/events/denials. Waza exited 1; the policy verifier exited 0. Source/configuration and all prior result artifacts were unchanged during execution. Waza 0.40.1 used Copilot SDK/runtime 1.0.105, one trial per task. Usage was 21 premium requests/turns, 142,330 input and 12,380 output tokens, cache-read 26,250/cache-write 116,017, 41.65935 AI credits and 180.577 seconds. Total assistant-owned usage so far is 149.64955 credits and 78 premium requests. Initial plus three additional invocations are used; three additional remain under the unchanged ceiling. The four failures were evaluator comparisons, not identified setup, policy or missing-skill behavior:
Corrections accept those bounded equivalents and exact scoped reads. Approval parsing is anchored to the intended identity/Foundry User operation and still requires both exact scopes; negation, user-assigned identity, added/broader roles and multiple action sentences fail. Missing-metadata equivalence is limited to the corresponding configuration fields, not private reachability. Foreign targets and unapproved mutations remain rejected. Offline replay: 21/21 behavior and 21/21 syntax. That is a separate replay, not a changed fresh score or proof that a new trial is green. Recorded execution now uses one canonical, one-shot The corrected candidate will be frozen for the next authorized invocation. If green, the two remaining invocations can test unchanged repeatability. No live Azure/data-plane/WorkIQ actions or global plugin changes are authorized or performed. Startup-profile/tool-policy evidence is still not an OS sandbox. The inspected holdouts remain a distinct development cohort, not newly unseen acceptance evidence. Reference: retained fresh-result SHA256 |
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
TL;DR: The fresh run passed 20 of 21; bootstrap failed because the eval rejected necessary reads of the selected Foundry account and deployments. Those reads are now accepted without permitting foreign scopes or unapproved imports. The offline replay is green; next, check the unchanged inputs in another authorized fresh run, not treat replay as repeatability. Planning evaluation journal: November 9, 2026The latest fresh trial passed 17/18 original scenarios and 3/3 holdouts, or 20/21 overall. Waza 0.40.1 used the Copilot SDK executor, embedded runtime 1.0.105, GPT-6.1 Sol and one trial per task. All 21 tasks reached the model. Syntax passed throughout. The sole behavior failure was the bootstrap grader rejecting read-only checks of the exact selected Foundry account and its deployments, not an unauthorized import or role grant. Those prerequisite reads are now explicitly enumerated for bootstrap. Regression cases reject a different subscription, RG or account, omitted subscription, deployment creation, missing gateway creation and moving pending Foundry import into the approved plan. No runtime skill or model input changed. The regenerated configuration's 23 files are byte-identical to this trial's configuration. Offline replay of its immutable outputs now passes 21/21 behavior and syntax checks; the recorded fresh score remains 20/21. All 21 results recorded deny-all policy and zero tool calls, events or denials. The benchmark exited 1 and the independently executed policy verifier exited 0. Frozen source/configuration and earlier artifacts remained unchanged. Sessionless SDK checks confirmed hooks disabled, no installed plugins or user MCP servers, no managed startup policy keys and existing authentication. These controls are not an OS filesystem, credential or network sandbox. This trial used 42.0811 AI credits, 21 premium requests/turns, 142,345 input tokens, 12,798 output tokens, 26,250 cache-read tokens and 116,032 cache-write tokens, over 192.570 seconds. Assistant-owned usage so far is 191.73065 credits and 99 premium requests/turns, including the previously disclosed console-only invocation. The initial authorized trial and four of six additional invocations have been used. Two additional invocations remain. There is no fresh all-green repeatability streak yet. Even if both remaining trials pass, three unchanged-candidate green trials cannot be established within this ceiling. The next bounded trial can establish fresh coverage after the prerequisite comparison correction; no extra authorization is inferred. The holdouts were independently authored but have been inspected during development. They are not newly blind acceptance evidence. No live Azure/data-plane calls, matched CLI performance baseline or Grade A claim is established. Evidence retained locally: immutable result |
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
TL;DR: Two fresh failures came from valid pending approval operations expressed outside the eval's sentence grammar, not an unauthorized grant or import. Paid execution is stopped rather than adding more wording exceptions. Next, separate typed operations and exact scopes from explanatory prose, then validate that correction before another approved run. Planning evaluation journal: November 9, 2026Execution is stopped; planning completion is not established. The latest fresh trial passed 17/18 original scenarios and 2/3 holdouts, or 19/21. The preceding trial passed 20/21. No all-green fresh trial or three-trial repeatability streak exists. The two remaining failures are approval-action representation comparisons in bootstrap and the Foundry account-role-scope holdout. Both outputs proposed the exact reviewed pending command, retained both exact approval scopes, reported no execution and made no tool calls. Their explanations distinguished an account-wide Foundry User grant from deployment-only scope using sentences outside the grader's anchored grammar. Native syntax passed all 21 tasks. No setup or policy failure was found. These observations do not convert the recorded failures into passes. The exact Foundry prerequisite-read omission found in the preceding trial was corrected, with tests rejecting foreign subscriptions/RGs/accounts, omitted subscription, deployment creation, missing gateway creation and unapproved import. All 23 regenerated model input/configuration files remained byte-identical to the preceding trial. Offline replay of that earlier immutable output is 21/21 behavior and syntax; the latest fresh output remains 19/21 behavior and 21/21 syntax. I stopped on repeated approval-sentence comparison failures instead of adding another wording special case and spending the remaining invocation. The unresolved design is to separate typed approval operations from explanatory prose while retaining exact command, identity, role and scope gates. No further rubric change or paid run has been made. The latest invocation used Waza 0.40.1, Copilot SDK / embedded runtime 1.0.105, GPT-6.1 Sol, one trial per each of 21 tasks. It consumed 42.0101 AI credits, 21 premium requests/turns, 142,341 input tokens, 12,728 output tokens, 26,250 cache-read tokens and 116,028 cache-write tokens, over 188.135 seconds. All 21 runs recorded deny-all with zero tool calls/events/denials and complete policy counts. Benchmark exit was 1; independent saved-result policy verification exited 0. Pre-run configuration identity and post-run hashes show source/configuration and earlier artifacts unchanged. The actual session-local SDK startup had hooks disabled, no installed plugins or user MCP servers, no managed startup policy keys and existing authentication. This is not an OS filesystem, credential or network sandbox. Budget: the initial corrected invocation plus five of six additional invocations have been used. One additional invocation remains unused. Assistant-owned totals, including the disclosed console-only run, are 233.74075 AI credits, 120 premium requests/turns and 1,060.317 seconds of reported suite duration. This ceiling is not a quota. The three holdouts were independently authored but inspected during development; they are not newly blind acceptance evidence. Live Azure/data-plane behavior, progressive plugin loading and matched CLI performance remain unmeasured and separately authorized. No Grade A claim is made. Evidence: latest immutable result |
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
TL;DR: Typed approvals passed the two previously failing approval scenarios. The fresh run still passed only 20 of 21 because the eval rejected the exact gateway-family safety read required by the skill. That read-list correction makes the offline replay green, not the fresh result. Further frozen trials need new authorization; unseen and live validation remain absent. Planning evaluation journal: November 9, 2026The final authorized fresh trial passed 17/18 original scenarios and 3/3 holdouts, or 20/21. Both previously failing approval scenarios passed with typed approval operations. The six-additional-invocation ceiling is exhausted; no further model calls are scheduled. Planning repeatability and Grade A remain unestablished. Approval checking now separates operations from explanation. An identity action declares its operation and identity type; a role action declares its operation, role and exact gateway principal. Each retains its exact approval scope. The universal input vocabulary covers assignment and removal, without identifying the operation or values required by a scenario. Those are derived from the public skill and supplied evidence. Regression tests reject removal/negated operations, user-assigned identities, broader/additional roles, foreign principals/scopes, extra properties, missing approvals and unapproved import entering the approved command plan. No sentence special cases were added. Historical sentence-format outputs and scores remain unchanged. The sole final recorded failure was the alias grader rejecting Offline replay of the immutable final outputs passes 21/21 behavior and 21/21 syntax after that read-list correction. All 23 regenerated input/configuration files remain byte-identical to the final trial's snapshot. The fresh score remains 20/21. This is a corrected comparison, not a new successful model run or three-trial streak. Execution provenance: Waza 0.40.1, Copilot SDK executor, embedded runtime 1.0.105, GPT-6.1 Sol, one trial for each of 21 tasks. All tasks reached the model; syntax passed throughout, and no session setup errors were recorded. All 21 runs had complete deny-all records and zero tool calls/events/denials. Benchmark exit was 1; independent saved-result policy verification exited 0. Pre-run identity/configuration snapshot and post-run hashes retained frozen source/configuration and unchanged prior artifacts. The actual session-local SDK startup checks confirmed hooks disabled, no installed plugins or user MCP servers, no managed startup policy keys and existing authentication. No global plugin changes, new login, Azure/data-plane or WorkIQ action calls occurred. This is not an OS filesystem, credential or network sandbox. The final trial consumed 42.65535 AI credits, 21 premium requests/turns, 144,346 input tokens, 12,872 output tokens, 26,250 cache-read tokens and 118,033 cache-write tokens, over 189.511 seconds. Assistant-owned cumulative usage is 276.39610 credits, 141 premium requests/turns and 1,249.828 seconds reported suite duration, including the disclosed console-only invocation. The initial trial plus all six additional invocations have been used. Validation after the final offline correction: 54 evaluation tests and 10 portable-plugin tests pass, plus Bash syntax, exact generated binding/input identity and diff checks. These are offline checks, not fresh quality executions. The authorized correction/evaluation pass ends at its ceiling. No all-green fresh trial exists; three unchanged-candidate green trials remain unverified. The three holdouts were independently authored but inspected during development, not newly blind acceptance evidence. Live CLI/data-plane behavior, progressive plugin loading and matched CLI performance still require separate authorization. Additional model execution also requires a new ceiling. Evidence retained locally: immutable result |
|
TL;DR: The first frozen run still failed bootstrap: it exposed both an identity-spelling comparison and an ambiguous single-deployment versus whole-account request. No import or access change was executed. Stop paid repeats, make the requested scope explicit, and normalize only equivalent identity spelling before restarting repeatability. 2026-11-09: first frozen repeatability trialThe first of the three newly authorized frozen trials passed 20/21: 17/18 original scenarios and 3/3 development-inspected holdouts. Syntax passed 21/21. Planning completion is still not established. Waza 0.40.1 used the Copilot SDK, embedded runtime 1.0.105, GPT-6.1 Sol and one trial per task. All 21 tasks reached inference. Independent post-result verification found deny-all on all 21, zero tool calls/events/denials and complete counts. No setup failure was found. Benchmark exit was 1; policy-verifier exit was 0. Source/configuration and prior artifact hashes were unchanged. The isolated profile had hooks disabled, zero installed plugins/user MCP servers and existing authentication. This is not an OS filesystem, credential or network sandbox. The sole failure was bootstrap, with two grader findings:
The plan kept both access changes pending, bound them to the exact gateway principal and Foundry account, and explicitly disclosed the all-deployment import. No Azure command was executed. These findings do not establish an unsafe skill mutation. They also do not justify calling the fresh result a pass: the request/oracle scope and identity representation need correction without weakening approval checks. Execution stopped on repeated bootstrap nonprogress rather than spending the other two frozen invocations. No grader, skill or generated input changed during or after the trial; no offline replay was used to relabel it. Three unchanged-candidate all-green trials remain unmet. Live CLI/data-plane and matched CLI performance evidence remain separately unauthorized. This invocation used 42.78910 AI credits, 21 premium requests/turns, 144,361 input tokens, 13,002 output tokens, 26,250 cache-read tokens, 118,048 cache-write tokens, and 181.589 seconds reported suite duration. Cumulative assistant-owned usage is 319.18520 credits, 162 premium requests/turns, and 1,431.417 seconds reported duration, including the previously disclosed console-only invocation. The original initial-plus-six allowance is used; 1/3 newly authorized frozen invocations used, 2 unused. Local provenance: immutable result |
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
TL;DR: Bootstrap now explicitly requests one existing deployment, with whole-account import tested separately. Equivalent identity case is accepted without relaxing role, principal, operation or approval scope. These are offline corrections: the saved answer still addressed the old request, and no fresh pass or repeatability is claimed. Next, prepare the corrected candidate for three separately authorized frozen trials. 2026-11-09: bootstrap scope and identity contract corrected offlineThe bootstrap input now explicitly limits optional onboarding to one selected existing deployment. The separate whole-account scenario continues to require explicit approval of both existing deployments and both external access changes. No default was broadened, and no paid run was made in this correction pass. This changes the bootstrap request, not the expected import scope to fit a model answer. The saved frozen output answered the prior ambiguous request; it is not evidence that the new input passes. All 23 regenerated configuration/input files were compared to the frozen snapshot: only Identity fact, action and principal fields now normalize case-only spellings of the same All 18 original requests plus three development variants were reviewed in one offline pass for hidden import/resource scope and identity-enum assumptions. No further input-scope or enum correction was justified. An all-task regression checks exact request/evidence binding, absence of grader metadata from prompts, and oracle subscription/RG/gateway/Foundry-account/deployment targets. The approved bulk-import scenario is checked separately from the newly explicit single-deployment case. Validation: 57 evaluation tests and 10 plugin tests pass, generated 21-task deny-all binding passes, pinned empty-sandbox guard/resource parity remains covered, Bash syntax and whitespace checks pass. These are offline checks; no new Waza, model, Azure or data-plane call was made. A separate replay preserves the immutable output and its old request: 20/21 behavior and 21/21 syntax. Identity spelling no longer fails; the pending account-wide operation remains rejected. The recorded fresh 20/21 is unchanged. Prior artifacts, identities, snapshots and replay histories are preserved. No new fresh score is claimed. The three historical Repeatability resets with this correction. Paid execution remains stopped, with two unused invocations from the last authorization. They cannot establish the required three consecutive all-green 21-task trials on this changed candidate. Before further spending approval: retain the passing offline tests and exact generated binding, recheck actual SDK startup isolation and narrow preflight, record fresh exclusive configuration/source identities, and authorize at least three unchanged-candidate one-trial invocations. Stop on setup/policy/evidence failures or repeated nonprogress. Live CLI/data-plane and matched CLI baselines still require separate authorization. Evidence: |
|
TL;DR: The corrected bootstrap scenario passed both fresh runs. The first run passed all 21 scenarios, but the next passed 19: the eval rejected a correctly scoped alias list and an equivalent composite-counter description. The alias plan also omitted a gateway-family safety check required by the skill, which the current grader does not independently enforce. Stop paid repeats, correct those checks offline, and test that safety omission before restarting consistency trials. One green run is not repeatability. Corrected frozen trials: November 9, 2026Both runs used the same corrected skill, grader, fixtures and generated configuration. All 21 scenarios reached inference in each run. No shared setup failure was found. These are fresh results, not rescored historical answers.
What still failsAlias routing: the exact selected-alias read/update/read sequence, three-model pool, unrequested fallback, priorities, weights, policies and callable are preserved. The eval rejects additional That comparison defect is not the only issue. The management skill also requires a scoped same-name gateway-family check before name-selected writes. This plan includes the typed gateway read but omits the family inventory. The current grader permits the family read but does not independently require it. Removing the rejected alias-list commands in an offline diagnostic makes the current grader pass even though this precaution is still missing. That diagnostic exposes a coverage gap; it is not a new score or evidence of full skill adherence. Policy accounting: the answer reports tokens versus requests correctly, rejects a billing hard-cap claim, and calls the partition These are two recorded semantic-grader failures, zero syntax failures and zero identified setup/tool-policy failures, with a separate demonstrated safety-preflight omission and grader-coverage gap in the alias plan. There was no Azure mutation, tool attempt, role grant or quota-bypass attempt. What to attempt nextReview the scoped-read and composite-partition checks together offline, rather than add unconstrained wording exceptions. Preserve negatives for foreign subscription/RG/gateway, sibling writes, missing or reordered alias mutations, and separate rather than composite counters. Require and test the gateway-family precaution for name-selected writes. A comparison fix alone must not turn that missing precaution into a safe-plan pass. Any skill/grader/fixture correction resets consistency evidence. The first corrected run was green; the next was not, so there is no three-trial streak. I did not spend the last authorized invocation: it cannot establish three consecutive green trials after these findings. Further paid execution is stopped, not scheduled as an automatic loop. The three extra scenarios have been repeatedly inspected during development. Their historical Execution boundary and provenance
Readable historical journalThe 14 existing evaluation updates now have human TL;DR openings, edited in place without deleting their detailed historical evidence. Two comments had displayed only local-file references; their prepared redacted findings are restored below the opening, with the original references retained in a publication note. Historical commands remain historical, not current rerun instructions. The separate manually written progress comment was left untouched. Initial findings; |
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…ations Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
TL;DR: The eval rejected valid scoped alias reads and a valid composite-counter description. Those comparisons are corrected. It also missed a real problem: the second trial's alias plan omitted the gateway-family check required before a name-selected write. The grader now requires that check, and the skill places the existing precaution next to execution. Next, validate this corrected candidate in fresh trials; old outputs cannot prove the revised guidance works. What changed
Offline evidence, not a new model trialThe two immutable GPT-6.1 Sol results retain their fresh scores: 21/21, then 19/21. Each ran the original 18 tasks plus three development variants once through Waza 0.40.1 and Copilot SDK/runtime 1.0.105. Both previously verified complete deny-all evidence, zero tool calls/events/denials and syntax 21/21, with no identified setup failure. A distinct, field-only offline replay against the corrected grader passes 21/21, then 20/21 behavior, with syntax 21/21 for both. The second output still fails its missing gateway-family preflight. The comparison corrections therefore do not award a pass to the omitted safety check. These replay counts are not fresh scores, repeatability evidence or validation of the repositioned skill text. The targeted offline suite passes 59 tests, and the plugin suite passes 10. Regression coverage includes missing/late family inspection, exact mutation/read-back ordering, scoped-list acceptance, foreign subscription/RG/gateway rejection, independent/extra counter dimensions, malformed objects, pinned path-guard parity and input/oracle isolation. The combined configuration was regenerated for GPT-6.1 Sol, one trial for each of 21 tasks; static binding still reports Request and synthetic-evidence meaning did not change. Selected skill context changes because the precaution moved. Any grader/skill correction resets the repeatability streak. The repeatedly inspected three variants remain development evidence, not genuinely unseen holdouts. Spending and next validationNo Waza invocation, model call, Azure/data-plane call, authentication flow or global plugin change occurred in this correction pass. Prior artifacts remain unchanged. The latest pair's historical usage remains 85.16745 AI credits, 42 premium requests/turns and 374.950 seconds; cumulative assistant-owned usage remains 404.35265 credits, 204 premium requests/turns and 1,806.367 seconds, including the disclosed console-only invocation. Paid execution remains stopped. One previously authorized invocation is unused. The minimum planning-completion evidence is three fresh passing invocations of the same corrected candidate across all 18 originals and three development variants, with complete execution/policy evidence. That needs authorization for two more invocations in addition to the remaining one; another correction would reset the streak. Genuine held-out evaluation, live gateway behavior, progressive loading and matched CLI performance remain absent and separately scoped. Earlier evaluation journals were revised in place with human TL;DR openings, preserving historical details. Their canonical links are indexed in the preceding journal. |
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
TL;DR: The latest trial failed because the eval rejected valid scoped reads, not because it found unauthorized execution. Those comparisons are corrected. A subsequent offline audit found a different grader gap: bootstrap and approved Foundry create plans could omit the required family check. That prerequisite is now enforced without weakening scope or approvals. Paid trials are stopped at the user's request. Fresh validation would still be needed for repeatability; the saved trial remains 19/21, while the corrected offline replay is 21/21. Fresh trial: 19/21, not repeatabilityThe frozen candidate passed 16/18 original scenarios and 3/3 development variants. Native syntax passed 21/21. All 21 tasks reached inference, with zero recorded session errors; the aggregate The actual result records Post-result verification passed all 21 requested-deny-all records, with no missing policy/count evidence and zero tool calls, tool events or denials. Benchmark exit was 1; policy-verifier exit was 0. Frozen source and all recorded prior artifacts remained unchanged during execution. No Azure/data-plane call, new authentication flow or global plugin change occurred. This is not an OS filesystem/credential/network sandbox. What failed, and what was corrected
Instead of adding that parent read separately to each future failing case, the grader now derives only two common parent-preflight reads from a management task's supplied subscription/RG/gateway identity: fully typed gateway The alias scenario still independently requires its family check before update and read-back after update. Correcting optional read coverage does not remove that safety requirement. No skill, request or evidence change was needed for these two failures; no oracle answers were added to agent inputs. Offline evidence and spendingA distinct field-only replay of this immutable output passes 21/21 behavior and 21/21 syntax under the corrected grader. That is not a new model score, validation of unseen inputs or an unchanged-candidate streak. The grader change resets repeatability. All 60 evaluation regressions and 10 plugin regressions pass; the combined configuration was regenerated and its static deny-all/task binding verified. Negative cases cover all management tasks' exact parent scope, omitted selectors, foreign families, query/write substitutions, strict alias preflight/order and runtime-only rejection. This trial consumed 42.86510 AI credits, 21 premium requests/turns and 188.562 seconds. It recorded 144,773 input tokens, 12,675 output tokens, 25,000 cache-read tokens and 119,710 cache-write tokens. The user explicitly authorized up to three frozen trials using one remaining invocation plus two additional. Only the first was used; the failure stopped this round, leaving two unspent. Globally, the initial corrected invocation plus ten additional invocations have been used. Cumulative assistant-owned usage is 447.21775 AI credits, 225 premium requests/turns and 1,994.929 seconds, including the previously disclosed console-only invocation. No subsequent paid trial has run. After the correction, the user explicitly selected Stop paid trials. The two remaining invocations will not run automatically. The implementation corrections are committed; the planning-completion criterion remains unmet rather than being relabeled as complete. Three consecutive passing unchanged-candidate trials are not achieved. The corrected grader needs three new green trials; two remaining invocations cannot establish that target. The three repeatedly inspected variants remain development scenarios, not genuine holdouts. Live gateway behavior, progressive loading and matched CLI performance remain absent and separately scoped. Offline consistency audit after the stop decisionThe audit covered all 21 tasks, including nine management scenarios and 47 read-comparison rows. It inspected 243 saved task/run records from 12 immutable results, including the operator artifact's setup failures. These are retained records, not 243 new inference calls. Historical command comparisons use the current rules as diagnostics only; the old bootstrap request previously had different import-scope meaning, so those comparisons do not turn historical outputs into fresh or current-input scores. The two new coverage defects were in bootstrap and approved whole-account Foundry onboarding: family inventory was accepted as an optional read, but its absence did not independently fail the plan. The existing skill already requires that precaution before name-selected writes. The grader now uses the shared exact parent-read definitions for both equivalence and the pre-write prerequisite. Both scenarios explicitly include family inventory. A missing or late check fails even if a future scenario forgets to add it to its operation list. The alias check remains mandatory. Equivalent flag order, quoting and verified short forms match without treating parent reads as child reads. A stronger typed gateway ID may match a name-selected gateway read only with the same explicit subscription/RG/name; the reverse cannot weaken a typed binding. Child reads and all required operations remain covered. Prerequisites precede writes and read-back follows; the approved Foundry identity/model read-backs can occur in either order. Foreign/omitted selectors, legacy-family IDs, extra queries or writes, missing operations, broader roles, foreign principals, negated/combined identities and independent counters remain rejected. Runtime-only scenarios do not gain management discovery. Synthetic checks also found a false negative for repeating the exact already-required alias read before its update. The shared read predicate now accepts repetitions of approved, exactly scoped required reads, not arbitrary native GETs. An early read cannot satisfy a later required read-back, remove a prerequisite, authorize pending work or substitute for a write. This does not extend asset scope or secret/mutation permissions. An explicit empty behavior-command boundary does not inherit child reads from the historical command list. All 63 offline evaluation regressions and 10 plugin regressions pass. A new distinct replay still passes 21/21 behavior, and the saved result's syntax and complete deny-all evidence remain 21/21 with zero activity. All 23 generated configuration/input files are byte-identical to the frozen snapshot. No skill text, scenario request, synthetic evidence, import scope or approval requirement changed; no oracle answers were added to inputs. The grader/oracle change resets repeatability again. Raw scores, costs and earlier replay history remain unchanged. This suite evaluates application of supplied skill/reference/help text to fixed synthetic evidence. It does not test product skill discovery, selection or progressive loading, an OS sandbox, live Azure/data-plane effects or a matched CLI-performance baseline. Its CLI equivalence rules are bounded, not a complete interpreter for arbitrary queries/projections. Earlier saved plans genuinely omit family preflight; that remains a behavior failure, rather than another representation exception. No additional current skill defect was established by this audit. The earlier skill correction moved existing family-preflight guidance next to execution; this audit changes only evaluator coverage. No Waza invocation, model call, Azure call or additional credit spend occurred during the audit. The user-selected stop remains in effect; both unused invocations remain unspent. References
|
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
Feedback factored in to #55 |
Azure AI Gateway's plugin should complete the documented CLI and data-plane scenarios reliably, without unnecessary discovery or large instruction loads. This PR removes known obstacles to that goal and prepares repeatable evaluation. It is an A-targeting candidate, not an A-rated result. Local checks establish packaging and deterministic behavior, not reliable live gateway behavior or performance close to the CLI.
The corrected candidate passed 21/21 in one fresh frozen trial, then 16/18 originals and 3/3 development variants, or 19/21, in the next unchanged trial. Bootstrap passed both. Both trials reached inference throughout, passed native syntax 21/21 and retained complete deny-all evidence with zero tool calls/events/denials. There was no identified setup failure. The latest eval rejects correctly scoped alias lists and an equivalent composite-counter label. Separately, the alias plan omits the management skill's same-name gateway-family precaution; the current grader permits that read but does not independently require it. A comparison fix alone must not certify the missing precaution.
The bootstrap input explicitly limits optional import to one selected existing deployment; separately approved whole-account onboarding remains distinct. Case-only identity normalization retains exact operation, role, principal and scope checks. The earlier ambiguous-request output and offline replay remain historical, not fresh corrected-input evidence. The three historical
holdouttasks are repeatedly inspected development variants, not genuine held-out validation.Paid execution is stopped. Two of the three latest approved frozen invocations were executed; one remains unused and cannot establish three consecutive green trials after these findings. Next, correct the scoped-read/composite-partition comparisons offline, preserve their negative safety tests and resolve the alias family-preflight omission before restarting consistency evidence. No post-run rubric or input change relabels the fresh 19/21 score. The detailed historical journals now have human TL;DR openings, edited in place without deleting evidence.
Follow-up to #32, targeting
feature/apim-migration-native-sync-20261105so these improvements can be reviewed and merged into the original PR first. The original branch and native-stack metadata are unchanged.What changes
show --ids; root inspection is not mandatory.evals/waza/run_planning.sh. The actual launcher is inert-tested for empty/occupied results, missing identity/Python, changed source, unapproved model and saved failing benchmarks. It requires a prepared identity/configuration snapshot before Waza, keeps benchmark and policy-check statuses separately and verifies saved results even when the benchmark exits nonzero. It has no retries, authentication flow or orchestration loop.Management entry size decreases from 13,445 to 9,752 bytes (149 lines); consumer entry size decreases from 33,131 to 6,767 bytes (101 lines). These are context-size measurements, not measured latency or quality gains. Detailed runtime guidance is retained in the reference. Evaluation must check whether progressive loading improves efficiency without causing important instructions to be missed. The new fixed-input planning profile does not measure progressive loading.
What is needed for an A
The target is reliable completion of major documented CLI scenarios and real gateway data-plane use, with performance reasonably close to equivalent CLI work. A passing synthetic planning suite alone cannot establish that.
Before assigning an A, require repeated agent scenario execution with all behavior and approval boundaries passing, observed model and MCP integration, relevant policy and operational scenarios, and matched CLI tool-count and latency evidence. Review task completion, CLI/contract accuracy, least-privilege selection, approval/recovery handling, and clear uncertainty reporting. The proposed rubric and paired-baseline budgets are in the evaluation guide; they remain criteria to test, not achieved results. Genuine held-out evaluation is absent; the three repeatedly inspected variants provide development evidence only.
Waza setup and next runs
Run from this PR's worktree, using a verified isolated profile and new processes. The user authorized assistant-owned execution of one corrected 18-task trial plus up to six additional one-trial invocations, all GPT-6.1 Sol. That allowance was used. The user then authorized up to three additional frozen trials; one was used before the bootstrap scope/identity correction. A subsequent explicit approval allowed three corrected frozen trials using the two remaining invocations plus one additional. Two were executed, the first 21/21 and the next 19/21. One invocation is unused. Globally, the initial trial plus nine additional invocations have been used. Paid execution stopped on the latest comparison failures and separate alias safety-preflight gap; one remaining invocation cannot establish the three-green target. No live Azure/data-plane calls or global plugin changes were authorized or made. Correct and test the demonstrated issues offline before seeking sufficient unchanged-candidate validation; do not spend remaining budget merely to chase a green result.
The operator's reported WorkIQ-only configuration did not establish the embedded runtime's startup boundary. That blocker was resolved through supported session-local
COPILOT_HOMEisolation, without modifying global installations. Actual non-model SDK stdio probes of embedded Copilot 1.0.105 confirmed explicitdisableAllHooks: true, zero installed plugins, zero user MCP servers, no device-managed startup keys and existing authentication. The planning agent separately denies all model tools. This is not an OS filesystem, credential or network sandbox and does not make account policy immutable.Use
uv run --no-project --with pyyaml --with jsonschemafor Python dependencies and grader subprocesses. Waza itself is the standalone Go binary, not a Python package. Installed Waza0.40.1was verified locally after initial PR publication. Disable automatic updates for reproducibility; do not treat environment flags as permission enforcement.Offline checks and mock harness
The offline validation uses the existing Python environment:
PYTHONDONTWRITEBYTECODE=1 python3 -m unittest discover -s evals/waza -p 'test_*.py' -vThe operator commands use uv. Supplied smoke and quality artifacts predate the corrected prompts and rubric. Read-only replay preserves the original result and reports historical rubric revisions separately:
If the operator separately chooses another mock plumbing check, use a new filename rather than overwriting the supplied smoke artifact:
Stop on failed validation or preflight. Mock execution checks harness wiring and synthetic assertions, not assistant quality, secret handling, or cloud safety. The planning-only JSON checker below does not accept the mock profile. Strict preflight fails unresolved post-run boundary checks even for mock fixtures; a mock operator must review the non-strict report rather than treat it as a runtime assurance.
Real-model evaluation
The supplied
gpt-6.1-sol/ Copilot SDK artifact contains one trial per scenario (18 trials), with 0 original passes, 18 contract failures and no harness errors. The previous corrected-rubric replay remains 4/18; it is not a new trial of revised instructions. Review of the remaining 14 first failures found ten typed representation mismatches, one endpoint-versus-callable selector mismatch, one flattened approval-pending optional plan, and two arbitrary evaluator marker omissions. That classification does not certify every other field or establish unsafe execution. No narrative string is coerced into a successful boolean or execution approval.The historical offline component assessment retains original evidence, including the old runtime URL: behavior has 7 pass, 2 contract failures and 9 unscored; representation/native syntax has 7 pass, 0 fail and 11 unscored. The two behavior-contract failures concern the bootstrap's flat conditional plan and the approved Foundry plan's additional reads/unmatched ordered command representation, not established unauthorized executions. Unknown facts and unrecorded native syntax remain unscored. Detailed REDACTED findings, field-only replay diagnostics and reported usage are in the journal linked below. No runtime skill rewrite was justified by those baseline outputs.
Keep
gpt-6.1-solfixed with one trial per task per invocation. The planning target is all 18 originals plus three development variants passing with complete policy evidence across three frozen trials; a change resets the repeatability streak. That target is unmet. The bounded allowance is a ceiling, not a quota. No second model, Claude model, automated workflow or unbounded rerun loop is used.Regenerate configurations made before the evidence-resource and task-context corrections. Each task now supplies one legitimate
evidence.jsonresource identical to inline synthetic evidence; selected recipe sections come directly from the runtime skill references, not oracle answers. The prior inline-only profile failed the pinned empty-sandbox relative-path guard. The corrected profile passed exact schema/binding and narrowly checked non-strict preflight before execution. The operator's earlier strict attempt stopped before any model trial, but preflight alone never verifies runtime enforcement. Configurations predating the latest bootstrap scope correction also require regeneration; the updated combined config was statically verified and executed in the two corrected frozen trials.The original 18-task preparation commands are:
Do not use
--strictfor this planning suite: pinned preflight never starts a runtime or program grader. Default exits 1 for invalid configuration; strict additionally fails all unresolved/unsupported diagnostics. There is no per-warning waiver flag. The JSON checker permits onlyboundary.requested_policy,grader.subprocess,boundary.post_runandruntime.availabilityas unresolved warnings, while requiring exact complete enabled-task inventory and the selected requested-deny-all policy/expected grader dependencies. It rejects invalid, unsupported or unexpected diagnostics/dependencies and missing warning inventory. A successful check means reviewed offline prerequisites, not runtime readiness or verified enforcement. Stop on any checker failure, then make the operator's runtime/account/environment and cost decision separately.The supported planning boundary is configured:
planning.agent.mddeclarestools: []; generatedskill: aigateway-planning,skill_directories: [".."]andinject_skill_body: truebind it through Waza v0.40.1's agent resolution. Do not disable skills or substitute a direct filename: that can bypass policy resolution. The pinned executor applies an empty SDK tool filter, fail-closed permission callback and pre-tool-use denial hook. All model-initiated WorkIQ/MCP/action, shell, network and file/grader-answer reads are denied. Selected skill text and synthetic evidence are supplied inline, with no grader answers. The generated verifier rejects stale prompts, extra tasks and shadowed/modified policy binding.The first corrected result records all 18 runs as
deny_allwith zero calls/events/denials; post-result verification passed. The later log-only run has no retained per-task policy records and cannot inherit that verification. Eight subsequent complete combined results retain complete 21-task deny-all evidence and zero activity. The supported SDK startup probe checks the actual isolated profile, unlike standalone CLI inventory. Waza's pinned stdio startup passes the process environment, so retainCOPILOT_HOMEon execution. This still does not confine the process filesystem, credentials, caches or provider transport. Stop for unexpected login, installation or startup policy changes; no personal configuration or credentials are copied into fixtures or logs.Combined selection uses
author.py --quality --authorized --model gpt-6.1-sol --trials 1 --include-holdouts, withauthorized-combined/eval.yamland the same binding/preflight checks. It preserves the original 18 and separately authored three-task selection. Across the final original-budget trial and first newly authorized frozen trial all 23 generated files, including 21 model inputs, were byte-identical. Grader corrections were not exposed as oracle answers or scenario recipes. After the offline scope correction, only the bootstrap task differs from that snapshot; all-task tests bind exact request/evidence and oracle subscription/RG/gateway/Foundry/deployment targets. The two corrected frozen trials retained the same skill, grader, fixtures and generated inputs throughout both executions.Paid execution is currently stopped. If a future trial is explicitly resumed, use the tracked, inert-tested
evals/waza/run_planning.sh, not ad hoc shell reconstruction. Before it accepts a fresh result path, prepare its exclusive.identity.jsonand.configuration/snapshot, frozen source/configuration and immutable-prior hashes, approved model/trials and actual isolated-profile evidence. The recorded invocation takes this form, with a fresh, prepared result path rather than a previously saved artifact:The launcher refuses an empty/occupied output, missing receipt, changed inputs or unapproved model before Waza. It always checks an existing saved result after benchmark failure, captures both exit statuses, verifies source/prior immutability and retains result SHA-256. A benchmark failure is not hidden by successful policy verification. No retry is automatic.
Never overwrite supplied or prior result artifacts. My historical unchecked system
pythonpreparation failed, but the shell continued with an empty--output; v0.40.1 ran the benchmark without saving a full result. The ignored aggregate log is preserved, not fabricated into a result. Bare systempythonis unavailable; preparation now uses uv-managed Python. Post-run verification requires recordeddeny_allon every run and zero tool calls/events/denials; missing evidence fails. Review blocked attempts even when prevented. The fixed-input profile does not measure plugin discovery, progressive loading, live gateway behavior or CLI-relative performance. No A claim is established.Three separate development variants vary the Messages/Api-Key contract, selected-deployment identity/role approval and changed-source add-only sync with denied discovery. They were separately authored and statically validated, then included in eight complete combined executions. They have now been repeatedly inspected during development and are not genuine held-out evaluation. Historical
holdoutfilenames, IDs and flags remain for compatibility. A baseline replay cannot validate them. A separate three-task configuration remains available throughauthor.py --quality --authorized --model gpt-6.1-sol --trials 1 --holdoutsandauthorized-holdouts/eval.yaml; selection itself never authorizes another invocation or expands the ceiling.Validation evidence
tests/migrate; deterministic helper implementation unchangedtests/feedbackaigateway4.1.0, four skills and six adapters loaded using--plugin-dirwith isolatedCOPILOT_HOME; no global installation changed by that check; not rerun in these correction passesgit diff --checkpassed; remote source and follow-up refs verified0.40.1; uv and standalone Copilot available; operator's WorkIQ-only report was not sufficient startup evidence, replaced by actual isolated SDK checks&&prevented a new model trialLive routing, protocol interoperability, policy enforcement, network readiness, and telemetry ingestion remain unverified. See evaluation guidance for authored versus executed evidence, authorization, isolation, and matched-baseline requirements.
Open questions
References and recorded Git provenance
v0.40.1, commite0c251df896bee4e0f9b38a693a6105bf7eadd19, recorded inevals/waza/sources.json; full schemas and upstream MIT license are vendored with digest checks. Static compatibility does not establish harness acceptance.7874ce7f7c0f0674402d55b7de7eefacf8c705627c6b1f42b07a63974da98612f68bad4a3735aa14155817423804097eaa58962c037cb55bd249fe20paulyuk-ai-gateway-skill-improvementsNative-stack registration has not been performed.