Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,10 @@ The release workflow uses a tag's section here as its release notes (a
hand-written `docs/releases/<tag>.md` wins if present), so keeping this file
current is part of cutting a release.

## v0.302.74.77

- Prefer `rlm` for independent file reads, JSON row counts, and field extraction. Direct tools, shell fallback, and `--no-rlm` remain available (#1915).

## v0.302.74.76

- Raise `gpt-6.2-astra` subscription context window to 500k on `codex` and `chatgpt-new` (offline rows, subscription clamp, and live catalog discovery when `max_context_window` advertises a wider window; ADR 0292 addendum).
Expand Down
7 changes: 7 additions & 0 deletions docs/adr/0351-prefer-rlm-for-supported-work.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# 0351. Prefer RLM for supported work

Status: accepted

[0140](0140-rlm-is-listed-when-available.md) makes RLM available from the first turn, but availability alone can lead the model to start a shell process for work RLM already handles.

Lead with a narrow preference in the shared RLM tool description, not in the always-on system prompt. Prefer one RLM call for independent reads, JSON row counts, and field projection. Keep direct tools and the shell available for builds, tests, Git commands, and transforms RLM cannot express. Keep `--old` and `--no-rlm` unchanged. Forcing tool choice or rewriting shell commands would interfere with legitimate fallbacks.
1 change: 1 addition & 0 deletions docs/adr/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -375,6 +375,7 @@ record only when you need the evidence or the edge cases.
| [0348](0348-persistent-python-repl-and-unharnessed-mode.md) | Persistent per-owner Python 3 worker (`python_repl`, folded in `native_fold`) gives sessions cross-turn globals, `_` expression results, fd 1/2 capture, and timeout resilience without turn-1 schema overhead; opt-in `--unharnessed` / `/unharnessed` (`CLI > env > .harness/settings.json`, ACP/JSON parity) exposes only `python` with a single-tool prompt, while `--no-local-tools`, `plan_mode`, and hard policies block both. |
| [0349](0349-native-flint-runtime-is-external.md) | Native Flint is a built-in local OpenAI-compatible provider, while its runtime and model files remain separately installed; use only validated loopback endpoints and never cloud-fallback. |
| [0350](0350-nonblocking-catalog-refresh-and-slash-refresh.md) | Dynamic provider catalogs (including Codex) stay 0 ms non-blocking at boot while stale-while-revalidating in the background past 5 minutes and activating landed caches in-session; `graff/models` adds bounded `{"force": true}`, and `/refresh` refreshes all signed-in catalogs across REPL and ACP. |
| [0351](0351-prefer-rlm-for-supported-work.md) | The shared RLM catalog entry prefers RLM for independent reads, JSON row counts, and field projection. It does not force tool choice or remove direct tools or the shell fallback. |

## When to write one

Expand Down
2 changes: 2 additions & 0 deletions evals/harness_behavior.jsonl
Original file line number Diff line number Diff line change
Expand Up @@ -118,6 +118,8 @@
{"args": [], "env": {"GRAFF_NO_CODEDB_GUARD": "1"}, "prompt": "Run the scripted RLM ordering fixture and report only observed results.", "workspace_seed": {"target.txt": "old"}, "id": "rlm-nested-shell-failure-status", "why": "A nested shell failure propagates error status and prevents later nested mutation.", "script": [{"tool": "rlm", "arguments": {"code": "print(bash(\"printf NESTED_FAILURE; exit 7\"), write_file(path=\"marker\", content=\"ran\"))"}}, {"text": "The nested command failed."}], "verify_command": ["python3", "-c", "from pathlib import Path; assert not Path('marker').exists()"], "assert": [{"event": {"type": "tool_call_finished", "name": "rlm", "is_error": true}}, {"request_contains": {"index": 1, "text": "NESTED_FAILURE"}}]}
{"args": [], "env": {"GRAFF_NO_CODEDB_GUARD": "1"}, "prompt": "Run the scripted RLM ordering fixture and report only observed results.", "workspace_seed": {"target.txt": "old"}, "id": "rlm-pending-host-blocks-tail-completion", "why": "Pending nested background work preserves pending status, stops later mutation and cannot satisfy same-batch completion. The headless run then waits for the job and continues with its exit instead of ending.", "script": [{"tools": [{"tool": "rlm", "arguments": {"code": "print(bash(command=\"sleep 0.2; exit 0\", run_in_background=true), write_file(path=\"marker\", content=\"ran\"))"}}, {"tool": "attempt_completion", "arguments": {"result": "PREMATURE_COMPLETION"}}]}, {"text": "Verification is still pending."}, {"text": "Verification finished: the check exited 0."}], "verify_command": ["python3", "-c", "from pathlib import Path; assert not Path('marker').exists()"], "assert": [{"event": {"type": "tool_call_finished", "name": "rlm", "is_error": false}}, {"event": {"type": "tool_call_finished", "name": "attempt_completion", "is_error": true}}, {"no_event": {"type": "tool_call_finished", "name": "attempt_completion", "is_error": false}}, {"request_contains": {"index": 2, "text": "exited 0: sleep 0.2; exit 0"}}, {"final_text_contains": "Verification finished: the check exited 0."}]}
{"id": "rlm-indexed-print-refuses-and-recovers", "why": "Unsupported Python-style result indexing must report an error without pretending to execute the nested host or executing later side effects; the supported text binding form recovers.", "args": ["--lean"], "prompt": "Run the scripted checks and report only observed results.", "script": [{"tool": "rlm", "arguments": {"code": "print(bash(\"printf ran > nested-marker\")[\"stdout\"])\nx = write_file(path=\"tail-marker\", content=\"ran\")"}}, {"tool": "rlm", "arguments": {"code": "r = bash(\"printf VERIFIED\")\nprint(r)"}}, {"text": "Verified using the supported text result."}], "verify_command": ["python3", "-c", "from pathlib import Path; assert not Path(\"nested-marker\").exists(); assert not Path(\"tail-marker\").exists()"], "assert": [{"event": {"type": "tool_call_finished", "name": "rlm", "is_error": true}}, {"events_at_most": {"match": {"type": "tool_call_finished", "is_error": true}, "count": 1}}, {"request_contains": {"index": 1, "text": "host results are text"}}, {"event": {"type": "tool_result", "name": "rlm", "is_error": false, "text": "VERIFIED"}}, {"final_text_contains": "Verified using the supported text result."}]}
{"id":"rlm-routing-preference","why":"The first request advertises a narrow RLM preference without removing the shell fallback.","args":[],"prompt":"Read the synthetic JSON fixture and return its row count.","workspace_seed":{"rows.json":"[{\"id\":1},{\"id\":2}]"},"script":[{"tool":"rlm","arguments":{"code":"rows = read_file(\"rows.json\")\nprint(len(rows))"}},{"text":"2"}],"assert":[{"request_contains":{"index":0,"text":"Prefer one RLM call for independent reads"}},{"request_contains":{"index":0,"text":"Use shell for builds, tests, git, and unsupported transforms"}},{"event":{"type":"tool_result","name":"rlm","is_error":false,"text":"2"}}]}
{"id":"rlm-routing-structured-opt-out","why":"Structured-only mode does not advertise the RLM preference.","args":["--old"],"prompt":"Read the synthetic fixture.","workspace_seed":{"rows.json":"[]"},"script":[{"tool":"read_file","arguments":{"path":"rows.json"}},{"text":"The array is empty."}],"assert":[{"request_lacks":{"index":0,"text":"Prefer one RLM call for independent reads"}},{"event":{"type":"tool_result","name":"read_file","is_error":false,"text":"[]"}}]}
{"id": "gateway-failure-usage-unknown", "why": "A typed gateway failure with no usage must preserve the provider diagnostic and label totals as incomplete instead of implying a free zero-token call.", "args": ["-p", "summarize the fixture"], "prompt": "summarize the fixture", "script": [{"text": "partial", "stream_error": {"type": "gateway_error", "code": "request_failed", "message": "fixture request failed"}}], "expected_exit": 1, "assert": [{"stderr_contains": "gateway_error request_failed"}, {"stderr_contains": "known subtotal: 0 api call(s)"}, {"stderr_contains": "1 failed request attempt(s) without usage (tokens and cost unknown)"}]}
{"id": "gateway-retry-usage-remains-incomplete", "why": "A recovered retry retains known successful usage while the failed attempt remains explicitly unknown.", "prompt": "summarize the fixture", "script": [{"text": "partial", "stream_error": {"type": "api_error", "message": "Internal error during token parsing"}}, {"text": "recovered fixture"}], "assert": [{"event": {"type": "turn", "usage_complete": false, "unreported_failed_attempts": 1, "api_calls": 1}}, {"final_text_contains": "recovered fixture"}]}
{"id":"terminal-model-unavailable-does-not-retry","why":"#1191: a structured model_unavailable error is terminal when the gateway cannot serve the chosen model; a retry would repeat the same failure. Keep the request id so support can find the failed call.","prompt":"Say hello","script":[{"stream_error":{"type":"service_unavailable","code":"model_unavailable","message":"This model is not available right now. Choose another model or contact Codegraff support with the request ID.","request_id":"cg_req_fixture"}}],"assert":[{"events_at_most":{"match":{"type":"model_call_started"},"count":1}},{"event":{"type":"error","message":"Model lmstudio is currently unavailable. Switch models with /model, or try again later. Technical details — api error: (service_unavailable model_unavailable) This model is not available right now. Choose another model or contact Codegraff support with the request ID. [cg_req_fixture]"}},{"no_event":{"type":"model_call_finished","ok":true}}]}
Expand Down
Loading
Loading