You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat(inference): opt-in native tool calls for local runtimes (local_ai.native_tool_calling) - #8680
Teeny's builder agent (a cron agent job) runs glm-5.3-flash on Surplus through the local-openai runtime. Every local runtime is treated as having no native tools, in three places: TurnModelSource::supports_native_tool_calling, the crate-native build path (!is_local), and tinyinference's build_local_runtime_chat_model. So the session renders the JSON-in-tag text protocol and the harness strips tools off the wire.
On Surplus that fails. In builder run 1957 the first call worked, and every later call came back with the model's whole {"arguments": {...}, "name": "shell"} body as the tool name: unknown tool \` (arguments: {})`. The model spent the rest of the run reasoning about the format, and every builder job ended "abandoned: the builder never reported back".
Fed the exact failing bodies (arguments before name, arguments as a JSON string, multi-line), tinytools' text parser decodes every one correctly, in batch and in 1/3/7/16-char streamed fragments. The bad calls did not come from our parser.
Probes against Surplus (glm-5.3-flash, OpenAI-compatible), run on the Teeny box:
No tools, model asked to emit JSON-in-tag: no tool_calls; the text stays text.
tools=[shell], a normal ask: a clean native call, finish_reason=tool_calls, id call_7df8…, name shell, args {"command":"ls"}. Native tool calling works for this model on Surplus.
tools sent plus "don't use native calls, write JSON-in-tag", non-stream: content came back verbatim, tool_calls null.
Same as 3 with stream=true: only thinking chunks and no answer. Surplus attaches response_error: {format_error, content_validation: ["valid_think_chunk"], tool_calls_validation: [], extra_tags: ["only_thinking"]}. Surplus validates and rewrites output server-side, so the text protocol plus thinking is fragile there.
tools sent, native calls wanted, stream=true: finish_reason=tool_calls, a clean shell call with {"command":"ls -la"}, and response_error.tool_calls_validation=["valid_tool_call"]. Native tool calling also works on the streamed path, which is the one local-openai uses.
What
local_ai.native_tool_calling: bool (serde default false). Settable through config.update_local_ai_settings and inference.update_local_settings, and declared in both controller schemas.
One helper, provider_string_native_tools: cloud providers get native tools, local runtimes only when the flag is set. Session assembly (supports_native_tool_calling) and the crate-native turn build both use it.
local_runtime.rs builds through build_local_runtime_chat_model_with_native_tools(…, config.local_ai.native_tool_calling), so the model's profile agrees with the session.
config::ops::tests::model_and_local_ai_tests::apply_local_ai_settings_updates_lm_studio_provider_fields now also sets and asserts native_tool_calling.
RED: error[E0609]: no field native_tool_calling on type LocalAiConfig and error[E0560]: LocalAiSettingsPatch has no field named native_tool_calling. GREEN: all three pass.
cargo test -p openhuman --lib -- config:: inference:: schema: 16 tests fail on this box, and the same 16 fail on main (comm of the two failure lists is empty). They come from the box's security module being unavailable, for example apply_local_ai_settings_persists_api_key fails with Failed to encrypt local_ai.api_key: security module unavailable: secret approval unavailable. Clippy shows nothing in the touched files.
Consumer: teenydroid/brain sets native_tool_calling: true in bin/spine for its Surplus endpoint.
Summary by CodeRabbit
New Features
Local AI settings now include an option to enable native tool calling. It is off by default; when enabled, supported local models can use native tool calls instead of the text-based protocol. You can update this setting alongside other local AI settings, and its value is retained when applying configuration changes.
Add tests asserting that local-openai endpoints report native tool calling only when the operator opts in, both at session assembly and on the built model profile, and that the local AI settings patch applies the flag. This guards against gateways misreading the text protocol's tool call bodies.
Auto-committed-on: cuttlefish
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Adds a `native_tool_calling` setting to the local AI config, exposed through the settings update RPC and schema, which sends native `tools` to the local runtime and reads structured tool calls instead of the JSON-in-tag text protocol. It defaults to off because many Ollama and LM Studio models reject or mishandle `tools`, but hosted OpenAI-compatible gateways behind `local-openai` need it since their template parsing can return the tool body as the tool name.
Auto-committed-on: cuttlefish
Co-authored-by: Medulla <medulla@tinyhumans.ai>
⚠️Review of 85e21d441ca0 stopped because the pull request closed or changed.
The last completed review (6c410264d6e17c54ff500f45e597777f408e4eaf) is shown below.
medium: Cover the cloud half of provider_string_native_tools (crates/openhuman-core/src/inference/turn_models.rs:339) Problem: `provider_string_native_tools` returns `true` for every non-local provider string regardless of `local_ai.native_tool_calling`: Suggested fix:
No concrete fix was supplied.
Verdict: Ready for maintainer review · Priority: medium Coverage: Reviewed 12 of 13 files; not reviewed: vendor/tinyagent….
Diagnostics and supporting details
Full summary
Pull request purpose (title): feat(inference): opt-in native tool calls for local runtimes (local_ai.native_tool_calling). Tiny Sweeper completed its review; deterministic results follow.
Unreviewed paths
vendor/tinyagents
Review snapshot
Change surface
Files
Review signal
Count
Production
9
Active findings
0
Tests
3
Noted findings
1
Documentation
0
Resolved findings
0
Configuration
0
Pending checks/questions
0
Completeness: Complete Test assessment: Test coverage is assessed from changed tests and lane evidence; execution is not claimed without trusted check data.
What changed
Local runtimes previously never received native tools in three places: `TurnModelSource::supports_native_tool_calling`, the crate-native turn build (previously `!is_local`), and tinyinference's local runtime model builder. This PR adds a serde-defaulted `local_ai.native_tool_calling` boolean to `LocalAiConfig` that is settable through `apply_local_ai_settings` / the update controllers and declared in the inference controller schema and catalog. A new helper `provider_string_native_tools` decides native-tool support as cloud providers yes, local runtimes only when the flag is set; session assembly and the crate-native turn build both call it, and `local_runtime.rs` now builds through `build_local_runtime_chat_model_with_native_tools(…, config.local_ai.native_tool_calling)` so the model profile matches the session. The default remains off, so existing local setups are unaffected.
Features
Added — Opt-in `local_ai.native_tool_calling` config flag: Adds a default-off boolean that lets operators choose native tool calls for local runtimes instead of the JSON-in-tag text protocol; existing Ollama/LM Studio behavior is unchanged because the default is false. (crates/openhuman-core/src/config/schema/local_ai.rspub struct LocalAiConfig {, crates/openhuman-core/src/config/schema/local_ai.rsimpl Default for LocalAiConfig {)
Added — Config patch field and apply path for `native_tool_calling`: Allows the flag to be set through the settings update path, logging the change when applied. (crates/openhuman-core/src/config/ops/model.rspub struct LocalAiSettingsPatch {, crates/openhuman-core/src/config/ops/model.rspub async fn apply_local_ai_settings()
Added — Controller schema exposure of `native_tool_calling`: Declares the flag as an optional boolean input so RPC callers can set it, and the field is accepted by both settings update controllers. (crates/openhuman-core/src/config/schemas/schema_defs/inference.rspub(super) fn lookup(function: &str) -> Option<ControllerSchema> {, crates/openhuman-core/src/inference/schemas/catalog.rspub fn schemas(function: &str) -> ControllerSchema {, crates/openhuman-core/src/config/schemas/helpers.rspub(super) struct LocalAiSettingsUpdate {, crates/openhuman-core/src/inference/schemas/settings_handlers.rspub(super) struct InferenceUpdateLocalSettingsParams {)
Modified — `provider_string_native_tools` helper drives native-tool decisions: Session assembly and the crate-native turn build now share one predicate returning true for cloud providers and true for local runtimes only when the operator opted in, keeping session capability, harness dialect and wire behavior consistent. (crates/openhuman-core/src/inference/turn_models.rsimpl TurnModelSource {)
Modified — Local runtime model built with native-tools builder and flag: The factory now constructs the local chat model via `build_local_runtime_chat_model_with_native_tools(…, config.local_ai.native_tool_calling)`, so the model's profile agrees with session assembly and the harness sends/expects tools consistently. (crates/openhuman-core/src/inference/provider/factory/local_runtime.rsfn try_create_local_runtime_chat_model_from_string_with_identity(, crates/openhuman-core/src/inference/provider/factory/local_runtime.rsuse super::*;)
Internal refactor — Derived `Debug`/`Default` updates for `LocalAiConfig`: Keeps debug output and default construction in sync with the new field. (crates/openhuman-core/src/config/schema/local_ai.rsimpl std::fmt::Debug for LocalAiConfig {, crates/openhuman-core/src/config/schema/local_ai.rsimpl Default for LocalAiConfig {)
Tests
unit — Session assembly reports native tool calling for a `local-openai` runtime only after the operator sets `local_ai.native_tool_calling`; default is false.: Directly exercises the session-assembly predicate at its default and opted-in state, covering the local half of the new helper. The tests lane noted the cloud half of `provider_string_native_tools` is not exercised by any test in the diff, so a regression making cloud providers return false would go unnoticed. (crates/openhuman-core/src/inference/turn_models_native_tests.rsasync fn injected_turn_without_profile_keeps_conservative_capabilities() {)
unit — A `local-openai` model profile has `tool_calling = false` by default and `true` once `local_ai.native_tool_calling` is enabled, verifying the wire-level profile choice.: Covers the flag-off default and the opted-in wire profile through the factory. The tests lane could not verify the vendored symbol `build_local_runtime_chat_model_with_native_tools` (submodule not checked out), so the call site's arity/argument order could not be confirmed from this environment. (crates/openhuman-core/src/inference/provider/factory_crate_native_diagnostics_tests.rsasync fn non_openrouter_openai_compatible_endpoints_get_no_baked_provider_option)
Before merge
None.
Agent review details
critique
Conclusion: Success
Scope reviewed: all assigned evidence
Lane summary: Reviewed 12 files; 0 findings. _The code index is behind this pull request (indexed at `e9475735c0f7`), so retrieved context may be out of date._
security
Conclusion: Success
Scope reviewed: all assigned evidence
Lane summary: Reviewed 12 files; 0 findings. _The code index is behind this pull request (indexed at `e9475735c0f7`), so retrieved context may be out of date._
tests
Conclusion: Success
Scope reviewed: all assigned evidence
Positive: The tests lane is Answered and characterizes the behavioral change as small and coherent, pinned by two new tests covering the session-assembly predicate and the wire-level profile, with the config plumbing covered by an extended existing assertion.
Lane summary: The behavioural change is small, coherent, and pinned by two new tests: `a_local_runtime_reports_native_tools_only_when_opted_in` covers the session-assembly predicate, and `local_openai_model_calls_tools_natively_only_when_opted_in` covers the wire-level profile (the flag off by default, on when opted in). The config plumbing (patch type, controller, schema defs, catalog) is a straight field pass-through with an existing assertion added in `apply_local_ai_settings_updates_lm_studio_provider_fields`. The one gap is `provider_string_native_tools`'s cloud half (`!is_local_provider_string(provider_string) || …`), which is not exercised by any test in the diff; the new tests only feed a `local-openai:` string, so a regression that made cloud providers return `false` would go unnoticed. I could not verify the vendor symbol `build_local_runtime_chat_model_with_native_tools` (the submodule is not checked out), so I cannot confirm the arity or argument order of that call site from here. _The code index is behind this pull request (indexed at `e9475735c0f7`), so retrieved context may be out of date._
commits
Conclusion: Success
Scope reviewed: all assigned evidence
Positive: The commits lane is Answered and found no sensitive content in committed files.
Lane summary: No sensitive content in committed files. _The code index is behind this pull request (indexed at `e9475735c0f7`), so retrieved context may be out of date._
description
Conclusion: Success
Scope reviewed: all assigned evidence
Lane summary: The description matches the diff: it adds an opt-in `local_ai.native_tool_calling` flag, threads it through config ops, controller schemas and helpers, and uses one helper (`provider_string_native_tools`) in both session assembly and the crate-native turn build, with the local-runtime factory passing the flag to the vendored builder. I could not inspect the vendored tinyinference/tinyagents definitions (`build_local_runtime_chat_model_with_native_tools`, `is_local_provider_string`) because those submodules are not checked out at the reviewed commit, so I could not verify the argument order or the exact semantics of the helper they rely on; nothing else in the diff is inconsistent with what the description claims. _The code index is behind this pull request (indexed at `e9475735c0f7`), so retrieved context may be out of date._
No actionable comments were generated in the recent review.
Walkthrough
The change adds an opt-in native_tool_calling setting for local AI. Settings updates carry the value into local-AI configuration. Local model construction and tool-calling support reporting use the setting.
Changes: Local Native Tool Calling
Layer / File(s)
Summary
Setting contract and update flow crates/openhuman-core/src/config/schema/local_ai.rs, crates/openhuman-core/src/config/ops/model.rs, crates/openhuman-core/src/config/schemas/*, crates/openhuman-core/src/inference/schemas/*, crates/openhuman-core/src/config/ops_model_and_local_ai_tests.rs
Local-AI configuration defaults native_tool_calling to false. Settings update inputs and patches carry the optional value. Applying a supplied value updates configuration. A settings test checks that the value is enabled.
Local model construction and support crates/openhuman-core/src/inference/provider/factory/local_runtime.rs, crates/openhuman-core/src/inference/turn_models.rs, crates/openhuman-core/src/inference/*native*tests.rs, vendor/tinyagents
The local runtime factory passes the setting to a native-tools-aware builder. Turn-model support reporting follows the setting for local providers. Tests cover default and enabled behavior. The vendor/tinyagents submodule reference changes.
This adds an opt-in setting that defaults to off, so existing Ollama and LM Studio behavior is unchanged. No merge-blocking risk was identified in the supplied changes.
Pre-merge checks: 4
Check name
Status
Explanation
Description Check
Check skipped - CodeRabbit’s high-level summary is enabled.
Title check
The title names the main change: an opt-in native tool calling setting for local runtimes, exposed as local_ai.native_tool_calling. The raw summary shows a new config flag, a provider helper change, a…
Linked Issues check
Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check
Check skipped because no linked issues were found for this pull request.
Full details: Title check
Explanation
The title names the main change: an opt-in native tool calling setting for local runtimes, exposed as local_ai.native_tool_calling. The raw summary shows a new config flag, a provider helper change, and controller/schema plumbing, all of which match the title. The title is 91 characters, slightly above a short title, but it stays specific and readable.
Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
A rabbit taps a setting with a paw, Native tools now follow what it saw. Local models read the switch, Off by default, enabled with a twitch. The test confirms the toggle’s way, Then hops through code to end the day.
Recent review info
Run configuration
Source mode: Auto
Repository: Unavailable
Branch: Unavailable
File: No configuration file from this repository applied
Commit: Unavailable
Layers: Organization UI
Review profile: CHILL
Plan: Advanced
Run ID: 35a1b350-974b-4fba-8fa0-50f58ae81092
Commits
Reviewing files that changed from the base of the PR and between 6c41026 and d9c0b19.
Files selected for processing: 1
vendor/tinyagents
Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 7 remain after this review.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
The factory API depends on unmerged tinyinference#154 and tinyagents#612, and the final gitlinks and combined update still need validation because the supplied diff omits the vendor/tinyagents gitlink.
No open issues 🎉
What changed in this PR
Adds opt-in native tool calling for local inference runtimes, allowing compatible endpoints to use structured tool calls instead of the text protocol. The setting defaults to false, preserving existing behavior.
Changes:
Adds local_ai.native_tool_calling to configuration and both settings-update APIs.
Aligns session capability reporting and local model construction with the setting.
Adds regression coverage for default-off behavior, opt-in capabilities, and configuration updates.
107-107: Major · Functional Correctness · Heavy lift
Unverified finding
Verification ran but could not confirm this finding. It is shown for review, not as a verified issue.
Do not carry the opt-in to a different endpoint.
If an operator enables native tools for one endpoint and then switches to a provider or model that rejects tools, the flag remains true. The new endpoint therefore receives native tools and can reject agent requests. The test at crates/openhuman-core/src/config/ops_model_and_local_ai_tests.rs, Lines 567–576, demonstrates a provider change that leaves the flag untouched.
Scope the opt-in to the endpoint or model, or clear it when those settings change unless the update explicitly supplies the flag.
Review info
Run configuration
Source mode: Auto
Repository: Unavailable
Branch: Unavailable
File: No configuration file from this repository applied
Commit: Unavailable
Layers: Organization UI
Review profile: CHILL
Plan: Advanced
Run ID: b095bec1-a662-4a58-9190-9a8bf731804e
Commits
Reviewing files that changed from the base of the PR and between b4359e0 and 6c41026.
Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 9 remain after this review.
tinysweeper (cloud half of provider_string_native_tools):a_local_runtime_reports_native_tools_only_when_opted_in now also asserts that openai:gpt-4o reports native tools with the local flag off.
CodeRabbit ("do not carry the opt-in to a different endpoint", marked unverified): I'm declining this one, deliberately. The flag is operator config for the local-runtime slot, like api_key and num_ctx, which also carry across a provider change. For local-openai the endpoint comes from LOCAL_OPENAI_URL in the environment, not this RPC, so the core can't tell when the endpoint changed. Silently resetting the flag would quietly put a working native-tools endpoint back on the text protocol, which is the failure this PR fixes. The capability how-to now says to turn it off before switching to a model that rejects native tools, and Teeny re-sets it on every boot (bin/spine, SPINE_NATIVE_TOOLS).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
featureNet-new user-facing capability or product behavior.priority: p2Soon. Real but survivable — a rough edge, a gap, a thing that will bite later.rust-coreCore Rust runtime in src/: CLI, core_server, shared infrastructure.
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Teeny's builder agent (a cron agent job) runs
glm-5.3-flashon Surplus through thelocal-openairuntime. Every local runtime is treated as having no native tools, in three places:TurnModelSource::supports_native_tool_calling, the crate-native build path (!is_local), and tinyinference'sbuild_local_runtime_chat_model. So the session renders the JSON-in-tag text protocol and the harness stripstoolsoff the wire.On Surplus that fails. In builder run 1957 the first call worked, and every later call came back with the model's whole
{"arguments": {...}, "name": "shell"}body as the tool name:unknown tool \` (arguments: {})`. The model spent the rest of the run reasoning about the format, and every builder job ended "abandoned: the builder never reported back".Fed the exact failing bodies (arguments before name, arguments as a JSON string, multi-line), tinytools' text parser decodes every one correctly, in batch and in 1/3/7/16-char streamed fragments. The bad calls did not come from our parser.
Probes against Surplus (glm-5.3-flash, OpenAI-compatible), run on the Teeny box:
tools, model asked to emit JSON-in-tag: notool_calls; the text stays text.tools=[shell], a normal ask: a clean native call,finish_reason=tool_calls, idcall_7df8…, nameshell, args{"command":"ls"}. Native tool calling works for this model on Surplus.toolssent plus "don't use native calls, write JSON-in-tag", non-stream: content came back verbatim,tool_callsnull.stream=true: only thinking chunks and no answer. Surplus attachesresponse_error: {format_error, content_validation: ["valid_think_chunk"], tool_calls_validation: [], extra_tags: ["only_thinking"]}. Surplus validates and rewrites output server-side, so the text protocol plus thinking is fragile there.toolssent, native calls wanted,stream=true:finish_reason=tool_calls, a cleanshellcall with{"command":"ls -la"}, andresponse_error.tool_calls_validation=["valid_tool_call"]. Native tool calling also works on the streamed path, which is the onelocal-openaiuses.What
local_ai.native_tool_calling: bool(serde defaultfalse). Settable throughconfig.update_local_ai_settingsandinference.update_local_settings, and declared in both controller schemas.provider_string_native_tools: cloud providers get native tools, local runtimes only when the flag is set. Session assembly (supports_native_tool_calling) and the crate-native turn build both use it.local_runtime.rsbuilds throughbuild_local_runtime_chat_model_with_native_tools(…, config.local_ai.native_tool_calling), so the model's profile agrees with the session.vendor/tinyagentsto232d2d6(chore(vendor): move tinyinference to ae1f09d (local runtime native tools) tinyagents#612 → feat(openai): let a host turn native tool calls on for a local runtime tinyinference#154).The default is unchanged, so Ollama and LM Studio users see no difference.
Tests
inference::turn_models::tests::native_tests::a_local_runtime_reports_native_tools_only_when_opted_ininference::provider::factory::factory_tests::crate_native_diagnostics_tests::local_openai_model_calls_tools_natively_only_when_opted_inconfig::ops::tests::model_and_local_ai_tests::apply_local_ai_settings_updates_lm_studio_provider_fieldsnow also sets and assertsnative_tool_calling.RED:
error[E0609]: no field native_tool_calling on type LocalAiConfiganderror[E0560]: LocalAiSettingsPatch has no field named native_tool_calling. GREEN: all three pass.cargo test -p openhuman --lib -- config:: inference:: schema: 16 tests fail on this box, and the same 16 fail onmain(commof the two failure lists is empty). They come from the box's security module being unavailable, for exampleapply_local_ai_settings_persists_api_keyfails withFailed to encrypt local_ai.api_key: security module unavailable: secret approval unavailable. Clippy shows nothing in the touched files.Merge order
tinyhumansai/tinyinference#154 → tinyhumansai/tinyagents#612 (then move its tinyinference gitlink to the merge commit) → this PR (then move
vendor/tinyagentsto #612's merge commit). Independent: tinyhumansai/tinyagents#613 (a clear corrective when a call's name is not a tool name).Consumer: teenydroid/brain sets
native_tool_calling: trueinbin/spinefor its Surplus endpoint.Summary by CodeRabbit