diff --git a/documentation/README.md b/documentation/README.md index afa15b6..1a1aece 100644 --- a/documentation/README.md +++ b/documentation/README.md @@ -21,6 +21,7 @@ Connect Lynkr to your development tools: - **[Claude Code CLI Setup](claude-code-cli.md)** - Configure Claude Code CLI to use Lynkr - **[Claude Desktop Setup](claude-desktop.md)** - Route the Claude Desktop app through Lynkr (macOS gateway profile, model picker as tier selector) - **[Tier Pinning](tier-pinning.md)** - Use any desktop client's model picker as a routing tier selector (the generalized technique behind the Claude Desktop and Codex integrations) +- **[Switching Models](model-switching.md)** - Pick a tier from any client with the `lynkr-*` model ids, see which model served each request (status line, badge, headers, dashboard), and give each CLI its own model - **[opencode Integration](opencode.md)** - Per-tier context windows via `lynkr run opencode` (picker doubles as tier pin; compaction always matches the serving model) - **[OpenMausBot Integration](openmausbot.md)** - Run OpenMausBot bots through Lynkr's OpenAI-compatible endpoint (engine config, client profile, no tool injection) - **[Context Window Header](context-window-header.md)** - `X-Lynkr-Context-Window`: the served model's real context window on every response, so clients can compact against the model actually serving them diff --git a/documentation/codex-cli.md b/documentation/codex-cli.md index afaf6d9..78f9a6f 100644 --- a/documentation/codex-cli.md +++ b/documentation/codex-cli.md @@ -152,6 +152,33 @@ enabled = true --- +## Pinning a Routing Tier + +With tier routing enabled (all four `TIER_*` set), the `model` Codex sends +can pin a tier instead of leaving the choice to content scoring: + +```toml +# ~/.codex/config.toml +model = "lynkr-reasoning" # or lynkr-simple / lynkr-medium / lynkr-complex +``` + +Or per run: `codex -m lynkr-simple`. + +| `model` | Served by | +|---|---| +| `lynkr-auto` | Content-based tier routing (no pin) | +| `lynkr-simple` | `TIER_SIMPLE` | +| `lynkr-medium` | `TIER_MEDIUM` | +| `lynkr-complex` | `TIER_COMPLEX` | +| `lynkr-reasoning` | `TIER_REASONING` | + +A `lynkr-*` id wins over the `model_reasoning_effort` Codex sends with each +request, and `lynkr-auto` turns off the effort-based pin entirely. Any other +model id keeps the `(model, effort)` mapping described in +[tier-pinning.md](tier-pinning.md). Both wire APIs behave the same. + +--- + ## Wire API Formats Codex supports two API formats: diff --git a/documentation/faq.md b/documentation/faq.md index a01fc87..a632d6f 100644 --- a/documentation/faq.md +++ b/documentation/faq.md @@ -166,6 +166,16 @@ export FALLBACK_PROVIDER=databricks --- +### How do I switch models, or see which model answered? + +- **Switch:** set your client's model to `lynkr-simple`, `lynkr-medium`, `lynkr-complex` or `lynkr-reasoning` to pin that tier, or `lynkr-auto` for automatic routing (e.g. `claude --model lynkr-reasoning`, or `model = "lynkr-simple"` in Codex's `config.toml`). +- **See:** add `lynkr-statusline` to Claude Code, set `LYNKR_VISIBLE_ROUTING=true` for a badge in every reply, or read the `X-Lynkr-Tier` / `X-Lynkr-Model` / `X-Lynkr-Provider` headers on `/v1/messages` responses. +- **Per CLI:** give each client a different `lynkr-*` id and point those tiers at the models you want. + +See [Switching Models](model-switching.md) for details. + +--- + ### What is MODEL_PROVIDER and do I still need it? `MODEL_PROVIDER` sets a single static provider for all requests. When you set `MODEL_PROVIDER=ollama`, every request goes to Ollama regardless of complexity. diff --git a/documentation/model-switching.md b/documentation/model-switching.md new file mode 100644 index 0000000..76589a0 --- /dev/null +++ b/documentation/model-switching.md @@ -0,0 +1,155 @@ +# Switching Models and Seeing Which One Served + +Lynkr picks a model for every request from the four tiers you configure +(`TIER_SIMPLE`, `TIER_MEDIUM`, `TIER_COMPLEX`, `TIER_REASONING`). This page +answers the everyday questions about that choice: + +- [How do I switch model from my client?](#switch-model-from-your-client) +- [How do I see which model answered?](#see-which-model-served-a-request) +- [How do I change the routing rules?](#change-the-routing-rules) +- [How do I use a different model for each CLI?](#a-different-model-for-each-cli) + +Picking a tier needs tier routing enabled, i.e. all four `TIER_*` +variables set (see [Routing & Model Tiering](routing.md)). + +--- + +## Switch Model From Your Client + +By default every request is scored and sent to the tier that fits it. To +choose the tier yourself, set the client's model to one of Lynkr's tier ids: + +| Model id | Served by | +|---|---| +| `lynkr-auto` | Automatic, content-based routing (the default behaviour) | +| `lynkr-simple` | `TIER_SIMPLE` | +| `lynkr-medium` | `TIER_MEDIUM` | +| `lynkr-complex` | `TIER_COMPLEX` | +| `lynkr-reasoning` | `TIER_REASONING` | + +The ids work on every endpoint (`/v1/messages`, `/v1/chat/completions`, +`/v1/responses`), so the same trick works in any client that lets you type a +model name: + +| Client | How to pick a tier | +|---|---| +| Claude Code | `claude --model lynkr-reasoning`, `/model lynkr-simple` inside a session, or `export ANTHROPIC_MODEL=lynkr-complex` | +| opencode | `lynkr run opencode` writes the five entries; pick one in opencode's model picker ([details](opencode.md)) | +| Codex CLI | `model = "lynkr-medium"` in `~/.codex/config.toml`, or `codex -m lynkr-medium` ([details](codex-cli.md#pinning-a-routing-tier)) | +| Cline, Continue, Kilo Code, curl, SDKs | Use `lynkr-*` as the model name of the OpenAI-compatible provider | +| Claude Desktop | Its model picker is mapped to tiers ([details](claude-desktop.md)) | +| Codex Desktop | The model + effort picker is mapped to tiers ([details](tier-pinning.md)) | + +Switch back to automatic routing with `lynkr-auto`. + +A tier pin selects the **tier**, and the tier's model comes from your +`TIER_*` configuration. On `/v1/messages` (Claude Code, opencode) a few +explicit signals still override the pin for that one request: force phrases +such as `ultrathink` or `security audit`, high-risk edits, and a follow-up +that scores clearly above the pinned tier. See +[Tier Pinning](tier-pinning.md#scope-and-caveats) for the full scope. + +--- + +## See Which Model Served a Request + +**Status line (Claude Code).** One line under the prompt after every turn, +with no token cost. Add to `~/.claude/settings.json`: + +```json +"statusLine": { "type": "command", "command": "lynkr-statusline" } +``` + +``` +◆ Opus 4.8 · MEDIUM → minimax-m3:cloud (ollama) · today $0.06 · cache 67% +``` + +**Badge in the reply (any client).** Set `LYNKR_VISIBLE_ROUTING=true` and +every reply starts with the tier, model and provider that served it: + +``` +*[Lynkr] COMPLEX → GLM-4.7 (zai) · score 58* +``` + +**Response headers.** Anthropic-format responses (`/v1/messages`) carry +`X-Lynkr-Tier`, `X-Lynkr-Model` and `X-Lynkr-Provider` (full list in +[Routing Headers](routing.md#routing-headers)); non-streaming +`/v1/chat/completions` responses carry `X-Lynkr-Routed-Model` and +`X-Lynkr-Routed-Provider`: + +```bash +curl -si http://localhost:8081/v1/messages \ + -H "Content-Type: application/json" \ + -H "x-api-key: dummy" \ + -H "anthropic-version: 2023-06-01" \ + -d '{"model":"lynkr-auto","max_tokens":64,"messages":[{"role":"user","content":"hi"}]}' \ + | grep -i '^x-lynkr' +``` + +**Dashboard.** `http://localhost:8081/dashboard` shows recent requests, +spend and routing decisions per tier, model and provider. + +**Dry run.** `lynkr route --preview` explains how a request *would* route +(every signal, the chosen tier and model) without calling a provider: + +```bash +echo '{"messages":[{"role":"user","content":"rename this variable"}]}' \ + | lynkr route --preview - +``` + +--- + +## Change the Routing Rules + +- **Which model a tier uses** — edit the `TIER_*` lines in `.env`. The file + in the directory Lynkr was started from is watched and reloaded on save, + so no restart is needed (disable with `HOT_RELOAD_ENABLED=false`): + + ```bash + TIER_SIMPLE=ollama:qwen2.5-coder:7b + TIER_MEDIUM=openai:gpt-4o-mini + TIER_COMPLEX=zai:GLM-4.7 + TIER_REASONING=openrouter:anthropic/claude-opus-4.1 + ``` + +- **Which phrases force a tier** — see [Force Patterns](routing.md#force-patterns). +- **How requests are scored** — see [Routing & Model Tiering](routing.md) + for the scoring modes, thresholds and the tier config file. + +--- + +## A Different Model for Each CLI + +Give each client its own tier id, and point those tiers at the models you +want. For example, Claude Code on a local Ollama model and opencode on GLM: + +```bash +# .env +# SIMPLE → Claude Code, COMPLEX → opencode +TIER_SIMPLE=ollama:qwen2.5-coder:7b +TIER_MEDIUM=openai:gpt-4o-mini +TIER_COMPLEX=zai:GLM-4.7 +TIER_REASONING=openrouter:anthropic/claude-opus-4.1 +ZAI_API_KEY=your-zai-api-key +``` + +```bash +# Claude Code: pinned to the SIMPLE tier (Ollama) +ANTHROPIC_MODEL=lynkr-simple claude + +# opencode: pick "lynkr-complex" in the model picker (GLM) +lynkr run opencode +``` + +Keep in mind: + +- There are four tiers, so up to four clients can each have a model of + their own; clients pinned to the same tier share its model. +- Changing a `TIER_*` line changes the model for every client pinned to that + tier, and for automatic routing too. +- Clients set to `lynkr-auto` keep full automatic routing across all four + tiers. +- On `/v1/messages` the overrides described in + [Switch Model From Your Client](#switch-model-from-your-client) still + apply, so an `ultrathink` turn from a client pinned to SIMPLE goes to the + REASONING tier for that request. diff --git a/documentation/tier-pinning.md b/documentation/tier-pinning.md index 5df37f0..c750192 100644 --- a/documentation/tier-pinning.md +++ b/documentation/tier-pinning.md @@ -28,7 +28,7 @@ Unrecognized ids/values always resolve to *no pin* (fall through to normal routing) — the resolver never guesses (see `documentation/claude-desktop.md` for the incident-shaped reasoning behind exact-match-only resolution). -## Two client shapes, two mechanisms +## Client shapes and their mechanisms ### 1. Gateway-advertised ids (Claude Desktop) — `src/routing/model-slots.js` @@ -52,6 +52,30 @@ shown in the UI as effort labels like "Light"). Lynkr pins on the `src/api/openai-router.js` on both `/v1/chat/completions` and `/v1/responses`. +### 3. Lynkr's own tier ids (any client) — `src/routing/model-slots.js` + +For clients whose model id you set yourself (a CLI flag, a config file, a +custom-model entry in a picker), Lynkr accepts five virtual ids on **every** +ingress — Anthropic `/v1/messages` and OpenAI `/v1/chat/completions` + +`/v1/responses`: + +| `model` | Tier | +|---|---| +| `lynkr-auto` | none — full content-based routing | +| `lynkr-simple` | `SIMPLE` | +| `lynkr-medium` | `MEDIUM` | +| `lynkr-complex` | `COMPLEX` | +| `lynkr-reasoning` | `REASONING` | + +Matching is case-insensitive. On the OpenAI routes a virtual id outranks the +Codex `(model, effort)` mapping above: Codex attaches a `reasoning.effort` +to every request, so `lynkr-simple` + effort `high` pins SIMPLE, and +`lynkr-auto` keeps content routing instead of letting the effort pin every +turn. Resolution for the OpenAI routes lives in +`resolveOpenAIRequestPin()` (`src/routing/openai-model-slots.js`). For +per-client setups built on these ids see +[model-switching.md](model-switching.md). + ## Generalizing to a new client The pattern reduces to three questions: @@ -69,9 +93,9 @@ The pattern reduces to three questions: content scoring. A picker pin is advisory-input-turned-explicit — an unmapped value must degrade to the default, never to a wrong pin. -Both resolvers return `null` for anything unrecognized, and both feed the -same downstream pin mechanism (`_forceProvider`), so a new client mapping is -one small module + one resolver call at its ingress point. +Every resolver returns `null` for anything unrecognized, and all of them +feed the same downstream pin mechanism (`_forceProvider`), so a new client +mapping is one small module + one resolver call at its ingress point. ## Scope and caveats diff --git a/package.json b/package.json index f8fa607..ebecc1c 100644 --- a/package.json +++ b/package.json @@ -38,7 +38,7 @@ "start:supervised": "while true; do node index.js 2>&1 | npx pino-pretty --sync; code=$?; echo \"[supervisor] lynkr exited ($code) — restarting in 3s\"; sleep 3; done", "lint": "eslint src index.js", "test": "npm run test:unit && npm run test:performance", - "test:unit": "LYNKR_KNN_DIR=/tmp/lynkr-test-knn DATABRICKS_API_KEY=test-key DATABRICKS_API_BASE=http://test.com LOG_FILE_ENABLED=false node --test test/routing.test.js test/hybrid-routing-integration.test.js test/retry-logic.test.js test/sse-transformer.test.js test/passthrough-stream.test.js test/passthrough-mode.test.js test/openrouter-error-resilience.test.js test/format-conversion.test.js test/azure-openai-config.test.js test/azure-openai-format-conversion.test.js test/azure-openai-routing.test.js test/azure-openai-streaming.test.js test/azure-openai-error-resilience.test.js test/azure-openai-integration.test.js test/openai-integration.test.js test/atlas-integration.test.js test/cheaperinference-integration.test.js test/atmorouter-integration.test.js test/runcrate-integration.test.js test/openscale-integration.test.js test/standardcompute-integration.test.js test/toon-compression.test.js test/gcf-compression.test.js test/llamacpp-integration.test.js test/resilience.test.js test/telemetry-routing.test.js test/memory/store.test.js test/memory/surprise.test.js test/memory/extractor.test.js test/memory/search.test.js test/memory/retriever.test.js test/memory/distiller.test.js test/memory/distiller-freeze.test.js test/memory/wiki.test.js test/memory/skills-cache.test.js test/memory/tencentdb-launcher.test.js test/distill.test.js test/large-payload.test.js test/prompt-cache-injection.test.js test/risk-analyzer.test.js test/interaction-block.test.js test/preflight.test.js test/token-reduction.test.js test/session-affinity.test.js test/cache-state.test.js test/cache-switch-cost.test.js test/lens-recommendations.test.js test/model-registry-cost.test.js test/output-format-guard.test.js test/tier-fallback.test.js test/wrap.test.js test/init.test.js test/tool-call-response-metadata.test.js test/degradation.test.js test/routing-telemetry-columns.test.js test/sticky-routing.test.js test/knn-ambiguous-escalate.test.js test/deescalator.test.js test/client-profiles.test.js test/openai-ingress-tool-injection.test.js test/tool-result-compression-gate.test.js test/strip-internal-fields.test.js test/complexity-tool-subtraction.test.js test/bandit.test.js test/routing-propensity.test.js test/reward-pipeline.test.js test/knn-cold-start.test.js test/calibration.test.js test/feedback-loop.test.js test/session-fingerprint.test.js test/side-request-guards.test.js test/verifier.test.js test/intent-score.test.js test/difficulty-classifier.test.js test/classifier-setup.test.js test/usage-stats.test.js test/loop-guard.test.js test/moonshot-model-mapping.test.js test/baidu-model-mapping.test.js test/tenant-policy-ingress-parity.test.js test/decide.test.js test/embeddings-degradation.test.js test/health-probe.test.js test/stuck-detector.test.js test/onnx-embedder.test.js test/ope.test.js test/hierarchical-budget.test.js test/token-rate-limit.test.js test/otel-export.test.js test/mcp-broker.test.js test/compression-budget.test.js test/gpt-utils.test.js test/dedup-observe-only.test.js test/context-window-header.test.js test/token-budget-auto.test.js test/opencode-setup.test.js test/auth-mode-first-party.test.js test/harness-envelope.test.js test/task-ledger.test.js test/jev-router.test.js test/jev-routing.test.js test/force-patterns.test.js test/upstream-fidelity.test.js test/tool-schema-compression.test.js test/passthrough-route.test.js test/quota-ledger.test.js test/shortfall-semantic.test.js test/routing-decisions.test.js test/routing-corpus.test.js test/switch-gate-grounding.test.js", + "test:unit": "LYNKR_KNN_DIR=/tmp/lynkr-test-knn DATABRICKS_API_KEY=test-key DATABRICKS_API_BASE=http://test.com LOG_FILE_ENABLED=false node --test test/routing.test.js test/hybrid-routing-integration.test.js test/retry-logic.test.js test/sse-transformer.test.js test/passthrough-stream.test.js test/passthrough-mode.test.js test/openrouter-error-resilience.test.js test/format-conversion.test.js test/azure-openai-config.test.js test/azure-openai-format-conversion.test.js test/azure-openai-routing.test.js test/azure-openai-streaming.test.js test/azure-openai-error-resilience.test.js test/azure-openai-integration.test.js test/openai-integration.test.js test/atlas-integration.test.js test/cheaperinference-integration.test.js test/atmorouter-integration.test.js test/runcrate-integration.test.js test/openscale-integration.test.js test/standardcompute-integration.test.js test/toon-compression.test.js test/gcf-compression.test.js test/llamacpp-integration.test.js test/resilience.test.js test/telemetry-routing.test.js test/memory/store.test.js test/memory/surprise.test.js test/memory/extractor.test.js test/memory/search.test.js test/memory/retriever.test.js test/memory/distiller.test.js test/memory/distiller-freeze.test.js test/memory/wiki.test.js test/memory/skills-cache.test.js test/memory/tencentdb-launcher.test.js test/distill.test.js test/large-payload.test.js test/prompt-cache-injection.test.js test/risk-analyzer.test.js test/interaction-block.test.js test/preflight.test.js test/token-reduction.test.js test/session-affinity.test.js test/cache-state.test.js test/cache-switch-cost.test.js test/lens-recommendations.test.js test/model-registry-cost.test.js test/output-format-guard.test.js test/tier-fallback.test.js test/wrap.test.js test/init.test.js test/tool-call-response-metadata.test.js test/degradation.test.js test/routing-telemetry-columns.test.js test/sticky-routing.test.js test/knn-ambiguous-escalate.test.js test/deescalator.test.js test/client-profiles.test.js test/openai-ingress-tool-injection.test.js test/openai-virtual-tier-pin.test.js test/tool-result-compression-gate.test.js test/strip-internal-fields.test.js test/complexity-tool-subtraction.test.js test/bandit.test.js test/routing-propensity.test.js test/reward-pipeline.test.js test/knn-cold-start.test.js test/calibration.test.js test/feedback-loop.test.js test/session-fingerprint.test.js test/side-request-guards.test.js test/verifier.test.js test/intent-score.test.js test/difficulty-classifier.test.js test/classifier-setup.test.js test/usage-stats.test.js test/loop-guard.test.js test/moonshot-model-mapping.test.js test/baidu-model-mapping.test.js test/tenant-policy-ingress-parity.test.js test/decide.test.js test/embeddings-degradation.test.js test/health-probe.test.js test/stuck-detector.test.js test/onnx-embedder.test.js test/ope.test.js test/hierarchical-budget.test.js test/token-rate-limit.test.js test/otel-export.test.js test/mcp-broker.test.js test/compression-budget.test.js test/gpt-utils.test.js test/dedup-observe-only.test.js test/context-window-header.test.js test/token-budget-auto.test.js test/opencode-setup.test.js test/auth-mode-first-party.test.js test/harness-envelope.test.js test/task-ledger.test.js test/jev-router.test.js test/jev-routing.test.js test/force-patterns.test.js test/upstream-fidelity.test.js test/tool-schema-compression.test.js test/passthrough-route.test.js test/quota-ledger.test.js test/shortfall-semantic.test.js test/routing-decisions.test.js test/routing-corpus.test.js test/switch-gate-grounding.test.js", "test:memory": "LYNKR_KNN_DIR=/tmp/lynkr-test-knn DATABRICKS_API_KEY=test-key DATABRICKS_API_BASE=http://test.com node --test test/memory/store.test.js test/memory/surprise.test.js test/memory/extractor.test.js test/memory/search.test.js test/memory/retriever.test.js test/memory/distiller.test.js test/memory/distiller-freeze.test.js test/memory/wiki.test.js test/memory/skills-cache.test.js test/memory/tencentdb-launcher.test.js", "test:new-features": "LYNKR_KNN_DIR=/tmp/lynkr-test-knn DATABRICKS_API_KEY=test-key DATABRICKS_API_BASE=http://test.com node --test test/passthrough-mode.test.js test/openrouter-error-resilience.test.js test/format-conversion.test.js", "test:performance": "LYNKR_KNN_DIR=/tmp/lynkr-test-knn DATABRICKS_API_KEY=test-key DATABRICKS_API_BASE=http://test.com node test/hybrid-routing-performance.test.js && DATABRICKS_API_KEY=test-key DATABRICKS_API_BASE=http://test.com node test/performance-tests.js", diff --git a/src/api/openai-router.js b/src/api/openai-router.js index aeb82c9..073bb41 100644 --- a/src/api/openai-router.js +++ b/src/api/openai-router.js @@ -629,33 +629,34 @@ router.post("/chat/completions", async (req, res) => { } if (clientProfile) anthropicRequest._clientProfile = clientProfile; - // Explicit model/effort pin: Codex's model picker (or any OpenAI-format - // caller) can name a real OpenAI model + reasoning.effort combo that - // resolveTierForOpenAIModel recognizes. Mirrors router.js's model-id-pin + // Explicit tier pin: a Lynkr virtual tier id (`lynkr-simple` … + // `lynkr-reasoning`) in `model`, or a real OpenAI model + reasoning.effort + // combo from Codex's model picker. Mirrors router.js's model-id-pin // block for the Anthropic /v1/messages route — this one is needed // separately because these OpenAI-shaped handlers call // orchestrator.processMessage() directly and never go through router.js's // pin/classifier code at all. See ../routing/openai-model-slots.js. try { - const { resolveTierForOpenAIModel } = require("../routing/openai-model-slots"); + const { resolveOpenAIRequestPin } = require("../routing/openai-model-slots"); const _effort = req.body?.reasoning?.effort ?? req.body?.reasoning_effort ?? null; - const _pinTier = resolveTierForOpenAIModel(req.body?.model, _effort); - if (_pinTier) { + const _pin = resolveOpenAIRequestPin(req.body?.model, _effort); + if (_pin) { const { getModelTierSelector } = require("../routing"); - const _sel = getModelTierSelector().selectModel(_pinTier, null); + const _sel = getModelTierSelector().selectModel(_pin.tier, null); anthropicRequest._forceProvider = _sel.provider; if (_sel.model) anthropicRequest._tierModel = _sel.model; - anthropicRequest._tierName = _pinTier; - anthropicRequest._forcedMethod = "openai_model_effort_pin"; + anthropicRequest._tierName = _pin.tier; + anthropicRequest._forcedMethod = _pin.method; logger.debug({ model: req.body?.model, effort: _effort, - tier: _pinTier, + tier: _pin.tier, + method: _pin.method, provider: _sel.provider, - }, "[OpenAI Router] Explicit model/effort pin — scoring bypassed"); + }, "[OpenAI Router] Explicit tier pin — scoring bypassed"); } } catch (err) { - logger.debug({ err: err.message }, "[OpenAI Router] model/effort pin failed"); + logger.debug({ err: err.message }, "[OpenAI Router] tier pin failed"); } const session = getSession(sessionId); @@ -1968,30 +1969,31 @@ router.post("/responses", async (req, res) => { } if (clientProfile) anthropicRequest._clientProfile = clientProfile; - // Explicit model/effort pin — see the identical block in /chat/completions - // above for the full rationale. This is the route Codex Desktop actually - // uses (wire_api="responses"), so this is the one that matters for its - // model picker specifically. + // Explicit tier pin — see the identical block in /chat/completions above + // for the full rationale. This is the route Codex (CLI and Desktop) + // actually uses (wire_api="responses"), so this is the one that matters + // for its model picker and `model = "lynkr-…"` config specifically. try { - const { resolveTierForOpenAIModel } = require("../routing/openai-model-slots"); + const { resolveOpenAIRequestPin } = require("../routing/openai-model-slots"); const _effort = req.body?.reasoning?.effort ?? req.body?.reasoning_effort ?? null; - const _pinTier = resolveTierForOpenAIModel(req.body?.model, _effort); - if (_pinTier) { + const _pin = resolveOpenAIRequestPin(req.body?.model, _effort); + if (_pin) { const { getModelTierSelector } = require("../routing"); - const _sel = getModelTierSelector().selectModel(_pinTier, null); + const _sel = getModelTierSelector().selectModel(_pin.tier, null); anthropicRequest._forceProvider = _sel.provider; if (_sel.model) anthropicRequest._tierModel = _sel.model; - anthropicRequest._tierName = _pinTier; - anthropicRequest._forcedMethod = "openai_model_effort_pin"; + anthropicRequest._tierName = _pin.tier; + anthropicRequest._forcedMethod = _pin.method; logger.debug({ model: req.body?.model, effort: _effort, - tier: _pinTier, + tier: _pin.tier, + method: _pin.method, provider: _sel.provider, - }, "[OpenAI Router] Explicit model/effort pin (responses) — scoring bypassed"); + }, "[OpenAI Router] Explicit tier pin (responses) — scoring bypassed"); } } catch (err) { - logger.debug({ err: err.message }, "[OpenAI Router] model/effort pin failed (responses)"); + logger.debug({ err: err.message }, "[OpenAI Router] tier pin failed (responses)"); } const session = getSession(sessionId); diff --git a/src/routing/openai-model-slots.js b/src/routing/openai-model-slots.js index 91eec81..567d585 100644 --- a/src/routing/openai-model-slots.js +++ b/src/routing/openai-model-slots.js @@ -40,9 +40,19 @@ * handlers never go through — they call orchestrator.processMessage() * directly). * + * Lynkr's own virtual tier ids (`lynkr-auto`, `lynkr-simple` … + * `lynkr-reasoning`, see ./model-slots.js VIRTUAL_TIER_IDS) are resolved + * here too, by resolveOpenAIRequestPin(), so an OpenAI-format client (Codex + * CLI, Cline, Continue, Kilo, …) can pin a tier by model id exactly like an + * Anthropic-format one. They take precedence over the (model, effort) + * mapping: naming a Lynkr tier is the most explicit pick a client can make, + * and Codex attaches a `reasoning.effort` to every request regardless. + * * @module routing/openai-model-slots */ +const { VIRTUAL_TIER_IDS } = require("./model-slots"); + // OpenAI's publicly documented naming convention for distilled/cheap model // variants — NOT confirmed present in this app's specific dropdown (its // real options are Sol/Terra/Luna/5.5/5.2, no mini/nano seen), kept only @@ -79,4 +89,35 @@ function resolveTierForOpenAIModel(model, effort) { return null; } -module.exports = { EFFORT_TIER, SMALL_MODEL_PATTERN, resolveTierForOpenAIModel }; +/** + * Resolve the tier pin an OpenAI-format request carries, and how it was + * picked. This is what the /chat/completions and /responses handlers call. + * + * Precedence: + * 1. A Lynkr virtual tier id in `model` — `lynkr-simple` … `lynkr-reasoning` + * pin their tier (method `model_id_pin`, the same label router.js uses + * for the Anthropic route). `lynkr-auto` is an explicit request for + * content-based routing, so it returns null WITHOUT consulting the + * effort mapping — otherwise the effort Codex sends on every request + * would silently pin every "auto" turn. + * 2. The (model, effort) mapping of resolveTierForOpenAIModel() (method + * `openai_model_effort_pin`). + * + * @param {string|null|undefined} model - body.model + * @param {string|null|undefined} effort - body.reasoning.effort / reasoning_effort + * @returns {{tier: string, method: string}|null} null when nothing pins — + * caller should fall through to normal content scoring. + */ +function resolveOpenAIRequestPin(model, effort) { + if (typeof model === "string") { + const normalized = model.toLowerCase(); + if (Object.prototype.hasOwnProperty.call(VIRTUAL_TIER_IDS, normalized)) { + const tier = VIRTUAL_TIER_IDS[normalized]; + return tier ? { tier, method: "model_id_pin" } : null; + } + } + const tier = resolveTierForOpenAIModel(model, effort); + return tier ? { tier, method: "openai_model_effort_pin" } : null; +} + +module.exports = { EFFORT_TIER, SMALL_MODEL_PATTERN, resolveTierForOpenAIModel, resolveOpenAIRequestPin }; diff --git a/test/openai-virtual-tier-pin.test.js b/test/openai-virtual-tier-pin.test.js new file mode 100644 index 0000000..a9d139a --- /dev/null +++ b/test/openai-virtual-tier-pin.test.js @@ -0,0 +1,223 @@ +/** + * Lynkr virtual tier ids on the OpenAI-compatible endpoints. + * + * `lynkr-simple` … `lynkr-reasoning` (src/routing/model-slots.js + * VIRTUAL_TIER_IDS) pin their tier on /v1/messages, and the docs promise + * they "work from any client". The OpenAI-shaped handlers never go through + * router.js's pin code, though — they only knew Codex's (model, effort) + * mapping — so a Codex CLI / Cline / Continue user who set + * `model = "lynkr-reasoning"` got plain content scoring instead of the tier + * they asked for. + * + * Rules under test: + * 1. resolveOpenAIRequestPin(): virtual ids pin their tier (case- + * insensitive) and outrank the effort mapping; `lynkr-auto` is an + * explicit "no pin" that also suppresses the effort mapping; everything + * else keeps resolving exactly as resolveTierForOpenAIModel() did. + * 2. End-to-end through the real Express app (orchestrator mocked, same + * harness as openai-ingress-tool-injection.test.js): both + * /v1/chat/completions and /v1/responses hand the orchestrator a payload + * forced onto the pinned tier's configured provider/model. + */ + +const test = require('node:test'); +const assert = require('node:assert/strict'); +const http = require('http'); + +process.env.DATABRICKS_API_KEY = process.env.DATABRICKS_API_KEY || 'test-key'; +process.env.DATABRICKS_API_BASE = process.env.DATABRICKS_API_BASE || 'http://test.com'; +process.env.LOG_FILE_ENABLED = 'false'; +// Distinct provider:model per tier so every assertion identifies the tier. +process.env.TIER_SIMPLE = 'ollama:qwen2.5-coder:7b'; +process.env.TIER_MEDIUM = 'openai:gpt-4o-mini'; +process.env.TIER_COMPLEX = 'openai:gpt-4o'; +process.env.TIER_REASONING = 'openrouter:anthropic/claude-opus-4.1'; + +const { + resolveOpenAIRequestPin, + resolveTierForOpenAIModel, +} = require('../src/routing/openai-model-slots'); +const { VIRTUAL_TIER_IDS } = require('../src/routing/model-slots'); + +const TIER_TARGETS = { + SIMPLE: { provider: 'ollama', model: 'qwen2.5-coder:7b' }, + MEDIUM: { provider: 'openai', model: 'gpt-4o-mini' }, + COMPLEX: { provider: 'openai', model: 'gpt-4o' }, + REASONING: { provider: 'openrouter', model: 'anthropic/claude-opus-4.1' }, +}; + +// --- 1. resolver ---------------------------------------------------------------- + +test('every virtual tier id pins its tier with the model_id_pin method', () => { + for (const [id, tier] of Object.entries(VIRTUAL_TIER_IDS)) { + if (!tier) continue; + assert.deepEqual(resolveOpenAIRequestPin(id, null), { tier, method: 'model_id_pin' }, id); + } +}); + +test('virtual tier ids match case-insensitively, like resolveTierForModelId', () => { + assert.deepEqual(resolveOpenAIRequestPin('Lynkr-Reasoning', null), { tier: 'REASONING', method: 'model_id_pin' }); + assert.deepEqual(resolveOpenAIRequestPin('LYNKR-SIMPLE', 'high'), { tier: 'SIMPLE', method: 'model_id_pin' }); +}); + +test('a virtual tier id outranks the reasoning-effort mapping', () => { + // Codex attaches an effort to every request; the named tier must win. + assert.deepEqual(resolveOpenAIRequestPin('lynkr-simple', 'high'), { tier: 'SIMPLE', method: 'model_id_pin' }); + assert.deepEqual(resolveOpenAIRequestPin('lynkr-reasoning', 'minimal'), { tier: 'REASONING', method: 'model_id_pin' }); +}); + +test('lynkr-auto is an explicit no-pin and suppresses the effort mapping', () => { + for (const effort of [null, 'minimal', 'low', 'medium', 'high']) { + assert.equal(resolveOpenAIRequestPin('lynkr-auto', effort), null, `effort=${effort}`); + } +}); + +test('non-virtual ids keep the (model, effort) mapping, labelled openai_model_effort_pin', () => { + const cases = [ + ['gpt-5.6-sol', 'low'], + ['gpt-5.6-sol', 'high'], + ['gpt-5.6-mini', null], + ['gpt-4o-nano', 'high'], + ]; + for (const [model, effort] of cases) { + const tier = resolveTierForOpenAIModel(model, effort); + assert.ok(tier, `${model}/${effort} should pin via the effort mapping`); + assert.deepEqual(resolveOpenAIRequestPin(model, effort), { tier, method: 'openai_model_effort_pin' }); + } +}); + +test('unrecognised or missing input resolves to no pin', () => { + assert.equal(resolveOpenAIRequestPin('gpt-5.6-sol', null), null); + assert.equal(resolveOpenAIRequestPin('gpt-5.6-sol', 'turbo'), null); + assert.equal(resolveOpenAIRequestPin('lynkr-ultra', null), null, 'only the known virtual ids pin'); + assert.equal(resolveOpenAIRequestPin(undefined, undefined), null); + assert.equal(resolveOpenAIRequestPin(null, null), null); + assert.equal(resolveOpenAIRequestPin(42, null), null); +}); + +// --- 2. end-to-end through the real app ----------------------------------------- + +function mockOrchestrator(record) { + const modulePath = require.resolve('../src/orchestrator'); + delete require.cache[modulePath]; + require.cache[modulePath] = { + id: modulePath, + filename: modulePath, + loaded: true, + exports: { + async processMessage(args) { + record.push(args.payload); + return { + body: { + id: 'msg_test', + model: 'test-model', + content: [{ type: 'text', text: 'ok' }], + stop_reason: 'end_turn', + usage: { input_tokens: 1, output_tokens: 1 }, + }, + }; + }, + }, + }; +} + +function startTestApp() { + delete require.cache[require.resolve('../src/server')]; + delete require.cache[require.resolve('../src/api/router')]; + delete require.cache[require.resolve('../src/api/openai-router')]; + const { createApp } = require('../src/server'); + const app = createApp(); + return new Promise((resolve) => { + const server = app.listen(0, '127.0.0.1', () => resolve(server)); + }); +} + +function postJson(port, urlPath, body) { + return new Promise((resolve, reject) => { + const data = JSON.stringify(body); + const req = http.request( + { + host: '127.0.0.1', + port, + path: urlPath, + method: 'POST', + headers: { + 'Content-Type': 'application/json', + 'Content-Length': Buffer.byteLength(data), + }, + }, + (res) => { + let chunks = ''; + res.on('data', (c) => (chunks += c)); + res.on('end', () => resolve({ status: res.statusCode, body: chunks })); + } + ); + req.on('error', reject); + req.write(data); + req.end(); + }); +} + +/** Boot the app with a recording orchestrator, POST once, return the payload the orchestrator saw. */ +async function capture(t, urlPath, body) { + const payloads = []; + mockOrchestrator(payloads); + const server = await startTestApp(); + t.after(() => server.close()); + const { port } = server.address(); + const res = await postJson(port, urlPath, body); + assert.equal(res.status, 200, res.body); + assert.equal(payloads.length, 1, `expected exactly one orchestrator call, got ${payloads.length}`); + return payloads[0]; +} + +function assertPinned(payload, tier, method) { + assert.equal(payload._tierName, tier); + assert.equal(payload._forceProvider, TIER_TARGETS[tier].provider); + assert.equal(payload._tierModel, TIER_TARGETS[tier].model); + assert.equal(payload._forcedMethod, method); +} + +function assertUnpinned(payload) { + assert.equal(payload._forceProvider, undefined, 'no pin → content routing decides'); + assert.equal(payload._tierName, undefined); + assert.equal(payload._forcedMethod, undefined); +} + +const chat = (model, extra = {}) => ({ model, messages: [{ role: 'user', content: 'hi' }], ...extra }); +const responses = (model, extra = {}) => ({ model, input: 'hi', ...extra }); + +test('/v1/chat/completions: model "lynkr-reasoning" pins the REASONING tier', async (t) => { + const payload = await capture(t, '/v1/chat/completions', chat('lynkr-reasoning')); + assertPinned(payload, 'REASONING', 'model_id_pin'); +}); + +test('/v1/chat/completions: model "lynkr-simple" outranks reasoning_effort "high"', async (t) => { + const payload = await capture(t, '/v1/chat/completions', chat('lynkr-simple', { reasoning_effort: 'high' })); + assertPinned(payload, 'SIMPLE', 'model_id_pin'); +}); + +test('/v1/responses: model "lynkr-medium" with reasoning.effort "high" pins MEDIUM', async (t) => { + const payload = await capture(t, '/v1/responses', responses('lynkr-medium', { reasoning: { effort: 'high' } })); + assertPinned(payload, 'MEDIUM', 'model_id_pin'); +}); + +test('/v1/responses: model "lynkr-auto" with reasoning.effort "high" stays unpinned', async (t) => { + const payload = await capture(t, '/v1/responses', responses('lynkr-auto', { reasoning: { effort: 'high' } })); + assertUnpinned(payload); +}); + +test('/v1/chat/completions: model "lynkr-auto" stays unpinned', async (t) => { + const payload = await capture(t, '/v1/chat/completions', chat('lynkr-auto')); + assertUnpinned(payload); +}); + +test('/v1/responses: the Codex (model, effort) pin is unchanged for real model ids', async (t) => { + const payload = await capture(t, '/v1/responses', responses('gpt-5.6-sol', { reasoning: { effort: 'high' } })); + assertPinned(payload, 'REASONING', 'openai_model_effort_pin'); +}); + +test('/v1/chat/completions: a real model id without effort stays unpinned', async (t) => { + const payload = await capture(t, '/v1/chat/completions', chat('gpt-4o')); + assertUnpinned(payload); +});