aimux

A unified LLM access layer written in Rust. One API for 327 provider names.

aimux is a Rust implementation of a unified LLM provider access layer. It collapses the HTTP APIs of every AI provider into a single dyn LanguageModel interface that anything upstream can call.

Unlike rig or langchain, aimux does not build agent loops, RAG, or orchestration — it focuses exclusively on unifying service access. That is the difference: aimux is an access layer, those are orchestration layers.

---

For planned milestones and their current status, see the roadmap.

Why aimux

• 327 provider names — 282 registry-backed OpenAI-compatible presets
  • 45 vendor packages, covering text and other modalities. Local engines such as Ollama/vLLM are presets. provider_names() lists the Rust names; the generated catalogue lives in docs/api/providers.md.
• Unified, object-safe interface — the LanguageModel trait supports Box<dyn> so providers are interchangeable without changing call sites.
• Full multimodal — text, streaming, tool calling, embeddings, image, speech, transcription, video, reranking, files — plus realtime transcription sessions (RFC-0028: push audio, pull transcript parts, retryable timeouts).
• Observability built in (new in 0.3.0) — record every request/response pair to JSONL and replay it offline (RFC-0023), group calls into sessions (RFC-0024), and detect provider prompt-cache hits (RFC-0015).
• Composite models (new in 0.3.0) — RouterModel routes each call with fallback (RFC-0021); MoaModel aggregates parallel reference models mixture-of-agents style (RFC-0022). Both are plain LanguageModels.
• Config-driven provider registry — provider_registry.json describes each registry-backed OpenAI-compatible provider (base URL, env var, auth mode, template parameters); the embedded JSON is parsed once into a runtime descriptor table. Rust uses create_provider(name, PresetSettings) for presets and vendor packages; bindings retain provider(name, ...). Packages follow the AI SDK shape: XxxProviderSettings + create_xxx() + a default xxx().
• Fast and small — Rust core, release profile tuned for binary size (lto, codegen-units=1, panic="abort", strip, opt-level="z").
• 8 language bindings from one core: Node, Python, Swift, Kotlin, Flutter, Go, Java, C — plus aimux-web, a browser console for model-call testing and trace visualization (RFC-0029).
• Hermetic tests — 2,800+ cassettes replay real API responses; no network or API keys required.

Performance

Benchmarked against the official OpenAI SDKs on the same machine, same mock server, same abstraction layer (HTTP + JSON, no orchestration). Full results and methodology in docs/PERF-RESULTS.md.

	aimux	OpenAI SDK	aimux faster
Node.js (single req)	0.101 ms	1.488 ms	14.7×
Python (single req)	0.080 ms	0.595 ms	7.5×

Sustained stress (2000 requests, 200 KB context, 50 KB response)

	aimux rps	SDK rps	aimux P99	SDK P99	RSS growth
Node.js (32 cores)	1512	563¹	1.92 ms	3.96 ms	+23 MB vs +103 MB
Python	1393	987	0.94 ms	1.37 ms	+0 MB vs +8 MB

¹ vs Vercel AI SDK (not apples-to-apples — AISDK adds Zod validation, middleware, and telemetry per request).

Why it's fast

• Rust core — reqwest connection pool, no GC, no runtime pauses.
• Zero memory growth — Python aimux RSS did not grow a single byte across 2000 requests; Node grew only 2 MB.
• Stable tail latency — no GC pauses means P99 stays flat even under CPU contention; the JS SDK's P99 spikes to 12.87 ms on a single core.
• C ABI overhead is low — serialization is ~50% of overhead only on large payloads; in real LLM requests (3–10 s) it is <0.1%.

Architecture

aimux/
├── aimux-core            # Core abstractions: LanguageModel / Provider / Message / StreamPart
├── aimux-providers       # Provider implementations — registry-backed + typed (docs/api/providers.md)
├── aimux-stream          # SSE parsing
├── aimux-provider-utils  # One-exchange HTTP helpers, response handlers, API-key loading, streamed tool-call tracking
├── aimux-ffi             # C ABI (opaque handles + JSON results + owned aimux_error_t *) for non-native bindings
└── tools/                # aimux-cli (cache probe) · aimux-replay · aimux-web (console)

           ┌─ native path ──→ aimux-core + aimux-providers (direct Rust types + async)
bindings ──┤
           └─ C ABI path  ──→ aimux-ffi (handles + JSON results + owned errors)

Installation

Rust (the core library):

cargo add aimux-core aimux-providers

Crate	Description	crates.io
aimux-core	Core abstractions: LanguageModel / Provider / Message / StreamPart	crates.io
aimux-providers	Provider implementations — registry-backed + typed	crates.io
aimux-stream	SSE parsing	crates.io
aimux-provider-utils	One-exchange HTTP helpers, typed response handlers, streamed tool-call tracking	crates.io
aimux-ffi	C ABI for non-native bindings	crates.io

Node.js:

npm install @arcships/aimux

import { openai, generateText } from '@arcships/aimux'

const model = await openai(process.env.OPENAI_API_KEY!, 'gpt-4o')
const result = await generateText(model, 'Explain Rust ownership in one sentence.')
console.log(result.text)

The package ships a typed wrapper (generateText / streamText) on top of the raw napi API. Need the raw JSON-string interface? Use import { openai } from '@arcships/aimux/raw'.

Quick start

use aimux_core::prelude::*;
use aimux_providers::openai::openai;

#[tokio::main]
async fn main() -> Result<(), AiMuxError> {
    // The default provider loads OPENAI_API_KEY when a request is made.
    let model = openai().chat("gpt-4o");

    let result = generate_text(
        &model,
        "Explain Rust ownership in one sentence.",
        GenerateTextOptions::default(),
    ).await?;

    println!("{}", result.text);
    Ok(())
}

Streaming

use futures::StreamExt;

let result = stream_text(
    &model,
    "Write a haiku about Rust.",
    GenerateTextOptions::default(),
).await?;

let mut stream = result.stream;
while let Some(part) = stream.next().await {
    match part? {
        TextStreamPart::TextDelta { text, .. } => print!("{}", text),
        TextStreamPart::Finish { .. } => println!("\n[done]"),
        _ => {}
    }
}

Record & replay (new in 0.3.0)

Every request/response pair can be captured to JSONL and replayed offline — regression tests, incident forensics, and CI without network or keys (RFC-0023):

use aimux_core::recording::{init_recording, JsonlRecorder};
use std::sync::Arc;

// One line turns on recording for the process (every binding has the same
// switch; secrets are redacted before hitting disk).
init_recording(Some(Arc::new(JsonlRecorder::new("./recordings"))));

let result = generate_text(&model, "Explain Rust ownership.", Default::default()).await?;
// → ./recordings/recordings.jsonl now holds the full request + response,
//   replayable offline with the `aimux-replay` CLI.

Related: sessions group calls with session_id (RFC-0024), and cache-hit tracing detects prompt-cache reuse from provider headers (RFC-0015, surfaced by aimux-cli).

Composite models: router & MoA (new in 0.3.0)

use aimux_core::composite::ChildModel;
use aimux_core::moa::{MoaConfig, MoaModel};
use aimux_core::router::{FallbackPolicy, RouterConfig, RouterModel, RuleRouter};

// Router: pick one model per call, fall back on failure (RFC-0021).
let provider = aimux_providers::openai::openai();
let children: Vec<ChildModel> = vec![
    Arc::new(provider.chat("primary-model-id")) as ChildModel,
    Arc::new(provider.chat("secondary-model-id")) as ChildModel,
];
let routed = RouterModel::new(
    children.clone(),
    Box::new(RuleRouter),
    FallbackPolicy::OnError,
    RouterConfig::default(),
);

// MoA: fan out to references, aggregate their answers (RFC-0022).
let moa = MoaModel::new(
    children,
    Arc::new(provider.chat("primary-model-id")) as ChildModel,
    MoaConfig::default(),
);

// Both are plain LanguageModel — every API and binding works unchanged.
let result = generate_text(&routed, "…", Default::default()).await?;

Realtime transcription sessions (new in 0.3.0)

WebSocket-based streaming transcription with push-audio / next-part / retryable timeouts, in every binding (RFC-0028). TypeScript:

import { openaiTranscription, startTranscriptionSession } from '@arcships/aimux/raw'

const model = await openaiTranscription(process.env.OPENAI_API_KEY!, 'gpt-realtime-whisper')
const session = await startTranscriptionSession(model, null)

session.pushAudio(pcmChunk) // push audio as it arrives

for (;;) {
  try {
    const raw = await session.nextPart(500)
    if (raw === null) break                       // stream ended
    const part = JSON.parse(raw)
    if ('TranscriptDelta' in part) process.stdout.write(part.TranscriptDelta.delta)
  } catch (e: any) {
    if (e.retryable === false && e.message.includes('timeout')) continue
    throw e                                       // real failure — typed error
  }
}
session.close()

Switch providers

use aimux_core::provider_registry::create_provider_registry;
use aimux_providers::{create_provider, default_providers, provider_names, PresetSettings};

// Keys are loaded when a request is made.
let provider = create_provider("deepseek", PresetSettings::default())?;
let model = provider.language_model("deepseek-chat")?;

// Or assemble the built-in providers and resolve a provider:model id.
let registry = create_provider_registry(default_providers(), Default::default());
let model = registry.language_model("deepseek:deepseek-chat")?;
let names = provider_names();
// model usage is identical — it's all dyn LanguageModel

Rust by-name creation covers vendor packages and all 282 presets. Each vendor also has create_xxx(XxxProviderSettings {..}) and a default instance xxx(). Bindings keep their own provider(name, ...) API. Runtime model listing uses Provider::discovery() and then ProviderDiscovery::list_models() — see docs/api/rust.md.

Provider coverage

The generated catalogue is docs/api/providers.md. The accepted Rust names come from provider_names(). The rough shape of the roster:

• Native protocol — OpenAI, Anthropic, Google, Bedrock, Vertex, Azure, Cohere, Mistral, Groq, DeepSeek, xAI, Anthropic-AWS, Voyage
• OpenAI-compatible (registry-backed) — Fireworks, Together, Perplexity, OpenRouter, Ollama Cloud, Alibaba Tongyi, Zhipu, Baidu, Tencent, Moonshot, SiliconFlow…
• Local and Vertex-hosted presets — Ollama, vLLM, SGLang, Llama.cpp, LiteLLM Proxy, Vertex-hosted DeepSeek/Qwen/Llama…
• Responses API for text — xAI and Hugging Face (no chat-completions entry)
• Speech / transcription — ElevenLabs, Deepgram, AssemblyAI, AWS Polly, Cartesia, Hume, Gladia, RevAI, LMNT, Fal
• Image / video — Black Forest Labs, Replicate, Luma, Prodia, KlingAI, Recraft, Stability, RunwayML
• Embeddings / rerank / search — Voyage, Jina, Tavily, Exa, Firecrawl, Serper, SearXNG, You.com…

Full list with per-category counts: docs/api/providers.md.

Language bindings

aimux ships 8 bindings that share the same Rust core:

Binding	Path	Tool	Get it	Native library
Node.js	native	napi-rs v3	npm install @arcships/aimux — npm	bundled in the package, nothing to do
Python	native	PyO3 + maturin	pip install arcships-aimux — PyPI	bundled in the wheel, nothing to do
Go	C ABI	cgo (static link)	go get github.com/arcships/aimux/bindings/go then go generate	auto-downloaded .a from GitHub Releases
Swift	C ABI	SPM	SPM: https://github.com/arcships/aimux (from: "0.3.0")	libaimux_ffi.dylib — see guide
Kotlin	C ABI	JNA	ai.arcships:aimux-kotlin — Maven Central · requires JDK 17+ (0.3.0)	libaimux_ffi.so/.dylib or aimux_ffi.dll on the JNA search path — see guide
Java	C ABI	JNA	ai.arcships:aimux-java — Maven Central	same as Kotlin — see guide
Flutter	C ABI	dart:ffi	aimux on pub.dev — Flutter plugin, native core embedded (publisher arcships.ai)	iOS/Android 开箱即用，桌面端开发/测试 — see guide
C / C++	C ABI	direct link	.so/.dylib/.dll + aimux-ffi.h from GitHub Releases	link against it — see guide

See bindings/README.md and the API docs.

Testing

cargo test -p aimux-providers --tests

Tests run on cassette playback — no network and no keys. See rfc/0003-test-cassette.md.

Documentation

Doc	Contents
docs/API.md	API overview — shared reference + links to per-language guides
docs/api/reference.md	API reference — public types & functions lookup
docs/contributing/adding-a-provider.md	Adding a provider — the three-case checklist (registry row / new protocol / single modality)
docs/api/providers.md	Provider list — every provider with entry points and the canonical counts (generated)
docs/api/	Per-language API guides — Node.js, Python, Rust, Go, C/C++, Swift, Kotlin, Flutter
docs/error-model.md	错误模型 — 跨语言错误形态与兼容性约定
docs/PROJECT-OVERVIEW.md	Project overview, design decisions, benchmarks
docs/PERF-RESULTS.md	Performance benchmark results
docs/aimux-vs-aisdk-node.md	Node.js DX comparison vs Vercel AI SDK
docs/README.md	Documentation index

Design docs (RFCs)

RFC	Contents
0001	Multi-language bindings (Node/Swift/Kotlin/Flutter/Python)
0002	Config descriptor & thin-wrapper improvements
0003	Test cassette scheme
0004	Full provider inventory & implementation status
0005	Protocol conversion & adaptation layer
0006	Provider minimum acceptance, core contract, tests
0007	Search model trait
0008	Multimodal bindings design
0009	Request resilience (shared client / jitter / timeout)
0010	Performance vs Vercel AI SDK benchmark
0011	Go bindings (cgo static link + push callback → channel)
0012	Source dedup (product source −25%)
0013	Java bindings (JNA + raw/typed two-layer API)
0016	Align with Vercel AI SDK (capability gaps)
0017	Unified provider config & request body overrides (DX)
0018	Codex subscription channel provider (evaluation)
0019	Session affinity lightweight support
0014	Logging (AIMUX_LOG controls)
0015	Cache-hit tracing & audit (TraceLayer / verdict evaluator)
0020	External OpenAI-compatible provider config (runtime registry overrides)
0021	RouterModel — composite model routing with fallback
0022	MoaModel — single-fanout mixture-of-agents
0023	Request recording & replay (JSONL, ring, redaction)
0024	Session grouping & query APIs
0025	aimux-cli cache-probe client
0026	OpenAI-compatible output format
0027	Model catalogue & list_models
0028	Realtime transcription streaming (WS sessions)
0029	aimux-web browser console

Contributing

Contributions are welcome! Read CONTRIBUTING.md for the development setup, testing workflow, provider/binding conventions, and the pull request process. Please follow the Code of Conduct.

License

MIT