Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -150,6 +150,7 @@ Each integration links to its installation instructions and API coverage. The li
| OpenRouter | [`@telemetry-dev/openrouter`](packages/openrouter) | [`telemetry-dev-openrouter`](sdks/python-openrouter) |
| Vercel AI SDK | [`@telemetry-dev/ai-sdk`](packages/ai) | |
| TanStack AI | [`@telemetry-dev/tanstack-ai`](packages/tanstack-ai) | |
| LangChain and LangGraph | [`@telemetry-dev/langchain`](packages/langchain) | |
| LiteLLM | | [`telemetry-dev-litellm`](sdks/python-litellm) |
| MCP | [`@telemetry-dev/mcp`](packages/mcp) | |
| Eve | [`@telemetry-dev/eve`](packages/eve) | |
Expand All @@ -160,7 +161,7 @@ Each integration links to its installation instructions and API coverage. The li

Some integrations need no package from this repository, because the tool exports OpenTelemetry
itself or through a third-party library: Claude Code, Claude Platform, Claude SDK native spans,
Hermes Agent, OpenClaw, CrewAI, LangChain and LangGraph, and Pydantic AI. Their guides are in
Hermes Agent, OpenClaw, CrewAI, LangChain and LangGraph in Python, and Pydantic AI. Their guides are in
[Integrations](https://telemetry.dev/docs/integrations).

Other languages can use an existing OTLP/HTTP exporter.
Expand Down
3 changes: 2 additions & 1 deletion docs/sdk-conformance.md
Original file line number Diff line number Diff line change
Expand Up @@ -116,7 +116,8 @@ process-wide bound (TypeScript `batch.maxStreamBytes`, Python `max_stream_bytes`
while no other stream holds bytes in it, not even the few bytes a started stream holds before any output, a stream may grow to 16 MiB even past it); output past
either bound is not kept, and its span gets `telemetry.dev.capture.truncated = true`;
instrumentation scopes `@telemetry-dev/sdk` and `telemetry_dev` (both
classify as framework `otel`).
classify as framework `otel`, except a span with a `langchain.run.id` attribute, which
`@telemetry-dev/langchain` sets and which classifies as framework `langchain`).

The SDK allowlists above drop unknown keys in SDK usage objects. Ingest reports
`unrecognized_usage_attribute` for unknown numeric `gen_ai.usage.*`, `ai.usage.*`, and
Expand Down
21 changes: 21 additions & 0 deletions packages/langchain/LICENSE
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
MIT License

Copyright (c) 2026 telemetry.dev

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
85 changes: 85 additions & 0 deletions packages/langchain/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
# @telemetry-dev/langchain

LangChain.js and LangGraph.js instrumentation for telemetry.dev. A LangChain callback handler records each run as an OpenTelemetry GenAI span through `@telemetry-dev/sdk`.

Docs: https://telemetry.dev/docs/integrations/langchain-langgraph

## Install

```sh
pnpm add @telemetry-dev/sdk @telemetry-dev/langchain @langchain/core
```

Requires `@langchain/core` 1.x and `@telemetry-dev/sdk`, which needs Node.js 20.19.0 or later.

Set `TELEMETRY_DEV_API_KEY` for telemetry export.

## Trace a graph or a chain

Call `init()` once, then give the handler to the run. LangChain hands the handler down to each child run: graph nodes, chains, model calls, tools, and retrievers.

```ts
import { init } from "@telemetry-dev/sdk";
import { TelemetryDevCallbackHandler } from "@telemetry-dev/langchain";

init({ serviceName: "support-agent" });

const agent = graph.compile().withConfig({
callbacks: [new TelemetryDevCallbackHandler()],
});

await agent.invoke(
{ messages: [{ role: "user", content: "Where is order 48213?" }] },
{ configurable: { thread_id: "thread-1" }, metadata: { userId: "user-1" } },
);
```

`withConfig()` attaches the handler to each run of `agent`. To trace only one run, pass the handler in the config of that run: `agent.invoke(input, { callbacks: [handler] })`.

In a short script, call `await shutdown()` from `@telemetry-dev/sdk` before the process exits. Otherwise the last spans can stay in the export queue.

The handler records nothing until `init()` runs with an API key.

## Spans

| LangChain run | Span name | `gen_ai.operation.name` |
| --------------------------- | --------------------------- | ----------------------- |
| LangGraph graph | `invoke_agent <graph name>` | `invoke_agent` |
| Graph node, chain, runnable | the run name | `function` |
| Chat model call | `chat <model>` | `chat` |
| Text completion model call | `text_completion <model>` | `text_completion` |
| Tool | `execute_tool <tool name>` | `execute_tool` |
| Retriever | the retriever name | `function` |

The handler does not record runs that LangChain tags `langsmith:hidden`, such as the `__start__` step of LangGraph. Their child runs go under the nearest recorded run.

Each span has `langchain.run.id`, the id of its LangChain run. telemetry.dev uses this attribute to show the span as a LangChain span.

A graph span records the input of the graph. If the state of the graph has a `messages` list, the output of the graph span is the last message, which is the answer of the agent. Otherwise the output is the full state. The node, model, and tool spans record each step.

Model spans record the request model, the provider (from `ls_provider`), temperature, maximum tokens, top-p, stop sequences, the input and output messages, the response model, response id, finish reason, and token usage from `usage_metadata`. Input token counts include cache reads and cache writes. telemetry.dev calculates the cost from the token usage.

Model spans also record the type and name of each tool bound to the model in `gen_ai.tool.definitions`. This attribute does not include the descriptions or schemas of the tools, and the capture options do not change it.

Streamed model calls also record the time to the first chunk and the time between chunks.

## Sessions, users, and metadata

The handler reads these keys from the metadata of a run:

- `thread_id`, `sessionId`, `session_id`, or `conversation_id` sets `gen_ai.conversation.id`. LangGraph copies `configurable.thread_id` into the metadata of each run.
- `userId` or `user_id` sets `user.id`.

The two attributes go on each span of the run. The other metadata keys of the app go on the top span as `td.metadata.<key>`. The handler ignores the keys that LangChain and LangGraph add, such as `ls_*`, `langgraph_*`, and `checkpoint_ns`.

`propagateAttributes()` from `@telemetry-dev/sdk` also operates. Explicit metadata keys on the run take priority.

## Nesting

When a run starts in an active span of the app, for example in `startActiveSpan()` or `observe()` from `@telemetry-dev/sdk`, the top span of the run goes under that span.

The handler sets `awaitHandlers`. Thus LangChain calls the handler before it continues the run, and each span starts in the context of the caller. The handler does only in-memory work, and the SDK exports the spans in the background.

## Capture

The `captureInput`, `captureOutput`, `mask`, and `maxAttributeLength` options of `init()` apply to the content that the handler records. With capture off, spans keep the names, model, usage, and errors. The tool names, the run names, and the metadata of the app are still sent.
70 changes: 70 additions & 0 deletions packages/langchain/package.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
{
"name": "@telemetry-dev/langchain",
"version": "0.1.0",
"description": "LangChain.js and LangGraph.js instrumentation for telemetry.dev: a callback handler that records chains, graph nodes, model calls, tools, and retrievers as OpenTelemetry GenAI spans.",
"keywords": [
"genai",
"langchain",
"langgraph",
"llm",
"observability",
"opentelemetry",
"telemetry",
"tracing"
],
"homepage": "https://telemetry.dev",
"license": "MIT",
"repository": {
"type": "git",
"url": "git+https://github.com/telemetry-dev/telemetry.git",
"directory": "packages/langchain"
},
"files": [
"dist",
"src"
],
"type": "module",
"exports": {
".": {
"bun": "./src/index.ts",
"types": "./src/index.ts",
"development": "./src/index.ts",
"default": "./dist/index.mjs"
},
"./package.json": "./package.json"
},
"publishConfig": {
"access": "public",
"exports": {
".": {
"types": "./dist/index.d.mts",
"import": "./dist/index.mjs",
"default": "./dist/index.mjs"
},
"./package.json": "./package.json"
}
},
"scripts": {
"build": "pnpm exec vp pack",
"dev": "pnpm exec vp pack --watch",
"test": "vp test",
"check": "vp check",
"prepublishOnly": "vp run build"
},
"devDependencies": {
"@langchain/core": "catalog:",
"@langchain/langgraph": "catalog:",
"@opentelemetry/api": "catalog:",
"@opentelemetry/sdk-trace-base": "catalog:",
"@telemetry-dev/sdk": "workspace:*",
"@types/node": "catalog:",
"typescript": "catalog:",
"vite-plus": "catalog:",
"vitest": "catalog:",
"zod": "catalog:"
},
"peerDependencies": {
"@langchain/core": ">=1.0.0 <2",
"@telemetry-dev/sdk": "workspace:^"
}
}
192 changes: 192 additions & 0 deletions packages/langchain/src/generation.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,192 @@
import {
type BaseMessage,
isAIMessage,
isBaseMessage,
isToolMessage,
} from "@langchain/core/messages";
import type { UsageMetadata } from "@langchain/core/messages";
import type { LLMResult } from "@langchain/core/outputs";
import type { GenerationFields, SpanFields, TokenUsage } from "@telemetry-dev/sdk";

import { plain, plainMessage } from "./plain.ts";

type Metadata = Record<string, unknown>;

// LangChain `ls_provider` values whose OpenTelemetry provider name is different.
const PROVIDERS: Record<string, string> = {
amazon_bedrock: "aws.bedrock",
azure: "azure.ai.openai",
fireworks: "fireworks_ai",
google_genai: "gcp.gen_ai",
google_vertexai: "gcp.vertex_ai",
mistralai: "mistral_ai",
together: "together_ai",
};

/**
* The request fields of a model run. LangChain puts the model, provider, and main sampling
* parameters in the `ls_*` metadata of each chat model, and the full request in
* `invocation_params`.
*/
export function generationRequest(
extraParams: Metadata | undefined,
metadata: Metadata | undefined,
): GenerationFields & { attributes?: Record<string, string> } {
const params = record(extraParams?.["invocation_params"]);
const provider = string(metadata?.["ls_provider"]);
const tools = toolDefinitions(params?.["tools"]);

return {
attributes: tools.length > 0 ? { "gen_ai.tool.definitions": JSON.stringify(tools) } : undefined,
model:
string(metadata?.["ls_model_name"]) ??
string(params?.["model"]) ??
string(params?.["model_name"]) ??
string(params?.["modelName"]),
provider: provider === undefined ? undefined : (PROVIDERS[provider] ?? provider),
temperature: number(metadata?.["ls_temperature"]) ?? number(params?.["temperature"]),
maxTokens:
number(metadata?.["ls_max_tokens"]) ??
number(params?.["max_tokens"]) ??
number(params?.["max_completion_tokens"]),
topP: number(params?.["top_p"]) ?? number(params?.["topP"]),
stopSequences: strings(metadata?.["ls_stop"]),
};
}

/** The response fields of a model run, from its first generation. */
export function completionFields(output: LLMResult): SpanFields {
const generation = output.generations[0]?.[0];

if (!generation) return {};

const message: BaseMessage | undefined =
"message" in generation && isBaseMessage(generation.message) ? generation.message : undefined;

const ai = message && isAIMessage(message) ? message : undefined;
const response = record(message?.response_metadata);
// A streamed Anthropic reply keeps the model and stop reason of the message here.
const extra = record(message?.additional_kwargs);
const info = record(generation.generationInfo);
const llmOutput = record(output.llmOutput);

return {
output: message ? [plainMessage(message)] : generation.text,
usage: usageOf(ai?.usage_metadata) ?? legacyUsage(record(llmOutput?.["tokenUsage"])),
responseModel:
string(response?.["model_name"]) ??
string(response?.["model"]) ??
string(extra?.["model"]) ??
string(llmOutput?.["model_name"]),
responseId: string(response?.["id"]) ?? providerId(message?.id),
finishReason:
string(response?.["finish_reason"]) ??
string(response?.["stop_reason"]) ??
string(extra?.["stop_reason"]) ??
string(info?.["finish_reason"]),
};
}

/** Tool input arrives as text: JSON for a structured tool. */
export function toolInput(input: string): unknown {
try {
return JSON.parse(input);
} catch {
return input;
}
}

/** A tool called with a tool call returns a ToolMessage; record its content. */
export function toolOutput(output: unknown): unknown {
if (isBaseMessage(output) && isToolMessage(output)) return output.content;

return plain(output);
}

/**
* The type and name of each tool in the request, in the provider's format: OpenAI
* `{type, function: {name}}`, Anthropic `{name}` or a server tool `{type, name}`, Gemini
* `{functionDeclarations: [{name}]}`, and Bedrock `{toolSpec: {name}}`. Descriptions and schemas
* stay out, so that the tool list does not depend on content capture.
*/
function toolDefinitions(tools: unknown): Array<{ type: string; name: string }> {
if (!Array.isArray(tools)) return [];

return tools.flatMap((item: unknown) => {
const tool = record(item);
const declarations = tool?.["functionDeclarations"];

if (Array.isArray(declarations)) {
return declarations.flatMap((declaration: unknown) => {
const name = string(record(declaration)?.["name"]);

return name ? [{ type: "function", name }] : [];
});
}

const name =
string(record(tool?.["function"])?.["name"]) ??
string(record(tool?.["toolSpec"])?.["name"]) ??
string(tool?.["name"]);

if (!name) return [];
const type = string(tool?.["type"]);

return [{ type: type === undefined || type === "custom" ? "function" : type, name }];
});
}

function usageOf(usage: UsageMetadata | undefined): TokenUsage | undefined {
if (!usage) return undefined;
const input = usage.input_token_details;
const output = usage.output_token_details;

return {
inputTokens: usage.input_tokens,
outputTokens: usage.output_tokens,
totalTokens: usage.total_tokens,
cacheReadInputTokens: input?.cache_read,
cacheCreationInputTokens: input?.cache_creation,
reasoningOutputTokens: output?.reasoning,
textInputTokens: input?.text,
textOutputTokens: output?.text,
imageInputTokens: input?.image,
imageOutputTokens: output?.image,
audioInputTokens: input?.audio,
audioOutputTokens: output?.audio,
};
}

// Older integrations report usage only in `llmOutput.tokenUsage`.
function legacyUsage(usage: Metadata | undefined): TokenUsage | undefined {
if (!usage) return undefined;

return {
inputTokens: number(usage["promptTokens"]),
outputTokens: number(usage["completionTokens"]),
totalTokens: number(usage["totalTokens"]),
};
}

// LangChain gives a message without a provider id the id of its run, as "run-<uuid>".
function providerId(id: string | undefined): string | undefined {
return id?.startsWith("run-") ? undefined : id;
}

function record(value: unknown): Metadata | undefined {
return value !== null && typeof value === "object" ? (value as Metadata) : undefined;
}

function string(value: unknown): string | undefined {
return typeof value === "string" && value.length > 0 ? value : undefined;
}

function number(value: unknown): number | undefined {
return typeof value === "number" && Number.isFinite(value) ? value : undefined;
}

function strings(value: unknown): string[] | undefined {
return Array.isArray(value) && value.length > 0 && value.every((item) => typeof item === "string")
? value
: undefined;
}
Loading
Loading