Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 20 additions & 1 deletion docs/sdk-conformance.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ repository, not this SDK workspace. Paths below refer to that product repository

| SDK field (TS / Py) | Wire attribute | Notes |
| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `type: "span"` | `gen_ai.operation.name = "function"` | Always explicit: a propagated `gen_ai.conversation.id` would otherwise trip the ingest's `hasGenAi` → `"llm"` heuristic |
| `type: "span"` | `gen_ai.operation.name = "function"` | Always explicit, so the ingest never guesses it from other `gen_ai.*` keys, such as a caller-set `gen_ai.agent.name` |
| `type: "generation"` | `gen_ai.operation.name = "chat"` | |
| `type: "tool"` | `gen_ai.operation.name = "execute_tool"` | |
| `type: "agent"` | `gen_ai.operation.name = "invoke_agent"` | |
Expand Down Expand Up @@ -54,6 +54,25 @@ repository, not this SDK workspace. Paths below refer to that product repository
| the item run | `gen_ai.conversation.id = "<experiment id>:<item id>:<run index>"` | one session and one trace for each run; replaces an ambient or process session, and the run drops the active span |
| the root span of an item run | name `experiment <experiment name>`, `gen_ai.operation.name = "function"` | `input` is the item input; `output` is the task result; gated by capture flags |

`error.type` and `exception.type` are the error's class name, with two exceptions that both SDKs
apply the same way. An error with an HTTP status (400–599) and a code that is not only digits records the code,
such as openai's `insufficient_quota` on a `RateLimitError`. A generic provider error class
(TypeScript `Error`, `APIError`, `ApiError`, `AI_APICallError`, `AI_StreamProviderError`; Python
`Exception`, `APIError`, `APIStatusError`, `ClientError`, `ServerError`, `EventStreamError`) records
its code, else its HTTP status as a string (`429`), else the provider's error type
(`overloaded_error` from an Anthropic stream; Python also reads botocore's `response.Error.Code`,
with its first letter in upper case, so a Bedrock stream's `modelStreamErrorException` records
`ModelStreamErrorException` as in TypeScript). TypeScript records the AI SDK's `AI_RetryError` as
its last error. A thrown value that is not an Error keeps the fallback it had before: `Error` in the
core SDKs, `tool_error` or `error` in the AI SDK integration.

A failure that a provider reports in a response body or a stream event, not as an HTTP error, is
recorded with the same rule: the integrations build it as a plain `Error` (TypeScript) or
`Exception` (Python) with the provider's `code` and a `type` that names the failure. Thus a
`response.failed` with `rate_limit_exceeded` records `rate_limit_exceeded`, one without a code
records `response.failed`, an OpenRouter numeric code records it as an HTTP status (`502`), and a
failed Google video operation records the `google.rpc.Code` name of its status (`RESOURCE_EXHAUSTED`).

`SpanHandle.serializeContent(value, key)` / `SpanHandle.serialize_content(value, key)` returns
what `input` / `output` would emit for `value` under `key`: mask → JSON stringify → truncate. It
returns `undefined` / `None` when the mask throws, the value cannot be serialized, or there is no
Expand Down
2 changes: 2 additions & 0 deletions packages/ai/src/batch.ts
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,7 @@ function wrapBatch<F extends Callable>(
if (failed) {
attempt(() => {
const failure = failureRecord(config, error, "Batch request failed");
attempt(() => span.setAttribute("error.type", failure.errorType));
attempt(() => span.setStatus({ code: SpanStatusCode.ERROR, message: failure.message }));
attempt(() => span.recordException(failure.exception));
});
Expand Down Expand Up @@ -237,6 +238,7 @@ export const wrapGetBatchResults = <F extends Callable>(
if (failed) {
attempt(() => {
const failure = failureRecord(config, error, "Batch request failed");
attempt(() => span.setAttribute("error.type", failure.errorType));
attempt(() => span.setStatus({ code: SpanStatusCode.ERROR, message: failure.message }));
attempt(() => span.recordException(failure.exception));
});
Expand Down
3 changes: 2 additions & 1 deletion packages/ai/src/integration_v6.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ import {
} from "@opentelemetry/api";
import {
createGenerationEmitter,
errorTypeOf,
type GenerationEmitter,
type GenerationEmitterOverrides,
jsonAttr,
Expand Down Expand Up @@ -443,7 +444,7 @@ function createV6Hooks(config: ResolvedConfig, emitter: GenerationEmitter) {
e.error instanceof Error ? e.error.message : unknownErrorMessage(e.error),
) ?? "Tool execution failed";

const errorType = e.error instanceof Error ? e.error.name : "tool_error";
const errorType = errorTypeOf(e.error, "tool_error");
span.setStatus({ code: SpanStatusCode.ERROR });
span.setAttribute("error.type", errorType);
span.addEvent("exception", {
Expand Down
3 changes: 2 additions & 1 deletion packages/ai/src/integration_v7.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ import {
} from "@opentelemetry/api";
import {
createGenerationEmitter,
errorTypeOf,
type GenerationEmitter,
type GenerationEmitterOverrides,
jsonAttr,
Expand Down Expand Up @@ -557,7 +558,7 @@ function createV7Hooks(
fallback = "Tool execution failed",
): ProviderToolFailure => {
try {
if (value instanceof Error && value.name.length > 0) errorType = value.name;
errorType = errorTypeOf(value, errorType);
} catch {
return { errorType, message: fallback };
}
Expand Down
1 change: 1 addition & 0 deletions packages/ai/src/media.ts
Original file line number Diff line number Diff line change
Expand Up @@ -158,6 +158,7 @@ function wrapMedia<F extends Callable>(
if (failed) {
attempt(() => {
const failure = failureRecord(config, error, "Media generation failed");
attempt(() => span.setAttribute("error.type", failure.errorType));
attempt(() => span.setStatus({ code: SpanStatusCode.ERROR, message: failure.message }));
attempt(() => span.recordException(failure.exception));
});
Expand Down
35 changes: 21 additions & 14 deletions packages/ai/src/shared.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
import type { Exception } from "@opentelemetry/api";
import { jsonAttr } from "@telemetry-dev/otel";
import { errorTypeOf, jsonAttr } from "@telemetry-dev/otel";

import type { ResolvedConfig, TelemetryDevOptions } from "./config.ts";

Expand Down Expand Up @@ -59,15 +59,18 @@ export function contentAttr(
}

/**
* What a failed wrapper call records as its span status message and exception. Without a mask
* that is the error itself; with one it is the masked message, or `fallback` when the mask omits
* it or throws, and no stack trace, which repeats the message.
* What a failed wrapper call records as its span's `error.type`, status message, and exception.
* `errorType` follows errorTypeOf and is also the exception's type. Without a mask the message is
* the error itself; with one it is the masked message, or `fallback` when the mask omits it or
* throws, and no stack trace, which repeats the message.
*/
export function failureRecord(
config: Pick<ResolvedConfig, "mask" | "onError">,
error: unknown,
fallback: string,
): { message: string; exception: Exception } {
): { errorType: string; message: string; exception: Exception } {
const errorType = errorTypeOf(error);

if (!config.mask) {
let message = fallback;

Expand All @@ -77,27 +80,31 @@ export function failureRecord(
// An error whose string conversion throws is still recorded, with the fallback text.
}

return { message, exception: error instanceof Error ? error : message };
let exception: Exception = message;

try {
if (error instanceof Error) {
exception = { name: errorType, message: error.message, stack: error.stack };
}
} catch {
// A throwing message or stack getter leaves the status message as the exception.
}

return { errorType, message, exception };
}

let name: string | undefined;
let detail: string | undefined;

try {
if (error instanceof Error) {
name = typeof error.name === "string" && error.name.length > 0 ? error.name : undefined;
detail = error.message;
} else {
detail = unknownErrorMessage(error);
}
detail = error instanceof Error ? error.message : unknownErrorMessage(error);
} catch {
detail = undefined;
}

const message =
(detail ? contentAttr(config, "exception.message", detail) : undefined) || fallback;

return { message, exception: { name: name ?? "Error", message } };
return { errorType, message, exception: { name: errorType, message } };
}

/** Reports to onError without throwing, including for thrown values String() cannot convert. */
Expand Down
55 changes: 55 additions & 0 deletions packages/ai/tests/integration_v7.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ import {
TraceIdRatioBasedSampler,
} from "@opentelemetry/sdk-trace-base";
import { AlsContextManager, sessionSpanContext } from "@telemetry-dev/otel";
import { APICallError, RetryError, StreamProviderError } from "ai";
import { AsyncLocalStorage } from "node:async_hooks";
import { expect, test, vi } from "vitest";

Expand Down Expand Up @@ -2751,6 +2752,60 @@ test("onError marks root and open step failed and a later onEnd for the same cal
expect(step.attributes["gen_ai.request.model"]).toBe("gpt-4o");
});

const apiCallError = (statusCode: number, data: unknown) =>
new APICallError({
message: "failed",
url: "https://api.example.com",
requestBodyValues: {},
statusCode,
data,
});

test.each([
{
name: "a retried quota error gives the provider's code",
error: new RetryError({
message: "Failed after 3 attempts",
reason: "maxRetriesExceeded",
errors: [
apiCallError(429, { error: { type: "insufficient_quota", code: "insufficient_quota" } }),
],
}),
expected: "insufficient_quota",
},
{
name: "an API call error without a code gives its HTTP status",
error: apiCallError(529, { type: "error", error: { type: "overloaded_error" } }),
expected: "529",
},
{
name: "a stream error without a status gives the provider's type",
error: new StreamProviderError({
message: "Overloaded",
type: "overloaded_error",
isRetryable: true,
}),
expected: "overloaded_error",
},
])("onError: $name", async ({ error, expected }) => {
const { spanBatches, overrides } = makeCapture();
const integ = telemetryDev(baseOptions, overrides);
const callId = "call-provider-error";

integ.onStart?.({
callId,
operationId: "ai.generateText",
provider: "openai",
modelId: "gpt-4o",
});
integ.onStepStart?.({ callId, provider: "openai", modelId: "gpt-4o", stepNumber: 0 });
await integ.onError?.({ callId, error });

const spans = spanBatches[0]!;
expect(spans.find((s) => s.kind === SpanKind.CLIENT)!.attributes["error.type"]).toBe(expected);
expect(spans.find((s) => s.kind === SpanKind.INTERNAL)!.attributes["error.type"]).toBe(expected);
});

test("onAbort ends the open step, keeps root UNSET, and emits generation.summary aborted", async () => {
const { spanBatches, overrides } = makeCapture();
const integ = telemetryDev(baseOptions, overrides);
Expand Down
1 change: 1 addition & 0 deletions packages/ai/tests/wrappers.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -627,6 +627,7 @@ test("failures whose string conversion throws are still recorded", async () => {
const exception = span.events.find((event) => event.name === "exception");

expect(span.status).toEqual({ code: SpanStatusCode.ERROR, message: expected[index] });
expect(span.attributes["error.type"]).toBe("RangeError");
expect(exception?.attributes).toMatchObject({
"exception.type": "RangeError",
"exception.message": "disk full",
Expand Down
20 changes: 20 additions & 0 deletions packages/anthropic/tests/anthropic.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -690,6 +690,26 @@ test("messages.create streaming preserves events and records aggregated text usa
).toBe(true);
});

test("an error event mid-stream records the provider's error type", async () => {
const spans = setupSpans();
const overloaded = {
type: "error",
error: { type: "overloaded_error", message: "Overloaded" },
};
const fake = createFakeFetch(namedSseResponse([streamEvents()[0]!, overloaded]));

const stream = await clientWith(fake.fetch).messages.create({
model: "claude-sonnet-4-6",
max_tokens: 64,
messages: [{ role: "user", content: "Say hello" }],
stream: true,
});

// The SDK throws its generic APIError, whose class names no cause.
await expect(collectStream(stream)).rejects.toMatchObject({ type: "overloaded_error" });
expect((await exportedSpan(spans)).attributes["error.type"]).toBe("overloaded_error");
});

test("stream capture retains no output when output capture is disabled", async () => {
const spans = setupSpans(undefined, false);
const capture = vi.spyOn(telemetrySdk, "boundedCaptureDetails");
Expand Down
Loading
Loading