Repository navigation
feat(sdk): record the provider's error code in error.type - #220
Merged
Merged
Conversation
A failed model call recorded its error class, which often names no cause: openai's RateLimitError is also an exhausted quota, Anthropic's APIError from a stream's error event, @google/genai's ApiError, and the AI SDK's APICallError and RetryError say nothing about why the call failed. Both SDKs now record, in error.type and exception.type: - the error's code when it has an HTTP status and a code that is not only digits (insufficient_quota, context_length_exceeded); - for a generic provider error class, its code, else its HTTP status (429), else the provider's error type (overloaded_error); - for the AI SDK's RetryError, the type of the last error it retried; - otherwise the class name, as before. The core SDK, the AI SDK and TanStack AI integrations, and the Python SDK share the rule. The comments and the conformance doc no longer say a propagated gen_ai.conversation.id makes the ingest guess "llm"; the ingest stops doing that.
A failure that a provider reports in a response body or a stream event, not as an HTTP error, recorded error.type Error or RuntimeError. This covered openai and OpenRouter response.failed and stream error events, OpenRouter chat stream errors, openai Realtime errors and failed image streams, LiteLLM Responses failures, and failed Google video operations. The integrations now build these failures with the provider's code and a type that names the failure, so error.type is the code (such as rate_limit_exceeded), an OpenRouter numeric code as an HTTP status (502), the google.rpc.Code name of a video operation (RESOURCE_EXHAUSTED), or else the failure's kind (response.failed). A TypeScript image stream that ends with a failed event now marks its span failed, as in Python. The AI SDK media and batch spans now set error.type with the same rule. In Python, a Bedrock stream's EventStreamError records the exception name, ModelStreamErrorException, as the TypeScript SDK does.
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A failed model call now records the provider's error code in
error.typeandexception.typewhen its class name says nothing about the cause. Before, an exhausted openai quota and a rate limit were bothRateLimitError, and an Anthropic stream that failed withoverloaded_error, a Gemini 429, and an AI SDKRetryErrorgave generic class names. telemetry.dev groups failures and alert incidents by error type, so these failures could not be told apart.The TypeScript SDKs and the Python SDK use the same rule, in
errorTypeOf(packages/otel/src/errors.ts) and_error_type(sdks/python/src/telemetry_dev/_spans.py):insufficient_quotaorcontext_length_exceededon an openai error.429), else the provider's error type (overloaded_error). The generic classes are listed indocs/sdk-conformance.md. In Python, a botoClientErrorgives itsresponse.Error.Code.AI_RetryErrorrecords the type of the last error that it retried.Numeric codes, such as Z.ai's
"1302", are not used, because a number names no cause and the class (RateLimitError) does. A getter on a third-party error that throws gives the fallback type and does not break the span.The core SDKs, the AI SDK v6 and v7 integrations (also the media and batch spans, which set no
error.typebefore), and TanStack AI call the shared function, so integrations that end spans through the core SDK, such as LangChain, get the rule too.Failures that a provider reports in a response body or a stream event, not as an HTTP error, recorded
ErrororRuntimeError. The integrations now build them with the provider's code and a type that names the failure:error.typeresponse.failed, streamerrorevent, openai Realtime errorrate_limit_exceeded, elseresponse.failedorresponse.error502response.failedgoogle.rpc.Codename, such asRESOURCE_EXHAUSTEDModelStreamErrorException, as in TypeScript, in place ofEventStreamErrorA TypeScript openai image stream that ends with a failed event now marks its span failed, as the Python SDK does.
After the release, an alert that groups incidents by error type sees the new values as new error types one time, for example
insufficient_quotain place ofRateLimitError.Tests through the real provider clients with a fake network: openai (TypeScript and Python) for
insufficient_quota,context_length_exceeded, a 500 with no code, and a numeric code; Anthropic (TypeScript and Python) for a stream that fails withoverloaded_error; Google GenAI (TypeScript 400, Python 429); and the AI SDK v7 forRetryError, anAPICallError529, and a stream provider error. The existing tests of each body or stream failure now check the code, and new tests cover a failed image stream, a failed Realtime response, and a failed Google video operation in TypeScript.I ran
vp test runin the changed packages (openai also withOPENAI_SDK_VERSION=6),vp check, and pytest, ruff, and pyright in each Python package.