diff --git a/docs.json b/docs.json index e73a1a1..c5170c1 100644 --- a/docs.json +++ b/docs.json @@ -199,6 +199,7 @@ "divider": true, "pages": [ "v3/llms/listing-models", + "v3/llms/model-leaderboard", "v3/llms/chat-completions", "v3/llms/responses", "v3/llms/embeddings", diff --git a/v3/llms/model-leaderboard.mdx b/v3/llms/model-leaderboard.mdx new file mode 100644 index 0000000..5ad410a --- /dev/null +++ b/v3/llms/model-leaderboard.mdx @@ -0,0 +1,214 @@ +--- +title: "LLM Model Leaderboard" +icon: "chart-line" +description: "Query the public /v3/observatory/models endpoints for anonymized LLM latency, throughput, uptime, error rates, and effective token prices." +--- +import { TechArticleSchema } from "/snippets/TechArticleSchema.mdx"; + + + +Query the public `/v3/observatory/models` endpoints to inspect anonymized, cross-tenant LLM performance, reliability, and effective token prices across providers and regions. + +## Endpoints + +``` +GET /v3/observatory/models +GET /v3/observatory/models/timeseries +``` + +Both endpoints are public and unauthenticated: no API key is required. Requests are rate-limited at 60 requests per minute. + +## Leaderboard: `GET /v3/observatory/models` + +Returns one anonymized rollup row per `(provider, model)` pair over the requested time window, ranked by `sort_by`. By default, metrics cover chat-family subfeatures (`chat`, `anthropic_messages`, `responses`) so non-chat modalities are not mixed into the rows. + + +```bash cURL +curl "https://api.edenai.run/v3/observatory/models?begin=2026-11-01T00:00:00Z&end=2026-11-15T00:00:00Z&top_n=10" +``` + +```python Python +import requests + +response = requests.get( + "https://api.edenai.run/v3/observatory/models", + params={ + "begin": "2026-11-01T00:00:00Z", + "end": "2026-11-15T00:00:00Z", + "top_n": 10, + }, +) + +for row in response.json()["response"]: + print(row["provider"], row["model"], row["latency_p50"], row["uptime"]) +``` + +```javascript JavaScript +const params = new URLSearchParams({ + begin: "2026-11-01T00:00:00Z", + end: "2026-11-15T00:00:00Z", + top_n: "10", +}); + +const response = await fetch( + `https://api.edenai.run/v3/observatory/models?${params}` +); +const { response: rows } = await response.json(); +rows.forEach((row) => console.log(row.provider, row.model, row.uptime)); +``` + + +### Query parameters + +| Parameter | Required | Default | Description | +| --- | --- | --- | --- | +| `begin` | Yes | — | Start of the window (ISO 8601 datetime). Must be before `end`. | +| `end` | Yes | — | End of the window (ISO 8601 datetime). Maximum span between `begin` and `end` is 400 days. | +| `provider` | No | — | Filter to a single provider. | +| `model` | No | — | Filter to a single model. Model aliases are resolved to their target model. | +| `subfeature` | No | Chat family | Filter to a specific subfeature. | +| `region` | No | — | Filter to a specific region. | +| `endpoints` | No | — | Comma-separated `/` IDs (up to 50). Scopes the rows to those endpoints and adds effective prices and token shares. | +| `by_region` | No | `false` | Break down rows per region and include `region` on each row. | +| `by_subfeature` | No | `false` | Break down rows per subfeature and include `subfeature` on each row. | +| `sort_by` | No | `usage_share` | Metric to rank by: `usage_share`, `error_rate`, `rate_limit_rate`, `caller_fault_rate`, `latency_p50`/`p75`/`p90`/`p95`/`p99`, `ttft_p50`/`p75`/`p90`/`p95`/`p99`, `throughput_p50`/`p75`/`p90`/`p95`/`p99`, `tool_call_failure_rate`, `tool_format_error_rate`, `structured_output_failure_rate`, `structured_format_error_rate`, `cost_sum`, `uptime`, `provider_fault_uptime`, `uptime_parity`. | +| `order` | No | `desc` | Sort direction: `desc` or `asc`. | +| `top_n` | No | `50` | Maximum number of rows to return (`1` to `200`). | + +## Effective prices and token shares + +Listed rates on [`GET /v3/models`](/v3/llms/listing-models) show a provider's base pricing, while **effective prices** measure what customers actually paid per token (the billed amount divided by the tokens paid for), reflecting [prompt caching](/v3/llms/prompt-caching) savings, cache writes, and long-context rates. + +Pass `endpoints` to `/v3/observatory/models` with a comma-separated list of `/` IDs (up to 50) to compare a model's endpoints in a single query. A provider name has no slash, while a model name may contain slashes (for example, `together/deepseek-ai/DeepSeek-V3.2`), and model aliases are resolved to their target model. + + +```bash cURL +curl "https://api.edenai.run/v3/observatory/models?begin=2026-11-01T00:00:00Z&end=2026-11-15T00:00:00Z&endpoints=openai/gpt-5.6-sol,azure/gpt-5.6-sol" +``` + +```python Python +import requests + +response = requests.get( + "https://api.edenai.run/v3/observatory/models", + params={ + "begin": "2026-11-01T00:00:00Z", + "end": "2026-11-15T00:00:00Z", + "endpoints": "openai/gpt-5.6-sol,azure/gpt-5.6-sol", + }, +) + +for row in response.json()["response"]: + print( + row["provider"], + row["effective_input_cost_per_token"], + row["effective_output_cost_per_token"], + row["token_share"], + ) +``` + +```javascript JavaScript +const params = new URLSearchParams({ + begin: "2026-11-01T00:00:00Z", + end: "2026-11-15T00:00:00Z", + endpoints: "openai/gpt-5.6-sol,azure/gpt-5.6-sol", +}); + +const response = await fetch( + `https://api.edenai.run/v3/observatory/models?${params}` +); +const { response: rows } = await response.json(); +rows.forEach((row) => + console.log( + row.provider, + row.effective_input_cost_per_token, + row.effective_output_cost_per_token, + row.token_share + ) +); +``` + + +When `endpoints` is provided, each row includes five additional fields (in USD per token for prices, and `0..1` fractions across the rows in scope for shares): + +| Field | Description | +| --- | --- | +| `effective_input_cost_per_token` | Measured USD paid per prompt token (uncached, cache reads, and cache writes). | +| `effective_output_cost_per_token` | Measured USD paid per output token. | +| `token_share` | This row's fraction of all paid tokens across the endpoints in the query. | +| `input_token_share` | This row's fraction of prompt tokens across the endpoints in the query (the weight used to average input prices). | +| `output_token_share` | This row's fraction of output tokens across the endpoints in the query (the weight used to average output prices). | + +## Price and performance timeseries: `GET /v3/observatory/models/timeseries` + +Returns per-provider or per-region time series for a single model. Pass `prices=true` to include `effective_input_cost_per_token` and `effective_output_cost_per_token` on each point. + + +```bash cURL +curl "https://api.edenai.run/v3/observatory/models/timeseries?begin=2026-11-01T00:00:00Z&end=2026-11-15T00:00:00Z&model=gpt-5.6-sol&interval=day&prices=true" +``` + +```python Python +import requests + +response = requests.get( + "https://api.edenai.run/v3/observatory/models/timeseries", + params={ + "begin": "2026-11-01T00:00:00Z", + "end": "2026-11-15T00:00:00Z", + "model": "gpt-5.6-sol", + "interval": "day", + "prices": "true", + }, +) + +for series in response.json()["response"]: + print(series["provider"], len(series["points"])) +``` + +```javascript JavaScript +const params = new URLSearchParams({ + begin: "2026-11-01T00:00:00Z", + end: "2026-11-15T00:00:00Z", + model: "gpt-5.6-sol", + interval: "day", + prices: "true", +}); + +const response = await fetch( + `https://api.edenai.run/v3/observatory/models/timeseries?${params}` +); +const { response: seriesList } = await response.json(); +seriesList.forEach((s) => console.log(s.provider, s.points.length)); +``` + + +### Query parameters + +| Parameter | Required | Default | Description | +| --- | --- | --- | --- | +| `begin` | Yes | — | Start of the window (ISO 8601 datetime). Must be before `end`. | +| `end` | Yes | — | End of the window (ISO 8601 datetime). Maximum span is 400 days. | +| `model` | Yes | — | Model name to query. Model aliases are resolved to their target model. | +| `provider` | No | — | Filter to a single provider. | +| `subfeature` | No | Chat family | Filter to a specific subfeature. | +| `region` | No | — | Filter to a specific region. | +| `interval` | No | `day` | Bucket size: `day`, `week`, or `month`. Hourly granularity is not supported (`400`). | +| `group_by` | No | `provider` | Series dimension: `provider` or `region`. | +| `prices` | No | `false` | When `true`, adds `effective_input_cost_per_token` and `effective_output_cost_per_token` to each point. | + +## Anonymity threshold + +Both endpoints enforce k-anonymity per row and per timeseries point: a bucket must have at least 20 calls across at least 3 distinct organizations before it appears. + +Effective prices and token shares only count calls whose input and output bill was split, and those split calls must clear the same k-anonymity threshold on their own. Until a row or timeseries point clears that threshold, its price and share fields are `null`, and the row contributes zero weight to other endpoints' shares. Raw call and token counts are never returned.