Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs.json
Original file line number Diff line number Diff line change
Expand Up @@ -199,6 +199,7 @@
"divider": true,
"pages": [
"v3/llms/listing-models",
"v3/llms/model-leaderboard",
"v3/llms/chat-completions",
"v3/llms/responses",
"v3/llms/embeddings",
Expand Down
214 changes: 214 additions & 0 deletions v3/llms/model-leaderboard.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,214 @@
---
title: "LLM Model Leaderboard"

Check warning on line 2 in v3/llms/model-leaderboard.mdx

View check run for this annotation

Mintlify / Mintlify Validation (edenai) - vale-spellcheck

v3/llms/model-leaderboard.mdx#L2

Did you really mean 'Leaderboard'?
icon: "chart-line"
description: "Query the public /v3/observatory/models endpoints for anonymized LLM latency, throughput, uptime, error rates, and effective token prices."

Check warning on line 4 in v3/llms/model-leaderboard.mdx

View check run for this annotation

Mintlify / Mintlify Validation (edenai) - vale-spellcheck

v3/llms/model-leaderboard.mdx#L4

Did you really mean 'anonymized'?
---
import { TechArticleSchema } from "/snippets/TechArticleSchema.mdx";

<TechArticleSchema
title={"LLM Model Leaderboard"}
description={"Query the public /v3/observatory/models endpoints for anonymized LLM latency, throughput, uptime, error rates, and effective token prices."}
path="v3/llms/model-leaderboard"
articleSection="LLMs"
about={"LLM API"}
proficiencyLevel="Intermediate"
keywords={["Eden AI", "LLM API", "model leaderboard", "effective prices", "latency", "uptime"]}
datePublished="2026-11-16T00:00:00Z"
dateModified="2026-11-16T00:00:00Z"
/>

Query the public `/v3/observatory/models` endpoints to inspect anonymized, cross-tenant LLM performance, reliability, and effective token prices across providers and regions.

Check warning on line 20 in v3/llms/model-leaderboard.mdx

View check run for this annotation

Mintlify / Mintlify Validation (edenai) - vale-spellcheck

v3/llms/model-leaderboard.mdx#L20

Did you really mean 'anonymized'?

## Endpoints

```
GET /v3/observatory/models
GET /v3/observatory/models/timeseries
```

Both endpoints are public and unauthenticated: no API key is required. Requests are rate-limited at 60 requests per minute.

## Leaderboard: `GET /v3/observatory/models`

Check warning on line 31 in v3/llms/model-leaderboard.mdx

View check run for this annotation

Mintlify / Mintlify Validation (edenai) - vale-spellcheck

v3/llms/model-leaderboard.mdx#L31

Did you really mean 'Leaderboard'?

Returns one anonymized rollup row per `(provider, model)` pair over the requested time window, ranked by `sort_by`. By default, metrics cover chat-family subfeatures (`chat`, `anthropic_messages`, `responses`) so non-chat modalities are not mixed into the rows.

Check warning on line 33 in v3/llms/model-leaderboard.mdx

View check run for this annotation

Mintlify / Mintlify Validation (edenai) - vale-spellcheck

v3/llms/model-leaderboard.mdx#L33

Did you really mean 'anonymized'?

Check warning on line 33 in v3/llms/model-leaderboard.mdx

View check run for this annotation

Mintlify / Mintlify Validation (edenai) - vale-spellcheck

v3/llms/model-leaderboard.mdx#L33

Did you really mean 'rollup'?

Check warning on line 33 in v3/llms/model-leaderboard.mdx

View check run for this annotation

Mintlify / Mintlify Validation (edenai) - vale-spellcheck

v3/llms/model-leaderboard.mdx#L33

Did you really mean 'subfeatures'?

<CodeGroup>
```bash cURL
curl "https://api.edenai.run/v3/observatory/models?begin=2026-11-01T00:00:00Z&end=2026-11-15T00:00:00Z&top_n=10"
```

```python Python
import requests

response = requests.get(
"https://api.edenai.run/v3/observatory/models",
params={
"begin": "2026-11-01T00:00:00Z",
"end": "2026-11-15T00:00:00Z",
"top_n": 10,
},
)

for row in response.json()["response"]:
print(row["provider"], row["model"], row["latency_p50"], row["uptime"])
```

```javascript JavaScript
const params = new URLSearchParams({
begin: "2026-11-01T00:00:00Z",
end: "2026-11-15T00:00:00Z",
top_n: "10",
});

const response = await fetch(
`https://api.edenai.run/v3/observatory/models?${params}`
);
const { response: rows } = await response.json();
rows.forEach((row) => console.log(row.provider, row.model, row.uptime));
```
</CodeGroup>

### Query parameters

| Parameter | Required | Default | Description |
| --- | --- | --- | --- |
| `begin` | Yes | — | Start of the window (ISO 8601 datetime). Must be before `end`. |

Check warning on line 75 in v3/llms/model-leaderboard.mdx

View check run for this annotation

Mintlify / Mintlify Validation (edenai) - vale-spellcheck

v3/llms/model-leaderboard.mdx#L75

Did you really mean 'datetime'?
| `end` | Yes | — | End of the window (ISO 8601 datetime). Maximum span between `begin` and `end` is 400 days. |

Check warning on line 76 in v3/llms/model-leaderboard.mdx

View check run for this annotation

Mintlify / Mintlify Validation (edenai) - vale-spellcheck

v3/llms/model-leaderboard.mdx#L76

Did you really mean 'datetime'?
| `provider` | No | — | Filter to a single provider. |
| `model` | No | — | Filter to a single model. Model aliases are resolved to their target model. |
| `subfeature` | No | Chat family | Filter to a specific subfeature. |

Check warning on line 79 in v3/llms/model-leaderboard.mdx

View check run for this annotation

Mintlify / Mintlify Validation (edenai) - vale-spellcheck

v3/llms/model-leaderboard.mdx#L79

Did you really mean 'subfeature'?
| `region` | No | — | Filter to a specific region. |
| `endpoints` | No | — | Comma-separated `<provider>/<model>` IDs (up to 50). Scopes the rows to those endpoints and adds effective prices and token shares. |
| `by_region` | No | `false` | Break down rows per region and include `region` on each row. |
| `by_subfeature` | No | `false` | Break down rows per subfeature and include `subfeature` on each row. |

Check warning on line 83 in v3/llms/model-leaderboard.mdx

View check run for this annotation

Mintlify / Mintlify Validation (edenai) - vale-spellcheck

v3/llms/model-leaderboard.mdx#L83

Did you really mean 'subfeature'?
| `sort_by` | No | `usage_share` | Metric to rank by: `usage_share`, `error_rate`, `rate_limit_rate`, `caller_fault_rate`, `latency_p50`/`p75`/`p90`/`p95`/`p99`, `ttft_p50`/`p75`/`p90`/`p95`/`p99`, `throughput_p50`/`p75`/`p90`/`p95`/`p99`, `tool_call_failure_rate`, `tool_format_error_rate`, `structured_output_failure_rate`, `structured_format_error_rate`, `cost_sum`, `uptime`, `provider_fault_uptime`, `uptime_parity`. |
| `order` | No | `desc` | Sort direction: `desc` or `asc`. |
| `top_n` | No | `50` | Maximum number of rows to return (`1` to `200`). |

## Effective prices and token shares

Listed rates on [`GET /v3/models`](/v3/llms/listing-models) show a provider's base pricing, while **effective prices** measure what customers actually paid per token (the billed amount divided by the tokens paid for), reflecting [prompt caching](/v3/llms/prompt-caching) savings, cache writes, and long-context rates.

Pass `endpoints` to `/v3/observatory/models` with a comma-separated list of `<provider>/<model>` IDs (up to 50) to compare a model's endpoints in a single query. A provider name has no slash, while a model name may contain slashes (for example, `together/deepseek-ai/DeepSeek-V3.2`), and model aliases are resolved to their target model.

<CodeGroup>
```bash cURL
curl "https://api.edenai.run/v3/observatory/models?begin=2026-11-01T00:00:00Z&end=2026-11-15T00:00:00Z&endpoints=openai/gpt-5.6-sol,azure/gpt-5.6-sol"
```

```python Python
import requests

response = requests.get(
"https://api.edenai.run/v3/observatory/models",
params={
"begin": "2026-11-01T00:00:00Z",
"end": "2026-11-15T00:00:00Z",
"endpoints": "openai/gpt-5.6-sol,azure/gpt-5.6-sol",
},
)

for row in response.json()["response"]:
print(
row["provider"],
row["effective_input_cost_per_token"],
row["effective_output_cost_per_token"],
row["token_share"],
)
```

```javascript JavaScript
const params = new URLSearchParams({
begin: "2026-11-01T00:00:00Z",
end: "2026-11-15T00:00:00Z",
endpoints: "openai/gpt-5.6-sol,azure/gpt-5.6-sol",
});

const response = await fetch(
`https://api.edenai.run/v3/observatory/models?${params}`
);
const { response: rows } = await response.json();
rows.forEach((row) =>
console.log(
row.provider,
row.effective_input_cost_per_token,
row.effective_output_cost_per_token,
row.token_share
)
);
```
</CodeGroup>

When `endpoints` is provided, each row includes five additional fields (in USD per token for prices, and `0..1` fractions across the rows in scope for shares):

| Field | Description |
| --- | --- |
| `effective_input_cost_per_token` | Measured USD paid per prompt token (uncached, cache reads, and cache writes). |

Check warning on line 146 in v3/llms/model-leaderboard.mdx

View check run for this annotation

Mintlify / Mintlify Validation (edenai) - vale-spellcheck

v3/llms/model-leaderboard.mdx#L146

Did you really mean 'uncached'?
| `effective_output_cost_per_token` | Measured USD paid per output token. |
| `token_share` | This row's fraction of all paid tokens across the endpoints in the query. |
| `input_token_share` | This row's fraction of prompt tokens across the endpoints in the query (the weight used to average input prices). |
| `output_token_share` | This row's fraction of output tokens across the endpoints in the query (the weight used to average output prices). |

## Price and performance timeseries: `GET /v3/observatory/models/timeseries`

Check warning on line 152 in v3/llms/model-leaderboard.mdx

View check run for this annotation

Mintlify / Mintlify Validation (edenai) - vale-spellcheck

v3/llms/model-leaderboard.mdx#L152

Did you really mean 'timeseries'?

Returns per-provider or per-region time series for a single model. Pass `prices=true` to include `effective_input_cost_per_token` and `effective_output_cost_per_token` on each point.

<CodeGroup>
```bash cURL
curl "https://api.edenai.run/v3/observatory/models/timeseries?begin=2026-11-01T00:00:00Z&end=2026-11-15T00:00:00Z&model=gpt-5.6-sol&interval=day&prices=true"
```

```python Python
import requests

response = requests.get(
"https://api.edenai.run/v3/observatory/models/timeseries",
params={
"begin": "2026-11-01T00:00:00Z",
"end": "2026-11-15T00:00:00Z",
"model": "gpt-5.6-sol",
"interval": "day",
"prices": "true",
},
)

for series in response.json()["response"]:
print(series["provider"], len(series["points"]))
```

```javascript JavaScript
const params = new URLSearchParams({
begin: "2026-11-01T00:00:00Z",
end: "2026-11-15T00:00:00Z",
model: "gpt-5.6-sol",
interval: "day",
prices: "true",
});

const response = await fetch(
`https://api.edenai.run/v3/observatory/models/timeseries?${params}`
);
const { response: seriesList } = await response.json();
seriesList.forEach((s) => console.log(s.provider, s.points.length));
```
</CodeGroup>

### Query parameters

| Parameter | Required | Default | Description |
| --- | --- | --- | --- |
| `begin` | Yes | — | Start of the window (ISO 8601 datetime). Must be before `end`. |

Check warning on line 200 in v3/llms/model-leaderboard.mdx

View check run for this annotation

Mintlify / Mintlify Validation (edenai) - vale-spellcheck

v3/llms/model-leaderboard.mdx#L200

Did you really mean 'datetime'?
| `end` | Yes | — | End of the window (ISO 8601 datetime). Maximum span is 400 days. |

Check warning on line 201 in v3/llms/model-leaderboard.mdx

View check run for this annotation

Mintlify / Mintlify Validation (edenai) - vale-spellcheck

v3/llms/model-leaderboard.mdx#L201

Did you really mean 'datetime'?
| `model` | Yes | — | Model name to query. Model aliases are resolved to their target model. |
| `provider` | No | — | Filter to a single provider. |
| `subfeature` | No | Chat family | Filter to a specific subfeature. |

Check warning on line 204 in v3/llms/model-leaderboard.mdx

View check run for this annotation

Mintlify / Mintlify Validation (edenai) - vale-spellcheck

v3/llms/model-leaderboard.mdx#L204

Did you really mean 'subfeature'?
| `region` | No | — | Filter to a specific region. |
| `interval` | No | `day` | Bucket size: `day`, `week`, or `month`. Hourly granularity is not supported (`400`). |
| `group_by` | No | `provider` | Series dimension: `provider` or `region`. |
| `prices` | No | `false` | When `true`, adds `effective_input_cost_per_token` and `effective_output_cost_per_token` to each point. |

## Anonymity threshold

Both endpoints enforce k-anonymity per row and per timeseries point: a bucket must have at least 20 calls across at least 3 distinct organizations before it appears.

Check warning on line 212 in v3/llms/model-leaderboard.mdx

View check run for this annotation

Mintlify / Mintlify Validation (edenai) - vale-spellcheck

v3/llms/model-leaderboard.mdx#L212

Did you really mean 'timeseries'?

Effective prices and token shares only count calls whose input and output bill was split, and those split calls must clear the same k-anonymity threshold on their own. Until a row or timeseries point clears that threshold, its price and share fields are `null`, and the row contributes zero weight to other endpoints' shares. Raw call and token counts are never returned.

Check warning on line 214 in v3/llms/model-leaderboard.mdx

View check run for this annotation

Mintlify / Mintlify Validation (edenai) - vale-spellcheck

v3/llms/model-leaderboard.mdx#L214

Did you really mean 'timeseries'?
Loading