Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,17 +55,17 @@ Every link on every published page is checked by `tests/test_links.py`, which re
The JSON, YAML and TOML blocks under `v3/integrations/` are configuration a reader pastes into another tool, so nothing runs them. `tests/test_config_blocks.py` parses each one and checks every name inside it. See `tests/README.md` for the details.

- A block must parse as the language its fence declares. If it is a fragment or a menu of alternative values for one key rather than a whole file, mark it `{/* skip-test: why */}` like any other block. The model ids inside it are still checked.
- Every model a config names must exist in the live catalog, which is the union of seven routes. Do NOT check a new model id against `/v3/models` alone: that route is chat only, and an embeddings or image model will look invalid.
- Every model a config names must exist in the live catalog: the union of seven routes, each read with `view=endpoints` (`provider/model` ids, including the provider aliases and `@region` ids) and `view=models` (model names and `~` aliases) when it has both. Do NOT check a new model id against `/v3/models` alone: that route is chat only, and an embeddings or image model will look invalid.
- A base URL is checked by requesting `/models` underneath it, because `https://api.edenai.run/v3` on its own answers 404 by design.
- What counts as a model id is decided by the catalog's own provider list, not a hardcoded one, so `image/gif` and `ghcr.io/open-webui/open-webui:main` are correctly not models. A new provider needs no change here.
- What counts as a model id is decided by the catalog, not a hardcoded list. A `provider/model` id counts when the catalog has its provider, so `image/gif` and `ghcr.io/open-webui/open-webui:main` are correctly not models, and a new provider needs no change here. A model name counts only after a tool's `edenai/` prefix (`edenai/~gemini-flash-latest`), never as a bare word in prose, since names like `base` and `o3` are ordinary words.
- `nginx` blocks are out of scope: no parser reads them and they name no models.

## What the Integration Pages' Tools Send

A config that parses can still point its tool at Eden AI in a way the tool cannot use, so `tests/test_integrations.py` checks the request each page's tool sends, and `tests/test_integration_clis.py` runs the real Claude Code, Codex and OpenCode CLIs with the page's settings. See `tests/README.md` for the details.

- A new page under `v3/integrations/` needs an entry in `TOOLS` in `tests/integrations.py`: the protocol its tool speaks (chat completions, Anthropic Messages, Responses) and whether it sends tool definitions. The test fails until it has one. Find out from the tool's source or docs; do not default to chat completions.
- A setting that holds a model (`model`, `titleModel`, `RAG_EMBEDDING_MODEL`, `--model`) names the whole `provider/model` id from the catalog. Tools send the value as it stands, and a bare `whisper-1` is rejected.
- A setting that holds a model (`model`, `titleModel`, `RAG_EMBEDDING_MODEL`, `--model`) names a model exactly as the catalog lists it: a model name (`gpt-4o`, `~gemini-flash-latest`) or a `provider/model` id. Tools send the value as it stands, so a name the catalog lacks, such as `whisper-1`, is rejected.
- A page for a coding agent names only models whose catalog entry has `supports_function_calling`, because the agent sends its tools with every request.
- Every model a page names is checked, in prose and tables as well as code. Do NOT name a retired model as an example.
- The CLI tests read the Claude Code page's `export` lines, the Codex page's `config.toml` block and the OpenCode page's config generator. Restructuring those pages means keeping those shapes, or updating the test that reads them.
Expand Down
25 changes: 17 additions & 8 deletions tests/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -324,7 +324,10 @@ three things that can be wrong with a config:
- **Every model it names is one Eden AI serves.** The catalog is the union of
seven routes, because `/v3/models` is hardcoded to the chat endpoints and
lists chat models only. A config that sets an embeddings model alongside its
chat models would be called broken by `/v3/models` alone.
chat models would be called broken by `/v3/models` alone. Every route but
videos is read twice, asking for each view by name: `view=endpoints` for the
`provider/model` ids, including the provider aliases and `@region` ids, and
`view=models` for the model names and `~` aliases.
- **Every Eden AI URL points at something.** A base URL is checked by asking
for `/models` underneath it, since a base URL on its own answers 404 by
design and always will.
Expand All @@ -339,7 +342,11 @@ data rather than by a list. A config block is full of strings with a slash in
them that are not models: docker volume mounts, image tags, MIME types, URLs.
A shape rule drops what cannot be an id at all, and then only what begins with
a provider the catalog actually lists is treated as a model. That is why
`image/gif` is not a model reference and nothing had to say so.
`image/gif` is not a model reference and nothing had to say so. A model name
has no provider, and many are ordinary words (`base`, `o3`), so it is never
read out of text on its own. It counts after a tool's `edenai/` prefix
(`edenai/~gemini-flash-latest`), when the catalog lists it or it starts with
`~`.

A block that is not a whole file, a menu of alternative values for one key or
a single line of a larger document, carries the same
Expand Down Expand Up @@ -368,12 +375,14 @@ integration page fails until it has an entry.

- **Without credentials**, every model the page names outside a config block
is one Eden AI serves (`test_config_blocks.py` covers the ones inside), every
setting that holds a model names a whole `provider/model` id, and an agent
is only pointed at models whose catalog entry says they support function
calling. A setting is any key or variable ending in `model` or `models`, or
qualified after the word (`TASK_MODEL_EXTERNAL`), and a list holds a setting
per model: `RAG_EMBEDDING_MODEL=text-embedding-3-small` is invisible to a
scan for `provider/model` strings, and the tool sends the value as it stands.
setting that holds a model names one exactly as the catalog lists it (a
model name or a `provider/model` id), and an agent is only pointed at models
whose catalog entry says they support function calling. A setting is any key
or variable ending in `model` or `models`, or qualified after the word
(`TASK_MODEL_EXTERNAL`), and a list holds a setting per model:
`AUDIO_STT_MODEL=whisper-1` is invisible to a scan for `provider/model`
strings, the tool sends the value as it stands, and the catalog has no
`whisper-1`.
- **With the sandbox token**, the request each page's tool sends: its
protocol, its auth and the page's lead model, streamed, with tool
definitions for an agent. One per page and kind of call, which costs nothing.
Expand Down
82 changes: 62 additions & 20 deletions tests/config_blocks.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,10 +14,17 @@
actually has, which the catalog itself supplies. That is why "image/gif" is
not a model reference and nothing had to say so.

A name Eden AI routes, such as "gpt-4o", has no provider half, and many are
ordinary words ("base", "o3"), so a bare name is never read out of text. It
counts only after a tool's "edenai/" prefix, and only when the catalog lists
it or a leading "~" marks it as an alias.

The catalog is the union of seven routes. /v3/models is hardcoded to the chat
endpoints, so it lists chat models only, and a config block may name any of
the rest: Continue configures an embeddings model next to its chat models, and
checking that one against /v3/models alone would call it broken.
checking that one against /v3/models alone would call it broken. Each route is
read in both of its views, because a page may name a provider id or a routed
name.
"""

import hashlib
Expand Down Expand Up @@ -90,16 +97,25 @@
"videos/models",
)

# Every route but videos lists its models in two views, and the default is
# moving from one to the other, so each is asked for by name. ?view=endpoints
# has an entry per provider id ("openai/gpt-4o", "azure/gpt-4o@eu") and is the
# only one with the provider aliases ("anthropic/claude-sonnet-latest").
# ?view=models has an entry per routed name ("gpt-4o", "~gemini-flash-latest").
_CATALOG_VIEWS = ("endpoints", "models")
_SINGLE_VIEW_ROUTES = frozenset({"videos/models"})

# OpenCode and Kilo Code write the name of the provider they configured, then
# the Eden AI id, which contains a slash of its own. Those tools split on the
# first slash only, so that is what happens here.
# the Eden AI id, which may contain a slash of its own. Those tools split on
# the first slash only, so that is what happens here.
TOOL_PREFIX = "edenai/"

# Anything that could be "a/b" or "a/b/c". Run over the raw text rather than a
# parsed document, because a block that is a menu of alternatives does not
# parse and its model ids still have to be checked. It never ends on a period,
# because no id does and a sentence that ends on one does.
_CANDIDATE_RE = re.compile(r"[A-Za-z0-9_.@-]+(?:/[A-Za-z0-9_.@-]+)+(?<!\.)")
# because no id does and a sentence that ends on one does. A "~" can open the
# part after a slash, for an alias behind the tool prefix: "edenai/~glm-latest".
_CANDIDATE_RE = re.compile(r"[A-Za-z0-9_.@-]+(?:/~?[A-Za-z0-9_.@-]+)+(?<!\.)")

# A JavaScript module path has the shape of an id and can start with a
# provider: import type { X } from 'openai/resources/chat/completions', or
Expand Down Expand Up @@ -166,8 +182,9 @@ def _candidates_in(text: str) -> list[str]:
if _MODULE_SPECIFIER_RE.search(text, 0, match.start()):
continue
token = match.group(0)
provider = normalize_model_id(token).split("/", 1)[0]
if _PROVIDER_SEGMENT_RE.match(provider):
model_id = normalize_model_id(token)
# No slash left means the tool prefix came off a routed name.
if "/" not in model_id or _PROVIDER_SEGMENT_RE.match(model_id.split("/")[0]):
candidates.append(token)
return candidates

Expand All @@ -193,13 +210,20 @@ def models_in(text: str) -> list[str]:
providers = catalog_providers()
seen: dict[str, None] = {}
for candidate in _candidates_in(text):
if normalize_model_id(candidate).split("/", 1)[0] in providers:
model_id = normalize_model_id(candidate)
if "/" in model_id:
named = model_id.split("/", 1)[0] in providers
else:
# A routed name has no provider to vouch for it, and the tool
# prefix starts other things too: edenai/policy/<name> on OpenClaw.
named = model_id.startswith("~") or model_id in catalog_ids()
if named:
seen.setdefault(candidate)
return list(seen)


def model_references(block: dict) -> set[str]:
"""The candidates that begin with a provider Eden AI really has."""
"""The candidates that name a model, by its provider or by a routed name."""
return set(models_in(block["code"]))


Expand All @@ -221,7 +245,8 @@ def catalog_routes(base_url: str = CATALOG_BASE_URL) -> dict[str, dict[str, dict
The route is what says which request a model answers: an id under
embeddings/models takes an embeddings call and nothing else. Each entry is
kept whole, because the chat entries carry the capabilities a tool relies
on, function calling first among them.
on, function calling first among them. A routed name ("gpt-4o") sits
beside the provider ids, with its endpoints' capabilities unioned.

Production's unless a caller names another API: readers use production,
so that is the catalog a page is checked against.
Expand All @@ -247,19 +272,31 @@ def _catalog_routes(base_url: str) -> dict[str, dict[str, dict]]:


def _fetch_catalog(base_url: str) -> dict[str, dict[str, dict]]:
# Fetched together, because seven requests one after another hold up
listings = [
(route, view)
for route in MODEL_CATALOG_ROUTES
for view in ((None,) if route in _SINGLE_VIEW_ROUTES else _CATALOG_VIEWS)
]
# Fetched together, because a dozen requests one after another hold up
# collection.
with ThreadPoolExecutor(len(MODEL_CATALOG_ROUTES)) as pool:
entries = pool.map(
lambda route: _catalog_route(base_url, route), MODEL_CATALOG_ROUTES
with ThreadPoolExecutor(len(listings)) as pool:
fetched = list(
pool.map(lambda listing: _catalog_listing(base_url, *listing), listings)
)
return dict(zip(MODEL_CATALOG_ROUTES, entries))
catalog: dict[str, dict[str, dict]] = {route: {} for route in MODEL_CATALOG_ROUTES}
for (route, _), entries in zip(listings, fetched):
catalog[route].update(entries)
return catalog


def _catalog_route(base_url: str, route: str) -> dict[str, dict]:
def _catalog_listing(base_url: str, route: str, view: str | None) -> dict[str, dict]:
with requests.Session() as session:
session.mount("https://", HTTPAdapter(max_retries=_CATALOG_RETRY))
response = session.get(f"{base_url}/{route}", timeout=TIMEOUT)
response = session.get(
f"{base_url}/{route}",
params={"view": view} if view else None,
timeout=TIMEOUT,
)
response.raise_for_status()
payload = response.json()
entries = payload["data"] if isinstance(payload, dict) else payload
Expand All @@ -268,13 +305,18 @@ def _catalog_route(base_url: str, route: str) -> dict[str, dict]:

@cache
def catalog_ids() -> frozenset[str]:
"""Every model id Eden AI serves, across all seven catalog routes."""
"""Every model id and routed name Eden AI serves, across all seven routes."""
return frozenset(
model_id for entries in catalog_routes().values() for model_id in entries
)


@cache
def catalog_providers() -> frozenset[str]:
"""The provider half of every id in the catalog."""
return frozenset(model_id.split("/", 1)[0] for model_id in catalog_ids())
"""The provider half of every provider/model id in the catalog.

A routed name has no provider half: "whisper" is a model, not a provider.
"""
return frozenset(
model_id.split("/", 1)[0] for model_id in catalog_ids() if "/" in model_id
)
5 changes: 3 additions & 2 deletions tests/integrations.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,9 @@

- the models it names, anywhere: a config block, an export line, a table
cell, a sentence telling the reader which one to pick;
- the settings that hold a model, which have to name one in full, because a
bare "whisper-1" is invisible to a scan that looks for provider/model.
- the settings that hold a model, read whole, because a name with no
provider ("gpt-4o", or a wrong one such as "whisper-1") is invisible to a
scan that looks for provider/model.

Which request a model takes comes from the catalog rather than the page: an id
listed under embeddings/models answers an embeddings call, and a chat model
Expand Down
25 changes: 25 additions & 0 deletions tests/test_config_blocks.py
Original file line number Diff line number Diff line change
Expand Up @@ -227,6 +227,20 @@ def test_an_id_with_several_slashes_keeps_all_but_the_tool_prefix():
)


def test_a_routed_name_behind_the_tool_prefix_is_a_model_reference():
"""How OpenCode writes one: the tool prefix, then the name or alias."""
block = {"code": '"model": "edenai/~gemini-flash-latest"\n- edenai/gpt-4o\n'}

assert model_references(block) == {"edenai/~gemini-flash-latest", "edenai/gpt-4o"}


def test_a_bare_word_is_not_a_model_reference():
"""Routed names include ordinary words, so only the tool prefix marks one."""
block = {"code": '{"model": "o3", "size": "base"}\nedenai/policy/<name>\n'}

assert model_references(block) == set()


# --- eden urls -------------------------------------------------------------


Expand Down Expand Up @@ -267,11 +281,22 @@ def test_the_catalog_covers_more_than_the_chat_models():
assert "openai/text-embedding-3-small" in catalog_ids()


def test_the_catalog_holds_both_views():
"""A routed name and an alias, a region-pinned id and a provider alias."""
ids = catalog_ids()

assert "gpt-4o" in ids
assert "~gemini-flash-latest" in ids
assert "azure/gpt-4o@eu" in ids
assert "anthropic/claude-sonnet-latest" in ids


def test_the_catalog_names_providers_and_not_mime_types():
providers = catalog_providers()

assert "openai" in providers
assert "image" not in providers
assert "gpt-4o" not in providers


# --- the docs themselves ---------------------------------------------------
Expand Down
12 changes: 6 additions & 6 deletions tests/test_integrations.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,9 @@
What runs where, cheapest first:

- Every pull request, no credentials: every model the page names anywhere
is one Eden AI serves, every setting that holds a model names one in full,
and an agent is only pointed at models whose catalog entry says they take
tools.
is one Eden AI serves, every setting that holds a model names one exactly
as the catalog lists it, and an agent is only pointed at models whose
catalog entry says they take tools.
- Every pull request, sandbox token: for each page, the request its tool
sends, with the page's lead model and the auth the tool uses. The sandbox
answers with a canned completion, so this proves the request is served,
Expand Down Expand Up @@ -317,14 +317,14 @@ def test_a_model_named_on_an_integration_page_is_one_eden_ai_serves(case):


@pytest.mark.parametrize("case", _SETTING_CASES, ids=_setting_id)
def test_a_model_setting_names_a_whole_model_id(case):
def test_a_model_setting_names_a_model_the_catalog_lists(case):
page, setting = case

assert normalize_model_id(setting.value) in catalog_ids(), (
f"{source(page)} line {setting.line}: {setting.key} is set to "
f"'{setting.value}', which is not a model Eden AI serves. The tool "
f"sends the value as it stands, so it needs the full provider/model id "
f"from the catalog."
f"sends the value as it stands, so it needs a model name or a "
f"provider/model id exactly as the catalog lists it."
)


Expand Down
Loading
Loading