feat: add LiteLLM client and agent

[Prodman Devokadev (prodmanpd)]

Summary

Adds a LiteLLM client and agent, so a problem can be evaluated against models the registry has no dedicated client for.

LiteLLM gives one interface to 100+ providers. The same model name works two ways: directly, where LiteLLM routes each model to its own provider using that provider's key (LITELLM_MODEL=anthropic/claude-opus-4-7 plus ANTHROPIC_API_KEY is the whole setup); or through a self-hosted gateway when LITELLM_API_BASE is set, keeping provider keys server-side and adding centralized cost tracking, budgets, rate limiting and fallbacks across a long benchmark run.

LiteLLMClient needs no endpoint at all. For a benchmark whose value is comparing agents across models, removing the infrastructure prerequisite for multi-vendor evaluation is the point.

Changes

• clients/utils/llm.py: new LiteLLMClient, matching the existing client contract exactly (Cache, inference(), run()), appended alongside its siblings.
• clients/litellm_agent.py (new): LiteLLMAgent, mirroring generic_openai.py including the __main__ runner.
• clients/registry.py: one row, "litellm": LiteLLMAgent.
• pyproject.toml: litellm in the existing clients group, so poetry install --without clients is unaffected.
• tests/clients/test_litellm_client.py (new): 19 tests.

drop_params=True is set on every call: the shared temperature / top_p / n settings are not accepted by every provider, and dropping the unsupported ones lets one configuration serve every model.

Tests

$ python -m unittest discover tests/clients
Ran 19 tests in 0.667s
OK

Live, through the real LiteLLMAgent.get_action(), against a self-hosted gateway on an SRE prompt:

Two upstream vendors answer correctly through one code path and one key. The last case has no vendor key here, so it demonstrates routing: the error proves the SDK went to Anthropic rather than to a gateway.

Example

# Direct: no gateway needed.
export LITELLM_MODEL=anthropic/claude-opus-4-7
export ANTHROPIC_API_KEY=...
python3 clients/litellm_agent.py

# Or through a gateway, with spend centrally tracked.
export LITELLM_MODEL=claude-opus-4-7
export LITELLM_API_BASE=http://localhost:4000
export LITELLM_API_KEY=sk-...
python3 clients/litellm_agent.py

[Prodman Devokadev (prodmanpd)]

@microsoft-github-policy-service agree