Skip to content

fix(cli): raise a clear error when conformance replay recordings are missing - #7129

Open
chelsealong wants to merge 1 commit into
google:mainfrom
chelsealong:fix-7128-conformance-test-missing-replay-plugin
Open

chelsealong wants to merge 1 commit into
google:mainfrom
chelsealong:fix-7128-conformance-test-missing-replay-plugin

Conversation

@chelsealong

Copy link
Copy Markdown
Contributor

Fixes #7128

Root cause

adk conformance test talks to an already-running adk web server over
HTTP (AdkWebServerClient, default http://127.0.0.1:8000). It never
starts a server itself, so whichever plugins the server was started with
determine what conformance operations actually work.

  • Record mode needs RecordingsPlugin loaded server-side (via
    adk web --extra_plugins=google.adk.cli.plugins.recordings_plugin.RecordingsPlugin)
    to capture LLM/tool interactions into generated-recordings.yaml.
  • Replay mode (adk conformance test) needs the different
    ReplayPlugin (google.adk.cli.plugins.replay_plugin.ReplayPlugin) to
    load those recordings into session state (_adk_replay_recordings) so
    _ConformanceTestGemini can mock the model.

ReplayPlugin is never referenced anywhere outside its own module (only
covered by its own unit tests), so there is no built-in or documented way
to discover that it's required. The issue's repro follows the only
plugin hinted at in the workflow — RecordingsPlugin — and reuses the
same running server for both the record and test phases. Because that
server never loaded ReplayPlugin, _ConformanceTestGemini.__init__
read config.get('_adk_replay_recordings') as None and crashed with an
unrelated AttributeError when indexing into it:

Error: Replay verification failed: AttributeError: 'NoneType' object has no attribute 'recordings'

Fix

_ConformanceTestGemini.__init__ now checks for the missing recordings
and raises a ReplayVerificationError that names the actual problem and
how to fix it (start the server with ReplayPlugin loaded), instead of
crashing on an unrelated attribute access a few lines later. This follows
the same "turn a cryptic runtime error into an actionable one" pattern
already used elsewhere in this codebase (e.g. the ImportError raised
for VertexAiSearchTool's GCP bypass).

I did not change the server/plugin wiring itself (e.g. auto-registering
ReplayPlugin/RecordingsPlugin on every adk web server) since that
changes production request-path behavior for all adk web users and is
a larger design call than a minimal bug fix should make unilaterally.

Test plan

Added tests/unittests/cli/conformance/test_conformance_test_google_llm.py:

  • test_missing_recordings_raises_actionable_error — constructs
    _ConformanceTestGemini with a config missing _adk_replay_recordings
    (the exact state produced when ReplayPlugin isn't loaded) and asserts
    a ReplayVerificationError mentioning ReplayPlugin is raised.
  • test_present_recordings_does_not_raise — sanity check that a
    properly populated config still constructs fine.

Verified the first test reproduces the issue's exact AttributeError
against the pre-fix code (via git checkout HEAD~1 -- <file>, confirmed
failing, then restored the fix):

E     AttributeError: 'NoneType' object has no attribute 'recordings'
src/google/adk/cli/conformance/_conformance_test_google_llm.py:248: AttributeError
1 failed, 1 passed in 0.84s

With the fix applied:

tests/unittests/cli/conformance/test_conformance_test_google_llm.py::test_missing_recordings_raises_actionable_error PASSED
tests/unittests/cli/conformance/test_conformance_test_google_llm.py::test_present_recordings_does_not_raise PASSED
2 passed in 0.82s

Also ran the full tests/unittests/cli suite (includes existing
conformance and replay-plugin tests) to check for regressions:

1048 passed, 5 skipped, 3 xfailed in 50.61s

isort/pyink report no formatting changes needed on the touched files.

AI assistance disclosure

This change was developed with the help of Claude Code (Anthropic).

🤖 Generated with Claude Code

…missing

adk conformance test crashed with AttributeError: 'NoneType' object has
no attribute 'recordings' when the ADK web server it talks to over HTTP
wasn't started with ReplayPlugin loaded (e.g. only RecordingsPlugin was
passed via --extra_plugins, which is what record mode needs but replay
does not use). _ConformanceTestGemini now raises a ReplayVerificationError
that names the missing plugin and the --extra_plugins flag needed to fix
it, instead of crashing on an unrelated attribute access.

Fixes google#7128
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Conformance testing produces AttributeError: 'NoneType' object has no attribute 'recordings'

2 participants