Skip to content

LLM-as-judge default config triggers google-genai's AFC warning in every eval run #7146

Description

@roanny

Summary

Every rubric-based eval run logs this google-genai warning once per process:

WARNING google_genai.models: Direct use of automatic function calling (AFC) in AsyncModels.generate_content is not recommended. Instead, we recommend to use AFC in AsyncChat.send_message. Similarly, direct use of AFC in AsyncModels.generate_content_stream is not recommended. ...

The agent under evaluation does not cause it. It comes from the LLM-as-judge request.

Where it comes from

LlmAsJudge builds the judge request with config=self._judge_model_options.judge_model_config or genai_types.GenerateContentConfig() (src/google/adk/evaluation/llm_as_judge.py:224-225 on main at ce53a36c0a, and the same in 2.9.0 and 2.9.1).

That default config sets neither tools nor automatic_function_calling. google-genai's AsyncModels.generate_content therefore takes its AFC branch and logs the warning:

  • _extra_utils.should_disable_afc returns False when automatic_function_calling is unset.
  • With no tools there are no AFC-incompatible tool indexes, so the direct _generate_content path is skipped.

An agent's own model calls do not hit this. ADK sends tools as function_declarations, which google-genai marks AFC-incompatible, so those calls go straight to _generate_content.

How it was observed

In an eval suite running rubric_based_*_quality_v1 metrics with gemini-3.5-flash as judge (google-adk 2.9.0, google-genai as resolved by it):

  • The warning appears once per pytest process.
  • It always appears right after the agent's inference for the case ends and right before the rubric verdicts.
  • It never appears during the agent's own model calls.
  • The deployed agent's logs, same code without the eval harness, contain no occurrence over 7 days.

Suggestion

The judge never needs automatic function calling. Its default config could disable it explicitly:

config=self._judge_model_options.judge_model_config
or genai_types.GenerateContentConfig(
    automatic_function_calling=genai_types.AutomaticFunctionCallingConfig(disable=True)
),

This does not change the request sent to the model (the field is client-side only) and removes a warning that points users at their own agent. Users who pass judge_model_config are unaffected.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions