Skip to content

feat: support pre-populated session events in SessionInput for eval cases (#4896) - #7097

Open
Anurag-M1 wants to merge 1 commit into
google:mainfrom
Anurag-M1:feat/eval-session-input-events
Open

Anurag-M1 wants to merge 1 commit into
google:mainfrom
Anurag-M1:feat/eval-session-input-events

Conversation

@Anurag-M1

Copy link
Copy Markdown

Please ensure you have read the contribution guide before creating a pull request.

Link to Issue or Description of Change

1. Link to an existing issue (if applicable):

Problem:
When evaluating LLM agents on mid-conversation turns (e.g. evaluating queries that rely on conversational context like "Send that information to my phone" or "Now book the second option"), there was previously no way to seed prior conversation events into the evaluation session.

  • SessionInput only exposed app_name, user_id, session_id, and state.
  • Putting prior turns into conversation forces those turns to be replayed through the LLM and evaluated/scored by autoraters, introducing non-deterministic variance and noise.
  • Setting prior context in state does not populate session.events, which is what LLM agents and compaction flows inspect for chat history.

Solution:

  1. Added events: Optional[list[Event]] = Field(default=None) to SessionInput in src/google/adk/evaluation/eval_case.py.
  2. In _get_or_create_eval_session in src/google/adk/evaluation/evaluation_generator.py, iterate over initial_session.events and append them via await session_service.append_event(session=session, event=event). This records events in session.events, applies any state deltas, and works seamlessly with both in-memory and persistent session services.
  3. Supported events and session_id in convert_eval_set_to_pydantic_schema in src/google/adk/evaluation/local_eval_sets_manager.py for legacy JSON format compatibility.
  4. Added comprehensive unit tests in test_eval_case.py, test_evaluation_generator.py, and test_local_eval_sets_manager.py.

Testing Plan

Unit Tests:

  • I have added or updated unit tests for my change.
  • All unit tests pass locally.

pytest summary:

======================= 879 passed, 7 warnings in 15.23s =======================
tests/unittests/evaluation/test_eval_case.py: 24 passed
tests/unittests/evaluation/test_evaluation_generator.py: 55 passed
tests/unittests/evaluation/test_local_eval_sets_manager.py: 45 passed

Manual End-to-End (E2E) Tests:
Verified round-tripping SessionInput serialization to JSON and execution through EvaluationGenerator._generate_inferences_from_root_agent with pre-seeded Event instances in session.events.

Checklist

  • I have read the CONTRIBUTING.md document.
  • I have performed a self-review of my own code.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have added tests that prove my fix is effective or that my feature works.
  • New and existing unit tests pass locally with my changes.
  • I have manually tested my changes end-to-end.
  • Any dependent changes have been merged and published in downstream modules.

…ases (google#4896)

Closes google#4896

- Add `events: Optional[list[Event]] = Field(default=None)` to `SessionInput` in `eval_case.py`.
- Append initial session events to the newly created session in `_get_or_create_eval_session` within `evaluation_generator.py`.
- Support `events` and `session_id` in `convert_eval_set_to_pydantic_schema` in `local_eval_sets_manager.py`.
- Add unit tests in `test_eval_case.py`, `test_evaluation_generator.py`, and `test_local_eval_sets_manager.py`.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support pre-populated session events in SessionInput for eval cases

2 participants