Skip to content

feat(studio): add preference-driven Agent quality management - #1137

Open
evanlowe wants to merge 1 commit into
volcengine:mainfrom
evanlowe:feat/studio-agent-quality
Open

evanlowe wants to merge 1 commit into
volcengine:mainfrom
evanlowe:feat/studio-agent-quality

Conversation

@evanlowe

@evanlowe evanlowe commented Sep 20, 2026

Copy link
Copy Markdown
Contributor

Adds a Quality management entry in Agent details, starting from user-defined evaluation preferences. Users can review and adjust model-generated dataset and evaluator preferences before generating the two components together. Existing Evaluations and feedback remain unchanged.

  • Add independent, selectable suggestions to all ten preference text areas, with shared Agent context preparation, caching, cancellation and field-level retry
  • Generate 50–300 evaluation cases in bounded parallel batches, validate counts and duplicate inputs, and display the cases in a drawer
  • Generate overall, tool capability and Skill evaluators concurrently, with vertically stacked cards, loading placeholders, dimension accordions and Agent-specific scoring rationales and prompts
  • Ground generation in available topology, root and child instructions, tools, Skills, knowledge bases, memory and environment metadata, preserving ownership and information gaps
  • Share the Agent form generation model for Volcengine and BytePlus, use structured output with thinking disabled, and return complete captured provider diagnostics with credentials redacted
  • Add preference-linked component filtering, outdated indicators, partial-success recovery, input validation, bilingual copy and matching WebUI assets

Preferences, datasets and evaluator definitions currently live in page memory. Persistence, evaluation execution and result analysis are outside this change; the results tab shows an unavailable state.

Validation:

  • CI passed: Python 3.10 and 3.12 unit suites, frontend/backend gates, release bundle verification, license checks, pre-commit and secret scanning
  • Synced with current origin/main before submission
  • Frontend: all 1,235 tests passed; production build, packaged-asset verification and locale consistency checks passed
  • Backend: 171 targeted tests passed, 3 skipped, including both providers, authorization, schemas, parallel generation, cancellation and redacted diagnostics
  • Pre-commit across all files passed, including Ruff and secret scanning
  • Pyright passed for the quality backend, new tests and shared planner model changes; the existing CLI file retains the same 33 diagnostics reproduced on main, with no new diagnostics
  • Browser checks covered suggestions, manual edits, partial results, retry and narrow layouts; live generation checks covered recommendations and a 50-case dataset

Additional startup checks passed 32 tests, with test_complete_studio_app_defers_model_catalog_network_runtime failing locally. The same failure was reproduced on an unchanged checkout of main.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant