feat(gooddata-eval): add --tag to put custom tags on Langfuse traces

[janmatzek]

gd-eval run gets a repeatable --tag option that adds custom tags to the Langfuse traces gd-eval writes. The gen-ai agent's own traces are not tagged.

Ticket: GDAI-2568

What this implements

gd-eval run ... --langfuse-dataset <ds> --langfuse --tag acme --tag nightly

Path	Tags on the trace
single-turn	gd-eval, <test_kind>, <provider_type>, effort-<level>, acme, nightly
agentic	gd-eval, <testing_framework>, acme, nightly

Custom tags come after the automatic ones. Blank tags are rejected, and duplicates are dropped. A run without --tag writes the same tags as before.

Decisions

1. The agentic path carries the tags in run_metadata_extra["tags"].
   Every evaluate_agentic_* already accepts run_metadata_extra, so no signature changes (cli/agentic_runner.py, core/agentic/_langfuse.py).

• The tags also show in the experiment-run metadata.
• Library callers can set tags the same way.

2. Tags apply wherever traces are written (best effort).

• Single-turn: only with --langfuse.
• Agentic: whenever LANGFUSE_* is set, also without --langfuse.
  --tag without --langfuse is not an error.

Test plan

• make -C packages/gooddata-eval format lint type-check: pass
• TEST_ENVS=py314 make -C packages/gooddata-eval test: 1623 passed, 1 skipped
• Manual run against a local stack with a 4-item Langfuse dataset (2 single-turn, 2 agentic):
  • with --tag: custom tags present on both paths, each once
  • without --tag: only the automatic tags
  • with --tag, without --langfuse: tags on the agentic traces only

Risk

low — opt-in CLI option; no change when it is not used.

🤖 Generated by Claude

Summary by CodeRabbit

• New Features
  • Added a repeatable --tag option to attach custom tags to Langfuse traces. Tags are trimmed, blank values are rejected, and duplicates are removed while preserving order.
  • Custom tags are included in traces for both single-turn and agentic evaluations, after automatic tags.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Organization UI
• Review profile: CHILL
• Plan: Essentials
• Run ID: 0346a870-b575-4cfe-8e3e-05fe63b0683c

---

📥 Commits

Reviewing files that changed from the base of the PR and between 0d95295 and a540bcd.

---

📒 Files selected for processing (10)

• packages/gooddata-eval/src/gooddata_eval/cli/agentic_runner.py
• packages/gooddata-eval/src/gooddata_eval/cli/main.py
• packages/gooddata-eval/src/gooddata_eval/core/agentic/_langfuse.py
• packages/gooddata-eval/src/gooddata_eval/core/config.py
• packages/gooddata-eval/src/gooddata_eval/core/langfuse/experiment.py
• packages/gooddata-eval/src/gooddata_eval/core/langfuse/sink.py
• packages/gooddata-eval/tests/test_agentic_observe_experiment.py
• packages/gooddata-eval/tests/test_agentic_runner.py
• packages/gooddata-eval/tests/test_cli.py
• packages/gooddata-eval/tests/test_langfuse_sink.py

---

Included review availability: This review used your included allowance. 4 included reviews remain after this review. Your included PR review attempts over the past 7 days set your current allowance at 5 reviews per hour.

---

---

📝 Walkthrough

Walkthrough

The CLI now accepts repeatable custom tags for Langfuse traces. It normalizes and deduplicates tags, then passes them to single-turn and agentic evaluation paths. Trace spans combine custom tags with automatic tags.

Changes

Custom Trace Tags

Layer / File(s)	Summary
Parse and route CLI tags packages/gooddata-eval/src/gooddata_eval/core/config.py, packages/gooddata-eval/src/gooddata_eval/cli/main.py, packages/gooddata-eval/tests/test_cli.py	The CLI accepts repeatable --tag values, strips whitespace, rejects blank values, and deduplicates tags in first-seen order. It passes the resulting tuple to the single-turn sink and agentic runner.
Forward tags through agentic evaluation packages/gooddata-eval/src/gooddata_eval/cli/agentic_runner.py, packages/gooddata-eval/tests/test_agentic_runner.py	The agentic runner creates run_metadata_extra from non-empty tags and forwards it to each evaluator.
Build trace tags packages/gooddata-eval/src/gooddata_eval/core/langfuse/experiment.py, packages/gooddata-eval/src/gooddata_eval/core/langfuse/sink.py, packages/gooddata-eval/src/gooddata_eval/core/agentic/_langfuse.py, packages/gooddata-eval/tests/test_langfuse_sink.py, packages/gooddata-eval/tests/test_agentic_observe_experiment.py	trace_tags removes empty and duplicate values while preserving order. Single-turn and agentic spans add custom tags after their automatic tags.

Priority: ⬇️ Low

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant CLI
  participant RunConfig
  participant LangfuseSink
  participant AgenticRunner
  participant Evaluator
  participant TraceSpan
  CLI->>RunConfig: Store normalized tag tuple
  RunConfig->>LangfuseSink: Pass tags to single-turn sink
  LangfuseSink->>TraceSpan: Combine automatic and custom tags
  RunConfig->>AgenticRunner: Pass tags to agentic run
  AgenticRunner->>Evaluator: Forward run_metadata_extra
  Evaluator->>TraceSpan: Add metadata tags after automatic tags

Loading

---

Merge Risk: ⚪ Minimal · up to a540b

No actionable issue remains identified; the change is mergeable after normal checks.

Pre-merge checks | 4 | 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 22.73% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 44 functions across 10 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check		Check skipped - CodeRabbit’s high-level summary is enabled.
Title check		The title clearly and concisely describes the main change: adding the repeatable --tag option for custom Langfuse trace tags.
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

---

• Fix all pre-merge checks with AI

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

_{I’m a rabbit with tags in my pack,
I trim every space from the stack.
Duplicates hop out of the line,
While trace tags join in order, just fine.
I nibble a leaf, then inspect the track.}

_{Comment @coderabbitai help to get the list of available commands.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 84.48%. Comparing base (a1ae1c2) to head (a540bcd).
⚠️ Report is 4 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1869      +/-   ##
==========================================
+ Coverage   84.19%   84.48%   +0.28%     
==========================================
  Files         333      334       +1     
  Lines       23261    23673     +412     
==========================================
+ Hits        19585    19999     +414     
+ Misses       3676     3674       -2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.