[opentelemetry-instrumentation-genai-langchain] Capture number of inference and tool calls on agent invocations by rads-1996 · Pull Request #1001 · open-telemetry/opentelemetry-python-genai

rads-1996 · 2026-10-28T18:02:37Z

Description

Fixes #598

Type of change

Please delete options that are not relevant.

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

How has this been tested?

Please describe the tests that you ran to verify your changes. Provide
instructions so we can reproduce. List any relevant details for your test
configuration.

Test A

Checklist

See CONTRIBUTING.md
for the style guide, changelog guidance, and more.

Followed the style guidelines of this project
Changelog updated if the change requires an entry
Unit tests added
Documentation updated

opentelemetry-pr-dashboard · 2026-10-28T18:03:20Z

Pull request dashboard status

Waiting on maintainers · refreshed 2026-11-09 17:41 UTC

Merge when ready.

Status above doesn't look right?

Just replied or pushed? Anything around or after the refresh time above may not be picked up yet — give it a few minutes.
Anything look wrong? Report it with what you expected; it helps us improve the dashboard.

Copilot

🟡 Changes recommended

Agent call counts can miss direct model calls or callbacks from retained managers, and can overcount calls inside tools or retrievers.

3 open issues

What changed in this PR

Adds per-agent inference and tool call metrics to LangChain instrumentation, addressing #598 through the shared GenAI invocation utilities.

Changes:

Initializes agent call counts and increments them from model and tool callbacks.
Adds metric assertions for agent classification, nested agents, tool failures, and conformance scenarios.
Raises the shared utility dependency floor and adds a changelog entry.

All paths below are under instrumentation/opentelemetry-instrumentation-genai-langchain/.

File	Description
`tests/test_tools.py`	Checks tool counts on failures and nearest-agent attribution.
`tests/test_callback_handler.py`	Checks zero counts and unnamed-agent inference metrics.
`tests/requirements.oldest.txt`	Uses the local utility package pending release.
`tests/conformance/agent.py`	Validates both call-count histograms and expected totals.
`src/opentelemetry/instrumentation/genai/langchain/invocation_manager.py`	Initializes counts and resolves the nearest parent agent.
`src/opentelemetry/instrumentation/genai/langchain/callback_handler.py`	Counts text-model, chat-model, and tool starts.
`pyproject.toml`	Raises the utility dependency minimum to `1.4b0.dev`.
`.changelog/1001.added`	Documents the new agent call-count feature.

_{Review effort: Balanced}

Copilot

🟡 Changes recommended

Agent call counts can be omitted for AgentExecutor, lost under concurrency, or double-counted for delegated models, with missing failure and oldest-version test coverage.

5 open issues

3 resolved since last review

_{Review effort: Balanced}

Copilot

🟡 Changes recommended

Non-announced agents omit zero-tool-call observations, biasing the tool-call metric toward runs that invoke tools.

1 open issue

Record zero-call observations for non-announced agents · New

5 resolved since last review

_{Review effort: Balanced}

Copilot

🔵 Needs a closer look

Observed model and tool calls for supported non-graph agents are discarded, so their call-count metrics are not emitted.

No open issues 🎉

1 resolved since last review

Record zero-call observations for non-announced agents

_{Review effort: Balanced}

lmolkova · 2026-11-06T03:15:22Z

                if token is not None:
                    context.run(detach, token)

+    def _wrap_attempt_config(


Sorry, my earlier retry comment sent this in the wrong direction. LangChain retries/fallbacks each produce their own chat span, so every traced inference is already one logical call from our side. Could we drop the retry dedup (_InferenceCallScope, the patch_config/BaseChatModel.invoke/middleware patches, start_inference_attempt) and just increment the agent counter in on_chat_model_start when we start an inference span? That keeps the metric consistent with the spans.

Makes sense, made revisions to the code.

lmolkova · 2026-11-06T03:15:22Z

        ):
            for method in (
                "on_chat_model_start",
+                "on_llm_start",


question: this looks unrelated now that on_llm_start isn't counted. Is it left over from the earlier version? If so, could we drop it along with the llm case in test_execution_context.py, and send it as a separate fix if it's still needed?

Yes, it was left over from the previous iteration. Removed it in the updated version.

lmolkova · 2026-11-07T01:42:45Z

+@pytest.mark.asyncio
+@pytest.mark.parametrize("asynchronous", [False, True], ids=["sync", "async"])
+@pytest.mark.parametrize("streaming", [False, True], ids=["invoke", "stream"])
+@pytest.mark.parametrize("strategy", ["retry", "fallback"])
+@pytest.mark.parametrize("failure", [False, True])
+@pytest.mark.parametrize("use_tool", [False, True])
+@pytest.mark.parametrize(
+    "use_middleware",
+    [
+        False,
+        pytest.param(
+            True,
+            marks=pytest.mark.skipif(
+                not hasattr(langchain.agents, "create_agent"),
+                reason="Model middleware requires LangChain 1.x",
+            ),
+        ),
+    ],
+)
+async def test_native_agent_retries_count_traced_inferences(


Now that retry dedup is gone, counting is just "one per chat span", so this 64-case retry/fallback/middleware matrix (plus test_agent_inference_counts_direct_fallback_stream and test_independent_model_calls_share_one_agent_node) mostly re-tests the old design. Could we drop them and keep test_native_agent_call_counts with a single retry case? Those tests would also fit better in an agent test module than in test_tools.py.

Addressed. Created a new test_agent suite and added the test there.

opentelemetry-pr-dashboard Bot mentioned this pull request Oct 28, 2026

Pull Request Dashboard #171

Open

rads-1996 force-pushed the langcahin-agent-call-counts branch from 7528d97 to 5b7ad0c Compare October 28, 2026 19:10

rads-1996 requested a balanced review from Copilot October 28, 2026 19:28

Copilot started reviewing on behalf of rads-1996 October 28, 2026 19:29

View session

Copilot AI reviewed Oct 28, 2026

View reviewed changes

rads-1996 requested a balanced review from Copilot October 28, 2026 20:34

Copilot started reviewing on behalf of rads-1996 October 28, 2026 20:34

View session

Copilot AI reviewed Oct 28, 2026

View reviewed changes

rads-1996 force-pushed the langcahin-agent-call-counts branch from 71e3eda to 362c7bc Compare October 29, 2026 17:59

rads-1996 requested a balanced review from Copilot October 29, 2026 20:11

Copilot started reviewing on behalf of rads-1996 October 29, 2026 20:12

View session

Copilot AI reviewed Oct 29, 2026

View reviewed changes

Comment thread ...tation-genai-langchain/src/opentelemetry/instrumentation/genai/langchain/callback_handler.py Resolved

rads-1996 requested a balanced review from Copilot October 29, 2026 21:32

Copilot started reviewing on behalf of rads-1996 October 29, 2026 21:32

View session

Copilot AI reviewed Oct 29, 2026

View reviewed changes

rads-1996 marked this pull request as ready for review October 29, 2026 21:51

rads-1996 requested a review from a team as a code owner October 29, 2026 21:51

lmolkova mentioned this pull request Oct 31, 2026

[opentelemetry-instrumentation-genai-langchain] Some model calls produce no inference span #1023

Open

lmolkova reviewed Oct 31, 2026

View reviewed changes

Comment thread ...tation-genai-langchain/src/opentelemetry/instrumentation/genai/langchain/callback_handler.py Resolved

Comment thread ...tation-genai-langchain/src/opentelemetry/instrumentation/genai/langchain/callback_handler.py Resolved

lmolkova mentioned this pull request Oct 31, 2026

Clarify that gen_ai.invoke_agent.inference_calls counts logical inference operations, not retries open-telemetry/semantic-conventions-genai#662

Closed

rads-1996 force-pushed the langcahin-agent-call-counts branch from 8b01ee1 to cf8a014 Compare November 2, 2026 21:10

lzchen reviewed Nov 2, 2026

View reviewed changes

Comment thread ...tion-genai-langchain/src/opentelemetry/instrumentation/genai/langchain/invocation_manager.py Resolved

rads-1996 force-pushed the langcahin-agent-call-counts branch from cf8a014 to 53e21f2 Compare November 5, 2026 17:25

lmolkova reviewed Nov 6, 2026

View reviewed changes

rads-1996 force-pushed the langcahin-agent-call-counts branch from 53e21f2 to e955027 Compare November 6, 2026 17:04

lmolkova approved these changes Nov 7, 2026

View reviewed changes

rads-1996 added 3 commits November 9, 2026 09:32

Record Langchain inference and tool call counts

e60286e

Target root agent in call-count tests

46720f8

Omit call-count metrics for agents with unknown coverage

08ebba1

rads-1996 added 6 commits November 9, 2026 09:32

Exclude retry attempts from agent inference counts

79d2854

Deduplicate logical LangChain inference calls

bfd57b5

Align LangChain agent counts with traced inferences

843fdec

Remove leftover text-completion counter test

911da18

Trim agent counter regression

db675b8

Address test suite feedback

a7ea024

rads-1996 force-pushed the langcahin-agent-call-counts branch from 13102fe to a7ea024 Compare November 9, 2026 17:32

Conversation

rads-1996 commented Oct 28, 2026

Description

Type of change

How has this been tested?

Checklist

Uh oh!

opentelemetry-pr-dashboard Bot commented Oct 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull request dashboard status

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lmolkova Nov 6, 2026

Choose a reason for hiding this comment

Uh oh!

rads-1996 Nov 6, 2026

Choose a reason for hiding this comment

Uh oh!

lmolkova Nov 6, 2026

Choose a reason for hiding this comment

Uh oh!

rads-1996 Nov 6, 2026

Choose a reason for hiding this comment

Uh oh!

lmolkova Nov 7, 2026

Choose a reason for hiding this comment

Uh oh!

rads-1996 Nov 9, 2026

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

opentelemetry-pr-dashboard Bot commented Oct 28, 2026 •

edited

Loading