chore(release): prepare 0.27.0b2 with the sgp-obs log-duplication fix

[stephen-wang24]

Base branch

This targets stephen/agentex-sdk-obs-beta-base (cut from main at a71fa670), not main and not next. There is no plan to merge this to the mainline yet: the sgp-obs wiring was merged to next in #518 and reverted again in #522, and a batch of issues found while measuring it on sgp-dev is still open. This branch is where the wiring survives and gets fixed; the diff below is unchanged by the re-target.

What this branch is

• feat(obs): wire sgp-obs from the SDK for traces, metrics and logs #518's sgp-obs wiring (traces, metrics, logs) as a single commit
• the fixes below, found by measuring 0.27.0b1 on sgp-dev
• Python package versions set to 0.27.0b2, 0.27.0-b2 in the release-please manifest

Fixed in this update

An agent's own loggers printed every record twice

make_logger attaches a handler to each module's own logger. sgp-obs' logs pipeline replaces the handlers on the root logger and leaves named loggers alone, so both print. #518 handled that with a sweep plus a latch — but only the latch ignored the logger's name. The sweep matched the agentex prefix, so a logger created before init_sgp_obs under any other name kept its handler.

Agents call make_logger(__name__) from their own modules, and project.acp (the module that builds the ACP server, in every scaffold) logs at import — always before init. So every record it logged was emitted twice, 80µs apart:

	copy 1 (leaf handler)	copy 2 (pipeline)
identity key	name	logger
trace_id / span_id	absent	present
source / agent_id	absent	present
request_id	present	absent

The leaf copy is emitted before the pipeline's filters, so it also escapes the allowlist and the truncation. Measured on sgp-dev: 123 of 3361 lines on dbt-assistant (3.7%), 1 line on sync-simple, 0 on all six obs-test agents.

The SDK cannot know an agent's package name, so make_logger now marks each handler it attaches and the sweep takes back exactly those, on a logger of any name. The agentex.* prefix rule stays, where every handler is ours by definition. A handler this SDK did not attach is still left alone — litellm's three loggers and anything else keep what their owner set up.

route_agentex_loggers_to_root is now route_loggers_to_root, since "agentex loggers" is what the bug was.

request_id would have disappeared with the duplicate

The leaf handler was the only writer of request_id (via CustomJSONFormatter). That is why dbt-assistant showed request_id on 5.2% of lines against trace_id on 90.6% — different copies of the same records. Removing the duplicate removes the field's only writer, so the ACP middleware now binds its id into sgp-obs' shared correlation context, which the pipeline enriches every record from.

Binding rather than stamping the record is deliberate, and not a style choice: with request_id already set on the record, logger.info(..., extra={"request_id": ...}) raises KeyError: Attempt to overwrite 'request_id' in LogRecord from the stdlib. Verified — an earlier draft of this fix used a record factory and broke exactly that call. The enrich stage runs on a copy of the record at handler time instead, and treats a hand-set value as authoritative.

sgp-obs can also fill that context from its own RequestIdMiddleware. Binding the SDK's id keeps one generator for the value, so the id in the logs is the id ctx_var_request_id gives application code and the id x-request-id carried in.

Version

0.27.0b1 is not merely prepared, it is installed: the six obs-test agents pin agentex-sdk==0.27.0b1 and dbt-assistant runs it. Leaving these fixes under that version would make one version name two artifacts, so this is 0.27.0b2. Further fixes can ride b2 while it is unpublished.

Release scope

Still prepares, does not publish. Do not run publish-pypi.yml until the public-vs-private distribution path is confirmed; if public PyPI is selected, dispatch for agentex-sdk only. Agents pinning 0.27.0b1 need their pin moved before they pick this up.

Validation

• ruff check . — clean
• pyright -p . — 0 errors
• pytest — 1687 passed, 1622 skipped
• the duplicate reproduced before the fix and does not after: project.acp printed 2 lines, now 1; the agentex.* logger printed 1 throughout
• request_id bind/reset checked against the real sgp_obs.context module, not only the test stub: bound inside the request, matching ctx_var_request_id, cleared after

New tests: 5 in test_logging_handover.py (an agent-named logger prints once; only our handler comes off a shared logger; a non-propagating logger of ours is left alone; the latch covers agent loggers) and 8 in tests/test_request_id_correlation.py.

Not fixed here

From the same sgp-dev measurement, the following are not SDK regressions and are out of scope for this branch:

Finding	Owner
SGP tracing export 401 on every agent using the sgp team's api-key secret	invalid aws-dev-sgp-api-key-declan secret — a raw urllib probe with no SDK in the path 401s identically
obs-test agents log 0% trace_id	pre-existing; those agents never ran ddtrace, which is what populates log trace context
gen_ai_client_token_usage absent on dbt-assistant	openai-agents omits stream usage when base_url is not official OpenAI

Greptile's four findings on the b1 commit are also still open: the Dockerfile template's private-index secret, the unbounded shutdown flush, sync LiteLLM calls skipping metrics, and worker startup failures skipping cleanup.

🤖 Generated with Claude Code

Confidence Score: 2/5

This PR is not safe to merge until the private-index token is isolated and observability shutdown is bounded.

Findings

1.   Dependency builds can steal tokens ▶
2.  Exporter flush can block shutdown ▶
3.  Sync LiteLLM calls skip metrics ▶
4.  Startup failures skip cleanup ▶

Fix with agent prompt

### Issue 1
src/agentex/lib/cli/templates/default/Dockerfile.j2:40-44
`requirements.txt` can choose a source package whose build code runs inside this same `RUN`. That code inherits the credentialed `UV_DEFAULT_INDEX` and can read `codeartifact-pip-conf` while it is mounted. Keep the secret out of dependency build processes, or install only trusted private artifacts before running project-controlled builds. The same pattern appears in the other requirements-based Dockerfile templates.

**How this was verified:** Project-controlled requirements run build code while the private-index secret and credentialed environment value are both available.

### Issue 2
src/agentex/lib/core/observability/sgp_obs_setup.py:339
`sgp_obs.shutdown` does blocking exporter work, but this await has no deadline. If an exporter or collector stalls, ACP and Temporal shutdown can wait past the pod grace period. `asyncio.to_thread` also leaves a stalled default-executor thread to be joined at process exit. Give the whole flush a hard budget and use an exit path that cannot be held open by the worker thread.

### Issue 3
src/agentex/lib/core/adapters/llm/adapter_litellm.py:22-30
Only the async methods use `inference_call`. A caller using the public sync `completion` or `completion_stream` API with a native LiteLLM provider gets no GenAI timing, usage, or failure metrics because that route never reaches the OpenAI client instrumentor. Wrap both sync paths with the matching recorder and add sync tests.

### Issue 4
src/agentex/lib/core/temporal/workers/worker.py:237-239
`init_sgp_obs` runs before the cleanup `try`. If health startup, registration, client creation, workflow checks, or `Worker` construction fails, the new providers never reach `shutdown_sgp_obs`, so buffered startup telemetry and exporter resources are left behind. Start the `try` immediately after initialization so every later exit drains what was created.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Agentex now supports optional sgp-obs observability and private package installs while preparing both packages for the 0.27.0b1 beta release. The observability setup stays inactive unless an agent installs the private package and enables the matching environment settings.

• Adds metrics, traces, logs, and shutdown flushing across ACP and Temporal runtimes.
• Adds vendor-aware, fail-open GenAI metrics around litellm calls.
• Lets scaffold Dockerfiles use an optional CodeArtifact secret for private packages.
• Updates package and release metadata for the beta version.

Diagram

sequenceDiagram
    participant Build as Agent image build
    participant Broker as Package broker
    participant UV as uv install
    participant ACP as ACP server
    participant Worker as Temporal worker
    participant Obs as sgp-obs
    participant Backend as Telemetry backend

    Broker-->>Build: Mount private index secret
    Build->>UV: Install project dependencies
    UV->>UV: Run selected package build code
    Build->>ACP: Start server
    ACP->>Obs: init_sgp_obs(app)
    Build->>Worker: Start worker process
    Worker->>Obs: init_sgp_obs()
    ACP->>Worker: Dispatch Temporal work
    Worker->>Obs: Record model and runtime telemetry
    Obs->>Backend: Export signals
    par ACP shutdown
        ACP->>ACP: Drain async and sync spans
        ACP->>Obs: shutdown_sgp_obs()
    and Worker shutdown
        Worker->>Worker: Drain sync spans
        Worker->>Obs: shutdown_sgp_obs()
    end

Loading

_{Reviews (1) · Last reviewed commit: "chore(release): prepare 0.27.0b1"}

[stephen-wang24]

@greptileai

[greptile-apps]

requirements.txt can choose a source package whose build code runs inside this same RUN. That code inherits the credentialed UV_DEFAULT_INDEX and can read codeartifact-pip-conf while it is mounted. Keep the secret out of dependency build processes, or install only trusted private artifacts before running project-controlled builds. The same pattern appears in the other requirements-based Dockerfile templates.

How this was verified: Project-controlled requirements run build code while the private-index secret and credentialed environment value are both available.

Knowledge Base Used: Command-line workflows

Prompt To Fix With AI

This is a comment left during a code review.
Path: src/agentex/lib/cli/templates/default/Dockerfile.j2
Line: 40-44

Comment:
`requirements.txt` can choose a source package whose build code runs inside this same `RUN`. That code inherits the credentialed `UV_DEFAULT_INDEX` and can read `codeartifact-pip-conf` while it is mounted. Keep the secret out of dependency build processes, or install only trusted private artifacts before running project-controlled builds. The same pattern appears in the other requirements-based Dockerfile templates.

**How this was verified:** Project-controlled requirements run build code while the private-index secret and credentialed environment value are both available.

**Knowledge Base Used:** [Command-line workflows](https://app.greptile.com/scale-ai/-/custom-context/knowledge-base/scaleapi/scale-agentex-python/-/docs/command-line-workflows.md)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[greptile-apps]

sgp_obs.shutdown does blocking exporter work, but this await has no deadline. If an exporter or collector stalls, ACP and Temporal shutdown can wait past the pod grace period. asyncio.to_thread also leaves a stalled default-executor thread to be joined at process exit. Give the whole flush a hard budget and use an exit path that cannot be held open by the worker thread.

Knowledge Base Used:

• Observability
• Tracing pipeline

Prompt To Fix With AI

This is a comment left during a code review.
Path: src/agentex/lib/core/observability/sgp_obs_setup.py
Line: 339

Comment:
`sgp_obs.shutdown` does blocking exporter work, but this await has no deadline. If an exporter or collector stalls, ACP and Temporal shutdown can wait past the pod grace period. `asyncio.to_thread` also leaves a stalled default-executor thread to be joined at process exit. Give the whole flush a hard budget and use an exit path that cannot be held open by the worker thread.

**Knowledge Base Used:**
- [Observability](https://app.greptile.com/scale-ai/-/custom-context/knowledge-base/scaleapi/scale-agentex-python/-/docs/observability.md)
- [Tracing pipeline](https://app.greptile.com/scale-ai/-/custom-context/knowledge-base/scaleapi/scale-agentex-python/-/docs/tracing-pipeline.md)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[greptile-apps]

init_sgp_obs runs before the cleanup try. If health startup, registration, client creation, workflow checks, or Worker construction fails, the new providers never reach shutdown_sgp_obs, so buffered startup telemetry and exporter resources are left behind. Start the try immediately after initialization so every later exit drains what was created.

Knowledge Base Used:

• Temporal execution
• Observability

Prompt To Fix With AI

This is a comment left during a code review.
Path: src/agentex/lib/core/temporal/workers/worker.py
Line: 237-239

Comment:
`init_sgp_obs` runs before the cleanup `try`. If health startup, registration, client creation, workflow checks, or `Worker` construction fails, the new providers never reach `shutdown_sgp_obs`, so buffered startup telemetry and exporter resources are left behind. Start the `try` immediately after initialization so every later exit drains what was created.

**Knowledge Base Used:**
- [Temporal execution](https://app.greptile.com/scale-ai/-/custom-context/knowledge-base/scaleapi/scale-agentex-python/-/docs/temporal-execution.md)
- [Observability](https://app.greptile.com/scale-ai/-/custom-context/knowledge-base/scaleapi/scale-agentex-python/-/docs/observability.md)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[stephen-wang24]

https://github.com/scaleapi/scale-agentex-python/actions/runs/35035099786 published to public pypi landed https://pypi.org/project/agentex-sdk/#history