[Feature] Support external OTLP collector for trace runtime

[matrix72c]

Summary

Allow XTuner processes to export spans to an externally managed, network-reachable OTLP collector while preserving the existing driver-local collector and viewer defaults.

Motivation

The default trace path introduced in #1946 uses a collector on the driver's loopback address. In a multi-pod deployment, a remote Ray actor resolves 127.0.0.1 to its own pod, so it cannot reach the driver's collector and its spans are missing from the JSONL and Jaeger traces.

This PR allows the producers to use a routable collector Service instead:

driver / Ray actors in multiple pods
        | OTLP/gRPC
        v
external collector Service
        |-- file exporter --> shared traces.jsonl --> XTuner viewer
        `-- exporter ------> Jaeger or another backend

Configuration

TraceConfig(
    enabled=True,
    external_otlp_endpoint="http://otel-collector.namespace.svc:4317",
    external_trace_jsonl_path="/shared/traces/traces.jsonl",
    xtuner_viewer_enabled=True,
)

• external_otlp_endpoint is propagated to Ray child processes so every process exports to the same reachable collector.
• The external collector is deployed, configured, started, and stopped outside XTuner.
• external_trace_jsonl_path only tells the driver-side viewer where to read the collector-owned JSONL file; it does not configure or start the collector and is not propagated to Ray child processes.
• When the viewer is enabled, the path must refer to the same underlying file written by the external collector and must be visible from the driver, for example through a shared volume.
• The viewer filters a shared JSONL file by service.name and XTuner run.id, so the file may contain spans from multiple runs.
• If the external collector only exports to Jaeger or another backend, leave xtuner_viewer_enabled=False; no JSONL path is required.
• Viewer JSONL loading is deferred until the first request so the external collector may create the file after the viewer process starts.

Compatibility

When external_otlp_endpoint is unset, local collector startup, per-run traces.jsonl output, viewer startup, and shutdown behavior remain unchanged. External collectors are never started or stopped by XTuner.

Tests

• 8 focused tests covering local defaults, external collector lifecycle, endpoint propagation to Ray children, collector ownership of JSONL, deferred viewer loading, shared JSONL viewer configuration, and invalid configuration combinations.
• ruff check.
• compileall.
• git diff --check.

[YanhuiDua]

最简单的修复是否是 driver 提供的endpoint为其真实IP，而非 127.0.0.1，这个问题能否一起帮忙修复下？支持external collector 本身没问题。

[YanhuiDua]

@claude review

[github-actions]

Claude:

Summary

本 PR 新增 external_otlp_endpoint / external_trace_jsonl_path,使多 Pod 部署下所有进程可导出到同一个外部 OTLP collector,同时把 driver 本地 collector 的 endpoint 从 127.0.0.1 改为可路由地址,兼容路径(未配置 external 时)行为保持不变。整体方向合理、默认行为未破坏,但在「viewer 先于 JSONL 文件启动」这一本 PR 新引入的窗口上存在未处理异常且无单测覆盖,另有 collector 模式被拆成多个相关联可选字段导致的设计摩擦。

Main Flowchart after this PR

flowchart TD
    A[configure_trace_runtime] --> B[_build_trace_runtime_handle]
    B --> C{external_otlp_endpoint 已配置?}
    C -- 是 --> D["endpoint = external<br/>trace_jsonl_path = external 或 None<br/>port = None<br/>start_local_collector = False"]
    C -- 否 --> E["创建 run_dir/traces/traces.jsonl<br/>endpoint = http://_local_advertised_host:port<br/>start_local_collector = True"]
    D --> F[handle.start]
    E --> F
    F --> G{"mode==driver 且 start_local_collector?"}
    G -- 是 --> H["三处防御式 RuntimeError 校验<br/>_OTelCollector.start"]
    G -- 否 --> I[_configure_tracer_provider]
    H --> I
    I --> J{viewer 已启用?}
    J -- 是 --> K["校验 trace_jsonl_path<br/>启动 viewer 子进程"]
    J -- 否 --> L[env_vars 注入 Ray 子进程]
    K --> M["viewer: payload_cache 延迟加载<br/>首个请求才 stat + load"]
    M --> N{JSONL 文件已存在?}
    N -- 否 --> O["FileNotFoundError 未捕获<br/>do_GET 抛栈,连接被重置"]
    N -- 是 --> P[返回 payload]
    K --> L

    style D fill:#cfe8ff
    style E fill:#cfe8ff
    style G fill:#cfe8ff
    style H fill:#ffe0b2
    style M fill:#cfe8ff
    style O fill:#ffcdd2

Loading

说明:蓝色为本 PR 改动节点,橙色为设计摩擦点,红色为 Warning 级缺陷点。

核心原理实现与单测

核心行为均有真实代码路径覆盖,mock 只限于项目外部依赖(_configure_tracer_provider、_OTelCollector.start),符合项目测试约定:

• external endpoint 向 Ray 子进程传播:test_ray_child_inherits_external_endpoint_without_trace_jsonl 走真实 ensure_trace_runtime_from_env。
• 本地 collector 默认路径不变:test_local_collector_remains_the_default 验证 start_local_collector、端口与可路由 endpoint。
• 外部 JSONL 由 collector 拥有、XTuner 不创建也不传播:test_external_trace_jsonl_is_owned_by_collector_and_not_propagated。
• viewer 子进程在 JSONL 尚不存在时可启动:test_external_viewer_process_starts_before_trace_jsonl_exists。
• 两处非法配置组合:两个 ValidationError 用例。

缺口在于「JSONL 尚未生成时对 viewer 发起真实请求」这一新窗口没有任何用例(见下文单测建议)。

抽象与信息隐藏评估

• Warning xtuner/v1/rl/trace/runtime.py:341-342,358-376,488-508:collector 模式被编码为 start_local_collector / collector_port / trace_jsonl_path 三个相互关联的可选字段,不变量在 _build_trace_runtime_handle 建立却在 start() 用三处防御式 RuntimeError 重复校验,规则 Locality 被拆到两处。

公开 Interface 的线性业务流程评估

• Warning xtuner/v1/rl/trace/runtime.py:358-376:_TraceRuntimeHandle.start() 的主流程被三处低层级的 None 校验打断,不再是「启动 collector → 配置 provider → 启动 viewer」的线性叙述。

单测建议

• Warning tests/rl/test_trace.py:99-148:test_external_viewer_lazily_loads_trace_jsonl 只在写入文件之后发起请求,本 PR 新启用的「文件尚未创建时提供服务」这一核心行为完全没有请求级覆盖。

其他 Issues

• Warning recipe/trace/viewer/server.py:180-204:移除预热加载后,JSONL 尚未生成期间每个请求都会在 _source_signature 的 path.stat() 或 load_jaeger_traces_from_otel_jsonl 处抛异常,do_GET 未捕获,客户端得到连接重置而非 5xx 或空视图。
• Warning xtuner/v1/rl/trace/runtime.py:83-94:_local_advertised_host() 吞掉所有异常并回退到 gethostbyname(gethostname()),在 Debian/Ubuntu 上常返回 127.0.1.1,会无声地退回本 PR 要修复的 driver 不可达问题且没有任何日志。
• Warning xtuner/v1/rl/trace/runtime.py:184:公开导出的 TraceRuntime.trace_jsonl_path 由 Path 放宽为 Path | None 且 inherited 模式下一律为 None(含未改动的本地 collector 默认场景),同时 XTUNER_OTEL_JSONL_PATH 从 TRACE_ENV_KEYS 移除,仓库内无消费者但下游读取方会静默拿到 None。

Verdict

REQUEST_CHANGES