Skip to content

[Feature] Instrument sandbox agents and SessionServer tracing - #2040

Open
matrix72c wants to merge 2 commits into
InternLM:mainfrom
matrix72c:feat/sandbox-otel-instrumentation
Open

[Feature] Instrument sandbox agents and SessionServer tracing#2040
matrix72c wants to merge 2 commits into
InternLM:mainfrom
matrix72c:feat/sandbox-otel-instrumentation

Conversation

@matrix72c

@matrix72c matrix72c commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Summary

Add OpenTelemetry instrumentation to the rollout, sandbox, and SessionServer paths so a request trace remains connected across Ray actors, sandbox stages, and the proxy lifecycle.

Motivation

The existing rollout and SessionServer paths already expose the live boundaries needed for tracing, but their OTel context was not consistently propagated across Ray calls or represented in sandbox and proxy spans. This made cross-process request traces incomplete even though the underlying training behavior was correct.

Implementation

  • Propagate the active OTel context through Ray generate_group calls for a single RolloutState or a concrete list/tuple group, while restoring internal carrier and call-chain fields after the call.
  • Bridge the existing sandbox stage timer to live OTel spans with stable names and a bounded attribute schema. Legacy JSONL enter/exit records and timing semantics remain unchanged.
  • Trace the SessionServer proxy lifecycle as prepare_request, backend_roundtrip, and record_response under one request span. Incoming aiohttp headers are used as the parent carrier, and the backend carrier is injected inside the roundtrip span.

Compatibility

  • No training behavior or request/response payload format changes.
  • No prompt, response body, or raw header attributes are exported.
  • No duplicate duration/ok telemetry is added; OTel owns duration and logical failures use error=true.
  • This PR instruments existing live call sites only. It does not add synthetic trace reconstruction, trajectory polling, timeline storage, or a flush API.

Tests

  • Focused rollout propagation, nested cleanup, sandbox bridge, SessionServer, and mixed-case W3C carrier tests.
  • Existing SingleTurnAgentLoop and producer compatibility tests: 29 passed.
  • ruff check on all changed Python files.
  • compileall on changed modules and tests.
  • git diff --check and final diff review against upstream/main.

@matrix72c
matrix72c force-pushed the feat/sandbox-otel-instrumentation branch 2 times, most recently from a340b7d to cdee4cc Compare August 24, 2026 14:09
@windreamer
windreamer requested a review from YanhuiDua August 24, 2026 23:16
@matrix72c
matrix72c force-pushed the feat/sandbox-otel-instrumentation branch from cdee4cc to 482a44b Compare August 25, 2026 02:44
@matrix72c matrix72c changed the title [Feat] Add sandbox and black-box agent tracing [Feat] Instrument sandbox agents and SessionServer tracing Aug 25, 2026
@matrix72c matrix72c changed the title [Feat] Instrument sandbox agents and SessionServer tracing [Feature] Instrument sandbox agents and SessionServer tracing Sep 2, 2026
@matrix72c
matrix72c force-pushed the feat/sandbox-otel-instrumentation branch from 1ebdb7c to 84ea888 Compare September 7, 2026 03:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant