[lvaroqui]

Add codspeed run --mode telemetry, an experimental mode that collects the OpenTelemetry spans and logs the benchmarks export, and uploads them with the results.

It is a separate pass rather than a flag on the walltime run: nothing has to keep the export out of the measured runs, and each benchmark runs once instead of once per walltime iteration. Combine it with a measuring mode, e.g. --mode telemetry,walltime.

Analysis executor. The mode runs on a new generic analysis executor: it runs the benchmarks once each without a profiler, answering the integration as an analysis mode, with analysis instruments hooked onto it. An instrument sets up before the process is spawned, receives the integration's start and stop commands, and stops once the process exits. The executor saves the benchmark markers as ExecutionTimestamps.msgpack, and fails when no benchmark ran unless --allow-empty is set. Uploads carry executor: "analysis" and the enabled instruments, so later analysis instruments only add an instrument. CODSPEED_RUNNER_MODE is telemetry in this mode, which cargo codspeed run reads to pick its telemetry build.

Telemetry instrument.

• It starts an OTLP/HTTP receiver on a free localhost port and passes its URL to the benchmark in CODSPEED_OTEL_ENDPOINT. The receiver accepts protobuf and JSON exports, plain or gzipped, on /v1/{traces,logs,metrics}. Each trace and log request is decompressed and written to disk before it is acknowledged, so everything an exited process sent is kept. Metrics are acknowledged and dropped.
• Once the process exits, the requests are decoded into otel/spans.msgpack.zst and otel/logs.msgpack.zst in the profile folder, one row per span or log, formatted like the OpenTelemetry Collector's ClickHouse exporter. A request that fails to decode is skipped with a warning, and a file is only written when it has rows. The output of a previous run in a reused --profile-folder is removed first.
• The codspeed.monotonic_start_ns/codspeed.monotonic_end_ns span attributes and the codspeed.monotonic_time_ns log attribute, set by the integrations on the monotonic clock of the benchmark markers, move to their own row fields. With the markers, they tell which benchmark emitted each span or log. A warning counts the rows that lack them.

File format. runner_shared::telemetry defines the files, shared by the runner and the parsers: a zstd stream (16 MB window, long-distance matching, since rows repeat their resource attributes far apart) of positional MessagePack records. Records are versioned like UnwindData, with v1 fixtures that must keep decoding. TelemetryReader decodes a file from any Read as its bytes arrive.

Testing. tests/telemetry.rs builds the divan telemetry bench of a codspeed-rust checkout next to this repository, runs it through codspeed run --mode telemetry, and checks that every span and log falls within the rounds of the benchmark that emitted it, keeps its severity and call site, and that the benchmark without telemetry has no rows. CI has no codspeed-rust checkout, so it is ignored by default: cargo test --test telemetry -- --ignored. Set CODSPEED_OTEL_TEST_PROFILE_FOLDER to keep the profile folder for inspection. scripts/run-telemetry-example.sh runs the same example with --mode telemetry,walltime and uploads it.

Closes COD-3863

[codspeed]

Merging this PR will not alter performance

2 warnings

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ 8 benchmarks are dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

✅ 41 untouched benchmarks
⏩ 6 skipped benchmarks¹

---

_{Comparing cod-3863-featrunner-collect-otlp-with-rotel-and-upload-it-as-parquet (c40417d) with main (d9612e3)}

Footnotes

1. 6 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[greptile-apps]

Confidence Score: 4/5

[Medium impact] The change affects runs that enable telemetry mode and the profile data they upload.

The harness release and download pin need updating before telemetry works for plain commands.

Findings

1.  Plain telemetry commands fail ▶

Fix with agent prompt

### Issue 1
crates/exec-harness/src/lib.rs:72
Plain `Exec` targets still use the downloaded `exec-harness` version `1.4.1`. `AnalysisExecutor` passes `CODSPEED_RUNNER_MODE=telemetry`, but that released harness only accepts the old modes, so it exits before running the command. Adding support in this workspace does not update the downloaded binary.

Release the updated harness and bump its version and installer pin.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Adds experimental --mode telemetry through a new analysis executor.

• Telemetry runs save benchmark spans and logs with their timings.
• Telemetry rows use a shared, versioned format for writing and reading.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Telemetry mode] --> B[Analysis executor]
  B --> C[Benchmark process]
  C -->|OTLP HTTP| D[Local receiver]
  C -->|Benchmark markers| E[Execution timestamps]
  D --> F[Stored requests]
  F --> G[Compressed span and log files]
  E --> H[Analysis upload]
  G --> H

Loading

_{Reviews (2) · Last reviewed commit: "fix(otel): drop the telemetry output of ..." · Reviewed by Greptile}

[greptile-apps]

Plain telemetry commands fail

Plain Exec targets still use the downloaded exec-harness version 1.4.1. AnalysisExecutor passes CODSPEED_RUNNER_MODE=telemetry, but that released harness only accepts the old modes, so it exits before running the command. Adding support in this workspace does not update the downloaded binary.

Release the updated harness and bump its version and installer pin.

Knowledge Base Used:

• Benchmark execution engine
• Execution harness and instrumentation bindings

Prompt To Fix With AI

This is a comment left during a code review.
Path: crates/exec-harness/src/lib.rs
Line: 72

Comment:
**Plain telemetry commands fail**

Plain `Exec` targets still use the downloaded `exec-harness` version `1.4.1`. `AnalysisExecutor` passes `CODSPEED_RUNNER_MODE=telemetry`, but that released harness only accepts the old modes, so it exits before running the command. Adding support in this workspace does not update the downloaded binary.

Release the updated harness and bump its version and installer pin.

**Knowledge Base Used:**
- [Benchmark execution engine](https://app.greptile.com/codspeed/-/custom-context/knowledge-base/codspeedhq/codspeed/-/docs/benchmark-execution.md)
- [Execution harness and instrumentation bindings](https://app.greptile.com/codspeed/-/custom-context/knowledge-base/codspeedhq/codspeed/-/docs/execution-harness-and-instrumentation.md)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.