Skip to content

feat(memtrack): pause producers under ring pressure - #543

Open
not-matthias wants to merge 13 commits into
cod-3222-add-ebpf-based-dwarffp-unwindingfrom
feat/memtrack-pause-worker
Open

not-matthias wants to merge 13 commits into
cod-3222-add-ebpf-based-dwarffp-unwindingfrom
feat/memtrack-pause-worker

Conversation

@not-matthias

Copy link
Copy Markdown
Member

Adds opt-in, best-effort backpressure for memtrack's BPF ring buffers.

When a ring crosses its watermark, BPF latches the episode and stops the winning tracked producer. A userspace coordinator then stops the owned process tree using pidfds, drains the event, stack, mapping, and attach pipelines, clears the latch, and resumes the tree. Finite timeouts terminate the tracked tree and fail the capture; 0 remains the default and preserves drop detection, while inf waits until recovery. Detected ring loss still fails the capture.

The change also:

  • lowers ring polling latency from 10 ms to 1 ms
  • keeps attach/startup resume ownership coordinated with pressure pauses
  • propagates the timeout through the runner's experimental memory-mode flag
  • avoids x86 per-CPU private BPF stack corruption during nested uprobes by keeping hash scratch in the unpublished ring record and reducing BPF stack use

This PR is stacked on #522.

Verification

  • cargo fmt --all -- --check
  • cargo clippy --release -p memtrack -p codspeed-runner -- -D warnings
  • Runtime tests are deferred to CI.

@codspeed

codspeed Bot commented Sep 22, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

✅ 33 untouched benchmarks
⏩ 4 skipped benchmarks1


Comparing feat/memtrack-pause-worker (31b6da1) with cod-3222-add-ebpf-based-dwarffp-unwinding (2152fe2)

Open in CodSpeed

Footnotes

  1. 4 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

Share one poll interval across the event, stack and attach pollers and
lower it from 10ms to 1ms so bursts drain before the rings fill.
A full allocation-stack ring loses stack records the same way a full
event ring loses events, so a run that overflowed it must fail the same
incompleteness check.
The FNV lanes lived on the BPF stack. Large kprobe-family programs may
spill that to per-CPU storage, which a nested uprobe on the same CPU can
overwrite mid-capture, corrupting the hash. Accumulate the lanes in the
not-yet-submitted ring record instead, which is private to this
reservation.
After every event or stack submission, check the ring's fill level. When
it crosses the watermark, latch the first over-watermark timestamp in the
`pressure_since` map and SIGSTOP the producing tracked task. While the
latch is set, every further event from a tracked task re-stops it, so a
producer resumed by another party cannot keep filling the ring.

Userspace reads the latch via `pressure_since()` and clears it with
`clear_pressure()` once the tree is quiesced and the rings are drained.
The BPF gate is compiled out unless `blocking_enabled` is set, so this is
inert until a consumer turns it on.
Replace the poller's ad-hoc `Sender<Sender<()>>` control channel with a
`PollerHandle` that any thread can clone to request a deadline-bounded
drain and wait for its acknowledgement. A drain is acked only when the
ring is empty and every item read before the request has been resolved
and forwarded; a ring stopped at an uncommitted reservation is retried
within the remaining budget.

The threaded poller routes requests through its resolver thread, and the
attach worker services them between batches so a drain also means every
queued attach request has been processed. `wait_all_stopped` treats
exited threads as stopped, since a tree-wide sweep will meet zombies.
The tracker is about to own a second background worker; name the existing one after what it does.
A per-session PressureWorker polls the BPF pressure latch. On an episode
it stops the whole tracked process tree, drains the attach, event and
stack pipelines through their drain handles, clears the latch and resumes
the tree, all within the configured blocking timeout. The attach worker
keeps owning the resume of pids it stopped; the kernel latch re-stops any
producer resumed mid-pause, so no cross-worker resume protocol is needed.

Exceeding the timeout or any failure during the pause kills the tree and
fails the run, which is still preferable to silently reporting an
incomplete trace. `Session::finish` joins the worker before the pollers
it drains are dropped and surfaces its failure.

Configured with `--blocking-timeout` / `CODSPEED_MEMTRACK_BLOCKING_TIMEOUT`
(`0`, `inf`, or a humantime duration); disabled by default, in which case
overflows are reported after the run as before. The runner enables it
with 15s.
@not-matthias
not-matthias force-pushed the feat/memtrack-pause-worker branch from 15e7943 to 89556be Compare September 23, 2026 12:23
Log per-phase timings (stop, drain, resume, total) for each pressure episode and the episode count when the worker exits, so pause overhead can be attributed to the phase that dominates it.

Add a poll_interval_ms tracker option (env CODSPEED_MEMTRACK_POLL_INTERVAL_MS, default 1ms) for the event and stack pollers. A slow poller lets the event ring cross its watermark on demand without rebuilding the BPF program with a smaller ring. Drain requests still wake the poller immediately, so the pause path is unaffected. The attach poller keeps its fixed interval.

Add an alloc_storm fixture and a pressure test that runs it with a fast poller, a slow poller without blocking, and a slow poller with blocking. The test asserts that the non-blocking run drops events and the blocking run loses none.
@not-matthias
not-matthias marked this pull request as ready for review September 23, 2026 12:42
@not-matthias
not-matthias added this pull request to stack #545 September 23, 2026 12:43
@greptile-apps

greptile-apps Bot commented Sep 23, 2026

Copy link
Copy Markdown

RetriggerConfidence Score: 4/5

The PR is not safe to merge until attach processing can no longer resume producers during pressure draining and the explicit Rust logging rule is satisfied.

Fix All in Claude CodeFindings

  1. P1 Attach Work Resumes Producers
  2. P2 Direct Stderr Logging Added
Fix with agent prompt
### Issue 1
crates/memtrack/src/ebpf/attach_worker.rs:200
When an attach request is pending during a pressure episode, this barrier calls `process_batch`, whose guards unconditionally send `SIGCONT` before acknowledging completion. The pressure coordinator drains the attach worker before the event and stack pollers, so the target can resume producing events while those rings are supposed to remain quiet. Sustained allocation can refill or overflow the rings, or make finite recovery time out and kill the tracked process tree. Resume ownership must remain coordinated with the pressure pause, or the tree must be stopped and checked again after the attach barrier.

### Issue 2
crates/memtrack/tests/pressure_tests.rs:43
This test uses `eprintln!` here and again at lines 59, 62, 70, and 79. That violates the repository directive: “Never use `println!`/`eprintln!` in Rust; use tracing macros (`info!`/`debug!`/`warn!`/`error!`).” Replace these diagnostics with tracing macros; this repository requirement must be satisfied before merging.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

This PR adds opt-in memtrack ring-buffer backpressure, including a BPF pressure latch, pidfd-based process-tree suspension, timeout-bounded pipeline barriers, runner configuration, and pressure integration coverage.

  • Adds 0, finite-duration, and inf timeout parsing and propagation.
  • Coordinates event, stack, mapping, and attach pipeline draining during pressure episodes.
  • Moves stack-hash scratch into unpublished ring records and reduces normal polling latency.
  • One resume-ownership race remains between attach processing and pressure recovery.
  • The new pressure test also violates the repository’s Rust logging rule.
Diagram
%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[BPF ring crosses watermark] --> B[Latch pressure episode]
  B --> C[Pressure worker scans tracked tree]
  C --> D[SIGSTOP tracked processes]
  D --> E[Drain attach pipeline]
  E --> F[Process pending attach requests]
  F --> G[Attach guards send SIGCONT]
  G --> H[Drain event and stack rings]
  H --> I[Clear pressure latch]
  I --> J[SIGCONT owned tree]
  G -. producer may refill rings .-> H
Loading

Reviews (1) · Last reviewed commit: "fixup! feat(memtrack): latch ring pressu..."

continue;
}
let mut batch = self.rx.try_iter().flatten().collect();
if let Err(error) = self.process_batch(&mut batch, known) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Attach Work Resumes Producers

When an attach request is pending during a pressure episode, this barrier calls process_batch, whose guards unconditionally send SIGCONT before acknowledging completion. The pressure coordinator drains the attach worker before the event and stack pollers, so the target can resume producing events while those rings are supposed to remain quiet. Sustained allocation can refill or overflow the rings, or make finite recovery time out and kill the tracked process tree. Resume ownership must remain coordinated with the pressure pause, or the tree must be stopped and checked again after the attach barrier.

Knowledge Base Used: eBPF memory tracker

Prompt To Fix With AI
This is a comment left during a code review.
Path: crates/memtrack/src/ebpf/attach_worker.rs
Line: 200

Comment:
**Attach Work Resumes Producers**

When an attach request is pending during a pressure episode, this barrier calls `process_batch`, whose guards unconditionally send `SIGCONT` before acknowledging completion. The pressure coordinator drains the attach worker before the event and stack pollers, so the target can resume producing events while those rings are supposed to remain quiet. Sustained allocation can refill or overflow the rings, or make finite recovery time out and kill the tracked process tree. Resume ownership must remain coordinated with the pressure pause, or the tree must be stopped and checked again after the attach barrier.

**Knowledge Base Used:** [eBPF memory tracker](https://app.greptile.com/codspeed/-/custom-context/knowledge-base/codspeedhq/codspeed/-/docs/ebpf-memory-tracker.md)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fix in Claude Code Fix in Codex

Comment thread crates/memtrack/tests/pressure_tests.rs
…ING_STATS

Each ring poller can now report how fast its ring is written and read,
sampled from libbpf's mmapped producer/consumer positions
(ring__producer_pos, ring__consumer_pos, ring__avail_data_size). This needs
no BPF changes and no syscalls: a sample is a few memory loads per poll tick.

With CODSPEED_MEMTRACK_RING_STATS=1, every poll thread logs a line per
second at debug level and a whole-run summary at info level on shutdown:

  stacks ring (1s): wrote 818.7 MB/s, read 664.5 MB/s, drain 61398.1 MB/s,
  peak backlog 159.7 MiB (31.2%), busy 1.1% (max tick 1.02 ms)

- wrote: producer position delta over wall time. It counts every reserved
  byte, including record headers and records the BPF side discards, so it
  measures ring pressure rather than artifact size.
- read: consumer position delta over wall time.
- drain: bytes consumed per second of poll-thread busy time, i.e. the read
  rate the poller can sustain.
- peak backlog: largest unconsumed backlog seen at the start of a poll tick.
- busy / max tick: share of wall time spent consuming, and the slowest tick.

When the variable is unset, the poll loop does one Option check per tick.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant