feat(memtrack): pause producers under ring pressure - #543
not-matthias wants to merge 13 commits into
Conversation
Merging this PR will not alter performance
|
Share one poll interval across the event, stack and attach pollers and lower it from 10ms to 1ms so bursts drain before the rings fill.
A full allocation-stack ring loses stack records the same way a full event ring loses events, so a run that overflowed it must fail the same incompleteness check.
The FNV lanes lived on the BPF stack. Large kprobe-family programs may spill that to per-CPU storage, which a nested uprobe on the same CPU can overwrite mid-capture, corrupting the hash. Accumulate the lanes in the not-yet-submitted ring record instead, which is private to this reservation.
After every event or stack submission, check the ring's fill level. When it crosses the watermark, latch the first over-watermark timestamp in the `pressure_since` map and SIGSTOP the producing tracked task. While the latch is set, every further event from a tracked task re-stops it, so a producer resumed by another party cannot keep filling the ring. Userspace reads the latch via `pressure_since()` and clears it with `clear_pressure()` once the tree is quiesced and the rings are drained. The BPF gate is compiled out unless `blocking_enabled` is set, so this is inert until a consumer turns it on.
Replace the poller's ad-hoc `Sender<Sender<()>>` control channel with a `PollerHandle` that any thread can clone to request a deadline-bounded drain and wait for its acknowledgement. A drain is acked only when the ring is empty and every item read before the request has been resolved and forwarded; a ring stopped at an uncommitted reservation is retried within the remaining budget. The threaded poller routes requests through its resolver thread, and the attach worker services them between batches so a drain also means every queued attach request has been processed. `wait_all_stopped` treats exited threads as stopped, since a tree-wide sweep will meet zombies.
The tracker is about to own a second background worker; name the existing one after what it does.
A per-session PressureWorker polls the BPF pressure latch. On an episode it stops the whole tracked process tree, drains the attach, event and stack pipelines through their drain handles, clears the latch and resumes the tree, all within the configured blocking timeout. The attach worker keeps owning the resume of pids it stopped; the kernel latch re-stops any producer resumed mid-pause, so no cross-worker resume protocol is needed. Exceeding the timeout or any failure during the pause kills the tree and fails the run, which is still preferable to silently reporting an incomplete trace. `Session::finish` joins the worker before the pollers it drains are dropped and surfaces its failure. Configured with `--blocking-timeout` / `CODSPEED_MEMTRACK_BLOCKING_TIMEOUT` (`0`, `inf`, or a humantime duration); disabled by default, in which case overflows are reported after the run as before. The runner enables it with 15s.
15e7943 to
89556be
Compare
Log per-phase timings (stop, drain, resume, total) for each pressure episode and the episode count when the worker exits, so pause overhead can be attributed to the phase that dominates it. Add a poll_interval_ms tracker option (env CODSPEED_MEMTRACK_POLL_INTERVAL_MS, default 1ms) for the event and stack pollers. A slow poller lets the event ring cross its watermark on demand without rebuilding the BPF program with a smaller ring. Drain requests still wake the poller immediately, so the pause path is unaffected. The attach poller keeps its fixed interval. Add an alloc_storm fixture and a pressure test that runs it with a fast poller, a slow poller without blocking, and a slow poller with blocking. The test asserts that the non-blocking run drops events and the blocking run loses none.
|
| continue; | ||
| } | ||
| let mut batch = self.rx.try_iter().flatten().collect(); | ||
| if let Err(error) = self.process_batch(&mut batch, known) { |
There was a problem hiding this comment.
When an attach request is pending during a pressure episode, this barrier calls process_batch, whose guards unconditionally send SIGCONT before acknowledging completion. The pressure coordinator drains the attach worker before the event and stack pollers, so the target can resume producing events while those rings are supposed to remain quiet. Sustained allocation can refill or overflow the rings, or make finite recovery time out and kill the tracked process tree. Resume ownership must remain coordinated with the pressure pause, or the tree must be stopped and checked again after the attach barrier.
Knowledge Base Used: eBPF memory tracker
Prompt To Fix With AI
This is a comment left during a code review.
Path: crates/memtrack/src/ebpf/attach_worker.rs
Line: 200
Comment:
**Attach Work Resumes Producers**
When an attach request is pending during a pressure episode, this barrier calls `process_batch`, whose guards unconditionally send `SIGCONT` before acknowledging completion. The pressure coordinator drains the attach worker before the event and stack pollers, so the target can resume producing events while those rings are supposed to remain quiet. Sustained allocation can refill or overflow the rings, or make finite recovery time out and kill the tracked process tree. Resume ownership must remain coordinated with the pressure pause, or the tree must be stopped and checked again after the attach barrier.
**Knowledge Base Used:** [eBPF memory tracker](https://app.greptile.com/codspeed/-/custom-context/knowledge-base/codspeedhq/codspeed/-/docs/ebpf-memory-tracker.md)
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.…ING_STATS Each ring poller can now report how fast its ring is written and read, sampled from libbpf's mmapped producer/consumer positions (ring__producer_pos, ring__consumer_pos, ring__avail_data_size). This needs no BPF changes and no syscalls: a sample is a few memory loads per poll tick. With CODSPEED_MEMTRACK_RING_STATS=1, every poll thread logs a line per second at debug level and a whole-run summary at info level on shutdown: stacks ring (1s): wrote 818.7 MB/s, read 664.5 MB/s, drain 61398.1 MB/s, peak backlog 159.7 MiB (31.2%), busy 1.1% (max tick 1.02 ms) - wrote: producer position delta over wall time. It counts every reserved byte, including record headers and records the BPF side discards, so it measures ring pressure rather than artifact size. - read: consumer position delta over wall time. - drain: bytes consumed per second of poll-thread busy time, i.e. the read rate the poller can sustain. - peak backlog: largest unconsumed backlog seen at the start of a poll tick. - busy / max tick: share of wall time spent consuming, and the slowest tick. When the variable is unset, the poll loop does one Option check per tick.
Adds opt-in, best-effort backpressure for memtrack's BPF ring buffers.
When a ring crosses its watermark, BPF latches the episode and stops the winning tracked producer. A userspace coordinator then stops the owned process tree using pidfds, drains the event, stack, mapping, and attach pipelines, clears the latch, and resumes the tree. Finite timeouts terminate the tracked tree and fail the capture;
0remains the default and preserves drop detection, whileinfwaits until recovery. Detected ring loss still fails the capture.The change also:
This PR is stacked on #522.
Verification
cargo fmt --all -- --checkcargo clippy --release -p memtrack -p codspeed-runner -- -D warnings