Repository navigation

Make the benchmarks able to resolve real effects - #15

Merged

brunoborges merged 19 commits into

mainfrom

improve-benchmark-statistics

Aug 5, 2026

Merged

Make the benchmarks able to resolve real effects#15
brunoborges merged 19 commits into
mainfrom
improve-benchmark-statistics

Commits on Aug 5, 2026
- Make the focused benchmark able to resolve real effects
  
  brunoborges
  and
  Copilot
  committed
  f14f9aa
- Keep the wrapper cache key stable between seeding and measuring
  
  brunoborges
  and
  Copilot
  committed
  1a624da
- Restore one shared cache entry in both arms
  
  brunoborges
  and
  Copilot
  committed
  7761291
- Stop the matrix from measuring its own cache contention
  
  brunoborges
  and
  Copilot
  committed
  01f3f0d
- Apply the paired methodology to the version sweep and JDK cache
  
  brunoborges
  and
  Copilot
  committed
  ebd51cf
- Keep PetClinic's build from linting the benchmark scripts
  
  brunoborges
  and
  Copilot
  committed
  319c61b
- Stop one stalled runner from holding a benchmark run open
  
  brunoborges
  and
  Copilot
  committed
  a4fc67a
- Discard the first slot of every measurement job
  
  brunoborges
  and
  Copilot
  committed
  8cd13cf
- Stop ranking the uncached versions against main
  
  brunoborges
  and
  Copilot
  committed
  efd11fd
- Stop a stalled slot from being reported as an effect
  
  brunoborges
  and
  Copilot
  committed
  77334eb
- Bound each measured setup so a stalled restore costs one runner
  
  brunoborges
  and
  Copilot
  committed
  c017f73
- Bound how long a failed cache download can stall a slot
  
  brunoborges
  and
  Copilot
  committed
  5f3c985
- Discard stalled runners in the version sweep too
  
  brunoborges
  and
  Copilot
  committed
  de4e834
- Give the configuration matrix a verdict it can support
  
  brunoborges
  and
  Copilot
  committed
  e5f870c
- Fail on missing report inputs instead of publishing undefined
  
  brunoborges
  and
  Copilot
  committed
  8a3c51c
- Compare version benchmarks within compatible cohorts
  
  brunoborges
  and
  Copilot
  committed
  ac70095
- Rank only the versions that do the same work
  
  brunoborges
  and
  Copilot
  committed
  2b3afa4
- Report the sweep as main against each version, not the reverse
  
  brunoborges
  and
  Copilot
  committed
  a967acc
- Ask each benchmark a question it can answer
  
  brunoborges
  and
  Copilot
  committed
  e1cb348

Commits

Commits on Aug 5, 2026