Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 8 additions & 12 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,26 +12,23 @@ wake up to a branch of accepted commits and a log of every experiment,
including the failures.

Inspired by [karpathy/autoresearch](https://github.com/karpathy/autoresearch).
This is a Python port of [g4lb/autor3search-go](https://github.com/g4lb/autor3search-go),
its Go sibling — same design, same guarantees, a different metric source: where
the Go harness reads `ns/op` out of `go test -bench`, this one reads per-round
timings out of [pytest-benchmark](https://pytest-benchmark.readthedocs.io/).
The metric source is [pytest-benchmark](https://pytest-benchmark.readthedocs.io/):
the harness reads per-round timings out of its JSON output.

> **Status: early but working.** Validated against one real library —
> [`humanize`](https://github.com/python-humanize/humanize) — where it found and
> kept a genuine 5.81% win across its 15 benchmarks (hoisting a per-call
> `import math` out of nine function bodies; `clamp` −23.2%, `apnumber` −14.9%,
> no regressions). That is one library, not the three the Go original was
> exercised against, so treat this as a tool that inherited a validated design
> and has begun earning its own record rather than one that already has it.
> no regressions). That is one library, so treat this as a tool that has begun
> earning its record rather than one that already has it.
>
> Every number in this README is a real measurement taken on the machine that
> wrote it, never an illustration. Where a number would have been guessed, there
> is no number.
>
> The decision procedure — the scoring rules, the Bonferroni correction, the
> asymmetric regression guard, the four exit codes — is carried over unchanged
> from the Go original. If you run this against your own project, the harness's
> asymmetric regression guard, the four exit codes — is fixed, not tuned per
> project. If you run this against your own project, the harness's
> `results.tsv` and `report` output are the honest record of what it actually did
> there; that is rather the point of the whole design.

Expand Down Expand Up @@ -391,10 +388,9 @@ abandoned. Plain `stop` is unaffected and behaves identically everywhere.
you about less. One residual race is real: a grandchild started in the
microseconds between spawning a benchmark subprocess and putting it in its job
object is outside that job and would survive a timeout kill. Windows offers no
way to create a process directly into a job through `subprocess`, and the Go
original has the same window.
way to create a process directly into a job through `subprocess`.

Ported from the Go original:
Accepted limits of the design:

- No attempt to make the harness tamper-proof against a same-user attacker.
The worktree-integrity check catches accidental clobbering and careless
Expand Down
2 changes: 1 addition & 1 deletion src/autor3search_python/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -174,7 +174,7 @@ def validate(cfg: Config) -> None:
raise ConfigError(f"gc must be one of {list(ALLOWED_GC)}, got {cfg.gc!r}")
if _ITERATION_COUNT_FORM.match(str(cfg.benchtime)):
raise ConfigError(
f"benchtime {cfg.benchtime!r} uses go test's fixed-iteration-count form (Nx), "
f"benchtime {cfg.benchtime!r} uses the fixed-iteration-count form (Nx), "
f"which is deliberately unsupported: a fixed count makes rounds incomparable, "
f"because a candidate that is twice as fast finishes in half the wall time and "
f"is therefore measured under different thermal conditions — exactly what the "
Expand Down
2 changes: 1 addition & 1 deletion src/autor3search_python/runner.py
Original file line number Diff line number Diff line change
Expand Up @@ -309,7 +309,7 @@ def python_run(self, *args: str) -> Result:
return self.run(self.python, *args)

def compile_gate(self, paths: Sequence[str]) -> Result:
"""The syntax gate. `go build`'s role: is this even valid code?"""
"""The syntax gate: is this even valid code, before anything runs it?"""
return self.python_run("-m", "compileall", "-q", "-x", _compile_exclude(), *paths)

def import_gate(self, modules: Sequence[str]) -> Result:
Expand Down
8 changes: 4 additions & 4 deletions src/autor3search_python/stats.py
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
"""The statistics behind a verdict.

The Go original gets these from golang.org/x/perf/benchmath. Implementing them
here keeps the harness free of a scipy dependency, and — more usefully — keeps
the exact small-sample behavior under this project's own tests, since small
samples are the entire operating regime (`count` defaults to 10 per side).
These are implemented here rather than taken from scipy: it keeps the harness
dependency-free, and — more usefully — keeps the exact small-sample behavior
under this project's own tests, since small samples are the entire operating
regime (`count` defaults to 10 per side).
"""

from __future__ import annotations
Expand Down
2 changes: 1 addition & 1 deletion tests/test_config.py
Original file line number Diff line number Diff line change
Expand Up @@ -130,7 +130,7 @@ def test_parse_duration_rejects(text):


def test_benchtime_rejects_the_iteration_count_form(tmp_path):
"""go test's `100x` form is a real flag value someone may copy across; say why not."""
"""`100x` is a plausible thing to type for a benchtime; say why it is refused."""
with pytest.raises(config.ConfigError, match="fixed-iteration"):
config.load(write(tmp_path, 'benchtime = "100x"\n'))

Expand Down
7 changes: 3 additions & 4 deletions tests/test_e2e.py
Original file line number Diff line number Diff line change
Expand Up @@ -72,10 +72,9 @@ def test_full_run(demo_repo):
# times the largest no-op delta observed, and well clear of the KEEP step
# below, which turns a quadratic per-character concatenation into a single
# join and measures -25.8% (score 0.7418) on the machine this was written
# on. The Go original raises the same floor for the same reason; it uses
# 15%, against a fixture whose true effect is an order of magnitude rather
# than this one's quarter, so the ratio here is the thing being matched,
# not the number.
# on. What matters is that ratio — the floor sitting well inside the gap
# between the two effects — not the absolute number, so a fixture with a
# different true effect would want a different floor.
raised = cfg_path.read_text().replace("min_effect_pct = 1.0", "min_effect_pct = 10.0")
assert "min_effect_pct = 10.0" in raised, "init's config no longer has the key this rewrites"
cfg_path.write_text(raised)
Expand Down
4 changes: 2 additions & 2 deletions tests/test_runner.py
Original file line number Diff line number Diff line change
Expand Up @@ -43,8 +43,8 @@ def test_timeout_is_reported_not_raised(tmp_path):


def test_timeout_kills_grandchildren(tmp_path):
"""go test runs its benchmark as a grandchild; pytest does too. A survivor
burns CPU and corrupts every later measurement on the machine."""
"""pytest runs its benchmark as a grandchild. A survivor burns CPU and
corrupts every later measurement on the machine."""
marker = tmp_path / "alive.txt"
# marker.as_posix(): a Windows path interpolated raw would put "\\U" of
# C:\\Users into the inner script's own non-raw string literal, and the
Expand Down
52 changes: 39 additions & 13 deletions tests/test_templates.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,26 @@
import pathlib
import re

from autor3search_python import pipeline, templates, verdict

# Names that do not exist in this project: files it never writes, a config
# format it does not read, metrics it does not report. Naming one in the
# agent-facing template or the README sends the reader after something that
# is not there.
ABSENT_NAMES = (
"autor3search-go",
"go.mod",
"go.sum",
"_test.go",
"config.yaml",
"ns/op",
"allocs_delta",
"go test",
)

# Prose that would explain this project as a port rather than on its own terms.
PORT_FRAMING = ("Go original", "Go sibling", "Go harness", "Python port")


def test_program_md_exit_code_table_agrees_with_verdict():
"""A stale table would send the agent branching on the wrong exit code."""
Expand Down Expand Up @@ -32,20 +51,27 @@ def test_program_md_documents_every_exit_code():
assert token in text


def test_program_md_names_no_go_artifacts():
"""A stale Go reference would send the agent looking for a file that is not there."""
def test_program_md_names_nothing_absent():
"""A name for a file that is not there sends the agent looking for it."""
text = templates.program_md()
for stale in (
"autor3search-go",
"go.mod",
"go.sum",
"_test.go",
"config.yaml",
"ns/op",
"allocs_delta",
"go test",
):
assert stale not in text
for absent in ABSENT_NAMES:
assert absent not in text


def test_readme_names_nothing_absent():
"""The README stands on its own; it does not explain this project as a port."""
text = (pathlib.Path(__file__).parent.parent / "README.md").read_text(encoding="utf-8")
for absent in ABSENT_NAMES + PORT_FRAMING:
assert absent not in text


def test_shipped_source_names_nothing_absent():
"""Error messages and docstrings reach users too, so they get the same guard."""
src = pathlib.Path(__file__).parent.parent / "src"
for path in sorted(src.rglob("*.py")):
text = path.read_text(encoding="utf-8")
for absent in ABSENT_NAMES + PORT_FRAMING:
assert absent not in text, f"{path.relative_to(src)} names {absent!r}"


def test_program_md_forbids_editing_conftest():
Expand Down
Loading