Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ jobs:
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, macos-latest]
os: [ubuntu-latest, macos-latest, windows-latest]
python: ["3.11", "3.12", "3.13"]
runs-on: ${{ matrix.os }}
steps:
Expand Down
114 changes: 76 additions & 38 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,39 +39,72 @@ timings out of [pytest-benchmark](https://pytest-benchmark.readthedocs.io/).

## Start here

> **POSIX only (Linux, macOS). Windows is not supported and not tested.**
> `autor3search-python` has never been run on Windows, and `eval` refuses to
> start there unless you explicitly override it. Three guarantees depend on
> POSIX process groups and are simply absent there:
>
> - the concurrency guard that stops two evals from running against the same
> pinned baseline at once;
> - `stop --force`'s ability to signal a running eval at all;
> - killing a timed-out benchmark's whole process tree, rather than leaking
> grandchildren that keep burning CPU.
>
> `doctor` reports all three by name if you run it anyway. See
> [Limitations](#limitations) for the override.
Open your coding agent inside the Python repository you want to make faster,
and paste this:

```text
Install and run autor3search-python on this repository, then optimize it.

Setup:
1. uv tool install autor3search-python (or: pipx install autor3search-python)
2. autor3search-python init
Show me the benchmarks it discovered. If it reports none, STOP and tell me:
this tool can only optimize what it can measure.
3. git add -A && git commit -m "autor3search-python init"
4. autor3search-python doctor
Show me any warnings. If the machine looks unfit to measure, stop and ask me
before continuing.
5. autor3search-python baseline -tag <today, e.g. sep7>
This copies the repository into a pinned worktree and freezes what the
verdict depends on, so it takes a moment.

Then:
6. Read program.md in this repository, in full. It is your instruction set for
the rest of this run. Follow it exactly.

Rules for the whole run:
- One hypothesis per commit. Commit before each experiment, then run
`autor3search-python eval --json` and apply its verdict before touching
anything else: KEEP means the commit stays; anything else (DISCARD, FAIL,
CRASH) means `git reset --hard HEAD~1`.
- Never edit program.md, .autor3search/config.toml, results.tsv, any test or
benchmark file, conftest.py, pyproject.toml, or a lockfile. They are not
yours.
- Never pass -force to any autor3search-python command. (I may run
`autor3search-python stop -force` myself; that one is mine, not yours.)
- Print one context line before each experiment, so I can see where you are:
[exp <n> | <branch> | vs <measure_commit> | stop: autor3search-python stop]

Run the loop until I stop you. I stop you by running
`autor3search-python stop` in my own terminal — you will see it as
"stop_requested": true in a verdict. When you do: apply that verdict, do not
start another experiment, run `autor3search-python report`, summarize what you
tried, and exit the loop.
```

If you are a coding agent that has just been pointed at this README, this is
everything you need:
That's the whole handoff. The agent installs the tool, discovers your
benchmarks, freezes a baseline, and then follows `program.md` — generated for
your repository by `init` — which tells it how to run the keep-or-discard loop.
`program.md` names the benchmarks in scope, spells out the KEEP/DISCARD/FAIL/
CRASH contract, lists everything the agent must never touch, and ends with a
bank of generic Python performance ideas for when the agent is out of
hypotheses.

```bash
# 1. Install the harness.
uv tool install autor3search-python # or: pipx install autor3search-python
What you get back: one commit per accepted change on a branch named
`autor3search-python/<tag>`, and a `results.tsv` recording every experiment
that was tried, including the ones that failed. `autor3search-python report`
summarizes it.

# 2. From the repository you want to optimize:
autor3search-python init # discovers benchmarks, writes config + program.md
git add -A && git commit -m "autor3search-python init"
autor3search-python doctor # is this machine fit to measure? (informational)
autor3search-python baseline -tag <a-short-tag>
Two things worth knowing before you start it:

# 3. Read program.md. It is the complete instruction set — the loop, the exit
# codes, the JSON contract, what you may and may not edit. Follow it exactly.
```
- **It needs benchmarks.** This optimizes what it can measure, and refuses to
guess: `init` looks for pytest-benchmark benchmarks and tells you plainly
when it finds none.
- **Numbers are only as good as the machine.** Run `doctor` and read it. A
thermally throttled laptop on battery produces noise dressed as data.

Everything past this point is for the human setting the run up, or for
understanding what the agent in step 3 is actually bound by.
understanding what the agent in step 6 is actually bound by.

## The idea

Expand Down Expand Up @@ -344,17 +377,22 @@ them should not be over-read:

## Limitations

**POSIX only (Linux, macOS); Windows is not supported and not tested.** See
the note under [Start here](#start-here) for what specifically breaks —
`doctor` names the same three gaps as a FAIL when run on a non-POSIX
platform. `eval` refuses to start there at all, because the whole point of
this tool is a number you can stand behind and it cannot produce a
trustworthy one on a platform none of its safety mechanisms have ever run
on. Set `AUTOR3SEARCH_PYTHON_ALLOW_UNSUPPORTED_PLATFORM=1` to run it anyway,
having read the three gaps above. `stop --force` also refuses to signal a
running eval on such a platform (it would otherwise crash trying), and
prints instead: the graceful stop request is still written, and you should
interrupt the agent yourself.
**Windows works, with one difference worth knowing.** CI runs the full suite on
`windows-latest` alongside Linux and macOS. The run claim is a real lock there
(`msvcrt`, not the unconditional success it used to report), and a job object
gives `eval` the killable process tree a process group gives it elsewhere — so
a benchmark that hits its timeout takes its pytest subprocess and every
grandchild with it, and `stop -force` reaches the benchmark rather than
orphaning it. The difference: `stop -force` there is **immediate rather than a
request**. Windows offers no signal a benchmark can act on mid-round, so the
eval is ended rather than asked, and it does not get to record what it
abandoned. Plain `stop` is unaffected and behaves identically everywhere.
`doctor` also has no load average or CPU governor to read there, so it warns
you about less. One residual race is real: a grandchild started in the
microseconds between spawning a benchmark subprocess and putting it in its job
object is outside that job and would survive a timeout kill. Windows offers no
way to create a process directly into a job through `subprocess`, and the Go
original has the same window.

Ported from the Go original:

Expand Down
1 change: 1 addition & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ classifiers = [
"Operating System :: POSIX",
"Operating System :: POSIX :: Linux",
"Operating System :: MacOS :: MacOS X",
"Operating System :: Microsoft :: Windows",
]

[project.scripts]
Expand Down
27 changes: 0 additions & 27 deletions src/autor3search_python/cli/eval.py
Original file line number Diff line number Diff line change
Expand Up @@ -35,19 +35,6 @@
from autor3search_python.cli import runctx
from autor3search_python.cli.main import EXIT_USAGE

# eval's whole product is a number the human can stand behind. On a
# non-POSIX platform it cannot deliver one: claim_eval's concurrency guard
# always reports success (two evals can run against the same pinned
# baseline at once), `stop --force` cannot signal a running eval to abandon
# its experiment, and a timed-out benchmark leaks grandchild processes that
# keep burning CPU and corrupting every later measurement. A silent wrong
# number is worse than no number, so eval refuses outright rather than
# producing one doctor already warned about. Set this to run anyway.
ALLOW_UNSUPPORTED_PLATFORM_ENV = "AUTOR3SEARCH_PYTHON_ALLOW_UNSUPPORTED_PLATFORM"

# Computed once, same as runstop._POSIX and runner._POSIX.
_POSIX = os.name == "posix"


def best_bench_delta(deltas: Sequence[benchio.Delta]) -> float:
"""The largest single-benchmark improvement, percent. 0.0 for no deltas."""
Expand Down Expand Up @@ -138,20 +125,6 @@ def run(args: list[str]) -> int:
)
opts = parser.parse_args(args)

if not _POSIX and not os.environ.get(ALLOW_UNSUPPORTED_PLATFORM_ENV):
print(
"autor3search-python eval: refusing to run on a non-POSIX platform. This "
"harness has never been run or tested here, and cannot deliver a trustworthy "
"verdict: the concurrency guard cannot detect a second eval already running "
"against the same pinned baseline, `stop --force` cannot signal this eval to "
"abandon its experiment, and a timed-out benchmark would leak grandchild "
"processes that keep burning CPU and corrupt every later measurement. Run "
"`autor3search-python doctor` for details. Set "
f"{ALLOW_UNSUPPORTED_PLATFORM_ENV}=1 to run anyway.",
file=sys.stderr,
)
return EXIT_USAGE

try:
ctx = runctx.resolve(opts.directory, opts.tag)
except runctx.ContextError as e:
Expand Down
60 changes: 38 additions & 22 deletions src/autor3search_python/cli/stop.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,11 @@
additionally signals the running eval to abandon the experiment — that
experiment is lost, because nothing was measured, and no results.tsv row is
written for it. Ctrl+C on the agent is equivalent to -force.

Off POSIX, -force is immediate rather than a request: there is no signal an
eval can act on mid-benchmark, so the process is ended instead of asked, and it
does not get to report what it abandoned. Plain `stop` behaves identically
everywhere.
"""

from __future__ import annotations
Expand All @@ -20,8 +25,9 @@
from autor3search_python.cli import runctx
from autor3search_python.cli.main import EXIT_OK, EXIT_USAGE

# Computed once, same as runstop._POSIX: `_signal_group` below calls
# os.killpg, which does not exist off this platform.
# Computed once, same as runstop._POSIX: it selects between the two ways
# -force can reach a running eval — `_signal_group` calls os.killpg, which
# does not exist off this platform, and `_terminate` ends the process instead.
_POSIX = os.name == "posix"


Expand All @@ -42,6 +48,21 @@ def _signal_group(pid: int) -> None:
os.killpg(group_signal_target(pid), signal.SIGINT)


def _terminate(pid: int) -> None:
"""End the eval outright, for a platform with no signal it can act on.

os.kill is TerminateProcess on Windows: there is no SIGINT a benchmark
loop could notice mid-round, so the process is stopped rather than asked.
It takes the pid itself, never the negated one a process group needs.
Everything the eval was measuring dies with it — the job objects holding
each benchmark's process tree are kill-on-close, and the last handle to
them goes when the eval does.
"""
if pid <= 1:
raise ValueError(f"pid {pid} is not a process this command will signal")
os.kill(pid, signal.SIGTERM)


def run(args: list[str]) -> int:
parser = argparse.ArgumentParser(prog="autor3search-python stop")
parser.add_argument("-C", dest="directory", default=".", help="repository root")
Expand Down Expand Up @@ -98,28 +119,23 @@ def run(args: list[str]) -> int:
runstop.clear_eval_pid(ctx.state_dir)
else:
if running:
if not _POSIX:
# `_signal_group` calls os.killpg, which does not exist on this
# platform (AttributeError, not a clean refusal) — so this must
# be caught before it is ever called, not after it raises.
print(
f"autor3search-python stop: cannot signal the running eval (pid {pid}) "
"on this platform — stop --force relies on POSIX process groups, which "
"do not exist here.",
file=sys.stderr,
)
print(
"The graceful stop request has still been written and will be read at "
"the next verdict. To abandon the current experiment now, interrupt the "
"running agent yourself (e.g. Ctrl+C in its terminal).",
file=sys.stderr,
)
return EXIT_USAGE
try:
_signal_group(pid)
print(f"signalled the running eval (pid {pid}) to abandon its experiment.")
if _POSIX:
_signal_group(pid)
print(f"signalled the running eval (pid {pid}) to abandon its experiment.")
else:
# `_signal_group` would call os.killpg, which does not exist
# here at all (an AttributeError traceback, not a refusal),
# so the branch is taken before it is ever reached.
_terminate(pid)
print(f"terminated the running eval (pid {pid}) and everything it started.")
print(
"On this platform -force is immediate rather than a request: there is "
"no signal a benchmark can act on mid-round, so the eval was ended "
"rather than asked, and did not get to report what it abandoned."
)
except (ValueError, OSError) as e:
print(f"autor3search-python stop: could not signal pid {pid}: {e}", file=sys.stderr)
print(f"autor3search-python stop: could not stop pid {pid}: {e}", file=sys.stderr)
return EXIT_USAGE
else:
print("no eval is running; the stop request is written and will be read next time.")
Expand Down
24 changes: 12 additions & 12 deletions src/autor3search_python/doctor.py
Original file line number Diff line number Diff line change
Expand Up @@ -36,9 +36,9 @@

_MISSING_MODULE_RE = re.compile(r"No module named ['\"]([\w.]+)['\"]")

# Computed once, same as runstop._POSIX and runner._POSIX: this is the one
# thing that determines whether the concurrency guard, `stop --force`, and
# the timeout's process-group kill actually work.
# Computed once, same as runstop._POSIX and runner._POSIX: it selects which
# mechanism carries the concurrency guard, `stop --force`, and the timeout's
# tree kill, and it is the one thing that changes what `stop --force` means.
_POSIX = os.name == "posix"


Expand Down Expand Up @@ -119,15 +119,15 @@ def check_platform() -> Finding:
if not _POSIX:
return Finding(
"platform",
"this platform is not POSIX; this harness has never been run or tested here, "
"and three of its guarantees are silently absent rather than merely degraded. "
"(1) The concurrency guard in claim_eval always reports success, so two evals "
"can run against the same pinned baseline worktree at once. (2) `stop --force` "
"cannot signal the running eval's process group here, so it cannot stop one. "
"(3) A benchmark that times out has only its direct child killed, not its "
"process group, so its grandchildren keep running and burn CPU, corrupting "
"every later measurement. Do not trust a number produced here.",
Severity.FAIL,
"this platform is not POSIX. The run claim is a real lock and a job object "
"gives a timed-out benchmark the killable process tree a process group gives "
"it elsewhere, so the guarantees hold; two differences are worth knowing. "
"`stop --force` is immediate here rather than a request — there is no signal "
"an eval can act on mid-benchmark, so it is ended rather than asked and does "
"not report what it abandoned (plain `stop` is unaffected). And this check "
"knows less about the machine here: no load average, no CPU governor, so it "
"warns you about less than it would on Linux.",
Severity.WARN,
)
if sys.platform == "darwin":
return Finding(
Expand Down
15 changes: 14 additions & 1 deletion src/autor3search_python/gitx.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,13 +19,20 @@ def _git(d: str | Path, *args: str) -> str:
otherwise successful command — lfs filter warnings, advice hints, a user's
own hooks — never gets parsed as part of the result. stderr appears in the
error message only when the command fails.

The encoding is named rather than left to the locale, which is what
`text=True` alone would use: git speaks UTF-8 on every platform, but a
default Windows install decodes with cp1252, and a repository holding a
`café.py` then reported a `caf?.py` that matches nothing on disk to every
gate that reads a diff.
"""
try:
proc = subprocess.run(
["git", *args],
cwd=str(d),
capture_output=True,
text=True,
encoding="utf-8",
timeout=_TIMEOUT,
)
except OSError as e:
Expand All @@ -38,7 +45,13 @@ def _git(d: str | Path, *args: str) -> str:


def root(d: str | Path) -> str:
return _git(d, "rev-parse", "--show-toplevel")
"""The repository root, in this platform's own path form.

git answers with POSIX separators everywhere, Windows included. Callers
that wrap this in Path() do not care, but `doctor` prints it and the run's
state directory is keyed on the repository path — one form, not two.
"""
return str(Path(_git(d, "rev-parse", "--show-toplevel")))


def head_commit(d: str | Path) -> str:
Expand Down
6 changes: 5 additions & 1 deletion src/autor3search_python/profile.py
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,11 @@ def _display_site(file: str, line: int, root: Path | None) -> str:
display = file
if root is not None:
try:
display = str(Path(file).resolve().relative_to(root))
# One separator in the report, on every platform: `_elide` cuts on
# "/" when a site is too long, and a Windows-native "pkg\mod.py"
# has no such boundary to cut on — it would be truncated
# mid-component instead of at a directory.
display = Path(file).resolve().relative_to(root).as_posix()
except ValueError:
display = file
return _elide(f"{display}:{line}", _SITE_WIDTH)
Expand Down
Loading
Loading