fix(core): keep a refused inotify instance from wedging the process - #48366
Open
Nowaker wants to merge 1 commit into
Open
fix(core): keep a refused inotify instance from wedging the process#48366Nowaker wants to merge 1 commit into
Nowaker wants to merge 1 commit into
Conversation
@parcel/watcher builds its shared inotify backend inside the synchronous constructor of the N-API SubscribeRunner, which runs on whichever thread called subscribe() - for opencode, the JS main thread. When inotify_init1 fails there, InotifyBackend::start() throws before notifyStarted(), and Backend::handleError() only notifies watchers that are already registered. The watcher for this call is not registered until execute() runs, so nothing ever notifies mStartedSignal and Backend::run() waits on it forever. That parks the event loop, not just one fiber. It is why the existing Effect.timeout(SUBSCRIBE_TIMEOUT_MS) and Effect.catchCause around the subscription never fired, why nothing reached the log, and why the turn could not be interrupted: no timer, fiber or signal handler runs again once the main thread stops returning to the loop. Reaching that state needs no bug of ours. fs.inotify.max_user_instances is a per-uid ceiling shared with every other process the user runs, it defaults to 1024, and it cannot be raised without root. Where this was found, 954 instances were held by an unrelated herd of orphaned kbuildsycoca6 processes; every opencode turn afterwards hung on an assistant row with no parts, no error and no completion time. So ask the kernel for one instance first, give it straight back, and call subscribe() only if that succeeded. The probe and the call sit in one synchronous thunk so nothing runs between them, which keeps the window where another process could take the last instance as narrow as it can be made from this side. It cannot be closed entirely: only fixing @parcel/watcher, or owning the watcher in a killable subprocess, removes the race outright. What it does remove is the permanent silent wedge - a refused instance now logs a warning naming the errno, and the turn carries on without file watching. EMFILE does not identify which ceiling was hit, the per-process descriptor limit or the per-uid inotify one, so the raw errno is logged without interpreting it. The probe needs bun:ffi, so it loads only on the inotify branch and only when that import succeeds. A runtime without it keeps the existing behaviour rather than silently losing file watching. AI-Tool: opencode AI-Model: anthropic/claude-opus-5 AI-Platform: darwin AI-Harness: Vibeterm 123ec5e1-dirty AI-Session-ID: ses_f76ce0cf6ffejNGTCJyk7ooYc5
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue for this PR
Closes #16610
Same deadlock as the still-open #37111, but this only fixes one of its two triggers - see the end of the description.
Type of change
What does this PR do?
You open opencode in a git repo, type a prompt, press Enter - and nothing happens, forever. No error, no toast, nothing in the log, and Esc-Esc won't cancel it. Menus and the model picker still respond, so it doesn't look broken, it looks like the model is thinking. Restarting doesn't help and a fresh session does the same. #16610 hit this a few releases back as a blank screen at startup; today the UI paints fine and the prompt simply never starts a turn.
The trigger is the machine being out of inotify instances.
fs.inotify.max_user_instancesis a per-uid ceiling, 1024 by default, shared with every other program you run, and raising it needs root. On the machine I found this on, 954 of the 1024 were held by a herd of orphaned KDEkbuildsycoca6processes - nothing to do with opencode, but opencode is what broke.Under that pressure
inotify_init1fails inside@parcel/watcher'ssubscribe(), which builds its shared backend in a synchronous N-API constructor on the calling thread - for us the JS main thread.InotifyBackend::start()throws beforenotifyStarted(), andBackend::handleError()only notifies watchers that are already registered, soBackend::run()waits onmStartedSignaland never returns. That parks the event loop rather than one fiber, which is why theEffect.timeout(SUBSCRIBE_TIMEOUT_MS)andEffect.catchCausealready wrapped around the subscription never fired and nothing reached the log: once the main thread stops returning to the loop, no timer, fiber or signal handler runs again. Only directories under version control are affected, because that is the only branch that reachessubscribe().The fix is to ask the kernel for one inotify instance, hand it straight back, and call
subscribe()only if that worked. The probe and the call sit in one synchronous thunk with nothing in between, so the window where another process takes the last instance is one expression wide. That window is not closed and I'm not claiming otherwise - only fixing@parcel/watcher, or owning the watcher in a killable subprocess, removes the race. What goes away is the permanent silent wedge: a refusal now logswatcher unavailable, continuing without itwith the errno, and the session continues without file watching. EMFILE doesn't say whether it was the per-process descriptor limit or the per-uid inotify ceiling, so the errno is logged as-is rather than interpreted. The probe needsbun:ffi, so it is imported only on the inotify branch and only if that import succeeds; a runtime without it keeps the current behaviour instead of silently losing file watching.What this does not fix: #37111's other trigger, several concurrent subscribes deadlocking while instances are still available. The probe succeeds there and the deadlock still happens.
How did you verify your code works?
Three levels, each under real inotify exhaustion on Linux against the real
@parcel/watcher-linux-x64-glibc: the native boundary, a whole turn from source, and a whole turn in a compiled binary.1. Native boundary. Holding inotify instances open until
inotify_init1returned EMFILE:Step 3 is the bug in two lines: the statement after
subscribe()never executes.2. Whole turn, from source. Throwaway Ubuntu 24.04 VM (glibc 2.39,
max_user_instances128, 124 held until EMFILE).bun run ./src/index.ts run --pure 'say ok'in a fresh git repo, against a local OpenAI-compatible mock so a turn can complete without credentials. Same host, same pressure, same command; the only variable is this patch. Two runs each way, both reproduced:WARN watcher unavailable, continuing without it ... backend=inotify errno=24 code=EMFILEdevunpatchedINFO watcher backend, then silence3. Whole turn, compiled. Running from source does not cover
bun build --compile, and a failure there would be invisible: if the probe'sbun:ffiimport could not load, that failure is swallowed by design, leaving no probe and no message. Throwaway Arch VM (kernel 7.2.3, glibc 2.44,max_user_instances1024, 1020 held until EMFILE). Arch packages opencode, soopencode-binsupplies a binary built by the distro package maintainer, andscript/build.ts --singlesupplies patched and unpatched binaries built identically to each other:opencode-bin1.18.30INFO watcher backend, then silenceWARN watcher unavailable ... errno=24 code=EMFILEINFO watcher backend, then silenceUnpressured, all three complete the turn normally. So the defect is present in a released binary, the fix works inside
bun build --compileoutput, and the difference is this patch rather than the build method - the last two binaries came from the same command on the same host minutes apart.In every wedged run above, the last log line is
project copy refresh started, which is also the last line produced on the machine where this was originally found - so these reproduce the original failure rather than merely resembling it.Reproducing it yourself needs no root, because the ceiling is per-uid: open inotify instances as your own user until
inotify_init1returns EMFILE, then run opencode in any git directory. Before this patch the turn never starts; after it you get the warning and a working session with no file watching.Tests and gates.
packages/core/test/filesystem/watcher-preflight.test.tsis new, and fails with the preflight removed because the watcher then subscribes to both the root and.gitand logs no warning.bun test test/filesysteminpackages/coreis 38 pass on macOS; on Linuxwatcher.test.tsplus the newwatcher-preflight.test.tsare 8 pass, which is the first time the six pre-existing watcher tests have run against the real inotify backend rather than macOS fs-events. The repo's pre-pushturbo typecheckis 30/30.Screenshots / recordings
Not a UI change.
Checklist