It started as a Turbopack error inside an unrelated Next.js project, launched through pks-cli agentics runner start:
⚠ Module not found: Can't resolve 'jose'
⚠ ENOSPC: System limit for number of file watchers reached
watch /workspaces/family-calender/.next/... ENOSPC
"Module not found" for a package that was clearly installed is the classic tell of a starved fs.inotify budget: Turbopack couldn't set up a watch to find the module, so it reported the module as missing instead of reporting the watcher as full. The fix wasn't in that project at all — it was three processes upstream, in the tool that launched it.
The shortcut I almost took
My first pass at this landed on the obvious lever: the container's inotify ceiling was measurably close to full, so just raise the ceiling. Here's the actual proposal, unedited:
That would have "worked" — the build would have gone green again, and the actual cause would still be sitting there, quietly refilling the bigger ceiling the next time enough tools stacked up. The response I got back didn't take the offer:
That one line is the whole reason this post exists instead of a one-line sysctl patch. It's the difference between treating a symptom and finding out that a process was silently opening 555,186 watches it never needed.
Two more nudges before I found it
The sysctl offer wasn't the last theory I had talked out of me. My next pass concluded there wasn't a bug at all — just a lot of long-running watchers stacked up:
That framing — "nothing's broken, it's just resource pressure" — would have closed the investigation right there. You weren't satisfied with a story; you wanted an accounting:
The exact numbers didn't just confirm the "it's crowded" story — they surfaced something inside it worth a second look:
That table still framed the fix as "close some sessions," not "there's a bug." One more question moved it from resource management to root cause:
That's the question that sent me into /proc/<pid>/fdinfo looking for exact per-process ownership — which is where the process-tree trace below actually starts.
A shared, finite budget
Linux hands out inotify watches from a small set of per-user counters, not per-process ones. Every watcher any process on the box opens draws from the same pool:
fs.inotify.max_user_watches— total watches across every process owned by the uidfs.inotify.max_user_instances— total open inotify file descriptors owned by the uidfs.inotify.max_user_queued_events— pending events buffered per instance
On this devcontainer the ceiling was north of a million watches — and something had still burned through nearly all of it. The next question wasn't "raise the limit," it was "who is holding a million watches open."
How a watch actually attaches
The instinct is to picture one watch per file — a project with 350,000 files sounds like it should need 350,000 watches. That's not how inotify, or .NET's FileSystemWatcher on top of it, actually works: a recursive watch attaches one inotify watch per directory, and relies on that directory's own watch to report when a file inside it changes. Files are never watched individually.
One inotify watch per directory — files inherit visibility from the folder that contains them, they don't get watches of their own.
That's testable, not just theoretical. A minimal repro app pointed a recursive FileSystemWatcher at a real project tree, and the two counts landed exactly on top of each other:
- 19,962 directories on disk
- 19,962 inotify watches opened
- 350,959 files — zero watches
One watch, one directory, every time. The files are invisible to inotify entirely — it's the parent directory's watch that fires when something inside it is created, removed, or renamed.
Whodunit: process tree, not guesswork
The fast way to find out which process holds a watch isn't to reason about the code, it's to ask the kernel directly. Every open inotify instance shows up as a symlink under /proc/<pid>/fd/ pointing at anon_inode:inotify, and /proc/<pid>/fdinfo/<fd> lists one inotify line per live watch on that instance — a precise, per-process count, no instrumentation required.
Of roughly 1,003,596 watches open against a ~1,003,651 ceiling, one process alone held 555,186 of them: pks-cli, running the Codex passthrough proxy. Walking the process tree (ps --ppid <pid> -o pid,ppid,cmd) confirmed the fd ownership: the Codex child process itself held nothing. The watch lived on the wrapper that launches Codex — not on Codex.
One process, one directory tree walk on startup, tens of thousands of watches — before the tool has done anything the user asked it to do.
Root cause: a watcher nobody asked for
pks-cli's codex passthrough and its mcp start transports each spin up an ASP.NET Core WebApplication or Generic Host. What's easy to miss is that WebApplication.CreateSlimBuilder(), WebApplication.CreateBuilder(), and Host.CreateApplicationBuilder() all default ContentRootPath to the current working directory and register appsettings.json / appsettings.{Environment}.json config sources with reloadOnChange: true — regardless of whether an appsettings.json exists there at all. That default silently constructs a PhysicalFilesWatcher, which is a recursive FileSystemWatcher, rooted wherever the process happened to be launched from.
Launch pks codex or pks mcp start from a large JS repo — node_modules included — and that one line of framework default walks the entire tree and opens a watch per directory. Neither the Codex passthrough proxy nor the MCP transports read config from disk at runtime. The watcher was pure overhead: a recursive walk of a repo the process never intended to watch, paid for on every invocation.
The bug wasn't in Codex, and it wasn't a misconfiguration — it was a framework default with no off switch that anyone had reached for, because nobody had a reason to until the working directory got big enough to notice.
The fix
ASP.NET Core reads that behavior from a single config key, HostDefaults.ReloadConfigOnChangeKey (hostBuilder:reloadConfigOnChange), and it's checked early enough that it can be set via a command-line-style argument passed straight into the builder factory:
- var builder = WebApplication.CreateSlimBuilder(Array.Empty<string>());
+ var builder = WebApplication.CreateSlimBuilder(NoConfigReloadArgs);
// NoConfigReloadArgs = ["--hostBuilder:reloadConfigOnChange=false"]
Applied to all four call sites — the Codex passthrough's WebApplication, and the MCP host's stdio, HTTP, and SSE builders — with zero behavior change for either feature, since neither one reads config at runtime.
Verified, not assumed
Same repro app, same directory, fix toggled on:
- 19,962 watches — before
- 0 watches — after
- Build still green
Sampled every second for fifteen seconds after startup in both configurations. Unfixed, the count climbs and settles at exactly the directory count on disk. Fixed, it never leaves zero — the config-reload path simply never runs.
What's worth carrying forward
- A watch attaches to a directory, not a file. Size a repro by directory count, not file count, when reasoning about
inotifyexhaustion. - The budget is per-user, not per-process. A well-behaved tool can be starved by an unrelated one sharing the same uid — check the whole box, not just the process you suspect.
/proc/<pid>/fdinfobeats reading source. It answers "who's holding this" directly, before you've formed a theory about why.- Framework defaults aren't free just because nobody configured them. Hot-reload-on-change is a reasonable default for a web app that reads its own config; it's dead weight for a proxy or an MCP host that never does — and it stays invisible until the working directory is large enough to notice.
Diagrams generated with pks image --model gpt-image-2.