Repository navigation
Recursive fs.watch on a Windows directory junction produces inconsistent events #53903
Description
Version
v20.5.1
Platform
Microsoft Windows NT 10.0.22631.0 x64
Subsystem
fs
What steps will reproduce the bug?
Run the following script from within an empty directory:
const fs = require("fs");
const path = require("path");
const delay = (ms) => new Promise(resolve => setTimeout(resolve, ms));
async function main() {
console.log("Setting up real/link folders...");
try { fs.mkdirSync(path.join(__dirname, "real/sub/folder"), { recursive: true }); } catch { }
try { fs.unlinkSync(path.join(__dirname, "real/sub/folder/test.txt")); } catch { }
try { fs.symlinkSync(path.join(__dirname, "real"), path.join(__dirname, "link"), "junction"); } catch { }
await delay(1000);
console.log("Creating recursive watchers...");
let last = 0;
const watcher1 = fs.watch(path.join(__dirname, "real"), { persistent: true, recursive: true }, (e, f) => {
const now = Date.now();
const delta = now - last;
console.log(` [${`+${delta.toString()}`.padStart(6, " ")}ms] real:`, f, e);
});
const watcher2 = fs.watch(path.join(__dirname, "link"), { persistent: true, recursive: true }, (e, f) => {
const now = Date.now();
const delta = now - last;
console.log(` [${`+${delta.toString()}`.padStart(6, " ")}ms] link:`, f, e);
});
await delay(1000);
console.log("Creating 'real/sub/folder/test.txt'...");
last = Date.now();
fs.writeFileSync(path.join(__dirname, "real/sub/folder/test.txt"), "test", { encoding: "utf8" });
await delay(2000);
console.log("Deleting 'real/sub/folder/test.txt'...");
last = Date.now();
fs.unlinkSync(path.join(__dirname, "real/sub/folder/test.txt"));
await delay(2000);
console.log("Creating 'link/sub/folder/test.txt'...");
last = Date.now();
fs.writeFileSync(path.join(__dirname, "link/sub/folder/test.txt"), "test", { encoding: "utf8" });
await delay(2000);
console.log("Deleting 'link/sub/folder/test.txt'...");
last = Date.now();
fs.unlinkSync(path.join(__dirname, "link/sub/folder/test.txt"));
await delay(5000);
// Uncomment these lines to trigger observation:
//console.log("Observing the link directory...");
//process.cwd(path.join(__dirname, "link/sub/folder/"));
//fs.readdirSync(path.join(__dirname, "link/sub/folder/"));
//last = Date.now();
await delay(10000);
console.log("Closing watchers...");
watcher1.close();
watcher2.close();
await delay(1000);
}
main().catch(e => console.error(e));Note that creating directory junctions on Windows may require either Developer Mode or local admin permissions.
How often does it reproduce? Is there a required condition?
So long as no other process is actively observing link/sub/folder, deleting link/sub/folder/test.txt will not trigger a "change" event for link/sub/folder. However, the event will trigger if any of the following conditions are met:
- Navigating to
link/sub/folderwithin a command shell - Opening
link/sub/folderfrom Windows Explorer - Antivirus or other software actively monitoring
link/sub/folderfor changes - Uncommenting the commented section in the above script
Whenever any process is actively observing the directory link/sub/folder, then the "change" event for link/sub/folder will be raised.
The "change" event will not be raised if only link, link/sub, or real/sub/folder are being actively monitored, only when immediate parent directory of the deleted file is being actively monitored.
It does not matter whether you are recursively monitoring real or link, only that you delete from a path within link.
What is the expected behavior? Why is that the expected behavior?
When a file is deleted within a junction, "change" events for the parent directory are reported consistently.
What do you see instead?
When a file is deleted within a junction, "change" events for the parent directory are only reported if some process is actively observing that parent directory.
Additional information
This does not appear to be specific to NodeJS v20 as it occurs in every version of NodeJS I've tested back at least as far as 10.22.0 (I have not tested anything earlier).
Activity
huseyinacacak-janea commentedon Jul 26, 2024
I've investigated this issue and my findings are below.
Node uses
libuvto handle file operations and it usesReadDirectoryChangesWto watch a given directory recursively. I put debug points and prints around this function to see the events generated by the API. The situation is the same as you described in the issue. When I run Node to watch a directory and delete a file there, it only triggers an event for the file itself, not for the parent folder. However, When I have a file explorer open, it also triggers an event for the folder. It seems that this inconsistency comes directly from the Windows API. From what I've seen, Node doesn't delete or ignore any events.I've created an example to clearly see the behavior of this API function. This example is for watching a directory and only uses the API functions that are being used in Node. If you want, you can check the example here. When I ran this example, the situation was the same as it is in Node.
There is a lot of discussion on the internet about this API. Watching a directory doesn't work properly on other platforms as well. While searching online, I saw a module called
chokidar. It seems to be very popular and useful for this kind of operation. It basically traverses the folder and calls thefs.watchfor all the files and directories to overcome the limitations of the API. You could utilize this module.jakebailey commentedon Jul 27, 2024
We certainly know about chokidar (and all of the other alternatives); TypeScript is dependency free and wraps
fs.watchjust like third party watchers (with similar workarounds to optimize things). One of the things done is to avoid doing that full manual recursive watch. It's not clear that switching to doing it the slow way is better than fixing the platform, hence the issue. Perhaps this is a Windows bug?huseyinacacak-janea commentedon Jul 31, 2024
I'm able to reproduce this issue with the minimal example I've provided above. Therefore, it seems to be a Windows bug. I'll investigate more.
StefanStojanovic commentedon Aug 20, 2024
Hey everyone, sorry for the late response. Anyway, I investigated this more and can confirm that this behavior is due to how Windows works under the hood, more precisely due to the optimizations for writing to disk. Based on a few things I tried out (I tried writing to file from Node.js), I saw that:
fs.writeFileSyncorfs.appendFileSync(or their async versions) everything will work great, as these open the file, write to it, and close the returned file descriptor.const fd = fs.openSync(...)and then callfs.writeSync(fd, ...), the changes will not get emitted to the watcher. If you add a periodicfs.fsyncSync(fd), the changes will be emitted to the watcher every time it is called.The workarounds you suggested also worked with my example (also right-clicking the file in the explorer). All of those force Windows to flush writing to disk, thus events are emitted.
One way to overcome this would be to implement polling which is something I assume chokidar is doing.
StefanStojanovic commentedon Dec 18, 2024
Hey everyone, I've noticed that this issue had no activity in ~4 months. Since this seems like a Windows OS issue, I'll go ahead and close it. Feel free to reopen if needed.