Memory leak due to process.fork()?

[Reggino]

• Version: v8.5.0
• Platform: Linux

I am experiencing memory leakage in an app that usesprocess.fork() a lot. These child processes get sent messages via process.send() with a sendHandle and are terminated later on.

I did run into issues with memory management here. Some heap dumps show that even after the child-processes exited, the ChildProcess-instances are retained in the master process. I learned that using subprocess.disconnect() partly fixes that issue, but one more retainer can be found here:

node/lib/net.js

Line 1665 in 20259f9

Server.prototype._setupWorker = function(socketList) {

How, where and when should this socketList be removed from the _workers-array?

[cjihrig]

Do you have code that reproduces the memory leak?

[Reggino]

I tried to make a simplified setup of my app:

server.js

const http = require('http');
const { fork } = require('child_process');

const server = http.createServer((req, res) => {
    const workerProcess = fork('worker.js', [], {execArgv: []});
    workerProcess.on('message', (status) => {
        switch (status) {
            case 'ready':
                workerProcess.send({
                    method: req.method,
                    headers: req.headers,
                    path: req.path,
                    httpVersionMajor: req.httpVersionMajor,
                    query: req.query,
                }, res.socket);
                break;

            case 'done':
                res.socket.end();
        }
    });
});

server.listen(8080);

setInterval(() => {
  console.log(process.memoryUsage());
}, 10000);

worker.js

const http = require('http');

process.on('message', (req, socket) => {
    const res = new http.ServerResponse(req);
    res._finish = function _finish() {
        res.emit('prefinish');
        socket.end();
    };
    res.assignSocket(socket);
    socket.once('finish', () => {
        process.send('done', () => {
            process.exit(0);
        });
    });
    res.end(process.pid.toString());
});

process.send('ready');

When simulating load on this server using ApacheBench, memory usage of the server-process slowly rises.

ab -n 1000000 -c 50  http://localhost:8080/

You'll have to be patient: it does take a few hours on my machine to become critical...

[Reggino]

Node 8.6.0 has the same issue.

[Reggino]

When limiting the available memory for the server to let's say 100MB, it dies within a few minutes at a certain point:

node --max_old_space_size=100 server.js 

...

--- Last few GCs --->

[15664:0x39d9410]   504893 ms: Mark-sweep 252.2 (299.0) -> 252.2 (268.0) MB, 219.9 / 0.1 ms  (+ 0.0 ms in 0 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 220 ms) last resort 
[15664:0x39d9410]   505119 ms: Mark-sweep 252.2 (268.0) -> 252.2 (268.0) MB, 226.7 / 0.1 ms  last resort 


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x3b40f62a8799 <JSObject>
    2: text(aka utf8Text) [string_decoder.js:201] [bytecode=0x4695cc3cbd1 offset=29](this=0x54b29377d39 <StringDecoder map = 0x16700ae4d0a9>,buf=0x17c2ab58e4b1 <Uint8Array map = 0x139076a04e01>,i=0)
    4: write [string_decoder.js:85] [bytecode=0x4695cc3c8c1 offset=97](this=0x54b29377d39 <StringDecoder map = 0x16700ae4d0a9>,buf=0x17c2ab58e4b1 <Uint8Array map = 0x139076a04e01>)
    6: onread [int...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [node]
 2: 0x13576dc [node]
 3: v8::Utils::ReportOOMFailure(char const*, bool) [node]
 4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [node]
 5: v8::internal::Factory::NewStruct(v8::internal::InstanceType) [node]
 6: v8::internal::Factory::NewTuple3(v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>) [node]
 7: v8::internal::LoadIC::LoadFromPrototype(v8::internal::Handle<v8::internal::Map>, v8::internal::Handle<v8::internal::JSObject>, v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::Smi>) [node]
 8: v8::internal::LoadIC::GetMapIndependentHandler(v8::internal::LookupIterator*) [node]
 9: v8::internal::IC::ComputeHandler(v8::internal::LookupIterator*) [node]
10: v8::internal::LoadIC::UpdateCaches(v8::internal::LookupIterator*) [node]
11: v8::internal::LoadIC::Load(v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Name>) [node]
12: v8::internal::Runtime_LoadIC_Miss(int, v8::internal::Object**, v8::internal::Isolate*) [node]
13: 0x1745398046fd
Aborted (core dumped)

[bnoordhuis]

@Reggino Can you check that #15679 fixes the issue for you?

[Reggino]

@bnoordhuis Thanks! While i do think your change is part of the solution , this issue still isn't completely resolved: even now the server-process keeps leaking memory up untill it crashes...

[bnoordhuis]

Do you mean with the example you posted or with a larger application?

[Reggino]

The example...

[Reggino]

The issue still exists...