Repository navigation
Segmentation fault in v8::internal::compiler::(anonymous namespace)::MayAlias (10.15.2 from Debian Buster) #31484
Description
Activity
Is there any chance you could share a reproduction? Can you try other Node.js versions and compare the result (including, ideally, verifying that the 10.15.2 binary from https://nodejs.org/en/ or e.g.
nvmalso exhibits the same issue)?Is there any chance you could share a reproduction?
No. As I said it's a fairly complex application, containing a bunch of internal business logic. I started trying to rip out JavaScript code that I assume not to be executed within the staging system to reduce the footprint of the application, but so far without any real success. The biggest issue that I am not yet able to reliably trigger the segmentation fault. With an unmodified application it takes roughly 30 minutes to 1 hour to crash with the current traffic on that system.
I can offer to poke around within the core file. I might be able to provide the core file to a nodejs core developer in private. I would need to check with my manager and to clean up the configuration to not leak any private information, such as TLS certificates, though.
Can you try other Node.js versions and compare the result (including, ideally, verifying that the 10.15.2 binary from https://nodejs.org/en/ or e.g.
nvmalso exhibits the same issue)?Yes, I can try that. I just downloaded
https://nodejs.org/download/release/v10.15.2/node-v10.15.2-linux-x64.tar.gzonto the machine, adjusted the unit file and restarted the service. For my record: As of 23:53:34 UTC the new binary is running with an unmodified application.For my record: As of 23:53:34 UTC the new binary is running with an unmodified application.
That was quick. I started a traffic generator in parallel and the application crashed at 23:59:14 UTC.
Find below the stack trace with the node binary downloaded from nodejs.org:
Core was generated by `/original-node/node-v10.15.2-linux-x64/bin/node dist/server.js'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x0000000000cfe112 in v8::internal::compiler::(anonymous namespace)::MayAlias(v8::internal::compiler::Node*, v8::internal::compiler::Node*) [clone .part.54] () [Current thread is 1 (Thread 0x7f269affd700 (LWP 24775))] (gdb) bt #0 0x0000000000cfe112 in v8::internal::compiler::(anonymous namespace)::MayAlias(v8::internal::compiler::Node*, v8::internal::compiler::Node*) [clone .part.54] () #1 0x0000000000d0188c in v8::internal::compiler::LoadElimination::AbstractField::Kill(v8::internal::compiler::LoadElimination::AliasStateInfo const&, v8::internal::MaybeHandle<v8::internal::Name>, v8::internal::Zone*) const () #2 0x0000000000d03e84 in v8::internal::compiler::LoadElimination::ReduceStoreField(v8::internal::compiler::Node*) () #3 0x0000000000d08ba5 in v8::internal::compiler::LoadElimination::Reduce(v8::internal::compiler::Node*) () #4 0x0000000000d4a9c8 in v8::internal::compiler::(anonymous namespace)::SourcePositionWrapper::Reduce(v8::internal::compiler::Node*) () #5 0x0000000000c7ac5e in v8::internal::compiler::GraphReducer::Reduce(v8::internal::compiler::Node*) () #6 0x0000000000c7b051 in v8::internal::compiler::GraphReducer::ReduceTop() () #7 0x0000000000c7b4e1 in v8::internal::compiler::GraphReducer::ReduceNode(v8::internal::compiler::Node*) () #8 0x0000000000d56bb6 in v8::internal::compiler::LoadEliminationPhase::Run(v8::internal::compiler::PipelineData*, v8::internal::Zone*) () #9 0x0000000000d57b10 in v8::internal::compiler::PipelineImpl::OptimizeGraph(v8::internal::compiler::Linkage*) () #10 0x0000000000d57b50 in v8::internal::compiler::PipelineCompilationJob::ExecuteJobImpl() () #11 0x0000000000c109b1 in v8::internal::OptimizedCompilationJob::ExecuteJob() () #12 0x0000000000c0b3f0 in v8::internal::OptimizingCompileDispatcher::CompileTask::RunInternal() () #13 0x0000000000bc65f6 in v8::internal::CancelableTask::Run() () #14 0x0000000000962bb9 in node::BackgroundRunner(void*) () #15 0x00007f26a1292fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486 #16 0x00007f26a11c14cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95Edit: After saving the core file and restarting the traffic generator the application crashed at 00:03:13 UTC.
Can you open the core dump with gdb and post the output of
info registersanddisassemble? Thanks.Sure. Disassembly cut at some location after the current instruction that still fit on my terminal. I can give more disassembly if it is helpful.
From the Debian binary
(gdb) info registers rax 0x4670001 73859073 rbx 0x0 0 rcx 0x0 0 rdx 0x27 39 rsi 0x0 0 rdi 0x221fc90 35781776 rbp 0x7fe93fffe130 0x7fe93fffe130 rsp 0x7fe93fffe110 0x7fe93fffe110 r8 0x4 4 r9 0x7fe93407a250 140639577023056 r10 0x7fe93407eea8 140639577042600 r11 0x1 1 r12 0x221fc90 35781776 r13 0x1f2e570 32695664 r14 0x213d3f8 34853880 r15 0x7fe934079698 140639577020056 rip 0x7fe9488dcb6b 0x7fe9488dcb6b <v8::internal::compiler::(anonymous namespace)::MayAlias(v8::internal::compiler::Node*, v8::internal::compiler::Node*)+43> eflags 0x10206 [ PF IF RF ] cs 0x33 51 ss 0x2b 43 ds 0x0 0 es 0x0 0 fs 0x0 0 gs 0x0 0 k0 0x0 0 k1 0x0 0 k2 0x0 0 k3 0x0 0 k4 0x0 0 k5 0x0 0 k6 0x0 0 k7 0x0 0 (gdb) disassemble Dump of assembler code for function v8::internal::compiler::(anonymous namespace)::MayAlias(v8::internal::compiler::Node*, v8::internal::compiler::Node*): 0x00007fe9488dcb40 <+0>: push %rbp 0x00007fe9488dcb41 <+1>: mov %rsp,%rbp 0x00007fe9488dcb44 <+4>: push %r12 0x00007fe9488dcb46 <+6>: push %rbx 0x00007fe9488dcb47 <+7>: sub $0x10,%rsp 0x00007fe9488dcb4b <+11>: mov %fs:0x28,%rax 0x00007fe9488dcb54 <+20>: mov %rax,-0x18(%rbp) 0x00007fe9488dcb58 <+24>: xor %eax,%eax 0x00007fe9488dcb5a <+26>: mov $0x1,%eax 0x00007fe9488dcb5f <+31>: cmp %rsi,%rdi 0x00007fe9488dcb62 <+34>: je 0x7fe9488dcbc0 <v8::internal::compiler::(anonymous namespace)::MayAlias(v8::internal::compiler::Node*, v8::internal::compiler::Node*)+128> 0x00007fe9488dcb64 <+36>: mov 0x8(%rdi),%rax 0x00007fe9488dcb68 <+40>: mov %rsi,%rbx => 0x00007fe9488dcb6b <+43>: mov 0x8(%rsi),%rsi 0x00007fe9488dcb6f <+47>: mov %rdi,%r12 0x00007fe9488dcb72 <+50>: lea -0x20(%rbp),%rdi 0x00007fe9488dcb76 <+54>: mov %rax,-0x20(%rbp) 0x00007fe9488dcb7a <+58>: callq 0x7fe9484a74b0 <_ZNK2v88internal8compiler4Type5MaybeES2_@plt> 0x00007fe9488dcb7f <+63>: test %al,%al 0x00007fe9488dcb81 <+65>: je 0x7fe9488dcbc0 <v8::internal::compiler::(anonymous namespace)::MayAlias(v8::internal::compiler::Node*, v8::internal::compiler::Node*)+128> 0x00007fe9488dcb83 <+67>: mov (%rbx),%rdx 0x00007fe9488dcb86 <+70>: movzwl 0x10(%rdx),%edx 0x00007fe9488dcb8a <+74>: cmp $0x3a,%dx 0x00007fe9488dcb8e <+78>: je 0x7fe9488dcc20 <v8::internal::compiler::(anonymous namespace)::MayAlias(v8::internal::compiler::Node*, v8::internal::compiler::Node*)+224> 0x00007fe9488dcb94 <+84>: cmp $0xca,%dx 0x00007fe9488dcb99 <+89>: je 0x7fe9488dcc40 <v8::internal::compiler::(anonymous namespace)::MayAlias(v8::internal::compiler::Node*, v8::internal::compiler::Node*)+256> 0x00007fe9488dcb9f <+95>: cmp $0x27,%dx 0x00007fe9488dcba3 <+99>: je 0x7fe9488dcc20 <v8::internal::compiler::(anonymous namespace)::MayAlias(v8::internal::compiler::Node*, v8::internal::compiler::Node*)+224>from the nodejs.org binary
(gdb) info registers rax 0x4670001 73859073 rbx 0x331d740 53598016 rcx 0x331d760 53598048 rdx 0x4670001 73859073 rsi 0x3125940 51534144 rdi 0x331d740 53598016 rbp 0x7f269affc300 0x7f269affc300 rsp 0x7f269affc2e0 0x7f269affc2e0 r8 0x340cfc0 54579136 r9 0x340ce98 54578840 r10 0x0 0 r11 0x1 1 r12 0x0 0 r13 0x3125940 51534144 r14 0x331d740 53598016 r15 0x7f269407f458 139803669034072 rip 0xcfe112 0xcfe112 <v8::internal::compiler::(anonymous namespace)::MayAlias(v8::internal::compiler::Node*, v8::internal::compiler::Node*) [clone .part.54]+130> eflags 0x10202 [ IF RF ] cs 0x33 51 ss 0x2b 43 ds 0x0 0 es 0x0 0 fs 0x0 0 gs 0x0 0 k0 0x0 0 k1 0x0 0 k2 0x0 0 k3 0x0 0 k4 0x0 0 k5 0x0 0 k6 0x0 0 k7 0x0 0 (gdb) disassemble Dump of assembler code for function _ZN2v88internal8compiler12_GLOBAL__N_18MayAliasEPNS1_4NodeES4_.part.54: 0x0000000000cfe090 <+0>: push %rbp 0x0000000000cfe091 <+1>: mov %rsp,%rbp 0x0000000000cfe094 <+4>: push %r12 0x0000000000cfe096 <+6>: push %rbx 0x0000000000cfe097 <+7>: mov %rdi,%rbx 0x0000000000cfe09a <+10>: sub $0x10,%rsp 0x0000000000cfe09e <+14>: mov (%rsi),%rax 0x0000000000cfe0a1 <+17>: movzwl 0x10(%rax),%eax 0x0000000000cfe0a5 <+21>: cmp $0x3a,%ax 0x0000000000cfe0a9 <+25>: je 0xcfe0f0 <_ZN2v88internal8compiler12_GLOBAL__N_18MayAliasEPNS1_4NodeES4_.part.54+96> 0x0000000000cfe0ab <+27>: cmp $0xca,%ax 0x0000000000cfe0af <+31>: je 0xcfe198 <_ZN2v88internal8compiler12_GLOBAL__N_18MayAliasEPNS1_4NodeES4_.part.54+264> 0x0000000000cfe0b5 <+37>: cmp $0x27,%ax 0x0000000000cfe0b9 <+41>: je 0xcfe0f0 <_ZN2v88internal8compiler12_GLOBAL__N_18MayAliasEPNS1_4NodeES4_.part.54+96> 0x0000000000cfe0bb <+43>: mov (%rdi),%rdx 0x0000000000cfe0be <+46>: movzwl 0x10(%rdx),%edx 0x0000000000cfe0c2 <+50>: cmp $0x3a,%dx 0x0000000000cfe0c6 <+54>: je 0xcfe1e0 <_ZN2v88internal8compiler12_GLOBAL__N_18MayAliasEPNS1_4NodeES4_.part.54+336> 0x0000000000cfe0cc <+60>: cmp $0xca,%dx 0x0000000000cfe0d1 <+65>: je 0xcfe1c8 <_ZN2v88internal8compiler12_GLOBAL__N_18MayAliasEPNS1_4NodeES4_.part.54+312> 0x0000000000cfe0d7 <+71>: cmp $0x27,%dx 0x0000000000cfe0db <+75>: je 0xcfe1e0 <_ZN2v88internal8compiler12_GLOBAL__N_18MayAliasEPNS1_4NodeES4_.part.54+336> 0x0000000000cfe0e1 <+81>: mov $0x1,%eax 0x0000000000cfe0e6 <+86>: add $0x10,%rsp 0x0000000000cfe0ea <+90>: pop %rbx 0x0000000000cfe0eb <+91>: pop %r12 0x0000000000cfe0ed <+93>: pop %rbp 0x0000000000cfe0ee <+94>: retq 0x0000000000cfe0ef <+95>: nop 0x0000000000cfe0f0 <+96>: movzbl 0x17(%rsi),%eax 0x0000000000cfe0f4 <+100>: and $0xf,%eax 0x0000000000cfe0f7 <+103>: cmp $0xf,%eax 0x0000000000cfe0fa <+106>: je 0xcfe200 <_ZN2v88internal8compiler12_GLOBAL__N_18MayAliasEPNS1_4NodeES4_.part.54+368> 0x0000000000cfe100 <+112>: mov 0x20(%rsi),%r12 0x0000000000cfe104 <+116>: cmp %r12,%rbx 0x0000000000cfe107 <+119>: mov $0x1,%eax 0x0000000000cfe10c <+124>: je 0xcfe0e6 <_ZN2v88internal8compiler12_GLOBAL__N_18MayAliasEPNS1_4NodeES4_.part.54+86> 0x0000000000cfe10e <+126>: mov 0x8(%rbx),%rax => 0x0000000000cfe112 <+130>: mov 0x8(%r12),%rsi 0x0000000000cfe117 <+135>: lea -0x20(%rbp),%rdi 0x0000000000cfe11b <+139>: mov %rax,-0x20(%rbp) 0x0000000000cfe11f <+143>: callq 0xdce240 <_ZNK2v88internal8compiler4Type5MaybeES2_> 0x0000000000cfe124 <+148>: test %al,%al 0x0000000000cfe126 <+150>: je 0xcfe0e6 <_ZN2v88internal8compiler12_GLOBAL__N_18MayAliasEPNS1_4NodeES4_.part.54+86> 0x0000000000cfe128 <+152>: mov (%r12),%rdx 0x0000000000cfe12c <+156>: movzwl 0x10(%rdx),%edx 0x0000000000cfe130 <+160>: cmp $0x3a,%dx 0x0000000000cfe134 <+164>: je 0xcfe238 <_ZN2v88internal8compiler12_GLOBAL__N_18MayAliasEPNS1_4NodeES4_.part.54+424> 0x0000000000cfe13a <+170>: cmp $0xca,%dx 0x0000000000cfe13f <+175>: je 0xcfe210 <_ZN2v88internal8compiler12_GLOBAL__N_18MayAliasEPNS1_4NodeES4_.part.54+384> 0x0000000000cfe145 <+181>: cmp $0x27,%dx 0x0000000000cfe149 <+185>: je 0xcfe238 <_ZN2v88internal8compiler12_GLOBAL__N_18MayAliasEPNS1_4NodeES4_.part.54+424> 0x0000000000cfe14f <+191>: mov (%rbx),%rcx 0x0000000000cfe152 <+194>: movzwl 0x10(%rcx),%ecx 0x0000000000cfe156 <+198>: cmp $0x3a,%cx 0x0000000000cfe15a <+202>: je 0xcfe171 <_ZN2v88internal8compiler12_GLOBAL__N_18MayAliasEPNS1_4NodeES4_.part.54+225> 0x0000000000cfe15c <+204>: cmp $0xca,%cx 0x0000000000cfe161 <+209>: je 0xcfe270 <_ZN2v88internal8compiler12_GLOBAL__N_18MayAliasEPNS1_4NodeES4_.part.54+480> 0x0000000000cfe167 <+215>: cmp $0x27,%cx 0x0000000000cfe16b <+219>: jne 0xcfe0e6 <_ZN2v88internal8compiler12_GLOBAL__N_18MayAliasEPNS1_4NodeES4_.part.54+86> 0x0000000000cfe171 <+225>: movzbl 0x17(%rbx),%eaxNot sure why the disassembly shows mangled function names.
Thanks. So in both cases it crashes because the second function argument is a nullptr. What happens when you start node with
--noturbo_load_elimination?Is checking with the latest v12.x or v13.x an option for you? v10.x is still at V8 6.8 and that's positively ancient by now.
The bug may have been fixed in a newer release. v8/v8@b28637b seems like a good candidate.
What happens when you start node with
--noturbo_load_elimination?Started the unmodified application at 13:40:45 UTC using
/original-node/node-v10.15.2-linux-x64/bin/node --noturbo_load_elimination dist/server.js.Is checking with the latest v12.x or v13.x an option for you?
I can check within the staging system with arbitrary node versions (I already downloaded some random precompiled binaries from the Internet upon request from addaleax). For the production system I would have to check with my manager, not using the Debian Packages comes with quite a bit of operational complexity I'd like to avoid.
It's 21:44:50 UTC now. The process is now running since roughly 8 hours without the crash by specifying the
--noturbo_load_eliminationoption. Previously it would more or less reliably crash every 30 minutes to 1 hour. I guess it's safe to say that the option works around the issue. I don't expect a noticeable performance hit, so this might be the best solution for production.While researching the command line flag I also stumbled upon
--trace-turbo-load-elimination. Would you consider it useful to enable this option for debugging? I suppose it would log whenever that specific optimization kicks in?As a quick sanity check, let's see if the regression test from v8/v8@b28637b triggers a crash.
It does so I think we've found our culprit. I've opened #31507 to back-port the fix to v10.x.
FWIW: The CI appears to be behind OAuth I did not attempt to authorize. Not sure whether it was meant for me or just for tracking it yourself.
In any case I trust your judgement here. Thanks for the quick turn around time. If you'd like me to test some special build to verify I'm happy to do so if you give me instructions.
For us both, really. :-)
If you could build from source and test with the patch, that'd of course be great. Second-best is if you can try with the next v10.x release because I don't know how often Debian updates their node package.
If you could build from source and test with the patch, that'd of course be great.
I compiled my last node.js from source back in the 0.10.x days. I'll have a look.
because I don't know how often Debian updates their node package.
Not at all. Debian Buster never got an updated nodejs since 2019-04-17 (https://tracker.debian.org/pkg/nodejs). Not even for backported security fixes which is odd. I plan to file a bug with them to get the patch backported to Debian stable once it is verified fixed, though.
3 remaining items
Hm, that's sad news. If you're up for it, a debug build (
make -j8 -C out BUILDTYPE=Debug- binary atout/Debug/node) might help catch the bug earlier or produce more meaningful stack traces.I ran the regression test from the pull request with Debian's node, self compiled node w/o patch and self compiled w/ patch. None of those crashed.
Yes, that's expected. Node does a lot of JS bootstrapping that d8, the V8 shell, doesn't - and perturbs the environment the test runs in. Many of V8's regression tests rely on a pristine environment.
./configure --enable-d8 && make -j8builds d8 if you want to try it out.If you're up for it, a debug build (
make -j8 -C out BUILDTYPE=Debug- binary atout/Debug/node) might help catch the bug earlier or produce more meaningful stack traces.After adding 3G of swap file to that 1G of memory VM it compiled without being killed due to an out of memory condition and spat out 4.7G of artifacts, filling up the disk to ~100%.
That said: The application is now running on a debug binary … I hope.
Okay, it took 34 minutes for the Debug binary to crash. Compiled as
make -C out BUILDTYPE=Debugwith g++, copied just thenodebinary into a different folder.Core was generated by `/my-node/node dist/server.js'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x0000564d776eb69a in v8::internal::compiler::Node::type (this=0x0) at ../deps/v8/src/compiler/node.h:267 267 Type type() const { return type_; } [Current thread is 1 (Thread 0x7ff812af3700 (LWP 1795))] (gdb) bt #0 0x0000564d776eb69a in v8::internal::compiler::Node::type (this=0x0) at ../deps/v8/src/compiler/node.h:267 #1 0x0000564d776eb6fd in v8::internal::compiler::NodeProperties::IsTyped (node=0x0) at ../deps/v8/src/compiler/node-properties.h:186 #2 0x0000564d776eb72a in v8::internal::compiler::NodeProperties::GetType (node=0x0) at ../deps/v8/src/compiler/node-properties.h:188 #3 0x0000564d778027e5 in v8::internal::compiler::(anonymous namespace)::MayAlias (a=0x564d7a5977d0, b=0x0) at ../deps/v8/src/compiler/load-elimination.cc:39 #4 0x0000564d77802874 in v8::internal::compiler::(anonymous namespace)::MayAlias (a=0x564d7a5977d0, b=0x564d7a25e9b0) at ../deps/v8/src/compiler/load-elimination.cc:56 #5 0x0000564d778051ec in v8::internal::compiler::LoadElimination::AliasStateInfo::MayAlias (this=0x7ff812af2320, other=0x564d7a25e9b0) at ../deps/v8/src/compiler/load-elimination.cc:679 #6 0x0000564d77803db9 in v8::internal::compiler::LoadElimination::AbstractField::Kill (this=0x7ff7fc089810, alias_info=..., name=..., zone=0x564d7a4da140) at ../deps/v8/src/compiler/load-elimination.cc:372 #7 0x0000564d778050b0 in v8::internal::compiler::LoadElimination::AbstractState::KillFields (this=0x7ff7fc08a780, object=0x564d7a5977d0, name=..., zone=0x564d7a4da140) at ../deps/v8/src/compiler/load-elimination.cc:654 #8 0x0000564d7780677a in v8::internal::compiler::LoadElimination::ReduceStoreField (this=0x7ff812af28f0, node=0x564d7a5fe5a0) at ../deps/v8/src/compiler/load-elimination.cc:979 #9 0x0000564d77802d26 in v8::internal::compiler::LoadElimination::Reduce (this=0x7ff812af28f0, node=0x564d7a5fe5a0) at ../deps/v8/src/compiler/load-elimination.cc:131 #10 0x0000564d7786bfcb in v8::internal::compiler::(anonymous namespace)::SourcePositionWrapper::Reduce (this=0x564d7a634648, node=0x564d7a5fe5a0) at ../deps/v8/src/compiler/pipeline.cc:651 #11 0x0000564d77735c77 in v8::internal::compiler::GraphReducer::Reduce (this=0x7ff812af2b30, node=0x564d7a5fe5a0) at ../deps/v8/src/compiler/graph-reducer.cc:85 #12 0x0000564d77736107 in v8::internal::compiler::GraphReducer::ReduceTop (this=0x7ff812af2b30) at ../deps/v8/src/compiler/graph-reducer.cc:152 #13 0x0000564d777359fe in v8::internal::compiler::GraphReducer::ReduceNode (this=0x7ff812af2b30, node=0x564d7a3d3828) at ../deps/v8/src/compiler/graph-reducer.cc:56 #14 0x0000564d77735bb6 in v8::internal::compiler::GraphReducer::ReduceGraph (this=0x7ff812af2b30) at ../deps/v8/src/compiler/graph-reducer.cc:78 #15 0x0000564d7786fb2e in v8::internal::compiler::LoadEliminationPhase::Run (this=0x7ff812af2c6f, data=0x564d7a193cd8, temp_zone=0x564d7a4da140) at ../deps/v8/src/compiler/pipeline.cc:1457 #16 0x0000564d7787473f in v8::internal::compiler::PipelineImpl::Run<v8::internal::compiler::LoadEliminationPhase> (this=0x564d7a193e58) at ../deps/v8/src/compiler/pipeline.cc:1037 #17 0x0000564d778715e9 in v8::internal::compiler::PipelineImpl::OptimizeGraph (this=0x564d7a193e58, linkage=0x564d7a1453d0) at ../deps/v8/src/compiler/pipeline.cc:1912 #18 0x0000564d7786ca87 in v8::internal::compiler::PipelineCompilationJob::ExecuteJobImpl (this=0x564d7a193b20) at ../deps/v8/src/compiler/pipeline.cc:847 #19 0x0000564d77662c2c in v8::internal::OptimizedCompilationJob::ExecuteJob (this=0x564d7a193b20) at ../deps/v8/src/compiler.cc:223 #20 0x0000564d7765882f in v8::internal::OptimizingCompileDispatcher::CompileNext (this=0x564d7a02e690, job=0x564d7a193b20) at ../deps/v8/src/compiler-dispatcher/optimizing-compile-dispatcher.cc:118 #21 0x0000564d776584c6 in v8::internal::OptimizingCompileDispatcher::CompileTask::RunInternal (this=0x564d7a228c40) at ../deps/v8/src/compiler-dispatcher/optimizing-compile-dispatcher.cc:69 #22 0x0000564d7758732f in v8::internal::CancelableTask::Run (this=0x564d7a228c40) at ../deps/v8/src/cancelable-task.h:148 #23 0x0000564d771cd011 in node::BackgroundRunner (data=0x564d79fbc640) at ../src/node_platform.cc:42 #24 0x00007ff8144c4fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486 #25 0x00007ff8143f34cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 (gdb) info registers rax 0x0 0 rbx 0x7ff7fc08fa00 140703062096384 rcx 0x0 0 rdx 0x0 0 rsi 0x0 0 rdi 0x0 0 rbp 0x7ff812af2140 0x7ff812af2140 rsp 0x7ff812af2140 0x7ff812af2140 r8 0x7ff7fc08f6a0 140703062095520 r9 0x1 1 r10 0x7c 124 r11 0x1 1 r12 0x564d7a4d5c60 94890764360800 r13 0x7fff36c0f2df 140734112002783 r14 0x7ff812af3700 140703442089728 r15 0x0 0 rip 0x564d776eb69a 0x564d776eb69a <v8::internal::compiler::Node::type() const+12> eflags 0x10206 [ PF IF RF ] cs 0x33 51 ss 0x2b 43 ds 0x0 0 es 0x0 0 fs 0x0 0 gs 0x0 0 k0 0x0 0 k1 0x0 0 k2 0x0 0 k3 0x0 0 k4 0x0 0 k5 0x0 0 k6 0x0 0 k7 0x0 0 (gdb) disassemble Dump of assembler code for function v8::internal::compiler::Node::type() const: 0x0000564d776eb68e <+0>: push %rbp 0x0000564d776eb68f <+1>: mov %rsp,%rbp 0x0000564d776eb692 <+4>: mov %rdi,-0x8(%rbp) 0x0000564d776eb696 <+8>: mov -0x8(%rbp),%rax => 0x0000564d776eb69a <+12>: mov 0x8(%rax),%rax 0x0000564d776eb69e <+16>: pop %rbp 0x0000564d776eb69f <+17>: retq End of assembler dump. (gdb) list 262 263 // Only NodeProperties should manipulate the op. 264 void set_op(const Operator* op) { op_ = op; } 265 266 // Only NodeProperties should manipulate the type. 267 Type type() const { return type_; } 268 void set_type(Type type) { type_ = type; } 269 270 // Only NodeMarkers should manipulate the marks on nodes. 271 Mark mark() const { return mark_; }Find attached the output of the Debug node binary running with
--trace-turbo-load-elimination. It shows the last seconds of output before the crash. Any private log output by the application itself has been redacted and replaced with*snip*. I have the core dump on record (the stack matches my previous comment).Thanks, I can see the general shape of the bug now.
The fourth stack frame is a
return MayAlias(a, b->InputAt(0));that I'm reasonably sure ought to be guarded onif (!b->IsDead()). There are a couple of similar calls that I need to check.If you're up for it, I can write a patch for you to try out.
If you're up for it, I can write a patch for you to try out.
Sure, I'm happy to test the patches you throw at me. It's not too much effort for me to test them, most of the time is spent waiting to find whether it crashes or not.
https://github.com/bnoordhuis/io.js/commit/473868ecc2.patch
Passes
make testlocally. Let's see if it also passes V8's test suite: https://ci.nodejs.org/job/node-test-commit-v8-linux/2834/I've applied that patch and recompiled node.js. The process was started with the new binary at 14:31:34 UTC. Let's wait and see.
It's 23:00:00 UTC now. The process is running fine since 8.5 hours. I guess it's safe to say that the patch indeed fixes my issue. Thanks.
- added a commit that references this issue
on Feb 2, 2020 Thanks for testing. I've opened #31613.
Reacted by Beth Griggs- added a commit that references this issue
on Feb 26, 2020
Debian Buster on amd64.
I'm seeing a more or less regular crash of a non-trivial application within libnode.so.64. It's running on a low-traffic staging system. Find below an excerpt from the syslog of the affected machine. Times are in UTC.
After becoming aware of the issue I made sure that the process could dump core and I also installed the relevant debug symbols. My understanding is that node crashes within v8's JIT (?) compiler:
I have the core dump on record and can provide additional information from the core file on request.
The application is launched using systemd. Find below the unit file for your reference:
Unit File (click to expand)
Find below a list of open files from the process after it was restarted automatically by systemd:
lsof -p $pid (click to expand)