Skip to content

Crash during Garbage collection in TLSWrap #17097

Closed
Closed
Crash during Garbage collection in TLSWrap#17097
Labels
c++Issues and PRs that require attention from people who are familiar with C++.tlsIssues and PRs related to the tls subsystem.

Description

@surajwy
surajwy
opened on Nov 17, 2017
  • Version:v6.11.0
  • Platform:Amazon Linux 4.9.27-14.31.amzn1.x86_64
  • OpenSSL version: OpenSSL 1.0.1k-fips

I am experiencing a crash in PostGarbageCollectionProcessing call.
I do not have any reproducible case. But it happens 1-2 times a day in production.
Whenever I have got this crash, we were connecting to AWS's endpoint.

Below is my stack trace:

(lldb) v8 bt
 * SBThread: tid = 0x0000  * frame #0: 0x000000000111b346 node`node::TLSWrap::~TLSWrap() + 358
    frame #1: 0x000000000111b421 node`node::TLSWrap::~TLSWrap() + 17
    frame #2: 0x0000000000c85e5c node`v8::internal::GlobalHandles::DispatchPendingPhantomCallbacks(bool) + 252
    frame #3: 0x0000000000c860ba node`v8::internal::GlobalHandles::PostGarbageCollectionProcessing(v8::internal::GarbageCollector, v8::GCCallbackFlags) + 42
    frame #4: 0x0000000000ca6e37 node`v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) + 535
    frame #5: 0x0000000000ca771a node`v8::internal::Heap::CollectGarbage(v8::internal::GarbageCollector, char const*, char const*, v8::GCCallbackFlags) + 330
    frame #6: 0x0000000000ca932e node`v8::internal::Heap::HandleGCRequest() + 222
    frame #7: 0x0000000000c55bac node`v8::internal::StackGuard::HandleInterrupts() + 796
    frame #8: 0x0000000000eb75fc node`v8::internal::Runtime_StackGuard(int, v8::internal::Object**, v8::internal::Isolate*) + 380
    frame #9: 0x00001ee4063079a7 <exit>

Contents of registers:

(lldb) register read
General Purpose Registers:
       rax = 0x0000000008617ac8
       rbx = 0x0000000008617950
       rcx = 0x1baffed00baffedf
       rdx = 0x0000000000000000
       rdi = 0x0000000000000000
       rsi = 0x0000000008617ac8
       rbp = 0x00007fff78159b10
       rsp = 0x00007fff78159b00
        r8 = 0x0000000000000001
        r9 = 0x00000000084fda90
       r10 = 0x0007312ef242a226
       r11 = 0x0000000000000246
       r12 = 0x0000000000000000
       r13 = 0x00000000000000ce
       r14 = 0x0000000000000001
       r15 = 0x0000000004dc9e80
       rip = 0x000000000111b346  node`node::TLSWrap::~TLSWrap() + 358
    rflags = 0x0000000000010246
        cs = 0x0000000000000033
        fs = 0x0000000000000000
        gs = 0x0000000000000000
        ss = 0x000000000000002b
        ds = 0x0000000000000000
        es = 0x0000000000000000

Disassembly of the top frame is attached here: disassembly.log

Activity

  1. bnoordhuis commented on Nov 17, 2017

    @bnoordhuis
    bnoordhuis
    on Nov 17, 2017
    Member

    Looks like the instruction pointer ended up in an all zeros page. Two questions:

    1. What node binary is this? What do node -p process.versions and node -p process.config print?
    2. Can you reproduce with the latest v6.x, v6.12.0?

  2. surajwy commented on Nov 17, 2017

    @surajwy
    surajwy
    on Nov 17, 2017
    Author
    1. I have downloaded this binary from nodejs.org.
    node -p process.versions
{ http_parser: '2.7.0',
  node: '6.11.0',
  v8: '5.1.281.102',
  uv: '1.11.0',
  zlib: '1.2.11',
  ares: '1.10.1-DEV',
  icu: '58.2',
  modules: '48',
  openssl: '1.0.2k' }

    node -p process.config
{ target_defaults: 
   { cflags: [],
     default_configuration: 'Release',
     defines: [],
     include_dirs: [],
     libraries: [] },
  variables: 
   { asan: 0,
     coverage: false,
     debug_devtools: 'node',
     force_dynamic_crt: 0,
     gas_version: '2.23',
     host_arch: 'x64',
     icu_data_file: 'icudt58l.dat',
     icu_data_in: '../../deps/icu-small/source/data/in/icudt58l.dat',
     icu_endianness: 'l',
     icu_gyp_path: 'tools/icu/icu-generic.gyp',
     icu_locales: 'en,root',
     icu_path: 'deps/icu-small',
     icu_small: true,
     icu_ver_major: '58',
     node_byteorder: 'little',
     node_enable_d8: false,
     node_enable_v8_vtunejit: false,
     node_install_npm: true,
     node_module_version: 48,
     node_no_browser_globals: false,
     node_prefix: '/',
     node_release_urlbase: 'https://nodejs.org/download/release/',
     node_shared: false,
     node_shared_cares: false,
     node_shared_http_parser: false,
     node_shared_libuv: false,
     node_shared_openssl: false,
     node_shared_zlib: false,
     node_tag: '',
     node_use_bundled_v8: true,
     node_use_dtrace: false,
     node_use_etw: false,
     node_use_lttng: false,
     node_use_openssl: true,
     node_use_perfctr: false,
     node_use_v8_platform: true,
     openssl_fips: '',
     openssl_no_asm: 0,
     shlib_suffix: 'so.48',
     target_arch: 'x64',
     uv_parent_path: '/deps/uv/',
     uv_use_dtrace: false,
     v8_enable_gdbjit: 0,
     v8_enable_i18n_support: 1,
     v8_inspector: true,
     v8_no_strict_aliasing: 1,
     v8_optimized_debug: 0,
     v8_random_seed: 0,
     v8_use_snapshot: true,
     want_separate_host_toolset: 0 } }
    1. Will try to reproduce this on latest v6.x on Monday. Will share the results.
      Also, FYI, we have also tried this on node v8.6 and this crash was still happening.

  3. bnoordhuis commented on Nov 17, 2017

    @bnoordhuis
    bnoordhuis
    on Nov 17, 2017
    Member

    Does your application contain sensitive information? If it doesn't, perhaps you can upload a core dump from the latest v6.x or v8.x? If you don't want to share the link publicly, my email is info@bnoordhuis.nl.

  4. mscdex
    added
    c++Issues and PRs that require attention from people who are familiar with C++.
    tlsIssues and PRs related to the tls subsystem.
    v6.x
    and removed
    v6.x
    on Nov 17, 2017

  5. surajwy commented on Nov 17, 2017

    @surajwy
    surajwy
    on Nov 17, 2017
    Author

    Yes, my app contains sensitive information. Won't be able to share. Sorry about that.

    We are trying to move our app from node v0.10.32 to an LTS node version. There are 2 more types of crashes and I will open separate git issues for them. We do have some add-on modules, but with these crash dumps, I am not able to pinpoint any issues with them.

  6. addaleax commented on Dec 10, 2017

    @addaleax
    addaleax
    on Dec 10, 2017
    Member

    @surajwy Is there a chance you could try the patch from #17478?

  7. surajwy commented on Dec 15, 2017

    @surajwy
    surajwy
    on Dec 15, 2017
    Author

    @addaleax, Yes, I will give it a try next week.

  8. apapirovski commented on Dec 18, 2017

    @apapirovski
    apapirovski
    on Dec 18, 2017
    Contributor

    This got fixed in #17478 but the RegExp didn't pick it up due to the struck out words. @surajwy feel free to reopen if the issue persists after building from master with that patch. Thanks!

  9. apapirovski
    closed this as completedon Dec 18, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    c++Issues and PRs that require attention from people who are familiar with C++.tlsIssues and PRs related to the tls subsystem.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions