Skip to content

Investigate flaky parallel/test-worker-memory #23277

Closed
#41867
Closed
Investigate flaky parallel/test-worker-memory#23277
#41867
Labels
flaky-testIssues and PRs involving tests that fail intermittently in CI.windowsIssues and PRs related to the Windows platform.workerIssues and PRs related to the worker_threads module and Worker API.

Description

@Trott
Trott
opened on Oct 5, 2018

https://ci.nodejs.org/job/node-test-commit-arm/18980/nodes=centos7-arm64-gcc6/console

00:03:00 not ok 2022 parallel/test-worker-memory
00:03:00   ---
00:03:00   duration_ms: 1.17
00:03:00   severity: crashed
00:03:00   exitcode: -7
00:03:00   stack: |-
00:03:00   ...

Activity

  1. Trott
    added
    flaky-testIssues and PRs involving tests that fail intermittently in CI.
    on Oct 5, 2018

  2. Trott commented on Oct 10, 2018

    @Trott
    Trott
    on Oct 10, 2018
    MemberAuthor

    Different:

    https://ci.nodejs.org/job/node-test-commit-linux/22212/nodes=ubuntu1804-docker/console

    00:51:34 not ok 2141 parallel/test-worker-memory
00:51:34   ---
00:51:34   duration_ms: 120.40
00:51:34   severity: fail
00:51:34   exitcode: -15
00:51:34   stack: |-
00:51:34     timeout

  3. Trott commented on Oct 10, 2018

    @Trott
    Trott
    on Oct 10, 2018
    MemberAuthor

    @nodejs/workers

  4. Trott commented on Oct 11, 2018

    @Trott
    Trott
    on Oct 11, 2018
    MemberAuthor

    https://ci.nodejs.org/job/node-test-binary-windows/20610/COMPILED_BY=vs2017,RUNNER=win2008r2-vs2017,RUN_SUBSET=0/console

    08:12:48 not ok 519 parallel/test-worker-memory
08:12:48   ---
08:12:48   duration_ms: 120.92
08:12:48   severity: fail
08:12:48   exitcode: 1
08:12:48   stack: |-
08:12:48     timeout
08:12:48   ...
  5. addaleax
    added
    workerIssues and PRs related to the worker_threads module and Worker API.
    on Oct 11, 2018

  6. addaleax commented on Oct 11, 2018

    @addaleax
    addaleax
    on Oct 11, 2018
    Member

    It’s unfortunate that none of these have any console output… -7 would be SIGBUS on arm Linux, if I am correct, which is something that could point to an actual bug.

  7. Trott commented on Oct 11, 2018

    @Trott
    Trott
    on Oct 11, 2018
    MemberAuthor

    It’s unfortunate that none of these have any console output… -7 would be SIGBUS on arm Linux, if I am correct, which is something that could point to an actual bug.

    We can add some console.log() calls to get an idea where the thing is getting hung up.

  8. Trott
    added a commit that references this issue on Oct 11, 2018

    test: add logging to test-worker-memory

    d60de05

  9. Trott commented on Oct 11, 2018

    @Trott
    Trott
    on Oct 11, 2018
    MemberAuthor

    Some minimal logging added in #23418.

  10. Trott
    added a commit that references this issue on Oct 11, 2018

    test: add logging to test-worker-memory

    f121ac0

  11. targos commented on Oct 16, 2018

    @targos
    targos
    on Oct 16, 2018
    Member

    Example failure with logging:

    timeout
run() called with n=15 (numWorkers=4)
run() called with n=15 (numWorkers=4)
run() called with n=15 (numWorkers=4)
run() called with n=15 (numWorkers=4)
run() called with n=14 (numWorkers=4)
run() called with n=14 (numWorkers=4)
run() called with n=14 (numWorkers=4)
run() called with n=14 (numWorkers=4)
run() called with n=13 (numWorkers=4)
run() called with n=13 (numWorkers=4)
run() called with n=13 (numWorkers=4)
run() called with n=13 (numWorkers=4)
run() called with n=12 (numWorkers=4)
run() called with n=12 (numWorkers=4)
run() called with n=12 (numWorkers=4)
run() called with n=12 (numWorkers=4)
run() called with n=11 (numWorkers=4)
run() called with n=11 (numWorkers=4)
run() called with n=11 (numWorkers=4)
run() called with n=11 (numWorkers=4)
run() called with n=10 (numWorkers=4)
run() called with n=10 (numWorkers=4)
run() called with n=10 (numWorkers=4)
run() called with n=10 (numWorkers=4)
run() called with n=9 (numWorkers=4)
run() called with n=9 (numWorkers=4)
run() called with n=9 (numWorkers=4)
run() called with n=9 (numWorkers=4)
run() called with n=8 (numWorkers=4)
run() called with n=8 (numWorkers=4)
run() called with n=8 (numWorkers=4)
run() called with n=8 (numWorkers=4)
run() called with n=7 (numWorkers=4)
run() called with n=7 (numWorkers=4)
run() called with n=7 (numWorkers=4)
run() called with n=7 (numWorkers=4)

    https://ci.nodejs.org/job/node-test-commit-linux-containered/nodes=ubuntu1604_sharedlibs_zlib_x64/7859/testReport/(root)/test/parallel_test_worker_memory/

  12. Trott commented on Oct 16, 2018

    @Trott
    Trott
    on Oct 16, 2018
    MemberAuthor

    Is there more logging we should add? Is it just taking a while? n keeps decreasing. So maybe the test just needs to be moved to sequential and/or have its iterations (or something else) reduced?

    Also happened in https://ci.nodejs.org/job/node-test-commit-linux-containered/7855/nodes=ubuntu1604_sharedlibs_zlib_x64/console:

    00:40:04 not ok 2193 parallel/test-worker-memory
00:40:04   ---
00:40:04   duration_ms: 120.161
00:40:04   severity: fail
00:40:04   exitcode: -15
00:40:04   stack: |-
00:40:04     timeout
00:40:04     run() called with n=15 (numWorkers=4)
00:40:04     run() called with n=15 (numWorkers=4)
00:40:04     run() called with n=15 (numWorkers=4)
00:40:04     run() called with n=15 (numWorkers=4)
00:40:04     run() called with n=14 (numWorkers=4)
00:40:04     run() called with n=14 (numWorkers=4)
00:40:04     run() called with n=14 (numWorkers=4)
00:40:04     run() called with n=14 (numWorkers=4)
00:40:04     run() called with n=13 (numWorkers=4)
00:40:04     run() called with n=13 (numWorkers=4)
00:40:04     run() called with n=13 (numWorkers=4)
00:40:04     run() called with n=13 (numWorkers=4)
00:40:04     run() called with n=12 (numWorkers=4)
00:40:04     run() called with n=12 (numWorkers=4)
00:40:04     run() called with n=12 (numWorkers=4)
00:40:04     run() called with n=12 (numWorkers=4)
00:40:04     run() called with n=11 (numWorkers=4)
00:40:04     run() called with n=11 (numWorkers=4)
00:40:04     run() called with n=11 (numWorkers=4)
00:40:04     run() called with n=11 (numWorkers=4)
00:40:04     run() called with n=10 (numWorkers=4)
00:40:04     run() called with n=10 (numWorkers=4)
00:40:04     run() called with n=10 (numWorkers=4)
00:40:04     run() called with n=10 (numWorkers=4)
00:40:04     run() called with n=9 (numWorkers=4)
00:40:04     run() called with n=9 (numWorkers=4)
00:40:04     run() called with n=9 (numWorkers=4)
00:40:04     run() called with n=9 (numWorkers=4)
00:40:04     run() called with n=8 (numWorkers=4)
00:40:04     run() called with n=8 (numWorkers=4)
00:40:04     run() called with n=8 (numWorkers=4)
00:40:04     run() called with n=7 (numWorkers=4)
00:40:04     run() called with n=8 (numWorkers=4)
00:40:04     run() called with n=7 (numWorkers=4)
00:40:04     run() called with n=7 (numWorkers=4)
00:40:04     run() called with n=6 (numWorkers=4)
00:40:04     run() called with n=7 (numWorkers=4)
00:40:04     run() called with n=6 (numWorkers=4)
00:40:04     run() called with n=5 (numWorkers=4)
00:40:04     run() called with n=6 (numWorkers=4)
00:40:04     run() called with n=6 (numWorkers=4)
00:40:04   ...
  13. jasnell
    added a commit that references this issue on Oct 17, 2018

    test: add logging to test-worker-memory

    0fa857f

  14. 36 remaining items

  15. Trott commented on Mar 6, 2021

    @Trott
    Trott
    on Mar 6, 2021
    MemberAuthor

    @gireeshpunathil Is there any chance the root cause has been fixed since we last visited this issue? This test hasn't failed recently.

  16. gireeshpunathil commented on Mar 6, 2021

    @gireeshpunathil
    gireeshpunathil
    on Mar 6, 2021
    Member

    @Trott - none that I am aware of. IMO, the issue cannot be comprehensively solved as long as we don't have a way to compute the actual / active / used memory by a process.

    However, if CI is consistently passing this, I guess we have no reason to keep this open. We can always re-open if we start to see this again.

  17. gireeshpunathil
    closed this as completedon Mar 6, 2021
  18. Trott
    reopened this on Apr 25, 2021

  19. Trott commented on Apr 25, 2021

    @Trott
    Trott
    on Apr 25, 2021
    MemberAuthor

    https://ci.nodejs.org/job/node-test-binary-windows-js-suites/9616/RUN_SUBSET=2,nodes=win2012r2-COMPILED_BY-vs2019-x86/console

    00:12:14 not ok 708 parallel/test-worker-memory # TODO : Fix flaky test
00:12:14   ---
00:12:14   duration_ms: 1.360
00:12:14   severity: crashed
00:12:14   exitcode: 3221225477
00:12:14   stack: |-
00:12:14     run() called with n=7.5 (numWorkers=8)
00:12:14     run() called with n=7.5 (numWorkers=8)
00:12:14     run() called with n=7.5 (numWorkers=8)
00:12:14     run() called with n=7.5 (numWorkers=8)
00:12:14     run() called with n=7.5 (numWorkers=8)
00:12:14     run() called with n=7.5 (numWorkers=8)
00:12:14     run() called with n=7.5 (numWorkers=8)
00:12:14     run() called with n=7.5 (numWorkers=8)
00:12:14     run() called with n=6.5 (numWorkers=8)
00:12:14     run() called with n=6.5 (numWorkers=8)
00:12:14     run() called with n=6.5 (numWorkers=8)
00:12:14     run() called with n=6.5 (numWorkers=8)
00:12:14     run() called with n=6.5 (numWorkers=8)
00:12:14     run() called with n=6.5 (numWorkers=8)
00:12:14     run() called with n=6.5 (numWorkers=8)
00:12:14     run() called with n=6.5 (numWorkers=8)
00:12:14     run() called with n=5.5 (numWorkers=8)
00:12:14     run() called with n=5.5 (numWorkers=8)
00:12:14     run() called with n=5.5 (numWorkers=8)
00:12:14     run() called with n=5.5 (numWorkers=8)
00:12:14     run() called with n=5.5 (numWorkers=8)
00:12:14     run() called with n=5.5 (numWorkers=8)
00:12:14     run() called with n=4.5 (numWorkers=8)
00:12:14     run() called with n=5.5 (numWorkers=8)
00:12:14     run() called with n=4.5 (numWorkers=8)
00:12:14     run() called with n=4.5 (numWorkers=8)
00:12:14     run() called with n=5.5 (numWorkers=8)
00:12:14     run() called with n=4.5 (numWorkers=8)
00:12:14     run() called with n=4.5 (numWorkers=8)
00:12:14     run() called with n=4.5 (numWorkers=8)
00:12:14     run() called with n=3.5 (numWorkers=8)
00:12:14     run() called with n=4.5 (numWorkers=8)
00:12:14     run() called with n=3.5 (numWorkers=8)
00:12:14     run() called with n=3.5 (numWorkers=8)
00:12:14     run() called with n=4.5 (numWorkers=8)
00:12:14     run() called with n=3.5 (numWorkers=8)
00:12:14     run() called with n=2.5 (numWorkers=8)
00:12:14     run() called with n=3.5 (numWorkers=8)
00:12:14     run() called with n=2.5 (numWorkers=8)
00:12:14     run() called with n=3.5 (numWorkers=8)
00:12:14     run() called with n=3.5 (numWorkers=8)
00:12:14     run() called with n=2.5 (numWorkers=8)
00:12:14     run() called with n=1.5 (numWorkers=8)
00:12:14     run() called with n=2.5 (numWorkers=8)
00:12:14     run() called with n=3.5 (numWorkers=8)
00:12:14     run() called with n=2.5 (numWorkers=8)
00:12:14     run() called with n=2.5 (numWorkers=8)
00:12:14     run() called with n=2.5 (numWorkers=8)
00:12:14     run() called with n=0.5 (numWorkers=8)
00:12:14     run() called with n=1.5 (numWorkers=8)
00:12:14     run() called with n=2.5 (numWorkers=8)
00:12:14     run() called with n=1.5 (numWorkers=8)
00:12:14     run() called with n=1.5 (numWorkers=8)
00:12:14     run() called with n=1.5 (numWorkers=8)
00:12:14     run() called with n=1.5 (numWorkers=8)
00:12:14     run() called with n=-0.5 (numWorkers=8)
00:12:14     done() called (finished=0)
00:12:14     run() called with n=0.5 (numWorkers=8)
00:12:14     run() called with n=1.5 (numWorkers=8)
00:12:14     run() called with n=0.5 (numWorkers=8)
00:12:14     run() called with n=0.5 (numWorkers=8)
00:12:14     run() called with n=0.5 (numWorkers=8)
00:12:14     run() called with n=0.5 (numWorkers=8)
00:12:14     run() called with n=0.5 (numWorkers=8)
00:12:14     run() called with n=-0.5 (numWorkers=8)
00:12:14     done() called (finished=1)
00:12:14     run() called with n=-0.5 (numWorkers=8)
00:12:14     done() called (finished=2)
00:12:14     run() called with n=-0.5 (numWorkers=8)
00:12:14     done() called (finished=3)
00:12:14     run() called with n=-0.5 (numWorkers=8)
00:12:14     done() called (finished=4)
00:12:14     run() called with n=-0.5 (numWorkers=8)
00:12:14     done() called (finished=5)
00:12:14     
00:12:14     
00:12:14     #
00:12:14     # Fatal error in , line 0
00:12:14     # Check failed: IsEmpty().
00:12:14     #
00:12:14     #
00:12:14     #
00:12:14     #FailureMessage Object: 08DFF4F4
00:12:14   ...
  20. Trott
    closed this as completedin 30bdee2on Feb 6, 2022
  21. ruyadorno
    added a commit that references this issue on Feb 8, 2022

    test: remove test-worker-memory flaky designation

    0d9248f
  22. danielleadams
    added 6 commits that reference this issue on Mar 2, 2022

    test: remove test-worker-memory flaky designation

    6a94a6b

    test: remove test-worker-memory flaky designation

    33f6d7a

    test: remove test-worker-memory flaky designation

    9b581a5

    test: remove test-worker-memory flaky designation

    708d932

    test: remove test-worker-memory flaky designation

    a8e4183
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    flaky-testIssues and PRs involving tests that fail intermittently in CI.windowsIssues and PRs related to the Windows platform.workerIssues and PRs related to the worker_threads module and Worker API.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions