Skip to content

async-hooks.test-writewrap is flaky #54809

Closed as not planned
Closed as not planned
async-hooks.test-writewrap is flaky#54809
Labels
async_hooksIssues and PRs related to the async hooks subsystem.flaky-testIssues and PRs involving tests that fail intermittently in CI.linuxIssues and PRs related to the Linux platform.staleIssues and PRs marked stale due to inactivity and scheduled for automatic closure.

Description

@anonrig
anonrig
opened on Sep 6, 2024

Test

async-hooks.test-writewrap

Platform

Linux x64

Console output

14:06:24 not ok 3869 async-hooks/test-writewrap
14:06:24   ---
14:06:24   duration_ms: 120029.98300
14:06:24   severity: fail
14:06:24   exitcode: -15
14:06:24   stack: |-
14:06:24     timeout
14:06:24   ...

Build links

Additional information

No response

Activity

  1. anonrig
    added
    flaky-testIssues and PRs involving tests that fail intermittently in CI.
    on Sep 6, 2024
  2. github-actions
    added
    linuxIssues and PRs related to the Linux platform.
    on Sep 6, 2024
  3. avivkeller
    added
    async_hooksIssues and PRs related to the async hooks subsystem.
    on Sep 6, 2024

  4. aduh95 commented on Sep 6, 2024

    @aduh95
    aduh95
    on Sep 6, 2024
    Contributor

    Reproducible locally (macOS ARM), looks like it's a native issue (CRASHED (Signal: 11) means SIGSEGV)

    tools/test.py --repeat 9999 test/async-hooks/test-writewrap.js
=== release test-writewrap ===                    
Path: async-hooks/test-writewrap
Command: out/Release/node /Users/duhamean/Documents/node/test/async-hooks/test-writewrap.js
--- CRASHED (Signal: 11) ---


=== release test-writewrap ===                    
Path: async-hooks/test-writewrap
Command: out/Release/node /Users/duhamean/Documents/node/test/async-hooks/test-writewrap.js
--- CRASHED (Signal: 11) ---


=== release test-writewrap ===                    
Path: async-hooks/test-writewrap
Command: out/Release/node /Users/duhamean/Documents/node/test/async-hooks/test-writewrap.js
--- CRASHED (Signal: 11) ---


=== release test-writewrap ===                    
Path: async-hooks/test-writewrap
Command: out/Release/node /Users/duhamean/Documents/node/test/async-hooks/test-writewrap.js
--- TIMEOUT ---


=== release test-writewrap ===                    
Path: async-hooks/test-writewrap
Command: out/Release/node /Users/duhamean/Documents/node/test/async-hooks/test-writewrap.js
--- TIMEOUT ---


=== release test-writewrap ===                    
Path: async-hooks/test-writewrap
Command: out/Release/node /Users/duhamean/Documents/node/test/async-hooks/test-writewrap.js
--- TIMEOUT ---


[08:07|% 100|+ 9993|-   6]: Done

  5. lpinca commented on Sep 7, 2024

    @lpinca
    lpinca
    on Sep 7, 2024
    Member

    I get no failures on macOS

    $ npx envinfo --system

  System:
    OS: macOS 14.6.1
    CPU: (16) x64 Intel(R) Xeon(R) W-2140B CPU @ 3.20GHz
    Memory: 7.72 GB / 32.00 GB
    Shell: 5.2.32 - /usr/local/bin/bash

$ tools/test.py --repeat 9999 test/async-hooks/test-writewrap.js
[03:03|% 100|+ 9999|-   0]: Done                  

All tests passed.

  6. avivkeller commented on Sep 8, 2024

    @avivkeller
    avivkeller
    on Sep 8, 2024
    Member

    FWIW I can't reproduce on Linux x64:

      System:
    OS: Linux 6.8 Kali GNU/Linux Rolling 2024.3
    CPU: (12) x64 AMD Ryzen 5 5600G with Radeon Graphics
    Memory: 21.71 GB / 27.30 GB
    Container: Yes
    Shell: 5.9 - /usr/bin/zsh

    $ tools/test.py --repeat 9999 test/async-hooks/test-writewrap.js
[02:35|% 100|+ 9999|-   0]: Done                  

All tests passed.

    Maybe it has to do with the amount of memory available?

  7. jasnell commented on Sep 8, 2024

    @jasnell
    jasnell
    on Sep 8, 2024
    Member

    I think with many of these timeout tests we should consider moving them into the sequential suite to see if that helps

  8. lpinca commented on Sep 8, 2024

    @lpinca
    lpinca
    on Sep 8, 2024
    Member

    @jasnell no, we will likely end up hiding a real bug. Read the discussions in #52959 and #52964.

  9. jasnell commented on Sep 8, 2024

    @jasnell
    jasnell
    on Sep 8, 2024
    Member

    @lpinca ... we don't have to actually land the PR. I want to determine if moving the test would make it less flaky which can be done with a PR and a few stress test runs. The PR doesn't actually have to land. That said, we have to get the CI back to a point where we aren't having to do 10-15 CI runs on a PR just to get it landed. If that means we freeze all development on everything else until CI is green again, let's do that.

  10. lpinca commented on Sep 8, 2024

    @lpinca
    lpinca
    on Sep 8, 2024
    Member

    I want to determine if moving the test would make it less flaky which can be done with a PR and a few stress test runs. The PR doesn't actually have to land.

    Ofc it will, it will not trigger the issue... The stress test CI does not run the test in parallel.

    That said, we have to get the CI back to a point where we aren't having to do 10-15 CI runs on a PR just to get it landed.

    Let's start by fixing the issues with the macOS machines which is what actually blocked development.

  11. lpinca commented on Sep 8, 2024

    @lpinca
    lpinca
    on Sep 8, 2024
    Member

    @jasnell try this

    python3 tools/test.py --repeat=10000 test/parallel/test-net-write-fully-async-hex-string.js -t 10

    or

    python3 tools/test.py --repeat=10000 test/parallel/test-buffer-tostring-rangeerror.js -t 10

    I think that if we find the reason why the process sometimes hangs we will be able to fix the timeout issue for many tests.

  12. jasnell commented on Sep 8, 2024

    @jasnell
    jasnell
    on Sep 8, 2024
    Member

    Let's start by fixing the issues with the macOS machines which is what actually blocked development.

    I can't do anything about the macOS machines right now. I can work on investigating these timeouts, which is what I'm trying to do.

    Ofc it will, it will not trigger the issue....

    Has that been tested? If so, can you point out where?

  13. jasnell commented on Sep 8, 2024

    @jasnell
    jasnell
    on Sep 8, 2024
    Member

    Going through the test-net-write-fully-async-hex-string.js test... I'm able to eliminate timeouts in local testing by reducing the size of the initial buffer allocation. If I keep the allocation under the default high water mark for the stream it appears to never time out. If the buffer passed into the write operation repeatedly is over the high water mark, it times out. The higher the allocation, the more frequently the test times out.

  14. jasnell commented on Sep 9, 2024

    @jasnell
    jasnell
    on Sep 9, 2024
    Member

    Running some local tests on test-net-write-fully-async-hex-string.js ... here are the results I'm getting locally

    Allocation Size Average Timeouts per 10000 repetitions Runs Timeout
    65600 8 10 60s
    65580 4 10 60s
    65570 0 10 60s
    65560 0 10 60s
    65550 0 10 60s

    So far this is the only variable that I've been able to identify that reliably alters the rate of timeout on this particular test.

    The reason for choosing 65550 is that it is fairly close (but still over the default high water mark for the stream. Setting the highwatermark to 65600 appears to reduce the rate of timeout when using 65600 as the allocation size.

    This makes me think that the tests that are timing out are due to some intersection between v8 GC and streams backpressure management -- I'm unable to reproduce any timeouts for any allocations smaller than the default high water mark size.

    /cc @mcollina @ronag @anonrig for visibility

  15. lpinca commented on Sep 9, 2024

    @lpinca
    lpinca
    on Sep 9, 2024
    Member

    @jasnell in my environment the issue seems different, see #52959 (comment).
    The server emits 'close', meaning that all data is flushed. It does not seem to be a backpressure issue. Anyway, let's continue the discussion on #52959 so as not to create noise here.

  16. github-actions commented on Apr 28, 2026

    @github-actions
    github-actionsbot
    on Apr 28, 2026 – with GitHub Actions
    Contributor

    This issue has been marked as stale due to 210 days of inactivity.
    It will be automatically closed in 30 days if no further activity occurs. If this is still relevant, please leave a comment or update it to keep it open.

  17. github-actions
    added
    staleIssues and PRs marked stale due to inactivity and scheduled for automatic closure.
    on Apr 28, 2026

  18. github-actions commented on May 29, 2026

    @github-actions
    github-actionsbot
    on May 29, 2026 – with GitHub Actions
    Contributor

    This issue has been automatically closed after 30 days of inactivity following its stale status (no activity for a total of 240 days).
    If this is still relevant, feel free to reopen it or leave a comment with additional details so we can continue the discussion.

  19. github-actions
    closed this as not plannedon May 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    async_hooksIssues and PRs related to the async hooks subsystem.flaky-testIssues and PRs involving tests that fail intermittently in CI.linuxIssues and PRs related to the Linux platform.staleIssues and PRs marked stale due to inactivity and scheduled for automatic closure.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions