Skip to content

HTTP2 server memory explodes and terminated #52952

Closed
Closed
HTTP2 server memory explodes and terminated#52952
Labels
http2Issues and PRs related to the http2 subsystem.

Description

@vilicvane
vilicvane
opened on May 12, 2024

Version

20.13.0 and later, including latest 22.x

Platform

Linux hostname 6.5.0-28-generic #29~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Apr 4 14:39:20 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

Subsystem

No response

What steps will reproduce the bug?

I may need some help to narrow down the problem as it happens sporadically.

Currently I only know that it happens after:

  1. A session created.
  2. The server pushStream to the client (and callback called).
  3. The client sends a request to create another stream and the server hangs.

I have tested with versions 20.10.0, 20.11.0, 20.12.0 and 20.12.1 and there's no problem. But since 20.13.0 the issue appears.

How often does it reproduce? Is there a required condition?

No response

What is the expected behavior? Why is that the expected behavior?

No response

What do you see instead?

I tried to use --inspect and pause on hanging (also the memory usage increases rapidly to several gigabytes and then terminated by OS I think) but no luck.

And tried --prof there's part of that looks suspicious:

   ticks parent  name
  27305   70.6%  /usr/bin/node
  24662   90.3%    JS: ^submitRstStream node:internal/http2/core:450:25
  24662  100.0%      JS: ^finishCloseStream node:internal/http2/core:1963:27
  24662  100.0%        JS: ^closeStream node:internal/http2/core:1930:21
  24662  100.0%          JS: ^_destroy node:internal/http2/core:2325:11
  24662  100.0%            JS: ^_destroy node:internal/streams/destroy:90:18

I am not a security guy but at first I thought it's something related to HTTP2 rapid reset attack and spent a whole day debugging towards that. At the end of the day I decided to downgrade Node.js and found out the older version just works.

Additional information

Also the HTTP2 server is on internet but generally only used by myself. Though I do saw a Russian IP connecting it periodically and that's also why I thought it was an attack.

But it also happens with only my own clients connected.

Activity

  1. VoltrexKeyva
    added
    http2Issues and PRs related to the http2 subsystem.
    on May 12, 2024

  2. benjamingr commented on May 13, 2024

    @benjamingr
    benjamingr
    on May 13, 2024
    Member

    @nodejs/http2

  3. mcollina commented on May 13, 2024

    @mcollina
    mcollina
    on May 13, 2024
    SponsorMember

    Can you provide steps to reproduce? We often need a reproducible example, e.g. some code that allows someone else to recreate your problem by just copying and pasting it. If it involves more than a couple of different file, create a new repository on GitHub and add a link to that.

  4. mcollina commented on May 13, 2024

    @mcollina
    mcollina
    on May 13, 2024
    SponsorMember

    These were the commits in v20.13.0 that showed the problem

    3bcd68337e] - http2: fix excessive CPU usage when using allowHTTP1=true (Eugene) #52713
    [e01015996a] - http2: fix h2-over-h2 connection proxying (Tim Perry) #52368
    [9f88736860] - http2: use internal addAbortListener (Chemi Atlow) #52081

  5. vilicvane commented on May 13, 2024

    @vilicvane
    vilicvane
    on May 13, 2024
    Author

    @mcollina I will do a further narrowing down and try to get a minimal repro later, was expecting some hint so I can reach it easier (as I've already spent a whole day on this issue so have to deprioritize it for now).

  6. moish83 commented on Dec 29, 2024

    @moish83
    moish83
    on Dec 29, 2024

    @vilicvane, we started using http2 for calling one of our third party vendors, we are calling ~150-200K request per hour. Since we moved those calls to http2 via got and http2-wrapper, we started seeing pod crashes/restarts every once in a while, and it seems to have the same pattern as described in this issue, a sudden spike in cpu and memory followed by a restart.
    We do see GOAWAY frames received from the server, not sure they are related though.
    NodeJS version 22.12.0

  7. mcollina commented on Feb 3, 2025

    @mcollina
    mcollina
    on Feb 3, 2025
    SponsorMember

    @vilicvane @moish83 can you please test with the latest v22 release? I think it might be fixed.

  8. moish83 commented on Feb 3, 2025

    @moish83
    moish83
    on Feb 3, 2025

    @mcollina, sure - will upgrade and update here

  9. vilicvane commented on Feb 3, 2025

    @vilicvane
    vilicvane
    on Feb 3, 2025
    Author

    @mcollina Sorry we have switched the relavent project to Rust recently, hope this helps for moish so that it can be closed.

  10. mcollina commented on Mar 21, 2025

    @mcollina
    mcollina
    on Mar 21, 2025
    SponsorMember

    Closing for now.

  11. mcollina
    closed this as completedon Mar 21, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    http2Issues and PRs related to the http2 subsystem.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions