Skip to content

More useful error than toString failed when Buffer length greater than kMaxLength? #3175

Closed
Closed
More useful error than toString failed when Buffer length greater than kMaxLength?#3175
Labels
bufferIssues and PRs related to the buffer subsystem.help wantedIssues that need assistance from volunteers or PRs that need help to proceed.

Description

@mhart
mhart
opened on Oct 4, 2015

I'm assuming there's some sort of memory constraint trying to convert buffers into strings?

$ uname -a
Darwin ReBuke-Pro.local 15.0.0 Darwin Kernel Version 15.0.0: Wed Aug 26 16:57:32 PDT 2015; root:xnu-3247.1.106~1/RELEASE_X86_64 x86_64
$ node --version
v4.1.1
$ node -p 'new Buffer(268435440).toString().length'
268435440

But:

$ node -p 'new Buffer(268435441).toString().length'
buffer.js:378
    throw new Error('toString failed');
    ^

Error: toString failed
    at Buffer.toString (buffer.js:378:11)
    at [eval]:1:23
    at Object.exports.runInThisContext (vm.js:54:17)
    at Object.<anonymous> ([eval]-wrapper:6:22)
    at Module._compile (module.js:434:26)
    at node.js:566:27
    at doNTCallback0 (node.js:407:9)
    at process._tickCallback (node.js:336:13)

Firstly, I'm not sure that this should even be an error (are we really only limited to 256MB? That's... not very much). This seems to have been addressed in the past, but I feel like a regression might have occurred: #1374 (comment)

But that aside, can we actually detect what the error is and report on it in a more detailed way? The current toString check provides little insight.

Activity

  1. bnoordhuis commented on Oct 4, 2015

    @bnoordhuis
    bnoordhuis
    on Oct 4, 2015
    Member

    256 MB is the limit for strings in V8 so no, it's not a regression. What would you report in more detail?

  2. mhart commented on Oct 4, 2015

    @mhart
    mhart
    on Oct 4, 2015
    ContributorAuthor

    Anything would be better than that TBH.

  3. mhart commented on Oct 4, 2015

    @mhart
    mhart
    on Oct 4, 2015
    ContributorAuthor

    It's not obvious to the user what's actually gone wrong.

    How about "Cannot create string larger than 256MB"? Seems a lot clearer to me.

  4. bnoordhuis commented on Oct 4, 2015

    @bnoordhuis
    bnoordhuis
    on Oct 4, 2015
    Member

    I think a pull request to that effect would be acceptable.

  5. mhart commented on Oct 4, 2015

    @mhart
    mhart
    on Oct 4, 2015
    ContributorAuthor

    But is that the only error that can occur? Looking at the code at the moment, the check is pretty basic – is there no way of getting more detail as to why result may be undefined?

  6. bnoordhuis commented on Oct 4, 2015

    @bnoordhuis
    bnoordhuis
    on Oct 4, 2015
    Member

    Not at the moment, no. The undefined value trickles down from StringBytes::Encode() in src/string_bytes.h. I don't think there are other code paths that return an empty Local<String> but I didn't check exhaustively.

  7. thefourtheye
    added
    bufferIssues and PRs related to the buffer subsystem.
    on Oct 4, 2015

  8. mhart commented on Oct 4, 2015

    @mhart
    mhart
    on Oct 4, 2015
    ContributorAuthor

    FWIW, v8 (I assume?) spits out a slightly better error when trying to concat strings:

    $ node -p 'var a = "a"; for (var i = 0; i < 27; i++) a = a + a; a.length'
134217728
$ node -p 'var a = "a"; for (var i = 0; i < 28; i++) a = a + a; a.length'
[eval]:1
var a = "a"; for (var i = 0; i < 28; i++) a = a + a; a.length
                                                ^

RangeError: Invalid string length
    at [eval]:1:49
    at Object.exports.runInThisContext (vm.js:54:17)
    at Object.<anonymous> ([eval]-wrapper:6:22)
    at Module._compile (module.js:434:26)
    at node.js:566:27
    at doNTCallback0 (node.js:407:9)
    at process._tickCallback (node.js:336:13)

    Is the size limit exposed at all? If it were I'd be happy to add a PR that checks, in the case of an undefined result, whether the Buffer length is greater than it.

  9. targos commented on Oct 4, 2015

    @targos
    targos
    on Oct 4, 2015
    Member

    I don't think it is exposed in JS. The exact limit is (1 << 28) - 16 (268435440) and is defined in

    node/deps/v8/include/v8.h

    Line 2083 in 64beab0

    static const int kMaxLength = (1 << 28) - 16;

  10. mhart
    changed the title [-]More useful error than `toString failed`?[/-] [+]More useful error than `toString failed` when Buffer length greater than kMaxLength?[/+] on Oct 4, 2015

  11. MylesBorins commented on Oct 6, 2015

    @MylesBorins
    MylesBorins
    on Oct 6, 2015
    Contributor

    I would love to take a stab at this if people are open to it.

    /cc @jasnell

  12. jasnell commented on Oct 6, 2015

    @jasnell
    jasnell
    on Oct 6, 2015
    Member

    Works for me!
    On Oct 6, 2015 12:49 PM, "Myles Borins" notifications@github.com wrote:

    I would love to take a stab at this if people are open to it.

    /cc @jasnell https://github.com/jasnell

    —
    Reply to this email directly or view it on GitHub
    #3175 (comment).

  13. trevnorris commented on Oct 6, 2015

    @trevnorris
    trevnorris
    on Oct 6, 2015
    Contributor

    @MylesBorins Seems the discussion addresses more than just the error message. Mind identifying the specifics of what you'll be working on?

  14. MylesBorins commented on Oct 6, 2015

    @MylesBorins
    MylesBorins
    on Oct 6, 2015
    Contributor

    I specifically was interested in a better error message when toString is called on a Buffer larger than kStringMaxLength

    That seemed like a pretty easy one to knock out (I've basically got it done already).

  15. mhart commented on Oct 6, 2015

    @mhart
    mhart
    on Oct 6, 2015
    ContributorAuthor

    @MylesBorins I think the discussion was around how to detect the limit though – have you exposed kMaxLength?

  16. 27 remaining items

  17. Trott commented on Mar 23, 2017

    @Trott
    Trott
    on Mar 23, 2017
    Member

    For what it's worth, the error message here has been marginally improved to Error: "toString()" failed (although I'm not sure why the quotation marks are used--that seems peculiar--but the addition of parentheses are a marginal improvement).

    And, of course, it still doesn't indicate why it failed though.

  18. anchnk
    added a commit that references this issue on May 6, 2017

    src: do proper StringBytes error handling

    143c891

  19. justinjdickow commented on Jul 9, 2017

    @justinjdickow
    justinjdickow
    on Jul 9, 2017

    I... don't understand why the limit is 256 MB :(

  20. mhart commented on Jul 9, 2017

    @mhart
    mhart
    on Jul 9, 2017
    ContributorAuthor

    There's an issue open on v8 too, if you'd like to star it: https://bugs.chromium.org/p/v8/issues/detail?id=6148

  21. justinjdickow commented on Jul 9, 2017

    @justinjdickow
    justinjdickow
    on Jul 9, 2017

    Can we rename the issue from "String length limit is small-ish" to "String length limit is small-af"

  22. mhart commented on Jul 24, 2017

    @mhart
    mhart
    on Jul 24, 2017
    ContributorAuthor

    Great news! v8 has upped the limit to ~1GB on 64-bit archs: https://chromium-review.googlesource.com/c/570047

    Hopefully it comes with a nicer error message when it fails too (although that's possibly still a Node.js responsibility?)

  23. joyeecheung
    closed this as completedin 289d152on Apr 7, 2018
  24. joyeecheung
    added a commit that references this issue on Apr 7, 2018

    src: migrate string_bytes.cc to throw errors with code

    63eb267

  25. joyeecheung commented on Apr 7, 2018

    @joyeecheung
    joyeecheung
    on Apr 7, 2018
    Member

    After 63eb267 this now returns a message Cannot create a string larger than 0x3fffffe7 bytes with the error code set to ERR_STRING_TOO_LARGE.

  26. soichih commented on Jan 7, 2019

    @soichih
    soichih
    on Jan 7, 2019

    I am running into this issue when I try to download .tar file via request and piping it to tar.x(). Does this issue imply that the maximum body size that I can download via request.get() is 1G for 64-bit machine?

  27. soichih commented on Jan 7, 2019

    @soichih
    soichih
    on Jan 7, 2019

    nvm.. I just had to set "encoding: null" ... which prevents the body to be passed to toString()

  28. sjarmak
    added a commit that references this issue on Jul 27, 2026

    src: add error code helpers to src/node_errors.h

    289d152
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bufferIssues and PRs related to the buffer subsystem.help wantedIssues that need assistance from volunteers or PRs that need help to proceed.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions