buffer.toString fails for buffers larger than 192MB

buffer.toString with base64 encoding fails for any buffer larger than or equal to 201326581 bytes (~192 MB).

See the following piece of code:

var buf1 = new Buffer(201326580),  //biggest buffer that works
    buf2 = new Buffer(201326581);  //~192 MB
buf1.toString('base64'); //works
buf2.toString('base64'); //fails!

For older node versions (like v0.12.2) this cause the process to crash with this error:

#
# Fatal error in d:\jenkins\workspace\nodejs-msi-julien\d8c2e2bb\deps\v8\src/handles.h, line 48
# CHECK(location_ != NULL) failed
#
Segmentation fault

For newer versions (like v5.2.0), it gives the following error:

buffer.js:401
    throw new Error('toString failed');
    ^

Probably a dupe of #3175

Slightly more representative tests:

Buffer(268435440).toString('binary');  // ok
Buffer(268435441).toString('binary');  // crashes in v0.12

Nominally a V8 bug due to the use of .ToHandleChecked() in deps/v8/src/api.cc. We could work around it by checking that length < 268435441 before calling v8::String::NewExternal() in src/string_bytes.cc.

I'm not going to work on it but if someone wants to pick it up, I'll review the patches.

@bnoordhuis

@bnoordhuis
I'll work on it.

fix: workaround for possible V8 bug fixes nodejs#4266

Is v4 also affected?

Yes. but it doesn't crash the process.

Unable to read a 40MB xlsx due to this.
mongod 3.0.6
Node 0.12.7

@RohanGada

@RohanGada Mind opening another issue with test case we can look at? Your issue will probably go unnoticed within an already closed issue.

EDIT: This isn't a closed issue yet. Don't know why I thought so.

Okay

Is this fixed ?

#4394 has yet to land.

Curious to know if there's been any progress on this yet? I'm running v5.6.0 and still seeing this behavior.

@indutny

Best I can figure this is a v8 bug. Unfortunately my v8 foo isn't good enough to track this down. /cc @indutny want to track down a v8 crash? :)

I had to bypass node and use bash with jq to parse large base64 objects through bash.

This issue is a duplicate of #3175

@davidrapin

@davidrapin It's not duplicate.
This is about an unhanded exception which breaks the process.

Ok got it, just found out what the v0.12 tag meant ;)

Message: Workaround Buffer.toString failure

Closing this given that support for v0.12 is coming to a close in two days and there will not be any further updates to the v0.12 branch.

Message: Again workaround Buffer.toString failure

Navigation Menu

Repository navigation

buffer.toString fails for buffers larger than 192MB #4266

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

buffer.toString fails for buffers larger than 192MB #4266

Description

Activity

ChALkeR commented on Dec 14, 2015

bnoordhuis commented on Dec 15, 2015

skilledDeveloper commented on Dec 15, 2015

silverwind commented on Dec 15, 2015

skilledDeveloper commented on Dec 15, 2015

RohanGada commented on Dec 27, 2015

trevnorris commented on Dec 28, 2015

RohanGada commented on Dec 29, 2015

kingroryg commented on Jan 11, 2016

silverwind commented on Jan 11, 2016

wmbutler commented on May 31, 2016

trevnorris commented on Jun 1, 2016

wmbutler commented on Aug 19, 2016

davidrapin commented on Oct 6, 2016

skilledDeveloper commented on Oct 6, 2016

davidrapin commented on Oct 6, 2016

jasnell commented on Dec 29, 2016

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions