Repository navigation
Faster Transform stream #33397
Description
Some thoughts on how to make Transform streams faster.
Part of the overhead of Transform (and PassThrough) is that it is actually 2 streams, one Writable and one Readable, both with buffering and state management, which are connected together.
We could try to skip this and implement Transform as a Readable which implements the Writable interface and proxies the naming.
e.g.
class FastTransform extends Readable {
constructor(options) {
super(options)
this._writableState = {
length: 0,
needDrain: false,
ended: false,
finished: false
}
}
get writableEnded () {
return this._writableState.ended
}
get writableFinished () {
return this._writableState.finished
}
_read () {
const rState = this._readableState
const wState = this._writableState
if (!wState.needDrain) {
return
}
if (wState.length + rState.length > rState.highWaterMark) {
return
}
wState.needDrain = false
this.emit('drain')
}
write (chunk) {
const rState = this._readableState
const wState = this._writableState
const len = chunk.length
wState.length += len
this._transform(chunk, null, (err, data) => {
wState.length -= len
if (err) {
this.destroy(err)
} else if (data != null) {
this.push(data)
}
this._read()
})
wState.needDrain = wState.length + rState.length > rState.highWaterMark
return wState.needDrain
}
end () {
const wState = this._writableState
wState.ended = true
if (this._flush) {
this._flush(chunk, (err, data) => {
const wState = this._writableState
if (err) {
this.destroy(err)
} else {
if (data != null) {
this.push(data)
}
this.push(null)
wState.finished = true
this.emit('finish')
}
})
} else {
this.push(null)
wState.finished = true
this.emit('finish')
}
}
}
// TODO: Make Writable[Symbol.hasInstance] recognize `FastTransform`.Making this fully backwards compatible with Transform might be difficult.
Activity
ronag commentedon May 14, 2020
Thoughts @nodejs/streams?
mcollina commentedon May 14, 2020
How much will we gain from this? I'm +1, possibly with a better name.
ronag commentedon May 14, 2020
@mcollina: ~60%
FasterTransform? 😄targos commentedon May 14, 2020
How does it compare to transforming with an async generator function?
ronag commentedon May 14, 2020
So using an async generator in
pipelinemight be significantly faster. Is it worth further optimizingTransform? Note that an optimization as suggested here does make it harder to maintain than today's rather trivial implementation.ronag commentedon May 17, 2020
pipelinewill convert aReadablesource into an async iterator and read from it that way. Might introduce unnecessary overhead. Will investigate that first.clshortfuse commentedon Jun 16, 2020
Apparently I should be using
pipeline. This does seem like a good replacement forPassThrough.