Skip to content

Split large write buffers into chunks libuv accepts - #793

Open
jensbjorgensen wants to merge 1 commit into
MagicStack:masterfrom
jensbjorgensen:stream_write_nbufs
Open

jensbjorgensen wants to merge 1 commit into
MagicStack:masterfrom
jensbjorgensen:stream_write_nbufs

Conversation

@jensbjorgensen

Copy link
Copy Markdown
Contributor

Summary

libuv 1.52.0 added uv__check_before_write() (libuv/libuv#5014), which makes
uv_write() and uv_try_write() return UV_EINVAL when a request carries
more than 1024*1024 buffers. uvloop 0.23.0 picked that up with the libuv
1.52.1 upgrade (#753).

UVStream appends every transport.write() to self._buffer while libuv
still has pending data for the stream, and _exec_write() then hands the
whole list to libuv as a single request. Any code that issues more than 1M
write() calls in one loop iteration, or one writelines() with more than
1M items, now gets the transport closed with OSError(22, 'Invalid argument')
instead of the data being sent. uvloop 0.22.1 (libuv 1.48.0) and vanilla
asyncio, which coalesces into a bytearray, both handle this fine.

We hit this in production on a server that streams a large snapshot to a
newly connected client as a few million small write() calls from one
callback; every snapshot died with EINVAL and the client reconnect loop
pegged a CPU.

Fix

_exec_write() now hands the buffered data to libuv in chunks of at most
1024*1024 buffers, one uv_write() request per chunk, and leaves the last
chunk to the existing code path. Writes under the limit are unaffected.

Reproducer

import asyncio
import uvloop


async def main():
    async def handler(reader, writer):
        while await reader.read(1 << 20):
            pass

    server = await asyncio.start_server(handler, '127.0.0.1', 0)
    reader, writer = await asyncio.open_connection(
        *server.sockets[0].getsockname())

    writer.writelines([b'x'] * (1024 * 1024 + 1))
    await writer.drain()
    print('ok')


uvloop.run(main())

On uvloop 0.23.0 the transport is closed with OSError(22, 'Invalid argument') and drain() raises ConnectionResetError: Connection lost.
With this change it prints ok. Same result with PYTHONASYNCIODEBUG=1.

Changes

- uvloop/handles/stream.pyx, stream.pxd: add __MAX_WRITE_BUFS and
  UVStream._exec_write_chunks(), called from _exec_write() when
  len(self._buffer) exceeds the limit.
- tests/test_tcp.py: test_many_queued_writes covers both the empty-queue
  path (writelines() of 1M+1 buffers on a fresh connection) and the
  queued path (1M+1 write() calls while libuv has data pending). It fails
  on master and passes with this change, on normal and UVLOOP_DEBUG builds.

Tested on Linux with 3.14 against libuv 1.53.0.

libuv >= 1.52 rejects uv_write()/uv_try_write() requests carrying more
than 1024*1024 buffers with UV_EINVAL.  A transport that keeps receiving
write() calls while libuv still has pending data accumulates every
buffer in UVStream._buffer and flushed them in a single request, so a
burst of more than 1M small writes in one loop iteration killed the
connection with OSError(22, 'Invalid argument').

Hand the buffered writes to libuv in chunks of at most 1024*1024
buffers, one uv_write() request each.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant