You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
create_subprocess_exec stdin writer does not write everything #312
Versions
This problem was already detected on my prod on linux.
platform darwin -- Python 3.7.4, pytest-5.3.5, py-1.8.1, pluggy-0.13.1
Platform:
linux/darwin
Can you reproduce the bug with PYTHONASYNCIODEBUG in env?:
yes
Does uvloop behave differently from vanilla asyncio? How?:
yes, test do not hang with pyloop.
When writing large amount of data in the stdin of a subprocess, we observe sometimes that the subprocess will hang trying to read from our app.
Trying to reproduce with a very small test, I find that, on macos, a limit of 8KiB.
Here is my test
# Standard Libraryimportasyncio.subprocessimportpytest@pytest.mark.parametrize("num_lines", range(8188, 8194,1))asyncdeftest_uvloopwrite(loop, num_lines):
proc=awaitasyncio.create_subprocess_exec("python", __file__,
stdout=asyncio.subprocess.PIPE,
stdin=asyncio.subprocess.PIPE)
data=b"\n"*num_lines+b"END\n"print(num_lines, len(data))
print("writing", proc.stdin.transport.get_write_buffer_limits())
proc.stdin.write(data)
print("drain")
awaitproc.stdin.drain()
# proc.stdin.close()print("waiting")
awaitproc.wait()
out=awaitproc.stdout.read()
assertint(out) ==num_lines# this small program is called by the testif__name__=="__main__":
importsysn=0whileTrue:
line=sys.stdin.readline()
ifn>8186:
print(n, repr(line), file=sys.stderr)
ifline=="END\n":
breakn+=1print(n)
Activity
1st1 commentedon Feb 11, 2020
Would appreciate you submitting a PR with a functional test (marked as x-fail) to demonstrate the problem.
tardyp commentedon Feb 12, 2020
done. actually I re-tested under linux, and found out a weird threshold of 219264 bytes
Add failing unit tests to reproduce issue MagicStack#312
Add failing unit tests to reproduce issue MagicStack#312
forcely add UV_HANDLE_READABLE on pipe_t
Add failing unit tests to reproduce issue MagicStack#312
Add failing unit tests to reproduce issue MagicStack#312
forcely add UV_HANDLE_READABLE on pipe_t
Add failing unit tests to reproduce issue MagicStack#312
forcely add UV_HANDLE_READABLE on pipe_t
Add failing unit tests to reproduce issue #312