Skip to content

create_subprocess_exec stdin writer does not write everything #312

Closed

Description

@tardyp
  • Python version:
    Darwin, python 3.7.4
# pipenv install pytest uvloop pytest-aiohttp
# pipenv run pip freeze
aiohttp==3.6.2
async-timeout==3.0.1
attrs==19.3.0
chardet==3.0.4
idna==2.8
importlib-metadata==1.5.0
more-itertools==8.2.0
multidict==4.7.4
packaging==20.1
pluggy==0.13.1
py==1.8.1
pyparsing==2.4.6
pytest==5.3.5
pytest-aiohttp==0.3.0
six==1.14.0
uvloop==0.14.0
wcwidth==0.1.8
yarl==1.4.2
zipp==2.2.0

Versions
This problem was already detected on my prod on linux.

platform darwin -- Python 3.7.4, pytest-5.3.5, py-1.8.1, pluggy-0.13.1

  • Platform:
    linux/darwin

  • Can you reproduce the bug with PYTHONASYNCIODEBUG in env?:
    yes

  • Does uvloop behave differently from vanilla asyncio? How?:

yes, test do not hang with pyloop.

When writing large amount of data in the stdin of a subprocess, we observe sometimes that the subprocess will hang trying to read from our app.

Trying to reproduce with a very small test, I find that, on macos, a limit of 8KiB.

Here is my test

# Standard Library
import asyncio.subprocess
import pytest

@pytest.mark.parametrize("num_lines", range(8188, 8194,1))
async def test_uvloopwrite(loop, num_lines):
    proc = await asyncio.create_subprocess_exec("python", __file__,
                                                        stdout=asyncio.subprocess.PIPE,
                                                        stdin=asyncio.subprocess.PIPE)
    data =  b"\n" * num_lines + b"END\n"
    print(num_lines, len(data))
    print("writing", proc.stdin.transport.get_write_buffer_limits())
    proc.stdin.write(data)
    print("drain")
    await proc.stdin.drain()
    # proc.stdin.close()
    print("waiting")
    await proc.wait()
    out = await proc.stdout.read()
    assert int(out) == num_lines

# this small program is called by the test
if __name__ == "__main__":
    import sys
    n = 0
    while True:
        line = sys.stdin.readline()
        if n > 8186:
            print(n, repr(line), file=sys.stderr)
        if line == "END\n":
            break
        n+=1
    print(n)

I run it with :

pipenv install pytest uvloop pytest-aiohttp
pipenv run pytest -s --aiohttp-loop=uvloop test_uvloop.py

When printing on the subprocess, I can see that the first 8191 bytes are seen by the subprocess, but the last one are never seen.

If I close the stdin (line commented in the test), it appears that it forces a buffer flush, and the test finishes.

Activity

  1. 1st1 commented on Feb 11, 2020

    @1st1
    Member

    Would appreciate you submitting a PR with a functional test (marked as x-fail) to demonstrate the problem.

  2. tardyp commented on Feb 12, 2020

    @tardyp
    ContributorAuthor

    done. actually I re-tested under linux, and found out a weird threshold of 219264 bytes

  3. added 5 commits that reference this issue on May 9, 2020
    ede1fb2
    39cd41f
    90b9e7f
    c6a0179
    88f9712
  4. added 2 commits that reference this issue on May 14, 2020
    5d41af8
    c392972
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions