The io module doesn't support non-blocking files

@pitrou

BPO	13322
Nosy	@pitrou, @vstinner, @benjaminp, @jab, @vadmium, @jstasiak, @bharel, @izbyshev
Files	blockingioerror.py blockingioerror.py write_blockingioerror.patch write_blockingioerror.patch write_blockingioerror.patch write_blockingioerror.patch nonblock-none.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2011-11-02.15:08:55.126>
labels = ['type-bug', 'library', '3.9']
title = "The io module doesn't support non-blocking files"
updated_at = <Date 2020-08-16.18:51:47.700>
user = 'https://bugs.python.org/sbt'

bugs.python.org fields:

activity = <Date 2020-08-16.18:51:47.700>
actor = 'bar.harel'
assignee = 'docs@python'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2011-11-02.15:08:55.126>
creator = 'sbt'
dependencies = []
files = ['23590', '23598', '23613', '23628', '23693', '23724', '37995']
hgrepos = []
issue_num = 13322
keywords = ['patch']
message_count = 43.0
messages = ['146841', '146878', '146936', '146940', '146998', '147003', '147004', '147009', '147011', '147012', '147013', '147023', '147027', '147028', '147048', '147061', '147071', '147242', '147387', '147405', '147409', '147664', '147682', '147875', '147916', '147923', '148074', '151937', '235023', '235249', '235316', '239371', '307763', '307764', '307770', '307773', '334044', '354337', '354339', '354341', '354342', '354345', '375519']
nosy_count = 14.0
nosy_names = ['pitrou', 'vstinner', 'benjamin.peterson', 'stutzbach', 'jab', 'neologix', 'abacabadabacaba', 'docs@python', 'python-dev', 'sbt', 'martin.panter', 'jstasiak', 'bar.harel', 'izbyshev']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue13322'
versions = ['Python 3.9']

Linked PRs

gh-57531 add BufferedReader.read() return value check for non-blocking stream mode #105224

According to the the documentation, BufferedReader.read() and BufferedWriter.write() should raise io.BlockingIOError if the file is in non-blocking mode and the operation cannot succeed without blocking.

However, BufferedReader.read() returns None (which is what RawIOBase.read() is documented as doing), and BufferedWriter.write() raises IOError with a message like

raw write() returned invalid length -1 (should have been 
between 0 and 5904)

I tested this on Linux with Python 2.6, 2.7 and 3.x.

Attached is a unit test.

BufferedReader.readinto() should also raise BlockingIOError according to the docs. Updated unittest checks for that also.

BTW, The documentation for BufferedIOBase.read() says that BlockingIOError should be raised if nothing can be read in non-blocking mode. BufferedReader inherits from BufferedIOBase and overrides the read() method. This is the documentation for BufferedReader.read():

    read([n])
        Read and return n bytes, or if n is not given or negative, 
        until EOF or if the read call would block in non-blocking mode.

This sentence is complete gobbledygook, and it makes no mention of what should happen if nothing can be read in non-blocking mode. So I presume behaviour for BufferedReader.read() should match the documented behaviour for BufferedIOBase.read().

Wierdly, it looks like BlockingIO is not raised anywhere in the code for the C implementation of io.

Even more wierdly, in the Python implementation of io, BlockingIOError is only ever raised by except clauses which have already caught BlockingIOError. So, of course, these clauses are dead code.

The only code in CPython which can ever succesfully raise BlockingIOError is MockNonBlockWriterIO.write() in test/test_io.py.

I don't know what the correct behaviour is for flush() and close() if you get EAGAIN. I think flush() should raise an error rather than blocking, and that close() should delegate to self.raw.close() before raising the error.

The docs say that read(), readinto() and write() can raise BlockingIOError. But what should readall() and readline() do? Should we just try to emulate whatever Python's old libc IO system did (with BlockingIOError replacing IOError(EAGAIN))?

Wierdly, it looks like BlockingIO is not raised anywhere in the code
for the C implementation of io.

That would explain why it isn't raised :)

This is a hairy issue: read(n) is documented as returning either n bytes or nothing. But what if less than n bytes are available non-blocking? Currently we return a partial read. readline() behaviour is especially problematic:

>>> fcntl.fcntl(r, fcntl.F_SETFL, os.O_NDELAY)
0
>>> rf = open(r, mode='rb')
>>> os.write(w, b'xy')
2
>>> rf.read(3)
b'xy'
>>> os.write(w, b'xy')
2
>>> rf.readline()
b'xy'

We should probably raise BlockingIOError in these cases, but that complicates the implementation quite a bit: where do we buffer the partial data? The internal (fixed size) buffer might not be large enough.

write() is a bit simpler, since BlockingIOError has a "characters_written" attribute which is meant to inform you of the partial success: we can just reuse that. That said, BlockingIOError could grow a "partial_read" attribute containing the read result...

Of course, we may also question whether it's useful to use buffered I/O objects around non-blocking file descriptors; if you do non-blocking I/O, you generally want to be in control, which means not having any implicit buffer between you and the OS.

(this may be a topic for python-dev)

This is a hairy issue

Indeed.

Performing partial read/write may sound imperfect, but using buffered I/O around non-blockind FD is definitely not a good idea.
Also, the advantage of the current approach is that at least, no data is ever lost (and changing the behavior to raise a BlockingIOError might break some code out there in the wild).

Note that Java's BufferedInputStream and ReadableByteChannel also return partial reads.

So I'm somewhat inclined to keep the current behavior (it would however probably be a good idea to update the documentation to warn about this limitation, though).

Also, the advantage of the current approach is that at least, no data
is ever lost

But what about the buggy readline() behaviour?

Note that Java's BufferedInputStream and ReadableByteChannel also
return partial reads.

Apparently, they are specified to, even for blocking streams (which I find a bit weird, and the language in the docs seems deliberately vague). Python's buffered read(), though, is specified to return the requested number of bytes (unless EOF happens).

No one has suggested raising BlockingIOError and DISCARDING the data when a partial read has occurred. The docs seem to imply that the partially read data should be returned since they only say that BlockingIOError should be raised if there is NOTHING to read. Clearly this should all be spelt out properly.

That leaves the question of whether, when there is NOTHING to read, BlockingIOError should be raised (as the docs say) or None should be returned (as is done now). I don't mind either way as long as the docs match reality.

The part which really needs addressing is partial writes. Currently, if a write fails with EAGAIN then IOError is raised and there is no way to work out how much data was written/buffered. The docs say that BlockingIOError should be raised with the e.args[2] set to indicate the number of bytes written/buffered. This at least should be fixed.

I will work on a patch.

But what about the buggy readline() behaviour?

Just tell people that if the return value is a string which does not end in '\n' then it might caused by EOF or EAGAIN. They can just call readline() again to check which.

The third arg of BlockingIOError is used in two quite different ways.

In write(s) it indicates the number of bytes of s which have been "consumed" (ie written to the raw file or buffered).

But in flush() and flush_unlocked() (in _pyio) it indicates the number of bytes from the internal buffer which have been written to the raw file.

I think this explains the following comment in write():

            # We're full, so let's pre-flush the buffer
            try:
                self._flush_unlocked()
            except BlockingIOError as e:
                # We can't accept anything else.
                # XXX Why not just let the exception pass through?
                raise BlockingIOError(e.errno, e.strerror, 0)

I don't think flush() should try to tell us how many bytes were flushed: we only need to know whether we need to try again.

Apparently, they are specified to, even for blocking streams (which
I find a bit weird, and the language in the docs seems deliberately
vague).

"""
As an additional convenience, it attempts to read as many bytes as possible by repeatedly invoking the read method of the underlying stream. This iterated read continues until one of the following conditions becomes true:

The specified number of bytes have been read,
The read method of the underlying stream returns -1, indicating end-of-file, or
The available method of the underlying stream returns zero, indicating that further input requests would block.
"""

As I understand it, it will return the number of bytes asked, unless EOF or EAGAIN/EWOULDBLOCK. It would seem reasonable to me to add the same note for non-blocking FDs to Python's read().

> But what about the buggy readline() behaviour?
Just tell people that if the return value is a string which does not
end in '\n' then it might caused by EOF or EAGAIN. They can just call
readline() again to check which.

Sounds reasonable.

No one has suggested raising BlockingIOError and DISCARDING the data
when a partial read has occurred.

The problem is that if we raise BlockingIOError, we can only buffer a limited amount of data.

The docs seem to imply that the partially read data should be returned
since they only say that BlockingIOError should be raised if there is
NOTHING to read. Clearly this should all be spelt out properly.

Agreed.

That leaves the question of whether, when there is NOTHING to
read, BlockingIOError should be raised (as the docs say) or None
should be returned (as is done now).

I don't have a string feeling: if we don't raise BlockingIOError on partial reads, then it probably makes sense to keep None.

Currently a BlockingIOError exception raised by flush() sets
characters_written to the number of bytes fushed from the internal
buffer. This is undocument (although there is a unit test which tests
for it) and causes confusion because characters_written has conflicting
meanings depending on whether the exception was raised by flush() or
write(). I would propose setting characters_written to zero on
BlockingIOError exceptions raised by flush(). Are there any reasons not
to make this change?

Also, the docs say that the raw file wrapped by
BufferedReader/BufferedWriter should implement RawIOBase. This means
that self.raw.write() should return None instead of raising
BlockingIOError. But the implementation tries to cope with
BlockingIOError coming from a raw write. In fact, the
MockNonBlockWriterIO class in unit tests is used as a raw file, but its
write() method raises BlockingIOError.

It would simplify matters a lot to insist that raw files implement
RawIOBase properly.

BTW, when I try to change characters_written of an existing
BlockingIOError exception using the pointer returned by
_buffered_check_blocking_error(), it appears not to work: the exception
continues to have characters_written == 0 -- not sure why...

I encountered the same problem

process = subprocess.Popen(["bash"], stdin=subprocess.PIPE, stdout=subprocess.PIPE, text=True)
os.set_blocking(process.stdout.fileno(), False)
process.stdout.read()

This leads to

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<frozen codecs>", line 321, in decode
TypeError: can't concat NoneType to bytes

It seems that this issue is more complicated than I imagined, but In any case, a read() operation should never raise a TypeError under any circumstances.

I think the crux of the issue lies in whether TextIOWrapper is a pure wrapper or a class with wrapping capabilities.

If it is a pure wrapper, it should only add decode/encode capabilities to the read/write functions. When encountering other situations, it should return an empty bytes (which should decode to an empty str) or None, or it should raise an error wrapper that returns/raises the output of rawbuffer unchanged, in order to maintain the same behavior as rawbuffer.

If it is a class with wrapping capabilities, it should have a consistent behavior, appropriately handling the return/raise of the raw buffer at EOF to maintain consistency in behavior.

I lean towards the class with wrapping capabilities option, as it reduces the hassle for downstream developers. However, for compatibility reasons, the pure wrapper option is also an enticing choice; it is harmless because it does not change the original behavior of TextIOWrapper. It merely chooses to return None in situations that would originally cause a TypeError. For existing code, this might just postpone the occurrence of TypeError, but it enables subsequent developers to handle situations where TypeError arises appropriately (rather than having to use try: ... expect TypeError: ...). This will not affect legacy code if they originally had no issues with TypeError.

#122933 (to fix gh-109523) adds more documentation around behavior + raise a BlockingIOError rather than TypeError for TextIO. Discuss thread around the change: https://discuss.python.org/t/handling-sys-stdin-read-in-non-blocking-mode/59633. That should help more cases here (open().read() which opens in buffered text mode by default, will raise a clearer exception)

#122933 (to fix gh-109523) adds more documentation around behavior + raise a BlockingIOError rather than TypeError for TextIO.

No doubt that BlockingIOError is more appropriate than TypeError, but this PR manifests that read() raises when there are no bytes available, whereas read(42) currently returns "". Also, readline() returns "".

Isn't this even more confusing than now?

A less invasive change would be to make them all return "", including read(). While this is consistent, it has the clear downside that the user has no way at all to distinguish EOF from "would block"...

I know this is a tricky issue with a history. If there's agreement that BlockingIOError is the way to go, then the behavior of read(n) and readline() should also be reevaluated. All of this has been discussed above:

If read(n) is changed to raise, then where to buffer the up to n-1 bytes internally? They could be added to the exception, but that's rather inconvenient for the user. The purpose is read(n) is typically to read into some kind of buffer of size at most n, and then it's okay to return a partial result. So it's probably best to keep the behavior?
Changing readline() is also difficult. What should it do with partial lines? (The io module doesn't support non-blocking files #57531 (comment)). Again, internal buffering is a bad idea. readline() could raise if there are no bytes available (not even a partial line) but that's not much better.

So perhaps the current behavior is the best what can be achieved without adding an entirely new API. The only caveat then is that read(n) will not raise while read() will raise.

In any case, and I think this is the lowest-hanging fruit: the docs can still be improved a lot. See #57531 (comment) for a nice overview.

The text added in gh-109523: Reading text from a non-blocking stream with read may now raise a BlockingIOError if the operation cannot immediately return bytes. #122933 can be improved, and for example, explain that read(42) will not raise but read() will raise. This behavior is specific to wrapping stdin or stdout/stderr of a subprocess, but I assume this is the most common use case, and so it's worth mentioning that specific case in the docs.
readline() should be documented to be able to return partial lines in the case of non-blocking streams.

Sorry, missed the notification for this comment.

I don't think there's an agreement that BlockingIOError is absolutely the way to go forward, just that it's a better exception if you accidentally end up in the stack with a blocking read than the previous "TypeError" without breaking existing code / having a global "this is what should do" + tests. In general the text case is not well tested or well defined so it's largely "The code does something". Tests and discussion to validate and help figure out the right consistent behavior would be really good.

For .read the docs of BufferedReader which is (usually) underneath TextIOWrapper didn't match the implementation (resolved by GH-130653). There is a similar issue in the FileIO and potentially RawIOBase docs which I plan to work on shortly so that the docs match implementation behavior. TextIOWrapper.buffer is sometimes a RawIOBase and sometimes a BufferedIOBase (see also: #85045)

With these there are two different cases which need to be handled:

"Blocking I/O" / EWOULDBLOCK
BufferedIOBase vs. RawIOBase retry behavior (Buffered tries to read/write "As much as posible" while RawIO matches POSIX read and write / "caller needs to retry"

An additional complexity here is that TextIOWrapper relies on the GIL and (in free threaded) critical sections to synchronize behavior, so need to take particular care around its members (ex. calls to buffer.read and/or buffer.write release the GIL), and the behavior is relied upon by existing codebases (ex: gh-119506, gh-118138).

For the GIL behavior I'd like (maybe 3.15, probably 3.16) to rework that at a bigger scale how the locking works in TextIOWrapper. In the shorter term it might make sense to use a more localized lock (see: https://discuss.python.org/t/bug-with-multi-threaded-print-write-causing-dropped-output-output-of-uninitialized-memory-assertion-failures/54538/4). In 3.15 I'm hoping to change how it works for BufferedIO and not certain I'll have time/bandwidth to also rework TextIOWrapper. Both are needed for a longer term goal of mine is to get print() to work better in multi-threaded code.

aside: I'm currently looking at the TextIOWrapper.write part of "Blocking I/O support" of this for gh-85393. So hopefully 3.15 will have improved behavior.

also: Lots of docs and history, but if you read through / look at it should be able to make progress :). Mainly linking here because I've been diving into a lot of this code and it's nice having all the disparate threads in one place to look back at.

pythongh-85393: Handle partial .write in TextIOWrapper

Fun fact: non-blocking text streams can raise spurious decode errors if less than a whole character is available in the stream. This can happen in Popen, for example, if the sender is buffering 8kB chunks and flushes the first chunk but pauses before flushing the second chunk and there happens to be a multi-byte character straddling that 8kB boundary.

It's technically a bug but I don't think there will be any interest from anyone unless they're already diving deep into this issue.

Navigation Menu

Repository navigation

The io module doesn't support non-blocking files #57531

Linked PRs

46 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

The io module doesn't support non-blocking files #57531

Description

Linked PRs

Activity

sbt commented on Nov 2, 2011

sbt commented on Nov 2, 2011

sbt commented on Nov 3, 2011

pitrou commented on Nov 3, 2011

neologix commented on Nov 4, 2011

pitrou commented on Nov 4, 2011

pitrou commented on Nov 4, 2011

sbt commented on Nov 4, 2011

sbt commented on Nov 4, 2011

sbt commented on Nov 4, 2011

neologix commented on Nov 4, 2011

sbt commented on Nov 4, 2011

46 remaining items

u-tung commented on Sep 13, 2024

cmaloney commented on Sep 14, 2024

real-or-random commented on Apr 23, 2025

cmaloney commented on May 21, 2025

sh1boot commented on Mar 24, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions