Repository navigation
asyncio signal handler receives signals from child processes #94454
Description
Activity
% ~/.pyenv/versions/3.10.4/bin/python asyncio_signal_bug.py 2>/dev/null 42747 Signal Handler set 42749 Worker 1 started 42750 Worker 2 started ^C42747 received signal: Signals.SIGINT ^C42747 received signal: Signals.SIGINT ^C42747 received signal: Signals.SIGINT ^C42747 received signal: Signals.SIGINT ^C42747 received signal: Signals.SIGINT ^C42747 received signal: Signals.SIGINT % ~/.pyenv/versions/3.10.4/bin/python asyncio_signal_bug.py --set-in-children 2>/dev/null 42755 Signal Handler set 42757 Worker 1 started 42758 Worker 2 started ^C42755 received signal: Signals.SIGINT ^C42755 received signal: Signals.SIGINT ^C42755 received signal: Signals.SIGINT ^C42755 received signal: Signals.SIGINT ^C42755 received signal: Signals.SIGINT ^C42755 received signal: Signals.SIGINTOn Mac it runs the same as per the docs.
Python signal handlers are always executed in the main Python thread of the main interpreter, even if the signal was received in another thread. This means that signals can’t be used as a means of inter-thread communication docs
Let me know if I have left something out while testing.I just tried on a different system with Python 3.6.9 on Ubuntu 18.04.6 LTS with Kernel 4.15.0-188 and could confirm the buggy behaviour (after adjusting for some asyncio API naming changes).
I expect, that it has either something to do with Mac handling
signalfd(the underlying syscall) slightly different or how your terminal issues SIGINT to child processes. I used Bash in my experiments, which obviously sends a SIGINT to the entire process group.Could you please try to manually send a signal to one of the worker processes with
kill -2 <worker PID>?For me this is also received in the parent process.
Also, please note that this issue does not concern an inter-thread, but an inter-process scenario.
Tried with bash 5.1. Also tried sending manual signal. The behaviour is the same ( see complete trace below without redirection to null ) with or without set children argument.
when manual kill signal is given the traceback does not spill on to terminal, there is no Process SpawnProcess-1/2 .
Maybe its difference with how Mac m1 and Linux kernels handle.bash-5.1$ ~/.pyenv/versions/3.10.4/bin/python asyncio_signal_bug.py 42834 Signal Handler set 42836 Worker 1 started 42837 Worker 2 started ^CProcess SpawnProcess-1: Process SpawnProcess-2: 42834 received signal: Signals.SIGINT Traceback (most recent call last): File "/Users/user1/.pyenv/versions/3.10.4/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/Users/user1/.pyenv/versions/3.10.4/lib/python3.10/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/Users/user1/.pyenv/versions/3.10.4/lib/python3.10/concurrent/futures/process.py", line 237, in _process_worker call_item = call_queue.get(block=True) File "/Users/user1/.pyenv/versions/3.10.4/lib/python3.10/multiprocessing/queues.py", line 102, in get with self._rlock: File "/Users/user1/.pyenv/versions/3.10.4/lib/python3.10/multiprocessing/synchronize.py", line 95, in __enter__ return self._semlock.__enter__() KeyboardInterrupt Traceback (most recent call last): File "/Users/user1/.pyenv/versions/3.10.4/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/Users/user1/.pyenv/versions/3.10.4/lib/python3.10/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/Users/user1/.pyenv/versions/3.10.4/lib/python3.10/concurrent/futures/process.py", line 237, in _process_worker call_item = call_queue.get(block=True) File "/Users/user1/.pyenv/versions/3.10.4/lib/python3.10/multiprocessing/queues.py", line 103, in get res = self._recv_bytes() File "/Users/user1/.pyenv/versions/3.10.4/lib/python3.10/multiprocessing/connection.py", line 221, in recv_bytes buf = self._recv_bytes(maxlength) File "/Users/user1/.pyenv/versions/3.10.4/lib/python3.10/multiprocessing/connection.py", line 419, in _recv_bytes buf = self._recv(4) File "/Users/user1/.pyenv/versions/3.10.4/lib/python3.10/multiprocessing/connection.py", line 384, in _recv chunk = read(handle, remaining) KeyboardInterrupt ^C42834 received signal: Signals.SIGINT ^C42834 received signal: Signals.SIGINT ^C42834 received signal: Signals.SIGINT ^C42834 received signal: Signals.SIGINT ^C42834 received signal: Signals.SIGINTI think I just found the reason:
https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods
spawn
The parent process starts a fresh Python interpreter process. The child process will only inherit those resources necessary to run the process object’s run() method. In particular, unnecessary file descriptors and handles from the parent process will not be inherited. Starting a process using this method is rather slow compared to using fork or forkserver.
Available on Unix and Windows. The default on Windows and macOS.fork
The parent process uses os.fork() to fork the Python interpreter. The child process, when it begins, is effectively identical to the parent process. All resources of the parent are inherited by the child process. Note that safely forking a multithreaded process is problematic.
Available on Unix only. The default on Unix.If a different process start method is used, this of course changes the behaviour!
Can you try adding:
import multiprocessing as mp
and in main:
mp.set_start_method('fork')When I set it to 'spawn' for me, I saw exectly the same behaviour you see.
True , spawn vs fork is the issue. I added fork as multiprocessing start method as you mentioned and the result is exactly what you found in beginning. But I am not sure why traceback is spilling over during spawn.
bash-5.1$ ~/.pyenv/versions/3.10.4/bin/python asyncio_signal_bug.py 42906 Signal Handler set 42907 Worker 1 started 42908 Worker 2 started ^C42906 received signal: Signals.SIGINT 42906 received signal: Signals.SIGINT 42906 received signal: Signals.SIGINT ^C42906 received signal: Signals.SIGINT 42906 received signal: Signals.SIGINT 42906 received signal: Signals.SIGINT bash-5.1$ ~/.pyenv/versions/3.10.4/bin/python asyncio_signal_bug.py --set-in-children 42909 Signal Handler set 42910 Worker 1 started 42911 Worker 2 started ^C42909 received signal: Signals.SIGINT ^C42909 received signal: Signals.SIGINT ^C42909 received signal: Signals.SIGINT ^C42909 received signal: Signals.SIGINT ^C42909 received signal: Signals.SIGINT ^C42909 received signal: Signals.SIGINTBut I am not sure why traceback is spilling over during spawn.
I guess, the worker processes terminate and thus print the interrupt output.
You should be able to check, if they are still running after the first ^C.
Workers are terminating and interrupt is not caught on first press of ctrl-c.
Looks like there is Base Exception asyncio issue with handling KeyboardInterruptWorkers are terminating and interrupt is not caught on first press of ctrl-c. Looks like there is Base Exception asyncio issue with handling KeyboardInterrupt
I'm not sure, but I think this is completely unrelated to the
signalfdissue I'm referring to?As a fix my issue, I would suggest adding something along the lines of:
if hasattr(os, 'register_at_fork'): def close_wakeup_fd(): signal.set_wakeup_fd(-1) self._signal_handlers.clear() os.register_at_fork(after_in_child=close_wakeup_fd)To: https://github.com/python/cpython/blob/main/Lib/asyncio/unix_events.py _UnixSelectorEventLoop.__init__
And I just noticed, that there may be other issues regarding the sockets
_ssockand_csock(from https://github.com/python/cpython/blob/7db1d2eaf367a1073191c80c7baeee41ae1f2f21/Lib/asyncio/selector_events.py BaseSelectorEventLoop) in the fork case:If a worker uses asyncio itself and they use the same socketpair for their internal wakeup, it might get messy!
asyncioandforkdon't work together. This seems to be a duplicate of #66285.I was able to reproduce the issue using the code below both on Ubuntu 22.04 and MacOS 12.6 Intel using Python 3.10.5. Sending
SIGTERMto the child also terminates the parent. Addingsignal.set_wakeup_fd(-1)to the child or setting the multiprocessing start method to spawn fix the problem.@kumaraditya303 I was reading the issue you mentioned but it doesn't look like a duplicate because the child isn't attempting to use asyncio's event loop.
import asyncio import multiprocessing import os import signal import time def child_signal_handler(signum, _frame): print(f"Child signal handler received signal {signum}") raise SystemExit(0) def child_main(): signal.signal(signal.SIGTERM, child_signal_handler) print(f"Child started, PID: {os.getpid()}") while True: time.sleep(1) def parent_signal_handler(signum): print(f"Parent signal handler received signal {signum}") raise SystemExit(0) async def parent_main(): loop = asyncio.get_running_loop() loop.add_signal_handler(signal.SIGTERM, lambda s=signal.SIGTERM: parent_signal_handler(s)) multiprocessing.set_start_method("fork") process = multiprocessing.Process(target=child_main) process.start() while True: await asyncio.sleep(2) os.kill(process.pid, signal.SIGTERM) if __name__ == "__main__": asyncio.run(parent_main())
I was reading the issue you mentioned but it doesn't look like a duplicate because the child isn't attempting to use asyncio's event loop.
The child is using the wakeup fd of the parent process since
asynciodoes not adds fork handlers to change the wakeup fd.
I was correct asasynciodoes not supportsfork, the child continues to use parent's wakeup fd.The OP isn't trying to use fork(), nor are they using asyncio in the handlers. But they are using a
ProcessPoolExecutorand the forked subprocess inherits some signal settings. It seems sensible to implement the OP's fix from #94454 (comment).The OP isn't trying to use fork(),
Multiprocessing fork context uses
forkunder the hood.It seems sensible to implement the OP's fix from #94454 (comment).
That could be done although there could be edge cases.
forkdoes not plays well with signal handlers and child watchers etc.TBH I don't know anything about multiprocessing and I'm not about to volunteer to become an expert there. I don't know if the responsibility would lay with asyncio or with multiprocessing. But one of these will have to accommodate the other, somehow. The OP's fix seems the simplest thing that could possibly work, so let's do that in 3.12 and see how it goes. (The worst case outcome is that there are still some cases that remain broken, for which we can iterate if someone reports an example.)
Bug report
Today I ran into a very weird behaviour in one of my projects, that uses asyncio and multiprocessing.
When I interrupted the program with a Ctrl-C on the terminal, the signal handler set in asyncio with
loop.add_signal_handlerwas triggered multiple times! Further inspection showed, that if I manually send aSIGINTto one of the worker child processes, it triggered the main process signal handler.I could trace the issue to the socket used by asyncio to receive the signals via
signal.set_wakeup_fd, which seems to receive the signal bytes also from the child processes.Running
signal.set_wakeup_fd(-1)at least once in the worker processes worked as a workaround for me but cannot be the final solution.Also, it seems, that other signal operations also can clear the issue (for example some automatically manager child processes did not trigger the bug, as they seem to ignore
SIGINT).In my project I used
concurrent.futures.ProcessPoolExecutor, but I don't think that matters, as long as the signal handler (and wakeup fd) is set before the child is started.Your environment
Python 3.10.4
Kali Linux rolling with kernel: Debian 5.16.18-1kali1 (2022-04-01) x86_64
Bug output
asyncio_signal_bug.py.txt
Notice, that in the first block, for every ^C, there are three signals received, whereas in the second block (if
signal.set_wakeup_fd(-1)was called in the workers) there is only one.