Repository navigation
Python launcher invoked with command "py" fails to start process #99442
Description
Activity
This error causes a failure to build MSI installer between version 3.7 and 3.10. (Example of GitHub Actions build failure in my personal project).
This is because WiX invokes
"py" -m pip install -U blurbduring build forexe.wixprojand recent GitHub-Hosted runner uses Python Launcher of version 3.11.0.Confirmed, I am seeing the same issue (see the linked rust issue report above).
Command: "py" "-V" argv0: py version: 3.11.0 SearchInfo.originalCmdLine: "py" -V SearchInfo.restOfCmdLine: -V SearchInfo.executablePath: (null) SearchInfo.scriptFile: (null) SearchInfo.executable: py" SearchInfo.executableArgs: (null) SearchInfo.company: (null) SearchInfo.tag: (null) SearchInfo.oldStyleTag: False SearchInfo.lowPriorityTag: False SearchInfo.allowDefaults: False SearchInfo.allowExecutableOverride: False SearchInfo.windowed: False SearchInfo.list: False SearchInfo.listPaths: False SearchInfo.help: False -V:3.11 C:\Users\Gustav\AppData\Local\Programs\Python\Python311\python.exe -V:3.10 C:\Users\Gustav\AppData\Local\Programs\Python\Python310\python.exe -V:3.9 C:\Users\Gustav\AppData\Local\Programs\Python\Python39\python.exe -V:3.8 C:\Users\Gustav\AppData\Local\Programs\Python\Python38\python.exe env.company: PythonCore env.tag: 3.11 # about to run: C:\Users\Gustav\AppData\Local\Programs\Python\Python311\py" -V Unable to create process using 'C:\Users\Gustav\AppData\Local\Programs\Python\Python311\py" -V': The system cannot find the file specified.In
parseCommandLine(), the value ofendpoints to just past the closing double quote if there's no "." found in the tail component.findArgumentEnd()could be modified to exclude the double quote from the end, since the double quote at the end isn't actually part of the command-line argument. It's a syntax character for argument parsing.Lines 565 to 580 in 88385b8
A couple of issues in
findArgumentLength(), which could also be resolved here without creating a new issue for them:- If the buffer doesn't start with a double quote, it needs to search for white space -- at least tab ("\t") in addition to space (" ").
- Since
findArgumentLength()is used byfindArgumentEnd(), which is only ever called to parse the first argument of the command line, it should not implement backslash escaping of double quotes. When parsing the first argument of the command line, bothCreateProcessW()andCommandLineToArgvW()enter quote mode if the first character is a double quote, in which case the argument ends at the next double quote, regardless of whether it's followed by whitespace. That's a bit simpler than the C runtime, but even the C runtime doesn't implement escaping of double quotes in the first argument.
The fix appears to have addressed part of my issue, but I'm still getting the same error in a slightly different case. I haven't yet been able to reduce the test case to something smaller, but if I use the
justcommand line utility, with ajustfileas follows:set windows-shell := ["pwsh.exe", "-NoLogo", "-Command"] test_py: #!py print(12)then running
just test_pygives the error:Unable to create process using 'C:\Users\Gustav\AppData\Local\Temp\justc2pOnD\py C:\Users\Gustav\AppData\Local\Temp\justc2pOnD\test_py': The system cannot find the file specified.If I add some debugging prints, I can see that the command being run was
"py" "C:\Users\Gustav\AppData\Local\Temp\justc2pOnD\test_py".Thanks for the quick extra test! This one looks like it's probably shebang parsing - can you include the whole output you get with PYLAUNCHER_DEBUG set?
Oh, this is probably the deliberate change of behaviour to make shebang lines only search PATH if you use
/usr/bin/envin them.If it falls through to the default case, we don't let you search for arbitrary executables. It will only look adjacent to the script file. The old behaviour was never documented, and isn't safe (the new behaviour is only mildly safer, but at least you can scan for
usr/bin/envand recognise when it's going to happen, rather than trying to prove a negative). The new documentation reads:Shebang lines that do not match any of these patterns are treated as Windows paths that are absolute or relative to the directory containing the script file. This is a convenience for Windows-only scripts, such as those generated by an installer, since the behavior is not compatible with Unix-style shells. These paths may be quoted, and may include multiple arguments, after which the path to the script and any additional arguments will be appended.
- added a commit that references this issue
on Nov 18, 2022 #!pyis a strange shebang, asking the launcher to recursively run itself. There should be a check in_useShebangAsExecutable()to prevent recursively running the launcher, as you did for the "#!/usr/bin/env" case insearchPath(). Or modifycheckShebang()to always check for this. Otherwise#!pywill blow up if the launcher is named "py.exe", and the script is beside it.11 remaining items
Unfortunately, other people demanded that
usr/bin/env python3should prioritise the PATH search over the version number, so we can't have perfect compatibility that way. But perhaps all thesepy -3cases would be quite happy with apython3on PATH override?if it's run directly with
./foo.pythen I just wanted to make sure it failed early if you only had python 2 available (the script is old enought that that was a concern at the time 😛).Note that if CMD or PowerShell is used instead of bash, then directly running a .py script will execute the associated command-line template for .py files, which is usually the "py.exe" launcher. If so, the launcher's implementation of "#!/usr/bin/env py" will search for "py.exe" in
PATH, find itself, and then opt to ignore the shebang and run the default Python. In most cases that should be 3.x, unless the user has configured 2.x as the default viaPY_PYTHONor "py.ini".Reacted by Steve DowerAh, so probably
#!/usr/bin/env py -3is what I'm after thenI just checked this using the MSYS2 version of bash and env. There's a snag. bash passes "py -3" as a single argument to "/usr/bin/env", which in turn requires the option "-S" to split this string into multiple command-line arguments. The launcher's implementation could support "#!/usr/bin/env -S", basically by just consuming the "-S", but currently it doesn't.
There's a limit on how much POSIX emulation we want to support, and I think
-Sis going too far. Especially sincepy -3isn't portable anyway -/usr/bin/env python3is the portable one, with/usr/bin/python3and/usr/local/bin/python3as valid though apparently less portable alternatives (and any of those can have a full tag specified on the end, so/usr/bin/python3.13-arm64if you really want).Any other paths not matching those templates can be added to the
[commands]section inpy.inialongside thepy.exeor in the user's%LocalAppData%\py.ini(which is not strictly the right place to put it, but historical reasons win). We don't need to have them all built in.Unfortunately, other people demanded that
usr/bin/env python3should prioritise the PATH search over the version number, so we can't have perfect compatibility that way. But perhaps all thesepy -3cases would be quite happy with apython3on PATH override?The launcher's "env" implementation falls back on a virtual "python[X[.Y[.exe]]]" command if the executable name can't be found on
PATH. For example:argv0: C:\Windows\py.exe version: 3.12.2 # Read 57 bytes from C:\Temp\test.py to find shebang line Shebang: /usr/bin/env python3 # Did not find python3.exe on PATH # Reading from C:\Users\user\AppData\Local\py.ini for commands/python3 # Did not find file C:\Users\user\AppData\Local\py.ini # Reading from C:\Windows\py.ini for commands/python3 # Did not find file C:\Windows\py.ini # Treating shebang command '3' as 'py -3'but may as well reopen to further add a special case for using py instead of the more portable #!/usr/bin/python3 (which would be the equivalent of #!py -3)
"py[.exe] [-X[.Y]]" could be reserved as a virtual command like "python[.X[.Y[.exe]]]". Then handling "#!/usr/bin/env py [-X[.Y]]" could default to a virtual command if either "py" isn't found or if the launcher finds itself.
And just for reference, the reason I'm using
py(orpy -3) instead ofpython3is that that's all I have on my PATH:$ which python which: no python in ... $ which python3 which: no python3 in ...Note that if CMD or PowerShell is used instead of bash, then directly running a .py script will execute the associated command-line template for .py files, which is usually the "py.exe" launcher. If so, the launcher's implementation of "#!/usr/bin/env py" will search for "py.exe" in
PATH, find itself, and then opt to ignore the shebang and run the default Python. In most cases that should be 3.x, unless the user has configured 2.x as the default viaPY_PYTHONor "py.ini".That's a good point. So perhaps the logic (if finding itself in the shebang line) needs to be:
- If no selection criteria were used by the current invocation of
pythen parse the shebang line for selection criteria (eg-3or whatever). - If the user specified any selection criteria then ignore the shebang line and go with what the user selected.
Or maybe the second case should merge the criteria and flag an error if there's a conflict - that is, if the user specifies
py -3 script.pyand the script says#!py -3.8then it should probably pick 3.8 rather than 3.11. Maybe with a--ignore-shebangoption or something if you really insist on running it withpy -2. (I'm assuming this will add complication to the implementation though, as it presumably doesn't have any way to combine criteria currently?)- If no selection criteria were used by the current invocation of
There's a limit on how much POSIX emulation we want to support, and I think
-Sis going too far. Especially sincepy -3isn't portable anyway -/usr/bin/env python3is the portable one, with/usr/bin/python3and/usr/local/bin/python3as valid though apparently less portable alternatives (and any of those can have a full tag specified on the end, so/usr/bin/python3.13-arm64if you really want).But even
/usr/bin/env python3isn't portable to bash on Windows, by default at least, since I don't think Python adds itself to PATH by default (at least my setup doesn't have it, and I don't think I'd have explicitly unset it). So you'd have to explicitly add it (either in Windows - for which I think there's an option in the installer - or in your.bashrcor something) to make that work there. I guess you could also add a script on your bash PATH calledpython3that forwards its arguments topy -3, but again, it won't happen out of the box.Has anyone ever considered/suggested porting
pyto non-Windows platforms? That would potentially allowpyto (eventually) become the cross-platform way to start a Python script... although to mix arguments like-3with using/usr/bin/envyou'd need to use#!/usr/bin/env -S py -3, but I don't think-Sis POSIX-specified 🤷♂️.So perhaps the logic (if finding itself in the shebang line) needs to be:
The logic you describe is basically what's already in place, though it's way more complicated because there are 4-5 different ways to set defaults. Shebangs show up somewhere in the middle.
I'm keen to not make this any more complicated, which is why about the only logic I'd consider is detecting
pyin the shebang and restarting processing with a new command line (and probably explicitly suppressing shebang handling, since we can't be sure that the original shebang line actually specified an option that would suppress it).Has anyone ever considered/suggested porting
pyto non-Windows platforms?Yes, there is also a
pycommand that is similar. However, its implementation is vastly different, and its roadmap is very different. It's not part of the core project, so there's not necessarily any alignment, and we can't direct its behaviour.Besides, one major difference is that it doesn't do shebangs at all, because it relies on the system to handle them. If you
py script-with-shebang.pywith that one, you get the defaultpybehaviour. The shebang handling really is a hack to help out Windows users running scripts that were originally intended for POSIX.But even /usr/bin/env python3 isn't portable to bash on Windows, by default at least, since I don't think Python adds itself to PATH by default
It depends exactly which emulation you're in. Cygwin brings its own Python build, which should include a
python3, and if you installed Python from the Windows Store then you'll get apython3on PATH (though last I heard, Cygwin and anything based on it - which seems to be every Windows Bash implementation - isn't capable of launching Store apps, so that's not much help). The environment is different enough and far enough out of our control that we don't support it, though we try not to do anything to deliberately break it.Or maybe the second case should merge the criteria and flag an error if there's a conflict - that is, if the user specifies
py -3 script.pyand the script says#!py -3.8then it should probably pick 3.8 rather than 3.11.I prefer the current behavior, which ignores the shebang if the launcher is executed with a "-X[.Y][-32|-64]" or "-V:tag" command-line option.
But even /usr/bin/env python3 isn't portable to bash on Windows, by default at least, since I don't think Python adds itself to PATH by default
Yes, there are issues for POSIX environments on Windows that have to be resolved manually. For Windows, with the shebang "#!/usr/bin/env python[X[.Y][-32|-64]][.exe]", if the launcher can't find the executable in
PATH, it instead handles the virtual "python[X[.Y][-32|-64]]" command. Except I just noticed the following bug in the code that ignores ".exe", given the shebang "#!/usr/bin/env python.exe":argv0: C:\Windows\py.exe version: 3.12.2 # Read 60 bytes from C:\Temp\test.py to find shebang line Shebang: /usr/bin/env python.exe # Did not find python.exe on PATH # Reading from C:\Users\user\AppData\Local\py.ini for commands/python.exe # Did not find file C:\Users\user\AppData\Local\py.ini # Reading from C:\Windows\py.ini for commands/python.exe # Did not find file C:\Windows\py.ini # Treating shebang command '.exe' as 'py -.exe' # Cannot select defaults for tag '.exe'I just noticed the following bug in the code that ignores ".exe", given the shebang "#!/usr/bin/env python.exe":
Whoops, yeah, that's a bug.
But at the same time, the intent of this limited shebang support isn't to provide an equivalent to what POSIX shells provide. It is really only meant to be limited to a few options that are compatible with other OSs so that developers who want to be compatible between both are able to achieve it.
I've chatted with Brett, who maintains the POSIX launcher, and he said that the shebang handling in that one is entirely limited to the same basic templates as on Windows in order to restrict Python version. Windows has to handle a few more features, but I'm pretty strongly leaning towards narrowing the scope back to what seems to have been the original intent: basic compatibility with POSIX shells.
What this would mean is that any shebang that doesn't work on a typical/reasonable Linux system is out of scope, even on Windows, and is unsupported. It doesn't mean that any shebang that does work there will be supported, but that the intent of shebang parsing is primarily to infer the
-V:x.yargument and nothing more. It also doesn't mean that existing support gets immediately removed, but it does draw the line against additional feature creep, such as that proposed in this issue.I'm not closing this immediately because I expect further discussion, and if/when some consensus is reached then it needs to be documented somewhere, but I'm no longer intending to change the behaviour in order to support
#! py -x.yshebangs.It depends exactly which emulation you're in. Cygwin brings its own Python build, which should include a
python3, and if you installed Python from the Windows Store then you'll get apython3on PATH (though last I heard, Cygwin and anything based on it - which seems to be every Windows Bash implementation - isn't capable of launching Store apps, so that's not much help). The environment is different enough and far enough out of our control that we don't support it, though we try not to do anything to deliberately break it.I'm using Git for Windows (which I think is built on top of MSYS2). It apparently doesn't come with a /usr/bin/python by default.
Personally for this use case, I'm not too fussed about
py foo.pydoing anything fancy with the shebang line at all. But if there's a valid Python install I'd rather it ran something rather than failing. That is, if I do have something weird or fancy in the shebang, it's unexpected thatpy foo.pycan fail to even lookup the interpreter. So maybe if py is unable to handle the shebang correctly it should just fall back to whatever processing it would normally do to locate the version of python to run, and prefer running something rather than nothing? (For example, in the case above, the shebang makes it look like it it's looking for an interpreter calledpyin the CWD, but there isn't such an executable, so fall back to whatever default python install it can find)Having said that, I'm not actually sure how/why my shebang was working with
./foo.py. From what I can tell, with a shebang of#!py, bash only looks in the current working directory (so you getUnable to create process using 'C:\test\py ./foo.py': The system cannot find the file specified.) but if you use#!py -3suddenly it does a PATH search and works - not sure if this is documented behaviour or a bug in bash 🤷♂️.Git Bash for Windows is about the most useless version of Bash, unfortunately, simply because it doesn't come with the full environment around it. As I said, we try not to actively break it, but there's very little we can do to make it work.
I suspect that when you
./foo.pyin Git Bash, it's processing the shebang line itself, which will follow its own rules. Most likely, it just does a command line replacement and re-runs it, which means you'll get a normalCreateProcesssearch.1 Becausepy.exe's handling of a bare command doesn't work like this, it will not get the same result.So what likely happens is that the
#!pyshebang is used to launchpy.exe foo.pywhich then reprocesses the shebang. But#!py -3shebang (when launched from Bash) launchespy.exe -3 foo.pywhich then ignores the shebang due to the-3, and so succeeds.Probably the best fix here is for Get Bash to detect the
/usr/bin/env python*pattern and remap it topy.exeitself. I'm not sure if that's possible to configure or if it'd need to be an upstream feature, but fundamentally the issue belongs to them.Alternatively you could configure a command in
py.inithat redirects apyshebang to a Python executable, which will override the script directory search we do for unknown commands. This appears to also be undocumented, unfortunately, but adding a[commands]section withpy=<path>ought to do it.Footnotes
-
Searches the application directory - bash.exe in this case - first, then
PATH, with a few added complexities that depend on your system configuration. ↩
Reacted by Tim72020-
I suspect that when you
./foo.pyin Git Bash, it's processing the shebang line itself, which will follow its own rules. Most likely, it just does a command line replacement and re-runs it, which means you'll get a normalCreateProcesssearch. Becausepy.exe's handling of a bare command doesn't work like this, it will not get the same result.So what likely happens is that the
#!pyshebang is used to launchpy.exe foo.pywhich then reprocesses the shebang. But#!py -3shebang (when launched from Bash) launchespy.exe -3 foo.pywhich then ignores the shebang due to the-3, and so succeeds.Oof! Yeah that sounds plausible and would explain the behaviour. Switching to
#!/usr/bin/env pyseems to be the simplest fix for me in this case as I don't need to worry about Linux support or anything. Thanks for the help 😃.The launcher referred to in this issue has now been removed (#153511), and so this issue is expired. Please use the Python install manager to install Python, and file any issues on the pymanager repo.
Bug report
When I invoke a python launcher of version 3.11.0 with
"py"command, a python launcher fails to start process.Step to reproduce
Install python 3.11.0 with py launcher for all users.
Invoke
"py"in command prompt.I expect to launch a python interpreter, but got a error as below
Debug output when enable PYLAUNCHER_DEBUG
I think this error seems to be similar to #95285. If I invoke
pyor"py.exe", a python interpreter starts well as I expected.In version 3.10.8, all of
"py",pyand"py.exe"start a python interpreter correctly.Your environment
Linked PRs