configure fails on linux for distributions using python 3 as default

Version: V7.0.0
Platform: Linux manjaro-vm 4.8.5-2-MANJARO deps: update openssl to 1.0.1j #1 SMP PREEMPT Sun Oct 30 11:26:24 UTC 2016 x86_64 GNU/Linux
Subsystem:

Currently the configure script executes via "python", this works perfectly fine on systems that use python 2.x as their default python but it fails on systems that use python 3.x. Would it be possible/feasible to adjust it to execute via "python2" instead of "python"? Alternatively an updated configure script that works on 2.x and 3.x python would be good.

Python 3 error when trying to run configure:

Is there possibly just a missing module for python3?

This also applies to Arch Linux on which python is actually python3. I think it'd be great if we could apply __future__ and friends to support both versions.

That won't help. configure does import gyp and gyp does not pretend to be python3-compatible. Changing the shebang to python2 breaks systems with no python2 binary. I think that includes OS X.

Indeed, macOS has no python2, only python2.6 and python2.7. The version from homebrew does feature all the symlinks including python2, but I'm not sure we can require that.

As for gyp: There's https://bugs.chromium.org/p/gyp/issues/detail?id=36, but it's unlikely gyp will ever gain support for Python3 as it's pretty much a dead project from what I understand.

@silverwind

There's a bunch of issues in the node-gyp repo that are similar to this. Unfortunately, the conclusion that @silverwind mentioned is likely what we can expect; upstream gyp won't get ported to python 3.

manjaro is based on arch linux and it has python2 and python2.7 links
This should be the same for arch linux but i'm not sure about other linux distributions.
For now I run the configure script directly via python 2.x.
Would it be possible to at least add a check for the python version and show an error message if python3 is detected with an advise to run the script via python2?

I don't think so. The python interpreter will throw syntax errors before the script even gets a chance to run.

The only way to work around that is to split off the main configure script into a separate script and leave only a loader stub that is legal python 2 + 3 but that seems like overcomplicating things when the python 2 prerequisite is listed in BUILDING.md.

I think it should be possible to make the file parseable in Python 3 and print a error when python >= 3. This should work as long as Python does not evaluate import before execution.

For the except issue in the original post, we can apply this workaround.

Can you guys help me check my first PR on node. If there is any problems, let me know.

A couple ideas:

We demand specifically python 2.7, and then do #!/usr/bin/env python2.7, or
We put some logic like this at the top of the script:

#!/usr/bin/env sh

# Find a python 2 executable:
python_executable="`which python2"
if [ -z "$python_executable" ]; then
  python_executable="`which python2.7`"
fi
if [ -z "$python_executable" ]; then
  python_executable="`which python2.6`"
fi
if [ -z "$python_executable" ]; then
  printf 'Error: could not find python 2 executable\n' >&2
  exit 1
fi

# Copy the remainder of this script (which is python) to a new file and execute it with
# python instead of sh:
tmpfile="`mktemp`"
cat "$0" | tail -n +22 > "$tmpfile"
chmod +x "$tmpfile"
exec "$python_executable" "tmpfile" $@

(1) is certainly simpler.

Αnother option is to special-case MacOS, doing something like:

case `uname` in
  darwin) python_executable="`which python2.7`" ;; # Or whatever uname returns on OS X; I don't have a live system handy.
  *) python_executable="`which python`";;
esac

@zenhack

This is fixed by #9657.

@zenhack generally good ideas, but I think the current solution is good enough as the user will know what to do from the error message. Spawning another python child process doesn't seem like the cleanest solution to me.

So, this actually does't fix the problem:

$ python2 configure 
creating icu_config.gypi
* Using ICU in deps/icu-small
Using version-specific floating patch tools/icu/patches/58/source/i18n/digitlst.cpp
creating icu_config.gypi
{ 'target_defaults': { 'cflags': [],
                       'default_configuration': 'Release',
                       'defines': [],
                       'include_dirs': [],
                       'libraries': []},
  'variables': { 'asan': 0,
                 'coverage': 'false',
                 'debug_devtools': 'node',
                 'force_dynamic_crt': 0,
                 'gas_version': '2.26',
                 'host_arch': 'x64',
                 'icu_data_file': 'icudt58l.dat',
                 'icu_data_in': '../../deps/icu-small/source/data/in/icudt58l.dat',
                 'icu_endianness': 'l',
                 'icu_gyp_path': 'tools/icu/icu-generic.gyp',
                 'icu_locales': 'en,root',
                 'icu_path': 'deps/icu-small',
                 'icu_small': 'true',
                 'icu_ver_major': '58',
                 'node_byteorder': 'little',
                 'node_enable_d8': 'false',
                 'node_enable_v8_vtunejit': 'false',
                 'node_install_npm': 'true',
                 'node_module_version': 51,
                 'node_no_browser_globals': 'false',
                 'node_prefix': '/usr/local',
                 'node_release_urlbase': '',
                 'node_shared': 'false',
                 'node_shared_cares': 'false',
                 'node_shared_http_parser': 'false',
                 'node_shared_libuv': 'false',
                 'node_shared_openssl': 'false',
                 'node_shared_zlib': 'false',
                 'node_tag': '',
                 'node_use_bundled_v8': 'true',
                 'node_use_dtrace': 'false',
                 'node_use_etw': 'false',
                 'node_use_lttng': 'false',
                 'node_use_openssl': 'true',
                 'node_use_perfctr': 'false',
                 'node_use_v8_platform': 'true',
                 'openssl_fips': '',
                 'openssl_no_asm': 0,
                 'shlib_suffix': 'so.51',
                 'target_arch': 'x64',
                 'uv_parent_path': '/deps/uv/',
                 'uv_use_dtrace': 'false',
                 'v8_enable_gdbjit': 0,
                 'v8_enable_i18n_support': 1,
                 'v8_inspector': 'true',
                 'v8_no_strict_aliasing': 1,
                 'v8_optimized_debug': 0,
                 'v8_random_seed': 0,
                 'v8_use_snapshot': 'true',
                 'want_separate_host_toolset': 0,
                 'want_separate_host_toolset_mkpeephole': 0}}
creating config.gypi
creating config.mk
/bin/sh: python: command not found
gyp: Call to 'python -c "import sys; print sys.byteorder"' returned exit status 127 while in /home/isd/scratch/node/deps/v8/src/v8.gyp. while loading dependencies of /home/isd/scratch/node/node.gyp while trying to load /home/isd/scratch/node/node.gyp
Error running GYP

There are a whole bunch of other python scripts in the tree that also hard-code the name of the python executable, so they'd all have to be run with python2, and it isn't entirely straightforward to do that short of patching the source tree, since they aren't invoked directly by the user. Here's a list:

$ grep -r '^#!/usr/bin/env python$' .
./deps/cares/build/gcc_version.py:#!/usr/bin/env python
./deps/cares/gyp_cares:#!/usr/bin/env python
./deps/npm/node_modules/node-gyp/gyp/setup.py:#!/usr/bin/env python
./deps/npm/node_modules/node-gyp/gyp/pylib/gyp/MSVSSettings_test.py:#!/usr/bin/env python
./deps/npm/node_modules/node-gyp/gyp/pylib/gyp/easy_xml_test.py:#!/usr/bin/env python
./deps/npm/node_modules/node-gyp/gyp/pylib/gyp/generator/ninja_test.py:#!/usr/bin/env python
./deps/npm/node_modules/node-gyp/gyp/pylib/gyp/generator/msvs_test.py:#!/usr/bin/env python
./deps/npm/node_modules/node-gyp/gyp/pylib/gyp/generator/xcode_test.py:#!/usr/bin/env python
./deps/npm/node_modules/node-gyp/gyp/pylib/gyp/mac_tool.py:#!/usr/bin/env python
./deps/npm/node_modules/node-gyp/gyp/pylib/gyp/input_test.py:#!/usr/bin/env python
./deps/npm/node_modules/node-gyp/gyp/pylib/gyp/common_test.py:#!/usr/bin/env python
./deps/npm/node_modules/node-gyp/gyp/pylib/gyp/flock_tool.py:#!/usr/bin/env python
./deps/npm/node_modules/node-gyp/gyp/pylib/gyp/__init__.py:#!/usr/bin/env python
./deps/npm/node_modules/node-gyp/gyp/pylib/gyp/win_tool.py:#!/usr/bin/env python
./deps/npm/node_modules/node-gyp/gyp/buildbot/buildbot_run.py:#!/usr/bin/env python
./deps/npm/node_modules/node-gyp/gyp/gyp_main.py:#!/usr/bin/env python
./deps/npm/node_modules/node-gyp/gyp/tools/pretty_sln.py:#!/usr/bin/env python
./deps/npm/node_modules/node-gyp/gyp/tools/pretty_gyp.py:#!/usr/bin/env python
./deps/npm/node_modules/node-gyp/gyp/tools/graphviz.py:#!/usr/bin/env python
./deps/npm/node_modules/node-gyp/gyp/tools/pretty_vcproj.py:#!/usr/bin/env python
./deps/npm/node_modules/node-gyp/gyp/gyptest.py:#!/usr/bin/env python
./deps/v8/test/test262/list.py:#!/usr/bin/env python
./deps/v8/test/test262/archive.py:#!/usr/bin/env python
./deps/v8/tools/unittests/run_perf_test.py:#!/usr/bin/env python
./deps/v8/tools/js2c.py:#!/usr/bin/env python
./deps/v8/tools/find-commit-for-patch.py:#!/usr/bin/env python
./deps/v8/tools/stats-viewer.py:#!/usr/bin/env python
./deps/v8/tools/run-tests.py:#!/usr/bin/env python
./deps/v8/tools/grokdump.py:#!/usr/bin/env python
./deps/v8/tools/gyp_flag_compare.py:#!/usr/bin/env python
./deps/v8/tools/dump-cpp.py:#!/usr/bin/env python
./deps/v8/tools/run-deopt-fuzzer.py:#!/usr/bin/env python
./deps/v8/tools/trace-maps-processor.py:#!/usr/bin/env python
./deps/v8/tools/eval_gc_nvp.py:#!/usr/bin/env python
./deps/v8/tools/release/auto_push.py:#!/usr/bin/env python
./deps/v8/tools/release/test_scripts.py:#!/usr/bin/env python
./deps/v8/tools/release/script_test.py:#!/usr/bin/env python
./deps/v8/tools/release/mergeinfo.py:#!/usr/bin/env python
./deps/v8/tools/release/check_clusterfuzz.py:#!/usr/bin/env python
./deps/v8/tools/release/releases.py:#!/usr/bin/env python
./deps/v8/tools/release/create_release.py:#!/usr/bin/env python
./deps/v8/tools/release/auto_tag.py:#!/usr/bin/env python
./deps/v8/tools/release/search_related_commits.py:#!/usr/bin/env python
./deps/v8/tools/release/test_search_related_commits.py:#!/usr/bin/env python
./deps/v8/tools/release/merge_to_branch.py:#!/usr/bin/env python
./deps/v8/tools/release/push_to_candidates.py:#!/usr/bin/env python
./deps/v8/tools/release/common_includes.py:#!/usr/bin/env python
./deps/v8/tools/release/roll_merge.py:#!/usr/bin/env python
./deps/v8/tools/release/test_mergeinfo.py:#!/usr/bin/env python
./deps/v8/tools/release/auto_roll.py:#!/usr/bin/env python
./deps/v8/tools/release/git_recipes.py:#!/usr/bin/env python
./deps/v8/tools/android-run.py:#!/usr/bin/env python
./deps/v8/tools/mingw-generate-makefiles.sh:#!/usr/bin/env python
./deps/v8/tools/callstats.py:#!/usr/bin/env python
./deps/v8/tools/perf-to-html.py:#!/usr/bin/env python
./deps/v8/tools/run-valgrind.py:#!/usr/bin/env python
./deps/v8/tools/concatenate-files.py:#!/usr/bin/env python
./deps/v8/tools/generate_shim_headers/generate_shim_headers.py:#!/usr/bin/env python
./deps/v8/tools/try_perf.py:#!/usr/bin/env python
./deps/v8/tools/gcmole/download_gcmole_tools.py:#!/usr/bin/env python
./deps/v8/tools/gcmole/run-gcmole.py:#!/usr/bin/env python
./deps/v8/tools/gcmole/parallel.py:#!/usr/bin/env python
./deps/v8/tools/verify_source_deps.py:#!/usr/bin/env python
./deps/v8/tools/isolate_driver.py:#!/usr/bin/env python
./deps/v8/tools/ll_prof.py:#!/usr/bin/env python
./deps/v8/tools/run.py:#!/usr/bin/env python
./deps/v8/tools/process-heap-prof.py:#!/usr/bin/env python
./deps/v8/tools/test-server.py:#!/usr/bin/env python
./deps/v8/tools/dev/v8gen.py:#!/usr/bin/env python
./deps/v8/tools/presubmit.py:#!/usr/bin/env python
./deps/v8/tools/gen-postmortem-metadata.py:#!/usr/bin/env python
./deps/v8/tools/objdump-v8:#!/usr/bin/env python
./deps/v8/tools/run_perf.py:#!/usr/bin/env python
./deps/v8/tools/gc-nvp-to-csv.py:#!/usr/bin/env python
./deps/v8/tools/sanitizers/sancov_formatter.py:#!/usr/bin/env python
./deps/v8/tools/sanitizers/sancov_merger.py:#!/usr/bin/env python
./deps/v8/tools/sanitizers/sanitize_pcs.py:#!/usr/bin/env python
./deps/v8/tools/gc-nvp-trace-processor.py:#!/usr/bin/env python
./deps/v8/tools/external-reference-check.py:#!/usr/bin/env python
./deps/v8/tools/testrunner/local/testsuite_unittest.py:#!/usr/bin/env python
./deps/v8/tools/testrunner/local/statusfile_unittest.py:#!/usr/bin/env python
./deps/v8/tools/testrunner/local/pool.py:#!/usr/bin/env python
./deps/v8/tools/testrunner/local/pool_unittest.py:#!/usr/bin/env python
./deps/v8/tools/testrunner/server/daemon.py:#!/usr/bin/env python
./deps/v8/tools/disasm.py:#!/usr/bin/env python
./deps/v8/tools/jsfunfuzz/download_jsfunfuzz.py:#!/usr/bin/env python
./deps/v8/tools/generate-builtins-tests.py:#!/usr/bin/env python
./deps/v8/gypfiles/landmines.py:#!/usr/bin/env python
./deps/v8/gypfiles/detect_v8_host_arch.py:#!/usr/bin/env python
./deps/v8/gypfiles/get_landmines.py:#!/usr/bin/env python
./deps/v8/gypfiles/vs_toolchain.py:#!/usr/bin/env python
./deps/v8/gypfiles/coverage_wrapper.py:#!/usr/bin/env python
./deps/v8/gypfiles/gyp_v8:#!/usr/bin/env python
./deps/v8/gypfiles/download_gold_plugin.py:#!/usr/bin/env python
./deps/uv/gyp_uv.py:#!/usr/bin/env python
./deps/v8_inspector/third_party/markupsafe/bench/runbench.py:#!/usr/bin/env python
./deps/v8_inspector/third_party/WebKit/Source/platform/inspector_protocol/CheckProtocolCompatibility.py:#!/usr/bin/env python
./deps/v8_inspector/third_party/WebKit/Source/platform/inspector_protocol/ConcatenateProtocols.py:#!/usr/bin/env python
./deps/v8_inspector/third_party/jinja2/scripts/jinja2-debug.py:#!/usr/bin/env python
./deps/v8_inspector/third_party/jinja2/scripts/make-release.py:#!/usr/bin/env python
./deps/v8_inspector/src/inspector/build/rjsmin.py:#!/usr/bin/env python
./deps/v8_inspector/src/inspector/build/generate_protocol_externs.py:#!/usr/bin/env python
./deps/v8_inspector/src/inspector/build/compile-scripts.py:#!/usr/bin/env python
./deps/v8_inspector/src/inspector/build/check_injected_script_source.py:#!/usr/bin/env python
./deps/v8_inspector/src/inspector/PRESUBMIT.py:#!/usr/bin/env python
./test/gc/node_modules/weak/build/gyp-mac-tool:#!/usr/bin/env python
./tools/js2c.py:#!/usr/bin/env python
./tools/mkssldef.py:#!/usr/bin/env python
./tools/install.py:#!/usr/bin/env python
./tools/cpplint.py:#!/usr/bin/env python
./tools/configure.d/nodedownload.py:#!/usr/bin/env python
./tools/gyp/setup.py:#!/usr/bin/env python
./tools/gyp/pylib/gyp/MSVSSettings_test.py:#!/usr/bin/env python
./tools/gyp/pylib/gyp/easy_xml_test.py:#!/usr/bin/env python
./tools/gyp/pylib/gyp/generator/ninja_test.py:#!/usr/bin/env python
./tools/gyp/pylib/gyp/generator/msvs_test.py:#!/usr/bin/env python
./tools/gyp/pylib/gyp/generator/xcode_test.py:#!/usr/bin/env python
./tools/gyp/pylib/gyp/mac_tool.py:#!/usr/bin/env python
./tools/gyp/pylib/gyp/input_test.py:#!/usr/bin/env python
./tools/gyp/pylib/gyp/common_test.py:#!/usr/bin/env python
./tools/gyp/pylib/gyp/flock_tool.py:#!/usr/bin/env python
./tools/gyp/pylib/gyp/__init__.py:#!/usr/bin/env python
./tools/gyp/pylib/gyp/win_tool.py:#!/usr/bin/env python
./tools/gyp/buildbot/buildbot_run.py:#!/usr/bin/env python
./tools/gyp/gyp_main.py:#!/usr/bin/env python
./tools/gyp/tools/pretty_sln.py:#!/usr/bin/env python
./tools/gyp/tools/pretty_gyp.py:#!/usr/bin/env python
./tools/gyp/tools/graphviz.py:#!/usr/bin/env python
./tools/gyp/tools/pretty_vcproj.py:#!/usr/bin/env python
./tools/gyp/gyptest.py:#!/usr/bin/env python
./tools/run-valgrind.py:#!/usr/bin/env python
./tools/icu/shrink-icu-src.py:#!/usr/bin/env python
./tools/compress_json.py:#!/usr/bin/env python
./tools/gyp_node.py:#!/usr/bin/env python
./tools/test.py:#!/usr/bin/env python
./tools/genv8constants.py:#!/usr/bin/env python
./tools/check-imports.py:#!/usr/bin/env python
./tools/specialize_node_d.py:#!/usr/bin/env python
./configure:#!/usr/bin/env python

The Archlinux PKGBUILD just applies a sed command to the whole source tree:

https://git.archlinux.org/svntogit/community.git/tree/trunk/PKGBUILD?h=packages/nodejs#n27

I agree the solution I proposed isn't pretty, and I wouldn't want to apply it to many files. It also looks like a lot of these are in dependencies that node pulls in in-tree? Probably don't want to be patching those.

This probably ought to be re-opened, since the problem persists. I don't have another solution immediately popping into my head, but will think on the matter.

@zenhack

For future reference, here's how I got it to work on Manjaro (~Arch):

mkdir /home/gcg/python2
ln -s /usr/bin/python2 /home/gcg/python2/python

Then I run tests with

PATH=/home/gcg/python2:$PATH make -j4 test

(credits go to @zenhack)

@gunar

@gunar: try PYTHON=/usr/bin/python2 ./configure instead.

@jbergstroem

@jbergstroem maybe we should add this info on the error print?

node/configure

Line 5 in 76ba3bf

sys.stdout.write("Please use either Python 2.6 or 2.7\n")

@silverwind

@silverwind sounds good to me!

@jbergstroem

@jbergstroem your method did not work :-(

> PYTHON=/usr/bin/python2 ./configure
Please use either Python 2.6 or 2.7
> uname -a
Linux zeno 4.4.45-1-MANJARO #1 SMP PREEMPT Thu Jan 26 11:32:11 UTC 2017 x86_64 GNU/Linux

But thanks!

Hmm, where's this PYTHON variable actually documented? Can't find it:

https://docs.python.org/3.7/using/cmdline.html#environment-variables

Sorry; its used in Makefile.

So we need to suggest PYTHON=path make too, right?

@silverwind

@silverwind said:
So we need to suggest PYTHON=path make too, right?

No; passing PYTHON=path to configure should write it to config.mk which gets included in Makefile. I was just providing a poor example.

Alright, so PYTHON=/usr/bin/python2 /usr/bin/python2 configure? 😆

Nevermind, I see 'PYTHON': sys.executable is already handed down. I'll land #11375 later as-is.

EDIT: I had a problem but when I updated to the latest 8.7.0 code it works.

If ./configure were to create a symlink called python that pointed towards whatever was used to run it, and were to prepend the directory containing it to the path, then that would work right?

So this was implemented in #16058, so this should now finally work without having to set PYTHON, even if your python is a python3 (as long as you have a python2 on your system).

Navigation Menu

Repository navigation

configure fails on linux for distributions using python 3 as default #9512

7 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

configure fails on linux for distributions using python 3 as default #9512

Description

Activity

silverwind commented on Nov 8, 2016

bnoordhuis commented on Nov 8, 2016

silverwind commented on Nov 8, 2016

jbergstroem commented on Nov 8, 2016

shannon-nii commented on Nov 9, 2016

bnoordhuis commented on Nov 9, 2016

silverwind commented on Nov 9, 2016

kalrover commented on Nov 17, 2016

zenhack commented on Nov 28, 2016

zenhack commented on Nov 28, 2016

silverwind commented on Nov 30, 2016

zenhack commented on Nov 30, 2016

7 remaining items

gunar commented on Feb 14, 2017

jbergstroem commented on Feb 14, 2017

silverwind commented on Feb 14, 2017

jbergstroem commented on Feb 14, 2017

gunar commented on Feb 14, 2017

silverwind commented on Feb 14, 2017

jbergstroem commented on Feb 16, 2017

silverwind commented on Feb 16, 2017

jbergstroem commented on Feb 16, 2017

silverwind commented on Feb 16, 2017

silverwind commented on Feb 16, 2017

Siecje commented on Oct 23, 2017

gibfahn commented on Oct 23, 2017

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions