node-gyp always pulls headers from internet

[rudicio]

After tracing through the code, I noticed that no matter what unless the --tarball flag is explicitly stated node-gyp will always attempt to pull the headers from the internet. The problem with this workaround is that if you try to install an existing npm module built with node-gyp, there is no means to specifying this flag.

It looks like fixing this would be as simple as adding a check if the file already exists in ~/.node-gyp for the architecture, or supporting an environment variable that specifies the path rather than a flag:

https://github.com/nodejs/node-gyp/blob/f6eab1f9e4b160b60b2736bb01a09c83785ba979/lib/install.js

[Fishrock123]

IIRC npm will accept something like --node-gyp-tarball.

[richardlau]

You can pass --nodedir=/path/to/node/headers to npm install, which should avoid downloading.

https://docs.npmjs.com/cli/install

The --nodedir=/path/to/node/source argument will allow npm to find the node source code so that npm can compile native modules.

[bnoordhuis]

What @richardlau said. npm passes on any switches it doesn't consume itself to node-gyp. npm install --tarball <..> probably works too.

[nickboldt]

Seeing this in a firewalled build:

yarn install v1.17.3
 $ node-gyp install
 gyp info it worked if it ends with ok
 gyp info using node-gyp@3.8.0
 gyp info using node@10.14.1 | linux | x64
 gyp http GET https://nodejs.org/download/release/v10.14.1/node-v10.14.1-headers.tar.gz
 gyp WARN install got an error, rolling back install
 gyp ERR! install error
 gyp ERR! stack Error: connect ECONNREFUSED 104.20.23.46:443
 gyp ERR! stack     at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1113:14)
 gyp ERR! System Linux 3.10.0-957.21.2.el7.x86_64
 gyp ERR! command "/usr/bin/node" "/usr/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "install"
 gyp ERR! cwd /home/theia-dev/theia-source-code
 gyp ERR! node -v v10.14.1
 gyp ERR! node-gyp -v v3.8.0
 gyp ERR! not ok

I can confirm that this works to prevent node-gyp from trying to download the headers:

NODE_VER=v10.14.1
curl -sSL https://nodejs.org/download/release/${NODE_VER}/node-${NODE_VER}-headers.tar.gz \
        -o /tmp/node-headers.tgz; \
npm config set tarball /tmp/node-headers.tgz

Now:

 yarn install v1.17.3
 $ node-gyp install
 gyp info it worked if it ends with ok
 gyp info using node-gyp@3.8.0
 gyp info using node@10.14.1 | linux | x64
 10.14.1
 gyp info ok
 [1/5] Validating package.json...
 [2/5] Resolving packages...
 error Couldn't find package "@eclipse-che/workspace-client@latest" required by "@eclipse-che/che-theia-hosted-plugin-manager-extension@7.2.0" on the "npm" registry.

So it's one step forward, one missing requirement in the npm offline cache back. :)

[DJViking]

This should not be closed. The --nodedir is a workaround, not a proper solution to the problem.

A proper solution would be to fix node-gyp with an configurable URL in the .npmrc where to retrieve the node-headers archive.
This solution works for others like we use.

chromedriver_cdnurl=https://nexus.company.com/javascript/chromedriver
phantomjs_cdnurl=https://nexus.company.com/javascript/
sass_binary_site=https://nexus.company.com/javascript/node-sass

The node-gyp could have something like this, to avoid going out on the Internet to retrieve the node headers.

[richardlau]

@DJViking You can use --dist-url to set the URL from which to download the headers. All node-gyp settings can be set via npm_config_ environment variables, so you should be able to set disturl=https://... in your .npmrc file.

[richardlau]

disturl in .npmrc works (there's an open issue about how the - is handled #2250). You can also set tarball to point directly at a locally available headers tarball.

[dovidweisz]

@nickboldt

Thank you, this fixed a sporadic issue I've been having for years now.

Is this documented anywhere? npm config set tarball

[nickboldt]

Does not appear to be in https://docs.npmjs.com/cli/v10/using-npm/config (or v8 or v9 either) so 🤷

See also #2800 ... seems this support might have been removed in 9.3

But #1154 (comment) suggests that env var npm_config_tarball seems to still work.