Download of jq binary fail when behind corporate proxy
olkinfra opened this issue · comments
Description
When using a corporate proxy (host has no direct access to internet) and node-jq inside an npm project (Mattermost Desktop App) as dependency, during postintall the download of jq binary fail with a timeout
Test Source
N/A
Error Message & Stack Trace
npm info run node-jq@2.3.4 postinstall { code: 1, signal: null }
npm verb stack Error: command failed
npm verb stack at ChildProcess.<anonymous> (/usr/local/lib/node_modules/npm/node_modules/@npmcli/promise-spawn/lib/index.js:53:27)
npm verb stack at ChildProcess.emit (node:events:514:28)
npm verb stack at maybeClose (node:internal/child_process:1091:16)
npm verb stack at ChildProcess._handle.onexit (node:internal/child_process:302:5)
npm verb pkgid node-jq@2.3.4
npm verb cwd /builds/infra/mattermost-desktop-app/mm_desktop
npm verb Linux 4.18.0-425.13.1.el8_7.x86_64
npm verb node v18.17.0
npm verb npm v9.6.7
npm ERR! code 1
npm ERR! path /builds/infra/mattermost-desktop-app/mm_desktop/node_modules/node-jq
npm ERR! command failed
npm ERR! command sh -c npm run install-binary
npm ERR! > node-jq@2.3.4 install-binary
npm ERR! > node scripts/install-binary.js
npm ERR!
npm ERR! Downloading jq from https://github.com/stedolan/jq/releases/download/jq-1.6/jq-linux64
npm ERR! npm verb cli /usr/local/bin/node /usr/local/bin/npm
npm ERR! npm info using npm@9.6.7
npm ERR! npm info using node@v18.17.0
npm ERR! npm verb title npm run install-binary
npm ERR! npm verb argv "run" "install-binary"
npm ERR! npm verb logfile logs-max:10 dir:/root/.npm/_logs/2023-07-28T13_09_09_659Z-
npm ERR! npm verb logfile /root/.npm/_logs/2023-07-28T13_09_09_659Z-debug-0.log
npm ERR! npm sill logfile done cleaning log files
npm ERR! GotError [RequestError]: connect ETIMEDOUT 140.82.121.3:443
npm ERR! at ClientRequest.<anonymous> (/builds/infra/mattermost-desktop-app/mm_desktop/node_modules/download/node_modules/got/index.js:182:22)
npm ERR! at Object.onceWrapper (node:events:629:26)
npm ERR! at ClientRequest.emit (node:events:514:28)
npm ERR! at TLSSocket.socketErrorListener (node:_http_client:501:9)
npm ERR! at TLSSocket.emit (node:events:514:28)
npm ERR! at emitErrorNT (node:internal/streams/destroy:151:8)
npm ERR! at emitErrorCloseNT (node:internal/streams/destroy:116:3)
npm ERR! at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
npm ERR! code: 'ETIMEDOUT',
npm ERR! host: 'github.com',
npm ERR! hostname: 'github.com',
npm ERR! method: 'GET',
npm ERR! path: '/stedolan/jq/releases/download/jq-1.6/jq-linux64',
npm ERR! protocol: 'https:',
npm ERR! url: 'https://github.com/stedolan/jq/releases/download/jq-1.6/jq-linux64'
npm ERR! }
npm ERR! npm verb exit 1
npm ERR! npm verb code 1
npm verb exit 1
npm verb unfinished npm timer reify 1690549637459
npm verb unfinished npm timer reify:build 1690549724138
npm verb unfinished npm timer build 1690549724143
npm verb unfinished npm timer build:deps 1690549724145
npm verb unfinished npm timer build:run:postinstall 1690549748951
npm verb unfinished npm timer build:run:postinstall:node_modules/node-jq 1690549749092
npm verb code 1
npm ERR! A complete log of this run can be found in: /root/.npm/_logs/2023-07-28T13_07_16_864Z-debug-0.logEnvironment
node-jq version : 2.3.4
npm version : 9.6.7
node version : 18.17.0
Host : Rocky Linux 8 (with proxy environment variables set)
Gitlab-runner on kubernetes (with proxy environment variables set at k8s level), when Gitlab trigger a pipeline it spawn a pod with an image node:lts-alpine (with proxy environment variables too)
Hello !
I know that there is an option that allow to skip binary download and manually point to the binary with an environment variable but with this method I have a permission issue which lead to another error.
I had the exact same issue at the step where electron module is downloaded, electron has implemented a variable (ELECTRON_GET_USE_PROXY) which trigger the use of global-agent with proxy environment variables retrieved via OS variables, would it be possible to do something similar ?
Best regards
Hi @olkinfra
Thanks for opening the issue. I'm not entirely familiar with your issue (using a proxy and having permission issues with downloading jq directly).
We use node-downloader-helper as a dependency to fetch the right jq architecture based on the OS. This script is here https://github.com/sanack/node-jq/blob/main/scripts/install-binary.js, feel free to take a look and see if there's any way we can setup the proxy there?
Hello @davesnx ,
Thank you for your quick reply, indeed it's possible to modify the script, node-download-helper allow to customize a request through httpRequestOptions (https://nodejs.org/api/http.html#http_http_request_options_callback) or httpsRequestOptions (https://nodejs.org/api/https.html#https_https_request_options_callback), you can configure the use of a proxy with the "agent" parameter (you can see an example in this discussion from node-download-helper project : hgouveia/node-downloader-helper#81). So if an agent like global-agent (or any other one) is initialized, you can then use it with those parameters.
I won't be able to suggest a fix as I am not a developer, but I can show you how electron implement it (I found it in their code) and I can also test a fix if one is produced :)
Since the install-script isn't too easy to configure (the only parameter provided is NODE_JQ_SKIP_INSTALL_BINARY to avoid running it). There's no way to configure the proxy agent from the CLI.
I suggest you to copy the install-script and add the proxy stuff needed, run the script before npm install and add NODE_JQ_SKIP_INSTALL_BINARY=true as env variable.
If that works, I will look into another way to configure the httpAgent for the install-script.
The problem is, in my context, that node-jq is downloaded as a dependency of another project so it's complicated to modify the script on the fly, if it's possible I will have to check how to achieve this. For the moment, as I am the company sysadmin I have temporarly enabled direct access to internet from my gitlab runner and I'm able to complete the install phase, but a proxy support in the future would be great.
Here from line 16 to 34 you can take a look at how it's done in Electron : https://github.com/electron/get/blob/4e0b055a7c8aa337215a8cd20880acc262d5a925/src/index.ts
utils.ts file add ability to read system environment's variables : https://github.com/electron/get/blob/4e0b055a7c8aa337215a8cd20880acc262d5a925/src/utils.ts
proxy.ts initialize a global-agent instance : https://github.com/electron/get/blob/4e0b055a7c8aa337215a8cd20880acc262d5a925/src/proxy.ts
You won't need to modify the script on the fly. Copy the script, modify it and run it before the npm installation.
Yes, there's ways to achieve the same level of configuration on httpAgent via env vars but the preferred way to deal with proxy issues on installation is to avoid using install-script.