Yarn install task failing consistently with [error]connect ETIMEDOUT 104.27.154.154:443

Question

Yarn install task failing consistently with [error]connect ETIMEDOUT 104.27.154.154:443

moizgh opened this issue 4 years ago · comments

Moiz Ghadiyali commented 4 years ago

Are you reporting a bug or a feature request ?

Bug
Feature request

Description

Yarn install task is failing consistently with [error]connect ETIMEDOUT 104.27.154.154:443

Expected behavior

Yarn install task should not fail

Reproduction

Insert reproduction steps here

MANDATORY FOR BUGS: Insert debug trace

2020-04-13T19:54:45.2933122Z ##[debug]Evaluating condition for step: 'Use Yarn 1.x'
2020-04-13T19:54:45.2933645Z ##[debug]Evaluating: SucceededNode()
2020-04-13T19:54:45.2933746Z ##[debug]Evaluating SucceededNode:
2020-04-13T19:54:45.2933898Z ##[debug]=> True
2020-04-13T19:54:45.2934038Z ##[debug]Result: True
2020-04-13T19:54:45.2934168Z ##[section]Starting: Use Yarn 1.x
2020-04-13T19:54:45.3045838Z ==============================================================================
2020-04-13T19:54:45.3045907Z Task : Yarn Tool Installer
2020-04-13T19:54:45.3046132Z Description : Installs a version of Yarn
2020-04-13T19:54:45.3046176Z Version : 3.0.1999
2020-04-13T19:54:45.3046202Z Author : Geek Learning
2020-04-13T19:54:45.3046229Z Help : More Information (Version 3.0.1999)
2020-04-13T19:54:45.3046274Z ==============================================================================
2020-04-13T19:54:45.5569889Z ##[debug]agent.TempDirectory=E:_work_temp
2020-04-13T19:54:45.5647965Z ##[debug]loading inputs and endpoints
2020-04-13T19:54:45.5651728Z ##[debug]loading ENDPOINT_AUTH_PARAMETER_SYSTEMVSSCONNECTION_ACCESSTOKEN
2020-04-13T19:54:45.5662381Z ##[debug]loading ENDPOINT_AUTH_SCHEME_SYSTEMVSSCONNECTION
2020-04-13T19:54:45.5665611Z ##[debug]loading ENDPOINT_AUTH_SYSTEMVSSCONNECTION
2020-04-13T19:54:45.5667371Z ##[debug]loading INPUT_ARGUMENTS
2020-04-13T19:54:45.5668994Z ##[debug]loading INPUT_CHECKLATEST
2020-04-13T19:54:45.5670231Z ##[debug]loading INPUT_INCLUDEPRERELEASE
2020-04-13T19:54:45.5671809Z ##[debug]loading INPUT_VERSIONSPEC
2020-04-13T19:54:45.5676156Z ##[debug]loading SECRET_SYSTEM_ACCESSTOKEN
2020-04-13T19:54:45.5681216Z ##[debug]loaded 8
2020-04-13T19:54:45.5694559Z ##[debug]Agent.ProxyUrl=undefined
2020-04-13T19:54:45.5694999Z ##[debug]Agent.CAInfo=undefined
2020-04-13T19:54:45.5696364Z ##[debug]Agent.ClientCert=undefined
2020-04-13T19:54:45.5696575Z ##[debug]Agent.SkipCertValidation=undefined
2020-04-13T19:54:45.5898111Z ##[debug]Agent.ProxyUrl=undefined
2020-04-13T19:54:45.5899518Z ##[debug]Agent.CAInfo=undefined
2020-04-13T19:54:45.5900468Z ##[debug]Agent.ClientCert=undefined
2020-04-13T19:54:45.5904256Z ##[debug]check path : E:_work_tasks\YarnInstaller_2ec1147b-5c33-41df-94b5-676758d87c1b\3.0.1999\node_modules\azure-pipelines-tool-lib\lib.json
2020-04-13T19:54:45.5906425Z ##[debug]adding resource file: E:_work_tasks\YarnInstaller_2ec1147b-5c33-41df-94b5-676758d87c1b\3.0.1999\node_modules\azure-pipelines-tool-lib\lib.json
2020-04-13T19:54:45.5907146Z ##[debug]system.culture=en-US
2020-04-13T19:54:45.6413679Z ##[debug]Agent.BuildDirectory=E:_work\1
2020-04-13T19:54:45.6428213Z ##[debug]versionSpec=1.x
2020-04-13T19:54:45.6430953Z ##[debug]checkLatest=false
2020-04-13T19:54:45.6432209Z ##[debug]includePrerelease=false
2020-04-13T19:54:45.6437709Z ##[debug]isExplicit: null
2020-04-13T19:54:45.6438052Z ##[debug]explicit? false
2020-04-13T19:54:45.6438133Z ##[debug]isExplicit: null
2020-04-13T19:54:45.6438279Z ##[debug]explicit? false
2020-04-13T19:54:45.6476714Z ##[debug]Agent.Version=2.164.6
2020-04-13T19:54:45.6477681Z ##[debug]Agent.ToolsDirectory=E:_work_tool
2020-04-13T19:54:45.6478356Z ##[debug]evaluating 0 versions
2020-04-13T19:54:45.6478827Z ##[debug]match not found
2020-04-13T19:54:45.6479307Z ##[debug]isExplicit: null
2020-04-13T19:54:45.6479444Z ##[debug]explicit? false
2020-04-13T19:54:45.6479542Z ##[debug]downloading: https://publicblobs.geeklearning.io/yarn/tarballsV2.json
2020-04-13T19:55:06.6731281Z ##[debug]task result: Failed
2020-04-13T19:55:06.6778158Z ##[error]connect ETIMEDOUT 104.27.154.154:443
2020-04-13T19:55:06.6787904Z ##[debug]Processed: ##vso[task.issue type=error;]connect ETIMEDOUT 104.27.154.154:443
2020-04-13T19:55:06.6788378Z ##[debug]Processed: ##vso[task.complete result=Failed;]connect ETIMEDOUT 104.27.154.154:443
2020-04-13T19:55:06.6810091Z ##[section]Finishing: Use Yarn 1.x

In order to allow us to quickly reproduce you issue please include a link to a reproduction repository

System information

VSTS or TFS:

VSTS
TFS

Agent:

Hosted
Custom

OS:

Windows
Mac
Linux

Yarn version:
Yarn version*

Node version:
Node version*

dedicad · Answer 1 · Wed Apr 15 2020 02:29:32 GMT+0800 (China Standard Time)

It seems that I also have an error with the yarn build task. Everything was working perfectly for several weeks until Sunday.
My system information are the same.

Stack Trace :

2020-04-14T04:08:50.2444442Z info There appears to be trouble with your network connection. Retrying...
2020-04-14T04:08:56.3383164Z info There appears to be trouble with your network connection. Retrying...
2020-04-14T04:09:39.3758843Z error An unexpected error occurred: "https://registry.yarnpkg.com/date-fns/-/date-fns-2.12.0.tgz: ESOCKETTIMEDOUT".

Edit

For our pipeline it was probably due to a slower connection (Azure side, or yarn/npm side, I don't know). So setting network-timeout to a higher value solved it. Here are some commands that could fix your issue.

Option 1 : With a global command
yarn config set network-timeout 600000 -g
Option 2 : With a config file
Create a .yarnrc at the root of your project with the following content :
network-timeout 600000

Network timeout for yarn are also shown for a lot of errors, linked with lock file present and so on, so you might want to check other issues that you could find online.
(for example : yarnpkg/yarn#5259)

Moiz Ghadiyali · Answer 2 · Wed Apr 15 2020 22:06:05 GMT+0800 (China Standard Time)

@dedicad Thank you for the pointers. We already have network-timeout 600000 set in the .yarnrc
This seems to occur on some agents quite consistently as compared to others. From the logs posted, it doesn't look like we are waiting for that long before failing.

Sandor · Answer 3 · Wed Apr 15 2020 22:09:12 GMT+0800 (China Standard Time)

hi @moizgh
what's the agent spec you arw using for your pipelines?

Moiz Ghadiyali · Answer 4 · Wed Apr 15 2020 22:23:45 GMT+0800 (China Standard Time)

windows 2016 server machines

Sandor · Answer 5 · Wed Apr 15 2020 22:25:57 GMT+0800 (China Standard Time)

ah so you are using self hosted build agents?

Sandor · Answer 6 · Wed Apr 15 2020 22:28:13 GMT+0800 (China Standard Time)

I could not reproduce the issue so far on my own agents (VSTS or self hosted).

From the logs it actually can't download the version file, It looks like cloudflare is having some issues https://www.cloudflarestatus.com/

Moiz Ghadiyali · Answer 7 · Wed Apr 15 2020 22:57:04 GMT+0800 (China Standard Time)

Yes. We are using self hosted machines. It impacts only certain agents in the pool quite consistently.
Is there retry logic we can introduce to improve the reliability?

Moiz Ghadiyali · Answer 8 · Thu Apr 16 2020 01:17:42 GMT+0800 (China Standard Time)

Added retries to the CI definition. Closing this issue for now. Will monitor progress and re-open if necessary.
Thank you all for the help with the investigation.