Since 2.3.0, webpack builds on CircleCI fail with "Error: Call retries were exceeded"
happypoulp opened this issue · comments
- Operating System: ubuntu:18.04 (from CircleCI)
- Node Version: v10.13.0
- NPM Version: 6.10.0
- webpack Version: 4.29.6
- terser-webpack-plugin Version: >= 2.3.0
Expected Behavior
Building webpack JS bundles on CircleCI should be working the same for version 2.2.3 and versions >= 2.3.0.
Actual Behavior
Builds succeeds for terser-webpack-plugin v2.2.3 and are not working for terser-webpack-plugin >= 2.3.0 (tested with v2.3.0 and v2.3.1).
The error returned when running a webpack command such as: webpack --mode production -p --progress --config webpack.config.js with terser-webpack-plugin v2.3.0 produces this error:
ERROR in javascripts/bundle/feature.js from Terser
Error: Call retries were exceeded
at ChildProcessWorker.initialize (/root/project/node_modules/terser-webpack-plugin/node_modules/jest-worker/build/workers/ChildProcessWorker.js:193:21)
at ChildProcessWorker.onExit (/root/project/node_modules/terser-webpack-plugin/node_modules/jest-worker/build/workers/ChildProcessWorker.js:263:12)
at ChildProcess.emit (events.js:182:13)
at Process.ChildProcess._handle.onexit (internal/child_process.js:240:12)
and this happens for multiple of the bundles that webpack tries to build.
Code
// Here is the config I am using for Terser plugin:
new TerserPlugin({
cache: true,
parallel: true,
sourceMap: true, // Using source-map is intentional
}),How Do We Reproduce?
The project I am building has more than 30 JS bundles and few of them are rather heavy (the heaviest is 1.6MB, average size is 600KB). Providing a reproducible example would be complicated to simulate the high number of JS files used and the overall heaviness of the bundles (maybe the issue is triggered in the source-map generation step).
But since the issue was introduced in a specific recent release (2.3.0), I hope that it will be easier to pinpoint the issue.
Please not that this issue is not reproducible on my local machine (a very recent MBP running Mac Mojave 10.14.6)
Thanks!
Actually I think my analysis was incorrect. It seems the issue also reproduces with version 2.2.3 (it just doesn't happen when debugging a failing build on CircleCI with SSH, only when running a "real" CircleCI build). Will close this issue as it is probably the same as #143 .
Can you try https://github.com/webpack-contrib/terser-webpack-plugin/releases/tag/v2.3.3? If problem still exists i will reopen the problem, thanks
@evilebottnawi I upgraded in my project to v2.3.3 but still had the same problem. Setting explicitly parallel: 2 resolved the issue.
Can you provide minimum reproducible test repo and information about environment - real cores and memory, config file for CircleCI
I cannot publish the code 😞The app that I'm talking about is quite large (+/- 4k files, 600+ react components, I don't count any external dependencies here).
While building, we compile server and client bundle with npm-run-all.
Job's resource: 2CPU/4096MB
circleci config.yml
version: 2
jobs:
build_dependencies:
docker:
- image: circleci/node:10.16.2-stretch
working_directory: ~/app-frontend
steps:
- checkout
- run: sudo apt-get update && sudo apt-get install -y autoconf libtool pkg-config nasm
- restore_cache:
keys:
- npm-dependencies-{{ checksum "package.json" }}
# fallback to using the latest cache if no exact match is found
- npm-dependencies-
- run: yarn --silent --frozen-lockfile
- save_cache:
key: npm-dependencies-{{ checksum "package.json" }}
paths:
- node_modules
- save_cache:
key: repo-{{ .Environment.CIRCLE_SHA1 }}
paths:
- ~/app-frontend
build:
docker:
- image: circleci/node:10.16.2-stretch
working_directory: ~/app-frontend
steps:
- restore_cache:
key: repo-{{ .Environment.CIRCLE_SHA1 }}
- run: CONFIG=dev yarn build
workflows:
version: 2
build:
jobs:
- build_dependencies:
filters:
branches:
ignore:
- develop
- master
tags:
ignore: /^v.*/
- build:
requires:
- build_dependencies
filters:
branches:
ignore:
- develop
- master
tags:
ignore: /^v.*/package.json
{
"build": "npm-run-all --parallel build:*",
"build:client": "NODE_ENV=production webpack --config webpack/webpack.client.js --env.prod",
"build:server": "NODE_ENV=production webpack --config webpack/webpack.server.js --env.prod"
}webpack terser config
const terserPlugin = new TerserJSPlugin({
terserOptions: {
parse: {
ecma: 8,
},
compress: {
ecma: 5,
warnings: false,
comparisons: false,
inline: 2,
drop_console: true,
},
mangle: {
safari10: true,
},
output: {
ecma: 5,
comments: false,
ascii_only: true,
},
},
parallel: isCircleCi() ? 2 : true,
cache: true,
sourceMap: true,
extractComments: true,
});Thanks for information, what is result of os.cpus() and lscpu?
circleci@<hash>:~$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 36
On-line CPU(s) list: 0-35
Thread(s) per core: 2
Core(s) per socket: 18
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz
Stepping: 4
CPU MHz: 3399.925
BogoMIPS: 6000.00
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 32K
L1i cache: 32K
L2 cache: 1024K
L3 cache: 25344K
NUMA node0 CPU(s): 0-35
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves ida arat pku ospke> os.cpus()[
{
model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
speed: 3397,
times: {
user: 26940600,
nice: 141900,
sys: 6673300,
idle: 84844700,
irq: 0
}
},
{
model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
speed: 3400,
times: {
user: 27778400,
nice: 153300,
sys: 7061100,
idle: 85539900,
irq: 0
}
},
{
model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
speed: 3389,
times: {
user: 28683800,
nice: 128700,
sys: 7030100,
idle: 85136100,
irq: 0
}
},
{
model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
speed: 3440,
times: {
user: 27663600,
nice: 121100,
sys: 6981800,
idle: 86086700,
irq: 0
}
},
{
model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
speed: 3116,
times: {
user: 29731200,
nice: 133300,
sys: 7820900,
idle: 83304900,
irq: 0
}
},
{
model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
speed: 3085,
times: {
user: 30912800,
nice: 134600,
sys: 8423700,
idle: 80364100,
irq: 0
}
},
{
model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
speed: 3355,
times: {
user: 30386700,
nice: 129000,
sys: 7978900,
idle: 82519400,
irq: 0
}
},
{
model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
speed: 3400,
times: {
user: 30117100,
nice: 140200,
sys: 8219200,
idle: 81339400,
irq: 0
}
},
{
model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
speed: 3287,
times: {
user: 29964800,
nice: 122800,
sys: 7932900,
idle: 82947300,
irq: 0
}
},
{
model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
speed: 3204,
times: {
user: 30102100,
nice: 134300,
sys: 8368700,
idle: 81286800,
irq: 0
}
},
{
model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
speed: 3287,
times: {
user: 30583800,
nice: 139200,
sys: 8267000,
idle: 82011400,
irq: 0
}
},
{
model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
speed: 3051,
times: {
user: 31621100,
nice: 118100,
sys: 8382300,
idle: 79922800,
irq: 0
}
},
{
model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
speed: 3399,
times: {
user: 30451100,
nice: 133400,
sys: 7990200,
idle: 82395800,
irq: 0
}
},
{
model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
speed: 3150,
times: {
user: 30496800,
nice: 152600,
sys: 7863100,
idle: 82457000,
irq: 0
}
},
{
model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
speed: 3400,
times: {
user: 30808600,
nice: 124900,
sys: 8456000,
idle: 81011800,
irq: 0
}
},
{
model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
speed: 3361,
times: {
user: 30607600,
nice: 137500,
sys: 7916100,
idle: 82274600,
irq: 0
}
},
{
model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
speed: 3399,
times: {
user: 30067600,
nice: 140000,
sys: 8477200,
idle: 81668900,
irq: 0
}
},
{
model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
speed: 3399,
times: {
user: 31656800,
nice: 118100,
sys: 8421500,
idle: 80762400,
irq: 0
}
},
{
model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
speed: 3395,
times: {
user: 29729600,
nice: 122000,
sys: 9043100,
idle: 80276600,
irq: 0
}
},
{
model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
speed: 3399,
times: {
user: 27666100,
nice: 125000,
sys: 8233300,
idle: 84423800,
irq: 0
}
},
{
model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
speed: 3394,
times: {
user: 28174800,
nice: 121000,
sys: 8073200,
idle: 84135900,
irq: 0
}
},
{
model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
speed: 3400,
times: {
user: 27988900,
nice: 125000,
sys: 8748100,
idle: 82338500,
irq: 0
}
},
{
model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
speed: 3286,
times: {
user: 26778200,
nice: 133100,
sys: 7375700,
idle: 86398500,
irq: 0
}
},
{
model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
speed: 3282,
times: {
user: 26562400,
nice: 124500,
sys: 7311000,
idle: 86705800,
irq: 0
}
},
{
model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
speed: 3282,
times: {
user: 27483200,
nice: 124300,
sys: 7304700,
idle: 85339500,
irq: 0
}
},
{
model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
speed: 3370,
times: {
user: 25486000,
nice: 123700,
sys: 7560800,
idle: 87532900,
irq: 0
}
},
{
model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
speed: 3370,
times: {
user: 26600400,
nice: 127400,
sys: 7662600,
idle: 85278300,
irq: 0
}
},
{
model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
speed: 3395,
times: {
user: 26358500,
nice: 131400,
sys: 7278300,
idle: 86987600,
irq: 0
}
},
{
model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
speed: 3399,
times: {
user: 25715800,
nice: 131500,
sys: 7525700,
idle: 86858100,
irq: 0
}
},
{
model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
speed: 3375,
times: {
user: 25976400,
nice: 142500,
sys: 6956500,
idle: 87683600,
irq: 0
}
},
{
model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
speed: 3379,
times: {
user: 25610900,
nice: 137300,
sys: 7254800,
idle: 87717200,
irq: 0
}
},
{
model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
speed: 3247,
times: {
user: 25417900,
nice: 136900,
sys: 7160800,
idle: 88048800,
irq: 0
}
},
{
model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
speed: 3340,
times: {
user: 26041700,
nice: 121100,
sys: 6681500,
idle: 87907000,
irq: 0
}
},
{
model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
speed: 3400,
times: {
user: 25104700,
nice: 138200,
sys: 6924000,
idle: 88608600,
irq: 0
}
},
{
model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
speed: 3398,
times: {
user: 25153000,
nice: 130900,
sys: 6785800,
idle: 88700200,
irq: 0
}
},
{
model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
speed: 3400,
times: {
user: 24982900,
nice: 151500,
sys: 6926600,
idle: 88630400,
irq: 0
}
}
]Looks CircleCI doesn't provide real available count of CPUs, here the request for improvement https://ideas.circleci.com/ideas/CCI-I-264
Unfortunately, we have no way to get the real value of available 😞 Your solution seems to be the only one right at the moment
👍 It'd be good to document that issue in the docs as a short explanation for parallel option.
Basically the trick with parallel came from @cjlarose #143 (comment)
Thanks for confirming that it's the proper way to resolve that.
@hinok Yes, will be great to documented that, I will do in near future, but you can be a champion 😄
This bug has been haunting us for some time in the https://github.com/automattic/wp-calypso project, so I did a deep dive trying to find the root cause.
And I figured out that the CircleCI container where the build runs is running out of memory, so the Linux kernel will select a "bad" process and kill it with SIGKILL (read here about how the most "bad" process is chosen).
The jest-worker library won't report why the worker process has exited and will merely try to restart it. It gives up after a few attempts (default maxRetries is 3) and the result is the Call retries were exceeded error.
There are other errors you can get that are caused by the same condition. For example, I've also seen this:
internal/child_process.js:394
throw errnoException(err, 'spawn');
^
Error: spawn ENOMEM
at ChildProcess.spawn (internal/child_process.js:394:11)
at spawn (child_process.js:540:9)
at Object.fork (child_process.js:108:10)
at ChildProcessWorker.initialize (/home/circleci/wp-calypso/node_modules/jest-worker/build/workers/ChildProcessWorker.js:137:44)
at ChildProcessWorker.onExit (/home/circleci/wp-calypso/node_modules/jest-worker/build/workers/ChildProcessWorker.js:263:12)
at ChildProcess.emit (events.js:223:5)
at Process.ChildProcess._handle.onexit (internal/child_process.js:272:12) {
errno: 'ENOMEM',
code: 'ENOMEM',
syscall: 'spawn'
}
Here the attempt to restart the process fails: the kernel outright refuses to spawn a new process, because there' not enough memory for it.
The jest-worker library could be improved to report failures better. I don't think that the terser-webpack-plugin itself can do anything better.
@jsnajdr thanks for feedback, can you open an issue in jest repo?
This bug has been haunting us for some time in the https://github.com/automattic/wp-calypso project, so I did a deep dive trying to find the root cause.
And I figured out that the CircleCI container where the build runs is running out of memory, so the Linux kernel will select a "bad" process and kill it with
SIGKILL(read here about how the most "bad" process is chosen).The
jest-workerlibrary won't report why the worker process has exited and will merely try to restart it. It gives up after a few attempts (defaultmaxRetriesis 3) and the result is theCall retries were exceedederror.There are other errors you can get that are caused by the same condition. For example, I've also seen this:
internal/child_process.js:394 throw errnoException(err, 'spawn'); ^ Error: spawn ENOMEM at ChildProcess.spawn (internal/child_process.js:394:11) at spawn (child_process.js:540:9) at Object.fork (child_process.js:108:10) at ChildProcessWorker.initialize (/home/circleci/wp-calypso/node_modules/jest-worker/build/workers/ChildProcessWorker.js:137:44) at ChildProcessWorker.onExit (/home/circleci/wp-calypso/node_modules/jest-worker/build/workers/ChildProcessWorker.js:263:12) at ChildProcess.emit (events.js:223:5) at Process.ChildProcess._handle.onexit (internal/child_process.js:272:12) { errno: 'ENOMEM', code: 'ENOMEM', syscall: 'spawn' }Here the attempt to restart the process fails: the kernel outright refuses to spawn a new process, because there' not enough memory for it.
The
jest-workerlibrary could be improved to report failures better. I don't think that theterser-webpack-pluginitself can do anything better.
We are having exactly the same issue and cannot figure out. Thanks for clarifying what is going on there...
This bug has been haunting us for some time in the https://github.com/automattic/wp-calypso project, so I did a deep dive trying to find the root cause.
And I figured out that the CircleCI container where the build runs is running out of memory, so the Linux kernel will select a "bad" process and kill it with
SIGKILL(read here about how the most "bad" process is chosen).The
jest-workerlibrary won't report why the worker process has exited and will merely try to restart it. It gives up after a few attempts (defaultmaxRetriesis 3) and the result is theCall retries were exceedederror.There are other errors you can get that are caused by the same condition. For example, I've also seen this:
internal/child_process.js:394 throw errnoException(err, 'spawn'); ^ Error: spawn ENOMEM at ChildProcess.spawn (internal/child_process.js:394:11) at spawn (child_process.js:540:9) at Object.fork (child_process.js:108:10) at ChildProcessWorker.initialize (/home/circleci/wp-calypso/node_modules/jest-worker/build/workers/ChildProcessWorker.js:137:44) at ChildProcessWorker.onExit (/home/circleci/wp-calypso/node_modules/jest-worker/build/workers/ChildProcessWorker.js:263:12) at ChildProcess.emit (events.js:223:5) at Process.ChildProcess._handle.onexit (internal/child_process.js:272:12) { errno: 'ENOMEM', code: 'ENOMEM', syscall: 'spawn' }Here the attempt to restart the process fails: the kernel outright refuses to spawn a new process, because there' not enough memory for it.
The
jest-workerlibrary could be improved to report failures better. I don't think that theterser-webpack-pluginitself can do anything better.
I think that this is the best explanation for the problem. I am also getting this error in my current project.
I am trying hard but am not able to solve it. All the details of my problem are in this #12899 issue.
If someone has solved it then please let me know.