FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory

[Basit]

I have moved my server from AWS to another company, now I keep getting following error. It works fine if I put a small number of files, but as soon as the files are for the complete website, it throws this error which it did not use to do on amazon ec2 server.

code:

gulp.task('publish', function() {

    var publisher = awspublish.create({
        region: config.awsS3Region,
        params: {
            Bucket: config.awsS3Bucket
        },
        credentials: {
            accessKeyId: config.awsS3AccessKeyId,
            secretAccessKey: config.awsS3SecretAccessKey
        }
    }, {
        cacheFileName: 's3-publish-assets-cache-list.json'
    });

    var headers = {
        'Cache-Control': 'max-age=315360000, no-transform, public'
    };

    return gulp.src(['web/**/**/**/*.{html,xml,js,css,gz,otf,eot,svg,ttf,woff,woff2,jpeg,jpg,png,gif,ico}'])
        .pipe(plugins.rename(function (path) {
            path.dirname = '/static/'+path.dirname;
        }))
        .pipe(parallelize(publisher.publish(headers), 10))
        .pipe(publisher.cache())
        .pipe(awspublish.reporter());
});

How can I fix this?

ubuntu@prod:~/prod$ gulp publish
[22:44:44] Using gulpfile ~/prod/gulpfile.js
[22:44:44] Starting 'publish'...
[22:44:45] [cache] static/safari-pinned-tab.svg
[22:44:53] [cache] static/android-chrome-192x192.png

<--- Last few GCs --->

[4615:0x2cdc510] 174495 ms: Mark-sweep 1789.5 (1855.3) -> 1789.2 (1857.8) MB, 1799.6 / 0.0 ms allocation failure GC in old space requested
[4615:0x2cdc510] 176438 ms: Mark-sweep 1789.2 (1857.8) -> 1789.1 (1826.8) MB, 1942.6 / 0.0 ms last resort GC in old space requested
[4615:0x2cdc510] 178171 ms: Mark-sweep 1789.1 (1826.8) -> 1789.1 (1826.8) MB, 1732.0 / 0.0 ms last resort GC in old space requested

<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x21c695c255e9
1: inflight [/home/ubuntu/prod/node_modules/inflight/inflight.js:~7] [pc=0xabb78e40656](this=0x2a5c739866f1 ,key=0x14798d37dff9 <String[111]: readdir\x00/home/ubuntu/prod/web/i/assets/uploads/2018/02/CroppedFocusedImage1200500-Norwich-low-1024x427.jpg\x00true>,cb=0x14798d37dfb1 <JSFunction (sfi = 0x192654d69ac1)>)
2: _readdir [/home/ubuntu/prod/node_modules/glob-stream/...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
1: node::Abort() [gulp publish]
2: 0x8d05bc [gulp publish]
3: v8::Utils::ReportOOMFailure(char const*, bool) [gulp publish]
4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [gulp publish]
5: v8::internal::Factory::NewFixedArray(int, v8::internal::PretenureFlag) [gulp publish]
6: v8::internal::HashTable<v8::internal::StringTable, v8::internal::StringTableShape>::NewInternal(v8::internal::Isolate*, int, v8::internal::PretenureFlag) [gulp publish]
7: v8::internal::HashTable<v8::internal::StringTable, v8::internal::StringTableShape>::New(v8::internal::Isolate*, int, v8::internal::PretenureFlag, v8::internal::MinimumCapacity) [gulp publish]
8: v8::internal::HashTable<v8::internal::StringTable, v8::internal::StringTableShape>::EnsureCapacity(v8::internal::Handlev8::internal::StringTable, int, v8::internal::PretenureFlag) [gulp publish]
9: v8::internal::StringTable::LookupString(v8::internal::Isolate*, v8::internal::Handlev8::internal::String) [gulp publish]
10: 0x104018e [gulp publish]
11: v8::internal::Runtime_KeyedGetProperty(int, v8::internal::Object**, v8::internal::Isolate*) [gulp publish]
12: 0xabb78c842fd
Aborted (core dumped)

[Gilles Piou]

Are your assets particularly heavy? I don't think this plugin retaining enough things in memory to cause this type of error. I would recommend investigating in a different area (maybe the max_old_space_size argument for Node, the memory available on your server...)

[Basit]

I have already tried max_old_space_size https://stackoverflow.com/questions/37272998/gulp-call-and-retry-last-allocation-failed-process-out-of-memory

It halts the complete system and CPU usage gets very high, but even after a long period of time, it continues to do still go through it.

I had the same server with the same assets on AWS and process used to take a minute or two, but after moving the server the process has become a pain unless there are only a few files.

[Basit]

There was a symbolic link in the project, which adds WordPress files as well, but not sure why it would make difference since we are only fetching images and styles only.

after removing the symbolic link, everything works fine for now on the local test, will update again, if online test still persists same.