Git memory issue with armv7 fanned containers
targos opened this issue · comments
08:51:12 + ssh-agent sh -c 'ssh-add **** && git fetch --no-tags binary_tmp@147.75.72.255:binary_tmp.git +refs/heads/jenkins-node-test-commit-arm-fanned-c7f8b1629123065c20362ca5c2be0552405fbc4a-binary-pi1p/cc-armv7:refs/remotes/jenkins_tmp'
08:51:12 Identity added: **** (****)
08:51:49 fatal: packfile .git/objects/pack/pack-dd8f2842296180e0ffb6376cb0696c662e80f52a.pack cannot be mapped: Cannot allocate memory
08:51:49 fatal: index-pack failed
The .git
folder takes quite a bit of space:
$ du -sh .git
324G .git
I'm not sure how to resolve this other than by wiping the workspace. How can we prevent the repository from growing in the future?
I tried to run git gc
but it also fails with a similar error.
I'm not sure how to resolve this other than by wiping the workspace. How can we prevent the repository from growing in the future?
Maybe we need to be running git gc
more often? I don't know the history but there's stuff commented out wrt gc in https://ci.nodejs.org/job/node-test-binary-armv7l/configure:
FWIW On jenkins-workspace-7 (where the actual git repo that is pulled from lives), we run a cleanup script on a five minute timer in addition to running git-delete-branches if node-test-commit-arm-fanned succeeds (but neither of those affect the local git checkout on the test machines).
build/ansible/roles/jenkins-workspace/tasks/main.yml
Lines 99 to 111 in 172558b
It might be related: some steps of the ARM jobs are taking a VERY long time, e.g. https://ci.nodejs.org/job/node-test-binary-armv7l/9848/RUN_SUBSET=native,nodes=debian11-armv7l/console took 3 hours:
19:04:14 + ssh-agent sh -c 'ssh-add **** && git fetch --no-tags binary_tmp@147.75.72.255:binary_tmp.git +refs/heads/jenkins-node-test-commit-arm-fanned-201ac057dc753c50d1f6fabc2c14ecc4a2bb9dd3-binary-pi1p/cc-armv7:refs/remotes/jenkins_tmp'
19:04:14 Identity added: **** (****)
22:18:59 From 147.75.72.255:binary_tmp
22:18:59 + eee6517716a8...89af234f8ab4 jenkins-node-test-commit-arm-fanned-201ac057dc753c50d1f6fabc2c14ecc4a2bb9dd3-binary-pi1p/cc-armv7 -> jenkins_tmp (forced update)