Jenkins marian-dev-cpu-avx2 #142 failed
marianminion opened this issue · comments
Build 'marian-dev-cpu-avx2' is failing!
Last 50 lines of build output:
[...truncated 50.47 KB...]
[01/26/2022 16:02:34] Test took 00:00:6.595s
[01/26/2022 16:02:34] Checking directory: tests/decoder/shortlist
[01/26/2022 16:02:34] Running setup script
[01/26/2022 16:02:34] Running tests/decoder/shortlist/test_shortlist_rnn_cpu.sh ... OK
[01/26/2022 16:02:44] Test took 00:00:10.479s
[01/26/2022 16:02:44] Checking directory: tests/decoder/wmt16
[01/26/2022 16:02:44] Running setup script
[01/26/2022 16:02:44] Running tests/decoder/wmt16/test_ende_cpu.sh ... OK
[01/26/2022 16:04:08] Test took 00:01:23.374s
[01/26/2022 16:04:08] Checking directory: tests/models/transformer
[01/26/2022 16:04:08] Running setup script
[01/26/2022 16:04:08] Running tests/models/transformer/test_hard_aligns_cpu.sh ... OK
[01/26/2022 16:04:20] Test took 00:00:12.486s
[01/26/2022 16:04:20] Checking directory: tests/models/wngt19
[01/26/2022 16:04:20] Running setup script
[01/26/2022 16:04:21] Running tests/models/wngt19/test_model_base_fbgemm_packed16.sh ... OK
[01/26/2022 16:04:34] Test took 00:00:13.753s
[01/26/2022 16:04:34] Running tests/models/wngt19/test_model_base_fbgemm_packed8.sh ... OK
[01/26/2022 16:04:45] Test took 00:00:11.122s
[01/26/2022 16:04:45] Checking directory: tests/models/wnmt18
[01/26/2022 16:04:45] Running setup script
[01/26/2022 16:04:46] Running tests/models/wnmt18/test_student_small.sh ... OK
[01/26/2022 16:04:57] Test took 00:00:11.293s
[01/26/2022 16:04:57] Running tests/models/wnmt18/test_student_small_aan.sh ... OK
[01/26/2022 16:05:07] Test took 00:00:10.538s
[01/26/2022 16:05:07] Running tests/models/wnmt18/test_student_small_aan_intgemm16.sh ... OK
[01/26/2022 16:05:16] Test took 00:00:9.061s
[01/26/2022 16:05:16] Running tests/models/wnmt18/test_student_small_aan_intgemm8.sh ... OK
[01/26/2022 16:05:25] Test took 00:00:8.078s
[01/26/2022 16:05:25] Checking directory: tests/scorer/scores
[01/26/2022 16:05:25] Running setup script
[01/26/2022 16:05:25] Running tests/scorer/scores/test_scores_cpu.sh ... OK
[01/26/2022 16:05:52] Test took 00:00:26.879s
[01/26/2022 16:05:52] Checking directory: tests/server
[01/26/2022 16:05:52] Running setup script
[01/26/2022 16:05:52] Running tests/server/test_ende_cpu.sh ... OK
[01/26/2022 16:06:32] Test took 00:00:40.135s
[01/26/2022 16:06:32] Checking directory: tests/training/restoring/multi-gpu
[01/26/2022 16:06:32] Running setup script
[01/26/2022 16:06:32] Running tests/training/restoring/multi-gpu/test_adam_sync_cpu.sh ... OK
[01/26/2022 16:07:01] Test took 00:00:28.887s
---------------------
Failed:
- tests/decoder/align/test_align_cpu.sh
Logs:
- /var/lib/jenkins/workspace/marian-dev-cpu-avx2/regression-tests/tests/decoder/align/test_align_cpu.sh.log
---------------------
Ran 19 tests in 00:05:20.578s, 18 passed, 0 skipped, 1 failed
Build step 'Execute shell' marked build as failure
Changes since last successful build:
After investigation, it happens because the machine is heavily loaded with many processes, so marian-scorer runs out of RAM memory and is killed.
Build was fixed!