run validate.py has a error: failed to allocate xxx from device: CUDA_ERROR_OUT_OF_MEMORY
Depblu opened this issue · comments
log info:
python validate.py --n_proc 2 --exp_dir ../../data/experiments/NGSIM-gail/ --params_filename itr_1000.npz --random_seed 42
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcublas.so.8.0 locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcudnn.so.5 locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcufft.so.8.0 locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcurand.so.8.0 locally
50 vehicles with H = 200
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE3 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE3 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
I tensorflow/core/common_runtime/gpu/gpu_device.cc:885] Found device 0 with properties:
name: GeForce GTX 1080
major: 6 minor: 1 memoryClockRate (GHz) 1.8095
pciBusID 0000:83:00.0
Total memory: 7.93GiB
Free memory: 7.70GiB
W tensorflow/stream_executor/cuda/cuda_driver.cc:590] creating context when one is currently active; existing: 0x641f8c0
I tensorflow/core/common_runtime/gpu/gpu_device.cc:885] Found device 0 with properties:
name: GeForce GTX 1080
major: 6 minor: 1 memoryClockRate (GHz) 1.8095
pciBusID 0000:83:00.0
Total memory: 7.93GiB
Free memory: 7.70GiB
W tensorflow/stream_executor/cuda/cuda_driver.cc:590] creating context when one is currently active; existing: 0x730cd40
I tensorflow/core/common_runtime/gpu/gpu_device.cc:885] Found device 1 with properties:
name: GeForce GTX 1080
major: 6 minor: 1 memoryClockRate (GHz) 1.8095
pciBusID 0000:84:00.0
Total memory: 7.93GiB
Free memory: 7.70GiB
I tensorflow/core/common_runtime/gpu/gpu_device.cc:885] Found device 1 with properties:
name: GeForce GTX 1080
major: 6 minor: 1 memoryClockRate (GHz) 1.8095
pciBusID 0000:84:00.0
Total memory: 7.93GiB
Free memory: 7.70GiB
I tensorflow/core/common_runtime/gpu/gpu_device.cc:906] DMA: 0 1
I tensorflow/core/common_runtime/gpu/gpu_device.cc:916] 0: Y Y
I tensorflow/core/common_runtime/gpu/gpu_device.cc:906] DMA: 0 1
I tensorflow/core/common_runtime/gpu/gpu_device.cc:916] 1: Y Y
I tensorflow/core/common_runtime/gpu/gpu_device.cc:916] 0: Y Y
I tensorflow/core/common_runtime/gpu/gpu_device.cc:916] 1: Y Y
I tensorflow/core/common_runtime/gpu/gpu_device.cc:975] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 1080, pci bus id: 0000:83:00.0)
I tensorflow/core/common_runtime/gpu/gpu_device.cc:975] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 1080, pci bus id: 0000:83:00.0)
I tensorflow/core/common_runtime/gpu/gpu_device.cc:975] Creating TensorFlow device (/gpu:1) -> (device: 1, name: GeForce GTX 1080, pci bus id: 0000:84:00.0)
I tensorflow/core/common_runtime/gpu/gpu_device.cc:975] Creating TensorFlow device (/gpu:1) -> (device: 1, name: GeForce GTX 1080, pci bus id: 0000:84:00.0)
E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 7.32G (7857173248 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 6.58G (7071455744 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 5.93G (6364310016 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 5.33G (5727878656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 4.80G (5155090432 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 4.32G (4639581184 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 3.89G (4175622912 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 3.50G (3758060544 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 3.15G (3382254336 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 2.83G (3044028928 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 2.55G (2739625984 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 2.30G (2465663232 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 2.07G (2219096832 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 1.86G (1997187072 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 1.67G (1797468416 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 1.51G (1617721600 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 1.36G (1455949568 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 1.22G (1310354688 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 1.10G (1179319296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 1012.22M (1061387520 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 911.00M (955248896 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 819.90M (859724032 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 737.91M (773751808 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 664.12M (696376832 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 597.71M (626739200 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 537.93M (564065280 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 484.14M (507658752 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 435.73M (456892928 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 392.15M (411203840 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
pid: 0 traj: 0 / 1031E tensorflow/stream_executor/cuda/cuda_blas.cc:372] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
W tensorflow/stream_executor/stream.cc:1390] attempting to perform BLAS operation using StreamExecutor without BLAS support
E tensorflow/stream_executor/cuda/cuda_blas.cc:372] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
W tensorflow/stream_executor/stream.cc:1390] attempting to perform BLAS operation using StreamExecutor without BLAS support
Traceback (most recent call last):
File "validate.py", line 381, in
random_seed=run_args.random_seed
File "validate.py", line 266, in collect
random_seed=random_seed
File "validate.py", line 188, in parallel_collect_trajectories
[res.get() for res in results]
File "validate.py", line 188, in
[res.get() for res in results]
File "/opt/conda/envs/rllab3/lib/python3.5/multiprocessing/pool.py", line 644, in get
raise self._value
multiprocessing.pool.MaybeEncodingError: Error sending result: '<multiprocessing.pool.ExceptionWithTraceback object at 0x7f23f5742710>'. Reason: 'PicklingError("Can't pickle <class 'module'>: attribute lookup module on builtins failed",)'
signal (15): Terminated
in expression starting at no file:0
do_futex_wait.constprop.3 at /lib/x86_64-linux-gnu/libpthread.so.0 (unknown line)
__new_sem_wait_slow.constprop.0 at /lib/x86_64-linux-gnu/libpthread.so.0 (unknown line)
sem_wait at /lib/x86_64-linux-gnu/libpthread.so.0 (unknown line)
semlock_acquire at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Modules/_multiprocessing/semaphore.c:310
signal (15): Terminated
in expression starting at no file:0
PyCFunction_Call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/methodobject.c:98
call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4758 [inlined]
PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289
_PyEval_EvalCodeWithName at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4071
PyEval_EvalCodeEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4092
function_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/funcobject.c:627
PyObject_Call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/abstract.c:2166
method_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/classobject.c:330
PyObject_Call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/abstract.c:2166
PyObject_CallFunctionObjArgs at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/abstract.c:2446
PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3160
fast_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4856 [inlined]
call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4783 [inlined]
PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289
_PyEval_EvalCodeWithName at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4071
PyEval_EvalCodeEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4092
function_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/funcobject.c:627
PyObject_Call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/abstract.c:2166
ext_do_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:5087 [inlined]
PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3328
fast_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4856 [inlined]
call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4783 [inlined]
PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289
fast_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4856 [inlined]
call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4783 [inlined]
PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289
fast_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4856 [inlined]
call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4783 [inlined]
PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289
_PyEval_EvalCodeWithName at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4071
PyEval_EvalCodeEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4092
at /root/code/julia-1.1.0/bin/../lib/julia/libLLVM-6.0.so (unknown line)
ZN4llvm15ScalarEvolution16isKnownPredicateENS_7CmpInst9PredicateEPKNS_4SCEVES5 at /root/code/julia-1.1.0/bin/../lib/julia/libLLVM-6.0.so (unknown line)
function_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/funcobject.c:627
PyObject_Call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/abstract.c:2166
method_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/classobject.c:330
PyObject_Call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/abstract.c:2166
ZN4llvm15ScalarEvolution25isKnownViaSimpleReasoningENS_7CmpInst9PredicateEPKNS_4SCEVES5 at /root/code/julia-1.1.0/bin/../lib/julia/libLLVM-6.0.so (unknown line)
ZN4llvm15ScalarEvolution27isImpliedCondOperandsHelperENS_7CmpInst9PredicateEPKNS_4SCEVES5_S5_S5 at /root/code/julia-1.1.0/bin/../lib/julia/libLLVM-6.0.so (unknown line)
ZN4llvm15ScalarEvolution21isImpliedCondOperandsENS_7CmpInst9PredicateEPKNS_4SCEVES5_S5_S5 at /root/code/julia-1.1.0/bin/../lib/julia/libLLVM-6.0.so (unknown line)
slot_tp_init at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/typeobject.c:6314
type_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/typeobject.c:905
PyObject_Call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/abstract.c:2166
do_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4989 [inlined]
call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4785 [inlined]
PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289
fast_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4856 [inlined]
call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4783 [inlined]
PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289
fast_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4856 [inlined]
call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4783 [inlined]
PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289
fast_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4856 [inlined]
call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4783 [inlined]
PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289
_PyEval_EvalCodeWithName at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4071
PyEval_EvalCodeEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4092
function_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/funcobject.c:627
PyObject_Call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/abstract.c:2166
method_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/classobject.c:330
PyObject_Call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/abstract.c:2166
ZN4llvm15ScalarEvolution13isImpliedCondENS_7CmpInst9PredicateEPKNS_4SCEVES5_S2_S5_S5 at /root/code/julia-1.1.0/bin/../lib/julia/libLLVM-6.0.so (unknown line)
ZN4llvm15ScalarEvolution27isLoopBackedgeGuardedByCondEPKNS_4LoopENS_7CmpInst9PredicateEPKNS_4SCEVES8 at /root/code/julia-1.1.0/bin/../lib/julia/libLLVM-6.0.so (unknown line)
_ZN4llvm15ScalarEvolution17getSignExtendExprEPKNS_4SCEVEPNS_4TypeEj at /root/code/julia-1.1.0/bin/../lib/julia/libLLVM-6.0.so (unknown line)
slot_tp_init at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/typeobject.c:6314
_ZN4llvm17simplifyUsersOfIVEPNS_7PHINodeEPNS_15ScalarEvolutionEPNS_13DominatorTreeEPNS_8LoopInfoERNS_15SmallVectorImplINS_14WeakTrackingVHEEERNS_12SCEVExpanderEPNS_9IVVisitorE at /root/code/julia-1.1.0/bin/../lib/julia/libLLVM-6.0.so (unknown line)
type_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/typeobject.c:905
PyObject_Call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/abstract.c:2166
do_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4989 [inlined]
call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4785 [inlined]
PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289
_PyEval_EvalCodeWithName at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4071
fast_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4866 [inlined]
call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4783 [inlined]
PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289
_PyEval_EvalCodeWithName at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4071
fast_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4866 [inlined]
call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4783 [inlined]
PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289
_PyEval_EvalCodeWithName at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4071
fast_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4866 [inlined]
call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4783 [inlined]
PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289
_PyEval_EvalCodeWithName at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4071
PyEval_EvalCodeEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4092
PyEval_EvalCode at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:777
run_mod at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/pythonrun.c:991 [inlined]
PyRun_FileExFlags at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/pythonrun.c:944
PyRun_SimpleFileExFlags at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/pythonrun.c:411
run_file at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Modules/main.c:328 [inlined]
Py_Main at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Modules/main.c:779
main at python (unknown line)
_ZN12_GLOBAL__N_114IndVarSimplify17simplifyAndExtendEPN4llvm4LoopERNS1_12SCEVExpanderEPNS1_8LoopInfoE.constprop.426 at /root/code/julia-1.1.0/bin/../lib/julia/libLLVM-6.0.so (unknown line)
_ZN12_GLOBAL__N_114IndVarSimplify3runEPN4llvm4LoopE.part.412 at /root/code/julia-1.1.0/bin/../lib/julia/libLLVM-6.0.so (unknown line)
__libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0x400998)
unknown function (ip: 0xffffffffffffffff)
Allocations: 112352957 (Pool: 112338583; Big: 14374); GC: 238
signal (11): Segmentation fault
in expression starting at no file:0
PyErr_Fetch at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/errors.c:328
PyObject_ClearWeakRefs at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/weakrefobject.c:909
_ZN12_GLOBAL__N_124IndVarSimplifyLegacyPass9runOnLoopEPN4llvm4LoopERNS1_13LPPassManagerE.part.413 at /root/code/julia-1.1.0/bin/../lib/julia/libLLVM-6.0.so (unknown line)
_ZN4llvm13LPPassManager13runOnFunctionERNS_8FunctionE at /root/code/julia-1.1.0/bin/../lib/julia/libLLVM-6.0.so (unknown line)
_ZN4llvm13FPPassManager13runOnFunctionERNS_8FunctionE at /root/code/julia-1.1.0/bin/../lib/julia/libLLVM-6.0.so (unknown line)
_ZN4llvm13FPPassManager11runOnModuleERNS_6ModuleE at /root/code/julia-1.1.0/bin/../lib/julia/libLLVM-6.0.so (unknown line)
ZN4llvm6legacy15PassManagerImpl3runERNS_6ModuleE at /root/code/julia-1.1.0/bin/../lib/julia/libLLVM-6.0.so (unknown line)
array_dealloc at /home/ilan/minonda/conda-bld/numpy-1.12_1486268766587/work/numpy-1.12.0/numpy/core/src/multiarray/arrayobject.c:384
pydecref at /root/.julia/packages/PyCall/ttONZ/src/PyCall.jl:121
pydecref at /root/.julia/packages/PyCall/ttONZ/src/PyCall.jl:126
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2219
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1571 [inlined]
run_finalizer at /buildworker/worker/package_linux64/build/src/gc.c:232
jl_gc_run_finalizers_in_list at /buildworker/worker/package_linux64/build/src/gc.c:318
jl_gc_run_all_finalizers at /buildworker/worker/package_linux64/build/src/gc.c:346
jl_atexit_hook at /buildworker/worker/package_linux64/build/src/init.c:249
jl_exit at /buildworker/worker/package_linux64/build/src/jl_uv.c:592
jl_exit_thread0_cb at /buildworker/worker/package_linux64/build/src/signals-unix.c:344
unknown function (ip: 0xffffffffffffffff)
Allocations: 112353419 (Pool: 112339045; Big: 14374); GC: 238
operator() at /buildworker/worker/package_linux64/build/src/jitlayers.cpp:335
E tensorflow/stream_executor/cuda/cuda_blas.cc:372] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
W tensorflow/stream_executor/stream.cc:1390] attempting to perform BLAS operation using StreamExecutor without BLAS support
addModule at /buildworker/worker/package_linux64/build/usr/include/llvm/ExecutionEngine/Orc/IRCompileLayer.h:57 [inlined]
addModule at /buildworker/worker/package_linux64/build/src/jitlayers.cpp:454
jl_add_to_ee at /buildworker/worker/package_linux64/build/src/jitlayers.cpp:679 [inlined]
jl_finalize_function at /buildworker/worker/package_linux64/build/src/jitlayers.cpp:687
E tensorflow/stream_executor/cuda/cuda_blas.cc:372] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
W tensorflow/stream_executor/stream.cc:1390] attempting to perform BLAS operation using StreamExecutor without BLAS support
getAddressForFunction at /buildworker/worker/package_linux64/build/src/codegen.cpp:1281
jl_generate_fptr at /buildworker/worker/package_linux64/build/src/codegen.cpp:1388
jl_compile_method_internal at /buildworker/worker/package_linux64/build/src/gf.c:1856 [inlined]
jl_fptr_trampoline at /buildworker/worker/package_linux64/build/src/gf.c:1863
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2219
deepcopy_internal at ./deepcopy.jl:78 [inlined]
deepcopy at ./deepcopy.jl:28 [inlined]
get_features at /home/lius/tmp/github/ngsim_env/julia/src/ngsim_env.jl:218
#reset#22 at /home/lius/tmp/github/ngsim_env/julia/src/ngsim_env.jl:125
unknown function (ip: 0x7f23cbe7e1a4)
jl_fptr_trampoline at /buildworker/worker/package_linux64/build/src/gf.c:1864
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2219
#reset at ./none:0 [inlined]
#reset#2 at /home/lius/tmp/github/ngsim_env/julia/src/env.jl:19 [inlined]
#reset at ./none:0
unknown function (ip: 0x7f23cbe7dcb9)
jl_fptr_trampoline at /buildworker/worker/package_linux64/build/src/gf.c:1864
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2219
f_kw_closure at /root/.julia/packages/PyCall/ttONZ/src/callback.jl:36
jl_fptr_trampoline at /buildworker/worker/package_linux64/build/src/gf.c:1864
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2219
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1571 [inlined]
jl_f__apply at /buildworker/worker/package_linux64/build/src/builtins.c:556
jl_f__apply_latest at /buildworker/worker/package_linux64/build/src/builtins.c:594
_pyjlwrap_call at /root/.julia/packages/PyCall/ttONZ/src/callback.jl:37
unknown function (ip: 0x7f23cbe1c443)
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2219
pyjlwrap_call at /root/.julia/packages/PyCall/ttONZ/src/callback.jl:49
unknown function (ip: 0x7f23cbe1bd76)
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2219
unknown function (ip: 0x7f23cbddbafc)
unknown function (ip: 0x7f23cbddb9ce)
PyObject_Call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/abstract.c:2166
ext_do_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:5087 [inlined]
PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3328
_PyEval_EvalCodeWithName at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4071
PyEval_EvalCodeEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4092
function_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/funcobject.c:627
PyObject_Call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/abstract.c:2166
ext_do_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:5087 [inlined]
PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3328
_PyEval_EvalCodeWithName at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4071
PyEval_EvalCodeEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4092
function_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/funcobject.c:627
PyObject_Call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/abstract.c:2166
ext_do_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:5087 [inlined]
PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3328
_PyEval_EvalCodeWithName at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4071
PyEval_EvalCodeEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4092
function_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/funcobject.c:627
PyObject_Call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/abstract.c:2166
ext_do_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:5087 [inlined]
PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3328
_PyEval_EvalCodeWithName at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4071
fast_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4866 [inlined]
call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4783 [inlined]
PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289
_PyEval_EvalCodeWithName at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4071
PyEval_EvalCodeEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4092
function_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/funcobject.c:627
PyObject_Call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/abstract.c:2166
ext_do_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:5087 [inlined]
PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3328
_PyEval_EvalCodeWithName at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4071
PyEval_EvalCodeEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4092
function_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/funcobject.c:627
PyObject_Call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/abstract.c:2166
ext_do_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:5087 [inlined]
PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3328
fast_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4856 [inlined]
call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4783 [inlined]
PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289
fast_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4856 [inlined]
call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4783 [inlined]
PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289
fast_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4856 [inlined]
call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4783 [inlined]
PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289
_PyEval_EvalCodeWithName at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4071
PyEval_EvalCodeEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4092
function_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/funcobject.c:627
PyObject_Call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/abstract.c:2166
method_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/classobject.c:330
PyObject_Call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/abstract.c:2166
slot_tp_init at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/typeobject.c:6314
type_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/typeobject.c:905
PyObject_Call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/abstract.c:2166
do_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4989 [inlined]
call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4785 [inlined]
PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289
fast_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4856 [inlined]
call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4783 [inlined]
PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289
fast_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4856 [inlined]
call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4783 [inlined]
PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289
fast_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4856 [inlined]
call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4783 [inlined]
PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289
_PyEval_EvalCodeWithName at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4071
PyEval_EvalCodeEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4092
function_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/funcobject.c:627
PyObject_Call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/abstract.c:2166
method_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/classobject.c:330
PyObject_Call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/abstract.c:2166
slot_tp_init at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/typeobject.c:6314
type_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/typeobject.c:905
PyObject_Call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/abstract.c:2166
do_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4989 [inlined]
call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4785 [inlined]
PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289
_PyEval_EvalCodeWithName at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4071
fast_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4866 [inlined]
call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4783 [inlined]
PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289
_PyEval_EvalCodeWithName at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4071
fast_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4866 [inlined]
call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4783 [inlined]
PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289
_PyEval_EvalCodeWithName at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4071
fast_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4866 [inlined]
call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4783 [inlined]
PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289
_PyEval_EvalCodeWithName at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4071
PyEval_EvalCodeEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4092
PyEval_EvalCode at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:777
run_mod at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/pythonrun.c:991 [inlined]
PyRun_FileExFlags at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/pythonrun.c:944
PyRun_SimpleFileExFlags at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/pythonrun.c:411
run_file at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Modules/main.c:328 [inlined]
Py_Main at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Modules/main.c:779
main at python (unknown line)
__libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0x400998)
unknown function (ip: 0xffffffffffffffff)
Allocations: 112471597 (Pool: 112457178; Big: 14419); GC: 238
signal (11): Segmentation fault
in expression starting at no file:0
PyErr_Fetch at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/errors.c:328
PyObject_ClearWeakRefs at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/weakrefobject.c:909
array_dealloc at /home/ilan/minonda/conda-bld/numpy-1.12_1486268766587/work/numpy-1.12.0/numpy/core/src/multiarray/arrayobject.c:384
pydecref_ at /root/.julia/packages/PyCall/ttONZ/src/PyCall.jl:121
pydecref at /root/.julia/packages/PyCall/ttONZ/src/PyCall.jl:126
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2219
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1571 [inlined]
run_finalizer at /buildworker/worker/package_linux64/build/src/gc.c:232
jl_gc_run_finalizers_in_list at /buildworker/worker/package_linux64/build/src/gc.c:318
jl_gc_run_all_finalizers at /buildworker/worker/package_linux64/build/src/gc.c:346
jl_atexit_hook at /buildworker/worker/package_linux64/build/src/init.c:249
jl_exit at /buildworker/worker/package_linux64/build/src/jl_uv.c:592
jl_exit_thread0_cb at /buildworker/worker/package_linux64/build/src/signals-unix.c:344
unknown function (ip: 0xffffffffffffffff)
Allocations: 112472059 (Pool: 112457640; Big: 14419); GC: 238
I encountered a similar issue (CUDA OUT OF MEMORY). One simple solution is just to uninstalled the tensorflow-gpu and reinstalled the tensorflow (so they will not use CUDA). There should be a more elegant way to handle this.