成功导入动态库，但依然没有使用GPU
shaoyangxu opened this issue · comments
2021-01-18 03:07:40.679672: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2021-01-18 03:07:40.701768: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:02:00.0 name: Tesla M40 24GB computeCapability: 5.2
coreClock: 1.112GHz coreCount: 24 deviceMemorySize: 22.41GiB deviceMemoryBandwidth: 268.58GiB/s
2021-01-18 03:07:40.701852: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2021-01-18 03:07:40.709300: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2021-01-18 03:07:40.709420: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2021-01-18 03:07:40.712022: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2021-01-18 03:07:40.713083: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2021-01-18 03:07:40.718227: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2021-01-18 03:07:40.719572: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2021-01-18 03:07:40.720228: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2021-01-18 03:07:40.722517: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2021-01-18 03:07:40.722557: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2021-01-18 03:07:41.218571: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-01-18 03:07:41.218648: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267] 0
2021-01-18 03:07:41.218664: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0: N
2021-01-18 03:07:41.222050: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 18066 MB memory) -> physical GPU (device: 0, name: Tesla M40 24GB, pci bus id: 0000:02:00.0, compute capability: 5.2)
2021-01-18 03:07:41.228471: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:196] None of the MLIR optimization passes are enabled (registered 0 passes)
2021-01-18 03:07:41.232040: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2599925000 Hz
2021-01-18 03:07:42.284596: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2021-01-18 03:07:42.496079: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
Epoch: 0001 train_loss= 2.99573 train_acc= 0.05126 val_loss= 2.99111 val_acc= 0.59947 time= 21.22448
Epoch: 0002 train_loss= 2.99065 train_acc= 0.61230 val_loss= 2.97835 val_acc= 0.43059 time= 18.53434
Epoch: 0003 train_loss= 2.97684 train_acc= 0.44437 val_loss= 2.96032 val_acc= 0.40407 time= 18.62268
从nvidida-smi和一轮的训练时间来看，我应该是没有用上GPU，想问一下why...
emm就算我用4块卡，也只是第一块卡占一丢丢显存