theano gpu not working
sedghi opened this issue · comments
Hi,
I've followed instructions of how to run theano gpu using the deepo; unfortuantely i'm not able to run the theano code with gpu. it uses the cpu instead
steps that I took
docker run --gpus all -it ufoym/deepo:theano bash
and I'm running the following test code (from theano documentations)
# Code to test if theano is using GPU or CPU
# Reference: https://stackoverflow.com/questions/34328467/how-can-i-use-my-gpu-on-ipython-notebook
import os
os.environ["MKL_THREADING_LAYER"] = "GNU"
from theano import function, config, shared, sandbox
import theano.tensor as T
import numpy
import time
vlen = 10 * 30 * 768 # 10 x #cores x # threads per core
iters = 1000
rng = numpy.random.RandomState(22)
x = shared(numpy.asarray(rng.rand(vlen), config.floatX))
f = function([], T.exp(x))
print(f.maker.fgraph.toposort())
t0 = time.time()
# if you're using Python3, rename `xrange` to `range` in the following line
for i in range(iters):
r = f()
t1 = time.time()
print("Looping %d times took %f seconds" % (iters, t1 - t0))
print("Result is %s" % (r,))
if numpy.any([isinstance(x.op, T.Elemwise) for x in f.maker.fgraph.toposort()]):
print('Used the cpu')
else:
print('Used the gpu')
I'm getting the following output which says it have used cpu instead of gpu:
/usr/local/lib/python3.6/dist-packages/theano/gpuarray/dnn.py:184: UserWarning: Your cuDNN version is more recent than Theano. If you encounter problems, try updating Theano or downgrading cuDNN to a version >= v5 and <= v7.
warnings.warn("Your cuDNN version is more recent than "
Using cuDNN version 7605 on context None
Mapped name None to device cuda0: TITAN RTX (0000:65:00.0)
[GpuElemwise{exp,no_inplace}(<GpuArrayType<None>(float32, vector)>), HostFromGpu(gpuarray)(GpuElemwise{exp,no_inplace}.0)]
Looping 1000 times took 0.180456 seconds
Result is [1.2317803 1.6187935 1.5227807 ... 2.2077181 2.2996776 1.623233 ]
Used the cpu
here is my nvidia-smi inside my docker
Tue May 5 19:41:17 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.82 Driver Version: 440.82 CUDA Version: 10.2 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 TITAN X (Pascal) Off | 00000000:17:00.0 Off | N/A |
| 23% 28C P8 8W / 250W | 2MiB / 12196MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 TITAN RTX Off | 00000000:65:00.0 On | N/A |
| 41% 33C P8 1W / 280W | 611MiB / 24217MiB | 4% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
+-----------------------------------------------------------------------------+
any help is appreciated
Sorry for the late reply.
By default, there exists a file /root/.theanorc
inside a deepo image with theano
. And it should be OK to use GPU in this case, though theano
is deprecated now. If you simply import theano
in the latest deepo image, there should be a warning like this:
/usr/local/lib/python3.6/dist-packages/theano/gpuarray/dnn.py:184: UserWarning: Your cuDNN version is more recent than Theano. If you encounter problems, try updating Theano or downgrading cuDNN to a version >= v5 and <= v7.
warnings.warn("Your cuDNN version is more recent than "
Using cuDNN version 8005 on context None
Mapped name None to device cuda0: NVIDIA GeForce RTX 2070 (0000:01:00.0)
We think the problem in your case is probably the lack of /root/.theanorc
, especially when a directory from your host is mounted into the /root
and therefore /root/.theanorc
does not exist any more.
You can, of course, create a /root/.theanorc
again with the following settings:
[global]
device = cuda
floatX=float32
or use theano
in this way
THEANO_FLAGS='device=cuda,floatX=float32' python -c "import theano"