TF examples take over all GPU memory

[Xingjian Shi]

I find that if we run the TF example, it will try to take over all of the available GPU memory (Refer: https://www.tensorflow.org/tutorials/using_gpu). This can cause troubles in public servers where lots of users are sharing the GPUs.

For example, when running klab-12-5-seq2seq.py, the GPU usage could be like this:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 375.26                 Driver Version: 375.26                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 1080    Off  | 0000:02:00.0     Off |                  N/A |
| 17%   47C    P2    44W / 200W |   7786MiB /  8113MiB |      9%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce GTX 1080    Off  | 0000:03:00.0     Off |                  N/A |
|  0%   41C    P2    43W / 200W |   7715MiB /  8113MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  GeForce GTX 1080    Off  | 0000:82:00.0     Off |                  N/A |
|  0%   44C    P2    43W / 200W |   7715MiB /  8113MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

I used the method suggested in https://www.tensorflow.org/tutorials/using_gpu and the memory will be allocated incrementally now.

(Adding the following lines after the import will solve the problem https://github.com/hunkim/DeepLearningZeroToAll/blob/master/Keras/klab-12-5-seq2seq.py#L12)

from keras.utils.vis_utils import plot_model
from keras.backend.tensorflow_backend import set_session
config = tf.ConfigProto()
config.gpu_options.allow_growth = True
set_session(tf.Session(config=config))

The GPU memory usage becomes:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 375.26                 Driver Version: 375.26                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 1080    Off  | 0000:02:00.0     Off |                  N/A |
|  0%   47C    P2    44W / 200W |    294MiB /  8113MiB |     12%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce GTX 1080    Off  | 0000:03:00.0     Off |                  N/A |
|  0%   41C    P8    14W / 200W |    115MiB /  8113MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  GeForce GTX 1080    Off  | 0000:82:00.0     Off |                  N/A |
|  0%   44C    P8    14W / 200W |    115MiB /  8113MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

Is there a way to enable this by default?

[Sung Kim]

Interesting.

Do you mean from keras.utils.vis_utils import plot_model help reduce memory?

How about just running TF without Keras. The same memory issue?

[Mo Kweon]

This is a normal behavior in Tensorflow

By default, TensorFlow maps nearly all of the GPU memory of all GPUs (subject to CUDA_VISIBLE_DEVICES) visible to the process. This is done to more efficiently use the relatively precious GPU memory resources on the devices by reducing memory fragmentation.

Anyway I don't see a problem because

1. For people coming to this repo, a situation like this is very rare.
2. If there isn't enough memory, it will show pool allocations errors. No harm is done
3. This is only a problem when you run Tensorflow first and try to do other things with the GPU later because there will be no memory left lol! However, if the memory is already occupied, Tensorflow won't run if there is not enough memory. (still will run if there is enough memory) In either case, no harm is done.

[Xingjian Shi]

@hunkim Configuring the memory allocation strategy of TF to be "allow_growth=True" can solve the problem.

@kkweon Currently I'm having the third problem. We will need to run multiple jobs on the same GPU or different GPUs. The default memory allocation behavior will be problematic in such scenario.

However, I feel that it's reasonable to ignore it now since most example scripts take less 1 minute to complete. In fact, I haven't noticed the problem until I begin to run the seq2seq example in Keras, which takes longer time.