TensorflowSharp with Tensorflow-GPU
Kjos opened this issue · comments
Describe the bug
I have TensorflowSharp running successfully on CPU; my model gives expected output.
I have Tensorflow 1.14 running successfully locally on GPU using CUDA 10.0 and CUDNN 7.6 (with python).
I would like to run TensorflowSharp on GPU, so replaced the libtensorflow.dll with _pywrap_tensorflow_internal.pyd (renamed to libtensorflow.dll) in 'packages\TensorFlowSharp.1.13.0\runtimes\win7-x64'. I also set the compile architecture to x64.
It loads the dll successfully, however it will always throw a CUDNN mismatch error.
To Reproduce
See above.
Note: I do not have time to create models, train models, create samples from scratch and then adding your 3-4 lines of code to a sample to debug your problem. You need to provide a complete test case.
Expected behavior
I would figure it would run on GPU fine, since it runs fine from python as well.
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):
- OS: Windows 10
- Browser [e.g. chrome, safari]
- Version [e.g. 22]
Smartphone (please complete the following information):
- Device: [e.g. iPhone6]
- OS: [e.g. iOS8.1]
- Browser [e.g. stock browser, safari]
- Version [e.g. 22]
Additional context
Add any other context about the problem here.
I filed this as a bug, however I can see how it is perhaps more so a feature as GPU support is not yet implemented.
Am I missing a DLL reference? The other threads about GPU support don't seem to have encountered this issue.
Error that is thrown:
(0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[{{node convolution}}]]
[[resize_4/ResizeBilinear/_3]]
(1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[{{node convolution}}]]
Note for others: replacing libtensorflow.dll in 'packages' directory requires rebuilding your solution.
Got it to work with 10! It's a bug within CUDA, not TF#.
You need to set the ProtoConfig to Allow Growth.
TFSessionOptions options = new TFSessionOptions();
unsafe
{
byte[] GPUConfig = new byte[] { 0x32, 0x02, 0x20, 0x01 };
fixed (void* ptr = &GPUConfig[0])
{
options.SetConfig(new IntPtr(ptr), GPUConfig.Length);
}
}
this.session = new TFSession(this.graph, options);
The speedup is amazing!
Got it to work with 10! It's a bug within CUDA, not TF#.
You need to set the ProtoConfig to Allow Growth.TFSessionOptions options = new TFSessionOptions(); unsafe { byte[] GPUConfig = new byte[] { 0x32, 0x02, 0x20, 0x01 }; fixed (void* ptr = &GPUConfig[0]) { options.SetConfig(new IntPtr(ptr), GPUConfig.Length); } } this.session = new TFSession(this.graph, options);The speedup is amazing!
What do I need to install to run GPU?