Cuda Dimension Mismatch
DrChromaticNo opened this issue · comments
when running this command
python2.7 make_image_analogy.py ~/Documents/imganal/examples/images/arch-A.jpg ~/Documents/imganal/examples/images/arch-Ap.jpg ~/Documents/imganal/examples/images/arch-B.jpg ~/Documents/imganal/out/img
on the arch example it runs for one pass (0x0 through 0x4) and then I get the following trace:
Traceback (most recent call last):
File "make_image_analogy.py", line 25, in
image_analogy.main.main(args, model_class)
File "build/bdist.linux-x86_64/egg/image_analogy/main.py", line 69, in main
model.build(a_image, ap_image, b_image, (1, img_num_channels, img_height, img_width))
File "build/bdist.linux-x86_64/egg/image_analogy/models/base.py", line 23, in build
File "build/bdist.linux-x86_64/egg/image_analogy/models/analogy.py", line 22, in build_loss
File "build/bdist.linux-x86_64/egg/image_analogy/models/base.py", line 51, in precompute_static_features
File "build/bdist.linux-x86_64/egg/image_analogy/models/base.py", line 60, in get_features
File "/usr/local/lib/python2.7/dist-packages/keras/backend/theano_backend.py", line 384, in call
return self.function(*inputs)
File "/usr/local/lib/python2.7/dist-packages/theano/compile/function_module.py", line 871, in call
storage_map=getattr(self.fn, 'storage_map', None))
File "/usr/local/lib/python2.7/dist-packages/theano/gof/link.py", line 314, in raise_with_op
reraise(exc_type, exc_value, exc_trace)
File "/usr/local/lib/python2.7/dist-packages/theano/compile/function_module.py", line 859, in call
outputs = self.fn()
ValueError: CudaNdarray_CopyFromCudaNdarray: need same dimensions for dim 2, destination=290, source=289
Apply node that caused the error: GpuIncSubtensor{InplaceSet;::, ::, int64:int64:, int64:int64:}(GpuAlloc{memset_0=True}.0, GpuElemwise{Composite{(i0 * ((i1 + i2) + Abs((i1 + i2))))}}[(0, 1)].0, Constant{1}, Constant{291}, Constant{1}, Constant{201})
Toposort index: 46
Inputs types: [CudaNdarrayType(float32, 4D), CudaNdarrayType(float32, 4D), Scalar(int64), Scalar(int64), Scalar(int64), Scalar(int64)]
Inputs shapes: [(1, 64, 292, 202), (1, 64, 289, 200), (), (), (), ()]
Inputs strides: [(0, 58984, 202, 1), (0, 57800, 200, 1), (), (), (), ()]
Inputs values: ['not shown', 'not shown', 1, 291, 1, 201]
Outputs clients: [[GpuContiguous(GpuIncSubtensor{InplaceSet;::, ::, int64:int64:, int64:int64:}.0)]]
all my requirements are up to date/correct (or at least pip install -r requirements.txt says so). I've had this project installed for awhile so it might be that the cruft from older versions is conflicting with this one (it's not in a venv or anything). I think it has something to do with the img heights and widths. If I run with an example where A is not the same dimensions as B, it crashes before the first pass... I vaguely remember this problem happening on an older version of this project but I forget how I fixed it. I'm not sure if it's me or the new version, any ideas?
Thanks for reporting. I remember there being some random issues with different sized inputs and theano CPU mode but it doesn't look like either apply here. Could you try with --a-scale-mode=match? Also it might be worth installing in a venv to make sure there's not some random garbage messing it up.
It works just fine with --a-scale-mode=match! wow!
For the record, I'm running in theano gpu mode
Cool glad it works! Could you tell me a little bit more about your setup: OS, CUDA version, are you using cuDNN?
BTW a-scale-mode forces both the A images and the B images to be the same size. I thought I had worked around this bug so this version I switched the default to ratio which scales down A the same amount B is scaled so that aspect ratio is preserved.
OS: Ubuntu 14.04 LTS, 64-bit
CUDA verison: (from nvcc --version): Built on Tue_Aug_11_14:27:32_CDT_2015
Cuda compilation tools, release 7.5, V7.5.17
cuDNN is not availible
Ah ha I was able to reproduce it by disabling cuDNN. I've pushed a new version to here and pypi which automatically sets a-scale-mode = 'match' if that config is detected. Thanks again for your help diagnosing this.
glad to help