Out of Memory?
zserlin opened this issue · comments
Has anyone run into this issue? I execute all of the commands as instructed but when I get to the execution of "python load_caffe_weights.py" I get the following error:
F0618 16:07:32.816975 43606 cudnn_conv_layer.cpp:52] Check failed: error == cudaSuccess (2 vs. 0) out of memory
*** Check failure stack trace: ***
Aborted (core dumped)
Here is the full runtime message:
$ python load_caffe_weights.py
...
I0618 16:07:31.268625 43606 layer_factory.hpp:77] Creating layer input
I0618 16:07:31.268640 43606 net.cpp:100] Creating Layer input
I0618 16:07:31.268646 43606 net.cpp:408] input -> data
I0618 16:07:31.268671 43606 net.cpp:150] Setting up input
I0618 16:07:31.268677 43606 net.cpp:157] Top shape: 1 3 300 300 (270000)
I0618 16:07:31.268682 43606 net.cpp:165] Memory required for data: 1080000
I0618 16:07:31.268687 43606 layer_factory.hpp:77] Creating layer data_input_0_split
I0618 16:07:31.268697 43606 net.cpp:100] Creating Layer data_input_0_split
I0618 16:07:31.268702 43606 net.cpp:434] data_input_0_split <- data
I0618 16:07:31.268708 43606 net.cpp:408] data_input_0_split -> data_input_0_split_0
I0618 16:07:31.268718 43606 net.cpp:408] data_input_0_split -> data_input_0_split_1
I0618 16:07:31.268726 43606 net.cpp:408] data_input_0_split -> data_input_0_split_2
I0618 16:07:31.268734 43606 net.cpp:408] data_input_0_split -> data_input_0_split_3
I0618 16:07:31.268743 43606 net.cpp:408] data_input_0_split -> data_input_0_split_4
I0618 16:07:31.268749 43606 net.cpp:408] data_input_0_split -> data_input_0_split_5
I0618 16:07:31.268757 43606 net.cpp:408] data_input_0_split -> data_input_0_split_6
I0618 16:07:31.268767 43606 net.cpp:150] Setting up data_input_0_split
I0618 16:07:31.268774 43606 net.cpp:157] Top shape: 1 3 300 300 (270000)
I0618 16:07:31.268779 43606 net.cpp:157] Top shape: 1 3 300 300 (270000)
I0618 16:07:31.268785 43606 net.cpp:157] Top shape: 1 3 300 300 (270000)
I0618 16:07:31.268791 43606 net.cpp:157] Top shape: 1 3 300 300 (270000)
I0618 16:07:31.268796 43606 net.cpp:157] Top shape: 1 3 300 300 (270000)
I0618 16:07:31.268802 43606 net.cpp:157] Top shape: 1 3 300 300 (270000)
I0618 16:07:31.268807 43606 net.cpp:157] Top shape: 1 3 300 300 (270000)
I0618 16:07:31.268812 43606 net.cpp:165] Memory required for data: 8640000
I0618 16:07:31.268816 43606 layer_factory.hpp:77] Creating layer Conv
I0618 16:07:31.268827 43606 net.cpp:100] Creating Layer Conv
I0618 16:07:31.268833 43606 net.cpp:434] Conv <- data_input_0_split_0
I0618 16:07:31.268841 43606 net.cpp:408] Conv -> Conv
I0618 16:07:31.691318 43606 net.cpp:150] Setting up Conv
I0618 16:07:31.691352 43606 net.cpp:157] Top shape: 1 32 150 150 (720000)
I0618 16:07:31.691355 43606 net.cpp:165] Memory required for data: 11520000
I0618 16:07:31.691367 43606 layer_factory.hpp:77] Creating layer Conv/relu
I0618 16:07:31.691376 43606 net.cpp:100] Creating Layer Conv/relu
I0618 16:07:31.691380 43606 net.cpp:434] Conv/relu <- Conv
I0618 16:07:31.691385 43606 net.cpp:395] Conv/relu -> Conv (in-place)
I0618 16:07:31.691552 43606 net.cpp:150] Setting up Conv/relu
I0618 16:07:31.691560 43606 net.cpp:157] Top shape: 1 32 150 150 (720000)
I0618 16:07:31.691563 43606 net.cpp:165] Memory required for data: 14400000
I0618 16:07:31.691570 43606 layer_factory.hpp:77] Creating layer conv/depthwise
I0618 16:07:31.691586 43606 net.cpp:100] Creating Layer conv/depthwise
I0618 16:07:31.691591 43606 net.cpp:434] conv/depthwise <- Conv
I0618 16:07:31.691598 43606 net.cpp:408] conv/depthwise -> conv/depthwise
I0618 16:07:31.709228 43606 net.cpp:150] Setting up conv/depthwise
I0618 16:07:31.709251 43606 net.cpp:157] Top shape: 1 32 150 150 (720000)
I0618 16:07:31.709254 43606 net.cpp:165] Memory required for data: 17280000
I0618 16:07:31.709265 43606 layer_factory.hpp:77] Creating layer conv/depthwise/relu
I0618 16:07:31.709275 43606 net.cpp:100] Creating Layer conv/depthwise/relu
I0618 16:07:31.709278 43606 net.cpp:434] conv/depthwise/relu <- conv/depthwise
I0618 16:07:31.709283 43606 net.cpp:395] conv/depthwise/relu -> conv/depthwise (in-place)
I0618 16:07:31.709456 43606 net.cpp:150] Setting up conv/depthwise/relu
I0618 16:07:31.709463 43606 net.cpp:157] Top shape: 1 32 150 150 (720000)
I0618 16:07:31.709467 43606 net.cpp:165] Memory required for data: 20160000
I0618 16:07:31.709472 43606 layer_factory.hpp:77] Creating layer conv/project
I0618 16:07:31.709491 43606 net.cpp:100] Creating Layer conv/project
I0618 16:07:31.709496 43606 net.cpp:434] conv/project <- conv/depthwise
I0618 16:07:31.709503 43606 net.cpp:408] conv/project -> conv/project
I0618 16:07:31.710234 43606 net.cpp:150] Setting up conv/project
I0618 16:07:31.710247 43606 net.cpp:157] Top shape: 1 16 150 150 (360000)
I0618 16:07:31.710253 43606 net.cpp:165] Memory required for data: 21600000
I0618 16:07:31.710264 43606 layer_factory.hpp:77] Creating layer conv_1/expand
I0618 16:07:31.710279 43606 net.cpp:100] Creating Layer conv_1/expand
I0618 16:07:31.710284 43606 net.cpp:434] conv_1/expand <- conv/project
I0618 16:07:31.710294 43606 net.cpp:408] conv_1/expand -> conv_1/expand
I0618 16:07:31.710984 43606 net.cpp:150] Setting up conv_1/expand
I0618 16:07:31.710995 43606 net.cpp:157] Top shape: 1 96 150 150 (2160000)
I0618 16:07:31.711000 43606 net.cpp:165] Memory required for data: 30240000
I0618 16:07:31.711009 43606 layer_factory.hpp:77] Creating layer conv_1/expand/relu
I0618 16:07:31.711021 43606 net.cpp:100] Creating Layer conv_1/expand/relu
I0618 16:07:31.711026 43606 net.cpp:434] conv_1/expand/relu <- conv_1/expand
I0618 16:07:31.711035 43606 net.cpp:395] conv_1/expand/relu -> conv_1/expand (in-place)
I0618 16:07:31.711334 43606 net.cpp:150] Setting up conv_1/expand/relu
I0618 16:07:31.711343 43606 net.cpp:157] Top shape: 1 96 150 150 (2160000)
I0618 16:07:31.711347 43606 net.cpp:165] Memory required for data: 38880000
I0618 16:07:31.711352 43606 layer_factory.hpp:77] Creating layer conv_1/depthwise
I0618 16:07:31.711367 43606 net.cpp:100] Creating Layer conv_1/depthwise
I0618 16:07:31.711372 43606 net.cpp:434] conv_1/depthwise <- conv_1/expand
I0618 16:07:31.711381 43606 net.cpp:408] conv_1/depthwise -> conv_1/depthwise
I0618 16:07:31.769933 43606 net.cpp:150] Setting up conv_1/depthwise
I0618 16:07:31.769959 43606 net.cpp:157] Top shape: 1 96 75 75 (540000)
I0618 16:07:31.769963 43606 net.cpp:165] Memory required for data: 41040000
I0618 16:07:31.769979 43606 layer_factory.hpp:77] Creating layer conv_1/depthwise/relu
I0618 16:07:31.769989 43606 net.cpp:100] Creating Layer conv_1/depthwise/relu
I0618 16:07:31.769994 43606 net.cpp:434] conv_1/depthwise/relu <- conv_1/depthwise
I0618 16:07:31.770002 43606 net.cpp:395] conv_1/depthwise/relu -> conv_1/depthwise (in-place)
I0618 16:07:31.770225 43606 net.cpp:150] Setting up conv_1/depthwise/relu
I0618 16:07:31.770233 43606 net.cpp:157] Top shape: 1 96 75 75 (540000)
I0618 16:07:31.770238 43606 net.cpp:165] Memory required for data: 43200000
I0618 16:07:31.770246 43606 layer_factory.hpp:77] Creating layer conv_1/project
I0618 16:07:31.770262 43606 net.cpp:100] Creating Layer conv_1/project
I0618 16:07:31.770267 43606 net.cpp:434] conv_1/project <- conv_1/depthwise
I0618 16:07:31.770279 43606 net.cpp:408] conv_1/project -> conv_1/project
I0618 16:07:31.771124 43606 net.cpp:150] Setting up conv_1/project
I0618 16:07:31.771138 43606 net.cpp:157] Top shape: 1 24 75 75 (135000)
I0618 16:07:31.771142 43606 net.cpp:165] Memory required for data: 43740000
I0618 16:07:31.771152 43606 layer_factory.hpp:77] Creating layer conv_1/project_conv_1/project_0_split
I0618 16:07:31.771163 43606 net.cpp:100] Creating Layer conv_1/project_conv_1/project_0_split
I0618 16:07:31.771169 43606 net.cpp:434] conv_1/project_conv_1/project_0_split <- conv_1/project
I0618 16:07:31.771176 43606 net.cpp:408] conv_1/project_conv_1/project_0_split -> conv_1/project_conv_1/project_0_split_0
I0618 16:07:31.771188 43606 net.cpp:408] conv_1/project_conv_1/project_0_split -> conv_1/project_conv_1/project_0_split_1
I0618 16:07:31.771198 43606 net.cpp:150] Setting up conv_1/project_conv_1/project_0_split
I0618 16:07:31.771204 43606 net.cpp:157] Top shape: 1 24 75 75 (135000)
I0618 16:07:31.771209 43606 net.cpp:157] Top shape: 1 24 75 75 (135000)
I0618 16:07:31.771214 43606 net.cpp:165] Memory required for data: 44820000
I0618 16:07:31.771219 43606 layer_factory.hpp:77] Creating layer conv_2/expand
I0618 16:07:31.771229 43606 net.cpp:100] Creating Layer conv_2/expand
I0618 16:07:31.771235 43606 net.cpp:434] conv_2/expand <- conv_1/project_conv_1/project_0_split_0
I0618 16:07:31.771245 43606 net.cpp:408] conv_2/expand -> conv_2/expand
I0618 16:07:31.772011 43606 net.cpp:150] Setting up conv_2/expand
I0618 16:07:31.772025 43606 net.cpp:157] Top shape: 1 144 75 75 (810000)
I0618 16:07:31.772032 43606 net.cpp:165] Memory required for data: 48060000
I0618 16:07:31.772040 43606 layer_factory.hpp:77] Creating layer conv_2/expand/relu
I0618 16:07:31.772049 43606 net.cpp:100] Creating Layer conv_2/expand/relu
I0618 16:07:31.772055 43606 net.cpp:434] conv_2/expand/relu <- conv_2/expand
I0618 16:07:31.772064 43606 net.cpp:395] conv_2/expand/relu -> conv_2/expand (in-place)
I0618 16:07:31.772256 43606 net.cpp:150] Setting up conv_2/expand/relu
I0618 16:07:31.772262 43606 net.cpp:157] Top shape: 1 144 75 75 (810000)
I0618 16:07:31.772267 43606 net.cpp:165] Memory required for data: 51300000
I0618 16:07:31.772272 43606 layer_factory.hpp:77] Creating layer conv_2/depthwise
I0618 16:07:31.772285 43606 net.cpp:100] Creating Layer conv_2/depthwise
I0618 16:07:31.772291 43606 net.cpp:434] conv_2/depthwise <- conv_2/expand
I0618 16:07:31.772300 43606 net.cpp:408] conv_2/depthwise -> conv_2/depthwise
I0618 16:07:31.868156 43606 net.cpp:150] Setting up conv_2/depthwise
I0618 16:07:31.868180 43606 net.cpp:157] Top shape: 1 144 75 75 (810000)
I0618 16:07:31.868185 43606 net.cpp:165] Memory required for data: 54540000
I0618 16:07:31.868199 43606 layer_factory.hpp:77] Creating layer conv_2/depthwise/relu
I0618 16:07:31.868218 43606 net.cpp:100] Creating Layer conv_2/depthwise/relu
I0618 16:07:31.868225 43606 net.cpp:434] conv_2/depthwise/relu <- conv_2/depthwise
I0618 16:07:31.868235 43606 net.cpp:395] conv_2/depthwise/relu -> conv_2/depthwise (in-place)
I0618 16:07:31.868607 43606 net.cpp:150] Setting up conv_2/depthwise/relu
I0618 16:07:31.868618 43606 net.cpp:157] Top shape: 1 144 75 75 (810000)
I0618 16:07:31.868623 43606 net.cpp:165] Memory required for data: 57780000
I0618 16:07:31.868629 43606 layer_factory.hpp:77] Creating layer conv_2/project
I0618 16:07:31.868647 43606 net.cpp:100] Creating Layer conv_2/project
I0618 16:07:31.868654 43606 net.cpp:434] conv_2/project <- conv_2/depthwise
I0618 16:07:31.868662 43606 net.cpp:408] conv_2/project -> conv_2/project
I0618 16:07:31.869462 43606 net.cpp:150] Setting up conv_2/project
I0618 16:07:31.869473 43606 net.cpp:157] Top shape: 1 24 75 75 (135000)
I0618 16:07:31.869478 43606 net.cpp:165] Memory required for data: 58320000
I0618 16:07:31.869493 43606 layer_factory.hpp:77] Creating layer conv_2/sum
I0618 16:07:31.869503 43606 net.cpp:100] Creating Layer conv_2/sum
I0618 16:07:31.869508 43606 net.cpp:434] conv_2/sum <- conv_1/project_conv_1/project_0_split_1
I0618 16:07:31.869515 43606 net.cpp:434] conv_2/sum <- conv_2/project
I0618 16:07:31.869523 43606 net.cpp:408] conv_2/sum -> conv_2
I0618 16:07:31.869544 43606 net.cpp:150] Setting up conv_2/sum
I0618 16:07:31.869549 43606 net.cpp:157] Top shape: 1 24 75 75 (135000)
I0618 16:07:31.869552 43606 net.cpp:165] Memory required for data: 58860000
I0618 16:07:31.869557 43606 layer_factory.hpp:77] Creating layer conv_3/expand
I0618 16:07:31.869570 43606 net.cpp:100] Creating Layer conv_3/expand
I0618 16:07:31.869575 43606 net.cpp:434] conv_3/expand <- conv_2
I0618 16:07:31.869581 43606 net.cpp:408] conv_3/expand -> conv_3/expand
I0618 16:07:31.870421 43606 net.cpp:150] Setting up conv_3/expand
I0618 16:07:31.870434 43606 net.cpp:157] Top shape: 1 144 75 75 (810000)
I0618 16:07:31.870437 43606 net.cpp:165] Memory required for data: 62100000
I0618 16:07:31.870446 43606 layer_factory.hpp:77] Creating layer conv_3/expand/relu
I0618 16:07:31.870457 43606 net.cpp:100] Creating Layer conv_3/expand/relu
I0618 16:07:31.870463 43606 net.cpp:434] conv_3/expand/relu <- conv_3/expand
I0618 16:07:31.870471 43606 net.cpp:395] conv_3/expand/relu -> conv_3/expand (in-place)
I0618 16:07:31.870661 43606 net.cpp:150] Setting up conv_3/expand/relu
I0618 16:07:31.870668 43606 net.cpp:157] Top shape: 1 144 75 75 (810000)
I0618 16:07:31.870672 43606 net.cpp:165] Memory required for data: 65340000
I0618 16:07:31.870678 43606 layer_factory.hpp:77] Creating layer conv_3/depthwise
I0618 16:07:31.870692 43606 net.cpp:100] Creating Layer conv_3/depthwise
I0618 16:07:31.870697 43606 net.cpp:434] conv_3/depthwise <- conv_3/expand
I0618 16:07:31.870707 43606 net.cpp:408] conv_3/depthwise -> conv_3/depthwise
I0618 16:07:31.974457 43606 net.cpp:150] Setting up conv_3/depthwise
I0618 16:07:31.974479 43606 net.cpp:157] Top shape: 1 144 38 38 (207936)
I0618 16:07:31.974483 43606 net.cpp:165] Memory required for data: 66171744
I0618 16:07:31.974491 43606 layer_factory.hpp:77] Creating layer conv_3/depthwise/relu
I0618 16:07:31.974501 43606 net.cpp:100] Creating Layer conv_3/depthwise/relu
I0618 16:07:31.974505 43606 net.cpp:434] conv_3/depthwise/relu <- conv_3/depthwise
I0618 16:07:31.974511 43606 net.cpp:395] conv_3/depthwise/relu -> conv_3/depthwise (in-place)
I0618 16:07:31.974733 43606 net.cpp:150] Setting up conv_3/depthwise/relu
I0618 16:07:31.974742 43606 net.cpp:157] Top shape: 1 144 38 38 (207936)
I0618 16:07:31.974747 43606 net.cpp:165] Memory required for data: 67003488
I0618 16:07:31.974752 43606 layer_factory.hpp:77] Creating layer conv_3/project
I0618 16:07:31.974771 43606 net.cpp:100] Creating Layer conv_3/project
I0618 16:07:31.974776 43606 net.cpp:434] conv_3/project <- conv_3/depthwise
I0618 16:07:31.974784 43606 net.cpp:408] conv_3/project -> conv_3/project
I0618 16:07:31.975775 43606 net.cpp:150] Setting up conv_3/project
I0618 16:07:31.975788 43606 net.cpp:157] Top shape: 1 32 38 38 (46208)
I0618 16:07:31.975793 43606 net.cpp:165] Memory required for data: 67188320
I0618 16:07:31.975802 43606 layer_factory.hpp:77] Creating layer conv_3/project_conv_3/project_0_split
I0618 16:07:31.975813 43606 net.cpp:100] Creating Layer conv_3/project_conv_3/project_0_split
I0618 16:07:31.975819 43606 net.cpp:434] conv_3/project_conv_3/project_0_split <- conv_3/project
I0618 16:07:31.975828 43606 net.cpp:408] conv_3/project_conv_3/project_0_split -> conv_3/project_conv_3/project_0_split_0
I0618 16:07:31.975838 43606 net.cpp:408] conv_3/project_conv_3/project_0_split -> conv_3/project_conv_3/project_0_split_1
I0618 16:07:31.975849 43606 net.cpp:150] Setting up conv_3/project_conv_3/project_0_split
I0618 16:07:31.975855 43606 net.cpp:157] Top shape: 1 32 38 38 (46208)
I0618 16:07:31.975862 43606 net.cpp:157] Top shape: 1 32 38 38 (46208)
I0618 16:07:31.975867 43606 net.cpp:165] Memory required for data: 67557984
I0618 16:07:31.975870 43606 layer_factory.hpp:77] Creating layer conv_4/expand
I0618 16:07:31.975883 43606 net.cpp:100] Creating Layer conv_4/expand
I0618 16:07:31.975888 43606 net.cpp:434] conv_4/expand <- conv_3/project_conv_3/project_0_split_0
I0618 16:07:31.975896 43606 net.cpp:408] conv_4/expand -> conv_4/expand
I0618 16:07:31.976830 43606 net.cpp:150] Setting up conv_4/expand
I0618 16:07:31.976842 43606 net.cpp:157] Top shape: 1 192 38 38 (277248)
I0618 16:07:31.976847 43606 net.cpp:165] Memory required for data: 68666976
I0618 16:07:31.976857 43606 layer_factory.hpp:77] Creating layer conv_4/expand/relu
I0618 16:07:31.976866 43606 net.cpp:100] Creating Layer conv_4/expand/relu
I0618 16:07:31.976872 43606 net.cpp:434] conv_4/expand/relu <- conv_4/expand
I0618 16:07:31.976881 43606 net.cpp:395] conv_4/expand/relu -> conv_4/expand (in-place)
I0618 16:07:31.977084 43606 net.cpp:150] Setting up conv_4/expand/relu
I0618 16:07:31.977090 43606 net.cpp:157] Top shape: 1 192 38 38 (277248)
I0618 16:07:31.977095 43606 net.cpp:165] Memory required for data: 69775968
I0618 16:07:31.977102 43606 layer_factory.hpp:77] Creating layer conv_4/depthwise
I0618 16:07:31.977113 43606 net.cpp:100] Creating Layer conv_4/depthwise
I0618 16:07:31.977118 43606 net.cpp:434] conv_4/depthwise <- conv_4/expand
I0618 16:07:31.977128 43606 net.cpp:408] conv_4/depthwise -> conv_4/depthwise
I0618 16:07:32.114236 43606 net.cpp:150] Setting up conv_4/depthwise
I0618 16:07:32.114261 43606 net.cpp:157] Top shape: 1 192 38 38 (277248)
I0618 16:07:32.114264 43606 net.cpp:165] Memory required for data: 70884960
I0618 16:07:32.114274 43606 layer_factory.hpp:77] Creating layer conv_4/depthwise/relu
I0618 16:07:32.114284 43606 net.cpp:100] Creating Layer conv_4/depthwise/relu
I0618 16:07:32.114287 43606 net.cpp:434] conv_4/depthwise/relu <- conv_4/depthwise
I0618 16:07:32.114293 43606 net.cpp:395] conv_4/depthwise/relu -> conv_4/depthwise (in-place)
I0618 16:07:32.114706 43606 net.cpp:150] Setting up conv_4/depthwise/relu
I0618 16:07:32.114718 43606 net.cpp:157] Top shape: 1 192 38 38 (277248)
I0618 16:07:32.114722 43606 net.cpp:165] Memory required for data: 71993952
I0618 16:07:32.114729 43606 layer_factory.hpp:77] Creating layer conv_4/project
I0618 16:07:32.114747 43606 net.cpp:100] Creating Layer conv_4/project
I0618 16:07:32.114753 43606 net.cpp:434] conv_4/project <- conv_4/depthwise
I0618 16:07:32.114763 43606 net.cpp:408] conv_4/project -> conv_4/project
I0618 16:07:32.115628 43606 net.cpp:150] Setting up conv_4/project
I0618 16:07:32.115640 43606 net.cpp:157] Top shape: 1 32 38 38 (46208)
I0618 16:07:32.115645 43606 net.cpp:165] Memory required for data: 72178784
I0618 16:07:32.115654 43606 layer_factory.hpp:77] Creating layer conv_4/sum
I0618 16:07:32.115665 43606 net.cpp:100] Creating Layer conv_4/sum
I0618 16:07:32.115671 43606 net.cpp:434] conv_4/sum <- conv_3/project_conv_3/project_0_split_1
I0618 16:07:32.115679 43606 net.cpp:434] conv_4/sum <- conv_4/project
I0618 16:07:32.115685 43606 net.cpp:408] conv_4/sum -> conv_4
I0618 16:07:32.115697 43606 net.cpp:150] Setting up conv_4/sum
I0618 16:07:32.115703 43606 net.cpp:157] Top shape: 1 32 38 38 (46208)
I0618 16:07:32.115708 43606 net.cpp:165] Memory required for data: 72363616
I0618 16:07:32.115713 43606 layer_factory.hpp:77] Creating layer conv_4_conv_4/sum_0_split
I0618 16:07:32.115720 43606 net.cpp:100] Creating Layer conv_4_conv_4/sum_0_split
I0618 16:07:32.115725 43606 net.cpp:434] conv_4_conv_4/sum_0_split <- conv_4
I0618 16:07:32.115734 43606 net.cpp:408] conv_4_conv_4/sum_0_split -> conv_4_conv_4/sum_0_split_0
I0618 16:07:32.115743 43606 net.cpp:408] conv_4_conv_4/sum_0_split -> conv_4_conv_4/sum_0_split_1
I0618 16:07:32.115753 43606 net.cpp:150] Setting up conv_4_conv_4/sum_0_split
I0618 16:07:32.115759 43606 net.cpp:157] Top shape: 1 32 38 38 (46208)
I0618 16:07:32.115764 43606 net.cpp:157] Top shape: 1 32 38 38 (46208)
I0618 16:07:32.115769 43606 net.cpp:165] Memory required for data: 72733280
I0618 16:07:32.115775 43606 layer_factory.hpp:77] Creating layer conv_5/expand
I0618 16:07:32.115794 43606 net.cpp:100] Creating Layer conv_5/expand
I0618 16:07:32.115799 43606 net.cpp:434] conv_5/expand <- conv_4_conv_4/sum_0_split_0
I0618 16:07:32.115808 43606 net.cpp:408] conv_5/expand -> conv_5/expand
I0618 16:07:32.116652 43606 net.cpp:150] Setting up conv_5/expand
I0618 16:07:32.116662 43606 net.cpp:157] Top shape: 1 192 38 38 (277248)
I0618 16:07:32.116667 43606 net.cpp:165] Memory required for data: 73842272
I0618 16:07:32.116677 43606 layer_factory.hpp:77] Creating layer conv_5/expand/relu
I0618 16:07:32.116685 43606 net.cpp:100] Creating Layer conv_5/expand/relu
I0618 16:07:32.116693 43606 net.cpp:434] conv_5/expand/relu <- conv_5/expand
I0618 16:07:32.116698 43606 net.cpp:395] conv_5/expand/relu -> conv_5/expand (in-place)
I0618 16:07:32.116886 43606 net.cpp:150] Setting up conv_5/expand/relu
I0618 16:07:32.116892 43606 net.cpp:157] Top shape: 1 192 38 38 (277248)
I0618 16:07:32.116896 43606 net.cpp:165] Memory required for data: 74951264
I0618 16:07:32.116902 43606 layer_factory.hpp:77] Creating layer conv_5/depthwise
I0618 16:07:32.116915 43606 net.cpp:100] Creating Layer conv_5/depthwise
I0618 16:07:32.116920 43606 net.cpp:434] conv_5/depthwise <- conv_5/expand
I0618 16:07:32.116930 43606 net.cpp:408] conv_5/depthwise -> conv_5/depthwise
I0618 16:07:32.262675 43606 net.cpp:150] Setting up conv_5/depthwise
I0618 16:07:32.262698 43606 net.cpp:157] Top shape: 1 192 38 38 (277248)
I0618 16:07:32.262701 43606 net.cpp:165] Memory required for data: 76060256
I0618 16:07:32.262717 43606 layer_factory.hpp:77] Creating layer conv_5/depthwise/relu
I0618 16:07:32.262727 43606 net.cpp:100] Creating Layer conv_5/depthwise/relu
I0618 16:07:32.262732 43606 net.cpp:434] conv_5/depthwise/relu <- conv_5/depthwise
I0618 16:07:32.262739 43606 net.cpp:395] conv_5/depthwise/relu -> conv_5/depthwise (in-place)
I0618 16:07:32.262974 43606 net.cpp:150] Setting up conv_5/depthwise/relu
I0618 16:07:32.262981 43606 net.cpp:157] Top shape: 1 192 38 38 (277248)
I0618 16:07:32.262984 43606 net.cpp:165] Memory required for data: 77169248
I0618 16:07:32.262989 43606 layer_factory.hpp:77] Creating layer conv_5/project
I0618 16:07:32.263005 43606 net.cpp:100] Creating Layer conv_5/project
I0618 16:07:32.263010 43606 net.cpp:434] conv_5/project <- conv_5/depthwise
I0618 16:07:32.263020 43606 net.cpp:408] conv_5/project -> conv_5/project
I0618 16:07:32.263936 43606 net.cpp:150] Setting up conv_5/project
I0618 16:07:32.263948 43606 net.cpp:157] Top shape: 1 32 38 38 (46208)
I0618 16:07:32.263953 43606 net.cpp:165] Memory required for data: 77354080
I0618 16:07:32.263962 43606 layer_factory.hpp:77] Creating layer conv_5/sum
I0618 16:07:32.263973 43606 net.cpp:100] Creating Layer conv_5/sum
I0618 16:07:32.263979 43606 net.cpp:434] conv_5/sum <- conv_4_conv_4/sum_0_split_1
I0618 16:07:32.263986 43606 net.cpp:434] conv_5/sum <- conv_5/project
I0618 16:07:32.263993 43606 net.cpp:408] conv_5/sum -> conv_5
I0618 16:07:32.264005 43606 net.cpp:150] Setting up conv_5/sum
I0618 16:07:32.264012 43606 net.cpp:157] Top shape: 1 32 38 38 (46208)
I0618 16:07:32.264017 43606 net.cpp:165] Memory required for data: 77538912
I0618 16:07:32.264022 43606 layer_factory.hpp:77] Creating layer conv_6/expand
I0618 16:07:32.264036 43606 net.cpp:100] Creating Layer conv_6/expand
I0618 16:07:32.264042 43606 net.cpp:434] conv_6/expand <- conv_5
I0618 16:07:32.264050 43606 net.cpp:408] conv_6/expand -> conv_6/expand
I0618 16:07:32.264948 43606 net.cpp:150] Setting up conv_6/expand
I0618 16:07:32.264961 43606 net.cpp:157] Top shape: 1 192 38 38 (277248)
I0618 16:07:32.264964 43606 net.cpp:165] Memory required for data: 78647904
I0618 16:07:32.264973 43606 layer_factory.hpp:77] Creating layer conv_6/expand/relu
I0618 16:07:32.264983 43606 net.cpp:100] Creating Layer conv_6/expand/relu
I0618 16:07:32.264989 43606 net.cpp:434] conv_6/expand/relu <- conv_6/expand
I0618 16:07:32.264995 43606 net.cpp:395] conv_6/expand/relu -> conv_6/expand (in-place)
I0618 16:07:32.265442 43606 net.cpp:150] Setting up conv_6/expand/relu
I0618 16:07:32.265455 43606 net.cpp:157] Top shape: 1 192 38 38 (277248)
I0618 16:07:32.265460 43606 net.cpp:165] Memory required for data: 79756896
I0618 16:07:32.265465 43606 layer_factory.hpp:77] Creating layer conv_6/depthwise
I0618 16:07:32.265480 43606 net.cpp:100] Creating Layer conv_6/depthwise
I0618 16:07:32.265486 43606 net.cpp:434] conv_6/depthwise <- conv_6/expand
I0618 16:07:32.265496 43606 net.cpp:408] conv_6/depthwise -> conv_6/depthwise
I0618 16:07:32.416790 43606 net.cpp:150] Setting up conv_6/depthwise
I0618 16:07:32.416812 43606 net.cpp:157] Top shape: 1 192 19 19 (69312)
I0618 16:07:32.416815 43606 net.cpp:165] Memory required for data: 80034144
I0618 16:07:32.416826 43606 layer_factory.hpp:77] Creating layer conv_6/depthwise/relu
I0618 16:07:32.416834 43606 net.cpp:100] Creating Layer conv_6/depthwise/relu
I0618 16:07:32.416838 43606 net.cpp:434] conv_6/depthwise/relu <- conv_6/depthwise
I0618 16:07:32.416846 43606 net.cpp:395] conv_6/depthwise/relu -> conv_6/depthwise (in-place)
I0618 16:07:32.417052 43606 net.cpp:150] Setting up conv_6/depthwise/relu
I0618 16:07:32.417058 43606 net.cpp:157] Top shape: 1 192 19 19 (69312)
I0618 16:07:32.417062 43606 net.cpp:165] Memory required for data: 80311392
I0618 16:07:32.417064 43606 layer_factory.hpp:77] Creating layer conv_6/project
I0618 16:07:32.417080 43606 net.cpp:100] Creating Layer conv_6/project
I0618 16:07:32.417085 43606 net.cpp:434] conv_6/project <- conv_6/depthwise
I0618 16:07:32.417096 43606 net.cpp:408] conv_6/project -> conv_6/project
I0618 16:07:32.418278 43606 net.cpp:150] Setting up conv_6/project
I0618 16:07:32.418292 43606 net.cpp:157] Top shape: 1 64 19 19 (23104)
I0618 16:07:32.418294 43606 net.cpp:165] Memory required for data: 80403808
I0618 16:07:32.418300 43606 layer_factory.hpp:77] Creating layer conv_6/project_conv_6/project_0_split
I0618 16:07:32.418308 43606 net.cpp:100] Creating Layer conv_6/project_conv_6/project_0_split
I0618 16:07:32.418311 43606 net.cpp:434] conv_6/project_conv_6/project_0_split <- conv_6/project
I0618 16:07:32.418316 43606 net.cpp:408] conv_6/project_conv_6/project_0_split -> conv_6/project_conv_6/project_0_split_0
I0618 16:07:32.418323 43606 net.cpp:408] conv_6/project_conv_6/project_0_split -> conv_6/project_conv_6/project_0_split_1
I0618 16:07:32.418329 43606 net.cpp:150] Setting up conv_6/project_conv_6/project_0_split
I0618 16:07:32.418334 43606 net.cpp:157] Top shape: 1 64 19 19 (23104)
I0618 16:07:32.418340 43606 net.cpp:157] Top shape: 1 64 19 19 (23104)
I0618 16:07:32.418342 43606 net.cpp:165] Memory required for data: 80588640
I0618 16:07:32.418347 43606 layer_factory.hpp:77] Creating layer conv_7/expand
I0618 16:07:32.418360 43606 net.cpp:100] Creating Layer conv_7/expand
I0618 16:07:32.418365 43606 net.cpp:434] conv_7/expand <- conv_6/project_conv_6/project_0_split_0
I0618 16:07:32.418373 43606 net.cpp:408] conv_7/expand -> conv_7/expand
I0618 16:07:32.419476 43606 net.cpp:150] Setting up conv_7/expand
I0618 16:07:32.419487 43606 net.cpp:157] Top shape: 1 384 19 19 (138624)
I0618 16:07:32.419489 43606 net.cpp:165] Memory required for data: 81143136
I0618 16:07:32.419495 43606 layer_factory.hpp:77] Creating layer conv_7/expand/relu
I0618 16:07:32.419500 43606 net.cpp:100] Creating Layer conv_7/expand/relu
I0618 16:07:32.419504 43606 net.cpp:434] conv_7/expand/relu <- conv_7/expand
I0618 16:07:32.419509 43606 net.cpp:395] conv_7/expand/relu -> conv_7/expand (in-place)
I0618 16:07:32.419693 43606 net.cpp:150] Setting up conv_7/expand/relu
I0618 16:07:32.419699 43606 net.cpp:157] Top shape: 1 384 19 19 (138624)
I0618 16:07:32.419703 43606 net.cpp:165] Memory required for data: 81697632
I0618 16:07:32.419708 43606 layer_factory.hpp:77] Creating layer conv_7/depthwise
I0618 16:07:32.419720 43606 net.cpp:100] Creating Layer conv_7/depthwise
I0618 16:07:32.419725 43606 net.cpp:434] conv_7/depthwise <- conv_7/expand
I0618 16:07:32.419734 43606 net.cpp:408] conv_7/depthwise -> conv_7/depthwise
I0618 16:07:32.744880 43606 net.cpp:150] Setting up conv_7/depthwise
I0618 16:07:32.744905 43606 net.cpp:157] Top shape: 1 384 19 19 (138624)
I0618 16:07:32.744909 43606 net.cpp:165] Memory required for data: 82252128
I0618 16:07:32.744918 43606 layer_factory.hpp:77] Creating layer conv_7/depthwise/relu
I0618 16:07:32.744928 43606 net.cpp:100] Creating Layer conv_7/depthwise/relu
I0618 16:07:32.744933 43606 net.cpp:434] conv_7/depthwise/relu <- conv_7/depthwise
I0618 16:07:32.744938 43606 net.cpp:395] conv_7/depthwise/relu -> conv_7/depthwise (in-place)
I0618 16:07:32.745450 43606 net.cpp:150] Setting up conv_7/depthwise/relu
I0618 16:07:32.745462 43606 net.cpp:157] Top shape: 1 384 19 19 (138624)
I0618 16:07:32.745468 43606 net.cpp:165] Memory required for data: 82806624
I0618 16:07:32.745474 43606 layer_factory.hpp:77] Creating layer conv_7/project
I0618 16:07:32.745491 43606 net.cpp:100] Creating Layer conv_7/project
I0618 16:07:32.745498 43606 net.cpp:434] conv_7/project <- conv_7/depthwise
I0618 16:07:32.745507 43606 net.cpp:408] conv_7/project -> conv_7/project
I0618 16:07:32.746654 43606 net.cpp:150] Setting up conv_7/project
I0618 16:07:32.746666 43606 net.cpp:157] Top shape: 1 64 19 19 (23104)
I0618 16:07:32.746671 43606 net.cpp:165] Memory required for data: 82899040
I0618 16:07:32.746681 43606 layer_factory.hpp:77] Creating layer conv_7/sum
I0618 16:07:32.746690 43606 net.cpp:100] Creating Layer conv_7/sum
I0618 16:07:32.746697 43606 net.cpp:434] conv_7/sum <- conv_6/project_conv_6/project_0_split_1
I0618 16:07:32.746704 43606 net.cpp:434] conv_7/sum <- conv_7/project
I0618 16:07:32.746712 43606 net.cpp:408] conv_7/sum -> conv_7
I0618 16:07:32.746723 43606 net.cpp:150] Setting up conv_7/sum
I0618 16:07:32.746731 43606 net.cpp:157] Top shape: 1 64 19 19 (23104)
I0618 16:07:32.746736 43606 net.cpp:165] Memory required for data: 82991456
I0618 16:07:32.746739 43606 layer_factory.hpp:77] Creating layer conv_7_conv_7/sum_0_split
I0618 16:07:32.746747 43606 net.cpp:100] Creating Layer conv_7_conv_7/sum_0_split
I0618 16:07:32.746753 43606 net.cpp:434] conv_7_conv_7/sum_0_split <- conv_7
I0618 16:07:32.746760 43606 net.cpp:408] conv_7_conv_7/sum_0_split -> conv_7_conv_7/sum_0_split_0
I0618 16:07:32.746768 43606 net.cpp:408] conv_7_conv_7/sum_0_split -> conv_7_conv_7/sum_0_split_1
I0618 16:07:32.746778 43606 net.cpp:150] Setting up conv_7_conv_7/sum_0_split
I0618 16:07:32.746783 43606 net.cpp:157] Top shape: 1 64 19 19 (23104)
I0618 16:07:32.746790 43606 net.cpp:157] Top shape: 1 64 19 19 (23104)
I0618 16:07:32.746795 43606 net.cpp:165] Memory required for data: 83176288
I0618 16:07:32.746800 43606 layer_factory.hpp:77] Creating layer conv_8/expand
I0618 16:07:32.746810 43606 net.cpp:100] Creating Layer conv_8/expand
I0618 16:07:32.746817 43606 net.cpp:434] conv_8/expand <- conv_7_conv_7/sum_0_split_0
I0618 16:07:32.746826 43606 net.cpp:408] conv_8/expand -> conv_8/expand
I0618 16:07:32.747963 43606 net.cpp:150] Setting up conv_8/expand
I0618 16:07:32.747975 43606 net.cpp:157] Top shape: 1 384 19 19 (138624)
I0618 16:07:32.747977 43606 net.cpp:165] Memory required for data: 83730784
I0618 16:07:32.747982 43606 layer_factory.hpp:77] Creating layer conv_8/expand/relu
I0618 16:07:32.747989 43606 net.cpp:100] Creating Layer conv_8/expand/relu
I0618 16:07:32.747992 43606 net.cpp:434] conv_8/expand/relu <- conv_8/expand
I0618 16:07:32.747997 43606 net.cpp:395] conv_8/expand/relu -> conv_8/expand (in-place)
I0618 16:07:32.748183 43606 net.cpp:150] Setting up conv_8/expand/relu
I0618 16:07:32.748189 43606 net.cpp:157] Top shape: 1 384 19 19 (138624)
I0618 16:07:32.748193 43606 net.cpp:165] Memory required for data: 84285280
I0618 16:07:32.748198 43606 layer_factory.hpp:77] Creating layer conv_8/depthwise
I0618 16:07:32.748211 43606 net.cpp:100] Creating Layer conv_8/depthwise
I0618 16:07:32.748216 43606 net.cpp:434] conv_8/depthwise <- conv_8/expand
I0618 16:07:32.748226 43606 net.cpp:408] conv_8/depthwise -> conv_8/depthwise
F0618 16:07:32.816975 43606 cudnn_conv_layer.cpp:52] Check failed: error == cudaSuccess (2 vs. 0) out of memory
*** Check failure stack trace: ***
Aborted (core dumped)
将engine: CAFFE前的注释取消试试
Yep that worked, upon closer inspection I realized I missed that part of the ReadMe. But yes, all I needed to do is uncomment all of the "#engine: CAFFE" lines in the deploy.prototxt and that fixed all of my problems.
That also worked for me!