Jetson memory issues

[Grant Timmerman]

Looks like there is not enough memory on the Jetson board. Here's the log:

Loading XML file ../blog-code/training_xml_web//Marcia_Turca.xml
    storage_map=getattr(self.fn, 'storage_map', None))

  File "/usr/local/lib/python2.7/dist-packages/theano/gof/link.py", line 314, in raise_with_op

    reraise(exc_type, exc_value, exc_trace)

  File "/usr/local/lib/python2.7/dist-packages/theano/compile/function_module.py", line 859, in __call__

    outputs = self.fn()

MemoryError: alloc failed

Apply node that caused the error: Alloc(TensorConstant{0.0}, Elemwise{add,no_inplace}.0, Elemwise{switch,no_inplace}.0, Elemwise{Composite{Switch(EQ(i0, i1), i2, i0)}}.0)

Toposort index: 226

Inputs types: [TensorType(float64, scalar), TensorType(int64, scalar), TensorType(int64, scalar), TensorType(int64, scalar)]

Inputs shapes: [(), (), (), ()]

Inputs strides: [(), (), (), ()]

Inputs values: [array(0.0), array(128L, dtype=int64), array(780L, dtype=int64), array(600L, dtype=int64)]

Outputs clients: [[forall_inplace,cpu,grad_of_scan_fn}(Elemwise{maximum,no_inplace}.0, Elemwise{mul}.0, Alloc.0, Alloc.0, Alloc.0, Subtensor{int64:int64:int64}.0, Subtensor{int64:int64:int64}.0, Subtensor{int64:int64:int64}.0, Subtensor{int64:int64:int64}.0, Alloc.0, Alloc.0, Alloc.0, Alloc.0, Alloc.0, Alloc.0, Alloc.0, Alloc.0, Alloc.0, Alloc.0, Alloc.0, Alloc.0, Alloc.0, Alloc.0, Alloc.0, Alloc.0, Alloc.0, Alloc.0, Layer.linear_matrix, Layer.linear_matrix, Layer.linear_matrix, Layer.linear_matrix, Layer.linear_matrix, Layer.linear_matrix, Layer.linear_matrix, Layer.linear_matrix, Elemwise{Cast{float64}}.0, InplaceDimShuffle{1,0}.0, InplaceDimShuffle{1,0}.0, InplaceDimShuffle{1,0}.0, InplaceDimShuffle{1,0}.0, InplaceDimShuffle{1,0}.0, InplaceDimShuffle{1,0}.0, InplaceDimShuffle{1,0}.0, InplaceDimShuffle{1,0}.0, InplaceDimShuffle{0,x}.0, InplaceDimShuffle{0,x}.0, InplaceDimShuffle{0,x}.0, InplaceDimShuffle{0,x}.0, InplaceDimShuffle{0,x}.0, InplaceDimShuffle{0,x}.0, InplaceDimShuffle{0,x}.0, InplaceDimShuffle{0,x}.0, Alloc.0, Alloc.0, Alloc.0, Alloc.0, Alloc.0, Alloc.0, Alloc.0, Alloc.0)]]



Backtrace when the node is created:

  File "/usr/local/lib/python2.7/dist-packages/theano/gradient.py", line 1113, in access_term_cache

    input_grads = node.op.grad(inputs, new_output_grads)



HINT: Use the Theano flag 'exception_verbosity=high' for a debugprint and storage map footprint of this apply node.

Training process nnn terminated!!!!

[Grant Timmerman]

According to free -m there's 3Gb of memory available on [e-mail]:

free -m
             total       used       free     shared    buffers     cached
Mem:          3853        707       3145          8        110        213
-/+ buffers/cache:        383       3470
Swap:            0          0          0