System.AccessViolationException on Invoke for TFLite model

[Freek Schoenmakers]

I'm trying to do some inferences for a TFLite model on Windows. I have a float[] filled with values as input, and want to get a float[] of values as output.

As I understand it, I need to provide the data to the Input tensor, and then get the data from the Output tensor after calling Invoke() on the Interpreter that has the TFLite model loaded as a FlatBuffer. The way I'm currently doing this is by doing the following:

Setting the input:

public void SetInput(float[] values) {
            Marshal.Copy(values, 0, interpreter.Inputs[0].DataPointer, values.Length);
        }

Calling Invoke:

public void Invoke() {
            Status status = interpreter.Invoke();
            if (status == Status.Error)
                throw new Exception("Invoke returned Error status!");
        }

Getting the output:

public float[] GetOutputArray() {
            return (float[])interpreter.Outputs[0].Data;
        }

I have verified that the data is being set correctly and that the output is also fine after calling Invoke. The whole thing actually works really well, but only for a short while. Usually after less than a minute (and several thousand successful inferences), Invoke will suddenly throw a System.AccessViolationException and crash the thread.

Note that all operations described here are happening on the same thread, the model is not being called multiple times from different threads or anything.

Am I not doing this right or is there an issue here? Note that the examples on the git here use methods specifically to convert images to an input, but as I'm not using images these are not suitable for me.

EDIT: Maybe also useful to know: the input float[] is always the same size, so I'm only doing AllocateTensors once, after intialising the model.

[Emgu]

One thing to check is that, in your code:

public void SetInput(float[] values) {
            Marshal.Copy(values, 0, interpreter.Inputs[0].DataPointer, values.Length);
        }

values.Length match the Size of the "interpreter.Inputs[0]".

I would also check and see if any of the objects being used are disposed when the access violation take place.

The following link may be helpful:
https://www.emgu.com/wiki/index.php/My_model_doesn%27t_work

Given that your model works well for a few thoundsand sucessfull inference, I would thinkg that "objects being used are disposed" is the most likely reason. Sound like garbage collector disposed an object while it is still being used some where in your code.

[Freek Schoenmakers]

You were correct, it was a Garbage Collection issue.

Apparently, the Interpreter object does not maintain an active reference to the FlatBufferModel object that is used to instantiate it, even though it does require it. Upon a GC, the model gets disposed and the interpreter no longer works.

I have fixed it on my end, but I would recommend keeping a reference in the Interpreter object as well, to avoid confusion. A GC should, in my humble opinion, not cause AccessViolationExceptions.

Thanks for your help!