Is it possible to make the MLMultiArray to image conversion faster via GPU?
sbhhbs opened this issue · comments
Currently, the implementation to convert MLMultiArray to image alloced buffer and uses for loops to fetch and fill the buffer. However it may be faster to run on GPU in parallel? It would be useful if CoreML model is rendered real time during a video stream.
If the MLMultiArray is laid out in a way that is similar to the image (RGBARGBA... etc) then you can simply copy it (possibly converting floats to bytes in the process). But often it will be laid out differently in memory and then this doesn't work. I don't know if using the GPU would make that faster (also, the GPU does not support doubles).
By the way, you can configure your CoreML model to output a CVPixelBuffer, which is way more efficient.
Hey, stumbled upon this thread while searching for a solution to a similar problem. How does one configure the model to output a CVPixelBuffer? I presume this is done in the converting process with coremltools?
I have searched all over the net for a solution but haven't been able to find one. I have trained a U-net segmentation model that accepts a 256x256 image and outputs a 256x256 mask which is returned in MLMultiArray and converting the array to an image is pain :/
You can load the mlmodel file into a Python script, then change the type of the output from a multi array to an image, and save the mlmodel again. This requires familiarity with the coremltools API.
Hey hollance, thanks for your reply. I have tried different parameters in the keras.convert method but no avail. The code I use to convert the model from Keras to CoreML is down below:
def convert_model(model_in): coreml_model = coremltools.converters.keras.convert(model_in, input_names=['image'], image_input_names='image', output_names='mask', image_scale=1/255.0) coreml_model.licence = 'MIT' coreml_model.short_description = 'Reads an image and outputs a segmentation mask' coreml_model.input_description['image'] = 'A 256x256 pixel image' coreml_model.output_description['mask'] = 'A 256x256 pixel mask' coreml_model.save(model_out) print('Model converted')
The optional output_names parameterer in keras.convert doesn't appear to change much since it only changes the name of the relevant output, are there any other parameters that affect the output that I am missing here?
What I meant was going a level lower than using the Keras converter and directly change the contents of the mlmodel file. I’m writing a chapter about this in the raywenderlich.com machine learning book, but it will be a while before that is out.
I see. Looking forward to reading that. I take it it's necessary to edit the mlmodel binary protobuf file? Any good pointers on reading up on how to do that until your book comes out?
You’ll have to dig through the machine learning section of the Apple Dev Forums, I’m afraid. I’d write something up but I don’t really have the time right now.
I've added an MLMultiArray to UIImage conversion function that uses vImage, which is about 10x faster than the nested loops.
When coreml outputs cvpixelbuffer, it always crashes accessing the output when using Objective-C