TopFormer (287) CoreML model does not take Image as input

[Owen Gregson]

Issue Type

Bug

OS

Mac OS

OS architecture

x86_64, armv7, armv6

Programming Language

Other

Framework

CoreML

Model name and Weights/Checkpoints URL

https://s3.ap-northeast-2.wasabisys.com/pinto-model-zoo/287_Topformer/resources.tar.gz

"model_coreml_float32.mlmodel"

Description

Our testing uses the Swift programming language, as CoreML models are designed to be integrated into Swift apps.

The Issue

The CoreML model should take an Image type as input but instead takes an MLMultiArray. Because of this, the model is effectively unusable in Swift, and any attempts to use it usually result in the error listed in the Log Output below. The error basically means that the model is unable to accept an Image as input, even though it is designed to process images.

What we tried

We attempted to change the input type ourselves using Python's coremltools, which seemed to have succeeded, but after importing the model into XCode... The editor was unable to build the project without encountering a build error. The error relates to something about the kernel size being incorrect for the reshape operation.

We also spent a long time trying to convert an ONNX version to CoreML ourselves, but due to the recent drop of ONNX -> CoreML support, we merely encountered issue after issue and ultimately failed.

We also tried a myriad of other methods to convert Tensorflow, PyTorch, and other model types to CoreML, but it was more of the same and we just encountered numerous issues that eventually made us give up.

If you wish to test the model in Swift, you can use the code I provided at the bottom of this issue to import the model, load it, then make a request to process something.

The Solution

However, all hope is not lost. I believe this issue should be simple to fix; whatever conversion process was originally used to convert the model into CoreML should be altered so that the model can properly accept Image as an input.

In addition, you could add (or replace the old model) file to the .tar.gz in the TopFormer folder in this repo so that others trying to use the CoreML version in their Swift app will not encounter runtime errors or build errors.

TLDR

The model needs to accept Image but accepts MLMultiArray instead. It needs to be re-converted appropriately.

If you'd like any more information about my process, what we tried, or something else, please let me know.

Relevant Log Output

Error Domain=com.apple.vis Code=15 "The model does not have a valid input feature of type image" UserInfo={NSLocalizedDescription=The model does not have a valid input feature of type image}

URL or source code for simple inference testing code

// Get the model file
guard let topFormerModel = Bundle.main.url(forResource: "model_coreml_float32", withExtension: "mlmodelc") else {
     print("Missing model file!")
     return
} do {
     // Load the model
     let segmentationModel = try VNCoreMLModel(for: MLModel(contentsOf: topFormerModel));
     segmentationRequest = VNCoreMLRequest(model: segmentationModel);
     } catch {
         print(error)
     }
// Send processing request to model
func makeMLRequest(sampleBuffer: CMSampleBuffer, request: VNRequest){
    let requestHandler = VNImageRequestHandler(cmSampleBuffer: sampleBuffer)
    do {
        try requestHandler.perform([request])
    } catch {
        print(error)
    }
}

[Owen Gregson]

Hello,

Still need some sort of resolution for this so I am posting another comment to bump it.

[Katsuya Hyodo]

If you seriously read my response to the issue you posted the other day and tried to address it seriously, there is nothing more I can explain. Good luck.

I am not a coding teacher.
Duplicate of: #344