Implementing Custom model?

[Colleen Dunion]

HI - I've been trying to work on implementing a custom model using this plugin - but having some issues and maybe you could assist?

Using this-
https://developers.google.com/ml-kit/vision/image-labeling/custom-models/ios

I created a custom model with https://www.tensorflow.org/lite/tutorials/model_maker_image_classification

I trying parsing together the implementation with the custom models/ios - but having issues.

Appreciate the plugin and would love some insight into adding the custom models!

[Marc Rousavy]

Hey!

This plugin is really just 30 lines of code at max, one for iOS and one for Android. I think to use custom models it'd be better to create those FP Plugins in your app instead directly.

[Colleen Dunion]

OK thanks! Will do that

[Colleen Dunion]

Actually had an easy time adding a few lines of code to this plugin to implement the custom model.

By following this model tutorial-https://colab.research.google.com/github/tensorflow/tensorflow/blob/master/tensorflow/lite/g3doc/tutorials/model_maker_image_classification.ipynb
and utilizing my own data for the model.
Then I just input the name of the tflite file in the pathforresource and its ready to go!

Thank you for your plugin - it was a great working example that allowed me to add the custom components!

#import <VisionCamera/FrameProcessorPlugin.h>
#import <VisionCamera/Frame.h>
#import <MLKit.h>

// Example for an Objective-C Frame Processor plugin

@interface ImageLabelerPlugin : NSObject

+ (MLKImageLabeler*) labeler;


@end

@implementation ImageLabelerPlugin


+ (MLKImageLabeler*) labeler {
  static MLKImageLabeler* labeler = nil;

  if (labeler == nil) {
  NSString *path = [[NSBundle mainBundle] pathForResource:@"model" ofType:@"tflite"];
    NSLog(@"PATH:", path);
    MLKLocalModel *localModel =
      [[MLKLocalModel alloc] initWithPath:path];

  MLKCustomImageLabelerOptions *options =
      [[MLKCustomImageLabelerOptions alloc] initWithLocalModel:localModel];
   NSLog(@"OPTIONS", options);
  options.confidenceThreshold = @(0.0);
  labeler =
      [MLKImageLabeler imageLabelerWithOptions:options];

    labeler = [MLKImageLabeler imageLabelerWithOptions:options];

  }
  return labeler;
}

static inline id labelImage(Frame* frame, NSArray* arguments) {

  MLKVisionImage *image = [[MLKVisionImage alloc] initWithBuffer:frame.buffer];
  NSLog(@"VISION CAMERA IMAGE LABELER: Testing this out:, %@", image);
  image.orientation = frame.orientation; // <-- TODO: is mirrored?

  NSError* error = nil;
 NSArray<MLKImageLabel*>* labels = [[ImageLabelerPlugin labeler] resultsInImage:image error:&error];

  if(error){
    NSLog(@"Error finding Labels: %@", error);
  } else {
    NSLog(@"No Error: %@", error);
  }

  NSMutableArray* results = [NSMutableArray arrayWithCapacity:labels.count];
  for (MLKImageLabel* label in labels) {
    [results addObject:@{
      @"label": label.text,
      @"confidence": [NSNumber numberWithFloat:label.confidence]
    }];
  }

  return results;
}

VISION_EXPORT_FRAME_PROCESSOR(labelImage)

@end

[Marc Rousavy]

Awesome, glad you're appreciating my projects :)