Giters

john-rocky / CoreML-Models

Converted CoreML Model Zoo.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

AnimeGANv2 to CoreML

ManluZhang opened this issue · comments

commented

Hi, thanks for the great work!

I'm trying to convert the AnimeGANv2 Shinkai.pb model to CoreML, and I've got some questions.

  1. Did you use the check points to generate a .pb file first, maybe using this script then convert it to coreml using coremltools?
  2. In coremltools, should I use ct.ImageType(shape=(1, 256, 256, 3)) as input format of the converter?

Here's my converter code:

image_inputs = ct.ImageType(shape=(1, 256, 256, 3))
mlmodel = ct.convert('./Shinkai_53.pb', inputs=[image_inputs],  source='tensorflow')
mlmodel.save('./output.mlmodel')

And then I use:

import coremltools.proto.FeatureTypes_pb2 as ft
spec = ct.utils.load_spec("output.mlmodel")
output = spec.description.output[0]
output.type.imageType.colorSpace = ft.ImageFeatureType.RGB
output.type.imageType.height = 256
output.type.imageType.width = 256
ct.utils.save_spec(spec, "new.mlmodel")

to convert the output format to Image.
After I drag the new mlmodel to Xcode, and preview the model with an image, the stylized image can't be generated. It seems to be loading forever.

screenshot

Do u have a cue on what's going wrong? Or could u please tell me how u convert the AnimeGANv2 models?
Again, thanks for your great work. Really appreciate it.

commented
commented

Hi. You need to permute output shape. 2022年2月15日(火) 20:34 Manlu Zhang @.>:
…
Hi, thanks for the great work! I'm trying to convert the AnimeGANv2 Shinkai.pb model https://github.com/TachibanaYoshino/AnimeGANv2/blob/master/pb_and_onnx_model/Shinkai_53.pb to CoreML, and I've got some questions. 1. Did you use the check points to generate a .pb file first, maybe using this script https://github.com/TachibanaYoshino/AnimeGANv2/blob/master/pb_and_onnx_model/animegan2pb.py then convert it to coreml using coremltools? 2. In coremltools, should I use ct.ImageType(shape=(1, 256, 256, 3)) as input format of the converter? Here's my converter code: image_inputs = ct.ImageType(shape=(1, 256, 256, 3)) mlmodel = ct.convert('./Shinkai_53.pb', inputs=[image_inputs], source='tensorflow') mlmodel.save('./output.mlmodel') And then I use: import coremltools.proto.FeatureTypes_pb2 as ft spec = ct.utils.load_spec("output.mlmodel") output = spec.description.output[0] output.type.imageType.colorSpace = ft.ImageFeatureType.RGB output.type.imageType.height = 256 output.type.imageType.width = 256 ct.utils.save_spec(spec, "new.mlmodel") to convert the output format to Image. After I drag the new mlmodel to Xcode, and preview the model with an image, the stylized image can't be generated. It seems to be loading forever. Do u have a cue on what's going wrong? Or could u please tell me how u convert the AnimeGANv2 models? Again, thanks for your great work. Really appreciate it. — Reply to this email directly, view it on GitHub <#11>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFRTLEDX3BNN5GRTVRERO3DU3I25NANCNFSM5OOH3SSQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you are subscribed to this thread.Message ID: @.>

Hi,
Thanks for the reply.
I'm actually new to machine learning. How do I permute the output shapes? Do you mean using the add_ permute function in coremltools?

commented
commented

Try this

spec = mlmodel.get_spec()
builder = ct.models.neural_network.NeuralNetworkBuilder(spec=spec)
builder.spec.description

builder.add_permute(name="permute",dim=[0,3,1,2], input_name={output name in printed description}, output_name="permute_out")
builder.add_squeeze(name="squeeze", input_name="permute_out", output_name="squeeze_out", axes = None, squeeze_all = True)
builder.add_activation(name="activation",non_linearity="LINEAR",input_name="squeeze_out",output_name="image",params=[127.5,127.5])
from coremltools.proto import FeatureTypes_pb2 as ft
builder.spec.description.output.pop()
builder.spec.description.output.add()
output = builder.spec.description.output[0]
output.name = "image"

output.type.imageType.colorSpace = ft.ImageFeatureType.ColorSpace.Value('RGB')
output.type.imageType.width = 256 
output.type.imageType.height = 256
commented

It works perfectly! Thanks a lot!!!

commented

Happy machine learning🐥

commented

@ManluZhang , can you please provide the CoreML model.

commented

@ManluZhang , can you please provide the CoreML model.

Yes, here.

commented

Hi, @john-rocky
I've converted the animeGANv2 Paprika checkpoints to coreML, and compared with the model that you provided in this repo.
The output of the same image looks quite different.
wecom-temp-1694df0f3d79bfb641c21a35960d29c3
Left is from your model, and the right one is from mine.
I also found that, the tensorflow inference of the animeGANv2 is very different after I converted it to coreML.
Any ideas why ?

My converting code:

    mlmodel = ct.convert(model=path, inputs=[ct.ImageType(shape=(1, 256, 256, 3))],
                         source='tensorflow')
    mlmodel.save(name)
    spec = ct.utils.load_spec(name)
    builder = ct.models.neural_network.NeuralNetworkBuilder(spec=spec)
    print(builder.spec.description)

    builder.add_permute(name="permute", dim=[0, 3, 1, 2], input_name="generator_G_MODEL_out_layer_Tanh", output_name = "permute_out")
    builder.add_squeeze(name="squeeze", input_name="permute_out", output_name="squeeze_out", axes=None,
                        squeeze_all=True)
    builder.add_activation(name="activation", non_linearity="LINEAR", input_name="squeeze_out", output_name="image",
                           params=[127.5, 127.5])

    builder.spec.description.output.pop()
    builder.spec.description.output.add()

    output = builder.spec.description.output[0]
    output.name = "image"
    output.type.imageType.colorSpace = ft.ImageFeatureType.ColorSpace.Value('RGB')
    output.type.imageType.width = 256
    output.type.imageType.height = 256

    ct.utils.save_spec(builder.spec, 'output.mlmodel')
commented

I think Activation is too high.

Try to change activation param. Like [255,0]

commented

Activation params are changed on the range of the output of the model.
If model has -1~1 output, We need * 125 +127.5.
If 0 ~ 1, we need *255.
I forgot AnimeGAN output range. So, please try...

commented

企业微信截图_0faab8b4-041b-499d-aed4-0d82fee7aa37
I've already tried [255,0], looks not same either.

commented

[127.5,0]?

commented

let me try

commented

企业微信截图_6a02804f-b280-4f1b-a876-54c2ff30e59a
too dark
maybe I should figure out the AnimeGAN output range first
do you know how to see the output range?

commented

Wait, I missed your conversion script.
I think you need add preprocessing input.

like

mlmodel = ct.convert('./frozen_model.pb',
                     inputs=[ct.ImageType(shape=[1,256,256,3],bias=[-1,-1,-1], scale=1/127)])

Then activate with [127.5,127.5]

About preprocessing, please see my article.

https://rockyshikoku.medium.com/generate-images-using-gan-on-ios-devices-1188e01283d2

commented

OK, thanks a lot !!!