Getting started with image input - can't figure it out

[Marin Todorov]

This is what I'm doing:

• cloned the repo
• went with Option 1 and downloaded the zip, unzipped the "bins" folder
• opened the SimpleDiffusion app and changed the URL in the task modifier to point to the local path:

            let path = URL(fileURLWithPath: "/Users/xyz/Downloads/bins")
            do {
                try await sd.prepModels(localUrl: path)
            }

• additionally set the inputImage to my input image:

            inputImage = NSImage(contentsOf: URL(fileURLWithPath: "/Users/xyz/IMG_1076.JPG"))?
                .cgImage(forProposedRect: nil, context: nil, hints: nil)

When I run the app, I set the prompt to "cat" and start the conversion. The process begins and then crashes with:

The file “first_stage_model.encoder.conv_in.weight.bin” couldn’t be opened because there is no such file.

[Morten Just]

You're doing it right. I have just uploaded a new pre-converted model to Google Drive, which also uses SD 1.5 and the encoder stuff for image to image.

https://drive.google.com/file/d/1EU_qxlF-p6XrxsoACvcI2CAmu45NphAC/view?usp=share_link

[Marin Todorov]

Thanks @mortenjust, I gave a try to the new download and it still crashes but with a different exception:

Model is ready
RAW progress 0.0
RAW progress 0.05
loc("mps_multiply_24"("(mpsFileLoc): /AppleInternal/Library/BuildRoots/810eba08-405a-11ed-86e9-6af958a02716/Library/Caches/com.apple.xbs/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Core/Files/MPSGraphUtilities.mm":228:0)): error: input types 'tensor<1x504x378x4xf16>' and 'tensor<1x64x64x4xf16>' are not broadcast compatible
loc("mps_multiply_24"("(mpsFileLoc): /AppleInternal/Library/BuildRoots/810eba08-405a-11ed-86e9-6af958a02716/Library/Caches/com.apple.xbs/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Core/Files/MPSGraphUtilities.mm":228:0)): error: 'mps.multiply' op failed to fold or infer return types
loc("mps_add_31"("(mpsFileLoc): /AppleInternal/Library/BuildRoots/810eba08-405a-11ed-86e9-6af958a02716/Library/Caches/com.apple.xbs/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Core/Files/MPSGraphUtilities.mm":228:0)): error: input types 'tensor<?x504x378x4xf16>' and 'tensor<1x64x64x4xf16>' are not broadcast compatible
loc("mps_add_31"("(mpsFileLoc): /AppleInternal/Library/BuildRoots/810eba08-405a-11ed-86e9-6af958a02716/Library/Caches/com.apple.xbs/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Core/Files/MPSGraphUtilities.mm":228:0)): error: 'mps.add' op failed to fold or infer return types
2022-11-24 19:16:22.048313+0100 SimpleDiffusion[50390:6618476] /AppleInternal/Library/BuildRoots/810eba08-405a-11ed-86e9-6af958a02716/Library/Caches/com.apple.xbs/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Core/Files/MPSGraphExecutable.mm, line 1584: error 'Error: MLIR pass manager failed'
/AppleInternal/Library/BuildRoots/810eba08-405a-11ed-86e9-6af958a02716/Library/Caches/com.apple.xbs/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Core/Files/MPSGraphExecutable.mm:1584: failed assertion `Error: MLIR pass manager failed'
/AppleInternal/Library/BuildRoots/810eba08-405a-11ed-86e9-6af958a02716/Library/Caches/com.apple.xbs/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Core/Files/MPSGraphExecutable.mm:1584: failed assertion `Error: MLIR pass manager failed'

[Marin Todorov]

I wonder if it expects a specific image format or something?

[Marin Todorov]

Oh yes, it expects a 512x512 image. Thanks for the help!