Automatically deriving mask for inpainting from alpha channel

[Vitaly Shukela]

Currently inpainting requires specifying two congruent images. It may be more convenient to just specify one image with erased regions (alpha channel).

Current UX:

$ convert q1.png -alpha extract q1_mask.png
$ texture-synthesis --out-size=171x248 --in-size=171x248 --out=w.png --inpaint=q1_mask.png generate q1.png

Desired UX:

$ texture-synthesis --out=w.png alpha-inpaint q1.png

[Austin Jones]

@vi @Jake-Shadle I have a prototype of generic pixel support (RGB/RGBA/Luma/LumaA) implemented on austinjones/texture-synthesis/multipixel, but it needs some work. It's just a quick exploration.

I tried handling alpha this way: Premultiply alpha and only consider color channels. This makes sense when later compositing: bright red with very low alpha and bright green with very low alpha are actually similar colors when added to anything. I didn't add alpha to the cost computation though, because the correlation with the color channels will cause bias in the cost metric.

There were some side effects:

• Session requires a type parameter (the pixel type). It has to be specified by the user whenever SessionBuilder::build() is called.
• There is a decent performance hit. Need to figure out why...
• The guided synthesis test fails.

Let me know if you are interested in the code. I can try to fix it up.

[Vitaly Shukela]

@austinjones, I don't understand the message.

The comment seems to be about improving alpha channel support in texture-synthesis.

But this proposal is about using alpha channel instead of separate file with the mask, for UX convenience. Approximately what the script with convert does, but from within texture-syntheses.

---

As far as experimentation shows, texture-synthesis already supports alpha channel:

begets

, with seemingly proper alpha channel.

[Vitaly Shukela]

For reference, this is my inpaint script I used for experimentation:

#!/bin/bash

if [ -z "$1" ]; then
    echo "Usage: inpaint src.png dst.png [text-synt args]"
    echo "Influential variables: K=50 (patch size) M=50 (randoms)"
    exit 1
fi

IN=$1
shift
OUT=$1
shift

S=$(identify "$IN" | cut -d' ' -f 3-3)
MASK=$(mktemp -u mask_XXXXXXXXXXX.png)

trap "rm -f \"$MASK\" " EXIT

convert  "$IN"  -alpha extract   -blur 0x1 -level 80%,100%  "$MASK"
#convert  "$IN"  -alpha extract "$MASK"

true ${K:=50}
true ${M:=50}

texture-synthesis --out-size="$S" --in-size="$S" --out="$OUT" --inpaint="$MASK" --k-neighs $K --m-rand $M "$@" generate "$IN"

[Austin Jones]

@vi Yes, you are right! I was trying to solve a different problem (support for example transparency).

I think what is supported right now is cost computation based on RGB, but pixel copies are on the full RGBA. If you add a bunch of transparent bright red, the cost is computed on bright red. Take a look at k_neighs_to_color_pattern - note the multiplier on the first line is 3, not 4. Transparent bright red is considered bright red when running the matching algorithm.

[Vitaly Shukela]

I think alpha channel support in matching should be tracked in the separate issue.

[Austin Jones]

Yeah, sounds like your use case is different @vi. Sorry for the confusion!

[Anastasia Opara]

Thanks for the suggestion! It sounds like a useful feature, which can be a part of pre-processing, or be an option if no mask is provided to the inpaint. We don't have any plans to add this atm, but pull requests are of course welcomed!

[Jake Shadle]

Closed by #60