Lengthy process or stuck?

Question

Lengthy process or stuck?

colbec opened this issue 6 years ago · comments

I wanted to see how the process works, so I set up an image of similar size to those used in the examples to set up inpainting on an RGB image. I used imread from PyPlot to read the image into memory and set up the BitArray to mask out an area and set the function in motion. Problem has been running for about 2 hours and I am wondering if it is stuck somewhere in a loop. Might be helpful to show progress of some kind if it is a genuinely lengthy process. When I have time I will simplify down to a smaller greyscale image to see if it will complete in a timely manner.

Júlio Hoffimann commented 6 years ago

See #1

Júlio Hoffimann · Answer 1 · Wed Apr 11 2018 05:15:57 GMT+0800 (China Standard Time)

Hi @colbec , thank you for opening the issue. The currently implemented algorithm by Crimisini et al. is computationally expensive, specially if the image is large and the patch size you pick is small. The idea is simple: at each iteration the entire image is scanned to find the best candidate patch, and the patch is pasted near the boundary defined by the mask.

The algorithm is the most powerful that I know of (in terms of the results it generates) when the masked region is large. All other alternative inpainting algorithms in the literature, which fit in a gradient-based category, present some blurriness with large masked regions.

What is the size of the image and the patch size that you are using? I suggest trying to increase the patch size to a point where the algorithm is feasible. If it still does not terminate, please share more details and I'd be happy to help debug it.

Additionally, there must be room for improvement in the code, I didn't optimize it yet for memory allocations, which is usually the bottleneck with Julia implementations. Please let me know if you find any spot in the source where things could be done differently. I am cc'ing other JuliaImages contributors to see if they can spot other inefficiencies as well. cc: @timholy , @Evizero , @andyferris, @rsrock , @mronian

When I migrated the repository to JuliaImages, it didn't have the contributors watching it by default.

Andy Ferris · Answer 2 · Wed Apr 11 2018 11:15:42 GMT+0800 (China Standard Time)

A couple things jumped out at me:

Applying a liberal sprinkling of @inbounds might make things faster. :)
Where you splat a vector from i=1:N into view this will be quite inefficient (possibly consider ntuple(f, Val(N)) and splatting that - you can splat tuples of known length "for free"). AFAICT this is in an inner loop in selectpatch. E.g. maybe try change

start  = [center[i] - (psize[i]-1) ÷ 2 for i=1:N]
finish = [center[i] +     psize[i] ÷ 2 for i=1:N]

[view(img, [start[i]:finish[i] for i=1:N]...) for img in imgs]

to

[view(img, ntuple(i -> @inbounds(return (center[i] - (psize[i]-1) ÷ 2) : (center[i] + psize[i] ÷ 2)), Val(N))...) for img in imgs]

(Please note I didn't actually try this - but basically for all your vectors of statically-known length N you could try replace them with a tuple and save an allocation each time, plus a lot of overhead with splatting).

Júlio Hoffimann · Answer 3 · Wed Apr 11 2018 13:46:39 GMT+0800 (China Standard Time)

Thank you @andyferris , I will try to find time to implement your suggestions, all of them are great.

Colin Beckingham · Answer 4 · Wed Apr 11 2018 21:30:33 GMT+0800 (China Standard Time)

@juliohm The original image I was working with is 1920x1080, I did not realize it was on the large size. The patch is about 700x500 so I guess I was a bit ambitious.
In order to experiment a bit I created the following as a test; the result is somewhat unexpected so I guess I must have mis-coded somewhere. The image is created, and is mainly grey except a red diagonal line from top left to bottom right and a blue line from top right to bottom left. The intention is to see if the algo can fill in the missing centre piece if taken out and restore the lines. But that does not happen. It does however return a result in 60 seconds on my machine, so it is a starting point. Hope it might be of some use:

# image inpainting
using ImageInpainting, PyPlot
ion()
tic()
n = 50 # image x and y dimension, square
img = zeros(n,n,3) .+ 0.5
# create the diagonal lines
for i in 1:n
    img[i,i,1] .= 1.0
    img[i,n-i+1,3] .= 1.0
end
imshow(img)
readline(STDIN) # wait to keep image displayed
if true
    h,w,d = size(img)
    # make a bit vector
    maska = BitArray(false for x in 1:h*w*d)
    # reshape it
    mask = reshape(maska,(h,w,d))
    # set the patch to true
    mask[20:30,15:40,:] .= true
    # do the inpainting
    out = inpaint(img,mask,Crimisini((11,11,3)))
    figure(2)
    imshow(out)
    readline(STDIN)
end
println("OK")
toc()

Júlio Hoffimann · Answer 5 · Thu Apr 12 2018 06:07:09 GMT+0800 (China Standard Time)

@colbec oh I see why it is taking forever. I didn't add support for color types yet (or images with multiple channels). The code also works with 3D images (with a single channel). This is what is happening: you think you are running a 2D inpainting problem, but behind the scenes the algorithm is trying to inpaint a 3D cube of values. It is much more expensive.

Could you please try to inpaint a single channel (grayscale) image to see if it works first? I didn't have the time to port the code to RGB or any color types in the Images.jl project, sorry.

Júlio Hoffimann · Answer 6 · Thu Apr 12 2018 06:19:06 GMT+0800 (China Standard Time)

For me it is taking 0.06 seconds with your example if I make it 1 channel:

using ImageInpainting
using PyPlot

n = 50
img = zeros(n,n) .+ 0.5

# create the diagonal lines
for i in 1:n
    img[i,i] .= 1.0
    img[i,n-i+1] .= 1.0
end

mask = falses(size(img))
mask[20:30,15:40] .= true

out = @time inpaint(img,mask,Crimisini((9,9)))

figure()
imshow(img)

figure()
imshow(mask)

figure()
imshow(out)

Colin Beckingham · Answer 7 · Thu Apr 12 2018 18:13:17 GMT+0800 (China Standard Time)

@juliohm Thanks for your comments and corrections. As you can imagine, the presented examples of Trump and lighthouse leads the reader to infer that 3 colour channels are tractable by the algorithm. I did try to process channels separately and recombine but ran into a few problems. I find that the algo is very sensitive to the tuple offered by the Crimisini, and also by the size and shape of the patch. Using mask[20:40,20:40] .= true together with Crimisini((9,9)) I get the following image:

I don't want to read too much into this, I have probably created a corner case and the literature is quite clear that some approaches are only suitable for certain kinds of problems. I'll lkeep exploring.

Júlio Hoffimann · Answer 8 · Thu Apr 12 2018 23:49:57 GMT+0800 (China Standard Time)

Thank you @colbec, I will add a note to the README mentioning that the multi-channel case is not implemented yet. Sorry for the confusion. Regarding the sensitivity to the patch size, that is true. The algorithm is sensitive specially with an image like the one above where there is basically no pattern to match (it is all empty with two thin lines on it). The issue is not the Crimisini algorithm itself, but the Euclidean distance that is used to find the best patch. In a real image, these pathological configurations should be harder to find.

Other algorithms out there may fit your purposes better, but as I mentioned before, I couldn't find one yet that outperforms Crimisini et al. for large masked regions. Contributions are very welcome.

Júlio Hoffimann · Answer 9 · Fri Apr 13 2018 09:34:16 GMT+0800 (China Standard Time)

I've added the observation to the README. I've also enabled all CPU cores in the FFT to gain some extra power. Closing the issue since it is clarified the current limitations. PRs are welcome to extend the code to color types. I will implement it at some point in the future otherwise.