Python

[victorca25]

I've implemented a couple of different smoothing filters, including WLS, TV, and RTV in Python and I just added the EAP with human mask to RTV and I think it mostly works (I still have to fix an issue with the bottom border of the image):

I now want to continue with the automatic mask creation, but I'm wondering about the information in the readme:

The codes are written in Matlab. I have spent one and a half months to convert those codes to python but finally failed. It is impossible to implement these tools in python. This is mainly because the sparse optimization in python is very weak.

Where exactly did you experience these problems? What should I expect? Any tips?

[victorca25]

The automatic knapsack masks work ok in Python:

But still have to do some more parameter testing to get better smooth results and fix the borders issues:

Any comment or suggestions will be greatly appreciated.

[lllyasviel]

Hi victorca25, your implementation looks cool! Some of my thoughts:

1. I have uploaded a python knapsack solver in this repo and you can see them in the last several commits. That was what we used in our ECCV submission. That solver is a better approximation for the knapsack problem and that codes are very readable. But note that the actual smoothing still happens in matlab. If you use that maybe you will need to manually copy image files between python and matlab backward and forward many times to compute one image.... But I have TV implemented in pure python and you can find it in my last several commits.

2. You have RTV implemented in python?! Wow that looks super cool. I have not find any correct python implementation of RTV yet on GitHub. If you need maybe you can share your python RTV implementations for me to take a look.

3. The biggest problem of python is that it does not have good enough solver for sparse matrix. If you implement WLS or TV maybe python is OK, but when you need L1 then it is extremely difficult. And unfortunately, L1 is the current best image smoothing algorithm when we want to maximize the quality and have enough time to wait for the results.

Not very sure whether you are familiar with a term called "affinity matrix". For example, let us say we have an image with shape [H,W,C] (height, width, channel). Because you have studied WLS so I assume that you know that pixels can share a "weight function" for example, if pixel (x1,y1) and (x2,y2) share a weight of w, we may write something like this
w=weight_function(x1,y1,x2,y2)
and an "affinity matrix" is a matrix that can record all weights, for example it can be of a shape [H,W,H,W], and
w= affinity_matrix[x1,y1,x2,y2]
means that the pixel (x1,y1) and (x2,y2) share a weight of w. You can see that "affinity_matrix" is extremely large and sparse (it may have lots of zero). Now if you want to compute the inverse matrix of "affinity_matrix", matlab only need minutes but python needs hours.
EAP is not the only project that suffers from this python limitation. Another example is "Semantic Soft Segmentation"(http://yaksoy.github.io/sss/) If you take a look at their code, you will find they also use a mixture of python and matlab as a workaround. If you have a very good solution maybe you can also help them.

[victorca25]

Thank you very much for your detailed feedback!

I'll study the knapsack solver and see what I can learn. I don't know for certain if my RTV code produces 1:1 the same results as the original, but it's at least close. I also used a sparse matrix in that case, similar to WLS and TV, but now I'll try with L0 that appears to be easier to understand than L1 and if that works I'll experiment with L1 as well.

I have some knowledge about graph theory and affinity matrices, but your explanation helps a lot to visualize the problem. If it goes too bad with that alternative, I'll try with different options for the solver to see if any is at least useable.

Once again, thanks for the comments! Let's see how it goes. Cheers!

[woctordho]

Hi all, I've also written a Python implementation here:
https://github.com/Lunatic-Works/neural-tools/blob/master/color_simplify.py

My friends and I have been using it for a while in our games. The parameters are not exactly the same as in this repo, but at least it works well visually.

Now if you want to compute the inverse matrix of "affinity_matrix", matlab only need minutes but python needs hours.

I think if we construct the matrix in the DIA sparse format, then convert it to CSR and use an iterative solver, it can finish in minutes.