lllyasviel / sketchKeras

an u-net with some algorithm to take sketch from paints

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

sketchKeras

An u-net with some algorithm to take the sketch from a painting.

requirement

  • Keras
  • Opencv
  • tensorflow/theano
  • numpy

download mod

see release

performance

Currently there are many edge-detecting algrithoms or nerual networks. But few of them has good performance on paintings, espatially those from comic or animate. Most of these existing methods just detect the edge and then add lines to the edge. However, we need a method to convert the painting to a sketch which looks like a painter drawed the outline of picture. It is important when we want to train a nerual networks to colorlize pictures.(Paper is on the way)

Here is a example of artificial sketch for reference.

goal

Here is a conclusion of existing methods to handle this problem.

  • use opencv and implement a high-pass effect to get the edge
  • train a nerual network (HED Edge Detect)/(PaintsChainer's lnet)
  • use this sketchKeras which combined algorithm and nerual networks

Take this pic as an example (Picture is from internet and I am finding the author.)

pic

If we use the high-pass algorithm via opencv or something else, we may get this one:

pic

As we can see, the result is far from artificial sketch. To achieve better performance, we may modify the parameters and enhance the pic, then it comes this one:

pic

The result is still not good. People like to add shadow to their drawing by add dense lines or points and these will become noise and disturb the high-pass algorithm. It is apprent that we can modify the parameters or use denoise methods to improve it, but drawings differ from one another and it is impossible to handle these automatically.

Then let us try the lnet of PaintsChainer (similar to HED)

pic

The result from nerual networks looks different from those from algorithm. However, this is still not so good. The author of PaintsChainer use threslod to avoid noise and normalize the line, as this:

pic

In this picture, we can see clearly that the noise, espeacilly near eyes and in the shadow of hair. "threslod" can filter some noise but some useful lines is also dropped. Last but not least, the lines are too coarse and thick. Here is a reference of thresloded artificial sketch:

pic

As we can see, though the picture is denoised by "threslod", it differs far from real artificial sketch. So it remains much improvement place in paintsChainer.

Finally, let us see the result of sketchKeras:

pic

pic

pic

pic

The four results are generated by sketchKeras. You can use your favorite one to generate your own training dataset for colorize networks.


another example

(picture is from internet and I am finding the author.) raw picture:

pic

opencv and high-pass (all detail and noise remained so it is not suitable for training a colorize network)

pic

opencv and high-pass enhanced (still not so good and we can see the pic is going to become a grayscaled detailed pic but not a sketch)

pic

paintsChainer's lnet (all detail and noise remained, espcially the hair)

pic

paintsChainer's lnet (thresloded) (just look at the hair)

pic

Pics below are generated by sketchKeras. It can drop some noise and unimportant detail to achieve better performance.

pic

pic

pic

ability to generate colored highlighted sketch

As you can see, sketchKeras has the ability to generate colored highlighted sketch.

pic

About

an u-net with some algorithm to take sketch from paints

License:Apache License 2.0


Languages

Language:Python 100.0%