Neural Style Transfer : Gallery

A couple weeks ago, I briefly described how a convolutional neural network (CNN) can be hacked to "understand" style from a source image, and "port" a similar style to an image. (See Codeur confiné - season 2 - Grand Finale)

Thanks to an optimized code and improved hardware, I was able to do execute a lot of different tests, and get a better intuition of what style is, from a CNN point of view.

My (current) gallery is available on Github here.

Here is the story of my journey through Neural Style Transfer experiments !

Experimental design

In order to analyse how style is understood, all the models were generated using the same protocol.

• Image to process is always the same : a picture of me, size 554x913
• CNN used is VGG-19, initialized with weights computed from MatConvNet
• When computing cost, the ratio content/style is one fourth.
• Gradient descent is performed over 2k iterations

(Processing time : about 10 hours / experiment)

Experiment 1 : Van Gogh - Self Portrait with Pallette

I first started to adapt style from Van Gogh's painting 'Self Portrait With Pallette'.

There are clearly visible patterns, like the blue color or the lot of small 'segments' (from CNN point of view).

This first experiment was quite successful :

• The main colors, blue and orange, from painting, are reused in the result image.
• The image itself has been redrawn using the 'small segment style

What was not ported

• My beard, my hairs remain dark
• My face isn't as pale as Vincent's
• The whole background is somehow kept
• And I don't have any pallette :)

Experiment 2 : Picasso - Portrait of Dora Maar

I moved to another experiment, with a painting from Picasso.

Here, color range is wider, with a bit more flashy yellow.

Some other visual patterns : the lines in the background, the curves used to draw Dora's hairs, ...

Interestingly, this experiment shows something quite different from Van Gogh's.

• Skin color is (partially) adapted
• Beard / hairs adopted the 'curve' style
• Existing in the background were transformed to get closer to Picasso's, though new lines were not added everywhere

What was not ported

• My face still looks like a real face
• Eye color : the green / orange thing.

Experiment 3 : Picasso - Guernica

One of the world's most famous painting : Guernica. The main patterns : colors of course, and the fact that "objects" are delimited using color differences.

Wow !

Colors : ok.

Some 'objects' are superimposed over my face (zoom to see them more in detail) : these aren't present in Guernica, though they could, regarding their style :)

What was not ported :

• My face still looks like a real face
• The original horse (or, what looks to me to to be a horse) was drawn using small vertical dashes pattern, it represents a fairly good proportion of the original painting, though it is not ported to the result.

Experiment 4 : Chagall - The creation of Man

This painting from Chagall is quite different from the previous experiments.

Although a human can get a sense of its style, the patterns are hard to detect for a CNN.

There are main colors like blue and yellow, but there is no visual pattern (like the segments / curves in previous tests), no object to detect, ...

And due to this, the style transfer won't be perfect :(

When you learn, there is no error, no failure.

The result clearly not looks like Chagall's source painting.

• The main colors are somehow changed to match source.
• If you zoom on my skin you may see some patterns that may be part of the origanl.

But that's all.

However, this helps to get a more accurate vision of what CNN considers as style and what is does not.

Experiment 5 : Mondrian

I don't exactly remember which painting from Mondrian I used.

However, it was some classical Mondrian style : uniformly colored squares and rectangles drawn by intersection of lines, un a white background.

Although it's not perfect, it looks nice (or interesting).

• My glasses became partially squared
• Colors within regions are not perfectly uniform yet, though it may be fixed through more iterations
• Almost-straight lines (like my mouth) are rendered using more straight lines

Experiment 6 : Klimt - The Kiss

In 'The kiss', there are visual patterns like :

• Yellow colors of course
• The rectangles (white, black)
• The colored circles (small flowers)
• The concentric circles

...

Most of the source styles were ported to the result where appropriate. For example:

• My shirt is now drawn using the "ground style" from source
• Some parts of my hairs / beard using dark rectangles
• The remaining parts of my hairs / beard using the same "ground style"
• And of course, the main color used is yellow.

What was not ported : the source background has a very recognizable style that cannot be found in the result, although it used a very large part of the original painting.

Experiment 7 : Kandinsky - Delicate Tension. No. 85

I was actually not expecting a very interesting result for this one, and I was very surprised.

The patterns in this image are

• The white color. Ok, it's the background, but it's also the most present color
• Purely geometrical shapes (segments, circles, triangles, ...)
• Colors : plenty of, but mainly used with the area enclosed in geometrical shapes.

The white color has been ported, or when not appropriate a gray was used instead.

Some parts were re-drawn using pseudo geometrical shapes.

Usage of colors is limited to the geometrical shapes.

What was not ported:

• The gray portions should have been white, or replaced by colored shaped
• Some parts are not perfect shapes (my glasses should have been circles for example)

Experiment 8 : Vasarely - Zèbres

At this point, I realized that colors are generally a pattern easier to detect and apply. Shapes and other patterns are generally well applied if they're delimited using different colors.

But what if I use something like Vasarely's Zebra ?

Patterns are easily identifiable : top-left to bottom-right lines, black or white, slightly deformed, just enough to let us guess a model behind :)

Ok, it's a bit blurry. Although it may be fixable with more iterations.

But it's impressive anyway : these are not almost-perfect lines from a corner to the opposite, although the global 'idea' of the painting is here : make the viewer guess my presence behind this set of lines.

Note that the diagonals are oriented differently. The only reason to this is that I use a rotated version of the zebra painting (to get a picture shape closer to my portrait's).

Experiment 9 : Moving circles

Vasarely produced a nice result. It's like an optical illusion, applied to a picture.

So, what if I try with a another kind of optical illusion ? Like the "moving circles" here ?

(No, it's not an animated Gif, circles aren't moving, but your brain decided that they are, so sort it out amongst you and your brain ^^)

Note that there are some style elements : the whole image is made of degenerated circles, black, green, blue, plus a white background.

Circles have different sizes, and orientations.

Well, result is... interesting.

It's not actually the optical illusion mode I initially expected to produce.

However, the whole picture have been redrawn using green or blue circles, different shapes, different orientation, ...

Another similarity : both sting eyes :)

Experiment 10 : Chagall - Windows, Hadassah Medical Center

I though stained glass drawings were worth a try.

• There are few colors, and shapes drawn with regular edges, not too many circles
• One shape generally has only one color inside.

Note: I only focused on the middle window (the orange-ish one).

In my humble opinion, the result is pretty impressive.

It somehow tried to split the image into regions, each of these being

• drawn as a polygon
• colored with one single color
• this color is picked from the one available in the window (or a similar one)

This experiment probably explains well what style is when detected the way I used.

Experiment 11 : Picasso - Bay of Cannes

Previous Picasso experiments were quite successful, so I decided to try another one.

I chose the Bay of Cannes, it probably has a less recognizable style than the previous tests, but let's try !

A less impressive result than the other Picasso...

Colors as perfectly kept, as usual.

But my skin is like tattooed using the Cannes' buildings style, my beard using palm tree leaves, ...

While Chagall's window explains well what style is, this one explains what style is not.

Note for later : it may be interesting to transfer style from a portrait to another, using a different network than CNN (one more trained on human faces).

Experiment 12 : Monet - Impression, Soleil levant

Next step : try to transfer impressionist style, using Monet's painting.

Mainly blue, a few orange.

Not that much recognizable patterns (the sea, maybe, but that's all).

Result is nice, but...

Well, it seems that the only operation performed was to adapt colors.

Ok, this is not true, but the result is less spectacular than previous tests !

Experiment 13 : Obey - Obama

When I started to work on this one, I already knew that the result won't be what we may have wanted.

From the CNN point of view, the only style applied here is the set of colors.

Hence, we should not expect more than a basic redraw, using the same color set.

Well, if you look closer at Shepard Fairey's paintings, you may notice some small sets of small parallel horizontal lines, generally over regions where color changes from one to another (using then lines of both colors).

But these lines are more here to create some new pseudo-colors using the basic palette. There are not parts of the contents.

On this result, well, colors are kep, and you can also see (the window on the right, my left glass, ...) that these small parallel lines were introduced, for some reasons I can't figure out :)

Experiment 14 : Braque - Le Viaduc à L'Estaque

This painting is not considered as part of the cubism period, more the pre-cubism one.

Anyway, based on all the previous experiments, it seems that we have here all the perfect ingredients to get a nice result :

• a small color set. Orange/yellow, green, that's almost all.
• painting pattern (small hashes in the background, or trees)
• use of geometrical shapes (houses are composed by different polygons).

So, let's try !

Looks nice ^^

As mentioned, it seems that every point has been mapped to a different color, the best match from the initial palette set for each point.

The tree-painting-style has been ported to my beard as well.

Note that I also performed a cubism test, my last experiment. It's quite different from this pre-cubism one.

Experiment 15 : Mosaic, Plato's academy

From all the previous tests, it appeared that limited colors + simple geometric shapes is a good combo to produce a nice result.

Hence, I gave a try to a roman mosaic

• There are indeed a very few different colors.
• And by definition, it's made of small squares !

But the result is ... interesting, and unexpected.

As usual, colors => ok.

But instead of having small squares, I have some kind of tiny squarish dots...

Experiment 16 : Picabia - La procession

As I said, for this last experiment, I tested cubism style applied to NST.

Well, again, this looks promising:

• Few colors
• Basic shapes

I already knew the result would be nice, without being able to figure it out exactly ^^

I like it ^^

The picture has been split into geometrical shape (polygon), and then each has been filled using a single color, picked from the palette set.

Conclusion and next step

Thanks to all these tests, it appears that the style that is well understood by the neural network is mainly a mix of color + shape.

That's why styles like cubism are well detected and transferred, while impressionism for example may be less well transfered.

Next step : change the underlying VGG-19 network and replace it (maybe by FaceNet), and then perform tests with portraits.