What is neural style transfer?
Neural style transfer is a technique that uses two images: A content image and a style image. The style image might be, for example, an artwork by a famous painter. A neural style transfer copies the texture, color, and other aspects of the style image and applies them to the content image. You can see an example of a content image, a style image, and a generated image here:
Images courtesy of Gatys et al., A Neural Algorithm of Artistic Style
Neural style transfer uses a pre-trained Convolutional Neural Network (CNN). Using your content image and your style image, you generate a new image that blends the content image and the style image. You begin with a simple white noise image, or use the content image or style image for optimization efficiency. Then you process the content image, style image, and generated images through the pre-trained neural network. Finally, you calculate loss functions at different layers.
Related information includes a link to an article called Intuitive Guide to Neural Style Transfer, which includes more detail.
There are three types of loss functions:
- Content loss function
- Style loss function
- Total loss function
Let’s look at each in turn.
The content loss function ensures that the content present in the content image is captured in the generated image. In a multiple layer CNN, lower layers are more focused on individual pixel values. Higher layers capture information about content. This means that we use the top CNN layer to define the content loss function in our illustration.
The style loss function ensures that the correlation of activations in all the layers is similar between the style image and the generated image.
The total loss function is the weighted sum of the content cost function and the style loss functions for the generated image. The weights are user-defined hyperparameters that control the amount of content and style that is injected into the generated image. Once the loss is calculated, it can be minimized using backpropagation. Backpropagation optimizes the randomly generated image into a piece of art.