ml -2 cw -2

Analyzing and Improving the Image Quality of StyleGAN

Authors:Tero Karras, NVIDIA Samuli Laine, NVIDIA Miika Aittala, NVIDIA Janne Hellsten, NVIDIA Jaakko Lehtinen, NVIDIA and Aalto UniversityTimo Aila, NVIDIA

2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

Index

Aim

Image quality and generator smoothness

Conclusion

GAN

Phase Artifact

References

Resolution Usage

StyleGAN

Droplet Artifact

Project of Latent Space to Images

Aim

"Fixing characteristic artifacts in StyleGAN and improving the quality further."

01

Generative Adversial Networks

70%

02

StyleGAN (2019)

• Data driven unconditional generative modeling
• Unconventional generator architecture
• Incorporates stochastic variation
• Generates high-resolution images

03

"But the SOTA has visible characteristic artifacts"

Distortion

Bluriness

Visual Anomalies

04

Issue - 1

Droplet Artifact

70%

+ info

Discriminator should have been able to detect the "Droplet Effect"

05

Problem Description

Problem: AdaIN Operation

Normalises Mean and Variance

Destroying information in the magnitude of features

Why?

Generator sneaks signal strength information

Hypothesis: Removal of Normalisation will remove the droplet artifact

06

Solution

Removing normalisation impacts scale-specific controls

07

Issue - 1

Droplet Artifact

Removes droplet effect and retains full controllability

70%

Remove the effect of s: Aim of Instance Normalisation

+ info

11

StyleGAN2 Improvement

Weight demodulation

11

Image quality and generator smoothness

PPL is inversely proportional to the Image Quality

12

Issue - 2

What conclusions can we draw from the relationship?

70%

1. Generator stretches the region of latent space to penalise broken images and improve the quality of images.
2. We cannot encourage low PPL as it would return a zero recall.

13

The Fix

Path Length Regularisation

Adapts the concept of "lazy regulariser". Uses the concepts of Jacobian matrix and Euclidean Norm to provide more effective generator models

Regularisation terms are computed less frequently. that the main loss function. Lowers computational cost

Lazy Regularisation

14

Issue - 3

Phase Artifact

70%

+ info

15

What is the potential cause?

Progressive Growing

1. Resolution of images are gradually increased at each step.
2. At each step, high frequency details are generated for the current resolution, which helps capture fine grain details.
3. This leads to the trained network having high frequency details.
4. The network becomes sensitive to small changes.
5. This compromises network shift invariance, which is the network's ability to identify shift in location or position.

Rises the need for change in Arhcitecture!

16

Comparison of Architectures

Progressive Growing

Using "skip" generator and a "residual" discriminator without progressive growing

17

StyleGAN2 Improvement

Without Progressive Growing

18

Resolution Usage

Start of the training the learning is similar to progressive growing. But at the end of training high resolution fails to dominate.

Solution: Doubling the number of feature maps.

Generator has to focus on "low-resolution" features and then slowly start focussing on finer details (key aspect of progressive growing).

The cause of the problem: Capacity problem

Works in line with "progressive growing" and the hypothesised "expectation"

The architectures discussed allow it

INFO

19

Projection of Images to Latent Space

• Use ramped-down noise during optimization to explore the latent space thoroughly.
• Optimize the stochastic noise inputs of StyleGAN generator.
• Regulate the stochastic noise inputs to avoid carrying coherent signals.

Detection of the generated image back to the source is important

Conclusions

• StyleGAN2 addressed image quality issues in StyleGAN and improved image quality in various datasets, especially in motion.
• It's easier to attribute a generated image to its source using StyleGAN2.
• Training performance was improved, with faster training time and energy usage comparable to the original StyleGAN.
• The project consumed a significant amount of electricity and computation resources.
• Future improvements in path length regularization could be explored, and finding ways to reduce training data requirements is important for practical deployment of GANs, especially with datasets with intrinsic variation.

21

References

1. Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., & Aila, T. (2019), Analyzing and improving the image quality of StyleGAN, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 8110-8119).
2. Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., & Aila, T. (2020). Supplemental Material: Analyzing and Improving the Image Quality of StyleGAN. NVIDIA
3. https://github.com/NVlabs/stylegan2

70%

" Thank you!"