image-to-image¶
pix2pix¶
Image-to-Image Translation with Conditional Adversarial Networks (CVPR 2017) by Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, Alexei A. Efros
https://phillipi.github.io/pix2pix/ |
Image-to-Image Demo
based on conditional GAN, with image as conditional&noise input
dataset: paired image (supervised), e.g. segmentation/edge/map -> real photo, real photo → map
discriminator: PatchGAN which also take conditional as pairs
DTN¶
Unsupervised Cross-Domain Image Generation (ICLR 2017)
Domain Transfer Network(DTN) with Identity Loss.
Identity Loss¶
first proposed in Equation 6 of this paper
The generator trained with this loss will often be more conservative for unknown content. – from CycleGAN FAQ
Spectral Regularization¶
Generative convolutional deep neural networks, e.g. popular GAN architectures, are relying on convolution based up-sampling methods to produce non-scalar outputs like images or video sequences. In this paper, we show that common up-sampling methods, i.e. known as upconvolution or transposed convolution, are causing the inability of such models to reproduce spectral distributions of natural training data correctly. This effect is independent of the underlying architecture and we show that it can be used to easily detect generated data like deepfakes with up to 100% accuracy on public benchmarks. To overcome this drawback of current generative models, we propose to add a novel spectral regularization term to the training optimization objective. We show that this approach not only allows to train spectral consistent GANs that are avoiding high frequency errors. Also, we show that a correct approximation of the frequency spectrum has positive effects on the training stability and output quality of generative networks.