Lectures
You will be able to download the lectures here. We will try to upload lectures prior to their corresponding classes.
-
01/31/2022 Data-Driven Graphics
tl;dr: Copy-paste smartly from large-scale data to compose an image [pdf] [pptx]
Reading list:
- Poisson Image Editing, Pérez et al. in SIGGRAPH, 2003
- Scene Completion using Millions of Photographs, Hays et al. in TOG, 2007
- CG2Real: Improving the Realism of Computer Generated Images using a Large Collection of Photographs, Johnson et al. in TVCG, 2010
- Modeling the shape of the scene: A holistic representation of the spatial envelope, Oliva et al. in IJCV, 2001
- Semantic photo synthesis, Johnson et al. in Computer Graphics Forum, 2006
- Sketch2Photo: internet image montage, Chen et al. in SIGGRAPHA, 2019
- Photo Clip Art, Lalonde et al. in SIGGRAPH, 2007
- ShadowDraw: Real-Time User Guidance for Freehand Drawing, Lee et al. in TOG, 2011
- AverageExplorer: Interactive Exploration and Alignment of Visual Data Collections, Zhu et al. in SIGGRAPH, 2014
-
02/02/2022 Data-Driven Graphics (student presentation) + image blending
tl;dr: Cool papers in data-driven graphics. [pdf] [pptx]
Reading list:
- Joint Bilateral Upsampling, 2007, presented by Jason Xu and Joyce Zhang
- Seam Carving for Content-Aware Image Resizing, Avidan et al. in SIGGRAPH, 2007, presented by Emma Liu
- Interactive Digital Photomontage, Agarwala et al. in SIGGRAPH, 2004
- GrabCut -Interactive Foreground Extraction using Iterated Graph Cuts, Rother et al. in SIGGRAPH, 2004
- Hybrid Images, Oliva et al. in SIGGRAPH, 2006
- Image Deformation Using Moving Least Squares, Schaefer et al. in SIGGRAPH, 2006
- Sketch2Photo: internet image montage, Chen et al. in SIGGRAPH Asia 2009
- Photo Clip Art, Lalonde et al. in SIGGARPH 2007
- Graphcut Textures: Image and Video Synthesis Using Graph Cuts, 2003
- ShadowDraw: Real-Time User Guidance for Freehand Drawing, 2011
- Summarizing Visual Data Using Bidirectional Similarity, 2008
-
02/07/2022 Convolutional Network for Image Synthesis
tl;dr: Convolutional networks aids in achieving higher quality images by leveraging higher-level knowledge of objects. [pdf] [pptx]
Reading list:
- Deep Learning Book, Chapter 6 and 9.
- Szeliski Book, Chapter 5.3 and 5.4.
- Gradient-based learning applied to document recognition, Lecun et al., Proc of IEEE, 1998.
- Receptive Fields of Single Neurones in the Cat’s Striate Cortex, Huber and Wiesel, J. Physiol, 1959.
- Learning to Generate Chairs, Tables and Cars with Convolutional Networks, Dosovitskiy et al., PAMI 2017 (CVPR 2015)
- Deconvolution and Checkerboard Artifacts
- Colorful Image Colorization, Zhang et al., ECCV 2016.
- Deep Learning, LeCun, Bengio, and Hinton, Nature 2015.
-
02/09/2022 Perceptual Loss, Generative Adversarial Networks (part 1) [pdf] [pptx]
Reading List:
- Szeliski Book, Chapter 5.5.3
- Generative Adversarial Networks, Goodfellow et al., NeurIPS 2014.
- GAN Tutorial (NeurIPS 2016), by Ian Goodfellow.
- ICCV 2017 Tutorial on GANs.
- CVPR 2018 Tutorial on GANs.
- Image style transfer using convolutional neural networks, Gatys et al., CVPR 2016.
- Perceptual Losses for Real-Time Style Transfer and Super-Resolution, Johnson et al., ECCV 2016.
- Generating images with perceptual similarity metrics based on deep networks. Dosovitskiy and Brox. NeurIPS, 2016
- The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. Zhang et al., CVPR 2018
- Image Manipulation with Perceptual Discriminators, Sungatullina et al., ECCV 2018.
-
02/14/2022 Generative Adversarial Networks (part 2) [pptx] [pdf]
Reading List:
- Szeliski Book, Chapter 5.5.3
- Generative Adversarial Networks, Goodfellow et al., NeurIPS 2014.
- GAN Tutorial (NeurIPS 2016), by Ian Goodfellow.
- ICCV 2017 Tutorial on GANs.
- CVPR 2018 Tutorial on GANs.
- Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks, Radford et al., ICLR 2016
- Wasserstein GAN, Arjovsky et al, 2017
- Improved Training of Wasserstein GANs, Gulrajani et al., NeurIPS 2017
- Least Squares Generative Adversarial Networks, Mao et al. 2017
- Progressive Growing of GANs for Improved Quality, Stability, and Variation, Karras et al, ICLR 2018
- Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks, Denton et al, 2015
- Spectral Normalization for Generative Adversarial Networks, Miyato et al., ICLR 2018
- A Style-Based Generator Architecture for Generative Adversarial Networks, Karras et al., CVPR 2019
- Analyzing and Improving the Image Quality of StyleGAN, Karras et al., 2020
- Alias-Free Generative Adversarial Networks, Karras et al., 2021
- Training Generative Adversarial Networks with Limited Data, Karras et al., 2020
- Differentiable Augmentation for Data-Efficient GAN Training, Zhao et al., 2020
- Projected GANs Converge Faster, Sauer et al., 2021
- Ensembling Off-the-shelf Models for GAN Training, Kumari et al., 2021
-
02/16/2022 Generative Models Zoo
tl;dr: Generative models are much more than GANs, e.g. variational autoencoder, normalizing flows, etc.) [pptx] [pdf]
Reading List:
- Deep Learning Book, Chapter 20
- Learning Deep Generative Models, Ruslan Salakhutdinov, Annual Review of Applied Statistics 2015
- Auto-Encoding Variational Bayes, Kingma and Welling, 2013
- Density estimation using Real NVP, Dinh et al, ICLR 2017
- Conditional Image Generation with PixelCNN Decoders, Oord et al, 2016
- Pixel Recurrent Neural Networks, Oord et al, 2016
- Generating Diverse High-Fidelity Images with VQ-VAE-2, Razavi et al., 2019
- Glow: Generative flow with invertible 1x1 convolutions, Kingma et al., 2018
- Denoising Diffusion Probabilistic Models, Ho et al., 2020
- Denoising Diffusion Implicit Models, Song et al., 2021
-
02/21/2022 Generative Models (student presentation)
tl;dr: Cool papers about generative models
Reading List:
- Deep Learning Book, Chapters 14 and 20
- A Style-Based Generator Architecture for Generative Adversarial Networks, Karras et al, CVPR 2019
- Large Scale GAN Training for High Fidelity Natural Image Synthesis (BigGAN), Brock et al., ICLR 2019
- Generating Diverse High-Fidelity Images with VQ-VAE-2, Razavi et al., 2019
- Conditional Image Generation with PixelCNN Decoders, Oord et al., NeurIPS 2016
- Glow: Generative Flow with Invertible 1x1 Convolutions, Kingma et al., ICLR 2018
- Analyzing and Improving the Image Quality of StyleGAN, Karras et al, 2019
- Autoencoding beyond pixels using a learned similarity metric, Larsen et al, CVPR 2016
- Denoising Diffusion Probabilistic Models, Ho et al., 2020
- Denoising Diffusion Implicit Models, Song et al., 2021
- Large scale adversarial representation learning (BigBiGAN), Donahue et al., ICLR 2019
- Alias-Free Generative Adversarial Networks (StyleGAN3), Karras et al., NeurIPS 2021
- SinGAN: Learning a Generative Model from a Single Natural Image (SinGAN), Shaham et al., ICCV 2019
- Score-Based Generative Modeling through Stochastic Differential Equations (SDE), Song et al., ICLR 2021
-
02/23/2022 Conditional GANs, Image-to-Image Translation (part 1) [pptx] [pdf]
Reading List:
- Image-to-Image Translation with Conditional Adversarial Networks, Isola et al, CVPR 2017
- Photographic Image Synthesis with Cascaded Refinement Networks, Chen et al, ICCV 2017
- Toward Multimodal Image-to-Image Translation, Zhu et al, NeurIPS 2017
- Multi-Agent Diverse Generative Adversarial Networks, Ghosh et al, CVPR 2018
- Diversity-Sensitive Conditional Generative Adversarial Networks, Yang et al, ICLR 2019
- High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs, Wang et al, CVPR 2018.
- cGANs with Projection Discriminator, Miyato et al., ICLR 2018
- Large Scale GAN Training for High Fidelity Natural Image Synthesis, ICLR 2019
- Semantic Image Synthesis with Spatially-Adaptive Normalization, Park et al, CVPR 2019
- Shapes and Context: In-the-Wild Image Synthesis & Manipulation, Bansal et al, CVPR 2019
-
02/28/2022 Conditional GANs, Image-to-Image Translation (Part 2) [pptx] [pdf]
Reading List:
- Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks, Zhu et al, ICCV 2017
- Unsupervised Image-to-Image Translation Networks, Liu et al., NeurIPS 2017
- Learning from Simulated and Unsupervised Images through Adversarial Training, Shrivastava et al, CVPR 2017
- Unsupervised Cross-Domain Image Generation, Taigman et al, ICLR 2017
- Multimodal Unsupervised Image-to-Image Translation, Huang et al., ECCV 2018
- Representation learning with contrastive predictive coding, Oord et al., arXiv 2018.
- Contrastive learning for unpaired image-to-image translation, Park et al., ECCV 2020
-
03/02/2022 Style and Content, Texture Synthesis (part 1)
tl;dr: How to control the style and content of your image with Deep Learning [pptx] [pdf]
Reading List:
- Separating Style and Content, Tenenbaum & Freeman, Neurips 1996
- Representation Learning with Contrastive Predictive Coding, van den Oord et al
- A Simple Framework for Contrastive Learning of Visual Representations, Chen et al, ICML 2020
- Contrastive Learning for Unpaired Image-to-Image Translation, Park et al, ECCV 2020
- Swapping Autoencoder for Deep Image Manipulation, Park et al., NeurIPS 2020.
- FineGAN: Unsupervised Hierarchical Disentanglement for Fine-Grained Object Generation and Discovery, Singh et al., CVPR 2019.
- Texture Synthesis by Non-parametric Sampling, Efros and Leung, ICCV 1999
- Image Quilting for Texture Synthesis and Transfer, Efros and Freeman, SIGGRAPH 2001.
- Image Analogies, Hertzmann et al, SIGGRAPH 2001
-
03/14/2022 Style and Content, Texture Synthesis (part 2)
tl;dr: How to control the style and content of your image with Deep Learning, part 2 [pdf] [pptx]
Reading List:
- PatchMatch: A Randomized Correspondence Algorithm for Structural Image Editing, Barnes et al, SIGGRAPH 2009
- Visual Attribute Transfer through Deep Image Analogy, Liao et al, SIGGRAPH 2017
- Deep Photo Style Transfer, Luan et al, CVPR 2017
- Non-Stationary Texture Synthesis by Adversarial Expansion, Zhou et al, SIGGRAPH 2018
- Deep Image Harmonization, Luan et al., EGSR 2018
- Cross-domain Correspondence Learning for Exemplar-based Image Translation, Zhang et al., CVPR 2020.
-
03/16/2022 Conditional Image Synthesis (Student Presentation)
tl;dr:
Reading List:
- Semi-parametric Image Synthesis, Qi et al, CVPR 2018
- StarGAN v2: Diverse Image Synthesis for Multiple Domains, Choi et al, CVPR 2020
- FUNIT: Few-Shot Unsupervised Image-to-Image Translation, Liu et al, ICCV 2019
- The Perception-Distortion Tradeoff, Blau et al, CVPR 2018
- Style Transfer for Headshot Portraits, Shih et al, SIGGRAPH 2014
- Image Analogies, Hertzmann et al., SIGGRAPH 2001
- Controlling Perceptual Factors in Neural Style Transfer, Gatys et al., CVPR 2017
- Visual attribute transfer through deep image analogy, Liao et al, SIGGRAPH 2017
- Deep Photo Style Transfer, Luan et al., CVPR 2017
- Non-stationary Texture Synthesis by Adversarial Expansion, Zhou et al., SIGGRAPH 2018
- Cross-domain Correspondence Learning for Exemplar-based Image Translation, Zhang et al., CVPR 2020
- Sean: Image synthesis with semantic region-adaptive normalization, Zhu et al., CVPR 2020
- A closed-form solution to photorealistic image stylization, Li et al., ECCV 2018
- Multimodal Conditional Image Synthesis with Product-of-Experts GANs, Huang et al., arXiv 2021
-
03/21/2022 Image Editing and Optimization
tl;dr: Use an optimization algorithm in a learned space to achieve photo manipulation [pptx] [pdf]
Reading List:
- Generative Visual Manipulation on the Natural Image Manifold, Zhu et al, ECCV 2016
- Neural Photo Editing with Introspective Adversarial Networks, Brock et al, ICLR 2017
- Semantic Photo Manipulation with a Generative Image Prior, Bau et al, SIGGRAPH 2019
- Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?, Abdal et al, ICCV 2019
- StyleGAN2: Analyzing and Improving the Image Quality of StyleGAN., Karras et al, CVPR 2020
- GAN Inversion: A Survey. Xia et al. 2021
-
03/23/2022 Image Editing and Optimization (part 2)
tl;dr: Use an optimization algorithm in a learned space to achieve photo manipulation [pptx] [pdf]
Reading List:
- GAN Dissection: Visualizing and Understanding Generative Adversarial Networks, Bau et al, ICLR 2019
- GANSpace: Discovering Interpretable GAN Controls, Härkönen et al, NeurIPS 2020
- StyleSpace Analysis: Disentangled Controls for StyleGAN Image Generation, Wu et al., CVPR 2021
- Exploiting Deep Generative Prior for Versatile Image Restoration and Manipulation, Pan et al., ECCV 2020
- On the “steerability” of generative adversarial networks, Jahanian et al, ICLR 2020
- Editing in Style: Uncovering the Local Semantics of GANs., Collins et al, CVPR 2020
- Closed-Form Factorization of Latent Semantics in GANs., Shen et al, CVPR 2021
- Pivotal Tuning for Latent-based Editing of Real Images, Roich et al., 2021
-
03/28/2022 Image Editing and Optimization (student presentations)
tl;dr: Use an optimization algorithm in a learned space to achieve photo manipulation
- GAN Dissection: Visualizing and Understanding Generative Adversarial Networks, Bau et al, ICLR 2019
- GANSpace: Discovering Interpretable GAN Controls, Härkönen et al, NeurIPS 2020
- Deep Image Prior, Ulyanov et al, CVPR 2018
- GANSpace: Discovering Interpretable GAN Controls, Härkönen et al, NeurIPS 2020
- Colorization using Optimization, Levin et al, SIGGRAPH 2004
- AppProp: all-pairs appearance-space edit propagation, An et al, SIGGRAPH 2008
- Image Smoothing via L0 Gradient Minimization, Xu et al, SIGGRAPH Asia 2011
- On the “steerability” of generative adversarial networks, Jahanian et al, ICLR 202
- TryOnGAN: Body-Aware Try-On via Layered Interpolation, Du et al, SIGGRAPH 2021
- StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery, Patashnik et al., ICCV 2021
- Encoding in style: a stylegan encoder for image-to-image translation, Richardson et al., CVPR 2021
- Rewriting a Deep Generative Model, Bau et al., ECCV 2020
- StyleGAN-NADA: CLIP-Guided Domain Adaptation of Image Generators, Gal et al., Arxiv 2021
-
03/30/2022 Face Modeling (part 1)
tl;dr: What special techniques can we use to generate and manipulate realistic faces? [pptx] [pdf]
Reading List:
- Active Appearance Models, Cootes et al, ECCV 1998
- Composite Portraits, Galton, Nature 1878
- Manipulating Facial Appearance through Shape and Color, Rowland & Perret, CG&A, 1995
- A Morphable Model For The Synthesis Of 3D Faces, Blanz & Vetter, SIGGRAPH 1999
- Eigenfaces for Recognition, Turk and Pentland, Journal of cognitive neuroscience, 1991
- Exploring Photobios, Kemelmacher-Shlizerman et al., SIGGRAPH 2011
-
04/04/2022 Face Modeling (part 2)
tl;dr: What special techniques can we use to generate and manipulate realistic faces? [pptx] [pdf]
Reading List:
- Reducing the dimensionality of data with neural networks, Hinton & Salakhutdinov, Science 2006
- Deep Video Portraits, Kim et al, SIGGRAPH 2018
- StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation, Choi et al, CVPR 2018
- StarGAN v2: Diverse Image Synthesis for Multiple Domains, Choi et al, CVPR 2020
- GANimation: Anatomically-aware Facial Animation from a Single Image, Pumarola et al, ECCV 2018
- Image2StyleGAN++: How to Edit the Embedded Images?, Abdal et al, CVPR 2020
- StyleRig: Rigging StyleGAN for 3D Control over Portrait Images, Tewari et al, CVPR 2020
-
04/06/2022 Face Modeling (student presentations)
tl;dr: What special techniques can we use to generate and manipulate realistic faces?
Reading List:
- Face2Face: Real-time Face Capture and Reenactment of RGB Videos, Theis et al, CVPR 2016
- SfSNet: Learning Shape, Reflectance and Illuminance of Faces ‘in the wild’, Sengputa et al, CVPR 2018
- paGAN: Real-time Avatars Using Dynamic Textures, Nagano et al, SIGGRAPH Asia 2018
- Single Image Portrait Relighting, Sun et al, SIGGRAPH 2019
- Intuitive, Interactive Beard and Hair Synthesis with Generative Models, Olzewski et al., CVPR 2020
- Time Travel Rephotography, Luo et al, arXiV 2020
- One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing, Wang et al., CVPR 2021
- Lifespan Age Transformation Synthesis, Or-el et al., ECCV 2020
- StyleFlow : Attribute-conditioned Exploration of StyleGAN-generated Images using Conditional Continuous Normalizing Flows, Abdal et al., ACM TOG 2021
- Text-based Editing of Talking-head Video, Fried et al., SIGGRAPH 2019
-
04/11/2022 3D-aware Synthesis (part I)
tl;dr: Techniques for synthesizing an image from different camera viewpoints [pptx] [pdf]
Reading List:
- QuickTime VR – An Image-Based Approach to Virtual Environment Navigation, Chen, SIGGRAPH 1995
- Stanford Lightfield Website, 2004-2014
- View Synthesis by Appearance Flow, Zhou et al, ECCV 2016
- Single-view view synthesis with multiplane images, Tucker et al, CVPR 2020
- NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis, Mildenhall et al, ECCV 2020
-
04/13/2022 3D-aware Synthesis (part II)
tl;dr: Synthesizing an image of a certain object category from different camera viewpoints [pptx] [pdf]
Reading List:
- Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling, Wu et al, NeurIPS 2017
- Visual Object Networks: Image Generation with Disentangled 3D Representation, Zhu et al, NeurIPS 2018
- Hologan: Unsupervised learning of 3d representations from natural images, Nguyen-Phuoc et al, ICCV 2019
- DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation, Park et al, CVPR 2019
- GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis, Schwarz et al, NeurIPS 2020
- pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis, Chan et al, CVPR 2021
- Editing Conditional Radiance Fields, Liu et al, ICCV 2021
- EG3D: Efficient Geometry-aware 3D Generative Adversarial Networks, Chan et al, arXiv 2021
- StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesis, Gu et al, arXiv 2021
-
04/18/2022 3D + Video (student presentations)
tl;dr: Presentations on exciting work in the 3d + video space!
Reading List:
- Interactive Video Cutout, Wang et al, SIGGRAPH 2005
- Video Object Annotation, Navigation, and Composition, Goldman et al, UIST 2008
- Recycle-GAN: Unsupervised Video Retargeting, Bansal et al, ECCV 2018
- Everybody Dance Now, Chan et al, ICCV 2019
- Adversarial Video Generation on Complex Datasets, Clark et al, arXiv 2019
- Text-based Editing of Talking-head Video, Fried at al, SIGGRAPH 2019
- Background Matting: The World is Your Green Screen, Sengupta et al, CVPR 2020
- World-Consistent Video-to-Video Synthesis, Mallya et al, ECCV 2020
- Third Time’s the Charm? Image and Video Editing with StyleGAN3, Alaluf et al, Arxiv 2021
- Learning to simulate dynamic environments with gamegan, Kim et al, CVPR 2020
- Instant Neural Graphics Primitives with a Multiresolution Hash Encoding, Muller et al, Arxiv 2022
- Light Field Photography with a Hand-held Plenoptic Camera, Ng et al, SIGGRAPH 2005
- Photo tourism: exploring photo collections in 3D, Snavely et al, SIGGRAPH 2006
- Unsupervised Learning of Depth and Ego-Motion from Video, Zhou et al, CVPR 2017
- Plenoxels, Yu et al, CVPR 2022
- Nerfies: Deformable Neural Radiance Fields, Park et al, ICCV 2021
- Unsupervised 3D Neural Rendering of Minecraft Worlds, Hao et al, ICCV 2021
- NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections, Martin-Brualla et al, CVPR 2021
- pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis, Chan et al, CVPR 2021
- EG3D: Efficient Geometry-aware 3D GANs, Chan et al, Arxiv 2021
-
-
04/23/2022
tl;dr: Video Synthesis and Editing [pptx] [pdf]
Reading List:
- Video Textures, Sch¨odl et al., SIGGRAPH 2000
- Controlled Animation of Video Sprites, Schodl et al., UIST 2008
- Generating Videos with Scene Dynamics, Vondrick et al., NeurIPS 2016
- Video-to-Video Synthesis, Vondrick et al., NeurIPS 2016
- Recycle-GAN: Unsupervised Video Retargeting, Bansal et al, ECCV 2018
- Adversarial Video Generation on Complex Datasets, Clark et al, arXiv 2019
- Everybody Dance Now, Chan et al, ICCV 2019
- Few-shot Video-to-Video Synthesis, Wang et al., NeurIPS 2019