Lectures

You will be able to download the lectures here. We will try to upload lectures prior to their corresponding classes.

• 

  02/01/2020 Introduction: How to Create an Image?
  tl;dr: Introduction to the course, how images are created and manipulated by humans and machines [pdf] [pptx]
  
• 

  02/03/2021 Pointwise Processing and Image Filtering
  tl;dr: Transformations on images which can be done on raw pixel values as well as image filtering techniques [pdf] [pptx]
  

  Reading List:

  • Szeliski Book, Chapters 3.1, 3.2, and 3.5
• 

  02/08/2021 Global and Local Image Warping
  tl;dr: Applying global and local geometric transformations to an image [pdf] [pptx]
  

  Reading list:

  • Szeliski Book, Chapter 3.6
  • Image Deformation using Moving Least Squares, Schaefer et al. in TOG, 2006
• 

  02/10/2021 Data-Driven Graphics
  tl;dr: Copy-paste smartly from large-scale data to compose an image [pdf] [pptx]
  

  Reading list:

  • Poisson Image Editing, Pérez et al. in SIGGRAPH, 2003
  • Scene Completion using Millions of Photographs, Hays et al. in TOG, 2007
  • CG2Real: Improving the Realism of Computer Generated Images using a Large Collection of Photographs, Johnson et al. in TVCG, 2010
  • Modeling the shape of the scene: A holistic representation of the spatial envelope, Oliva et al. in IJCV, 2001
  • Semantic photo synthesis, Johnson et al. in Computer Graphics Forum, 2006
  • Sketch2Photo: internet image montage, Chen et al. in SIGGRAPHA, 2019
  • Photo Clip Art, Lalonde et al. in SIGGRAPH, 2007
  • ShadowDraw: Real-Time User Guidance for Freehand Drawing, Lee et al. in TOG, 2011
  • AverageExplorer: Interactive Exploration and Alignment of Visual Data Collections, Zhu et al. in SIGGRAPH, 2014
• 02/15/2021 Data-Driven Graphics (student presentation)
  tl;dr: Cool papers in data-driven graphics.
  

  Reading list:

  • Interactive Digital Photomontage, Agarwala et al. in SIGGRAPH, 2004, Presentation by Rohan Rao and Harsh Sharma
  • GrabCut -Interactive Foreground Extraction using Iterated Graph Cuts, Rother et al. in SIGGRAPH, 2004
  • Hybrid Images, Oliva et al. in SIGGRAPH, 2006
  • Image Deformation Using Moving Least Squares, Schaefer et al. in SIGGRAPH, 2006, Presentation by Teddy Zhang
  • Seam Carving for Content-Aware Image Resizing, Avidan et al. in SIGGRAPH, 2007, Presentation by Konwoo Kim
  • Sketch2Photo: internet image montage, Chen et al. in SIGGRAPHA, 2019
  • Photo Clip Art, Lalonde et al. in SIGGRAPH, 2007, Presentation by Manuel Rodriguez
  • Coordinates for Instant Image Cloning, Farbman et al. in SIGGRAPH, 2009
• 

  02/17/2021 Convolutional Network for Image Synthesis
  tl;dr: Convolutional networks aids in achieving higher quality images by leveraging higher-level knowledge of objects. [pdf] [pptx]
  

  Reading list:

  • Deep Learning Book, Chapter 6 and 9.
  • Szeliski Book, Chapter 5.3 and 5.4.
  • Gradient-based learning applied to document recognition, Lecun et al., Proc of IEEE, 1998.
  • Receptive Fields of Single Neurones in the Cat’s Striate Cortex, Huber and Wiesel, J. Physiol, 1959.
  • Learning to Generate Chairs, Tables and Cars with Convolutional Networks, Dosovitskiy et al., PAMI 2017 (CVPR 2015)
  • Colorful Image Colorization, Zhang et al., ECCV 2016.
  • Deep Learning, LeCun, Bengio, and Hinton, Nature 2015.
• 

  02/22/2021 Perceptual Loss, Generative Adversarial Networks (part 1) [pdf] [pptx]
  

  Reading List:

  • Szeliski Book, Chapter 5.5.3
  • Generative Adversarial Networks, Goodfellow et al., NeurIPS 2014.
  • GAN Tutorial (NeurIPS 2016), by Ian Goodfellow.
  • ICCV 2017 Tutorial on GANs.
  • CVPR 2018 Tutorial on GANs.
  • Image style transfer using convolutional neural networks, Gatys et al., CVPR 2016.
  • Perceptual Losses for Real-Time Style Transfer and Super-Resolution, Johnson et al., ECCV 2016.
  • Generating images with perceptual similarity metrics based on deep networks. Dosovitskiy and Brox. NeurIPS, 2016
  • The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. Zhang et al., CVPR 2018
• 

  02/24/2021 Generative Adversarial Networks (part 2) [pptx] [pdf]
  

  Reading List:

  • Szeliski Book, Chapter 5.5.3
  • Generative Adversarial Networks, Goodfellow et al., NeurIPS 2014.
  • GAN Tutorial (NeurIPS 2016), by Ian Goodfellow.
  • ICCV 2017 Tutorial on GANs.
  • CVPR 2018 Tutorial on GANs.
  • Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks, Radford et al., ICLR 2016
  • Wasserstein GAN, Arjovsky et al, 2017
  • Improved Training of Wasserstein GANs, Gulrajani et al., NeurIPS 2017
  • Least Squares Generative Adversarial Networks, Mao et al. 2017
  • Progressive Growing of GANs for Improved Quality, Stability, and Variation, Karras et al, ICLR 2018
  • Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks, Denton et al, 2015
  • Spectral Normalization for Generative Adversarial Networks, Miyato et al., ICLR 2018
  • A Style-Based Generator Architecture for Generative Adversarial Networks, Karras et al., CVPR 2019
  • Analyzing and Improving the Image Quality of StyleGAN, Karras et al., 2020
• 

  03/01/2021 Other Generative Models
  tl;dr: Generative models are much more than GANs, e.g. variational autoencoder, normalizing flows, etc.) [pptx] [pdf]
  

  Reading List:

  • Deep Learning Book, Chapter 20
  • Learning Deep Generative Models, Ruslan Salakhutdinov, Annual Review of Applied Statistics 2015
  • Auto-Encoding Variational Bayes, Kingma and Welling, 2013
  • Density estimation using Real NVP, Dinh et al, ICLR 2017
  • Conditional Image Generation with PixelCNN Decoders, Oord et al, 2016
  • Pixel Recurrent Neural Networks, Oord et al, 2016
  • WaveNet: A Generative Model for Raw Audio, Oord et al, 2016
• 03/03/2021 Generative Models (student presentation)
  tl;dr: Cool papers about generative models [pptx] [pdf]
  

  Reading List:

  • Deep Learning Book, Chapters 14 and 20
  • A Style-Based Generator Architecture for Generative Adversarial Networks, Karras et al, CVPR 2019
  • Analyzing and Improving the Image Quality of StyleGAN, Karras et al, 2019
  • Autoencoding beyond pixels using a learned similarity metric, Larsen et al, CVPR 2016
  • Generating Images with Perceptual Similarity Metrics based on Deep Networks
  • Glow: Generative Flow with Invertible 1x1 Convolutions, Kingma et al., ICLR 2018
  • Generating Diverse High-Fidelity Images with VQ-VAE-2, Razavi et al., 2019
• 

  03/08/2021 Conditional GANs, Image-to-Image Translation (part 1) [pptx] [pdf]
  

  Reading List:

  • Image-to-Image Translation with Conditional Adversarial Networks, Isola et al, CVPR 2017
  • Photographic Image Synthesis with Cascaded Refinement Networks, Chen et al, ICCV 2017
  • High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs, Wang et al, CVPR 2018
  • Semantic Image Synthesis with Spatially-Adaptive Normalization, Park et al, CVPR 2019
  • Semi-parametric Image Synthesis, Qi et al, CVPR 2018
  • Shapes and Context: In-the-Wild Image Synthesis & Manipulation, Bansal et al, CVPR 2019
• 

  03/10/2021 Conditional GANs, Image-to-Image Translation (Part 2) [pptx] [pdf]
  

  Reading List:

  • Toward Multimodal Image-to-Image Translation, Zhu et al, Neurips 2017
  • Multi-Agent Diverse Generative Adversarial Networks, Ghosh et al, CVPR 2018
  • Diversity-Sensitive Conditional Generative Adversarial Networks, Yang et al, ICLR 2019
  • Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks, Zhu et al, ICCV 2017
  • Learning from Simulated and Unsupervised Images through Adversarial Training, Shrivastava et al, CVPR 2017
  • Unsupervised Cross-Domain Image Generation, Taigman et al, ICLR 2017
  • CyCADA: Cycle-Consistent Adversarial Domain Adaptation, Hoffman et al, ICML 2018
• 03/15/2021 Conditional Image Generation (student presentation)
  tl;dr: Cool recent papers in conditional image generation
  

  Reading List:

  • Few-Shot Unsupervised Image-to-Image Translation, Liu et al, ICCV 2019, Presentation by Jiaheng Hu
  • Diverse Image-to-Image Translation via Disentangled Representations, Lee at al, ECCV 2018, Presentation by Sanil Pande
  • Zero-Shot Text-to-Image Generation, Ramesh et al, Presentation by Jun Luo
  • Semi-parametric Image Synthesis, Qi et al, CVPR 2018
  • StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation, Choi et al, CVPR 2018
  • StarGAN v2: Diverse Image Synthesis for Multiple Domains, Choi et al, CVPR 2020
  • The Perception-Distortion Tradeoff, Blau et al, CVPR 2018
• 

  03/17/2021 Style and Content, Texture Synthesis (part 1)
  tl;dr: How to control the style and content of your image with Deep Learning [pptx] [pdf]
  

  Reading List:

  • Separating Style and Content, Tenenbaum & Freeman, Neurips 1996
  • Unsupervised Image-to-Image Translation Networks, Liu et al, Neurips 2017
  • Multimodal Unsupervised Image-to-Image Translation, Huang et al, ECCV 2018
  • Representation Learning with Contrastive Predictive Coding, van den Oord et al
  • A Simple Framework for Contrastive Learning of Visual Representations, Chen et al, ICML 2020
  • Contrastive Learning for Unpaired Image-to-Image Translation, Park et al, ECCV 2020
• 03/22/2021 Style and Content, Texture Synthesis (part 2)
  tl;dr: How to control the style and content of your image with Deep Learning, part 2 [pdf] [pptx]
  

  Reading List:

  • Separating Style and Content, Tenenbaum & Freeman, Neurips 1996
  • Image Analogies, Hertzmann et al, SIGGRAPH 2001
  • PatchMatch: A Randomized Correspondence Algorithm for Structural Image Editing, Barnes et al, SIGGRAPH 2009
  • Visual Attribute Transfer through Deep Image Analogy, Liao et al, SIGGRAPH 2017
  • Deep Photo Style Transfer, Luan et al, CVPR 2017
  • Non-Stationary Texture Synthesis by Adversarial Expansion, Zhou et al, SIGGRAPH 2018
• 03/24/2021 Texture Synthesis
  tl;dr: Further techniques and results in Texture Synthesis and Style Transfer [pptx] [pdf]
  

  Required Reading List:

  • Visual Attribute Transfer through Deep Image Analogy, Liao et al, SIGGRAPH 2017
  • Deep Photo Style Transfer, Luan et al, CVPR 2017
  • Non-Stationary Texture Synthesis by Adversarial Expansion, Zhou et al, SIGGRAPH 2018 Optional Reading List:
  • Separating Style and Content, Tenenbaum et al, Neurips 1996
  • Image Analogies, Hertzmann et al, SIGGRAPH 2001
  • PatchMatch: A Randomized Correspondence Algorithm for Structural Image Editing, Barnes et al, SIGGRAPH 2009
• 

  03/29/2021 Image Editing and Optimization
  tl;dr: Use an optimization algorithm in a learned space to achieve photo manipulation [pptx] [pdf]
  

  Required Reading List:

  • Generative Visual Manipulation on the Natural Image Manifold, Zhu et al, ECCV 2016
  • Neural Photo Editing with Introspective Adversarial Networks, Brock et al, ICLR 2017
• 

  04/05/2021 Image Editing and Optimization (part 2)
  tl;dr: Use an optimization algorithm in a learned space to achieve photo manipulation [pptx] [pdf]
  

  Required Reading List:

  • Semantic Photo Manipulation with a Generative Image Prior, Bau et al, SIGGRAPH 2019
  • Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?, Abdal et al, ICCV 2019
  • StyleGAN2: Analyzing and Improving the Image Quality of StyleGAN., Karras et al, CVPR 2020
  • GAN Dissection: Visualizing and Understanding Generative Adversarial Networks, Bau et al, ICLR 2019
  • GANSpace: Discovering Interpretable GAN Controls, Härkönen et al, NeurIPS 2020
  • On the “steerability” of generative adversarial networks, Jahanian et al, ICLR 2020
  • Editing in Style: Uncovering the Local Semantics of GANs., Collins et al, CVPR 2020
  • Closed-Form Factorization of Latent Semantics in GANs., Shen et al, CVPR 2021
• 04/07/2021 Image Editing and Optimization (student presentations)
  tl;dr: Use an optimization algorithm in a learned space to achieve photo manipulation
  

  Required Reading List:

  • Deep Image Prior, Ulyanov et al, CVPR 2018
  • GANSpace: Discovering Interpretable GAN Controls, Härkönen et al, NeurIPS 2020 Optional Reading List:
  • Colorization using Optimization, Levin et al, SIGGRAPH 2004
  • AppProp: all-pairs appearance-space edit propagation, An et al, SIGGRAPH 2008
  • Image Smoothing via L0 Gradient Minimization, Xu et al, SIGGRAPH Asia 2011
  • On the “steerability” of generative adversarial networks, Jahanian et al, ICLR 202
• 

  04/12/2021 Face Modeling (part 1)
  tl;dr: What special techniques can we use to generate and manipulate realistic faces? [pptx] [pdf]
  

  Reading List:

  • Active Appearance Models, Cootes et al, ECCV 1998
  • Composite Portraits, Galton, Nature 1878
  • Manipulating Facial Appearance through Shape and Color, Rowland & Perret, CG&A, 1995
  • A Morphable Model For The Synthesis Of 3D Faces, Blanz & Vetter, SIGGRAPH 1999
  • Eigenfaces for Recognition, Turk and Pentland, Journal of cognitive neuroscience, 1991
  • Exploring Photobios, Kemelmacher-Shlizerman et al., SIGGRAPH 2011
• 

  04/14/2021 Face Modeling (part 2)
  tl;dr: What special techniques can we use to generate and manipulate realistic faces? [pptx] [pdf]
  

  Reading List:

  • Reducing the dimensionality of data with neural networks, Hinton & Salakhutdinov, Science 2006
  • Deep Video Portraits, Kim et al, SIGGRAPH 2018
  • StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation, Choi et al, CVPR 2018
  • StarGAN v2: Diverse Image Synthesis for Multiple Domains, Choi et al, CVPR 2020
  • GANimation: Anatomically-aware Facial Animation from a Single Image, Pumarola et al, ECCV 2018
  • Image2StyleGAN++: How to Edit the Embedded Images?, Abdal et al, CVPR 2020
  • StyleRig: Rigging StyleGAN for 3D Control over Portrait Images, Tewari et al, CVPR 2020
• 

  04/19/2021 Face Modeling (student presentations)
  tl;dr: What special techniques can we use to generate and manipulate realistic faces?
  

  Reading List:

  • Face2Face: Real-time Face Capture and Reenactment of RGB Videos, Theis et al, CVPR 2016
  • SfSNet: Learning Shape, Reflectance and Illuminance of Faces ‘in the wild’, Sengputa et al, CVPR 2018
  • paGAN: Real-time Avatars Using Dynamic Textures, Nagano et al, SIGGRAPH Asia 2018
  • Single Image Portrait Relighting, Sun et al, SIGGRAPH 2019
  • Intuitive, Interactive Beard and Hair Synthesis with Generative Models, Olzewski et al., CVPR 2020
  • Time Travel Rephotography, Luo et al, arXiV 2020
• 

  04/21/2021 Novel View Synthesis
  tl;dr: Techniques for changing your (camera's) point of view [pptx] [pdf]
  

  Reading List:

  • QuickTime VR – An Image-Based Approach to Virtual Environment Navigation, Chen, SIGGRAPH 1995
  • Stanford Lightfield Website, 2004-2014
  • The Lumigraph, Gortler, SIGGRAPH 1996
  • Light field photography with a hand-held plenoptic camera, Ng, SIGGRAPH 2005
  • DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation, Park et al, CVPR 2019
  • NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis, Mildenhall et al, ECCV 2020
• 

  04/26/2021 Video Editing and Synthesis (part 1)
  tl;dr: How can we extend our techniques to the video setting? [pptx] [pdf]
  

  Reading List:

  • Video Textures, Sch¨odl et al, SIGGRAPH 2000
  • Controlled Animation of Video Sprites, Schodl et al, UIST 2008
  • Generating Videos with Scene Dynamics, Vondrick et al, NeurIPS 2016
  • Video-to-Video Synthesis, Vondrick et al, NeurIPS 2016
  • Few-shot Video-to-Video Synthesis, Wang et al, NeurIPS 2019
• 

  05/03/2021 Video Editing and Synthesis (Student Presentations)
  tl;dr: Presentations on exciting work in the video space!
  

  Reading List:

  • Interactive Video Cutout, Wang et al, SIGGRAPH 2005
  • Video Object Annotation, Navigation, and Composition, Goldman et al, UIST 2008
  • Recycle-GAN: Unsupervised Video Retargeting, Bansal et al, ECCV 2018
  • Everybody Dance Now, Chan et al, ICCV 2019
  • Adversarial Video Generation on Complex Datasets, Clark et al, arXiv 2019
  • Text-based Editing of Talking-head Video, Fried at al, SIGGRAPH 2019
  • Background Matting: The World is Your Green Screen, Sengupta et al, CVPR 2020
  • World-Consistent Video-to-Video Synthesis, Mallya et al, ECCV 2020