Computational Creativity

ECSE 4964/6964, Rensselaer Polytechnic Institute, Fall 2023

Instructor: Rich Radke

Time: Mondays & Thursdays 4:00 pm - 5:20 pm

Location: JEC 4104

Course Description: Creative applications of generative artificial intelligence have exploded in the past year, including image generation tools like Stable Diffusion and DALLE-2 and text generation tools like Chat-GPT3. This course will survey the theoretical foundations of these tools, focusing on generative models and self-supervised learning, as well as explore the historical and ethical considerations involving the procedural generation of art. Students will apply cutting-edge tools for generating creative content and critique each other’s work.

Prerequisites: A first course in machine learning (ECSE-4840 or ECSE-4850 or CSCI-4100 or equivalent). Having taken such a course implies an understanding of calculus, probability, and programming (e.g., the content in MATH-2010, ECSE-2500, and CSCI-1200). Additionally, a creative/artistic mindset and curiosity/enthusiasm for making fun images and videos is required!

Textbook: We will cover most of Generative Deep Learning: Teaching Machines To Paint, Write, Compose, and Play, 2nd Edition by David Foster, O’Reilly Media, June 2023, as well as many research papers that go beyond this text.

Topics: The topics we’ll discuss over the course of the semester include:

  • Introduction and historical overview
  • Algorithms for making art (1960-2010)
  • Variational auto-encoders (VAEs)
  • VQ-VAEs and image quality metrics
  • Generative adversarial networks (GANs)
  • Advanced GANs (e.g., ProGAN, StyleGAN, GANSpace)
  • Image-to-Image GANs (e.g., CycleGAN, pix2pix); GAN artists
  • Normalizing flow models
  • Denoising diffusion models
  • Neural style transfer; Deepdream
  • Neural language models and word embeddings
  • Word2vec and attention
  • Large language models and their implications
  • CLIP and its applications
  • DALLE-2 and Stable Diffusion
  • Diffusion developments (inpainting, DreamBooth, ControlNet)
  • Text to 3D (Dreamfusion, Point-E)
  • Music generation (MuseGAN, Magenta, Jukebox, Riffusion, Meta)
  • Graphic design layout generation and other extensions
  • Neural rendering fields (NeRF)

We will also feature several guest speakers from the intersections of art, law, and technology. We will also interleave discussions about the ethics of generative AI techniques throughout the technical material.

Grading and Policies

See more information at the Policies page (still tentative until classes begin).

Staff Contact: The best way to reach the staff is by posting to the class Discord, or mailing the professor directly at