15-387/86-375/675 Computational Perception

Carnegie Mellon University

Fall 2020

Course Description

The perceptual capabilities of even the simplest biological organisms are far beyond what we can achieve with machines. Whether you look at sensitivity, robustness, adaptability and generalizability, perception in biology just works, and works in complex, ever changing environments, and can make inference on the most subtle sensory patterns. Is it the neural hardware? Does the brain use a fundamentally different algorithm? What can we learn from biological systems and human perception?

In this course, we will study the biological and psychological data of biological perceptual systems, mostly the visual system, in depth, and then apply computational thinking to investigate the principles and mechanisms underlying natural perception. You will learn how to reason scientifically and computationally about problems and issues in perception, how to extract the essential computational properties of those abstract ideas, and finally how to convert these into explicit mathematical models and computational algorithms. The course is targeted to both neuroscience and psychology students who are interested in learning computational thinking, as well as computer science and engineering students who are interested in learning more about the neural and computational basis of perception. Prerequisites: First year college calculus, differential equations, linear algebra, basic probability theory and statistical inference, and programming experience (Matlab) are desirable.

Course Information

Instructors Office (Office hours) Email (Phone)
Tai Sing Lee (Professor) zoom: tai@cnbc.cmu.edu (412-268-1060)
Andrew Luo (TA) zoom: afluo@andrew.cmu.edu

Recommended Textbook

Classroom Etiquette

Grading Scheme 15-387 or 86-375

Evaluation% of Grade
Assignments 65
Midterm 10
Final Exam 15
Class Attendance and Participation 10
Term Paper or project (optional for 387) replacement grade for one homework or mid-term

Grading Scheme 86-675

Evaluation% of Grade
Assignments 65
Journal Club 30
Term Project 30
Exams (25) + class participation 30

Grading Scheme 86-375

Evaluation% of Grade
Assignments 40/65
Journal Club 30
Term Project/Paper 30
Exams (25) + class participation (5) 30


Term Project

Journal Club


Late Policy


Date Lecture Topic Relevant Readings Assignments
M 8/31 1. Introduction ch. 1, Marr  
W 9/2 2. Computational Approach ch 1 Marr, ch1, FS  
M 9/7 Label Day (no class)    
W 9/9 3. Retina FS ch 6 and ch 3, Gollisch and Meister Homework 1
M 9/14 4. Pyramid Burt and Adelson
W 9/16 5. Frequency Analysis FS Ch 4 and ch 5  
M 9/21 6. Representation FS ch 9, Shlens  
W 9/23 7. Source separation Fodiak, Olshausen Homework 2
M 9/28 8. Lightness and color ch 16, Land, Horn, Morel  
W 9/30 9. Intrinsic images and Retinex ch 17, Adelson, Weiss, Freeman  
M 10/5 10. Perceptual Systems ch 10 (brain maps), Van Essen  
W 10/7 11. Multi-sensory Integration ch 20 Homework 3;
M 10/12 12. Bayesian inference ch 13 (inference)  
W 10/14 Midterm    
M 10/19 13. Perceptual Organization ch 7 Project Proposal due
W 10/21 14. Features and Contours Geisler, Elders  
M 10/26 15. Texture Perception Julesz  
W 10/28 16. Texture Metamer Simoncelli Homework 4
M 11/2 17. Source Separation Bells and Sejnowski, Olshausen and Field  
W 11/4 18. Depth and Stereo Poggio  
M 11/9 19. Figure-Ground Perception ch 7  
W 11/11 20. Hierarchy Fukushima, Van Essen Homework 5
M 11/16 21. Deep Network LeCun, Hinton  
W 11/18 22. Analysis by Synthesis Tenenbaum  
M 11/23 23. Recurrent Feedback Lee and Mumford, Hinton  
W 11/25 Thanksgiving break    
M 11/30 24. Predictive Coding Rao and Ballard, Lotter and Cox HW 5 due
W 12/2 25. Motion Perception Weiss, Adelson  
M 12/7 26. Scene Analysis Oliva and Torralba  
W 12/9 27. Object and face perception Tenenbaum, Tsao and Freiwald Project and Term Paper Due
X 12/18 Final Exam and Presentations    

Journal Club

Week 1 September 11 Retina

Week 2 September 18. Retina Computation

Week 3 September 26. Scene Statistics and Perception

Week 4 October 2. Lightness Computation

Week 5 October 9. Retinex and Perception

Week 5 October 16. Contour Processing

Week 6 October 30. Art from the Eyes of Perception Science

Week 7 Nov 6. From texture synthesis to artistic style transfer

Week 8 Nov 13. Sensitivity of naturalistic texture in visual cortex

Week 9 Nov 20. Attention in the Brain and AI

Week 10 Dec 11. Affordance and Neural Scene Representation

Supplementary Readings

Vision, Perceptual Systems and Philosophy

Computations in the Retina

Neural Codes, Features and Representational Learning

Lightness and color perception, Retinex

Mid-level vision: Texture, depth and motion Perception

Visual Hierarchy, Object recognition, Abstract Representations

Feedback, Synthesis and Predictive Coding

Generative models, Art, Abstract Representations

Belief, memories and Association

Analysis by Synthesis

Composition and Grammar

Attention, Eye Movement and Routing

Emotion and Action on perception

Questions or comments: contact Tai Sing Lee
Last modified: August 2020, Tai Sing Lee