15-387/86-375/675 Computational Perception

Carnegie Mellon University

Fall 2020

Course Description

The perceptual capabilities of even the simplest biological organisms are far beyond what we can achieve with machines. Whether you look at sensitivity, robustness, adaptability and generalizability, perception in biology just works, and works in complex, ever changing environments, and can make inference on the most subtle sensory patterns. Is it the neural hardware? Does the brain use a fundamentally different algorithm? What can we learn from biological systems and human perception?

In this course, we will study the biological and psychological data of biological perceptual systems, mostly the visual system, in depth, and then apply computational thinking to investigate the principles and mechanisms underlying natural perception. You will learn how to reason scientifically and computationally about problems and issues in perception, how to extract the essential computational properties of those abstract ideas, and finally how to convert these into explicit mathematical models and computational algorithms. The course is targeted to both neuroscience and psychology students who are interested in learning computational thinking, as well as computer science and engineering students who are interested in learning more about the neural and computational basis of perception. Prerequisites: First year college calculus, differential equations, linear algebra, basic probability theory and statistical inference, and programming experience (Matlab) are desirable.

Course Information

Instructors Office (Office hours) Email (Phone)
Tai Sing Lee (Professor) zoom: tai@cnbc.cmu.edu (412-268-1060)
Andrew Luo (TA) zoom: afluo@andrew.cmu.edu

Recommended Textbook

Classroom Etiquette

Grading Scheme 15-387 or 86-375

Evaluation% of Grade
Assignments 65
Midterm 10
Final Exam 15
Class Attendance and Participation 10
Term Paper or project (optional for 387) replacement grade for one homework or mid-term

Grading Scheme 86-675

Evaluation% of Grade
Assignments 65
Journal Club 30
Term Project 30
Exams (25) + class participation 30

Grading Scheme 86-375

Evaluation% of Grade
Assignments 40/65
Journal Club 30
Term Project/Paper 30
Exams (25) + class participation (5) 30

Assignments

Term Project

Journal Club

Examinations

Late Policy

Syllabus

Date Lecture Topic Relevant Readings Assignments
  SENSORY CODING    
M 8/31 1. Introduction ch. 1, Marr  
W 9/2 2. Computational Approach ch 1 Marr, ch1, FS  
M 9/7 Label Day (no class)    
W 9/9 3. Retina FS ch 6 and ch 3, Gollisch and Meister Homework 1
M 9/14 4. Pyramid Burt and Adelson
W 9/16 5. Frequency Analysis FS Ch 4 and ch 5  
M 9/21 6. Representation FS ch 9, Shlens  
W 9/23 7. Source separation Fodiak, Olshausen Homework 2
  PERCEPTUAL INFERENCE    
M 9/28 8. Lightness and color ch 16, Land, Horn, Morel  
W 9/30 9. Intrinsic images and Retinex ch 17, Adelson, Weiss, Freeman  
M 10/5 10. Perceptual Systems ch 10 (brain maps), Van Essen  
W 10/7 11. Multi-sensory Integration ch 20 Homework 3;
M 10/12 12. Bayesian inference ch 13 (inference)  
W 10/14 Midterm    
M 10/19 13. Perceptual Organization ch 7 Project Proposal due
W 10/21 14. Features and Texture ch 2, Julesz, Simoncelli  
M 10/26 15. Texture Perception    
W 10/28 16. Depth and Stereo ch 18,19 Homework 4
M 11/2 17. Shape from Shading Horn, Zucker  
W 11/4 18. Motion Perception ch 14,15, Weiss  
  OBJECT AND SCENES    
M 11/9 19. Figure-Ground Perception ch 7  
W 11/11 20. Scene Analysis Torrelba, Oliva Homework 5
M 11/16 21. Objectd recognition ch 8 (objects), Sinha, LeCun, Hinton  
W 11/18 23. Analysis by Synthesis Mumford, Hinton  
M 11/23 22. Generative Models Active shape and appearance  
W 11/25 Thanksgiving break    
M 11/30 24. Relationshps and Composition ch 11. Yuille, Zhu and Mumford HW 5 due
W 12/2 25. Attention and routing ch 22, Hinton, Arathon, Olshausen  
M 12/7 26. Perception and Art    
W 12/9 Project Presentations   Project and Term Paper Due
X 12/X Final Exam and Presentations    

Journal Club

Week 1 September 11 Retina

Week 2 September 18. Retina Computation

Week 3 September 26. Scene Statistics and Perception

Week 4 October 2. Lightness perception

Supplementary Readings

Vision, Perceptual Systems and Philosophy

Computations in the Retina

Neural Codes, Features and Representational Learning

Lightness and color perception, Retinex

Mid-level vision: Texture, depth and motion Perception

Visual Hierarchy, Object recognition, Abstract Representations

Feedback, Synthesis and Predictive Coding

Generative models, Art, Abstract Representations

Belief, memories and Association

Composition and Grammar

Attention, Eye Movement and Routing

Emotion and Action on perception


Questions or comments: contact Tai Sing Lee
Last modified: August 2020, Tai Sing Lee