86-375/675 (15-387) Computational Perception

Carnegie Mellon University

Fall 2019

Course Description

The perceptual capabilities of even the simplest biological organisms are far beyond what we can achieve with machines. Whether you look at sensitivity, robustness, or sheer perceptual power, perception in biology just works, and works in complex, ever changing environments, and can pick up the most subtle sensory patterns. Is it the neural hardware? Does biology solve fundamentally different problems? What can we learn from biological systems and human perception?

In this course, we will first study the biological and psychological data of biological perceptual systems, particularly the visual system, in depth, and then apply computational thinking to investigate the principles and mechanisms underlying natural perception. The course will focus primarily on visual perception. You will learn how to reason scientifically and computationally about problems and issues in perception, how to extract the essential computational properties of those abstract ideas, and finally how to convert these into explicit mathematical models and computational algorithms. The course is targeted to neuroscience and psychology students who are interested in learning computational thinking, and computer science and engineering students who are interested in learning more about the neural basis of perception. Prerequisites: First year college calculus, some linear algebra, probability theory and programming experience are desirable.

Course Information

Instructors Office (Office hours) Email (Phone)
Tai Sing Lee (Professor) Mellon Inst. Rm 115 (Friday 1:30-2:30) tai@cnbc.cmu.edu (412-268-1060)
Mert Inan Mellon Inst. Rm 116 (Tuesdsy 4:30-5:30 p.m) merti@andrew.cmu.edu
Jiaxin Wang Mellon Institute Room 116 (Monday 4:30-5:30) jiaxinwa@andrew.cmu.edu

Recommended Textbook

Classroom Etiquette

Grading Scheme

Evaluation% of Grade
Assignments 65
Midterm 10
Final Exam 15
Class Attendance and Participation 10
Term Paper (optional) 10 (replacement grade)
675 Term Project 30
  • Grading scheme: A: > 88, B: > 75. C: > 65.


    Term Project


    Late Policy


    Date Lecture Topic Relevant Readings Assignments
    M 8/26 1. Introduction ch. 1, Marr  
    W 8/28 2. Computational Approach ch 1 Marr, ch1, FS  
    M 9/2 Label Day (no class)    
    W 9/4 3. Retina FS ch 6 and ch 3, Gollisch and Meister Homework 1
    M 9/9 4. Pyramid Burt and Adelson
    W 9/11 5. Frequency Analysis FS Ch 4 and ch 5  
    M 9/16 6. Representation FS ch 9, Shlens  
    W 9/18 7. Source separation Fodiak, Olshausen Homework 2
    M 9/23 8. Lightness and color ch 16, Land, Horn, Morel  
    W 9/25 9. Intrinsic images and Retinex ch 17, Adelson, Weiss, Freeman  
    M 9/30 10. Perceptual Systems ch 10 (brain maps), Van Essen  
    W 10/2 11. Multi-sensory Integration ch 20 Homework 3;
    M 10/7 12. Bayesian inference ch 13 (inference)  
    W 10/9 Midterm    
    M 10/14 13. Perceptual Organization ch 7 Project Proposal due
    W 10/16 14. Features and Texture ch 2, Julesz, Simoncelli  
    M 10/21 15. Texture Perception    
    W 10/23 16. Depth and Stereo ch 18,19 Homework 4
    M 10/28 17. Shape from Shading Horn, Zucker  
    W 10/30 18. Motion Perception ch 14,15, Weiss  
    M 11/4 19. Figure-Ground Perception ch 7  
    W 11/6 20. Scene Analysis Torrelba, Oliva  
    M 11/11 21. Objectd recognition ch 8 (objects), Sinha, LeCun, Hinton  
    W 11/13 23. Analysis by Synthesis Mumford, Hinton Homework 5
    M 11/18 22. Objective Modeling Active shape and appearance  
    W 11/20 24. Relationshps and Composition ch 11. Yuille, Zhu and Mumford  
    M 11/25 25. Attention and routing ch 22, Hinton, Arathon, Olshausen  
    W 11/27 Thanksgiving break    
    M 12/2 26. Perception and Art    
    W 12/4 Project Presentations   Homework 5 due, Project and Term Paper Due
    X 12/X Final Exam and Presentations    

    Supplementary Readings (Relevant to Understanding Lectures)

    Part 1: Vision, Perceptual Systems and Philosophy

    Part 2: Neural Codes, Features and Representational Learning

    Part 3: Lightness and color perception, Retinex

    Part 4: Mid-level vision: Texture, depth and motion Perception

    Part 5: Visual Hierarchy, Object recognition, Abstract Representations

    Part 6: Generative models, Art, Abstract Representations

    Part 7: Belief, memories and Association

    Part 8: Composition and Grammar

    Part 9: Attention, Eye Movement and Routing

    Journal Club Reading (September 13) Choices: Retina

    Questions or comments: contact Tai Sing Lee
    Last modified: Jan 2018, Tai Sing Lee