86-375/675 (15-387) Computational Perception

Carnegie Mellon University

Fall 2019

Course Description

The perceptual capabilities of even the simplest biological organisms are far beyond what we can achieve with machines. Whether you look at sensitivity, robustness, or sheer perceptual power, perception in biology just works, and works in complex, ever changing environments, and can pick up the most subtle sensory patterns. Is it the neural hardware? Does biology solve fundamentally different problems? What can we learn from biological systems and human perception?

In this course, we will first study the biological and psychological data of biological perceptual systems, particularly the visual system, in depth, and then apply computational thinking to investigate the principles and mechanisms underlying natural perception. The course will focus primarily on visual perception. You will learn how to reason scientifically and computationally about problems and issues in perception, how to extract the essential computational properties of those abstract ideas, and finally how to convert these into explicit mathematical models and computational algorithms. The course is targeted to neuroscience and psychology students who are interested in learning computational thinking, and computer science and engineering students who are interested in learning more about the neural basis of perception. Prerequisites: First year college calculus, some linear algebra, probability theory and programming experience are desirable.

Course Information

Instructors Office (Office hours) Email (Phone)
Tai Sing Lee (Professor) Mellon Inst. Rm 115 (Friday 11:30-12:30) tai@cnbc.cmu.edu (412-268-1060)
Mert Inan Mellon Inst. Rm 116 (Tuesdsy 4:30-5:30 p.m) merti@andrew.cmu.edu
Jiaxin Wang Mellon Institute Room 116 (Monday 4:30-5:30) jiaxinwa@andrew.cmu.edu
Either TA GHC (Sat 4:00-5:00) Jiaxin or Mert

Recommended Textbook

Classroom Etiquette

Grading Scheme 15-387

Evaluation% of Grade
Assignments 65
Midterm 10
Final Exam 15
Class Attendance and Participation 10
Term Paper (optional) replacement grade for one homework or midterm
  • Total points: 100
  • Grading scheme: A: > 88%, B: > 75%. C: > 65%.

    Grading Scheme 86-675

    Evaluation% of Grade
    Assignments 65
    Journal Club 30
    Term Project 30
    Midterm 10
    Final Exam 15
    Class Attendance and Participation 10
  • Total credit to count: 130.
  • Option 1: Term project + Journal Club + 35 points from homework + exams + attendance
  • Option 2: Term project + 65 points homework + exams + attenance
  • Option 3: Journal Club + 65 points homework + exams + attenance
  • Grading scheme: A: > 88%, B: > 75%. C: > 65%.

    Grading Scheme 86-375

    Evaluation% of Grade
    Assignments 65
    Participation 10
    Midterm 10
    Final Exam 15
    Class Attendance and Participation 10
  • Total credit to count: 100.
  • Option 1: 65 points homework + exams + attenance (just like 15-387)
  • Option 2: Journal Club + 35 points from homework + exams + attendance
  • Option 3: Term project + 35 points from homework + exams + attendance
  • Grading scheme: A: > 88%, B: > 75%. C: > 65%.

    Assignments

    Term Project

    Examinations

    Late Policy

    Syllabus

    Date Lecture Topic Relevant Readings Assignments
      SENSORY CODING    
    M 8/26 1. Introduction ch. 1, Marr  
    W 8/28 2. Computational Approach ch 1 Marr, ch1, FS  
    M 9/2 Label Day (no class)    
    W 9/4 3. Retina FS ch 6 and ch 3, Gollisch and Meister Homework 1
    M 9/9 4. Pyramid Burt and Adelson
    W 9/11 5. Frequency Analysis FS Ch 4 and ch 5  
    M 9/16 6. Representation FS ch 9, Shlens  
    W 9/18 7. Source separation Fodiak, Olshausen Homework 2
      PERCEPTUAL INFERENCE    
    M 9/23 8. Lightness and color ch 16, Land, Horn, Morel  
    W 9/25 9. Intrinsic images and Retinex ch 17, Adelson, Weiss, Freeman  
    M 9/30 10. Perceptual Systems ch 10 (brain maps), Van Essen  
    W 10/2 11. Multi-sensory Integration ch 20 Homework 3;
    M 10/7 12. Bayesian inference ch 13 (inference)  
    W 10/9 Midterm    
    M 10/14 13. Perceptual Organization ch 7 Project Proposal due
    W 10/16 14. Features and Texture ch 2, Julesz, Simoncelli  
    M 10/21 15. Texture Perception    
    W 10/23 16. Depth and Stereo ch 18,19 Homework 4
    M 10/28 17. Shape from Shading Horn, Zucker  
    W 10/30 18. Motion Perception ch 14,15, Weiss  
      OBJECT AND SCENES    
    M 11/4 19. Figure-Ground Perception ch 7  
    W 11/6 20. Scene Analysis Torrelba, Oliva Homework 5
    M 11/11 21. Objectd recognition ch 8 (objects), Sinha, LeCun, Hinton  
    W 11/13 23. Analysis by Synthesis Mumford, Hinton  
    M 11/18 22. Objective Modeling Active shape and appearance  
    W 11/20 24. Relationshps and Composition ch 11. Yuille, Zhu and Mumford HW 5 due
    M 11/25 25. Attention and routing ch 22, Hinton, Arathon, Olshausen  
    W 11/27 Thanksgiving break    
    M 12/2 26. Perception and Art    
    W 12/4 Project Presentations   Project and Term Paper Due
    X 12/X Final Exam and Presentations    

    Supplementary Readings (Relevant to Understanding Lectures)

    Part 1: Vision, Perceptual Systems and Philosophy

    Part 2: Neural Codes, Features and Representational Learning

    Part 3: Lightness and color perception, Retinex

    Part 4: Mid-level vision: Texture, depth and motion Perception

    Part 5: Visual Hierarchy, Object recognition, Abstract Representations

    Part 6: Generative models, Art, Abstract Representations

    Part 7: Belief, memories and Association

    Part 8: Composition and Grammar

    Part 9: Attention, Eye Movement and Routing

    Journal Club Reading (September 13) Computations in the Retina

    Journal Club Reading (September 20) Neural Codes: Sparse versus Redundant Codes

    Journal Club Reading (September 27) Recurrent Network and Neural Control

    Journal Club Reading (October 4) Predictive Coding


    Questions or comments: contact Tai Sing Lee
    Last modified: Jan 2018, Tai Sing Lee