This year, we will use the case-study approach that emphasizes on "active learning by doing", with strong focus on research. Graduate students will each or collaboratively engage in a research project, potentially forming a team with an undergraduate assistant. It is understood that the undergraduate student will be less prepared in math and coding, and it is my hope that the graduate students will provide help and advice to their partner in performing the homework exercises.
Some Matlab or programming background are required for 15-387 and 86-675. Or you have to learn it over time during the course.
Prerequisites: First year college calculus, some linear algebra, probability theory and programming experience are desirable. Discuss with instructor if you have any question.
| Instructors | Office (Office hours) | Email (Phone) |
|---|---|---|
| Tai Sing Lee (Professor) | Mellon Inst. Rm 115 (office hour TBA) and by appt | tai@cnbc.cmu.edu (412-268-1060) |
| Evaluation | % of Grade |
|---|---|
| Paper Presentation (2-3 times) | 30 |
| Midterm | 10 |
| Term Project | 50 |
| Course assistance | 10 |
| Weekly or biweekly resaerch meeting | your bonus |
| Evaluation | % of Grade |
|---|---|
| Homework (3-4) | 50 |
| Research Paper or Project Presentation | 20 |
| Midterm | 10 |
| Final Exam | 20 |
| Date | Lecture Topic | Relevant Readings | Assignments |
|---|---|---|---|
| Physiological and Computational Foundation | |||
| M 8/31 | Organization and Objectives | FS. Ch 1, Marr Ch 1 | |
| W 9/2 | Retina tunings | Marr ch1,2, Burt and Adelson | |
| M 9/7 | Labor day holiday | FS. Ch 1, Marr, Ch 1 | |
| W 9/9 | Laplacian pyramid, Research | FS Ch 6, Marcus Meister | |
| M 9/14 | Neuronal encoding and decoding | Signals and System Textbook | |
| W 9/16 | Paper 1: Retinal Codes | Meister, Nirenberg | |
| M 9/21 | Bayesian inference and belief propagation | Isard, Kelly, Yair | |
| W 9/23 | Paper 2: How we see the world with jittering eyes and spiking neurons? | Burak et al. PNAS 2010 | |
| M 9/28 | Visual Cortex and V1 models | FS ch 9,10 | |
| W 9/30 | Paper 3: Gabor, LN models, image representation | Daugman, Lee, Yan | |
| M 10/5 | Sparse and efficient codes | Olshausen and Lewicki | |
| W 10/7 | Paper 4: Variety of sparse code models | Olshausen, Sommer, Locke | |
| M 10/12 | Deep learning models | ||
| W 10/14 | Paper 5: CNN and compositional learning | Alex net, Ruslan Sakahutdinov | |
| M 10/19 | Midterm | SFN | |
| W 10/21 | Paper 6: Lightness and illusion | Adelson | |
| F 10/23 | Mid-Semester Break | ||
| Discriminative vs Generative models | |||
| M 10/26 | Recovering intrinisc images. | Land, Freeman, Weiss, Adelson | Midterm grade due | W 10/28 | Perceptual organization (grouping, segmentation, symmetry and texture) | FS ch 8, 13, 2 |
| M 11/2 | 3D perception (stereo, shading, texture) | FS Ch 18, 19 | |
| W 11/4 | Paper 7: Bayesian models of perception | Yuille, Bank, Geisler | M 11/9 | Space, time and Motion | FS ch 14,15 | < tr> | W 11/11 | Paper 8: predictive coding | Ballard, Lee, Friston, Bar | < tr> | M 11/16 | Art and Perception | < tr> | W 11/18 | Paper 9: Beauty, realism, forgery | Cavanagh, Zaki, Farid | < tr> | M 11/23 | Scene parsing, context and attention | Desimone, Torralba | < tr> | W 11/25 | Thanksgiving Holiday | < tr> | M 11/30 | Paper 10: Scene parsing, and stochastic grammar | Zhu, Yuille | < tr> | W 12/2 | Research Project | < tr> | M 12/7 | Research Project | < tr> | W 12/9 | Research Project | < tr> | F 12/11 | Last day of class |
| M 12/14 | FINAL EXAM WEEK | ||
| X X/XX | Project Presentation | ||
| W 12/23 | FINAL Grade due |
The following are some of the unifying themes of our investigation.
Theme 1: Scene statistics, sensory and cortical representationTo undersatnd perception, we must understand the natural environments which shape our brain and our perceptual computational machinery. Central to to understanding the neural basis of perceptual inference from a Bayesian perspective is understanding how the statistical regularities in natural scenes are encoded in cortical representation to serve as priors in the inference process. Natural images however are enormously complex and maybe best expressed in hierarchical forms. Thus, a major challenge in computational vision is to understand the basic vocabulary of images, and the computational rules with which elementary components can be composed to form successive compositional structures to encode the hierarchical priors of natural scenes. We will explore statistical models of images, as well as compositional models such as DBN (Deep belief net) and RCM (Recursive compositional models) for learning the hierarchical language of vision. We will explore how these hierarchical scene priors are encoded in neural tunings and neural connectivities to faciliate perceptual inference.
Theme 2: Probabilistic models and algorithms of perception
While perception has been popularly formulated in terms of Bayesian inference at the theoretical level, little is known about the computational algorithms and implementation of perceptual inference. We will explore mechanistic and normative models for motion, binocular stereo, texture, surface and contour perception, perceptual organization and hierarchical models for object recognition, drawing knowledge from work in computer vision and computational neural models. We will study a number of algorithms that have been effective in computer vision for performing learning and inference, including gradient descent, particle and Kalman filtering, MCMC sampling and mean field approximation, and explore the links between observed neural dynamics and these inference algorithms. We will explore various theoretical frameworks on how perceptual representations are encoded and represented in neuronal ensembles, and what the recurrent feedback abundant in the visual cortex could be or is doing computationally.
Theme 3: Relationship between Perception, computation and art
Visual perception and artistic expression are deeply connected at many levels. In fact, visual perception in the brain might involves both analysis and synthesis. That is, our perception is not simply analyzing what is out there, but an active synthesis of an internal mental representation of what is out there, sometimes leading to illusion and hallucination. We will explore this synthesis process and how it might be tied to aesthetics and art making. The integration of visual art and the experimental study of vision has its roots in formal analysis of paintings. Advances in our understanding of how our brain or perception works have lead to resurgence of interests in linking art with vision science. Here, we will explore some of the new links between neuroscience, computational vision and the art, with a view to enrich our understanding and making of arts -- how artistic expression is rooted in perceptual computation and how scientific understanding of vision have transformed arts over the centuries.