15-387/86-375/675 Computational Perception

Carnegie Mellon University

Fall 2020

Course Description

In this course, we will study the biological and psychological data of biological perceptual systems, mostly the visual system, in depth, and then apply computational thinking to investigate the principles and mechanisms underlying natural perception. You will learn how to reason scientifically and computationally about problems and issues in perception, how to extract the essential computational properties of those abstract ideas, and finally how to convert these into explicit mathematical models and computational algorithms. The course is targeted to both neuroscience and psychology students who are interested in learning computational thinking, as well as computer science and engineering students who are interested in learning more about the neural and computational basis of perception. Prerequisites: First year college calculus, differential equations, linear algebra, basic probability theory and statistical inference, and programming experience (Matlab) are desirable.

Course Information

• Class location and time: TP 1101 or Zoom (link announced on CANVAS), Monday/Wednesday 1:30 p.m - 2:50 p.m.
• Class recitation and journal club: zoom (same as class link), 1:30 p.m - 2:50 p.m.
• Website: http://www.cnbc.cmu.edu/~tai/cp20.html (course info and readings )
• Canvas: Lecture materials and Information would be on Canvas.

Recommended Textbook

• Handouts on Canvas .
• Frisby and Stone Seeing: The computational approach to biological vision . MIT Press, 2010 (recommended).
• Supplementary Simon Prince Computer Vision: Models, Learning, and Inference . Cambridge University Press , 2012. Downloadable at: http://www.computervisionmodels.com. More relevant to graduate students.

Classroom Etiquette

• If in person, refrain from using laptop and cell phones. If on zoom, please turn on your video.

Grading Scheme 15-387 or 86-375

• Total points: 100
• Grading scheme: A: > 88%, B: > 75%. C: > 65%.

Grading Scheme 86-675

• Total credit for 86-675: 125
• Term paper can be used to replace journal club or up to 30 points of homework grade
• Grading scheme: A: > 88%, B: > 75%. C: > 65%.

Grading Scheme 86-375

• Total credit for 86-375: 100
40 points out of three of five problem sets.
• Term paper or Journal Club
• Grading scheme: A: > 88%, B: > 75%. C: > 65%.

Assignments

• 5 homework assignments, each involves Matlab (or Python) exercises as well as reading/research assignments.

Term Project

• A term paper is available to undergraduate students. It involves in-depth reading (of a group of 5 papers) on a particular topic in computational perception. A student must work by himself, and can use this to replace up to 10 points of any assignment or midterm grade.
• Term project is an available option for 86-675 and counts for 30 points toward the total grade of 125 points. If you are interested in doing a term project, you should switch to 86-675. A term project must involve some computational experiments, using either downloadable softwares or programs you develop. All the codes should be documented and archived in github and submitted together with a term paper (8 pages) materials in a different pdf/doc file and/or matlab zip files. Term project should take about 30 hours to complete. Each student should work on his/her own project. A project proposal is due on October 16 with literature research and preliminary results and thus students are encouraged to discuss project ideas with professor from the very beginning of the semester.

Journal Club

• Journal club option is an available option for 86-675 students and count for 30 points toward the total grade. Each student is expected to present three to four times during the course of the semester and participate in 90% of the journal club.
• Term project option is an available option for 86-675 and counts for 30 points of a total grade of 125 points. If you are interested in doing a term project, you should switch to 86-675. A term project must involve some computational experiments, using either downloadable softwares or programs you develop. All the codes should be documented and archived in github and submitted together with a term paper (8 pages) materials in a different pdf/doc file and/or matlab zip files. Term project should take about 30 hours to complete. Each student should work on his/her own project. A team project might be approved but the project is expected to be more substantial and ambitious (a combined 60 hours of work) and each student should have a well defined task and component in the project and could receive a different grade. A project proposal is due on October 16 with literature research and preliminary results and thus students are encouraged to discuss project ideas with professor from the very beginning of the semester.

Examinations

• There will be a midterm (10 points) and a final exam (15 points) to test materials covered in the lectures and homework assignments.
• There will be 12 in-class Piazza pool test that is designed mostly to keep track of attendance as well as to test understanding. Each pool worths 1 point for 15-387/86-375 student and 1/2 point for 86-675 student, with a maximum score of 10 points and 5 points respectively.

Late Policy

• Each student will have 7 days grace period for late homework. This grace period can be used for one or multiple assignments or the term paper. Use it wisely and you cannot ask for more. CANVAS submission after the starting of class time of due day is considered late by one day.

Syllabus

Journal Club

Week 1 September 11 Retina

• Retina might be the best understood part of the brain in terms of detailed circuits and functions. The reality is actaully more complicated than what we learned in the textbook. We will read a classic paper by McCulloch and Pitt (the inventor of neuron in neural networks), as well as a review paper by Markus Meister rediscovering the complexity of retinal computation half a century later.
• Lettvin, Maturana, McCulloch and Pitt. (1959) What the frog's eye tells the frog's brain Proceedings of the IRE 1940-1959 .
• Gollisch and Meister (2010) Eye Smarter than Scientists Believed: Neural Computations in Circuits of the Retina Neuron 65: 151-164.
• Turner, Scheartz and Rieke (2018) Receptive field center-surround interactions mediate context-dependent spatial contrast encoding in the retina eLife DOI: 10.7554/eLife.38841

Week 2 September 18. Retina Computation

• In the last few years, deep learning and large dataset of neural data provide new approach to the computation in the retina as well. We will critically review some of the recent papers by Ganguli's group and discuss what new insights deep learning approach teach us about the retina that is new. Below are some example papers from the past.
• Ocko, Lindsey, Ganguli and Deny (2018) The emergence of multiple retinal cell types throughefficient coding of natural movies NIPS
• Maheswaranathan, Kastner, Baccus and Ganguli (218) Infeerring hidden struture in multilayered neural circuits. PLOS Computaitonal Biology 14(8):e1006291
• Maheswaranthan, .... Ganguli and Baccus (2018) Deep learning models reveal internal structure and diverse computations in the retina under natural scenes bioRxiv, June 8, 2018.
• McIntosh, Maheswaranathan, Nayebi, Ganguli and Baccus (2016) Deep Learning Models of the Retinal Response to Natural Scenes NIPS

Week 3 September 26. Scene Statistics and Perception

• Why we see visual illusion as we do? The idea is that the visual system is performing inference, and the statistical structures of our natural environment are encoded as priors that influence our perception. Here, we will study a couple papers that try to relate scene statistics and perception. Yilun Chen and Ravesh Sukhnandan will present.
• Catherine Q. Howe, Zhiyong Yang, and Dale Purves (2005) The Poggendorff illusion explained by natural scene geometry PNAS 2005 102(21) 7707-7712.
• Catherine Q. Howe, and Dale Purves (2004) Natural-scene geometry predicts the perception of angles and line orientation. PNAS 2004 101(23): 8745-50.

Week 4 October 2. Lightness Computation

• Why we see visual illusion as we do? This week, we deal with some illusion on lightness perception from both scene statistics and computational perspectives. Katherine Giesa and Zhiyun Gong will present.
• Yang Zhiyong, and Dale Purves (2004) .The statistical structure of natural light patterns determines perceived light intensity PNAS 2004 101(23) 8745-8750.
• Stella Yu. Angular Embedding: from jarring intensity difference to perceived luminance CVPR 2009
• Richard Murray (2020) A model of lightness perception guided by probabilistic assumptions about lighting and reflectance Journal of Vision 20(7): 28:1-22
• Allred and Brainard (2013) A Bayesian model of lightness perception that incorporates spatial variation in the illumination Journal of Vision 13(7):18, 1-18.

Week 5 October 9. Retinex and Perception

• This week, we continue our investigation of retinex with a review and a study on subjective experience on color perception, specifically on the viral discussion on the dress, Jason Zhang and Junqi Chen will present.
• John McCann (2017) Retinex at 50: color theory and spatial algorithms, a review Journal of Electronic Imaging 26(3) 031204(2017). https://doi.org/10.1117/1.JEI.26.3.031204
• Christoph Witzel, Chris Racey, J. Kevin O’Regan (2017) The most reasonable explanation of “the dress”: Implicit assumptions about illumination Journal of Vision February 2017, Vol.17, 1. doi:https://doi.org/10.1167/17.2.1
• Pascal Wallisch (2017) Illumination assumptions account for individual differences in the perceptual interpretation of a profoundly ambiguous stimulus in the color domain: “The dress” Journal of Vision Vol.17, 5. doi:https://doi.org/10.1167/17.4.5

Week 5 October 16. Contour Processing

• The perception of lightness and color depend on contrast, which often reflects surface or color discontinuity. This discontinuity forms "contours". Long contour provides reliable estimate of the color and lightness contrast as well as shown in some of the illusion. This week, we will explore some recent work on contour processing from both computational and biological perspective. Yilun Chen will present Malik's paper and Harsha Sridhar will present Wu's paper.
• Xiaofeng Ren, Charless C. Fowlkes, Jitendra Malik (2008) Learning probabilistic models for contour completion in natural images I. J. Computer Vision 77:47-63.
• Chen R. Wang Feng, H. Liang, Wu Li (2017) Synergistic processing of visual contours across cortical layers in V1 and V2 Neuron 96 (6): 1388-1402.

Week 6 October 30. Art from the Eyes of Perception Science

• Yilun Chen will finish presenting Malik's paper and Katherine Giesa will present a paper by Cavanagh on insights on art from the perspective of perception science.
• Cavanagh, P. (2005) The artist as neuroscientist. Nature, 434, 301-307.
• Ostrovsky, Y., Cavanagh, P., & Sinha, P. (2005). Perceiving illumination inconsistencies in scenes. Perception, 34, 1301-14.
• Sayim, B., & Cavanagh P. (2011). The art of transparency. i-Perception, 2, 679-696
• Perdreau, F. & Cavanagh, P. (2011). Do artists see their retinas? Frontiers in Human Neuroscience, 5:171

Week 7 Nov 6. From texture synthesis to artistic style transfer

• Joan will present the works of Matthias Bethge's lab on texture synthesis and artistic style transfer
• L. A. Gatys, A. S. Ecker, and M. Bethge (2015) Texture Synthesis Using Convolutional Neural Networks NIPS 28
• L. A. Gatys, A. S. Ecker, and M. Bethge (2017) Texture and art with deep neural networks Current Opinions in Neurbiology 46, 178-186.
• L. A. Gatys, A. S. Ecker, and M. Bethge (2016) Image Style Transfer Using Convolutional Neural Networks CVPR 2016

Week 8 Nov 13. Sensitivity of naturalistic texture in visual cortex

• Julia Lee will present the works of Jeremy Freeman on texture sensitivity in monkey V2.
• Freeman J, Ziemba CM, Heeger DJ, Simoncelli EP, Movshon JA. A functional and perceptual signature of the second visual area in primates. Nature Neuroscience. 16: 974-81.
• Freeman J, Simoncelli EP. (2011) Metamers of the ventral stream. Nature Neuroscience.
• Ziemba CM, Freeman J, Movshon JA, Simoncelli EP. (2016) Selectivity and tolerance for visual texture in macaque V2. PNAS

Week 9 Nov 20. Attention in the Brain and AI

• Yilun will review papers on attention, its understanding in neuroscience and application in AI
• Grace Lindsay (2020) Attention in psychology, neuroscience and machine learning Frotnier Computational Neuroscience April 2020.
• Luo and Manunsell (2019) Attention can be subdivided into neurobiological components corresponding to distinct behavioral effects PNAS 116(52) 26187-26194.
• Eric Knudsen (2018) Fundamental components of attention. Annual Review of Neuroscience

Week 10 Dec 11. Affordance and Neural Scene Representation

• We will conclude with perception and scene analysis. Jason Zhang will read the classical paper by Gibson on affordance, and Harsha will read DeepMind's new paper on Neural Scene Representation and rendering.
• Gibson's ch 8 of The Ecological Approach to Visual Perception Houghton Mifflin Press. 1979
• S. M. A. Eslami, D. J. Rezende, F. Besse, F. Viola, A. S. Morcos, M. Garnelo, A. Ruderman, A. A. Rusu, I. Danihelka, K. Gregor, D. P. Reichert, L. Buesing, T. Weber, O. Vinyals, D. Rosenbaum, N. Rabinowitz, H. King, C. Hillier, M. Botvinick, D. Wierstra, K. Kavukcuoglu, D. Hassabis, Neural scene representation and rendering. Science 360, 1204–1210 (2018).

Supplementary Readings

Vision, Perceptual Systems and Philosophy

• Teller, D. (2016) Vision and Visual System 130 MB) Chapter 13, 16 and 17
• Marr, D. (1982) Chapter 1: The Philosophy and the Approach Vision 8-38 .
• Gollisch and Meister (2010) Eye Smarter than Scientists Believed: Neural Computations in Circuits of the Retina Neuron 65: 150-164
• Van Essen, Anderson and Felleman (1992) Information processing in primate visual systems: an integrated approach Science 5043: 419-423.
• Fellman and Van Essen ( 1991) Distributed Hierarchical Processing the the Primate Cerebral Cortex Cerebral Cortex 1-47.

Computations in the Retina

• Gollisch and Meister (2010) Eye Smarter than Scientists Believed: Neural Computations in Circuits of the Retina Neuron 65: 150-164
• Green, Kim and Seung, "Analogous Convergence of Sustained and Transient Inputs in Parallel On and Off Pathways for Retinal Motion Computation" Cell Reports 14, 1892–1900, March 1, 2016
• Ocko, Lindsey, Ganguli and Deny (2018) The emergence of multiple retinal cell types throughefficient coding of natural movies NIPS
• Maheswaranathan, Kastner, Baccus and Ganguli (218) Infeerring hidden struture in multilayered neural circuits. PLOS Computaitonal Biology 14(8):e1006291
• Maheswaranthan, .... Ganguli and Baccus (2018) Deep learning models reveal internal structure and diverse computations in the retina under natural scenes bioRxiv, June 8, 2018.
• McIntosh, Maheswaranathan, Nayebi, Ganguli and Baccus (2016) Deep Learning Models of the Retinal Response to Natural Scenes NIPS

Neural Codes, Features and Representational Learning

• Olshausen and Field (1996) Emergence of simple-cell receptive field properties by learning a sparse code for natural images Nature 381: 607-609.
• Olshausen and Field (2004) Sparse coding of sensory inputs Current Opinions in Neurobiology 14:481-487.
• Vinje, W.E., Gallant, J.L. Sparse Coding and Decorrelation in Primary Visual Cortex During Natural Vision, Science Vol. 287, Issue 5456, pp. 1273-1276.
• Vinje and Gallant (2002) Natural stimulation of the non-classical receptive field increases information transmission efficiency in V1 J. Neuroscience 22(7): 2904-2915.
• Barlow (1954) Possible principles underying the transformation of sensory messages Sensory Communication MIT Press, p217-234.
• Barlow (2001) Redundancy reduction revisited Network: Computation in Neural Systems 12: 241-253.
• Shlens: PCA tutorial (2003) online
• Simoncelli: Geometric review of linear algebra (2017)) online
• Hyvarinen and Oja (1999) Independent component analysis: tutorial On-line tutorial https://research.ics.aalto.fi/ica/fastica/
• J. Puchalla, E. Schneidman, R. Harris, and M.J. Berry II, “Redundancy in the Population Code of the Retina”, Neuron 46: 493-504 (2005
• G. Tkacik, O. Marre, D. Amodei, W. Bialek, and M.J. Berry II, “Searching for Collective Behavior in Networks of Real Neurons”, PLoS Computational Biology 10(1): e1003408 (2014).
• J.S. Prentice, O. Marre, M.L. Ioffe, A.R. Loback, G. Tkacik, and M.J. Berry II, “Error-Robust Modes of the Retinal Population Code”, PLoS Computational Biology 12(11): e100514 (2016).
• E. Schneidman, M.J. Berry II, R. Segev, and W. Bialek, “Weak Pairwise Correlations Imply Strongly Correlated Network States in a Neural Population”, Nature 440: 1007-1012 (2006).

Lightness and color perception, Retinex

• Adelson, Ed, (2000) Lightness Perception and Lightness Illusion The New Cognitive Neuroscience, Gazzaniga ed. MIT Press. (2000).
• Land, E, (1977) The retinex theory of color vision Scientific America 1977
• Horn, B, (1974) Determininng lightness from an image Computer Graphics and Image Procwssing. 1974
• Morel JM, Petro AB, Sbert C. (2010) IEEE Trans Image Process. 19(11) 2825-37.
• Tappen, Freeman and Adelson (2005) Recovering intrinsic images from a single image IEEE PAMI. 27(9): 1459-1472.

Mid-level vision: Texture, depth and motion Perception

• Some classic papers by Simoncelli, Julesz -- Relevant for problem set 4
• Weiss, Simoncelli and Adelson (2002) Motion Illusion as optimal percepts Nature Neuroscience 5(6): 598-.
• Freeman and Simoncelli (2011) Metamers of the ventral stream. Nature Neuroscience 14(9): 1195-.
• Freeman et al. (2013) A functional and perceptual signature of the second visual area in primate Nature Neuroscience 16(7): 974-

Visual Hierarchy, Object recognition, Abstract Representations

• S. Geman (1999) Invariance and selectivity in the ventral visual pathway. Journal of Physiology Paris 100: 212-224.
• LeCun, Bengio and Hinton (2016) Deep Learning Nature 521: 436-444.
• Krizhevsky, I. Sutskever, and G. E. Hinton, (2012) ImageNet Classification with Deep Convolutional Neural Networks.” NIPS online.
• M. D. Zeiler and R. Fergus, “Visualizing and Understanding Convolutional Networks, ECCV. online.
• Kar K, Kubilius J, Schmidt K, Issa EB, DiCarlo JJ. Evidence that recurrent circuits are critical to the ventral stream’s execution of core object recognition behavior. Nature Neuroscience. 2019. doi: 10.1038/s41593-019-0392-5.

Feedback, Synthesis and Predictive Coding

• Bastos, A.M., et al. (2012) Canonical microcircuits for predictive coding. Neuron, 2012. 76(4): p. 695-711.
• Rao, R.P. and D.H. Ballard (1999) Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects. Nature neuroscience, 1999. 2(1): p. 79-87.
• Spratling, M.W., A review of predictive coding algorithms. Brain and cognition, 2017. 112: p. 92-97.
• Miller, K.D., Canonical computations of cerebral cortex. Current opinion in neurobiology, 2016. 37: p. 75-84..

Generative models, Art, Abstract Representations

• Salakhutdinov (2015) Learning Deep Generative Models Annual REview of Statistics and its applications 2: 361-85
• Gatys, L, Ecker, AS, Bethge, Matthias (2018) Texture and art with deep neural networks Trends in Neuroscience Submitted.
• Gatys, L, Ecker, AS, Bethge, Matthias (2016) Image style transfer using CNN CVPR (code) https://github.com/leongatys/NeuralImageSynthesis
• Bashivan P, Kar K, DiCarlo JJ. (2019) Neural population control via deep image synthesis. Science. 364(6439)
• Schmidhuber Jurgen. (1997) Low complexity art.
• Schmidhuber Jurgen. (2008) Driven by Compression Progress: A simple principle explains essential aspects of subjective beauty, novelty, surprise, interestingness, attention, curiosity, creativity, art, science, music, and jokes!
• Bilge Sayim and Patrick Cavangah (2011) What line draws reveal about the visual brain.

Belief, memories and Association

• Ramalingam, McManus, Li and Gilbert (2013) Top-down modulation of lateral interactions in visual cortex Journal of Neuroscience 33(5): 1773-1789.
• Li and Gilbert (2014) Top-down influences on visual processing Nature Review Neuroscience 14: 350.
• Yan et al. (2014) Perceptual training continously refines neuronal population codes in primary visual cortex Nature Neuroscience 17(1); 1380.
• Hinton and Sejnowski (1984) Learning and relearning in Boltzmann machine Parallel Distributed Processing
• Salakhutdinov (2015) Learning deep generative models Annual Review of Statistics and Its Application 2:361-85.
• Zhang et al. (2016) Relating functional connectivity in V1 neural circuits and 3D antural scenes using Boltzmann machines Vision Research 120: 121-131.
• Shushruth et al. (2012) Strong Recurrent Networks Compute the Orientation Tuning of Surround Modulation in the Primate Primary Visual Cortex. J. Neuroscience 32(1): 308-321.
• Tom Albright. (2012) On the Perception of Probable Things: Neural Substrates of Associative Memory, Imagery, and Perception Neuron 74. 227-245.

Analysis by Synthesis

• Ilker Yildirim, Mario Belledonne, Winrich Freiwald, Josh Tenenbaum (2020) Efficient inverse graphics in biological face processing Sci. Adv. 2020; 6 : eaax5979 4 March 2020
• J. Z. Leibo, Q. Liao, F. Anselmi, W. A. Freiwald, T. Poggio, View-tolerant face recognition and hebbian learning imply mirror-symmetric neural tuning to head orientation. Curr. Biol. 27, 62–67 (2017).
• T. D. Kulkarni, W. F. Whitney, P. kohli, J. Tenenbaum, Deep convolutional inverse graphics network, in Proceeding of the Advances in Neural Information Processing Systems (NIPS, 2015), pp. 2539–2547
• J. Wu, Y. Wang, T. Xue, X. Sun, W. T. Freeman, J. B. Tenenbaum, MarrNet: 3D shape reconstruction via 2.5D sketches, in Proceeding of the Advances in Neural Information Processing Systems (NIPS, 2017).

Composition and Grammar

• E. Bienenstock, S. Geman, and D. Potter. Compositionality, MDL Priors, and Object Recognition NIPS Advances in Neural Information Processing Systems 9. 1998.
• L. Zhu, C. Lin, H. Huang, Y. Chen, A.L. Yuille. Unsupervised Structure Learning: Hierarchical Recursive Composition, Suspicious Coincidence and Competitive Exclusion. Proceedings of the European Conference on Computer Vision. ECCV. (see also another paper by Zhu and Yuille with Bill Freeman.) 2008.
• S.C. Zhu and D. Mumford (2006) A Stochastic Grammar of Images Foundations and Trends in Computer Graphics and Vision 2(4): 259-362.

Attention, Eye Movement and Routing

• Olshausen, Anderson and Van Essen (1995) Mutliscale dynamic routing circuit for forming size- and position-invariant object recognition J. Neuroscience 2:45-62.
• David Arathorn (1995) Computation in the Higher Visual Cortices: Map-Seeking Circuit Theory and Application to Machine Vision online. Long version (book) by Stanford University Press as Map-Seeking Circuit.
• Sara Sabour, Nicholas Frosst and Geoffrey Hinto (2017) Dynamic routing between capsules NIPS
• Kovacs, I., Papathomas, T., Yang, M. and Feher, A. (1996). When the brain changes its mind: Interocular grouping during binocular rivalry. Proceedings of the National Academy of Sciences, 93(26), pp.15508-15511.

Emotion and Action on perception

• Niedenthal and Wood (2019) Does emotion influence visual perception? Depends on how you look at it. Cognition and Emotion, 33 (1),77-84.
• Gayet, Paffen, Belopolsky and Theeuwes and Van Der Stigchel (2016) Visual input signaling threat gains preferential access to awareness in a breaking continuous flash suppression paradigm. Cognition 149, 77-83.
• Siegel, Wormwood, Quigley and Barrett. (2018) Seeing what you feel: affect drives visual percpetion of structurally neural faces. Psychological Science 29(4):496-503.
• Phelps, Ling and Carrasco M. (2006) Emotion facilitates perception and potentiates the perceptual benefits of attention. Psychological Science. 17(4): 292-299.
• Wood, A., Lupyan G., Sherrin, S, Niedenthal, P. Alterating sensorimotor feedback disrupts visual discrimination of facial expressions. Psychonomic Bulletin & Review. Volume 23, Issue 4, pp 1150–1156.
• Gallese Vittorio (2013) Mirror Neurons and the Perception–Action Link. The Oxford Handbook of Cognitive Neuroscience: Volume 2: The Cutting Edges.
• Gibson, J. J. (1979). The ecological approach to visual perception (Boston: Houghton Mifflin)
• Loomis J M, Philbeck J W, (2008). Measuring spatial perception with spatial understanding and action”, in Embodiment, Ego-Space, and Action Eds. R L Klatzky, B MacWhinney, M Behrmann (Cambridge, MA: The MIT Press) pp. 1–44, here: p. 33.
• Philbeck and Witt (2015) Action-Specific Influences on Perception and Post-Perceptual Processes: Present Controversies and Future Directions. Pyschological Bull. 141(6) 1120-1144.
• Witt, Jessica (2011) Action's Effect on Perception. Current Directions in Psychological Science.