15-386/686 Neural Computation

Carnegie Mellon University

Spring 2020

Course Description

Neural Computation is an area of interdisciplinary study that seeks to understand how the brain learns and computes to achieve intelligence. It seeks to understand the computational principles and mechanisms of intelligent behaviors and mental abilities -- such as perception, language, motor control, decision making and learning -- by building artificial systems and computational models with the same capabilities. This course explores computational issues at multiple levels, from individual neurons to circuits and systems, with a view to bridging brain science and machine learning. It will cover basic models of neurons and circuits, computational models of learning, memories and inference, and quantitative approaches to neural system analysis in real and artifical systems. Concrete examples will be drawn from the visual system and the motor system, with emphasis on relating current deep learning research and the brain research, from hierarchical computation, attention, recurrent neural networks, to reinforcement learning. Students will learn to perform quantitative analysis as well as computational experiments using Matlab and deep learning platforms. No prior background in biology or machine learning is assumed. Prerequisites: 15-100, 21-120 or permission of instructor. 21-241 preferred but not required.

Course Information

Instructors Office (Office hours) Email (Phone)
Tai Sing Lee (Professor) MI 115. Friday TBA tai@cnbc.cmu.edu (412-268-1060)
Darby Losey (TA) Wean Hall 6423. Monday 4:30-5:30 loseydm@cmu.edu
Geyang Zhang (TA) Wean Hall 5312. Tuesday 4:30-5:30. geyangz@andrew.cmu.edu

Recommended Supplementary Textbook

Classroom Etiquette

386 Grading Scheme

Evaluation% of Grade
6 Assignments 70
Class participation/attendance/quizzes 10
Midterm 10
Final Exam 10
Optional term project (Replacement of 1 HM or Exam) 10
  • Grading scheme: A: > 88 B: > 75. C: > 65.
  • 686 Grading Scheme

    Evaluation% of Grade
    Assignments 70
    Class participation/attendance/quizzes 10
    Midterm 10
    Final Exam 10
    Option 1. Weekly Journal Club (reading / presentations) 25
    Option 2. Term project 25
    Option 3. Term project and journal club replace 25 assigment points (~2 assignments)
  • 686 students can choose one of the above three options for the 20 points.
  • Grading scheme: A: > 88 B: > 75. C: > 65.
  • Assignments

    Term Project

    Examinations

    Late Policy

    Syllabus

    Date Lecture Topic Relevant Readings Assignments
      LEARNING AND REPRSENTATION    
    M 1/13 1. Introduction NIH Brain Facts (chapter 1)  
    W 1/15 2. Neurons and Membranes McCulloch and Pitts (1943) HW 1 out
    M 1/20 Martin Luther King Day (no class)    
    W 1/22 3. Spikes and Cables Trappenberg Ch 1.1-2.2 (C) Matlab tutorial Wean 5201 4:30-5:30
    M 1/27 4. Synapse and Neural Net Trappenberg Ch 3.1  
    W 1/29 5. Neuron models and Perceptron F. Rosenblatt - Perceptron. HW 2 out, HW 1 in
    M 2/3 6. Synaptic plasticity Trappenberg Ch 4 Abbott and Nelson (2000)  
    W 2/5 7. Hebbian Learning Trappenberg Ch 4, HPK Ch 8 Oja (1982)  
    M 2/10 8. Features and Convolutions HPK Ch 8  
    W 2/12 9. Computaitonal Maps HPK Ch 9 Kohonen (1982) HW 3 out. HW 2 in;
    M 2/17 10. Source Separation Olshausen and Field (1997) (2004)  
    W 2/19 11. Belief Net Hinton and Salahutdinov (2006)  
    M 2/24 12. Bayesian inference Kersten and Yuille (2003) Weiss et al. (2002).  
    W 2/26 13. Review    
    M 3/2 14. Midterm    
    W 3/4 15. Population Codes Ma et al. (2006). Fiser et al. (2010). HW 3 in. HW 4 out
    M 3/9 Midterm grade, Spring break    
    W 3/11 Spring break    
    M 3/16 16. Memories and Mind reading Redish, Gallant  
    W 3/18 17. Brain Computer Interface Chase and Schwartz  
      ASSOCIATION and INTERACTION    
    M 3/23 18. Recurrent and Attractor network Hopfield and Tank (1986)  
    W 3/25 19. Memories and Cogntive Maps Wikenheiser and Redish (2015) Hassabis et al. (2007) HW 4 in. HW 5 out
    M 3/30 20. Hierarchy and Reverse Hierarchy    
    W 4/1 21. Convolutional Neural Networks    
    M 4/6 22. Feedback and recurrent Circuits    
    W 4/8 23. Generative Models   HW 5 in. HW 6 out.
    M 4/13 24. Normalization and Attention    
    W 4/15 25. Reinforcement Learning    
    F 4/17 Spring Carnival  
    M 4/20 26. Curiosity and Imagination    
    W 4/22 27. Emotion and Consciouseness   HW 6 in
    M 4/27 28. Review and Connecting the Dots    
    W 4/29 29. Final Examination    
    F 5/1 30. Project Presentation   Term paper deadline
    R 5/14 Final Grade due 4 p.m. for Graduates    
    T 5/19 Final Grade due 4 p.m.    

    Journal Club and Relevant Reading List

  • Every Friday 1:30-3:00, except 3/13 Spring Break, and 4/17 Spring Carnival. Total 13 weeks. Minimal attendance: 10 times (10 points). 20 points for 3-4 Presentations.
  • The papers below are reading choices of last year (see www.cnbc.cmu.edu/~tai/nc19.html). New papers will be added as we will explore some new topics, such as brain computer interface, mind readng, emotion and consciouness.

    Week 1 (1/24): Logical computation in Neurons

    Week 2 (1/31): Advances in Neural Circuits

    Week 3 (2/8): Computational models of Neural Circuits

    Week 4 (2/17): Sparse Coding and Reinforcement Learning

    Week 5 (2/24): Biological Plausible Deep Learning Algorithms

    Suggested paper choices from last year plus new papers:

    Computer and the Brain

    Relating Structures, Circuits and Functions -- Retina Example

    Week 4 (2/7): Synaptic Plasticity and Biological Learning

    Week 5 (2/14): Principles of Sensory Processing and Learning

    Week 6 and 7 (2/21, 2/28): Sparse Coding and its Implication in Neural Networks

    Week 8 (3/6): Neural Codes -- Variability and Covariability, Priors and Inferences

    Week 9 (3/20): Neural Decoding, Brain Computer Interface

    Week 10 (3/27): Attractors and Prototype Networks

    Week 11 (4/3): Deep and recurrent Neural Networks

    Week 12 (4/24): Generative Models, Feedback and Attention

    Week 13: (May 1) Reinforcement Learning, Curiosity Learning and Song Birds


    Questions or comments: contact Tai Sing Lee
    Last modified: spring 2019, Tai Sing Lee
  • N.