15-386/686 Neural Computation

Carnegie Mellon University

Spring 2019

Course Description

Neural Computation is an area of interdisciplinary study that seeks to understand how the brain learns and computes to achieve intelligence. It seeks to understand the computational principles and mechanisms of intelligent behaviors and mental abilities -- such as perception, language, motor control, decision making and learning -- by building artificial systems and computational models with the same capabilities. This course explores computational issues at multiple levels, from individual neurons to circuits and systems, with a view to bridging brain science and machine learning. It will cover basic models of neurons and circuits, computational models of learning, memories and inference, and quantitative approaches to neural system analysis in real and artifical systems. Concrete examples will be drawn from the visual system and the motor system, with emphasis on relating current deep learning research and the brain research, from hierarchical computation, attention, recurrent neural networks, to reinforcement learning. Students will learn to perform quantitative analysis as well as computational experiments using Matlab and deep learning platforms. No prior background in biology or machine learning is assumed. Prerequisites: 15-100, 21-120 or permission of instructor. 21-241 preferred but not required.

Course Information

Instructors Office (Office hours) Email (Phone)
Tai Sing Lee (Professor) MI 115. Friday 9:00-10:00 tai@cnbc.cmu.edu (412-268-1060)
Ziniu Wu (TA) GHC Tuesday 4:30-5:30 ziniuw@andrew.cmu.edu
Stella Yuan (TA) GHC Monday 5:30-6:30 yixiny@andrew.cmu.edu
Xuyang Fang (CA) no office hour xuyangf@andrew.cmu.edu

Recommended Supplementary Textbook

Classroom Etiquette

386 Grading Scheme

Evaluation% of Grade
5 Assignments 70
Midterm 10
Final Exam 20
Optional term project (Replacement of 1 HM or Midterm) 10-12
  • Grading scheme: A: > 88 B: > 75. C: > 65.
  • 686 Grading Scheme

    Evaluation% of Grade
    Option 1. 5 Assignments 35
    Option 2. Weekly Journal Club (reading / 3 presentations) 35
    Option 3. Term project 35
    Midterm 10
    Final Exam and/or Quiz 2 20
  • 686 students can choose two of the above three options, i.e. they have the three options to earn the 70 points: (1) term project + 5 problem sets; (2) term project + journal club (can miss at most 3 weeks); (3) journal club + 5 problem sets.
  • Grading scheme: A: > 88 B: > 75. C: > 65.
  • Assignments

    Term Project

    Examinations

    Late Policy

    Syllabus

    Date Lecture Topic Relevant Readings Assignments
      LEARNING AND REPRSENTATION    
    W 1/14 1. Introduction NIH Brain Facts (chapter 1)  
    M 1/16 2. Neurons and Membranes F. Rosenblatt - Perceptron.  
    M 1/21 Martin Luther King Day (no class)    
    W 1/23 3. Spikes and Cables Trappenberg Ch 1.1-2.2 (C) HW 1 out
    M 1/28 4. Synapse and Neural Net Trappenberg Ch 3.1  
    W 1/30 5. Polar Vortex (no class) McCulloch and Pitts (1943)  
    M 2/4 6. Synaptic plasticity Trappenberg Ch 4 Abbott and Nelson (2000)  
    W 2/6 7. Hebbian Learning Trappenberg Ch 4, HPK Ch 8 Oja (1982) HW 2 out, HW 1 in
    M 2/11 8. Features and Convolutions HPK Ch 8  
    W 2/13 9. Computaitonal Maps HPK Ch 9 Kohonen (1982)  
    M 2/18 10. Source Separation Olshausen and Field (1997) (2004)  
    W 2/20 11. Belief Net Hinton and Salahutdinov (2006) HW 2 in, HW 3 out.
    M 2/25 12. Neural Probability Codes Ma et al. (2006). Fiser et al. (2010).  
    W 2/27 13. Review    
    M 3/4 14. Midterm    
    W 3/6 15. Bayesian inference Kersten-Yuille (2003) Weiss et al. (2002).  
    M 3/11 Midterm grade, Spring break    
    W 3/13 Spring break    
      ASSOCIATION and INTERACTION    
    M 3/18 16. Hopfield and attractor network    
    W 3/20 17. Boltzmann Machines and Sampling    
    M 3/25 18. Memories and Cogntive Maps   HW3 HW4 out
    W 3/27 19. Multi-layer perceptron    
    M 4/1 20. Convolutional neural networks    
    W 4/3 21. Interpreting neural networks    
    M 4/8 22. More deep networks    
    W 4/10 23. Recurrent Circuits   HW4 in, HW5 out
    F 4/13 Spring Carnival    
    M 4/15 24. Markov Networks    
    W 4/17 25. Normalization  
    M 4/22 26. Feedback    
    W 4/24 27. Attention and Generative models   HW5 in
    M 4/29 28. Review    
    W 5/1 29. Reinforcement Learning   Term paper deadline
    F 5/3 30. no class   presentation deadline
    Tu 5/7 Final Exam    
    R 5/16 Final Grade due 4 p.m. for Graduates    
    T 5/21 Final Grade due 4 p.m.    

    Journal Club and Relevant Reading List

  • Every Friday 1:30-3:00, except 3/15 Spring Break, and 4/12 Spring Carnival. Total 13 weeks. Minimal attendance: 10 times (10 points). 20 points for 3-4 Presentations.

    Week 1 and 2 (1/18, 1/25): The Computer and the Brain

    Week 3 (2/1): Relating Structures, Circuits and Functions -- Retina Example

    Week 4 (2/8): Synaptic Plasticity and Biological Learning Rules

    Week 5 (2/15): More on Biological Learning Rules (BCM and PCA)

    Week 6 and 7 (2/22, 3/1): Sparse Coding and its Implication in Neural Networks

    Week 8 (3/8): Neural Codes -- Variability and Covariability

    Week 9 (3/18): Attractors and Memories

    Week 10 (3/25-4/5): Attractors and Prototype Networks

    Week 11 (4/13): (Spring Carnival) Recurrent Neural Networks and Target Propagation

    Week 12 (4/19): Generative Models and Transformer Networks

    Week: Imagination and Introspection Network

    Week: Reinforcement Learning, Curiosity Learning and Song Birds


    Questions or comments: contact Tai Sing Lee
    Last modified: spring 2019, Tai Sing Lee