[Overview] [Goals and Assessment] [Readings] [Software] [Syllabus: 1 2 3 4 5]

85-419/719: Introduction to Parallel Distributed Processing

Spring 2016, Tue/Thu 1:30-2:50am, Baker 336B

Instructor: David Plaut
Baker 254N, x85145
Office hours: By appt.
     TA: Bo Powers
Baker 455A, x8112
Office hours: Mon 11-noon; Wed 2-3pm, Psychology Dept. Lounge, 3rd Floor Baker Hall




The goal of the course is to introduce the basic principles of parallel distributed processing (also known as connectionist or neural network modeling) and to illustrate how these principles provide insight into human cognitive processing. In addition, the course will cover some issues in neural and cognitive development, cognitive impairments due to brain damage, and some basic computational issues. The course also attempts to introduce the general practice of studying cognition through computational modeling and analysis. There will be computer simulation exercises in addition to readings. Homework assignments will generally require you to report the results of simulations you have carried out, to analyze these results, and to think critically about some issues raised in the readings. There will also be a final project that will typically involve simulation modeling.

The course is divided into five sections. The first three cover basic topics in parallel distributed processing. For each of these, a homework assignment is handed out at the beginning of the section and is due at the end of the section (n.b. the third section's homework is split into two parts). At the end of the third section, you will also be required to submit a one-page proposal outlining the final project you intend to carry out. This will be returned with feedback at the beginning of the fourth section (right after Spring Break), and you will be expected to get started on your project immediately thereafter. You should be on the lookout throughout the earlier sections of the course for topics or issues that you find particularly interesting and would like to pursue in more detail in a project. The fourth section focuses on applications from a range of perceptual, linguistic and cognitive domains, and will be followed by a take-home essay (1200-1500 words) based on class lectures and readings. The final section will be devoted to a brief oral report from each student on the topic of their project. A 12-15 page final paper (5000-7000 words) based on the project is due at the end of this last section. There is no final exam for the course.

In general, there are assigned readings for each lecture that are intended to prepare you to participate in the class discussion for that day. In addition, there may be optional background readings (marked with "opt:" and in parentheses in the Syllabus) that serve either as the basis for the lecture, to present an alternative point of view, or simply to make available to you relevant material that we won't have time to cover in class. Optional readings are also a good source of ideas for projects. There are no required readings on days when something is due, but you are still expected to attend class, hand in your homework, and draw on the material you have already learned in order to participate in the discussion.

Course Goals and Assessment

Below are the broad goals of the course and how each is assessed (listed in brackets).

The grading in the class will be divided up as follows:

Homework 1:   10%
Homework 2:   15%
Homework 3:   10%
Homework 4:   10%
Project proposal:     5%
Take-home essay:   15%
Oral presentation/class participation:    5%
Final project   30%

Assignments should be uploaded within Canvas and are due at the beginning of class on the date listed in the Syllabus (usually a Tuesday). [Note: If you have difficulties uploading an assignment, you can email it to the course instructor.] Late penalties will be assessed as follows: Homeworks handed in late but before 5pm of the next day (usually a Wednesday) will be penalized by 5% of the total possible points (i.e., the graded score will be multiplied by 0.95); those handed in before 5pm of the following weekday (usually a Thursday, but a Monday if the homework was due on a Thursday) will be penalized by 10%; those handed in later than that but before graded papers are returned will be penalized by 15%. Papers may not be handed in for credit after other students' graded homeworks are returned and feedback is posted to the course webpage, unless you get explicit permission from the instructor. Late homeworks may be submitted to the instructor by email (pdf file). The 5% for class participation will be based on contributions to class discussions throughout the semester, and on the quality of the oral project report.


There is no required text for the course. All assigned and optional readings, as well as lecture slides, are available as downloadable pdf files from links in the Syllabus below. Other course materials (e.g., handouts, assignments, etc.) will be made available via links at the top of this web page. The following texts contain some of the course readings and may be useful as general references:


We will be using a software package called "Lens" (for Light Efficient Network Simulator), developed by former CMU CS graduate student Doug Rohde. Lens runs under Windows, Mac OSX, and Linux. The main website for Lens is http://tedlab.mit.edu/~dr/Lens/ but don't install Lens from that site. You can download a file containing a precompiled version of Lens here:

If you have any problems getting Lens running, contact the TA or instructor. After installing Lens, you should look at the online manual at http://tedlab.mit.edu/~dr/Lens/, particularly the instructions under "Running Lens" and the Tutorial Network under "Example Networks". The precompiled versions of Lens come with a offline (local) copy of the manual that can be accessed by pointing your web browser at Manual/index.html in the Lens directory.


This syllabus is subject to change throughout the course, so be sure to revisit this web page frequently.

Section 1: Processing and Constraint Satisfaction

Jan 12 (Tue): Overview and basic principles (slides) [HOMEWORK 0: Upload test file and Install Lens; see "Software" section in Overview] [HOMEWORK 1 POSTED]

Jan 14 (Thu): Lens tutorial [BRING LAPTOP]

Jan 19 (Tue): Constraint satisfaction

Jan 21 (Thu): Schema theory (slides)

Jan 26 (Tue): Psychological implications (slides) [HOMEWORK 1 DUE]

Section 2: Simple Learning and Distributed Representations

Jan 28 (Thu): Correlation-based learning (Hebb rule) (slides) [HOMEWORK 2 POSTED]

Feb 2 (Tue): Error-correcting learning (Delta rule)

Feb 4 (Thu): Distributed representations (slides)

Feb 9 (Tue): Psychological implications (slides) [HOMEWORK 2 DUE]

Section 3: Learning Internal Representations

Feb 11 (Thu): Back-propagation (slides) [HOMEWORK 3 POSTED]

Feb 16 (Tue): Temporal learning and recurrent networks (slides)

Feb 18 (Thu): Generalization and overfitting (slides)

Feb 23 (Tue): Boltzmann machines / Contrastive Hebbian learning [HOMEWORK 3 DUE] [HOMEWORK 4 POSTED] (slides)

Feb 25 (Thu): Unsupervised learning / generative models

Mar 1 (Tue): Deep learning (slides)

Mar 3 (Thu): Reinforcement learning / forward models (slides) [HOMEWORK 4 DUE] [PROJECT PROPOSAL DUE]

Mar 8 (Tue): NO CLASS (Spring Break)

Mar 10 (Thu): NO CLASS (Spring Break)

Section 4: Applications

Mar 15 (Tue): Cognitive development (slides)

Mar 17 (Thu): High-level vision and attention (slides)

Mar 22 (Tue): Semantics (slides)

Mar 24 (Thu): Memory and the hippocampus (slides)

Mar 29 (Tue): Language: Morphology (slides)

Mar 31 (Thu): Language: Word reading (slides)

Apr 5 (Tue): Language: Sentence processing (slides)

Apr 7 (Thu): Cognitive control and executive function [TAKE-HOME ESSAY POSTED]

Apr 12 (Tue): Future directions [TAKE-HOME ESSAY DUE]

Apr 14 (Thu): NO CLASS (Spring Carnival)

Section 5: Project Progress Reports

Apr 19 (Tue):
Apr 21 (Thu):
Apr 26 (Tue):
Apr 28 (Thu):