[Overview] [Goals and Assessment] [Readings] [Software] [Syllabus: 1 2 3 4 5]

85-419/719: Introduction to Parallel Distributed Processing

Spring 2017, Tue/Thu 10:30-11:50am, Baker A53

Instructor: David Plaut
Baker 254N
Office hours: By appt. (arrange by email)
     TA: Ven Popov
Baker 345H
Office hours: Wed 9-10am; Thu 12:30-1:30pm




The goal of the course is to introduce the basic principles of parallel distributed processing (also known as connectionist or neural network modeling) and to illustrate how these principles provide insight into human cognitive processing. In addition, the course will cover some issues in neural and cognitive development, cognitive impairments due to brain damage, and some basic computational issues. The course also attempts to introduce the general practice of studying cognition through computational modeling and analysis. There will be computer simulation exercises in addition to readings. Homework assignments will generally require you to report the results of simulations you have carried out, to analyze these results, and to think critically about some issues raised in the readings. There will also be a final project that will typically involve simulation modeling.

The course is divided into five sections. The first three cover basic topics in parallel distributed processing. For each of these, a homework assignment is handed out at the beginning of the section and is due at the end of the section (n.b. the third section's homework is split into two parts). At the end of the third section, you will also be required to submit a one-page proposal outlining the final project you intend to carry out. This will be returned with feedback at the beginning of the fourth section (right after Spring Break), and you will be expected to get started on your project immediately thereafter. You should be on the lookout throughout the earlier sections of the course for topics or issues that you find particularly interesting and would like to pursue in more detail in a project. The fourth section focuses on applications from a range of perceptual, linguistic and cognitive domains, and will be followed by a take-home essay (1000-1200 words) based on class lectures and readings. The final section will be devoted to a brief oral report from each student on the topic of their project. A 12-15 page final paper (5000-7000 words) based on the project is due one week after the end of classes. There is no final exam for the course.

In general, there are assigned readings for each lecture that are intended to prepare you to participate in the class discussion for that day. In addition, there may be optional background readings (marked with "opt:" and in parentheses in the Syllabus) that serve either as the basis for the lecture, to present an alternative point of view, or simply to make available to you relevant material that we won't have time to cover in class. Optional readings are also a good source of ideas for projects. There are no required readings on days when something is due, but you are still expected to attend class, hand in your homework, and draw on the material you have already learned in order to participate in the discussion.

Course Goals and Assessment

Below are the broad goals of the course and how each is assessed (listed in brackets).

The grading in the class will be divided up as follows:

Homework 1:   10%
Homework 2:   15%
Homework 3:   15%
Homework 4:   10%
Project proposal:     5%
Take-home essay:   10%
Oral presentation/class participation:    5%
Final project   30%

Assignments should be uploaded within Canvas and are due at the beginning of class on the date listed in the Syllabus (usually a Tuesday). [Note: If you have difficulties uploading an assignment, you can email it to the course instructor, but do so immediately as the time of the email will be taken as the time of submission.] Late penalties will be assessed as follows: Homeworks handed in late but before 5pm of the next day (usually a Wednesday) will be penalized by 5% of the total possible points (i.e., the graded score will be multiplied by 0.95); those handed in before 5pm of the following weekday (usually a Thursday, but a Monday if the homework was due on a Thursday) will be penalized by 10%; those handed in later than that but before graded papers are returned will be penalized by 15%. Papers may not be handed in for credit after other students' graded homeworks are returned and feedback is posted, except with explicit permission from the instructor. Late homeworks may be submitted to the instructor by email (pdf file). The 5% for class participation will be based on contributions to class discussions throughout the semester, and on the quality of the oral project report.

Academic integrity

All submitted work must be solely the product of your own original work for this course. You must not work with other students on homeworks, and you must not look at solutions to problems from previous semesters of this course (from past students or the instructor), even if you have access to such solutions. Your final paper must appropriately cite the sources on which it is based, particularly when text from a source is included verbatim or paraphrased. Out of class, you are encouraged to discuss issues and content related to the course with other students, as well as possible final paper topics. However, you must not discuss a specific homework assignment until after the submissions for it are graded and returned. The minimum penalty for a violation of academic integrity is to receive zero points on the relevant assignment, and all violations will be reported to the Office of Student Affairs for possible further disciplinary action.


There is no required text for the course. All assigned and optional readings, as well as lecture slides, are available as downloadable pdf files from links in the Syllabus below. Other course materials (e.g., handouts, assignments, etc.) will be made available via links at the top of this web page. The following texts contain some of the course readings and may be useful as general references:


We will be using a software package called "Lens" (for Light Efficient Network Simulator), developed by former CMU CS graduate student Doug Rohde. Lens runs under Windows, Mac OSX, and Linux. The main website for Lens is http://tedlab.mit.edu/~dr/Lens/ but don't install Lens from that site. You can download a file containing a precompiled version of Lens here:

If you have any problems getting Lens running, contact the TA or instructor. After installing Lens, you should look at the online manual at http://tedlab.mit.edu/~dr/Lens/, particularly the instructions under "Running Lens" and the Tutorial Network under "Example Networks". The precompiled versions of Lens come with a offline (local) copy of the manual that can be accessed by pointing your web browser at Manual/index.html in the Lens directory.


This syllabus is subject to change throughout the course, so be sure to revisit this web page frequently.

Section 1: Processing and Constraint Satisfaction

Jan 17 (Tue): Overview and basic principles (slides) [HOMEWORK 0: Install Lens] [HOMEWORK 1 POSTED]

Jan 19 (Thu): Lens tutorial [BRING LAPTOP]

Jan 24 (Tue): Constraint satisfaction

Jan 26 (Thu): Schema theory (slides)

Jan 31 (Tue): Psychological implications (slides) [HOMEWORK 1 DUE] [HOMEWORK 2 POSTED]

Section 2: Simple Learning and Distributed Representations

Feb 2 (Thu): Correlation-based learning (Hebb rule) (slides)

Feb 7 (Tue): Error-correcting learning (Delta rule)

Feb 9 (Thu): Distributed representations (slides)

Feb 14 (Tue): Psychological implications (slides) [HOMEWORK 2 DUE] [HOMEWORK 3 POSTED]

Section 3: Learning Internal Representations

Feb 16 (Thu): Back-propagation (slides)

Feb 21 (Tue): Temporal learning and recurrent networks (slides)

Feb 23 (Thu): Generalization and overfitting (slides)

Feb 28 (Tue): Boltzmann machines / Contrastive Hebbian learning [HOMEWORK 3 DUE] [HOMEWORK 4 POSTED] (slides)

Mar 2 (Thu): Unsupervised learning / generative models (slides)

Mar 7 (Tue): Deep learning (slides)

Mar 9 (Thu): Reinforcement learning / forward models (slides) [HOMEWORK 4 DUE] [PROJECT PROPOSAL DUE]

Mar 14 (Tue): NO CLASS (Spring Break)

Mar 16 (Thu): NO CLASS (Spring Break)

Section 4: Applications

Mar 21 (Tue): Cognitive development (slides)

Mar 23 (Thu): High-level vision and attention (slides)

Mar 28 (Tue): Semantics (slides)

Mar 30 (Thu): Memory and the hippocampus (slides)

Apr 4 (Tue): Language: Morphology (slides)

Apr 6 (Thu): Language: Sentence processing (slides)

Apr 11 (Tue): NO CLASS (Passover)

Apr 13 (Thu): Cognitive control and executive function (slides) [TAKE-HOME ESSAY POSTED]

Apr 18 (Tue): Future directions [TAKE-HOME ESSAY DUE]

Apr 20 (Thu): NO CLASS (Spring Carnival)

Section 5: Project Progress Reports

Apr 25 (Tue):
Apr 27 (Thu):
May 2 (Tue):
May 4 (Thu):