[Overview]
[Goals and Assessment]
[Readings]
[Software]
[Syllabus]
85-419/719: Introduction to Parallel Distributed Processing
Spring 2008, Tue/Thu 10:30-11:50am, Doherty 1211
Course webpage: http://www.cnbc.cmu.edu/~plaut/IntroPDP/
Announcements
- Links to lecture slides are in the syllabus listing for that day.
- Lens creates a postscript file when it prints to a file. The easist way
to view and print these files under Windows is with a postscript viewer
like GSView, which you can download from http://www.cs.wisc.edu/~ghost/gsview/.
- If installing Lens under Windows: Lens does not handle path
names with spaces well. A good way to avoid this problem, if you have
Administrator privileges, is to install Lens at the top level of the
main drive (i.e, C:\Lens). Also, to allow the internal "help" command
to find the relevant files, you should then set LENSDIR to C:\\Lens
(i.e., with a double slash). The specification of the Bin folder for
the PATH variable can still be a single slash.
Assignments
Overview
The goal of the course is to introduce the basic principles of parallel
distributed processing (also known as connectionist or neural network modeling)
and to illustrate how these principles provide insight into human perceptual,
linguistic, and cognitive behavior. In addition, the course will cover some
issues in neural and cognitive development, cognitive impairments due to brain
damage, and some basic computational issues. The course also attempts to
introduce the general practice of studying cognition through computational
modeling and analysis. There will be computer simulation exercises in addition
to readings. Homework assignments will generally require you to report the
results of simulations you have carried out, to analyze these results, and to
think critically about some issues raised in the readings. There will also be
a final project that will typically involve simulation modeling.
The course is divided into five sections. The first three cover basic
topics in parallel distributed processing. For each of these three sections,
a homework assignment is handed out at the beginning of the section and
is due at the end of the section. At the end of the third section (just
before Spring Break), you will also be required to submit a one-page
proposal outlining the final project you intend to carry out. This
will be returned with feedback after Spring Break and you will be expected to
get started on your project immediately thereafter. You should be on the
lookout throughout the earlier sections of the course for topics or issues
that you find particularly interesting and would like to pursue in more detail
in a project. The fourth section focuses on applications from a wide range of
perceptual, linguistic and cognitive domains, and will be followed by a
take-home exam based on class lectures and readings. The final section
(4 class meetings) will be devoted to brief oral reports from each student on
the progress of their project. A 10-15 page paper based on the final
project is due at the end of this last section. There is no final exam
for the course.
In general, there are assigned readings for each lecture that are intended
to prepare you to participate in the class discussion for that day. In
addition, there may be optional background readings (listed in parentheses and
marked with ``opt:'' in the Syllabus) that serve either as the basis
for the lecture, to present an alternative point of view, or simply to make
available to you relevant material that we won't have time to cover in class.
Optional readings are also a good source of ideas for projects. There are no
required readings on days when something is due, but you are still expected to
attend class, hand in your homework, and draw on the material you have already
learned in order to participate in the discussion.
[Back to Top]
Course Goals and Assessment
Below are the broad goals of the course and how each is assessed (listed in
brackets).
- Extend breadth of knowledge of cognitive psychology, including theoretical
perspectives, research findings, and applications [through assigned readings,
homeworks, and in-class discussions]
- Foster familiarity with diverse experimental paradigms used in psychology [through
hands-on experience with computational modeling, assessed via homework
assignments and the final project].
- Engender the ability to read and critique psychological articles [assessed
through homework assignments, take-home exam, and in-class discussions of
assigned readings].
- Improve skill in oral and written presentation [through the oral project
presentation and written final project report].
- Increase facility in designing psychological studies to address research questions
[through the design of a final modeling project].
- Foster critical thinking and creativity [through in-class discussions and the
formulation and execution of a final project].
The grading in the class will be divided up as follows:
| Homework 1: | 10% |
| Homework 2: | 15% |
| Homework 3: | 20% |
| Project Proposal: | 5% |
| Take-Home Exam: | 15% |
| Final Project: | 30% |
| Class Participation: | 5% |
Homeworks are due at the beginning of class on the date listed in
the Syllabus (usually a Tuesday). Homeworks handed in late but before 5pm of
the next day (usually a Wednesday) will be penalized by 5% (i.e., total points
multiplied by 0.95); those handed in before 5pm of the following weekday
(usually a Thursday, but a Monday if the homework was due on a Thursday) will
be penalized 10%; those handed in later than that but before graded homeworks
are returned will be penalized 20%. After other students' graded homeworks are
returned, homeworks may be handed in for credit only with permission of the
instructor. The 5% for class participation will be based on contributions to
class discussions throughout the semester, and on the quality of the oral
project report.
[Back to Top]
Readings
There is no required text for the course. All assigned and optional readings
are available as downloadable pdf files from links in the Syllabus below. Other course materials (e.g.,
handouts, assignments, etc.) will be made available via links at the top of
this web page.
The following texts contain some of the course readings and may be useful as
general references:
- PDP1:
Rumelhart, D. E., McClelland, J. L., & the PDP research group (1986).
Parallel distributed processing: Explorations in the microstructure
of cognition. Volume 1: Foundations. Cambridge, MA: MIT Press.
- PDP2:
McClelland, J. L., Rumelhart, D. E., & the PDP research group (1986).
Parallel distributed processing: Explorations in the microstructure
of cognition. Volume 2: Psychological and biological models.
Cambridge, MA: MIT Press.
- PDP Handbook:
McClelland, J. L. and Rumelhart, D. E. (1988). Explorations in
parallel distributed processing: A handbook of models, programs, and
exercises. Cambridge, MA: MIT Press.
- MPR:
McLeod, P., Plunkett, K. and Rolls, E. T. (1998). Introduction to
Connectionist Modelling of Cognitive Processes. Oxford: Oxford
University Press.
[Back to Top]
Software
We will be using a software package called "Lens" (for Light Efficient Network
Simulator), developed by former CMU CS graduate student Doug Rohde. Lens runs
under both Windows and Unix-based systems (including Linux and Mac OS X). The
main website for Lens is http://tedlab.mit.edu/~dr/Lens/.
If you're running one of the following operating systems, you can download a
file containing a precompiled version of Lens.
- Windows 95/98/Me/NT/2000/XP
Download the file
Lens-windows.zip
and use WinZip or a similar program to unzip the file. This will create
a directory called "Lens". Read the file "README.rtf" in this directory
for further instructions.
- Linux
Download the file
Lens-linux.tar.gz.
Open a terminal and untar it with the command "tar xzf
Lens-linux.tar.gz". This will create a directory called "Lens". Read
the text file README in this directory for further instructions.
- Mac OS X
Download the file
Lens-OSX.tar.gz.
Open a terminal and untar it with the command "tar xzf Lens-OSX.tar.gz". This will
create a directory called "Lens". Read the text file README in this
directory for further instructions.
If you're not using one of these platforms, or if for some reason the
precompiled version for your platform doesn't work (let me know if this
happens), you'll need to download Lens and compile it yourself. Instructions
for how to do this can be found at
http://tedlab.mit.edu/~dr/Lens/installing.html.
After installing Lens, you should look at the online manual at http://tedlab.mit.edu/~dr/Lens/,
particular the instructions under "Running Lens" and the
Tutorial Network under "Example Networks".
The precompiled versions of Lens come with a offline (local) copy of the
manual that can be accessed by pointing your web browser at
Manual/index.html in the Lens directory. Those installing Lens from
scratch can also download an offline copy of the manual from http://tedlab.mit.edu/~dr/Lens/Dist/lens-manual.tar.gz.
Just move this into your Lens directory and run "tar xzf
lens-manual.tar.gz".
[Back to Top]
Syllabus
This syllabus is subject to change throughout the course, so be sure to
revisit this web page frequently.
Section 1: Processing and Constraint Satisfaction
Jan 15 (Tue): Overview and basic principles
(slides)
[HOMEWORK 1 HANDED OUT]
[Install Lens (see Software section)]
Jan 17 (Thu): Lens tutorial
Jan 22 (Tue): Constraint satisfaction
Jan 24 (Thu): Schema theory
Jan 29 (Tue): Psychological implications
[HOMEWORK 1 DUE]
- (opt:
McClelland, J. L. & Rumelhart, D. E. (1981).
An interactive activation model of context effects in letter perception:
Part 1. An account of basic findings. Psychological Review,
88, 375-407.)
- (opt:
McClelland, J. L. & Elman, J. L. (1986).
The TRACE model of speech perception. PDP2, Chapter 15.)
- (opt:
Dell, G. S., Schwartz, M. F., Martin, N., Saffran, E. M., & Gagnon, D. A. (1997).
Lexical access in normal and aphasic speakers.
Psychological Review, 104, 801-838.)
Section 2: Simple Learning and Distributed Representations
Jan 31 (Thu): Hebb and Delta rules
(slides)
[HOMEWORK 2 HANDED OUT]
- McClelland, J. L., and Rumelhart, D. E. (1988).
The pattern associator. PDP Handbook, Chapter 4 (pp. 83-96).
- (opt:
McLeod, P., Plunkett, K. and Rolls, E. T. (1998).
Pattern association. MPR, Chapter 3.)
- (opt:
McLeod, P., Plunkett, K. and Rolls, E. T. (1998).
Autoassociation. MPR, Chapter 4.)
Feb 5 (Tue): Pattern association
(slides)
Feb 7 (Thu): Distributed representations
Feb 12 (Tue): Psychological implications
[HOMEWORK 2 DUE]
Section 3: Learning Internal Representations
Feb 14 (Thu): Back-propagation
(slides)
[HOMEWORK 3 HANDED OUT]
Feb 19 (Tue): Temporal learning and recurrent networks
Feb 21 (Thu): Generalization and overfitting
- (opt:
Morgan, N. & Bourlard, H. (1990).
Generalization and parameter estimation in feedforward nets: Some
experiments. In D. S. Touretzky (Ed.), Advances in neural
information processing systems 2. San Mateo: CA: Morgan Kaufmann,
630-637. )
- (opt:
le Cun, Y., Denker, J. S., & Solla, S. A. (1990).
Optimal brain damage.
In D. S. Touretzky (Ed.), Advances in neural information processing
systems 2. San Mateo: CA: Morgan Kaufmann, 598-605.)
- (opt:
Weigand, A. S., Rumelhart, D. E., & Huberman, B. A. (1991).
Generalization by weight-elimination with application to forcasting. In
R. P. Lippmann, J. E. Moody, & D. S. Touretzky (Eds.), Advances in
neural information processing systems 3. San Mateo: CA: Morgan
Kaufmann, 875-882.)
Feb 26 (Tue): Contrastive Hebbian learning
Feb 28 (Thu): Unsupervised learning
(slides)
Mar 4 (Tue): Reinforcement learning and forward models [lecture by C. Watson]
(slides)
- Barto, A. G. (1995).
Reinforcement learning; Reinforcement learning in
motor control. In M. A. Arbib (Ed.), The handbook of brain theory
and neural networks (pp. 804-813). Cambridge, MA: MIT Press.
- (opt:
Jordan, M. I. & Rumelhart, D. E. (1992).
Forward models: Supervised learning with a distal teacher.
Cognitive Science, 16, 307-354.)
- (opt:
Tesauro, G. (1995).
Temporal difference learning and TD-Gammon.
Communications of the ACM, 38, 58-68.)
Mar 6 (Thu): Psychological implications
(slides)
[HOMEWORK 3 DUE]
- (opt:
McClelland, J. L. (2001).
Failures to learn and their remediation: A Hebbian account. In
J. L. McClelland and R. S. Siegler (Eds.), Mechanisms of cognitive
development: Behavioral and neural perspectives, (pp. 97-121).
Mahwah, NJ: Lawrence Erlbaum Associates.)
- (opt:
McCandliss, B. D., Fiez, J. A., Protoapas, A., Conway, M., McClelland, J. L. (2001).
Success and failure in teaching the [r] [l] contrast to Japanese adults:
Tests of a Hebbian model of plasticity and stabilization in spoken
language perception. Cognitive, Affective, & Behavioral
Neuroscience, 2, 89-108.)
Mar 11, 13: NO CLASS (Spring Break)
Section 4: Applications
Mar 18 (Tue): Memory and the hippocampus
(slides)
[PROJECT PROPOSAL DUE]
Mar 20 (Thu): High-level vision and attention [lecture by M. Behrmann]
(slides)
- Mozer, M. C. and Sitton, M. (1998).
Computational modeling of spatial attention.
In H. Pashler (Ed.),
Attention (pp. 341-393). Hove, England: Psychology
Press/Erlbaum.
- (opt:
Mozer, M. C. and Behrmann, M. (1990).
On the interaction of selective attention and lexical
knowledge: A connectionist account of neglect dyslexia.
Journal of Cognitive Neuroscience, 2,
96-123.)
- (opt:
Behrmann, M., Zemel, R. S. and Mozer, M. C. (1998).
Object-based attention and occlusion: Evidence from normal
participants and a computational model. Journal of Experimental
Psychology: Human Perception and Performance, 24,
1101-1036.)
Mar 25 (Tue): Cognitive development
(slides)
- Munakata, Y. and McClelland, J. L. (2003).
Connectionist models of development. Developmental Science, 6,
413-429.
- (opt:
Munakata, Y., McClelland, J. L., Johnson, M. H. & Siegler, R. (1997).
Rethinking infant knowledge: Toward an adaptive process account of
successes and failures in object permanence tasks. Psychological
Review, 104, 686-713.)
Mar 27 (Thu): Language: Morphology
(slides)
-
The past-tense debate.
Trends in Cognitive Sciences, 2002, 6, 456-474.
[Pinker, S. & Ullman, M T.
The past and future of the past tense, 456-463;
McClelland, J. M. & Patterson, K.
'Words or Rules' cannot exploit the regularity in exceptions:
Reply to Pinker and Ullman, 464-465;
McClelland, J. M. & Patterson, K.
Rules or connections in past-tense inflections: What does the evidence
rule out?, 465-472;
Pinker, S. & Ullman, M T.
Combination and structure, not gradedness, is the issue: Reply to
McClelland and Patterson, 472-474.]
- (opt: Rumelhart, D. E. & McClelland, J. L. (1986).
On learning the past tenses of English verbs. PDP2, Chapter 18.)
Apr 1 (Tue): Language: Word reading
(slides)
- Plaut, D. C. (1999).
Computational modeling of word reading, acquired dyslexia, and remediation.
In R. Klein and P. A. McMullen (Eds.), Converging methods in
reading and dyslexia (pp. 339-372). Cambridge, MA: MIT Press.
- (opt:
Plaut, D. C., McClelland, J. L., Seidenberg, M. S., & Patterson, K. (1996).
Understanding normal and impaired word reading:
Computational principles in quasi-regular domains. Psychological
Review, 103, 56-115.)
- (opt:
Coltheart, M., Rastle, K., Perry, C. and Langdon, R. & Ziegler, J. (2001).
DRC: A dual route cascaded model of visual word recognition and reading
aloud. Psychological Review, 108, 204-256.)
Apr 3 (Thu): Language: Sentence processing
(slides)
- McClelland, J. L., St. John, M., & Taraban, R. (1989).
Sentence comprehension: A parallel distributed processing approach.
Language and Cognitive Processes, 4, 287-335.
- (opt:
Elman, J. L. (1993).
Learning and development in neural networks: The
importance of starting small. Cognition, 48, 71-99.)
- (opt:
Rohde, D. L. T., and Plaut, D. C. (1999).
Language acquisition in the absence of explicit negative evidence: How
important is starting small? Cognition, 72, 67-109.)
- (opt:
Frazier, L. (1987).
Sentence processing: A tutorial review.
In M. Coltheart (Ed.), Attention and performance XII: The
psychology of reading (pp. 559-586). Hillsdale, NJ: Erlbaum.)
Apr 8 (Tue): Semantics
(slides)
Apr 10 (Thu): Routine action
(slides)
[TAKE-HOME EXAM HANDED OUT (covering Mar 21 to Apr 13)]
Apr 15 (Tue): Cognitive control [lecture by C. Watson]
[TAKE-HOME EXAM DUE]
Apr 17 (Thu): NO CLASS (Spring Carnival)
Section 5: Project Progress Reports
Apr 22 (Tue):
Arizpe, Doersch, Doyle, Flynn, Lundin, Maas, Whitney, Zachariou
Apr 24 (Thu):
Austin, Farmer, Kramer, Lim, Navia, Sandler, Sheehan, Wen
Apr 29 (Tue):
Comer, Hwang, Iwanaga, Klionsky, McCortney, Sampson, Scozio, Stedenfeld
May 1 (Thu):
Bates, Danker, Kanter, Liu, Nowak, Mou, Shen, Taylor
May 2 (Fri) 5pm:
PROJECT PAPER DUE
[Back to Top]