Introduction to Parallel Distributed Processing
(85-419 & 85-719)

Overview

Studying cognition using PDP models is a challenging interdisciplinary endeavor requiring familiarity with notions from artificial intelligence, statistics, psychology, and neuroscience. Thus, students may find this course highly demanding. Mastery of the course materials will require extensive reading, puzzling through unfamiliar concepts, active participation in class discussions, learning new mathematical formalisms, many hours of hands-on experience building and analyzing computer simulations, and a willingness to view cognition in new ways.

This course consists of six sections. Each section involves about two or three weeks of readings, lectures, discussions, and computer simulation exercises. The first four sections cover the mechanics and formal properties of various PDP modeling methods, along with demonstrations of how these methods may be used to investigate many aspects of cognition. The fifth section surveys a number of cognitive domains in more detail, applying a PDP perspective to issues in language, memory, and brain damage. The last section provides an opportunity for students to learn from each other through the presentation of results from simulation projects conducted during the course.

Resources

Readings for this course are drawn primarily from three seminal tomes of this field:

These books are available in the bookstore. They are also on reserve in the Hunt Library.

One of the goals of this course is to introduce students to the PDP++ software package. This software system may be used to conduct computer simulations of PDP models. Use of this system will be required in order to complete course exercises and projects. Documentation for PDP++ may be found in:

An online version of this manual may be found on the World Wide Web at:

A printable PostScript version of the manual may be found at:

(Please consider the option of not printing this manual. It is about 300 pages.) Further information on obtaining, installing, and using PDP++ may be found on the the official PDP++ home page, which is located at: http://www.cnbc.cmu.edu/PDP++/PDP++.html.

Some suggested readings for this course do not appear in any of these books. These readings will be made available to students in a number of ways. First, one copy of each assigned reading will be placed on reserve in the Hunt Library. Also, two copies of each suggested paper will be placed in a file cabinet in the Psychology Department lounge in folders labeled with the course number (85-419) and the corresponding lecture date. Students are expected to remove these papers from the file cabinet for no more than 20 minutes for the purpose of making personal copies.

The reading schedule (found below) includes an extensive list of optional readings for each topic. Many of these foundational papers have been collected in the following volume:

This book has also been placed on reserve in the library.

As mentioned previously, course exercises will require extensive use of the PDP++ software package. This computer simulation environment has been installed on AFS on the CMU Andrew system. Unix versions have been provided - both for Sun computers and for PCs running Linux. Students may make use of Andrew workstations in CMU Computer Clusters (e.g., Wean 5200 Corridor) to conduct their class work. Information on these computer labs may be found at: http://www.cmu.edu/acs/clustweb/clusters.html. On these CMU machines PDP++ may be found at /afs/andrew/hss/psy/pdp-IVlib. Alternatively, students may make use of any other appropriate machine at their disposal. Students intending to install PDP++ on their own machines should consult the official PDP++ home page. Students are expected to be sufficiently familiar with an appropriate computing environment so as to be able to edit and manage files and to operate within a window-based graphical environment. This course includes instruction on the use of PDP++ software tools, but it does not include instruction on the use of the underlying computer operating environment.

This class has an official page on the World Wide Web, located at:

This web page will be used to announce updates to the class schedule as well as to distribute certain class materials. It should be consulted often.

Expectations & Evaluations

Learning can be greatly facilitated by interactions between students, and these interactions are encouraged. Students should feel free to discuss lecture topics, readings, project ideas, and even excercise assignments with each other. (The main exception to this is the Take Home Examination which will be administered late in the course.) The actual completion of project work and exercises, however, should be conducted on an individual basis. All assignments submitted for evaluation should reflect the understanding and effort of the individual student. Also, helpful conversations with fellow students should be explicitly mentioned in submitted assignments.

PDP methods are best learned through active experimentation. Thus, a number of assignments have been prepared to aid students in their understanding of PDP concepts and techniques. Four collections of exercises will be specified - one for each of the first four sections of the class. Exercise assignment descriptions will be disseminated at the beginning of each course section, and assignment solutions will be due from the students on the first day of the following course section. These exercises will require substantial effort to complete, and they will involve the use of the PDP++ simulation software. Student solutions to these assignments will be evaluated by the instructor and the teaching assistant, and appropriate feedback will be given to the students. Following the fifth section of the course a take home examination will be administered. This, too, will provide an opportunity for feedback from the course teachers. Lastly, and most importantly, students will be expected to complete a computer simulation project which demonstrates an understanding of PDP concepts in some novel and interesting way. Feedback will be given on the student's project proposal (a description of the project to be done), on an oral report concerning the project, and on a final 10-15 page paper based on the project.

The due dates for all assignments are listed in the course schedule, below. All assignments are due at the beginning of class on the given dates. Late assignments which arrive in the instructor's mailbox (in 115 Mellon Institute) before 5:00 P.M. on the day after a due date will be evaluated and will receive 90% of the credit for the assignment. Late assignments which arrive in the instructor's mailbox before 5:00 P.M. on the following business day (e.g., a Thursday if the assignment was due Tuesday; a Monday if the the assignment was due Thursday) will be evaluated and will receive 80% credit. Assignments which are later than this will not be evaluated, and no credit will be given.

Those students who are to receive grades for this course will have their work assessed as follows:

Student performance will be evaluated in comparison to that of other students, both past and present. In performing this comparison, undergraduate and graduate students will be evaluated separately. Class participation will be monitored throughout the term, with a special emphasis given to participation during the final section of the course - during project status reports.

Schedule

Information Processing in PDP Networks

	McClelland, J. L., Rumelhart, D. E., & Hinton, G. E. (1986) The Appeal of Parallel Distributed Processing. PDP, Chapter 1.
	McCulloch, W. S. & Pitts, W. (1943) A logical calculus of the ideas immanent in nervous activity. Neurocomputing, Chapter 2.
	von Neumann, J. (1958) The Computer and the Brain. Neurocomputing, Chapter 7.

January 14 : The PDP Framework for Information Processing

*	Assignment #1 Specification Distributed
	Rumelhart, D. E., McClelland, J. L., & Hinton, G. E. (1986) A General Framework for Parallel Distributed Processing. PDP, Chapter 2.
	McClelland, J. L. & Rumelhart, D. E. (1988) Introduction. Handbook, Chapter 1.
	McClelland, J. L. & Rumelhart, D. E. (1988) Interactive Activation and Competition. Handbook, Chapter 2, pages 11-17.

January 19 : Schema Theory & Soft Constraint Satisfaction

	Rumelhart, D. E., Smolensky, P., McClelland, J. L., & Hinton, G. E. (1986) Schemata and Sequential Thought Processes in PDP Models. PDP, Chapter 14.
	McClelland, J. L. & Rumelhart, D. E. (1988) Constraint Satisfaction in PDP Systems. Handbook, Chapter 3, pages 49-54.
	Selfridge, O. G. (1958) Pandemonium: a paradigm for learning. Neurocomputing, Chapter 9.

January 21 : Introduction to PDP++

*	Class meets in Computer Cluster.
	Noelle, D. C. (1999) PDP++ Basics. (available from instructor)
	Dawson, C. K., O'Reilly, R. C., & McClelland, J. L. (1997) Introduction to the PDP++ Software. The PDP++ Software Users Manual, Chapter 2.
	Dawson, C. K., O'Reilly, R. C., & McClelland, J. L. (1997) Conceptual Overview. The PDP++ Software Users Manual, Chapter 4.
	Dawson, C. K., O'Reilly, R. C., & McClelland, J. L. (1997) Guide to the Graphical User Interface (GUI). The PDP++ Software Users Manual, Chapter 7.

January 26 : Stochastic Constraint Satisfaction

	Hinton, G. E. & Sejnowski, T. J. (1986) Learning and Relearning in Boltzmann Machines. PDP, Chapter 7, pages 282-290.
	Smolensky, P. (1986) Information Processing in Dynamical Systems: Foundations of Harmony Theory. PDP, Chapter 6.
	McClelland, J. L. & Rumelhart, D. E. (1988) Constraint Satisfaction in PDP Systems. Handbook, Chapter 3, pages 68-73, 75-81.
	Hopfield, J. J. (1982) Neural networks and physical systems with emergent collective computational abilities. Neurocomputing, Chapter 27.
	Geman, S. & Geman, D. (1984) Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images. Neurocomputing, Chapter 37.

January 28 : Psychological Modeling

	McClelland, J. L. & Elman, J. L. (1986) Interactive Processes in Speech Perception: The TRACE Model. PDP, Chapter 15.
	McClelland, J. L. & Rumelhart, D. E. (1981) An interactive activation model of context effects in letter perception: Part 1. An account of basic findings. Neurocomputing, Chapter 25.
	Movellan, J. R. & McClelland, J. L. (1995) Stochastic interactive processing, channel separability, and optimal perceptual inference: An examination of Morton's Law. Technical Report PDP.CNS.95.4, Carnegie Mellon University Department of Psychology, Pittsburgh, PA (available at "ftp://cnbc.cmu.edu:/pub/pdp.cns/pdp.cns.95.4.ps.Z").
	Ballard, D. H., Hinton, G. E., & Sejnowski, T. J. (1983) Parallel visual computation. Nature, 306, pages 21-26.

Associational Learning in PDP Networks

*	Assignment #1 Due
*	Assignment #2 Specification Distributed
	McClelland, J. L. & Rumelhart, D. E. (1988) Learning in PDP Models: The Pattern Associator. Handbook, Chapter 4, pages 83-89.
	Noelle, D. C. (1999) Hebbian & Delta Rule Learning in PDP++. (available from instructor)
	Hebb, D. O. (1949) The first stage of perception: growth of the assembly. Neurocomputing, Chapter 4.
	Rosenblatt, F. (1958) The perceptron: a probabilistic model for information storage and organization in the brain. Neurocomputing, Chapter 8.
	Widrow, B. & Hoff, M. E. (1960) Adaptive switching circuits. Neurocomputing, Chapter 10.
	Minsky, M. & Papert, S. (1969) Perceptrons. Neurocomputing, Chapter 13.

February 04 : Model Fitting, Generalization, Pattern Classification, and Pattern Association

	Jordan, M. I. (1986) An Introduction to Linear Algebra in Parallel Distributed Processing. PDP, Chapter 9.
	Stone, G. O. (1986) An Analysis of the Delta Rule and the Learning of Statistical Associations. PDP, Chapter 11.
	McClelland, J. L. & Rumelhart, D. E. (1988) Learning in PDP Models: The Pattern Associator. Handbook, Chapter 4, pages 89-99.

February 09 : Distributed Representations

	Hinton, G. E., McClelland, J. L., & Rumelhart, D. E. (1986) Distributed Representations. PDP, Chapter 3.
	Hinton, G. E. (1981) Implementing semantic networks in parallel hardware. In G. E. Hinton & J. A. Anderson (Eds.), Parallel Models of Associative Memory, pages 161-188. Lawrence Erlbaum: Hillsdale, NJ.
	van Gelder, T. (1991) What is the 'D' in 'PDP'? An Overview of the Concept of Distribution. Chapter 3 in S. Stich, D. Rumelhart, & W. Ramsey (Eds.), Philosophy and Connectionist Theory. Lawrence Erlbaum: Hillsdale, NJ.
	van Gelder, T. (1992) Defining "Distributed Representation". Connection Science, 4, pages 175-191.

February 11 : Psychological Modeling

	McClelland, J. L. & Rumelhart, D. E. (1986) A Distributed Model of Human Learning and Memory. PDP, Chapter 17.
	Anderson, J. A., Silverstein, J. W., Ritz, S. A., & Jones, R. S. (1977) Distinctive features, categorical perception, and probability learning: some applications of a neural model. Neurocomputing, Chapter 22.
	Gluck, M. A. & Bower, G. H. (1988) From conditioning to category learning: An adaptive network model. Journal of Experimental Psychology: General, 117, pages 227-247.
	Sutton, R. S. & Barto, A. G. (1981) Toward a modern theory of adaptive networks: Expectation and prediction. Psychological Review, 88, pages 135-170.

Learning Internal Representations in PDP Networks

*	Assignment #2 Due
*	Assignment #3 Specification Distributed
	Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986) Learning Internal Representations by Error Propagation. PDP, Chapter 8.
	Hinton, G. E. (1989) Connectionist learning procedures. Artificial Intelligence, 40, pages 185-234.
	Rumelhart, D. E., Durbin, R., Golden, R., & Chauvin, Y. (1995) Backpropagation: The Basic Theory. Chapter 1 in Y. Chauvin & D. E. Rumelhart (Eds.), Backpropagation: Theory, Architectures, and Applications. Lawrence Erlbaum: Hillsdale, NJ.
	Rumelhart, D. E., Durbin, R., Golden, R., & Chauvin, Y. (1996) Backpropagation: The Basic Theory. Chapter 15 in Smolensky, P., Mozer, M. C., & Rumelhart, D. E. (Eds.), Mathematical Perspectives on Neural Networks. Lawrence Erlbaum: Hillsdale, NJ.

February 18 : Using the Backpropagation Algorithm

	McClelland, J. L. & Rumelhart, D. E. (1988) Training Hidden Units: The Generalized Delta Rule. Handbook, Chapter 5, pages 121-137.
	Noelle, D. C. (1999) Backpropagation in PDP++. (available from instructor)
	Dawson, C. K., O'Reilly, R. C., & McClelland, J. L. (1997) Tutorial Introduction (using Bp). The PDP++ Software Users Manual, Chapter 5.
	Dawson, C. K., O'Reilly, R. C., & McClelland, J. L. (1997) Backpropagation. The PDP++ Software Users Manual, Chapter 15, pages 202-211.

February 23 : Generalization, Overfitting, Regularization, and Catastrophic Forgetting

	McCloskey, M. & Cohen, N. J. (1989) Catastrophic interference in connectionist networks: The sequential learning problem. The Psychology of Learning and Motivation, 24, pages 109-165. (available from instructor)
	Morgan, N. & Bourlard, H. (1990) Generalization and parameter estimation in feedforward nets: Some experiments. In D. S. Touretzky (Ed.), Advances in Neural Information Processing Systems 2, pages 630-637. Morgan Kaufmann: San Mateo, CA. (available from instructor)
	le Cun, Y., Denker, J. S., & Solla, S. A. (1990) Optimal brain damage. In D. S. Touretzky (Ed.), Advances in Neural Information Processing Systems 2, pages 598-605. Morgan Kaufmann: San Mateo, CA.
	Weigand, A. S., Rumelhart, D. E., & Huberman, B. A. (1991) Generalization by weight-elimination with application to forcasting. In R. P. Lippmann, J. E. Moody, & D. S. Touretzky (Eds.), Advances in Neural Information Processing Systems 3, pages 875-882. Morgan Kaufmann: San Mateo, CA.

February 25 : Analysis Techniques and Common Backpropagation Architectures

	Pollack, J. B. (1990) Recursive Distributed Representations. Artificial Intelligence, 46, pages 77-105. (available from instructor)
	Cottrell, G. W., Munro, P., & Zipser, D. (1988) Image compression by back propagation: An example of extensional programming. In N. E. Sharkey (Ed.), Advances in Cognitive Science, Volume 3. Ablex: Norwood, NJ.
	Jordan, M. I. & Rumelhart, D. E. (1992) Forward models: Supervised learning with a distal teacher. Cognitive Science, 16, pages 307-354.
	Williams, R. J. (1986) The Logic of Activation Functions. PDP, Chapter 10.

March 02 : Temporal Learning Using Backpropagation (Steve Gotts)

	McClelland, J. L. & Rumelhart, D. E. (1988) Training Hidden Units: The Generalized Delta Rule. Handbook, Chapter 5, pages 155-158.
	Dawson, C. K., O'Reilly, R. C., & McClelland, J. L. (1997) Backpropagation. The PDP++ Software Users Manual, Chapter 15, pages 212-221.
	Elman, J. L. (1990) Finding structure in time. Cognitive Science, 14, pages 179-211.
	Jordan, M. I. (1986) Attractor dynamics and parallelism in a connectionist sequential machine. In Proceedings of the 8th Annual Conference of the Cognitive Science Society, pages 531-546. Lawrence Erlbaum: Hillsdale, NJ.
	Williams, R. J. & Zipser, D. (1995) Gradient-Based Learning Algorithms for Recurrent Networks and Their Computational Complexity. Chapter 13 in Y. Chauvin & D. E. Rumelhart (Eds.), Backpropagation: Theory, Architectures, and Applications. Lawrence Erlbaum: Hillsdale, NJ.
	Pearlmutter, B. (1989) Learning state space trajectories in recurrent neural networks. Neural Computation, 1, pages 263-269.

March 04 : Designing Networks with PDP++

*	Class meets in Computer Cluster.
	Noelle, D. C. (1999) Building a Backpropagation Simulation in PDP++. (available from instructor)

March 09 : Psychological Modeling

	Elman, J. L. (1993) Learning and development in neural networks: The importance of starting small. Cognition, 48, pages 71-99. (available from instructor)
	Elman, J. L., Bates, E. A., Johnson, M. H., Karmiloff-Smith, A., Parisi, D., & Plunkett, K. (1996) Ontogenetic development: A connectionist synthesis. Chapter 3 in Rethinking Innateness: A Connectionist Perspective on Development. MIT Press: Cambridge, MA.
	Munakata, Y., McClelland, J. L., & Siegler, R. (1997) Rethinking infant knowledge: Toward an adaptive process account of successes and failures in object permanence tasks. Psychological Review, 104, pages 686-713.

Non-supervised Learning Methods in PDP Networks

*	Assignment #3 Due
*	Assignment #4 Specification Distributed
	Rumelhart, D. E. & Zipser, D. (1986) Feature Discovery by Competitive Learning. PDP, Chapter 5.
	Kohonen, T. (1982) Self-organized formation of topologically correct feature maps. Neurocomputing, Chapter 30.

March 16 : Contrastive Hebbian Learning and the Boltzmann Machine

	Hinton, G. E. & Sejnowski, T. J. (1986) Learning and Relearning in Boltzmann Machines. PDP, Chapter 7, pages 290-317.
	Dawson, C. K., O'Reilly, R. C., & McClelland, J. L. (1997) Constraint Satisfaction. The PDP++ Software Users Manual, Chapter 16.
	Ackley, D. H., Hinton, G. E., & Sejnowski, T. J. (1985) A learning algorithm for Boltzmann machines. Neurocomputing, Chapter 38.
	Peterson, C. & Anderson, J. R. (1987) A mean field theory learning algorithm for neural nets. Complex Systems, 1, pages 995-1019.
	Movellan, J. R. & McClelland, J. L. (1993) Learning continuous probability distributions with symmetric diffusion networks. Cognitive Science, 17, pages 463-496.

March 18 : Reinforcement Learning

*	Project Proposals Due
	Barto, A. G. (1994) Adaptive Critics and the Basal Ganglia. Chapter 11 in J. C. Houk, J. L. Davis, & D. G. Beiser (Eds.), Models of Information Processing in the Basal Ganglia. MIT Press: Cambridge, MA. (available from instructor)
	Barto, A. G., Sutton, R. S., & Anderson, C. W. (1983) Neuronlike adaptive elements that can solve difficult learning control problems. Neurocomputing, Chapter 32.
	Sutton, R. S. & Barto, A. G. (1998) Reinforcement Learning: An Introduction. MIT Press: Cambridge, MA.

March 23 : Spring Break

March 25 : Spring Break

Psychological Modeling

*	Assignment #4 Due
	Crick, F. H. C. & Asanuma, C. (1986) Certain Aspects of the Anatomy and Physiology of the Cerebral Cortex. PDP, Chapter 20.
	Sejnowski, T. J. (1986) Open Questions About Computation in Cerebral Cortex. PDP, Chapter 21.
	Zipser, D. (1986) Biologically Plausible Models of Place Recognition and Goal Location. PDP, Chapter 23.
	Munro, P. W. (1986) State-Dependent Factors Influencing Neural Plasticity: A Partial Account of the Critical Period. PDP, Chapter 24.
	O'Reilly, R. C. (1996) Biologically plausible error-driven learning using local activation differences: The Generalized recirculation algorithm. Neural Computation, 8, pages 895-938.

April 01 : Complementary Memory Systems (James McClelland)

	McClelland, J. L., McNaughton, B. L., and O'Reilly, R. C. (1995) Why there are complementary learning systems in the hippocampus and neocortex: Insights from the successes and failures of connectionist models of learning and memory. Psychological Review, 102, pages 419-457. (available from instructor)
	McClelland, J. L. & Rumelhart, D. E. (1986) Amnesia and Distributed Memory. PDP, Chapter 25.

April 06 : Inflectional Morphology

	Rumelhart, D. E. & McClelland, J. L. (1986) On Learning the Past Tenses of English Verbs. PDP, Chapter 18.
	Pinker, S. & Prince, A. (1988) On language and connectionism: Analysis of a parallel distributed processing model of language acquisition. Cognition, 28, pages 73-193.
	Plunkett, K. & Marchman, V. (1991) U-shaped learning and frequency effects in a multi-layered perceptron: Implications for child language acquisition. Cognition, 38, pages 43-102.

April 08 : Neuropsychology and Dyslexia (David Plaut)

	Plaut, D. C. (1995) Double dissociation without modularity: Evidence from connectionist neuropsychology. Journal of Clinical and Experimental Neuropsychology, 17, pages 291-321. (available from instructor)
	Sejnowski, T. J. & Rosenberg, C. R. (1986) NETtalk: a parallel network that learns to read aloud. Neurocomputing, Chapter 40.
	Seidenberg, M. S. & McClelland, J. L. (1989) A distributed developmental model of word recognition and naming. Psychological Review, 96, pages 523-568.
	Plaut, D. C., McClelland, J. L., Seidenberg, M. S., & Patterson, K. (1996) Understanding normal and impaired word reading: Computational principles in quasi-regular domains. Psychological Review, 103, pages 56-115.
	Farah, M. J. (1994) Neuropsychological inference with an interactive brain: A critique of the locality assumption. Behavioral and Brain Sciences, 17, pages 43-104.

April 13 : Lecture Canceled

*	Last chance to catch up on readings.

April 15 : Syntax and Semantics

	McClelland, J. L. & Kawamoto, A. H. (1986) Mechanisms of Sentence Processing: Assigning Roles to Constituents. PDP, Chapter 19.
	St.~John, M. F. & McClelland, J. L. (1990) Learning and Applying Contextual Constraints in Sentence Comprehension. Artificial Intelligence, 46, pages 217-257. (available from instructor)

April 20 : Architectures for Cognition

	Rumelhart, D. E. & McClelland, J. L. (1986) PDP Models and General Issues in Cognitive Science. PDP, Chapter 4.
	Norman, D. A. (1986) Reflections on Cognition and Parallel Distributed Processing. PDP, Chapter 26.
	Rumelhart, D. E. & McClelland, J. L. (1986) Future Directions. PDP, Last Chapter.
	Smolensky, P. (1986) Neural and Conceptual Interpretation of PDP Models. PDP, Chapter 22.
	Fodor, J. A. & Pylyshyn, Z. W. (1988) Connectionism and cognitive architecture: A critical analysis. Cognition, 28, pages 3-71.
	Smolensky, P. (1988) On the proper treatment of connectionism. Behavioral and Brain Sciences, 11, pages 1-74.

Project Status Reports

*	Take Home Examination Distributed

April 27 : Oral Reports

*	Take Home Examination Due

April 29 : Oral Reports

May 04 : Written Project Reports Due

*	Reports must be in the instructor's mailbox in 115 Mellon Institute by 10:20 A.M. on this day.