85-419/719: Introduction to Parallel Distributed Processing

Announcements

• [Feb 14] Homework 2 is now due Thurs Feb 16. Also, instead of running commands like "loadExamples train.ex -s train" from the command-line interface, use the "Load Examples" button on the main control panel to load each example set (giving it a name in the pop-up window), and then click on "Training Set" (or "Testing Set") at the top of the control panel to select it as the training set (or testing set).
• [Feb 1] There is a new version of Lens-OSX.zip for Mac OS X in which the problem is fixed. Fixed versions for Windows and Linux will be available shortly.
• [Jan 24] Lens will crash (segfault) on a left-click to a unit in the Unit Display anytime after the palette has been changed. This is true on every platform (Linux, Windows, Mac OS X). The good news is that there's no real reason to left-click on a unit, so try not to (right-clicking to display weights is fine). I'll try to track this down and fix it if possible.

Assignments

• Homework 1 (feedback)
• Homework 2 (feedback)
• Homework 3 (feedback)
• Project proposal
• Take-home essay (due Tue Apr 17)

The following texts contain some of the course readings and may be useful as general references:

• PDP1: Rumelhart, D. E., McClelland, J. L., & the PDP research group (1986). Parallel distributed processing: Explorations in the microstructure of cognition. Volume 1: Foundations. Cambridge, MA: MIT Press.
• PDP2: McClelland, J. L., Rumelhart, D. E., & the PDP research group (1986). Parallel distributed processing: Explorations in the microstructure of cognition. Volume 2: Psychological and biological models. Cambridge, MA: MIT Press.
• PDP Handbook: McClelland, J. L. and Rumelhart, D. E. (1988). Explorations in parallel distributed processing: A handbook of models, programs, and exercises. Cambridge, MA: MIT Press.
• MPR: McLeod, P., Plunkett, K. and Rolls, E. T. (1998). Introduction to Connectionist Modelling of Cognitive Processes. Oxford: Oxford University Press.

• Windows
  Download the file Lens-windows.zip and use WinZip or a similar program to unzip the file. Inside the resulting folder will be a folder called "Lens". Read the file "README.rtf" in this directory for further instructions. Because of differences in the way Windows and Unix-based systems handled spaces, it will simply things if you put the Lens directory at the top level of the drive (i.e., C:\Lens).
• Mac OS X
  Download the file Lens-OSX.zip and double-click it to create an unzipped version. Inside will be a folder called "Lens". Read the text file README.txt in this folder for further instructions.
• Linux
  Download the file Lens-linux.tar.gz. Open a terminal and untar it with the command "tar xzf Lens-linux.tar.gz". This will create a directory called "Lens". Read the text file README in this directory for further instructions.

If you have any problems getting Lens running, contact the instructor. After installing Lens, you should look at the online manual at http://tedlab.mit.edu/~dr/Lens/, particular the instructions under "Running Lens" and the Tutorial Network under "Example Networks". The precompiled versions of Lens come with a offline (local) copy of the manual that can be accessed by pointing your web browser at Manual/index.html in the Lens directory.

Jan 17 (Tue): Overview and basic principles (slides)
[HOMEWORK 1 POSTED] [Install Lens (see Software section)]

• Rumelhart, D. E. (1989). The architecture of mind: A connectionist approach. In M. I. Posner (Ed.), Foundations of cognitive science (pp. 133-159). Cambridge, MA: MIT Press.

Jan 19 (Thu): Lens tutorial

Jan 24 (Tue): Constraint satisfaction

• McLeod, P., Plunkett, K. and Rolls, E. T. (1998). The attraction of parallel distributed processing for modelling cognition. MPR, Chapter 2.

Jan 26 (Thu): Schema theory

• Rumelhart, D. E., Smolensky, P., McClelland, J. L., & Hinton, G. E. (1986). Schemata and sequential thought processes in PDP models. PDP2, Chapter 14.

Jan 31 (Tue): Psychological implications (slides)
[HOMEWORK 1 DUE]

• (opt: McClelland, J. L. & Rumelhart, D. E. (1981). An interactive activation model of context effects in letter perception: Part 1. An account of basic findings. Psychological Review, 88, 375-407.)
• (opt: McClelland, J. L. & Elman, J. L. (1986). The TRACE model of speech perception. PDP2, Chapter 15.)
• (opt: Dell, G. S., Schwartz, M. F., Martin, N., Saffran, E. M., & Gagnon, D. A. (1997). Lexical access in normal and aphasic speakers. Psychological Review, 104, 801-838.)

Feb 2 (Thu): Hebb and Delta rules (slides)
[HOMEWORK 2 POSTED]

• McClelland, J. L., and Rumelhart, D. E. (1988). The pattern associator. PDP Handbook, Chapter 4 (pp. 83-96).
• (opt: McLeod, P., Plunkett, K. and Rolls, E. T. (1998). Pattern association. MPR, Chapter 3.)

Feb 7 (Tue): Hebb and Delta rules (continued)

Feb 9 (Thu): Distributed representations (slides)

• Hinton, G. E., McClelland, J. L., & Rumelhart, D. E. (1986). Distributed representations. PDP1, Chapter 3.
• (opt: Plaut, D. C. and McClelland, J. L. (2010). Locating object knowledge in the brain: A critique of Bowers's (2009) attempt to revive the grandmother cell hypothesis. Psychological Review, 117, 284-290.)

Feb 14 (Tue): Psychological implications (slides)
[HOMEWORK 2 DUE]

• McClelland, J. L. & Rumelhart, D. E. (1985). Distributed memory and the representation of general and specific information. Journal of Experimental Psychology: General, 114, 159-188.
• (opt: McLeod, P., Plunkett, K. and Rolls, E. T. (1998). Autoassociation. MPR, Chapter 4.)
• (opt: Farah, M. J. and McClelland, J. L. (1991). A computational model of semantic memory impairment: Modality specificity and emergent category specificity. Journal of Experimental Psychology: General, 120, 339-357.)

Feb 16 (Thu): Back-propagation (slides)
[HOMEWORK 3 POSTED]

• Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning internal representations by error propagation. PDP1, Chapter 8.
• (opt: Hinton, G. E. (1989). Connectionist learning procedures. Artificial Intelligence, 40, 185-234.)

Feb 21 (Tue): Temporal learning and recurrent networks (slides)

• Elman, J. L. (1990). Finding structure in time. Cognitive Science, 14, 179-211.
• (opt: Williams, R. J. & Zipser, D. (1995). Gradient-based learning algorithms for recurrent networks and their computational complexity. In Y. Chauvin & D. E. Rumelhart (Eds.), Back-propagation: Theory, architectures, and applications (pp. 433-486). Hillsdale, NJ: Erlbaum.)

Feb 23 (Thu): Generalization and overfitting (slides)

• (opt: Morgan, N. & Bourlard, H. (1990). Generalization and parameter estimation in feedforward nets: Some experiments. In D. S. Touretzky (Ed.), Advances in neural information processing systems 2. San Mateo: CA: Morgan Kaufmann, 630-637. )
• (opt: le Cun, Y., Denker, J. S., & Solla, S. A. (1990). Optimal brain damage. In D. S. Touretzky (Ed.), Advances in neural information processing systems 2. San Mateo: CA: Morgan Kaufmann, 598-605.)
• (opt: Weigand, A. S., Rumelhart, D. E., & Huberman, B. A. (1991). Generalization by weight-elimination with application to forcasting. In R. P. Lippmann, J. E. Moody, & D. S. Touretzky (Eds.), Advances in neural information processing systems 3. San Mateo: CA: Morgan Kaufmann, 875-882.)

Feb 28 (Tue): Contrastive Hebbian learning (slides)

• Hinton, G. E. & Sejnowski, T. J. (1986). Learning and relearning in Boltzmann Machines. PDP1, Chapter 7.

Mar 1 (Thu): Unsupervised learning (slides)

• Rumelhart, D. E. & Zipser, D. (1986). Feature discovery by competitive learning. PDP1, Chapter 5.
• (opt: Kohonen, T. (1990). The self-organizing map. Proceedings of the IEEE, 78, 1465-1480.)

Mar 6 (Tue): Reinforcement learning and forward models (slides)

• Barto, A. G. (1995). Reinforcement learning; Reinforcement learning in motor control. In M. A. Arbib (Ed.), The handbook of brain theory and neural networks (pp. 804-813). Cambridge, MA: MIT Press.
• (opt: Tesauro, G. (1995). Temporal difference learning and TD-Gammon. Communications of the ACM, 38, 58-68.)
• (opt: Jordan, M. I., and Rumelhart, D. E. (1992). Forward models: Supervised learning with a distal teacher. Cognitive Science, 16, 307-354.)

Mar 8 (Thu): Psychological implications
[PROJECT PROPOSAL DUE] [HOMEWORK 3 DUE]

• (opt: McClelland, J. L. (2001). Failures to learn and their remediation: A Hebbian account. In J. L. McClelland and R. S. Siegler (Eds.), Mechanisms of cognitive development: Behavioral and neural perspectives, (pp. 97-121). Mahwah, NJ: Lawrence Erlbaum Associates.)
• (opt: McCandliss, B. D., Fiez, J. A., Protoapas, A., Conway, M., McClelland, J. L. (2001). Success and failure in teaching the [r] [l] contrast to Japanese adults: Tests of a Hebbian model of plasticity and stabilization in spoken language perception. Cognitive, Affective, & Behavioral Neuroscience, 2, 89-108.)

Mar 13 (Tue): NO CLASS (Spring Break)

Mar 15 (Thu): NO CLASS (Spring Break)

---

Section 4: Applications

Mar 20 (Tue): Cognitive development (slides)

• Munakata, Y. and McClelland, J. L. (2003). Connectionist models of development. Developmental Science, 6, 413-429.
• (opt: Munakata, Y., McClelland, J. L., Johnson, M. H. & Siegler, R. (1997). Rethinking infant knowledge: Toward an adaptive process account of successes and failures in object permanence tasks. Psychological Review, 104, 686-713.)

Mar 22 (Thu): Memory and the hippocampus (slides)

• McClelland, J. L., McNaughton, B. L., and O'Reilly, R. C. (1995). Why there are complementary learning systems in the hippocampus and neocortex: Insights from the successes and failures of connectionist models of learning and memory. Psychological Review, 102, 419-457.

Mar 27 (Tue): High-level vision and attention [lecture by M. Behrmann] (slides)

• Mozer, M. C. and Sitton, M. (1998). Computational modeling of spatial attention. In H. Pashler (Ed.), Attention (pp. 341-393). Hove, England: Psychology Press/Erlbaum.
• (opt: Mozer, M. C. and Behrmann, M. (1990). On the interaction of selective attention and lexical knowledge: A connectionist account of neglect dyslexia. Journal of Cognitive Neuroscience, 2, 96-123.)
• (opt: Behrmann, M., Zemel, R. S. and Mozer, M. C. (1998). Object-based attention and occlusion: Evidence from normal participants and a computational model. Journal of Experimental Psychology: Human Perception and Performance, 24, 1101-1036.)

Mar 29 (Thu): Language: Morphology (slides)

• The past-tense debate. Trends in Cognitive Sciences, 2002, 6, 456-474. [Pinker, S. & Ullman, M T. The past and future of the past tense, 456-463; McClelland, J. M. & Patterson, K. 'Words or Rules' cannot exploit the regularity in exceptions: Reply to Pinker and Ullman, 464-465; McClelland, J. M. & Patterson, K. Rules or connections in past-tense inflections: What does the evidence rule out?, 465-472; Pinker, S. & Ullman, M T. Combination and structure, not gradedness, is the issue: Reply to McClelland and Patterson, 472-474.]
• (opt: Rumelhart, D. E. & McClelland, J. L. (1986). On learning the past tenses of English verbs. PDP2, Chapter 18.)

Apr 3 (Tue): Language: Word reading (slides)

• Plaut, D. C. (1999). Computational modeling of word reading, acquired dyslexia, and remediation. In R. Klein and P. A. McMullen (Eds.), Converging methods in reading and dyslexia (pp. 339-372). Cambridge, MA: MIT Press.
• (opt: Plaut, D. C., McClelland, J. L., Seidenberg, M. S., & Patterson, K. (1996). Understanding normal and impaired word reading: Computational principles in quasi-regular domains. Psychological Review, 103, 56-115.)
• (opt: Coltheart, M., Rastle, K., Perry, C. and Langdon, R. & Ziegler, J. (2001). DRC: A dual route cascaded model of visual word recognition and reading aloud. Psychological Review, 108, 204-256.)

Apr 5 (Thu): Language: Sentence processing (slides)

• McClelland, J. L., St. John, M., & Taraban, R. (1989). Sentence comprehension: A parallel distributed processing approach. Language and Cognitive Processes, 4, 287-335.
• (opt: Elman, J. L. (1993). Learning and development in neural networks: The importance of starting small. Cognition, 48, 71-99.)
• (opt: Rohde, D. L. T., and Plaut, D. C. (1999). Language acquisition in the absence of explicit negative evidence: How important is starting small? Cognition, 72, 67-109.)
• (opt: Frazier, L. (1987). Sentence processing: A tutorial review. In M. Coltheart (Ed.), Attention and performance XII: The psychology of reading (pp. 559-586). Hillsdale, NJ: Erlbaum.)

Apr 10 (Tue): Semantics [lecture by B. Armstrong] (slides)

• McClelland, J. L. and Rogers, T. T. (2003). The parallel distributed processing approach to semantic cognition. Nature Neuroscience Reviews, 4, 310-322.
• (opt: Plaut, D. C. (2002). Graded modality-specific specialization in semantics: A computational account of optic aphasia. Cognitive Neuropsychology, 19, 603-639.)

Apr 12 (Thu): Routine action (slides)
[TAKE-HOME ESSAY POSTED (covering Mar 20 to Apr 12)]

• Botvinick, M. and Plaut, D. C. (2004). Doing without schema hierarchies: A recurrent connectionist approach to normal and impaired routine sequential action. Psychological Review, 111, 395-429.
• (opt: Cooper, R. P., and Shallice, T. (2006). Hierarchical schemas and goals in the control of sequential behavior. Psychological Review, 113, 887-916.)
• (opt: Botvinick, M., and Plaut, D. C. (2006). Such stuff as habits are made on: A reply to Cooper and Shallice (2006). Psychological Review, 113, 917-928.)
• (opt: Cooper, R. P., and Shallice, T. (2006). Structured representations in the control of behavior cannot be so easily dismissed: A reply to Botvinick and Plaut (2006). Psychological Review, 113, 929-931.)

Apr 17 (Tue): Cognitive control and executive function
[TAKE-HOME ESSAY DUE]

• (opt: Rumelhart, D. E., Smolensky, P., McClelland, J. L., & Hinton, G. E. (1986). Schemata and sequential thought processes in PDP models. PDP2, Chapter 14, pages 38-57.)
• (opt: Cohen, J. D., Aston-Jones, G., and Gilzenrat, M. S. (2004). A system-level perspective on attention and cognitive control: Guided activation, adaptive gating, conflict monitoring, and exploitation versus exploration. In M. I. Posner (Ed.), Cognitive Neuroscience of Attention (pp. 71-90). New York: Guilford Press.)

Apr 19 (Thu): NO CLASS (Spring Carnival)

---

Section 5: Project Progress Reports