This network aims to understand the progressive differentiation of semantic representations by showing how the network's internal representations change as the model is exposed to information about the items in its training environment.Specifically, it reveals how development of conceptual knowledge begins with very broad distinctions, progressing to finer ones , resulting in the gradual elaboration of a conceptual hierarchy.
The network consists of two input layers called Item and Relation. Each unit in the Item layer corresponds to an individual item in the environment. Different relations are represented by individual units in the Relation layer and the various possible completions of three-element propositions are represented by individual units in the Attribute layer.When presented with a particular Item and Relation pair in the input, the network's task is to turn on the attribute units in the output that correspond to valid completions of the proposition. For example, when units corresponding to canary and can are activated in the input, the network must learn to activate the output units move, grow , fly and sing.
A range of +/- 0.9 was used to randomize initial weights. Training was done using the steepest descent algorithm with a batchSize of 1,learning rate 0.1 and without noise, weight decay or momentum.To ensure that the model's output responses relied entirely on input from other units in the network, we assigned to all units in the network , an untrainable bias of -2.
To see how internal representations develop in the network, we stopped training at different points during learning and stepped through the items, recording the states of the representation units for each.Using euclidean distance as a measure of similarity between representation patterns, we found that the network is first sensitive to broad semantic distinctions and only gradually picks up on more specific ones.
The network script and training file can be found on master and swan under the Examples directory. Alternatively, you can download it here