diff --git a/ch10/sem/README.md b/ch10/sem/README.md index df0ba58..331961f 100644 --- a/ch10/sem/README.md +++ b/ch10/sem/README.md @@ -2,7 +2,7 @@ Back to [All Sims](https://github.com/CompCogNeuro/sims) (also for general info # Introduction -This network is trained using Hebbian learning on paragraphs from an early draft of the *Computational Explorations* textbook, allowing it to learn about the overall statistics of when different words co-occur with other words, and thereby learning a surprisingly capable (though clearly imperfect) level of semantic knowlege about the topics covered in the textbook. This replicates the key results from the *Latent Semantic Analysis* research by [Landauer and Dumais (1997)](#references). This is an early version of the current GPT style models that process vast quantities of text and absorb an impressive amount of semantic knowledge from it. +This network is trained using Hebbian learning on paragraphs from an early draft of the *Computational Explorations* textbook, allowing it to learn about the overall statistics of when different words co-occur with other words, and thereby learning a capable (though clearly imperfect) level of semantic knowlege about the topics covered in the textbook. This replicates the key results from the *Latent Semantic Analysis* research by [Landauer and Dumais (1997)](#references). This is an early version of the current GPT style models that process vast quantities of text and absorb an impressive amount of semantic knowledge from it. The `Input` layer has one unit for each different word that appeared with a frequency of 5 or higher (and excluding purely function words like "the" etc) -- 1920 words in total. Each paragraph is presented as a single input pattern during training, with each word in the paragraph activated in the input (if the same word appears multiple times, it still just has the same unit activation). After each such paragraph, Hebbian learning between input and active `Hidden` layer neurons takes place, using our standard BCM-style learning mechanism, as explored earlier in the [v1rf](../../ch6/v1rf/README.md) and [self_org](../../ch4/self_org/README.md) projects. This model also includes recurrent lateral excitatory and inhibitory connections just like `v1rf`, which can induce a topological organization of neurons. Unlike in the visual model, the high-dimensional nature of semantics makes this somewhat harder to understand but nevertheless the same principles are likely at work.