Representations of ‘what’ and ‘when’ in the brain, language models and neurally-inspired deep networks
OA Version
Citation
Abstract
Language comprehension depends critically on a memory for events as a function of time - one must remember preceding words, sentences, and paragraphs to construct a coherent narrative. In the brain, memory for time seems to be related to two kinds of populations, called temporal context cells and time cells. These populations can be concisely described with a mathematical model (scale-invariant temporal history or SITH). One population, resembling temporal context cells, maintain a real Laplace transform of what happened when. Another population, resembling time cells, inverts the activity of this population to generate scale-invariant sequential activity. This study explored the problem of language through the lens of this computational framework. In Chapter 2, I showed that having a range of timescales is helpful for language models. I studied word2vec, a language model that generates a vector space of words from a text corpus by looking at the context around each word, at a fixed scale. I examined these vector spaces generated by word2vec with different scales of context. Different linguistic relations were best encoded at a wide range of scales. The semantic space around words also changed dramatically as a function of context. In Chapter 3, I showed that working memory models with conjunctive representations of ‘What’ and ‘When’, with continuous timescales, showed similar behavior to neural data. Low-dimensional representations of these neuronal populations, obeying equations for SITH, exhibited stable subspaces and rotational dynamics as observed in many regions in the cortex. Here I also introduced a biological implementation of these equations using continuous attractor networks. In Chapter 4, I introduced a deep recurrent neural network (RNN) with conjunctive representations of What and When (SITH-RNN), which can learn a toy hierarchical language. Such networks exhibited temporal receptive windows across layers, as observed in human cognitive neuroscience studies of language. When compared to more generic RNNs, SITH-RNN, with its specific set of inductive priors, generalized better to time-rescaled inputs with orders-of-magnitude fewer weights. This suggests that SITH-RNN, and deep architectures with neurally-inspired inductive priors in general, can be viable candidates for language models, and for studying language processing in the brain.
Description
2025
License
Attribution 4.0 International