How the (Tensor-) Brain uses Embeddings and Embodiment to Encode Senses and Decode Symbols

The brain might consist of thousands or even millions of modules that need to coordinate and exchange information. Some information needs to be conveyed to higher processing layers, maybe even reaching conscious attention. In the TB, an integrative layer is introduced in the form of the representation layer, which is a high-dimensional basis from which modules can write and read.

The TB assumes that the activation (firing rate) of an ensemble of neurons implements a population code. The activation pattern of all ensembles in the representation layer reflects the cognitive brain state (CBS). Some ensembles might be specific and represent, e.g., the color “red”. Others might be rather unspecific latent factors. Experimental results indicate that the CBS exhibits a meaningful clustering structure [46].

Let $n$ be the number of ensembles. Then for $i=1,\ldots,n$

Here, $\gamma_{i}\in(0,1)$ is the (post-) activation of ensemble $i$, and $q_{i}\in\mathbb{R}$ is its pre-activation. Thus, $\gamma_{i}=1$ means that ensemble $i$ is firing with maximal intensity, and $\gamma_{i}=0$ means that ensemble $i$ is firing at some basal rate or not at all. As discussed further down $q_{i}$ receives input signals, and $\gamma_{i}$ is the output activation. The CBS is the activation vector $\vec{\gamma}$ of the representation layer. In the TB, one assumes that $\mathrm{sig}(q_{i})=(1+\exp(-q_{i}))^{-1}$ is the logistic function.

Baars and his coworkers have introduced the concept of a global workspace [1], which is the foundation of some theories on consciousness [1, 8]. The global workspace is a functional hub of broadcast and integration that allows information to be disseminated across modules. The global workspace is sometimes referred to as the “theater of the brain” or its “mental canvas”. As noted by [1], the global workspace could be related to the blackboard in a multi-agent system, where computational modules share information. The global workspace provides a communication and interaction hub, which is suitable for modules dealing with higher levels of abstraction, where direct module-to-module interactions might not be appropriate.

Let’s consider vision. Visual inputs contribute to the pre-activations of the ensembles in the global workspace through the visual paths, so visual input writes to the representation layer. In reverse, the CBS might feed back to the visual path in the form of embodiment, so for the visual backward path, the global workspace acts as input, and early visual processing layers get activated.

Abstracting from vision, the modeling assumption is that the global workspacereceives signals from many brain modules, and brain modules, in turn, might be affected by the state of the global workspace. In particular, the global workspace receives input from all sensory modalities the agent can be aware of, like vision, hearing, taste, smell, touch, proprioception, and pain. It has been proposed that all brain states that can reach consciousness, including inner feelings and emotional states, contribute to the global workspace and, in reverse, can be affected by its state. However, not all modules in the brain directly affect the global workspace and, thus, might not directly reach conscious attention. Naturally, perceptual input plays a major role. But also the pain after injury and emotions associated with a memory.

In the TB, it is assumed that the global workspace is implemented by the representation layer. The representation layer is related to the hidden layer in a recurrent neural network (RNN), with some important modifications, as discussed in the following.

The representation layer in the TB exhibits feedback connections, which introduce dependencies between the activations of the ensembles and an internal memory. The model assumes, with $i=1,\ldots,n$

The index $\tau$ is a counter of a discrete-time operation of the brain. All operations that occur for a fixed $\tau$ are related to the same concept or episodic index. Here, $f_{i}^{\textit{NN}}(\vec{\gamma}^{(\tau-1)})$ is the $i$-th output of a neural network with linear output units that describes the evolution neural network, and $\mathbf{v}^{(\tau)}$ represents some input vector, originating, e.g., from a visual scene. $\mathbf{g}(\mathbf{v}^{(\tau)})$ is a deep convolutional neural network with linear output units that maps visual input to the representation layer.

In general, a particular $q_{i}^{(\tau)}$ might only depend on a subset of ensembles, permitting modularity. The direct dependency on $q_{i}^{(\tau-1)}$ models a skip connection, as commonly used in generative AI [15], and realizes a self-memory. This implements a recurrency that encourages stable activation patterns in the CBS.

The update is equivalent to an RNN update with skip connections. The main difference is that $f_{i}^{\textit{NN}}(\cdot)$ is a neural network that contains at least one hidden layer, which provides universal modeling capability. This hidden layer would not be part of a standard RNN.

The evolution neural network enables a prediction of a future CBS. In addition, it enables relationship modeling in semantic memory and episodic memory.

The symbolic index layer is a module of high cognitive relevance that interacts with the representation layer. It receives input from the representation layer but also feeds back to it. In the brain, an index (or pointer) might be realized by an ensemble of neurons, as before. A difference is that the index layer can enforce a sample-take-all behavior, a variant of the well-known winner-take-all concept.

Each index represents a symbol. We discuss three types of indices: concept indices, predicate indices, and episodic indices. Thus, we propose that an episodic index is also a symbol. In perception, we also use the term label for a symbol.

The indices that belong to concepts are symbolic in nature. Concepts can, e.g., be entities, classes, attributes, locations, inner emotional states, actions, and decisions. Concepts permit the recognition of stable patterns in the CBS. Thus, a cluster analysis would have the brain detect a repeated pattern for concepts like Dog, a particular dog Sparky, and attributes such as Black and Happy. In perception, concept indices might label a scene or a visual area of interest, which we will refer to as the region of interest (ROI).

Assume the CBS is $\vec{\gamma}$. An index $k$ receives inputs from the representation layer with

where $Y=k$ means that the index layer is in state $k$, i.e., only ensemble $k$ is active, and all other indices are inactive. The vector $\mathbf{a}_{k}=(a_{1,k},\ldots,a_{n,k})^{\top}$ is the typically sparse embedding vector of concept $k$, and $a_{0,k}$ is a bias term. In the architecture, $a_{i,k}$ is the weight on the link from ensemble $i$ in the representation layer to ensemble $k$ in the index layer. The softmax is defined as

The domain dom could be the set of all entities in case one only normalizes about entities. Similarly, it could be the set of all predicates, the set of all classes, or the set of all colors. Each domain defines a random variable. The set members should be mutually exclusive and complete labels, although this is not enforced, e.g., if two colors are assigned to the same entity. See the discussion in Section 7.4.3.²²2For the pCBS, Equation 4 approximates $P(Y=k|\textit{context})=\mathbb{E}_{P(X_{1},...,X_{n}|\textit{context})}P(Y=k|X_{1},...,X_{n})$ [42].

Indices reflect repeated patterns in the CBS and are identified by their embeddings and maybe their location in the brain. In animals, they do not have associated names like Dog, Sparky, Black and Happy. In humans, a subset might be identified with concepts in natural language.

The second type of indices refers to predicates. Consider that the agent analyses a scene. The CBS first focuses on an ROI that, e.g., contains Sparky, a Dog in the scene, then on a second ROI, that, e.g., contains Jack, a Person, in the scene. Then, an ROI is formed that encompasses both previous ROIs. In the context of the two previous ROIs, the third one might be labeled by their relationship, e.g., looksAt. Thus, the brain does not only consider labels of ROIs, but also labels their relationships. Here, $\mathbf{a}_{k}$ is the embedding vector of predicate $k$.

The third type of indices concerns episodic indices and refers to time instances. In the TB, an episodic index is introduced for each (relevant) time instance. In its simplest form, the embedding vector of index $t$ is the vector of pre-activations of the representation layer at that instance, i.e., $\mathbf{a}_{t}\leftarrow\mathbf{q}^{(t)}$. If the representation layer is dominated by visual input from a scene, it will store the resulting pre-activations of the CBS, including, e.g., the emotional state at that instance. Later, we will discuss how $\mathbf{a}_{t}$ is actually optimized.

In a deterministic CBS interpretation, Equation 4 describes the output of the system. Assume that the labels for Sparky, Dog, Black, and Happy have substantial activation. The agent might select the label with the highest probability, e.g., Sparky. The agent might also consider other likely labels, such as Dog, Black, and Happy. There is no feedback to the representation layer in either case. The remaining part of the brain is not informed about likely labels. This is the standard RNN setting. Also, given the CBS, all labels are independent.

Assume that the labels for Sparky, Dog, and Black exhibit substantial activation. In the sampling mode, the TB samples from the distribution in Equation 4, and might generate the label Sparky. In general, if index $k$ is sampled, then it is assumed that concept $k$ is represented in the representation layer.

The index layer works in a sample-take-all mode: only one ensemble is firing at an instance, suppressing the firing of all other indices; in the example the index Sparky would be firing. Technically, after sampling, the index layer’s state is a one-hot-vector.

If index $Y=k$ is firing, it activates the global workspace in a top-down manner with

One can interpret $a_{i,k}$ as the synaptic weight connecting ensemble $k$ in the index layer, with ensemble $i$ in the representation layer. In the pCBS interpretation, we again obtain independent Bernoulli distributions. This update equation was derived in [42]. The factor $\alpha$, with $0\leq\alpha\leq 1$ controls how much of $\mathbf{q}$ is preserved in the update. With $\beta=1$ and $\alpha=0$, it is the Heisenberg approximation and with $\alpha=1$, it is the Heisenberg approximation with an “added prior”. With $\alpha=1$ and $\beta=0$ we get an RNN update.

Thus, as reflected in the last equation and in Equation 4, connections are bi-partite and symmetrical. Strict symmetry is implied by some theoretical considerations [42], but it is not a strict requirement and is likely to be violated biologically. Note that with $\alpha=1$ the previous pre-activation is not eliminated but added to the embedding vector as a skip connection.

The process can be repeated several times by repeatedly sampling from Equation 4 and updating Equation 5. With $\alpha>0$, each sample is always conditioned on the previous ones. Each sample is a particular interpretation of the CBS, in the context of what has been sampled before. For example, labels might represent the entity Sparky, the next one Dog, then Black, Happy, and so on. Finally, with $\alpha=1$,

Thus, the index layer is part of a labeling engine. We also clearly see the difference to a standard RNN, where the last equation would not contain the sum, i.e., there is no feedback from the samples to the state. In the TB, the representation layer is explicitly informed about all generated samples. Also, subsequently generated samples will consider all the labels generated before. The activation of an index is an internal brain measurement, which is feeding back to the brain! All modules in the brain can learn about what was detected. Equation 7 shows the additive combination of embedding vectors found in word2vec [28], and also in TransE[4].

Consider that there is no sampling, and the CBS is updated to consider all possible labels, weighted by their probabilities. In [46, 42] the equation

has been introduced. Thus, one obtains an update of the CBS, even without sampling but by rather considering label probabilities. Note that the equation is a form of an attention mechanism with a skip connection from generative AI. The CBS $\vec{\gamma}$ is the query vector, and the embedding vectors $\mathbf{a}_{k},\forall k$ are the key and the value vectors.

Attention does not result in a symbolic interpretation since no index is active in isolation; an advantage is that the attention mechanism is fast and parallel and holistically considers all interpretations. The attention equation relates indices to related indices leading to a sharing of statistical strength. In [42] it is shown that the attention mechanism can be derived from a Heisenberg scenario and not from a Bayes scenario: in the latter, only actual samples would influence the CBS.

Sampling commits the brain to a specific interpretation. For example, if the black thing in the scene is sampled to be Sparky, then the brain can add a lot of background information about Sparky. But if it is sampled to be a Puma, other background information is added and the agent’s reaction might be quite different. In sampling, both options are explored in different sampling rounds. Sampling explores the joint dependencies between labels. Sampling and top-down inference are the basis for semantic memory. See Figure 4.

The sampling of the indices is an inherent feature of the TB. What initiates the sampling? Sampling appears to have many things in common with the collapse of the wave function in quantum mechanics. Is the brain a decoherent quantum computer? These questions are addressed in [42].

Consider the last equation but without the skip connection, i.e., $\alpha=0$. One can define an autoencoder with an encoder, which maps the CBS to the index layer activation, i.e., using Equation 4. The decoder then maps the index layer softmax activation back to the representation layer, using attention Equation 7 (again without the skip connection). Thus, the index layer is the bottleneck layer in an autoencoder where one considers all symbolic encodings weighted with their probabilities.

In perception, one can also start with the visual input. In the encoder, the visual input $\mathbf{v}$ is mapped to the index layer, via the visual processing and function $\mathbf{g}(\mathbf{v})$. Then the CBS is mapped to the index distribution, as before. In the decoder, the activation of the index layer is mapped back to the representation layer and then back to the visual input via an approximation to the inverse of the visual processing layer, $\mathbf{\hat{v}}=\mathbf{g}^{\text{inv}}(\vec{\gamma})$. This top-down processing is a form of grounding or embodiment: not only the representation layer but all visual processing layers are informed, e.g., about what it means to consider Sparky in the index layer.

Brain circuits are generally bi-directional in the sense that if a module connects to another module, there are also connections in the reverse directions. In the TB, the connections between the representation layer and the index layer are bi-directional. Now, also the perceptual processing pipeline is assumed bi-directional. The inverse perceptual processing pipeline might not be as precise: as we all experience, a visual recall of the concept of Sparky is not as strong and clear as actually observing Sparky in a scene.

The pseudocode is described in Algorithm 7.1, Algorithm 7.1, and Algorithm 7.1. In Algorithm 7.1, $\alpha=1$ and $\beta=1$, which is the Heisenberg approximation with an added prior. With $\alpha=0$, it is the Heisenberg approximation (without an added prior). With $\beta=0$ we get an RNN.

[h]Evolution Neural Network

[h]Input and Attention

[h]Encoding (bottom-up) and Decoding (top-down)

According to Tulving, who introduced the term [47]: Episodic memory stores information about general and personal events and concerns information we ”remember“. It is about past observations. In the context of vision, it would be the reconstruction and the decoding of a past scene.

In the cognitive literature, semantic memory is defined as representing facts that we know but don’t remember where we learned them. It is about statements that are true or false or probabilistic in nature. The logical or probabilistic statements are either time-invariant or are only slowly or rarely changing in time.

Consider that a friend describes a dog that you don’t know. Your episodic memory might provide you with information about previous encounters with dogs, but we seem to have an understanding of the concept Dog in semantic memory, even without the recall of specific episodic memories.

Triples form a knowledge graph (KG), where concepts are nodes and predicates link labels. Some triples would be certain (in the probabilistic sense described before), and some would be labeled with conditional probabilities. The episodic memory can be related to a temporal knowledge graph (tKG). Generative models for embedded knowledge graphs were also studied by [23].

As discussed, symbolic and subsymbolic representations interact tightly. One cannot think, e.g., of Sparky without activating a subsymbolic embedding of Sparky and an embodiment process that might activate earlier visual and other processing layers in a top-down process. Similarly, anything presented in the representation layer is constantly decoded into labels, e.g., Sparky, in a bottom-up process.

The processing of different brain modules can occur in parallel, particularly in the processing of perceptual paths. However, the symbolic decoding itself is a serial process. This is discussed in Dehaine  [8] as the serial bottleneck in consciousness. The importance of sampling is also recognized in [8] in the context of conscious perception. For example, the author states that “… consciousness is a slow sampler”.

The sequential nature of symbol generation in the TB might have been a prerequisite to the generation of natural language. The natural “outer” language might be based on the triple-oriented “inner” language. Individuals, to a large degree, share common concepts (of the “inner” language), even when they do not share a common language (“outer language”)! Embeddings might be specific to a “personal brain”, although recent research indicates that they as well might be shared to some degree [13, 38]. How eloquent is natural language? Part of natural language is quite fact oriented: Some managers might request, “Just give me the facts”. It has been speculated that even more sophisticated language consists of facts, glued together by fillers. This might be true for some texts generated by large language models.

Natural (outer) language can also be an input. It feeds to the representation layer and a sentence or a paragraph is like an episode to be encoded as an inner language. As with any episodic memory that has an embedding vector, we can recall a past sentence and its meaning. An input sentence thus becomes part of the memory systems.

Consider simple associations between indices and embeddings: An episodic memory embedding can be formed by simply storing the CBS at perception. Similarly, the embedding vector for a concept could be the simple average of all CBSs, when the concept was present. Indeed, this might have been the starting point in biological evolution. The TB actually uses self-supervised learning, which leads to better performance.

No external agent provides training data, i.e., symbolic labels. In the TB, perception itself produces labels by stochastic sampling and the brain then treats the generated labels as actual labels and trains on them. This is a type of bootstrap learning where model predictions are used as targets for learning; see the bootstrap Widrow-Hoff rule [16] and learning with pseudo labels [22]. The self-generated labels occur in different cost functions. [3] discusses different principled forms of self-supervised adaptation in deep learning.

Assume a scene $\mathbf{v}^{\textit{scene}}$ has been observed and then the focus was on an ROI with $\mathbf{v}^{\textit{ROI}}$. To simplify we do not include attention. The labels $s=\textit{Sparky}$ and $k=\textit{Dog}$ are generated.

For perception, the contribution to the log-likelihood (negative cost function) is $\log P(s|\mathbf{v}^{\textit{ROI}},\mathbf{v}^{\textit{scene}})+\log P(k|s,\mathbf{v}^{\textit{ROI}},\mathbf{v}^{\textit{scene}})$ which becomes

with $\mathbf{h}_{1}=\mathbf{g}(\mathbf{v}^{ROI})+\mathbf{g}(\mathbf{v}^{scene})+\mathbf{f}^{\textit{NN}}(\mathbf{g}(\mathbf{v}^{scene}))$. Let’s assume that it is the first time the first label $s=\textit{Sparky}$ has been observed. The first term in the cost function will attempt to align the embedding $\mathbf{a}_{s}$ with the visual input. Then, the second cost term will attempt to align $\mathbf{a}_{s}$ with the embedding of Dog. Thus the “dogginess” will be integrated in Sparky’s embedding.

For the episodic memory for time $t$, the contribution to the log-likelihood (negative cost function) is $\log P(s|t)+\log P(k|s,t)$ which becomes

with $\mathbf{h}_{2}=\mathbf{a}_{t}+\mathbf{f}^{\textit{NN}}(\mathbf{a}_{t})$.⁶⁶6If the embedding vector of the ROI $\mathbf{a}_{t_{roi}}$, can be included $\mathbf{h}_{2}=\mathbf{a}_{t_{roi}}+\mathbf{a}_{t}+\mathbf{f}^{\textit{NN}}(\mathbf{a}_{t})$.

For the semantic memory for entity $s$, the contribution to the log-likelihood (negative cost function) is

with $\mathbf{h}_{3}=\mathbf{\bar{a}}+\mathbf{f}^{\textit{NN}}(\mathbf{\bar{a}})$. This last cost term will also try to align the embedding of Sparky with the embedding of the concept Dog. When semantic memory recalls Sparky, it is likely that Dog will be sampled next. The embedding of the index Sparky will integrate all labels that were generated in context with the index Sparky.

We can also derive cost terms for triples with predicates other than type, like (Sparky, looksAt, Jack).⁷⁷7We can see that the equations are not commutative: First sampling $s$ and then $k$ leads to a different cost than if we first sample $k$ and then $s$.

A consolidated episodic memory recall is more consistent with the agent’s worldview: Past memories are made a bit more to fit the agent’s expectation. Memories that fit an agent’s understanding are much better to recall than, e.g., scenes that occurred in a foreign or alien place, where situations might not fit the agent’s expectations. We can also only verbally report about past scenes effectively if they fit the agent’s world understanding [6, 18].

In Section 6.6 we discussed how scene input can be used to train an autoencoder, which is instrumental to realize embodiment. Cost functions derived from a perceptual autoencoder might regularize and stabilize learning.

As discussed, the recall of an episodic or semantic memory involves the complex decoding of the embedding vectors. In this process samples are generated. The brain might decide to train on these samples. This provides an explanation to the well-known phenomena, that a memory trace can change when retrieved; in fact it can even be manipulated as shown by the work of Loftus [24, 25].

For existing indices, the predicted labels can be used for adapting the embeddings of those indices. But the brain also needs to form new indices. A new perceptual experience is represented by a newly formed episodic index and its learned embedding. At a slower rate, new entity indices are introduced, and even at a slower rate, new indices for classes and attributes are required. The neurobiological aspects of forming new indices are extensively discussed in [46].

Assume that the agent is occupied with planning a friend’s visit for the evening while driving a car. Suddenly, and maybe just for a few seconds, the traffic situation requires full attention. Afterward, the agent continues the evening planning. How can the brain do that?

A leading hypothesis suggests that it only appears that the mind can think of several things at the same time. At the conscious level, we rapidly switch between mental states. One might consider that cognitive attention is another serial bottleneck in the brain! An extreme case is multitasking, where we seem to be able to do two things at a time, like driving a car and participating in a conversation. Cognitive multitasking is an illusion, and the brain does multiplexing, rather than multitasking [29].

The TB would support the concept of multitasking by multiplexing. Returning to the example: The TB approach suggests that the evening-planning CBS is stored as the embedding vector of an episodic index. This CBS is then restored by activating this novel episodic index after the traffic situation is resolved.

Multitasking by multiplexing might have a capacity limit. The capacity of working memory has been estimated to be seven or less [30, 10, 5], which would mean that the mind can switch between seven or fewer mental states. In the context of the TB, cognitive control might consider seven or fewer episodic or concept indices and their embeddings in problem-solving.

A specific example, of course, is the usage of episodic and semantic memory during perception. In case of time permits, the mind can activate episodic or semantic memory to supplement the current perceptional activity with background information, e.g., to provide state information (recent episodic memory) and for decision guidance (remote episodic memory). See Figure 3. The multitasking between perception and memory can easily be achieved by the TB approach but might be quite difficult to achieve with other technical models. The samples generated by semantic memory can be considered as samples generated by the prior; samples generated from perception or episodic memory can be considered as data. In [46] this was related to Dirichlet fusion.

There seem to be a number of mental tasks that require a form of cognitive control. Multitasking is an obvious example: How does the brain decide to focus on which mental or physical task?

Another example is semantic memory: assume the agent’s thought is with Sparky. Should the mind stay with Sparky or digress to Jack (his owner), who just got married to Mary, who lives in Munich. Munich has the EnglishGarden with several nice BeerGardens. That reminds the agent to buy some beer on the way home.⁸⁸8This type of digression is infamous for the character J.D. in the sitcom ”Scrubs“. A third example is the serial selection of ROIs in an image [34].

Cognitive control seems to be one of the central faculties of consciousness and conscious decision making. Who is in charge? Technically, the module responsible for cognitive control might be optimized by a form of reinforcement learning.

By design, the TB is multimodal: We proposed that many modules communicate with the representation layer, in particular, different sensor modalities. A given modality might only contribute to a subset of the dimensions of the representation layer. Figure 4 illustrates that some dimensions might be mainly visual, others auditory, tactile, or abstract.

Embedding vectors might be sparse, as well. For example, the index Black will mostly interact with the visual dimensions of the representation layer, whereas the index Loud or Barking with auditory dimensions. However, one might also talk about a loud image, so concepts sometimes generalize across dimensions.

Consider the entity Sparky. The visual dimensions in the representation layer provide the mind with information about how he looks (e.g., his color, his appearance), his auditory dimensions (how his bark sounds and other sounds he makes), how it feels like touching him, and maybe some abstract dimensions. Thus the top-down inference is quite rich. See Figure 4.

Consider the example in Figure 4. The sensory input might be visual and auditory. So only the parts of the representation layer that respond to vision and sound are active. This is reflected in the bottom-up inference. If Sparky is in the scene, his index is sampled. Top-down inference is global in the sense that all regions in the representation layer which link to Sparky are activated, in the example auditory, vision, tactile and abstract dimensions. The firing of Sparky supports the firing of Dog. Only when Dog fires, then Mammal also fires, since mammal is not visual and there are no common abstract dimensions with Sparky directly but only with Dog. This activation of Mammal via the activation of Dog demonstrates chaining in symbolic embedded reasoning. Thus, the brain can conclude that Sparky is a Mammal, even without an adaptation of Sparky’s embedding.

Consider the same situation as before but assume that the embedding of Sparky has been adapted. This means that the embedding of the label Mammal will be integrated into the embedding of Sparky. Then the activation of the index for Sparky might already lead to a sampling of Mammal directly, without an intermediate sampling of Dog. It is the type of reasoning applied in recommendation systems [21] and is related to the process of materialization in logic and database theory.

Assume the agent has stored a scene from EnglishGarden at time $t_{1}$, with visual labels, e.g., Sunny, and a scene from Marienplatz at time $t_{2}$, with visual labels, e.g., Rainy. The episodic embeddings for both and also the generated visual labels might show no similarity.

A location module of the brain might tell the mind at the time of perception that both scenes happened in Munich. Vision and location-generated labels can now be shared. When the episodic embeddings are adapted they will integrate the embedding of Munich. For episodic index $t_{1}$, the embedding vector is adapted to also generate the label Munich; the same for episodic index $t_{2}$.

The two engrams become similar in terms of their embeddings. An episodic recall of one of them might activate the other one. Similarly, a semantic recall of the concept Munich might now activate both episodic memories.

We define symbolic reasoning as reasoning that does not explicitly exploit the embeddings of the symbolic indices. In Section 7.4.4, we discussed the symbolic decoding of semantic memory. It is solely based on the symbolic indices and does not exploit the embeddings. Symbolic decoding can represent binary statistics (e.g., between Dog and Black) and can do chaining: Assuming that Sparky was sampled in the scene, and then Dog, and this triggered the sampling of Mammal, then the mind can conclude that Sparky is a Mammal. This chaining operation can also be materialized by introducing a link from Sparky to Mammal directly. It is conceivable that more complex reasoning can be performed in the indices, e.g., based on (relational) Bayes nets.

In Section 7.4.2 we discussed that a sample can nonlinearly depend of previous samples, but that the dependency is not fully general. For this we need the evolution neural network. Assume that the object entity is identical to the subject entity, e.g., Sparky. Then, we can perform two samples in the entity’s role as a subject, e.g., Dog and Black, and one as an object, e.g., Cute. The latter one, in general, will have a nonlinear dependency on the outcomes of the first two, since in the transition from subject to object, the evolution neural network is applied. Thus, the brain can exploit the evolution operation to perform general mappings and achieve universal approximation ability.

What is the semantics of a concept like Sparky? The embedding of Sparky is its signature, or “DNA”. The decoding of the “DNA” involves the activation of the representation layer with Sparky’s embedding. The embodiment informs substantial parts of the brain about Sparky’s sensory and other properties: how does Sparky look like, how does he sound, how does it feel to pad him. Then semantic memory tells the brain about attributes of Sparky, e.g., that he is Black, and by the decoding of Sparky’s relationships, e.g., about his social network (owned by Jack, loved by Mary). All this is embedded and embodied as well. Then, past episodic memories involving Sparky might be recalled. Thus, substantial parts of the brain’s sensory processing modules, as well as many memories “decode” Sparky’s “DNA”. It is an adaptable signature that can only be decoded in the context of an individual’s brain. Thus, thinking intensely of Sparky triggers a firework of activities in the brain. A similar firework, of course, happens also during perception and episodic memory recall.

Intelligence is often associated with working memory and cognitive control. In Section 9, we indicated how working memory might control a few symbolic indices to contemplate a decision. Although perception and memory might be the basis, model-based reasoning might also play a significant role. We support the idea that human intelligence has many facets and relies on several functional modules.