Dog Chases Boy Example

Semantic Pointer Architecture
1

Hello,

I am entirely new to nengo and SpikNNs. I was reading through the ‘How to Build a Brain’ book and I’m trying to replicate the Structured Manipulation (in Section 4.4) where it has the grammatical syntax and the semantic pointer P:

“The dog chases the boy”

P = verb * chase + agent * dog + theme * boy

In the example, it is manipulating the semantic pointer, P, without explicitly decoding each Pointer (verb/agent/theme) using the following semantic pointer:

T = ~agent * theme + ~chase * hug + ~theme * agent

such that the manipulation occurs:

T * P = verb * hug + agent * boy + theme * dog + noise

“The boy hugs the dog”

Notation-wise, to my understanding, * is used for circular convolution (HRR), and ~ is used for the inverse (book notation uses ’ ).

However, when implementing this in a similar manner to the examples, the verb (hug) appears to be incorrect when decoding the result. The code I used is below:

import nengo
import nengo.spa as spa
from nengo.spa import Vocabulary
import numpy as np
    
dim = 128
rng = np.random.RandomState(0)
vocab = Vocabulary(dimensions=dim, rng=rng)

# add semantic pointers to the vocabulary
BOY = vocab.parse('BOY')
DOG = vocab.parse('DOG')
CHASE = vocab.parse('CHASE')
HUG = vocab.parse('HUG')

model = spa.SPA(label='Dog chases boy', vocabs=[vocab])
with model:
    model.P = spa.State(dimensions=dim, label='P')
    model.T = spa.State(dimensions=dim, label='T')
    model.Z = spa.State(dimensions=dim, label='Z')
    
    model.OutAgent = spa.State(dimensions=dim, label='OutAgent')
    model.OutVerb = spa.State(dimensions=dim, label='OutVerb')
    model.OutTheme = spa.State(dimensions=dim, label='OutTheme')
    
    model.agent = spa.State(dimensions=dim, label='agent')
    model.verb = spa.State(dimensions=dim, label='verb')
    model.theme = spa.State(dimensions=dim, label='theme')
    
    actions = spa.Actions(
        'P = agent + verb + theme',
        'T = ~agent * theme + ~CHASE * HUG + ~theme * agent',
        'Z = T * P',
        'OutAgent = Z * ~agent',
        'OutVerb = Z * ~verb',
        'OutTheme = Z * ~theme',
        )
        
    model.cortical = spa.Cortical(actions)
    
    model.input = spa.Input(agent='DOG', verb='CHASE', theme='BOY')

I have several variations of this code and they end up with similar results. Is there something that I am missing or misunderstanding?

2

You define P as

where I assume all of verb, chase, agent, dog, theme, boy are Semantic Pointers (we usually would write these with a capital letter or all uppercase to differentiate specific Semantic Pointers from variables or neural ensembles that can be assigned different Semantic Pointers).
but in your model you write

P = agent + verb + theme

where you use agent, verb, theme to refer to neural ensembles and thus act more like variables that are assigned DOG, CHASE, BOY. But you are not binding these Semantic Pointers to the Semantic Pointers denoting the role in the sentence.

3

Hmm, looks like I have variables/ensembles/states and the semantic pointers mixed up? So then adding AGENT, VERB, and THEME to the vocabulary of semantic pointers so that I am able to bind them, I can write something like this:

import nengo
import nengo.spa as spa
from nengo.spa import Vocabulary
import numpy as np

dim = 64
rng = np.random.RandomState(0)
vocab = Vocabulary(dimensions=dim, rng=rng)

BOY = vocab.parse('BOY')
DOG = vocab.parse('DOG')
CHASE = vocab.parse('CHASE')
HUG = vocab.parse('HUG')

AGENT = vocab.parse('AGENT')
VERB = vocab.parse('VERB')
THEME = vocab.parse('THEME')

model = spa.SPA(label='Dog chases boy', vocabs=[vocab])
with model:
    model.p = spa.State(dimensions=dim, label='p')
    model.t = spa.State(dimensions=dim, label='t')
    model.z = spa.State(dimensions=dim, label='z')
    
    model.out_agent = spa.State(dimensions=dim, label='out_agent')
    model.out_verb = spa.State(dimensions=dim, label='out_verb')
    model.out_theme = spa.State(dimensions=dim, label='out_theme')
    
    actions = spa.Actions(
        'p = VERB * CHASE + AGENT * DOG + THEME * BOY',
        't = ~AGENT * THEME + ~CHASE * HUG + ~THEME * AGENT',
        'z = p * t',
        'out_agent = z * ~AGENT',
        'out_verb = z * ~VERB',
        'out_theme = z * ~THEME',
        )
        
    model.cortical = spa.Cortical(actions)

I’ve removed the input since it doesn’t seem applicable and reduced the dimensions to have it run faster.
Here should p, t, and z be a semantic pointer as well (+ added to the vocab)? Or rather that does not matter since they’re simply holding the binded result as variables?

Also in the code above, the verb is still not the answer I expected (‘HUG’).

4

No, p, t, and z are variables (so to say) that hold Semantic Pointers (or combinations there of). They can change depending on the input, whereas the Semantic Pointers in your vocabulary stay constant.

I think, the model is actually working as intended, but you need to increase the dimensions sufficiently. I was trying it with d = 512 and the expected words were most similar to the outputs. However, the absolute similarity isn’t very high and other words are still close in similarity. That is because the way used to transform the sentence accumulates a lot of noise in the vector. You can see this by writing down the math by hand which gives you:
$$
z = A \circledast \mathit{BOY} + V \circledast \mathit{HUG} + T \circledast \mathit{DOG} + \left[V \circledast \mathit{CHASE} \circledast (\sim A \circledast T + \sim T \circledast A) + A \circledast \mathit{DOG} \circledast(\sim\mathit{CHASE} \circledast \mathit{HUG} + \sim T \circledast A) + T \circledast \mathit{BOY} \circledast (\sim A \circledast T + \sim\mathit{CHASE} \circledast \mathit{HUG})\right]
$$
Everything in the square brackets is noise. For brevity $A = \mathit{AGENT}, V = \mathit{VERB}, T = \mathit{THEME}$.