Executing multiple actions from a single condition

#1

Hey folks, as my first post-tutorial project I’m trying to build a system that will input phrases one word at a time, and then answer questions. So when I input (one word at a time)

JOHN KICK BALL PERIOD

JOHN goes into the the subject state (spa.State with feedback=1), KICK goes into verb state, and BALL goes into object state. PERIOD triggers putting the whole phrase into a memory state.

So I have (in part):

model.vision = spa.State(D)

model.verb = spa.State(D, feedback=1)
model.subject = spa.State(D, feedback=1)
model.object = spa.State(D, feedback=1)  

model.phrase = spa.State(D)
model.randompointer = spa.State(D)  

bg_actions = spa.Actions(
    'dot(vision, BUILT+ATE+KICK) --> verb=vision',
    'dot(vision, JOHN+SUSAN+MIKE) --> subject=vision',
    'dot(vision, HOUSE+SUSHI+BALL) --> object=vision',

    'dot(vision, PERIOD) --> memory=phrase*randompointer',
    )

cortical_actions = spa.Actions(
‘phrase = subject * WHO + verb * WHAT + object * TO_WHAT’,
)
model.cortical = spa.Cortical(cortical_actions)

What I would like to do is have PERIOD trigger multiple actions, namely

  1. put phrase into memory
  2. clear subject, verb, and object
  3. generate a new randompointer

So what is the best way to have one input trigger multiple actions? Also, what is the best way to generate a random pointer?

#2

To trigger multiple actions separate them by a comma. In your case, something like

'dot(vision, PERIOD) --> memory=phrase, reset=TRUE'

However, do you know how to reset a memory via inhibition? Would you like me to link you to an example?

Finally, could you elaborate on what you mean by a “random” pointer and what you plan to do with it?

#3

Hey Sean (I’m guessing),

  1. Comma, hunh. Who coulda guessed?
  2. Link to example of inhibition? Yes, please.
  3. Re: random pointer, must I explain my madness? Okay then:

The plan is to have multiple phrases in memory and be able to make random queries, so …

JOHN KICKED BALL PERIOD
SUSAN ATE SUSHI PERIOD
MIKE BUILT HOUSE PERIOD
WHO ATE SUSHI QUESTION

So the cortical function keeps “phrase” loaded with the most recent “subjectWHO + verbWHAT + object*TO_WHAT”

When PERIOD comes in, I want to tie that phrase to a random pointer so that I can retrieve that whole phrase later, thus “memory = phrase*randompointer”.

So when

WHO ATE SUSHI QUESTION

comes in, phrase will look like “WHOWHO + ATEWHAT + SUSHI*WHAT_TO”. Alternatively, the subject(WHO) part of the phrase will contain the subject from the last phrase, in this case, MIKE. That’s why I would prefer to reset the subject after PERIOD.

In any case, when QUESTION comes in, I try to get the random pointer most similar to the phrase, thus
phrasepointer = memory*~phrase

As cortical actions I have
answerphrase = memory*~phrasepointer
answer = answerphrase*~question

So has this been done already? I’m sure there is a better way. I’m all ears … or eyes … as it were.

*

#4

Uh, apparently there isn’t an example of inhibiting a memory, so give me a few days to wrangle that up.

In terms of saving multiple phrases in memory, this is somewhat similar to various working memory models created. The fact you have a series of pointers to put into memory resembles @xchoo’s working memory (which is just a basic SPA memory, but where each item is assigned a unitary pointer, which I’ll also need an example showing…) and/or @pblouw’s stack memory.

However, if you want to see models of the question answering task, this was actually @pblouw’s thesis.

#5

At the risk of overwhelming you with new information, the original Nengo SPA syntax isn’t very intuitive, especially since it’s all encoded in strings. If you like, you can switch to using the newly released nengo-spa Python package which might be nicer to you.

#6

Sean, thanks for the nengo_spa link. Unfortunately, it doesn’t seem to be as nengo.gui-friendly. I can’t figure out how to get a useful output.

#7

A new Nengo GUI version was just released today that should work with nengo_spa.

#8

Jan, I’m afraid I’m not particularly tech-savvy, so please bear with me. I’m using a MacBook Pro. I tried this from a command line:

Jamess-MBP:~ jamessmith$ pip install nengo_spa

and got this:

Requirement already satisfied: nengo_spa in ./anaconda/lib/python3.6/site-packages (0.4.1)

Requirement already satisfied: numpy>=1.8 in ./anaconda/lib/python3.6/site-packages (from nengo_spa) (1.11.3)

Requirement already satisfied: nengo>=2.7 in ./anaconda/lib/python3.6/site-packages (from nengo_spa) (2.7.0)

pillow 4.0.0 requires olefile, which is not installed.

When I fire up the GUI, it’s the same as before. Right clicking on a spa.State icon gives me only one option for output, which is “Output Value”, which is not very helpful.

Any ideas would be greatly appreciated.

James

#9

Try pip install --upgrade nengo-spa.

#10

I mean try pip install --upgrade nengo-gui because that’s the package you actually want to upgrade, but upgrading nengo-spa doesn’t hurt either.

#11

Awesome, this works, thanks … except … the icons don’t pick up the names like they used to, so

visionstate = spa.State(myvocab)

pointerstate = spa.State(myvocab)

Produces two icons, both labeled “State”.

Ideas?

James

#12

It works for me … and I’d need more detail to figure out why it doesn’t work for you.

#13

I’ll play with it over the weekend. Thanks for your help.

James

#14

I figured it out (problem with icons not picking up labels). In the old version (nengo.spa) you didn’t need to specify the “label” attribute because the variable name would get picked up. In the new version (nengo_spa) that doesn’t happen, so you need to specify each “label” attribute. Ah well.

#15

It should still work in the new version without explicitly specifying the label. Can you post your code?

#16

Here’s enough code to generate the issue:

#
#  Setup the environment
#

import nengo
import nengo_spa as spa
import numpy as np

dim = 64

with spa.Network() as model:

myvocab = spa.Vocabulary(dimensions=dim)
words = ['JOHN','SUSAN','MIKE',
        'BALL','SUSHI','HOUSE',
        'KICKED','ATE','BUILT',
        'WHO','WHAT','WHICH',
        'SUBJ','OBJ','VERB',
        'FIRST','SECOND','THIRD', 'PERIOD', 'QUESTION']
        
myvocab.populate(';'.join(words))
    #
    #   pre-tie subject/verb/object to initCap words
    #
myvocab.populate('John = JOHN*SUBJ;Susan=SUSAN*SUBJ;Mike=MIKE*SUBJ')
myvocab.populate('Kicked = KICKED*VERB;Ate=ATE*VERB;Built=BUILT*VERB')
myvocab.populate('Ball = BALL*OBJ;Sushi=SUSHI*OBJ;House=HOUSE*OBJ')
myvocab.populate('Who = WHO*SUBJ;What=WHAT*VERB;Which=WHICH*OBJ')



    #
    #   set up inputs
    #
        #
        #   these words will be input one at a time, 
        #   notice pre-tied form
        #
phraseSequence = ['John Kicked Ball PERIOD',
                  'Susan Ate Sushi PERIOD',
                  'Mike Built House PERIOD',
                  'Who Ate Sushi QUESTION']
                  
def visual_input(t):
    phraseindex = int(t/0.8) % len(phraseSequence)
    sequence = phraseSequence[phraseindex].split()
    index = int(t/0.2) % len(sequence)
    return sequence[index]
    
visionState = spa.State(myvocab)
visioninput = spa.Transcode(visual_input, output_vocab=myvocab)
visioninput >> visionState
#17

Thanks, I can reproduce the issue now and it has been a recent change. For now the best option is to provide the label.

#18

Cool. Ready for the next nengo_spa issue? Specifically …

When I use the GUI to create a visualization, such as a semantic pointer cloud, the visualization becomes unbound when I change the code. So if I change the code and re-run, the old visualization remains blank. Every time I change the code, I have to remove the old vizualizations and recreate each one. It’s a little tedious. Hope this helps.

#19

I can reproduce this and created a new issue on Github.