Numeric Vector Classification using SPA

[xchoo]

@michael

Can I ask some follow up questions on some of the points you mentioned?

Absolutely!

@michael

If I wanted to dynamically pass data to my model would you recommend using the data argument provided by nengo_dl (done in this example) or would you recommend using the above approach where you would create a function which returns the data to be fed into the model at each timestep?

I’d have to double check, but I think the data argument in NengoDL only applies to the sim.fit / sim.train (i.e., the TensorFlow wrapper functions). Since you’ll probably be simulating the network with the .run function, you’ll need to use a node with a (custom) function to feed in your data. I posted some pseudo-code demonstrating how to do this with a Python class in this forum post.

@michael

You had mentioned the approach used in Spaun where a classifier was used to classify the visual inputs (would this be both the MNIST images and the task character cues (A0, A1, etc.)?

That is correct! The same classifier network was trained on the MNIST images and used with the character cues (A, X, ?, >, <, etc.) as well. Interestingly (and I’ll go into more detail on this later), no additional training was needed for the network to classify the character cues (it was only trained on the MNIST images).

@michael

Would I be right in saying that for this approach I would be creating a DL network which generates the semantic pointer vectors based on the EMG input?

In essence, yes! There are multiple ways of doing this though. One easy approach is to train a DL / ML network to classify your EMG data into a set of output classes. Then you can apply a one-hot encoding schema to those classes to easily turn them into “semantic pointers”. You can then feed these one-hot encoded semantic pointers into a hetero-associative memory to map them into a semantic pointer space that is used by the rest of your model.

@michael

Is this how Spaun achieved it by using this encoding approach? I had read the Spaun 2.0 thesis paper where it talks about Spauns vision module making use of a visual semantic pointer classifier to achieve its tasks, does this semantic pointer classifier implement the same logic where it uses a DL network to encode the image data into a semantic pointer vector?

In Spaun, a slightly different approach (but essentially the same technique) from the one-hot method was used. Since the MNIST classifier was only trained on the MNIST digits (i.e., the output classes only go from 0 to 9), the one-hot encoding schema couldn’t be used. Fortunately, it turned out that the activity of the second-to-last layer was sufficient to use as the “intermediate” semantic pointer (I referred to this as the “visual semantic pointer” in the Spaun 2.0 thesis). As with the one-hot approach, this “intermediate” semantic pointer was used with a hetero-associative memory to map it on to the Spaun-wide semantic pointer space. I should note that there had to be some scaling applied to the activities of the second-to-last layer to get it into the right magnitudes (see the previous discussion about vector magnitudes) so that the hetero-associative memory could use them properly.