4 layer SNN using Nengo Core (2.80)

Examples & Tutorials
1

Hi!
I am new to Nengo and currently trying to build a 4 layer SNN for classification in Nengo core. The goal is to use this network for classification with fewer classes (2-4). I am trying to learn from the NengoExtras Single layer MNIST example and having difficulties. Please find below my attempted code. Any help would be very much appreciated. Thank you!

import matplotlib.pyplot as plt
import numpy as np
from numpy import random
from nengo_extras.data import load_mnist, one_hot_from_labels
from nengo_extras.matplotlib import tile
from nengo_extras.vision import Gabor, Mask
rng = np.random.RandomState(9)
import nengo


(X_train, y_train), (X_test, y_test) = load_mnist()

X_train = 2 * X_train - 1  # normalize to -1 to 1
X_test = 2 * X_test - 1  # normalize to -1 to 1
T_train = one_hot_from_labels(y_train, classes=10)

# --- set up network parameters
n_vis = X_train.shape[1]
n_out = T_train.shape[1]

# input layer
n_in = 784

ens_params = dict(
    eval_points=X_train,
    neuron_type=nengo.LIFRate(),
    intercepts=nengo.dists.Choice([0.1]),
    max_rates=nengo.dists.Choice([100]),
)


solver = nengo.solvers.LstsqL2(reg=0.01)
 

with nengo.Network(seed=3) as model:
    a = nengo.Ensemble(n_in, 784, **ens_params)
    v = nengo.Node(size_in=n_in)
    conn1 = nengo.Connection(a, v, synapse=None, solver=solver)

#2nd layer     
    A = nengo.Ensemble(784, 64)
    v1 = nengo.Node(size_in = 64)
    conn2 = nengo.Connection(A, v1, synapse=None, solver=solver)
    
#3rd layer   
    B = nengo.Ensemble(64, 64)
    v2 = nengo.Node(size_in=64)
    conn3 = nengo.Connection(B, v2)

#output layer  
    C = nengo.Ensemble(64, 10)
    v3 = nengo.Node(size_in=10)
    conn4 = nengo.Connection(C, v3, synapse=None, solver=solver)
    
    solver = nengo.solvers.Solver(weights=True)

with nengo.Simulator(model) as sim:
    sim.run(10)
    
def get_outs(sim, images):
    #reshape 

    # encode the images to get the ensemble activations
    _, acts1 = nengo.utils.ensemble.tuning_curves(a, sim, inputs=images)
    _, acts2 = nengo.utils.ensemble.tuning_curves(A, sim, inputs=images)
    _, acts3 = nengo.utils.ensemble.tuning_curves(B, sim, inputs=images)
    _, acts4 = nengo.utils.ensemble.tuning_curves(C, sim, inputs=images)


    # decode the ensemble activities using the connection's decoders
    npd1 = np.dot(acts1, sim.data[conn1].weights.T)
    npd2 = np.dot(acts2, sim.data[conn2].weights.T)
    npd3 = np.dot(acts3, sim.data[conn3].weights.T)
    npd4 = np.dot(acts4, sim.data[conn4].weights.T)
    return np.dot(acts4, sim.data[conn4].weights.T)

def get_error(sim, images, labels):
    # the classification for each example is index of 
    # the output dimension with the highest value
    indx = np.argmax(get_outs(sim, images), axis=1) != labels
    return indx

def print_error(sim):
    train_error = 100 * get_error(sim, X_train, y_train).mean()
    test_error = 100 * get_error(sim, X_test, y_test).mean()
    print("Train/test error: %0.2f%%, %0.2f%%" % (train_error, test_error))

#not sure if we need to use encoders, and if we must, then how to setup these?
#do we setup one encoder or need one for each layer?

encodersa = rng.normal(size=(n_in, 28 * 28))
a.encoders = encodersa

encodersA = rng.normal(size=(784, 8 * 8))
A.encoders = encodersA

encodersB = rng.normal(size=(64, 8 * 8))
B.encoders = encodersB

encodersC = rng.normal(size=(64, 10))
C.encoders = encodersC

tile(a.encoders.reshape(-1, 28, 28), rows=4, cols=6, grid=True)
tile(A.encoders.reshape(-1, 28, 28), rows=4, cols=6, grid=True)
tile(B.encoders.reshape(-1, 8, 8), rows=4, cols=6, grid=True)
tile(C.encoders.reshape(-1, 5, 2), rows=4, cols=6, grid=True)

print_error(sim)
2

Hi kvemuru,

If you’d like to do a four-layer SNN, my guess is that you want something that is trained end-to-end with backpropagation, as is done with a traditional ANN. For that, you’ll need to use NengoDL. There’s a NengoDL spiking MNIST example that can get you started there.

The code that you’ve posted looks like it’s taken from the NengoExtras single-layer MNIST example here. That example takes a bit of a different approach: it has a single layer, and does not train the network end-to-end. It uses a fixed set of weights from the input to the hidden layer, which we call the “encoders”. We then learn the weights from the hidden layer to the output, called the “decoders”, to minimize our classification loss. This method is based of the principles of the Neural Engineering Framework, which you can learn more about here.

In that example, I show a number of different ways of choosing the encoders, and the idea is to choose one of them. The “sparse Gabor filter encoders” perform the best, so that’s probably the distribution you want to use.

Let me know which approach you’re more interested in, and if you have more questions.

1 Like
3

Hi Eric,

Thank you for quickly reviewing the post and the helpful guidance to further develop the SNN network. Yes, this one is from Single-Layer MNIST example. My goal is to build a 4 layer network (if possible in the NEF encoder based approach) for inference using the weights from an identical network (I have the weights). Training is not necessary for the one I am trying to build, but if the weights can be further optimized/learned, that will help!

If you can take a look at the code and suggest some improvements, that would be great!
You can call me Krishna.
Thank you!
Best Regards,
Krishna Vemuru

4

Hi Krishna,

I’m not sure how those goals will fit together. If you have weights from another network, then you do not want to use the encoders generated by Nengo, since these are different weights and will give you different results. If you want to use the encoders generated by Nengo, then you’re not going to be able to make a multi-layer network; you’ll have to stick to the single-layer network described in the Single-layer MNIST example.

If you’re planning on using weights from another network, then you’ll probably want to use neuron to neuron connections (this allows you to specify the full dense weight matrix). For example:

weight_matrix = np.zeros((90, 100))  # replace this with your weights
a = nengo.Ensemble(100, 1)
b = nengo.Ensemble(90, 1)
nengo.Connection(a.neurons, b.neurons, transform=weight_matrix)

Here, I’ve got two ensembles, the first with 100 neurons, and the second with 90 neurons. (The 1 passed when creating the ensembles specifies the number of dimensions. If we’re only using the .neurons attribute when creating connections, then this doesn’t matter so we just set it to 1.)

We connect the .neurons of one ensemble to the .neurons of the other, indicating that we’re directly connecting the neurons of these ensembles, we’re not using the encoders or decoders that Nengo solves for by default. We provide the full connection weight matrix in weight_matrix.

In your code, you have created four ensembles that are not connected to one another. Each ensemble connects to a different node, so there is no link between the ensembles. Also, you do not have an input node, so nothing is providing input to any of the ensembles.

Let me know if this helps.

1 Like
5

Hi Eric,
Thanks again for clarifying when the Nengo encoders should be used (single-layer) and how one can build a a multi-layer network with an input of weight-matrix for each connection using nengo.Ensemble! This is very helpful! I revised the code with this approach, a few things are still not clear to me. Should we use the nengo.Node or the ens_params to connect the training data to input to the network. Should we still use the encode and decode of ensemble activation to get the index (in get_error) to calculate the training/test errors? Some more guidance would be very much appreciated! The revised code is pasted below, it seem to work. I will update the weight_matrix with non-zero values and try again. Thank you.
Best Regards,
Krishna

#Spiking Neural Network with 784,64,64,10 neuron layers for classification
#using MNIST dataset
#following the NengoExtras Single-layer MNIST example 
#%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
from numpy import random
from nengo_extras.data import load_mnist, one_hot_from_labels
from nengo_extras.matplotlib import tile
from nengo_extras.vision import Gabor, Mask
rng = np.random.RandomState(9)
import nengo


(X_train, y_train), (X_test, y_test) = load_mnist()

X_train = 2 * X_train - 1  # normalize to -1 to 1
X_test = 2 * X_test - 1  # normalize to -1 to 1
T_train = one_hot_from_labels(y_train, classes=10)

# --- set up network parameters
n_vis = X_train.shape[1]
n_out = T_train.shape[1]


#working with a small subset of data while coding the network
ns = 1000
X_train2 = X_train[0:ns,:]
y_train2 = y_train[0:ns,]
T_train2 = T_train[0:ns]

nt =200
X_test2 = X_test[0:nt,:]
y_test2 = y_train[0:nt]


# input layer
n_in = 784

ens_params = dict(
    eval_points=X_train2,
    neuron_type=nengo.LIFRate(),
    intercepts=nengo.dists.Choice([0.1]),
    max_rates=nengo.dists.Choice([100]),
)


solver = nengo.solvers.LstsqL2(reg=0.01)
 

with nengo.Network(seed=3) as model:
     
    #data = nengo.Node(X_train2)
    a = nengo.Ensemble(784, 784, **ens_params)
    #conn1 = nengo.Connection(data, a.neurons, )
    weight_matrix2 = np.zeros((64, 784))  # replace this with your weights
    b = nengo.Ensemble(64, 1)
    conn2 = nengo.Connection(a.neurons, b.neurons, transform=weight_matrix2)
    
    weight_matrix3 = np.zeros((64, 64))  # replace this with your weights
    c = nengo.Ensemble(64, 1)
    conn3 = nengo.Connection(b.neurons, c.neurons, transform=weight_matrix3)
    
    weight_matrix4 = np.zeros((10, 64))  # replace this with your weights
    d = nengo.Ensemble(10, 1)
    conn4 = nengo.Connection(c.neurons, d.neurons, transform=weight_matrix4)
    
    #solver = nengo.solvers.Solver(weights=True)

with nengo.Simulator(model) as sim:
    sim.run(10)

def get_error(sim, images, labels):
    # the classification for each example is index of 
    # the output dimension with the highest value
    indx = np.argmax(conn3.probeable[0]) != labels
    #indx =random.randint(0,2,1000)
    return indx

def print_error(sim):
    train_error = 100 * get_error(sim, X_train2, y_train2).mean()
    test_error = 100 * get_error(sim, X_test2, y_test2).mean()
    print("Train/test error: %0.2f%%, %0.2f%%" % (train_error, test_error))

print_error(sim)
6

For the input, you want to use a nengo.Node, probably with a nengo.process.PresentInput as the output. This will present a series of images, each for a length of presentation_time.

You’ll also need to add a probe to the output of your network. Probes are how we record output in Nengo. For example, d_probe = nengo.Probe(d.neurons).

After you run your network, the output of d.neurons will be available in sim.data[d_probe]. The first axis of this array will be the time dimension. Because you’re presenting each input for a length of time and then switching to the next one, to measure the accuracy, you need to determine the classification in each of these windows (i.e. the classification for each output). The last code block in this example gives you an idea of how to do this.

1 Like
7

Hi Eric!
Thanks a lot for all the quick responses!! I saw how Probe is used in the EnsembleArray example. https://www.nengo.ai/nengo/examples/networks/ensemble-array.html
I revised the code using nengo.Probe as you suggested and the last block from the example. My code works now!

#setup the probes
b_probe = nengo.Probe(b.neurons)
c_probe = nengo.Probe(c.neurons)
d_probe = nengo.Probe(d.neurons)

You are awesome!

Best Regards,
Krishna

8

Hi Eric,

The last code block in the ImageNet example with the link your latest email, we have this code:

blocks = sim.data[output_p].reshape(n_presentations, nt, n_classes)
choices = np.argsort(blocks[:, -20:, :].mean(axis=1), axis=1)
top5corrects = choices[:, -5:] == Y_test[:n_presentations, None]
top1accuracy = top5corrects[:, -1].mean()

First, I would like to understand the number 20 in [:, -20; :], is this same as nt?

Second, when I use the 2-class data, if we modify the code as below will that become a correct to calculate the accuracy?

top5corrects = choices[:, :] == Y_test[:n_presentations, None]
top1accuracy = top5corrects[:, 0].mean()

Thanks again!
Best Regards,
Krishna

9

The -20 takes the last 20 timesteps within each presentation. This is because when we first present the stimulus, there might be transients left over from the last stimulus, so we get rid of that time and just use the last part of the presentation when the activities have settled.

If you just have two classes, you don’t need to worry about doing the top 5 (that’s for ImageNet, so that we can see whether the correct class is in the top 5 guesses of the neural network).

For you, I’d just do something like

predictions = np.argmax(blocks[:, -20:, :].mean(axis=1), axis=1)
assert predictions.shape == Y_test[:n_presentations].shape
accuracy = (predictions == Y_test[:n_presentations]).mean()

I put the assert in there just to make sure they’re the same shape, so that Numpy doesn’t secretly do some broadcasting that we’re not expecting.

1 Like
10

Eric, Thanks a lot for the explanation of the -20 timesteps factor for the presentations and the simplified code for the accuracy calculation for the 2-class example! I am loving Nengo and hope to try some more networks. The code now looks like this. If you find any errors, please do let me know !
Best Regards,
Krishna

from random import sample
import matplotlib.pyplot as plt
import numpy as np
from numpy import random
from nengo_extras.data import load_mnist, one_hot_from_labels
g = np.random.RandomState(9)
import nengo

— load the data

img_rows, img_cols = 28, 28

(X_train, y_train), (X_test, y_test) = load_mnist()

X_train = 2 * X_train - 1 # normalize to -1 to 1
X_test = 2 * X_test - 1 # normalize to -1 to 1
#T_train = one_hot_from_labels(y_train, classes=10)

#select the 0s and 1s as the two classes from MNIST data
X_train2 = []
y_train2 = []
for i in range(0,5000):
if y_train[i] == 0:
X_train2.append(X_train[i])
y_train2.append(y_train[i])

for i in range(0,5000):
if y_train[i] == 1:
X_train2.append(X_train[i])
y_train2.append(y_train[i])

X_train2 = np.array(X_train2)
y_train2 = np.array(y_train2)
#############################
T_train2 = []
for i in range(0,1000):
oh = [0]
if y_train2[i] == 0:
oh.insert(0,1)
elif y_train2[i] == 1:
oh.insert(1,1)
#print(oh)
T_train2.append(oh)
T_train2 = np.array(T_train2)
#########################################################################
n_vis = 784 #number of pixels in each image
n_out = 2 #number of classes in the data

#split the data randomly into training and testing sets
l = 1000 #length of data
f = 1000 #number of elements you need
#generate a set of random indices for training data
indices = sample(range(l),f)

#finding the remaining samples of the population
idxall = []
for j in range(0,1000):
idxall.append(j)

idxall = np.array(idxall)
idx = np.array(indices)
idx2 = np.delete(idxall,idx)
indices2 = idx2.tolist()

#select a portation of the training data as test data
X_train2 = X_train2[indices]
y_train2 = y_train2[indices]
T_train2 = T_train2[indices]
X_test2 = X_train2[indices2]
y_test2 = y_train2[indices2]

classes = 2
ns = 1000
nt = 1000
presentation_time = 0.2

input layer

n_in = 784

ens_params = dict(
eval_points=X_train2,
neuron_type=nengo.LIFRate(),
intercepts=nengo.dists.Choice([0.1]),
max_rates=nengo.dists.Choice([100]),
)

solver = nengo.solvers.LstsqL2(reg=0.01)

with nengo.Network(seed=3) as model:
#a = nengo.Ensemble(784, 784, **ens_params)
picture = nengo.Node(nengo.processes.PresentInput(X_train2, presentation_time))
weight_matrix1 = random.random((64, 784)) # replace this with your weights
b = nengo.Ensemble(64, 1)
conn2 = nengo.Connection(picture, b.neurons, transform=weight_matrix1)

weight_matrix2 = random.random((64, 64))  # replace this with your weights
c = nengo.Ensemble(64, 1)
conn3 = nengo.Connection(b.neurons, c.neurons, transform=weight_matrix2)

weight_matrix3 = random.random((2, 64))  # replace this with your weights
d = nengo.Ensemble(2, 1)
conn4 = nengo.Connection(c.neurons, d.neurons, transform=weight_matrix3)

#setup the probes
b_probe = nengo.Probe(b.neurons)
c_probe = nengo.Probe(c.neurons)
d_probe = nengo.Probe(d.neurons)

with nengo.Simulator(model) as sim:
sim.run(50)

Plot the results

plt.figure()
plt.plot(sim.trange(), sim.data[c_probe])

#npt = int(presentation_time / sim.dt)
#print(npt)
n_classes = 2
blocks = sim.data[d_probe].reshape(ns, 50, n_classes)

predictions = np.argmax(blocks[:, -20:, :].mean(axis=1), axis=1)
assert predictions.shape == y_train2[:ns].shape
accuracy = (predictions == y_train2[:ns]).mean()
print(“Accuracy: %0.2f%%” % (100accuracy))
--------------------------------------------------------------------------------------strong text***strong text