Converting simple Keras network to Nengo Spiking neural network results in bad prediction

General Discussion Frequently Asked Questions
1

Hi,

I have a pretty simple 4 layer network with 3 input neurons, 2 hidden layers of 50 neurons each, and output layer of 5 neurons. All will RELU activation function.
All inputs and outputs are scaled to be between 0 to1 range.
The network is trained with SGD optimizer on 70000 samples using batch of 250
and MSE as a loss function.
15k samples for validation, and 15k samples for test
After several epochs the MSE loss of the network is under 0.02

Epoch 15/15
280/280 [==============================] - 1s 3ms/step - loss: 0.0200 - probe_loss: 0.0200 - val_loss: 0.0198 - val_probe_loss: 0.0198

But when all I do is change the RectifiledLinear activation function to SpikingRectifiedLinear the prediction results in MSE of 0.29, which is the output of data when almost all predicted by spiking network values are 0.
Network is “not working”
Playing with scale_firing_rates parameter has almost no effect.

I am attaching a simplified version of my code to demonstrate what i am doing.
I would be grateful to know if i am doing something wrong.
I am using python version 3.6.8rc1 and NengoDL 3.3.0
Thank you in advance!

def ConstructNetworkNengo(inputTrain,outputTrain,inputValidation,outputValidation,inputTest,outputTest):

steps = 1

inputTrainNgo = np.tile(inputTrain[:, None, :], (1, steps, 1))

outputTrainNgo = np.tile(outputTrain[:, None, :], (1, steps, 1))

inputValidationNgo = np.tile(inputValidation[:, None, :], (1, steps, 1))

outputValidationNgo = np.tile(outputValidation[:, None, :], (1, steps, 1))

inputTestNgo = np.tile(inputTest[:, None, :], (1, steps, 1))

#Build Network

input = tf.keras.Input(shape=(3,))

l1 = tf.keras.layers.Dense(50,activation=tf.nn.relu)(input)

l2 = tf.keras.layers.Dense(50,activation=tf.nn.relu)(l1)

output = tf.keras.layers.Dense(5,activation=tf.nn.relu)(l2)



model = tf.keras.Model(inputs=input, outputs=output)

converter = nengo_dl.Converter(

    model,

    swap_activations={tf.nn.relu: nengo.RectifiedLinear()},

 )

net = converter.net

nengo_input = converter.inputs[input]

nengo_output = converter.outputs[output]

# run training

with nengo_dl.Simulator(net, minibatch_size=250, seed=0) as sim:

    sim.compile(

        optimizer=tf.optimizers.SGD(),

        loss={nengo_output: tf.losses.mse},

    )


    sim.fit(

        inputTrainNgo, {nengo_output: outputTrainNgo},

        validation_data=(inputValidationNgo, outputValidationNgo),

        epochs=15,

    )

    sim.save_params("./nengo-model-relu-1")     

with nengo_dl.Simulator(net, seed=0) as sim:

    sim.load_params("./nengo-model-relu-1")

    data = sim.predict({nengo_input: inputTestNgo})



mse = MSE(outputTest, data[nengo_output][:, -1]) #normalized comparison

print('MSE Nengo prediction using RectifiedLinear of {} samples: {}'.format(len(inputTest), mse))


#Convert to Spiking

converter2 = nengo_dl.Converter(

    model,

    swap_activations={tf.nn.relu: nengo.SpikingRectifiedLinear()},

    #scale_firing_rates=10000,

)

net2 = converter2.net

nengo_input = converter2.inputs[input]

nengo_output = converter2.outputs[output]

with nengo_dl.Simulator(net2, seed=0) as sim2:

    sim2.load_params("./nengo-model-relu-1")

    data = sim2.predict({nengo_input: inputTestNgo})



mse = MSE(outputTest, data[nengo_output][:, -1]) #normalized comparison

print('MSE Nengo prediction using SpikingRectifiedLinear of {} samples: {}'.format(len(inputTest), mse))

return model
2

Hi @alex.hex, and welcome to the Nengo forums! :smiley:

Converting a “rate-based” Keras network to a Nengo spiking neural network requires a few steps in addition to changing the neuron type. Our NengoDL Keras-to-SNN example provides a step by step description on how to convert a standard Keras network to a spiking neural network (using NengoDL).

Taking a brief look at your code, a couple of points stand out to me:

  1. It looks like you are only running each input for 1 step. This is okay for a rate-based network since each “neuron” is able to report its instantaneous firing rate. However, for a spiking neural network, you need to wait for the neuron to fire, which can take a few timesteps to happen. This part of the example describes how to increase the input presentation time to account for this.
  2. It looks like you are not filtering the output of the network. Since the network is using spiking neurons, this would mean the output is very noisy (either a spike or no spike). Applying a synaptic filter on the output will smooth out the spikes and make it less noisy, and “better” for the training algorithm to adjust the weights.

Making these two changes should improve the training accuracy of network. :slight_smile:

3

Hi @xchoo,

Thank you very much for your help!
I’ve changed the timing aspect of my network and analyzed that safe number of steps is more than 10000 for my network. Since some neurons don’t fire before that.
But this did not significantly improve my prediction as compared to regular Relu net.
So i’ve added synapse smoothing, playing with synapse parameter from 0.1 to 0.0001 and this also didn’t get any close to good results. Then i played with firing rate, altering the “scale_firing_rates” parameter going from 10 to as big as 10000.
And still the spiking network prediction MSE is by factor larger than same non-spiking network with Relu.
Can it be that what i am trying to solve is not possible with spikes, or probably i am missing something basic here?

4

If your network was able to train and perform to your expectation using the rate-based ReLU neurons, I’d expect that a spiking network to be able to perform similarly. I’d advise taking the same “debugging” steps that was presented in the Keras-to-SNN example (i.e., generating spike plots and output plots) and compare them to the non-spiking case to see if these is any discernable reason why the spiking model is not performing well.

Another thing I’d definitely recommend checking is to ensure that you sim2.load_params code works with non-spiking neurons. If it doesn’t it might indicate that there may be an issue with the parameter saving and loading.

While you are debugging your network, it might also be helpful to post your full model (or a minimal version of your model that exhibits the same prediction accuracy mismatch between rate and spiking neurons), along with the data you used to train and test it. Without the full model and data, I can unfortunately only provide suggestions based on the code you provided, and from my look-over, there doesn’t seem to be anything obviously wrong with it.

5

Hi @xchoo,

I have tried to follow the Keras-to-SNN example, but unfortunately i am still missing something.
and the network doesn’t work as expected.
I have created a simplified fully functional ipynb example similar to keras-to-snn way to demonstrate the entire flow and the MSE difference.
It is located in GitHub:
https://github.com/alekshex/Nengo/blob/main/nengo_test1.ipynb

there you can clearly see
MSE Nengo run net prediction of 100 samples: 0.031192688108674975
against
MSE Nengo run net prediction of 100 samples: 8.26394990775321
when moving to spikes

I would very much appreciate your help, since currently i don’t know why it goes wrong.

6

Hmmm. Looking at your notebook, I managed to get decent results by combining the use of the synapse, increasing the n_steps and by increasing the scale_firing_rates:

RunNetwork(
    NNa,
    inputTestNgo,
    outputTest,        
    activation=nengo.SpikingRectifiedLinear(), 
    params_file="./nengo-model-relu-1",
    n_steps=200,
    synapse=0.01,
    scale_firing_rates=2500,
)

which gives me the result:

MSE Nengo run net prediction of 100 samples: 0.16374916352982077

Using the default parameters, the neurons in your network have quite a low firing rate, so I had to increase scale_firing_rates to 2500. If you use a lower scale_firing_rates number, you’ll need to increase the synapse parameter (a higher value for synapse help smooth the intermittent spikes out more). However, if you increase synapse, you’ll need to also increase the n_steps parameter, because a larger synapse value will make the output take longer to reach steady state.

As a side note, the synapse parameter does add a “start up” time the output of your network, and the size of the synapse determines how long it takes for the network outputs to reach steady state. Thus, when you compute the RMSE, you have to take this into account. This code demonstrates this effect, and note that it’s not even using spiking neurons:

RunNetwork(
    NNa,
    inputTestNgo,
    outputTest,        
    activation=nengo.RectifiedLinear(), 
    params_file="./nengo-model-relu-1",
    n_steps=200,
    synapse=0.01,
)

image

7

Hi @xchoo,

Thank you for your quick reply!
I did reach the same MSE, with synapse set to 0.1 and firing rate of 1000.
But there is a huge difference between MSE of 0.03 just in one epoch in non-spiking network,
and MSE of 0.16 in spiking.
I understand that there is a “start up” time for the output of the network, but when i calculate the MSE
I take the results from last timestamp (data[nengo_output][:n_test, -1]).
The network should already be stable at this point after 20000 steps.

In all my tries i never managed getting lower than MSE of 0.16 using spiking network.

But my goal is to make it equal to MSE of 0.03 which i get when use non-spiking network with relu.
MSE of 0.16 in my case means, that network will most probably predict wrongly the results and my use case of the network is something i cannot rely on in my application.
I also recall from the keras to snn example that increasing firing rate gets us closer to non-spiking network, and this maybe something that is better to be avoided. But in any case even with firing rate of 10000 MSE doesn’t get lower than 0.16

Is there anything that can be done to move from MSE of 0.16 to 0.03?
Or do you think we hit the limit here?

Best Regards,
Alex

8

Right! I had missed that. Although, I would also caution against calculating the MSE using just the last timestep, because in the spiking network, the output is kind of noisy, so you’d probably want to average it out across multiple timesteps.

The spiking network is inherently noisy, so improving the MSE below 0.16 may be hard to do. You can try increasing the scale_firing_rate parameter, and at the same time, increase the synapses parameter. As a sanity check you’ll want to run the exact same function call (with both parameters) using the non-spiking ReLU neurons to see what is the “maximum” MSE you can achieve when you convert it to use the spiking network.

One thing I did notice is that there may be an issue with NengoDL parameter loading and synapses.
I’ve noticed that if you run this code:

RunNetwork(
    NNa,
    inputTestNgo,
    outputTest,        
    activation=nengo.RectifiedLinear(), 
    params_file="./nengo-model-relu-1",
    n_steps=10,
    synapse=None,
)

but only modify the synapse to be synapse=0, the MSE goes from 0.03 to 0.09, even though, both networks should operate identically. A synapse of 0 applies no synaptic filter (just like when synapse=None), all it does it add a 1 timestep delay between the input of a neuron to the output of a neuron. However, since no synaptic filters are applied, both networks should give you the same result.

We’re still investigating this, and it may take a while to pin point the issue. Thus far, we have discovered that if you don’t load the params from file, the networks are identical, but otherwise they aren’t.