I need to rewrite a model from Keras to Nengo-Loihi and run it with a backend on the chip (nengo_loihi.Simulator). The model is a simple three-dense-layer network, but it’s not possible to import it directly from Keras to Loihi. I couldn’t find any examples of dense layers written in Nengo. Would it be possible to implement it? I would appreciate any suggestions and I attach a code snippet below.
dropoutRate = 0.25
inputArray = Input(shape=(input_shape,))
x = Dense(64, activation='relu')(inputArray)
x = Dropout(dropoutRate)(x)
x = Dense(32, activation='relu')(x)
x = Dropout(dropoutRate)(x)
x = Dense(32, activation='relu')(x)
x = Dropout(dropoutRate)(x)
output = Dense(5, activation='softmax')(x)
model = Model(inputs=inputArray, outputs=output)
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
An example of this in the context of a deep learning Loihi model is available here: https://www.nengo.ai/nengo-loihi/examples/keyword_spotting.html. Note that the connections are of the form nengo.Connection(layer_x.neurons, layer_y.neurons, ...). This example works by taking the pretrained model parameters from NengoDL and then directly mapping them onto the connection weights of a Nengo-Loihi model. I think that you would want to do something similar by extracting the weights from your trained Keras model and importing them directly into an equivalent spiking model. Note that there is no softmax in Loihi, so you would either want to approximate this in another way, or as an offline post-processing step.
For your reference there is another good example here: https://www.nengo.ai/nengo-loihi/examples/mnist_convnet.html, which performs the training directly on the model without going through this extra step of converting weights from one model format to another.
I was wondeing if it’s possible to use tf.nn.softmax_cross_entropy_with_logits_v2 just like in a convolutional example. The training would look as follows:
with nengo_dl.Simulator(model, minibatch_size=minibatch_size, seed=0) as sim:
sim.train(train_data, tf.train.RMSPropOptimizer(learning_rate=0.001), objective={out_p: crossentropy}, n_epochs=10)
I’m able to run the training and deploy the results to nengo-loihi, but can’t get lower error than 47%. I’m not sure if this is correct and would appreciate a comment on the idea.
I don’t see any obvious errors in your code. Is 47% the training error when you train with nengo_dl, or the testing error when you test with nengo_loihi?
How well does your Keras model do on the task? How bad is 47% (i.e. is it a binary decision task, where 47% is about chance, or did it actually learn something)?
One thing you can try is breaking down the problem. Start with a model in nengo_dl that is as close as possible to your Keras model. Specifically, use nengo.RectifiedLinear neurons. You also might want to try making all your neurons the same at the start, by putting something like:
I’ve chosen a max_rate of 200 here, because this is the inverse of your neuron amplitude of 0.005, so when combined should give you neurons that have an output of 1 when the input is 1 (i.e. they don’t affect the scale of the input).
If things do not train well in nengo_dl, then there’s some problem with how you set up your model that we’re missing (possibly some weird scaling going on somewhere). If it does train well, then you can try switching to LIFRate neurons (in nengo_dl) and see if that works, and then try running in spiking LIF neurons in nengo_dl, and finally try it on Loihi.
Hi Eric,
An error of 47% was achieved with nengo_dl but simulator with nengo_loihi shows 84% of error (not sure why yet). A task is to classify 5 types of particles for high energy physics and the best results got with this Keras model were ~74% of accuracy, so it actually learned something with nengo_dl.
Thank you for all suggestions. I’ll investigate the problem and try to play with different settings that you described. Good to know that idea is correct itself, probably I can just adjust the model better.
I verified the code and everything seems correct. The model performs really well in the simulations with nengo_dl, but still I encounter the problem in nengo_loihi. I was able to get 57,43% of acc with nengo_dl and even 70,8% after adding the synaptic filters. This is already very close to the best results on this task with DNNs in Keras. I’ve added the second dimension to the data such as in the examples (https://www.nengo.ai/nengo-loihi/examples/mnist_convnet.html), but after longer discussion with Eric we were not sure if this output is actually running in spiking neurons. Before the simulation I repeat the test data with input/target for a number of timesteps, so in my understanding the results should be with spikes. I would be very thankful for clarifying this part.
I’ve also added Probes to the layers and surprisingly the number of activated spikes with nengo_loihi is always 0. Same happens with LIF and SpikingRectifiedLinear neurons. It seems like the simulation in nengo_loihi doesn’t work at all, while I get good results with nengo_dl (as I understand also with spiking neurons). What could be the reason?
One problem I see is that you’re not actually providing any input data when you run it in nengo_loihi. In nengo_dl, when you call sim.train, you provide data as part of the train call that overrides your input node output. This is the same with sim.loss. This is why nengo_dl examples often just set nodes to output zeros, since these values are then overridden.
When you’re running in nengo_loihi, your simulation is going to be using the node as it was originally configured, i.e. outputting all zeros. This is why in the nengo_loihi MNIST example we use the PresentInput class to set up the node to present the test images each for a length of presentation_time. Look at where PresentInput is used in that example to see how you should be setting up your node.
Please forgive me a late answer. Indeed, the problem was an input data configuration. I knew it had to be something really obvious and eventually I was able to achieve up to 69,8% of acc on Loihi. Hopefully the results will be published what I could also share here later. Thank you very much for helping.