KerasSpiking and TensorFlow

[dadadima]

Hi everyone, I am a MSc student interested in Spiking Neural Networks.

First of all, I would like to thank the people working behind Nengo and the community. I have managed to find almost always the information and the knowledge I was seeking. However, this time I need to reach out for help and a suggestion.

The main reason for me to approach the technology of SNN is that I am very enthusiastic about the great possibilities in terms of energy savings due to the spiking nature of such networks. The main application I am exploring is time-series forecasting using SNN. I have two possible strategies:

• converting to NengoDL a DNN model that uses LSTM and runs on Tensorflow
• writing from scratch my new spiking model using either NengoDL or KerasSpiking

Since I am also interested in good quality predictions and I have been very impressed by the results obtained by LMU on the sequential-MNIST challenge, I am really thinking this is the best way forward for me. Moreover, the similarity of LMU to LSTM together with the numerous learning resources on the Nengo website also influenced my decision.

In the past couple of months, I have extensively explored Nengo and its application from the most basic example to the most complex ones related to NengoDL. For instance, I went through the Nengo examples, with particular attention to the one about LMU. I have also followed and reproduced most of the NengoDL examples, and again focusing on LMU. I have to say it’s quite easy and straightforward to follow through with your example and documentation, given the really nice quality and quantity, but unfortunately, I am not an expert at all in regards to TensorFlow. For this reason, I have struggled without any good progress in the past weeks by trying to converting the TensorFlow LSTM model I was given to a NengoDL model with LMU.

Given that I am more experienced with Keras, now I am thinking to go with KerasSpiking. However, I am not really sure about the possible downsides if I choose this approach. I have read the comparison page, but I am not really sure about the implication of the following sentence: "One particularly significant distinction is that KerasSpiking does not really integrate with the rest of the Nengo ecosystem (e.g., it cannot run models built with the Nengo API, and models built with KerasSpiking cannot run on other Nengo platforms) ". Lastly, I am really interested in the KerasSpiking example called “Estimating Model Energy”, because it could be really useful to prove my initial point. This could be another great reason to choose KerasSpiking, since I couldn’t find any other similar example for NengoDL.

My data looks very similar to this one.

What do you guys recommend for my case? Should I try out KerasSpiking and if yes is it possible to use it with LMUs? Any resource?
On the contrary, do you guys have any suggestions or resources I can use to start implementing my LMU NengoDL model based on my data? Would It be useful to post another question in the forum with my actual TensorFlow model and ask for more practical advice?

I am aware it is a long message, but I wanted to be complete and exhaustive rather than asking a poor question. I really appreciate any form of help or input, thank you all in advance.

[zerone]

Hello @dadadima, welcome to our community! What I understand from your post is that you are interested in time-series modelling with Nengo (and its derivatives). With respect to the following:

@dadadima

I have two possible strategies:

• converting to NengoDL a DNN model that uses LSTM and runs on Tensorflow
• writing from scratch my new spiking model using either NengoDL or KerasSpiking

Your first strategy might/will not work. Nengo-DL does not have a registered converter for LSTM layers. The only supported TensorFlow (TF) layers in Nengo-DL can be found here. As you can check, LSTM isn’t there.

Next, your second approach seems doable to me, albeit with slight tweaks. I can see that KerasSpiking has APIs referring the RNN cells, but I not entirely sure if KerasSpiking can be used to build TimeSeries models, as I too don’t find any relevant examples with it. With respect to using LMUs, it is certainly doable as demonstrated via relevant examples (although I haven’t used LMUs before).

Therefore my best bet would be to proceed with LMUs, attempting to build the model from scratch.

With respect to the following:

@dadadima

I am not really sure about the implication of the following sentence: "One particularly significant distinction is that KerasSpiking does not really integrate with the rest of the Nengo ecosystem (e.g., it cannot run models built with the Nengo API, and models built with KerasSpiking cannot run on other Nengo platforms) ".

KerasSpiking is supposed to be light weight, i.e. it doesn’t support the extensive functionalities as supported by Nengo-DL. Let’s suppose you have a TF network (to be converted to spiking network) and you want to include/replace some extra functionality in the converted TF network which would require Nengo (i.e. you want to tweak your converted spiking network); then your best friend is Nengo-DL since it allows you to interact with Nengo as well. Whereas, if you convert your TF network with KerasSpiking, you won’t be able to add new Nengo code in your KerasSpiking converted network. I hope this clarifies your doubt.

@dadadima

What do you guys recommend for my case? Should I try out KerasSpiking and if yes is it possible to use it with LMUs? Any resource?

As I mentioned above, my best bet would be to go with LMUs, although given that I find some reference to RNNs in KerasSpiking, I would also be interested in what experts have to say about it. Not sure if you came across this, if not… then you can find more info about building LMU models here.

BTW, depending on your preferences, you can also proceed with 1D or 2D CNNs for TimeSeries modelling in TF (you can find many resource on Google). If you are able to build such a TimeSeries model with Convolutional layers, Dense, AveragePooling etc. layers then you are most welcome to use Nengo-DL for the conversion of your rate network to spiking one.

[dadadima]

@zerone

Hello @dadadima, welcome to our community! What I understand from your post is that you are interested in time-series modelling with Nengo (and its derivatives). With respect to the following:

Hey @zerone, thanks for welcoming me! Yeah, correct, more precisely I would like to explore SNN for the task of time-series modeling and I have decided to do that with Nengo, given the friendliness of the framework due to the extensive set of resources.

@zerone

Next, your second approach seems doable to me, albeit with slight tweaks. I can see that KerasSpiking has APIs referring the RNN cells, but I not entirely sure if KerasSpiking can be used to build TimeSeries models, as I too don’t find any relevant examples with it.

I also couldn’t find any relevant information about building time-series models using KerasSpking. Maybe someone more expert on this topic can shed some light.

@zerone

With respect to using LMUs, it is certainly doable as demonstrated via relevant examples (although I haven’t used LMUs before)

One of the examples I have followed is this one, Legendre Memory Units in NengoDL. I think that this model has a spiking nature since it uses Nengo Neurons and other Nengo natives constructs. Is that correct? Moreover, I am having some difficulties in understanding how properly define my LMUCell class, especially the matrix definition.

@zerone

BTW, depending on your preferences, you can also proceed with 1D or 2D CNNs for TimeSeries modelling in TF (you can find many resource on Google). If you are able to build such a TimeSeries model with Convolutional layers, Dense, AveragePooling etc. layers then you are most welcome to use Nengo-DL for the conversion of your rate network to spiking one.

I have also came through this 3-level way of integrating a Keras model in a Nengo network: Integrating a Keras model into a Nengo network. Although, I am not really interested in the first level since it’s non-spiking by definition, I am really interested in the intermediate one, which seems a nice starting point.
I think I will try to start with what you suggested, 1D or 2D CNNs models for time-series and converting them in SNN using NengoDL. After that I will try to implement my LMU model from scratch.

Lastly, I am still really curious in evaluating my time-series models (both 1D/2D CNNs and LMUs) in relation to the energy consumption, in order to prove that SNN are a more efficient alternative. What are the suggestions of the community? Something similar to this seems very nice: Estimating model energy.

[zerone]

Hello @dadadima, with respect to the following,

@dadadima

I think that this model has a spiking nature since it uses Nengo Neurons and other Nengo natives constructs. Is that correct? Moreover, I am having some difficulties in understanding how properly define my LMUCell class, especially the matrix definition.

I believe other experts would be best to answer it, although I would like to point you here as well.

About the following,

@dadadima

I am really interested in the intermediate one, which seems a nice starting point.

my suggestion would be to first go through this tutorial and a related one here. The one you mentioned (about integrating a Keras model to Nengo network) is probably for advanced use cases when you would be required to interface a TF/Keras model with Nengo. Whereas, the ones I have linked will give you a focused and fundamental method to build SNNs.

And with respect to the following,

@dadadima

Lastly, I am still really curious in evaluating my time-series models (both 1D/2D CNNs and LMUs) in relation to the energy consumption, in order to prove that SNN are a more efficient alternative. What are the suggestions of the community?

I have few suggestions to make. To estimate energy consumption (in view of a converted 1D/2D/3D CNNs TF model), you can create the same model architecture with KerasSpiking and estimate the energy consumption - exactly the last link you mentioned. Note that KerasSpiking does not run your model on the actual neuromorphic hardware, rather it mathematically estimates the energy consumed assuming that your network is spiking. OR, if you have access to the Loihi hardware, you can port your TF model to NengoLoihi and then do a real time energy profiling! You can find more information here (again, I am not an expert on NengoLoihi… sorry!, in fact I am myself learning it gradually).

My next suggestion is, when it comes to energy profiling, your actual problem becomes more of a design problem. What I mean is, no doubt you can make your solution to the problem work as expected, but does it guarantee that it will consume minimal energy? No. Then you would have to take measures to properly design your solution, keeping in mind its energy usage as well. More often, you will end up making a tradeoff between the accuracy of your solution and energy consumed. E.g. if your spiking network (i.e. your solution) keeps spiking every timestep then it will have high accuracy (as it will effectively replicate the rate based TF model), but due to high spiking activity it will consume too much of energy. If you execute your spiking network longer (i.e. large number of timesteps), it will have high accuracy, but will end up using more energy. You may wish to answer this question as well… is your solution best suited for a single batch_size (i.e. batch_size = 1) predictions (e.g. in online setting)?

Overall, showing that energy efficient SNNs are better alternative, depends very much on the problem itself (of course when emphasizing on the real world applications).