LMU for Time series forecasting

Hi everyone,

I am new to Nengo DL. I have been trying to see if it is possible to make time-series forecasts using an SNN.

I tried to implement an LMU to forecast the time series. I have attached the code below:

LMU nengoDL.ipynb (149.5 KB)

but when I run the simulation on my own time series data(stock prices) then surprisingly again with this model I get the same prediction value(output) for all test samples and also get a huge loss.

I am not sure what I am doing wrong here, since I tried to tune the parameters but nothing seems to work. Would greatly appreciate it if someone could help. thank you.

Thank you

1 Like

Hi @acnh371, and welcome to the Nengo forums. :smiley:

In addition to the thread you have posted, there is another (more recent) thread that may be of interest to you as well: link.

With respect to the notebook you posted, I found that for the dummy dataset, the model needed a large about of epochs (~5000) to reach a decent test result. I also reduced the model parameters (by making the units and order both 30) in order to make sure that the LMU network wasn’t overfitting the relatively small training data set.

As for the actual (stock price) data, I would caution that such a simplistic model (i.e. just the LMU network) may not be enough to produce decent results. Additionally, I don’t even know if forecasting the stock price is possible with the dataset you used.

Hi @xchoo,

Thank you for your reply.

For the actual data(stock data). I have now changed the dataset and I am getting comparable results to LSTM.

I was wondering if it is possible to implement a spiking version of the LMU in nengoDL. I have seen an example using Nengo Loihi but I don’t have access to Loihi hardware.

Thank you

Hi @acnh371,

It should be possible to use the NengoDL converter to convert the trained LMU network into a spiking one. The one difficultly I can foresee is that the self.h population is a TensorNode, and you might need to replace this with a nengo.EnsembleArray (like in the NengoLoihi example) to get it to be fully spiking. I haven’t tested it myself, so it’s just a guess.

As a side note, NengoLoihi code (assuming you remove the nengo_loihi.set_defaults() and other NengoLoihi-specific code) can be run in regular Nengo and in NengoDL. All you need to do is to change the nengo_loihi.Simulator call to nengo.Simulator or nengo_dl.Simulator.

2 Likes

Hi @xchoo ,

Thank you for your reply.

Over the past few days, I tried really hard to convert the LMU into spiking but run into some errors. I have tried to convert KerasLMU into spiking using the nengo_dl.Converter method. I have attached my code below. Could you please tell me what I am doing wrong in my code?

Converting LMU to spikingLMU.ipynb (33.6 KB)

I get one of these two errors:
error 1

or

In the code, I provided a month earlier (top of the post) The LMUcell was fully defined in a function and you suggested to change the self.h to a nengo.EnsembleArray. I tried that but was not able to make it work.

I have looked in the forum and elsewhere but cannot find an example of LMU conversion into spiking. so I would greatly appreciate if your could help me with this problem.

Thank you

I spoke to the NengoDL and KerasLMU devs and from what I understand, trying to convert a KerasLMU model to a SNN using the NengoDL converter is a difficult process due to the differences between how TF (which is used to run KerasLMU) and Nengo represent time.

In order to “convert” a KerasLMU network into a spiking version, you’ll probably need to:

  1. Train the KerasLMU network in TF
  2. Save the network connection weights after training
  3. Construct an equivalent spiking network in NengoDL
  4. Load the network connection weights into the NengoDL model to run as a spiking network.

I haven’t done this process myself, but @travis.dewolf has and may have some code to share with you to facilitate this process.

Another approach to training a spiking LMU network is to use a Nengo network from the start (as in the NengoDL LMU example). However, as I mentioned before, you’ll need to replace the self.h population with a nengo.Ensemble in order to make it spiking. If you take this approach, you’ll need to manually specify the neuron parameters to emulate the defaults used by TF. I found that the following (a spiking version of the tanh neuron) works reasonably well:

self.h = nengo.Ensemble(units, 
                        1, 
                        neuron_type=nengo.RegularSpiking(nengo.Tanh(tau_ref=1)), 
                        gain=np.ones(units), 
                        bias=np.zeros(units)).neurons   

The gain and bias values are 1 and 0 by default in TF, and the tau_ref value limits the firing rate of the neuron to about 1, which is also default in TF. You can tweak these values to see if they improve your results. You can also substitute the neuron model with nengo.LIF or nengo.SpikingRectifiedLinear to use other neuron types besides a spiking Tanh neuron.

3 Likes

Hi @xchoo,

Thank you for your answer. Since your post, I have been working on this and I decided to go with the second approach you suggested as I am a bit pressed for time, and given the complications around the first approach, I thought the second approach would be better to try. I did replace the self.h as you suggested and for the standard network code that is available on nengo, I am getting 0 as the output for all test samples.

I, therefore, tried to pass my test samples for n_steps in my second experiment and took the final output spike as my prediction for each test sample, however, I am getting very strange results that fluctuate in a wide range. I have attached my working code below:

Converting LMU to spikingLMU - nengo.ipynb (30.8 KB)

For comparison, I have also included the normal LMU output using the non-spiking tanh function which produces good results. Could you kindly suggest what I might be doing wrong here or how I could change the network to improve the results?

Thank you

I took a look at your notebook, and there doesn’t seem to be anything outwardly wrong about the network itself. Rather, I think the issue stems with the probes that you are using to record the output data.

Because the spiking network requires filtering on the probes in order to smooth out the spike noise, you’ll need to ensure that the synapse on the filter is long enough to sufficiently filter the spikes (this depends on the spike rate of the neurons you are probing), but not too long that the output becomes too unresponsive. From what I can see, your probes have 2 problems:

  • With respect to the number of steps the network is run to get the output (100 steps from what I can see), the synapse on the output probe (0.1s) basically means that the output doesn’t really change much.
  • The amount of time between each input presentation needs to be long enough to allow the information to propagate through the entire network, and for the filtered output value to “stabilize”.

One thing I did note while playing around with your network is that there’s a lot of “ringing” in the output when using spiking neurons. I think that might have to do with the lack of a synapse on the h feedback connection, but I don’t have much experience with spiking LMU networks, so it’s only a guess. I’ll have to ask the LMU devs to see what else needs to be done (I’m guessing that a synapse needs to be put on the h feedback, and maybe the firing rate of the h ensemble needs to be increased somehow) to effectively use a spiking LMU network.

1 Like

Hi @xchoo,

Thank you for your response. I have tried to tweak things however was not able to obtain a satisfactory result, however, I am satisfied that I was able to make both LIF and LMU networks work for my task.

Thanks again for all your support.

1 Like

Sorry about the long delay responding to being tagged. I took a look at the notebook here, and noticed a few things.

  1. the parameter count between the TensorNode and the nengo.Ensemble with nengo.Tanh was different (you can check parameter count with sim.keras_model.summary()). The Ensemble had a trainable bias. To test that they performed the same I removed the RegularSpiking wrapper and set the bias to non-trainable and I get the same results in both cases.
  2. The reason you were getting 0 for output is because RegularSpiking was wrapping Tanh neurons with max firing rates of 1 (specified when you set tau_ref=1). While training NengoDL uses the rate mode version of the neuron, which outputs real numbers. But the spiking version (used when sim.predict is called) only outputs integers. Since the spiking neurons had a max firing rate of 1 spike per second, this lead to essentially never spiking and your output being all zeros.
  3. In addition to the max firing rate of 1Hz, you also need to set synapses on all of the connections outbound from lmu.h before running the sim.predict (the Nengo default synapse value is 0.005, may need to play around a bit with that value to get good results, depending on the firing rates of your spiking Tanh neurons). Also, when you add synapses it takes time for information to filter through the network, you’ll probably want to train and test on sequences longer than 3 time steps.

I’ve attached an updated version of the notebook with the network version that gets the same results with the Ensemble as with the TensorNode.
Converting LMU to spikingLMU - nengo.ipynb (29.4 KB)

2 Likes

Hi @travis.dewolf,

Thank you for your detailed answer.

1 Like

Hi everyone,

Sorry for this reply on an old thread.

I am working on the same use case as OP, time series forecasting using Spiking Neural Network (LMU).

Could you please explain me the difference between keras_lmu and class LMUCell(nengo.Network): that uses self.h = TensorNode. From what I understood from @xchoo replies these two implementations are both non-spiking. The first one is plain keras while the second one doesn’t use nengo.EnsembleArray() as self.h. Is this correct? Then what is the main difference between the two?

Also, is the only requirement to have a Spiking LMU defining self.h = nengo.EnsembleArray() or I do need more?

EDIT:

I also have troubles understanding a few more things related to @travis.dewolf answer:

  1. Why as soon as you set net.config[lmu.h.ensemble].trainable = True or (remove net.config[lmu.h.ensemble].trainable = False) the number of parameters count between the TensorNode and nengo.Ensemble are different? So to phrase it differently, how and why there is a trainable bias?

  2. Now that we have removed net.config[lmu.h.ensemble].trainable = False and the RegularSpiking wrapper, is still a spiking LMU?

  3. Could you show an example (even based on another use case than OP’s notebook) on how to overcome the 0s OP was getting? You mentioned that you 1Hz is too low since 1spike per second is not enough, but this alone won’t solve the problem. How can I set synapses on all the outbound connections from lmu.h?

Thanks so much for your help guys!

1 Like