Nengo DL time series forecasting with regressors

Hi,

I am new to spiking neural net. I am working to figure out a possible benefit of time series forecasting with spiking neural net. My question is that Nengo DL can be used for time series forecasting with regressors? I can follow a classification problem in example but my goal is to predict real values.

I have real-valued three regressors say X1, X2, X3, and one real-valued predictor.

Thanks,

Hi @happyjang7, welcome to the Nengo forum!

In Nengo DL, there isn’t a very strong distinction between classification and regression problems. See, for example, the plots at the bottom of the page; the correct number is the highest value, but all numbers have a real value associated with them. If you wanted each of those numbers to be a certain value rather than only wanting the chosen value to have the highest, you could use a different objective function. The MNIST example uses the tf.nn.softmax_cross_entropy_with_logits_v2 objective function, which gives good results for classification problems. The default objective function, mean squared error, would be appropriate for regression problems. Depending on the type of problem, other objective functions could give better performance. But the overall structure of the network and how it is built and trained within Nengo DL will be very similar regardless of whether you’re solving a classification or regression problem.

Hi @tbekolay.

Thanks for your reply. I will try it and ask if I have any further question.

Sounds good!

Hi, @tbekolay.

I have one more question. I have tried to make Multivariate CNN Models with nengo where there are 7 historical data points (input: 3 by 2, output: real number). As you recommended, I set objective function with mse. I set n_epoches as 1000 but the test result is very different from what I expected (y_test = 206.0161). Could you guide me where I missed with the below code written based on your tutorial?

=========================================================

import nengo
import tensorflow as tf
import numpy as np
import nengo_dl
import matplotlib.pyplot as plt
n_steps = 3
x_train = np.array([[[10, 15], [20, 25], [30, 35]],
                    [[20, 25], [30, 35], [40, 45]],
                    [[30, 35], [40, 45], [50, 55]],
                    [[40, 45], [50, 55], [60, 65]],
                    [[50, 55], [60, 65], [70, 75]],
                    [[60, 65], [70, 75], [80, 85]],
                    [[70, 75], [80, 85], [90, 95]]])
print(x_train.shape)
n_features = x_train.shape[2]
y_train = np.array([65, 85, 105, 125, 145, 165, 185])
x_test = np.array([[[80, 85], [90, 95], [100, 105]]])
y_test = np.array([206.0161])


with nengo.Network() as net:
    net.config[nengo.Ensemble].max_rates = nengo.dists.Choice([100])
    net.config[nengo.Ensemble].intercepts = nengo.dists.Choice([0])
    neuron_type = nengo.LIF(amplitude=0.01)
    nengo_dl.configure_settings(trainable=False)
    # the input node that will be used to feed in input images
    inp = nengo.Node([0] * n_steps * n_features)


    x = nengo_dl.tensor_layer(inp, tf.layers.conv1d, shape_in=(n_steps, n_features), filters=5, kernel_size=3)
    x = nengo_dl.tensor_layer(x, neuron_type)
    x = nengo_dl.tensor_layer(x, tf.layers.dense, units=1)

    out_p = nengo.Probe(x)
    out_p_filt = nengo.Probe(x, synapse=0.1)

minibatch_size = 1
sim = nengo_dl.Simulator(net, minibatch_size=minibatch_size)

# add the single timestep to the training data

train_data = {inp: np.reshape(x_train, (x_train.shape[0], 1, x_train.shape[1] * x_train.shape[2])),
              out_p: np.reshape(y_train, (y_train.shape[0], 1, 1))}

n_steps = 50
test_data = {inp: np.tile(np.reshape(x_test, (x_test.shape[0], 1, x_test.shape[1] * x_test.shape[2])), (1, n_steps, 1)),
             out_p_filt: np.tile(np.reshape(y_test, (y_test.shape[0], 1, 1)), (1, n_steps, 1))}


def objective(outputs, targets):
    return tf.losses.mean_squared_error(predictions=outputs, labels=targets)

opt = tf.train.RMSPropOptimizer(learning_rate=0.001)

sim.train(train_data, opt, objective={out_p: objective}, n_epochs=1000)
sim.run_steps(n_steps, data={inp: test_data[inp][:minibatch_size]})

print(sim.data[out_p][0][-1])
print(sim.data[out_p_filt][0][-1])
sim.close()

Hi @happyjang7,

@drasmuss wrote the tutorial, so I think that he’ll have a more thorough and useful answer than me, but here are a few observations.

Running the model initially gave me these results:

Training finished in 0:01:12 (loss: 15230.5898)                                
Simulation finished in 0:00:00                                                 
[6.475765]
[16.418745]

At the outset, you are constraining your model such that all neuron have the same maximum firing rate (100 Hz) and the same intercept value (0). This means that all neurons have the same tuning properties (i.e., the same gain and bias), which limits the computational power of the network. Additionally, Nengo DL cannot find better values for these properties because you’ve set all values to not be training with nengo_dl.configure_settings(trainable=False). We can remove these limitations by changing the first few lines of your network definition to:

with nengo.Network() as net:
    # net.config[nengo.Ensemble].max_rates = nengo.dists.Choice([100])
    # net.config[nengo.Ensemble].intercepts = nengo.dists.Choice([0])
    neuron_type = nengo.LIF(amplitude=0.01)
    # nengo_dl.configure_settings(trainable=False)

Which results in the following performance:

Training finished in 0:01:20 (loss: 397.2037)                                  
Simulation finished in 0:00:00                                                 
[2.1436608]
[49.544193]

Note the much lower loss value. It is still, however, a very high loss, and not very good performance. This is due to the range of you input/output values. Nengo works best when your input/output values range from -1 to 1; without tuning your network, it is difficult to go outside of this range, especially when the firing rate is constrained, and especially when using the LIF neuron model, which saturates at a certain point (i.e., it has a hard maximum firing rate that cannot be exceeded no matter what you do). Neuron saturation here is an especially bad thing because you’ve set your neuron model to have amplitude of 0.01, which means that each spike is only contributing a small amount to the overall value, so it is going to take a whole lot of spikes in order to raise the value up to 206. If we increase the amplitude of the LIF neuron, then we can achieve much better performance without changing the structure of the network at all.

with nengo.Network() as net:
    # net.config[nengo.Ensemble].max_rates = nengo.dists.Choice([100])
    # net.config[nengo.Ensemble].intercepts = nengo.dists.Choice([0])
    neuron_type = nengo.LIF(amplitude=1)
    # nengo_dl.configure_settings(trainable=False)

Which results in

Training finished in 0:01:21 (loss: 7.2059)                                    
Simulation finished in 0:00:00                                                 
[741.23987]
[90.66766]

Again and improvement, but still not great performance. A final way to get around the issue of neuron saturation is to use a different neuron model. The SpikingRectifiedLinear model is a good choice because it spikes, but can spike at any rate (i.e., it does not saturate).

with nengo.Network() as net:
    # net.config[nengo.Ensemble].max_rates = nengo.dists.Choice([100])
    # net.config[nengo.Ensemble].intercepts = nengo.dists.Choice([0])
    neuron_type = nengo.SpikingRectifiedLinear()
    # nengo_dl.configure_settings(trainable=False)
Training finished in 0:01:15 (loss: 0.1075)                                    
Simulation finished in 0:00:00                                                 
[189.40036]
[54.18062]

However, I will say that these are all relatively superficial changes to improve performance that jumped out at me from looking at the input/output arrays and those first four lines (I haven’t looked at the network architecture at all).

Hi @happyjang7,

If you take a look at the example you linked (https://machinelearningmastery.com/how-to-develop-convolutional-neural-network-models-for-time-series-forecasting/), you’ll notice a couple differences between your model and the one used there. 1) that model uses 64 filters in the convolutional layer, your model uses 5, 2) that model uses a kernel size of 2, your model uses a kernel size of 3. So the first step would be to set up your model to match the example, i.e. change it to

x = nengo_dl.tensor_layer(inp, tf.layers.conv1d, shape_in=(n_steps, n_features), filters=64, kernel_size=2)

The other issue is that because your target output value is quite large (200), it will take longer for the network output to converge to that value (as the spikes are filtered by the synaptic filter on the output probe out_p_filt). So you’ll need to run your network for longer during the test phase in order to get a good measure of the spiking output. All you have to do for that is change this line

n_steps = 300  # (from 50)

With those two changes, I get output that looks like

Training finished in 0:00:35 (loss: 6.6682)                                    
Simulation finished in 0:00:00                                                 
[245.87856]
[183.46375]

which is pretty close (note that it’s the second number we care about).

There are a lot of other things we could do to further improve performance (e.g. the model in the example you linked uses two dense layers and a max pooling layer, while you only have one dense layer after the convolution). And then there are various things you could do specific to spiking deep learning (e.g. using smoothed SoftLIF neurons https://www.nengo.ai/nengo-dl/config.html#lif-smoothing). And, like Trevor mentioned, normalizing your inputs to be in the range (-1, 1) would probably help performance as well. But hopefully that’s enough to get you going, let us know if you have any other questions!