Nengo ensembles vs nengo dl

Deep Learning
1

Hey, just a quick question, I seem to get different results with the ensembles and dl models that I think are similar. I mean I’m not 100% sure if they are identical, I followed the advice on a previous forum post - Need to download weights of fully connected network - below is the code and output images.

I was also looking at how to improve my model (i plan to transition into a cnn based model later, so just like stuff in nengo i haven’t utilised in the models shown) and/or move from SReLU to LIF but the performance was awful in comparison. i looked at the studywolf blog, but since it is using the converter, i am unsure how to replicate the “changing of the gains” value within the networks i have made.

Also just to ask all the questions, if I am looking to parse the data to feed it in sequentially, as it is histogram data i want it to retain the previous input and use the new input to help get a better results, should i look toward the LMUs to help retain this information?

with nengo.Network(seed=0) as net:
net.config[nengo.Ensemble].max_rates = nengo.dists.Choice([1000])
net.config[nengo.Ensemble].intercepts = nengo.dists.Choice([0])
net.config[nengo.Connection].synapse = None
neuron_type = nengo.SpikingRectifiedLinear(amplitude=0.001)
nengo_dl.configure_settings(stateful=False)

inp = nengo.Node(np.zeros(7999))

hidden = nengo_dl.Layer(
    tf.keras.layers.Dense(units=1024, activation=tf.nn.relu))(inp)
hidden = nengo_dl.Layer(neuron_type)(hidden)

hidden = nengo_dl.Layer(
    tf.keras.layers.Dense(units=512, activation=tf.nn.relu))(hidden)
hidden = nengo_dl.Layer(neuron_type)(hidden)

hidden = nengo_dl.Layer(
    tf.keras.layers.Dense(units=256, activation=tf.nn.relu))(hidden)
hidden = nengo_dl.Layer(neuron_type)(hidden)

hidden = nengo_dl.Layer(
    tf.keras.layers.Dense(units=4096, activation=tf.nn.relu))(hidden)

out = nengo_dl.Layer(tf.keras.layers.Dense(units=4096))(hidden)

out_p = nengo.Probe(out, label="out_p")
out_p_filt = nengo.Probe(out, synapse=0.01, label="out_p_filt")

Figure_1
Figure_11920×960 161 KB

Figure_2
Figure_21920×960 196 KB

one set of images is slightly more refined than the other like I am taking an extra step or smoothing the spikes somewhere

with nengo.Network(seed=0) as net:

net.config[nengo.Ensemble].max_rates = nengo.dists.Choice([1000])
net.config[nengo.Ensemble].intercepts = nengo.dists.Choice([0])
net.config[nengo.Connection].synapse = None
neuron_type = nengo.SpikingRectifiedLinear(amplitude=0.001)

nengo_dl.configure_settings(stateful=False)

inp = nengo.Node(np.zeros(7999))

hidden = nengo_dl.Layer(
    tf.keras.layers.Dense(units=1024, activation=tf.nn.relu))(inp)
hidden = nengo_dl.Layer(neuron_type)(hidden)

hidden = nengo_dl.Layer(
    tf.keras.layers.Dense(units=512, activation=tf.nn.relu))(hidden)
hidden = nengo_dl.Layer(neuron_type)(hidden)

hidden = nengo_dl.Layer(
    tf.keras.layers.Dense(units=256, activation=tf.nn.relu))(hidden)
hidden = nengo_dl.Layer(neuron_type)(hidden)

hidden = nengo_dl.Layer(
    tf.keras.layers.Dense(units=4096, activation=tf.nn.relu))(hidden)

out = nengo_dl.Layer(tf.keras.layers.Dense(units=4096))(hidden)

out_p = nengo.Probe(out, label="out_p")
out_p_filt = nengo.Probe(out, synapse=0.01, label="out_p_filt")
2

Hi Paul,

Can you describe more what the difference is between the two models you are running, and what the difference is you are seeing in their output? The two code samples you posted look the same to me (but I might be missing the difference), and the two sets of plots also look very similar (e.g., within the bounds of the variability I might expect to see due to different random initializations).

Under the hood what the converter is doing is scaling all the gains and biases of the Ensembles by x, and scaling the amplitude by 1/x. You can see the code implementing this here https://github.com/nengo/nengo-dl/blob/master/nengo_dl/converter.py#L512, as an example. You would want to do something similar in your own code if you wanted to implement the gain scaling manually.

Yes, LMUs would definitely be a good thing to try if you’re looking to work with time series data.

3

ooops just realised that my second code was just the first one, copy and paste failure. so it was supposed to be the ensembles. it was just to check i wasn’t adding another activation function by mistake.
In general the ensembles code as seen below delivers an end image with more background noise in comparison to the DL version (seen above) almost as if the DL version is running an extra relu or something somewhere

with nengo.Network(seed=seed) as net:
    # set up some default parameters to match the Keras defaults
    # net.config[nengo.Ensemble].gain = nengo.dists.Choice([1])
    # net.config[nengo.Ensemble].bias = nengo.dists.Choice([0])
    # net.config[nengo.Connection].synapse = None
    net.config[nengo.Connection].transform = nengo_dl.dists.Glorot()

    net.config[nengo.Ensemble].max_rates = nengo.dists.Choice([1000])
    net.config[nengo.Ensemble].intercepts = nengo.dists.Choice([0])
    net.config[nengo.Connection].synapse = None
    NN_neuron_type = nengo.SpikingRectifiedLinear(amplitude=0.001)


    # this is an optimization to improve the training speed,
    # since we won't require stateful behaviour in this example
    nengo_dl.configure_settings(stateful=False)


    # input node, same as before
    inp = nengo.Node(output=np.ones(7999))

    # add the first dense layer
    hidden = nengo.Ensemble(1024, 1, neuron_type=NN_neuron_type).neurons
    nengo.Connection(inp, hidden)

    hidden2 = nengo.Ensemble(512, 1, neuron_type=NN_neuron_type).neurons
    nengo.Connection(hidden, hidden2)

    hidden3 = nengo.Ensemble(254, 1, neuron_type=NN_neuron_type).neurons
    nengo.Connection(hidden2, hidden3)

    hidden4 = nengo.Ensemble(4096, 1, neuron_type=NN_neuron_type).neurons
    nengo.Connection(hidden3, hidden4)

    # add the linear output layer (using nengo.Node since there is
    # no nonlinearity)
    out = nengo.Node(size_in=4096)
    nengo.Connection(hidden4, out)

    # add a probe to collect output
    out_p = nengo.Probe(out)
    out_p_filt = nengo.Probe(out, synapse=0.1, label="out_p_filt")
4

Hi Paul,

The main difference that I can see, is that in the example with Ensembles, your last hidden layer (hidden4) is spiking; however, in your example with Layers, it’s not (you don’t call nengo_dl.Layer(neuron_type) after this layer). So in the Ensembles example, you have 4 spiking layers, whereas in the Layers example, you only have 3 (and one rate layer). This would cause some extra noise in the output.

5

Thanks!
yeah now i see it, couldn’t see the wood for the trees.