Nengo DL converter

Hi Paul,

We’ve just released an updated version of this example. It uses the new synapse converter argument to do the synapses. Give that a shot and see if it works for you (you’ll need the latest version of the of NengoDL master branch).

As for your error itself, I am able to replicate it. Looking into it a bit, it seems like there are some tensors that get combined by the optimizer, resulting in two smaller tensors rather than one larger one. In one network, they are combined, and in the other, they aren’t, giving problems when trying to load the parameters. What’s strange is that the network where they are combined is actually the one with synapses, which is the opposite of what I’d expect.

If I turn off the optimizer completely for both networks (by putting the code below right after you call the converter each time), then it works the way you originally had it. But it probably makes more sense to just use the new converter synapses argument if possible.

# turn off the optimizer completely, as to not change the number of tensors
with converter.net:
    nengo_dl.configure_settings(simplifications=[])

Yeah i am on it just now, then realised i need to pull the new version on nengo_dl.

I’ll report back shortly

i got one step further so the Synapses are working great now (that is when i don’t run out of RAM, needed to shrink the n_test to like 10 to not get a 15GB spike (i have 32GB on my machine).
Any other advice on how to save space?

But now i have run into the same problem with the scaled firing rate section of the new example code.

nengo.exceptions.SimulationError: Number of saved parameters in keras_to_snn_params (25) != number of variables in the model (24)

i might try the strided conv conversion you mentioned in another forum post previously just to neaten up the network, also avg pooling made my network underperform.

train_images = train_images.reshape((train_images.shape[0], 1, -1))
train_histograms = train_histograms.reshape((train_histograms.shape[0], 1, -1))
test_images = test_images.reshape((test_images.shape[0], 1, -1))
test_histograms = test_histograms.reshape((test_histograms.shape[0], 1, -1))

inp = Input(shape=(7999, 1))

conv1 = Conv1D(64, 3, activation='relu', padding='same', kernel_initializer='he_normal')(inp)
conv1 = Conv1D(64, 3, activation='relu', padding='same', kernel_initializer='he_normal')(conv1)
pool1 = AveragePooling1D(pool_size=(2))(conv1)
conv2 = Conv1D(128, 3, activation='relu', padding='same', kernel_initializer='he_normal')(pool1)
conv2 = Conv1D(128, 3, activation='relu', padding='same', kernel_initializer='he_normal')(conv2)
pool2 = AveragePooling1D(pool_size=(2))(conv2)
conv3 = Conv1D(256, 3, activation='relu', padding='same', kernel_initializer='he_normal')(pool2)
conv3 = Conv1D(256, 3, activation='relu', padding='same', kernel_initializer='he_normal')(conv3)
pool3 = AveragePooling1D(pool_size=(2))(conv3)
conv4 = Conv1D(512, 3, activation='relu', padding='same', kernel_initializer='he_normal')(pool3)
conv4 = Conv1D(512, 3, activation='relu', padding='same', kernel_initializer='he_normal')(conv4)
##### drop4 = Dropout(0.5)(conv4)
pool4 = AveragePooling1D(pool_size=(2))(conv4) # # LOOK TO CHANGE FOR STRIDED CONV INSTEAD WITH FOLLOWING CONV

conv5 = Conv1D(512, 3, activation='relu', padding='same', kernel_initializer='he_normal')(pool4)
conv5 = Conv1D(512, 3, activation='relu', padding='same', kernel_initializer='he_normal')(conv5)
# pool5 = MaxPooling1D(pool_size=(2))(conv5)

# drop5 = Dropout(0.5)(conv5)
# tensorflow.keras.layers.Reshape((3999,)))
flat = Flatten()(conv5)
dense = Dense(units=128)(flat)
dense = Dense(units=1024)(dense)
out = Dense(units=4096)(dense)

model = tf.keras.Model(inputs=inp, outputs=out)




converter = nengo_dl.Converter(model)


# Now we are ready to train the network. It's important to note that we are using standard (non-spiking) ReLU neurons
# at this point.
# 
# To make this example run a bit more quickly we've provided some pre-trained weights that will be downloaded below;
# set `do_training=True` to run the training yourself.

# In[ ]:


do_training = False
if do_training:
    with nengo_dl.Simulator(converter.net, minibatch_size=20) as sim:
        # run training
        sim.compile(
            optimizer=tf.keras.optimizers.Adam(learning_rate=0.0001, beta_1=0.9, beta_2=0.999, amsgrad=False),
            loss="mse",
            metrics=['accuracy'],
        )

        sim.fit(
            {converter.inputs[inp]: train_histograms},
            {converter.outputs[out]: train_images},
            validation_data=(
                {converter.inputs[inp]: test_histograms},
                {converter.outputs[out]: test_images},
            ),
            epochs=100,
        )

        # save the parameters to file
        sim.save_params("./keras_to_snn_params")
else:
    # download pretrained weights
    # urlretrieve(
    #     "https://drive.google.com/uc?export=download&"
    #     "id=1lBkR968AQo__t8sMMeDYGTQpBJZIs2_T",
    #     "keras_to_snn_params.npz")
    print("Loaded pretrained weights")


# After training for 2 epochs the non-spiking network is achieving ~98% accuracy on the test data, which is what we'd
# expect for a network this simple.
# 
# Now that we have our trained weights, we can begin the conversion to spiking neurons. To help us in this process
# we're going to first define a helper function that will build the network for us, load weights from a specified
# file, and make it easy to play around with some other features of the network.

# In[ ]:


def run_network(activation, params_file="keras_to_snn_params", n_steps=30,
                scale_firing_rates=1, synapse=None, n_test=10):
    # convert the keras model to a nengo network
    nengo_converter = nengo_dl.Converter(
        model,
        swap_activations={tf.nn.relu: activation},
        scale_firing_rates=scale_firing_rates,
        synapse=synapse,
    )
    
    # get input/output objects
    nengo_input = nengo_converter.inputs[inp]
    nengo_output = nengo_converter.outputs[out]
    
    # add a probe to the first convolutional layer to record activity
    with nengo_converter.net:
        conv1_probe = nengo.Probe(nengo_converter.layers[conv1])
            
    # repeat inputs for some number of timesteps ##### Could edit this to do sequence data with LMUs
    tiled_test_histograms = np.tile(test_histograms[:n_test], (1, n_steps, 1))
    
    # set some options to speed up simulation
    with nengo_converter.net:
        nengo_dl.configure_settings(stateful=False)
            
    # build network, load in trained weights, run inference on test histograms
    with nengo_dl.Simulator(
            nengo_converter.net, minibatch_size=10, 
            progress_bar=True) as nengo_sim:
        nengo_sim.load_params(params_file)
        data = nengo_sim.predict({nengo_input: tiled_test_histograms})
        
    # compute accuracy on test data, using output of network on
    # last timestep

    predictions = data[nengo_output][:, -1]
    mse = ((predictions - test_images[:n_test, 0, :])**2).mean()
    print("###################################################################################################################################################")
    print("\nTest Batch MSE: %.5f%%\n" % mse)
    print("Activation Type: %s" % activation)
    print("Number of Steps: %d" % n_steps)
#####################################################################################################################
    # plot the results


    for ii in range(4):
        fig = plt.figure(figsize=(12, 4))

        gs0 = gridspec.GridSpec(1, 3)
        gs00 = gridspec.GridSpecFromSubplotSpec(3, 3, subplot_spec=gs0[2])
        gs01 = gridspec.GridSpecFromSubplotSpec(2, 1, subplot_spec=gs0[0])
        # plt.figure(figsize=(12, 4))
        
        plt.subplot(gs01[0])
        plt.title("Input Histogram")
        plt.plot(test_histograms[ii, 0])
        plt.axis('off')
        plt.subplot(gs01[1])
        plt.title("GT image")
        plt.imshow(test_images[ii, 0].reshape(64, 64), cmap='viridis_r', vmin=0.02, vmax=1)
        plt.axis('off')

        plt.subplot(1, 3, 2)
        sample_neurons = np.linspace(
            0,
            data[conv1_probe].shape[-1],
            1000,
            endpoint=False,
            dtype=np.int32,
        )
        scaled_data = data[conv1_probe][ii, :, sample_neurons].T * scale_firing_rates
        if isinstance(activation, nengo.SpikingRectifiedLinear):
            scaled_data *= 0.001
            rates = np.sum(scaled_data, axis=0) / (n_steps * nengo_sim.dt)
            plt.ylabel('Number of spikes')
        else:
            rates = scaled_data
            plt.ylabel('Firing rates (Hz)')
        plt.xlabel('Timestep')
        plt.title(
            "Neural activities (conv1 mean=%dHz max=%dHz)" % (
                rates.mean(), rates.max())
        )
        plt.plot(scaled_data)

        for xx in range(3):
            for yy in range(3):
                plt.subplot(gs00[xx, yy])
                plt.title("Output predictions of ")
                # imgn = (3 * xx + yy + 1) * (n_steps / 10) + (n_steps / 10 - 1)
                plt.imshow((data[nengo_output][ii, int((3*xx+yy+1)*(n_steps/10)+(n_steps/10 -1))]).reshape(64, 64), cmap='viridis_r', vmin=0.02, vmax=1)
                # plt.legend([str(j) for j in range(10)], loc="upper left")
                # plt.xlabel('Timestep')
                # plt.ylabel("Probability")
                plt.axis('off')
            
        plt.tight_layout()
    plt.show()



for scale in [2, 10, 20]:
    print("Scale=%d" % scale)
    run_network(
        activation=nengo.SpikingRectifiedLinear(),
        scale_firing_rates=scale,
        synapse=0.01
    )
    # plt.show()

Two potential workarounds that I can see:

  1. Turn off the optimizer, as I described above. It might be that it’s just one of the optimization rules that is actually causing the problem, so you could play around with turning on some simplifications but not others. See here for the docs, and here for the list of default simplifications in code (at least one of which is the problem).
  2. Train the model in Keras, rather than in NengoDL. Then you don’t have to do saving and loading at all, or if you do, you can do it in Keras. Then, once it’s trained, run it through the converter to get your Nengo network.
  3. EDIT: A third one I just thought of, but haven’t tried, would be to put an almost 1 (e.g. 1.000001) value for scale_firing_rates the first time you call the converter (i.e. before training your network). That should have almost no effect on the network, but should be enough to keep the optimizer from touching those tensors. If you try this, let me know if it works.

We’ll look more into this problem, and see if there’s a more permanent fix on our side.

As for advice on how to save space, one thing you could try is dropping the minibatch size. It’s not large in either case, though, and n_steps or your number of dimensions (7999) aren’t too large either. So I’m not sure why you’re using so much memory.

With regards to max/average pooling, I avoid it completely now, and just use convolution with a stride of 2 (which is pretty much the same as convolution then average pooling, just more efficient). That said, they’re pretty much the same, so I wouldn’t expect that change to fix any performance problems. I’m not sure why switching max pooling to average pooling would be very detrimental to your performance, other than your network is relying on it more than most classic vision networks do (the All Conv paper basically showed that it’s not necessary for standard vision networks).

1 Like

Quick reply, both #1 and #3 works in the meantime, going to try #2 along with removing of pooling. I do agree striding or dilation seem like more efficient approaches to “pooling”.

Thanks very much. Plenty to work on now…Also, the new example is pretty neat so great timing with that.
:nerd_face: :nerd_face:

Hey,

So been playing with my model and its grew arms and legs now.
but i cant seem to solve a simple keras to nengo converter issue, dunno if im just missing the obv

but i am getting a
File "/home/paul/.local/lib/python3.6/site-packages/nengo_dl/converter.py", line 401, in __getitem__ return self.dict[self._get_key(key)] KeyError: <Reference wrapping <tf.Tensor 'input_1:0' shape=(None, 7999, 1) dtype=float32>>

which I don’t quite understand… i have just removed the if do training else use preloaded weights sections and just call a pretrained keras model as model and put it into the converter.

What else would I need to change in the run_network func ??

Hi Paul,

Can you attach your newest code that exhibits the problem? What do you mean by “call a pretrained keras model as model”? One possibility is that when you call nengo_converter.inputs[inp], the input inp that you’re using isn’t actually a part of the Keras model that you’ve passed to the converter. Is that the line in your script that’s causing the problem?

So essentially I’m trying to do the training outside of nengo now just to see and call the model in the converter

looking at my code slimmed down perhaps it’s odd there is a model = followed by another.

train_images = train_images.reshape((train_images.shape[0], 1, -1))
train_histograms = train_histograms.reshape((train_histograms.shape[0], 1, -1))
test_images = test_images.reshape((test_images.shape[0], 1, -1))
test_histograms = test_histograms.reshape((test_histograms.shape[0], 1, -1))



test_images = np.reshape(test_images, (test_images.shape[0], 1, 64, 64))
train_images = np.reshape(train_images, (train_images.shape[0], 1, 64, 64))

inp = Input(shape=(7999, 1))

conv1 = Conv1D(64, 3, activation='relu', padding='valid', kernel_initializer='he_normal')(inp)
conv1 = Conv1D(64, 3, activation='relu', padding='valid', kernel_initializer='he_normal')(conv1)
conv2 = Conv1D(128, 3, activation='relu', padding='valid', kernel_initializer='he_normal', strides=2)(conv1)
conv2 = Conv1D(128, 3, activation='relu', padding='valid', kernel_initializer='he_normal')(conv2)

conv3 = Conv1D(256, 3, activation='relu', padding='valid', kernel_initializer='he_normal', strides=2)(conv2)
conv3 = Conv1D(256, 3, activation='relu', padding='valid', kernel_initializer='he_normal')(conv3)
conv4 = Conv1D(512, 3, activation='relu', padding='valid', kernel_initializer='he_normal', strides=2)(conv3)
conv4 = Conv1D(512, 3, activation='relu', padding='valid', kernel_initializer='he_normal')(conv4)

conv5 = Conv1D(512, 3, activation='relu', padding='valid', kernel_initializer='he_normal', strides=2)(conv4)
conv5 = Conv1D(512, 242, activation='relu', padding='valid', kernel_initializer='he_normal')(conv5)

shape = tensorflow.keras.layers.Reshape((16, 16, 512))(conv5)

tconv1 = Conv2D(512, 3, activation='relu', padding='same', kernel_initializer='he_normal')(shape)
tconv1 = UpSampling2D(size=(2, 2))(tconv1)
tconv1 = Conv2D(512, 3, activation='relu', padding='same', kernel_initializer='he_normal')(tconv1)

tconv2 = Conv2D(256, 3, activation='relu', padding='same', kernel_initializer='he_normal')(tconv1)
tconv2 = UpSampling2D(size=(2, 2))(tconv2)
tconv2 = Conv2D(256, 3, activation='relu', padding='same', kernel_initializer='he_normal')(tconv2)

tconv3 = Conv2D(128, 3, activation='relu', padding='same', kernel_initializer='he_normal')(tconv2)
tconv3 = UpSampling2D(size=(2, 2))(tconv3)
tconv3 = Conv2D(128, 3, activation='relu', padding='same', kernel_initializer='he_normal')(tconv3)

tconv4 = Conv2D(64, 3, activation='relu', padding='same', kernel_initializer='he_normal')(tconv3)


out = Conv2D(1, 3, padding='same', kernel_initializer='he_normal')(tconv4)  # try without relu here?? activation='relu',

model = tensorflow.keras.Model(inputs=inp, outputs=out)

model = tensorflow.keras.models.load_model('upsample.h5')

converter = nengo_dl.Converter(model, scale_firing_rates=1.0001)





def run_network(activation, params_file="Keras_SNN_LIDAR_params2", n_steps=30,
                scale_firing_rates=1.0001, synapse=None, n_test=10):
    # convert the keras model to a nengo network
    nengo_converter = nengo_dl.Converter(
        model,
        swap_activations={tensorflow.nn.relu: activation},
        scale_firing_rates=scale_firing_rates,
        synapse=synapse,
    )
    # # turn off the optimizer completely, as to not change the number of tensors
    # with converter.net:
    #     nengo_dl.configure_settings(simplifications=[])

    # get input/output objects
    nengo_input = nengo_converter.inputs[inp]
    nengo_output = nengo_converter.outputs[out]

    # add a probe to the first convolutional layer to record activity
    with nengo_converter.net:
        conv1_probe = nengo.Probe(nengo_converter.layers[conv1])

    # repeat inputs for some number of timesteps ##### Could edit this to do sequence data with LMUs
    tiled_test_histograms = np.tile(test_histograms[:n_test], (1, n_steps, 1))

    # set some options to speed up simulation
    with nengo_converter.net:
        nengo_dl.configure_settings(stateful=False)

    # build network, load in trained weights, run inference on test histograms
    with nengo_dl.Simulator(
            nengo_converter.net, minibatch_size=10,
            progress_bar=True) as nengo_sim:
        # nengo_sim.load_params(params_file)
        data = nengo_sim.predict({nengo_input: tiled_test_histograms})

    # compute accuracy on test data, using the output of network on
    # last timestep

    predictions = data[nengo_output][:, -1]
    mse = ((predictions - test_images[:n_test, 0, :]) ** 2).mean()

When you call tensorflow.keras.models.load_model, it’s loading the whole model, including all the tensors inside it. So the layers you’ve defined earlier (including inp) aren’t actually part of that new model; it all gets overwritten by the second model = ....

So I would get rid of the whole model definition before the second model = (or at least comment it out for now). Then you still need a way to get those input and output tensors. You should be able to do inp = model.inputs[0] and out = model.outputs[0] to get handles to those input and output tensors.

Ahh okay, I get my mistake. Similarly, I take it model.layers[x] is how I refer to any of the Conv layers for probes then.

Yes, exactly.

An alternative is to define your model as usual, and then use model.save_weights and model.load_weights to save and load the weights. I’m not sure if that will work with your existing weights (i.e. if you can just load the weights from a checkpoint that’s been created with save); I feel like it should (I think the .h5 file is all the weights), but I’ve never tried. But if you retrain the model and save the weights with save_weights, that should definitely work. The downside is you’ve got to always make sure that your model definition matches the saved weights, since none of the information about your model architecture/definition is saved.

Yeah i already tried that, seem to work well.

When i run the SReLU, i dont get spike output from the probe, i haven’t been able to either way i tried, with all the models shown above. i just seems to show the same output as the relu version.

i also still get a few of the 0) Resource exhausted: OOM when allocating tensor with shape[10,120,511680] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[{{node TensorGraph/transpose_1}}]] issue, but I think I need to come up with ways of resetting everything after each simulation

plt.subplot(1, 3, 2)
        sample_neurons = np.linspace(
            0,
            data[conv1_probe].shape[-1],
            1000,
            endpoint=False,
            dtype=np.int32,
        )
        scaled_data = data[conv1_probe][ii, :, sample_neurons].T * scale_firing_rates
        if isinstance(activation, nengo.SpikingRectifiedLinear):
            scaled_data *= 0.001
            rates = np.sum(scaled_data, axis=0) / (n_steps * nengo_sim.dt)
            plt.ylabel('Number of spikes')
        else:
            rates = scaled_data
            plt.ylabel('Firing rates (Hz)')
        plt.xlabel('Timestep')
        plt.title(
            "Neural activities (conv1 mean=%dHz max=%dHz)" % (
                rates.mean(), rates.max())
        )
        plt.plot(scaled_data)
        plt.title("Output Predictions of Image Over Time")

What it looks like from your plot is that your output is just your biases, i.e. no spikes are making it through your network. Try using scale_firing_rates to scale things up.

Yeah the same thing seems to happen even at these scaled versions of 2 and 20

and when i try the regularizor i get

File "/home/paul/PycharmProjects/LIDAR/keras-snn.py", line 612, in <module> conv1_p: np.ones((train_images.shape[0], 1, conv1_p.size_in))
File "/home/paul/.local/lib/python3.6/site-packages/numpy/core/numeric.py", line 207, in ones a = empty(shape, dtype, order)

MemoryError: Unable to allocate 43.5 GiB for an array with shape (11402, 1, 511680) and data type float64

I seen for the recent INRC meeting that y’all made a spiking unet for segmentation ?
So i believe in the Nengo way!

can get similar failure to spike and only biases in the example by adding a good few conv layers.

What does it mean by the output just being bias, is that similar to overfitting from the model?

No. The problem is that when you train an ANN, it doesn’t care about what the scale of the outputs are if you’re training with ReLUs. If the range of the outputs for a layer is [0, 1] or if it’s [0, 100] doesn’t make a difference to the network, because the subsequent layer can just e.g. scale down its weights by a factor of 100 in the case where the range is [0, 100]. Furthermore, the ANN could have different ranges for different layers.

So if your ANN happens to choose a lower range for a layer (e.g. [0, 3]) then when you switch to spiking neurons, the neurons in that layer are going to fire between 0 and 3 spikes a second; not very many.

The first thing I would try is higher firing rates scaling. Go up to 100, 1000, or even higher, just to see if you can get some output.

The more principled way to do it is to actually measure the outputs of different layers, to see how quickly neurons are firing. In the example notebook, we do this by looking at the number of non-zero timesteps (i.e. the percent of timesteps that the neuron has a non-zero output), and turning that into a firing rate.

For U-Net and other larger networks that we’ve made, we always use the regularization method, because it provides more exact control over the firing rates, and works for LIF neurons (the firing rate scaling only works for ReLUs because it depends on the fact that they’re scale-invariant).

Looking more at your memory error, I see why you’re getting it now. The firing rate regularization in that example has an inefficient way of passing the firing rate targets; since you have the same target for all neurons, there’s no need to pass a giant array where every element is the same value. Take a look at how I do firing rate regularization in the reworked CIFAR-10 example here.

Amazing, Thanks for all the help.

My goal is to convert to LIF next, but i need to get the regularizer working anyway, i will muster on with that from the loihi example, i mean thats where i want to port the model in the end also, but ways to go!.

I had tried scaling upto 1000 without any difference, but i’ll try a few different things.

Cheers

Hi Paul!

Not sure if you are still having this issue of no spikes in the network, but I was running into the same problem trying to convert a u-net to spiking. I finally realized that the converter wasn’t actually converting the activation functions to SpikingRectictifiedLinear() (i.e. for ensemble in nengo_converter.net.ensembles: print(ensemble, ensemble.neuron_type) was showing all RectifiedLinear() outputs. To fix it, I had to switch over from using from keras.layers.Conv2D to tf.keras.layers.Conv2D. I also needed to switch from activation='relu' to activation='tf.nn.relu'. Small changes, but I think it had to do with the converter not recognizing the activation function.

1 Like

Hello @msanch35! You are correct, if you use activation="relu" and then swap it with following statement:

swap_activations={tf.nn.relu: nengo.SpikingRectifiedLinear()}

it won’t swap it. Rather you should use the following:

swap_activations={tf.keras.activations.relu: nengo.SpikingRectifiedLinear()}

while using “relu” in non-spiking network. This is adapted from Conversion of sequential model.

1 Like

Thanks so much to both @msanch35 and @zerone !!
That was exactly my problem, now if I scale the firing rate to like 10k I get actual spiking activity.

I had taken a while off the project until I had more time to implement a few things to try and get it working, but thanks to the magic of the forum it solved itself! (or well someone else solved it for me) :smiley:

now I might actually get to the LIF neuron stage

2 Likes