Loading saved keras model into NengoDL and converting it to Spiking network

Good Day,

I have a keras model which is trained and then converted to spiking network in a similar way as shown in the official keras to nengo example.
But the problem is when i enlarge the number of my hidden layers, train the model and save it using:
sim.save_params
then load it into nengo network, nengo simulator fails with error:
nengo.exceptions.SimulationError: Number of saved parameters in != number of variables in the model.

I tried methods #1 and #3 suggested in this thread for similar issue:

But they don’t work for my network.
All is left now is to try solution number #2, have all the training done in pure keras.

So my question is, once i do all the model training in keras and save it using:
keras.models.save_model,
and load it afterwards when i need it with:
keras.models.load_model

how can i still pass this trained keras model to converter, then convert it to spiking network
with code like this:

nengo_converter = nengo_dl.Converter(
        model,
        swap_activations={tf.nn.relu: nengo.SpikingRectifiedLinear()},
        scale_firing_rates=my_firing_rates,
        synapse=my_synapse,
    )

and then be able to Predict this model without training, but using the pre trained keras weights.

Thank you!

Best Regards,
Alex

Good day @alex.hex! :smiley:

Regarding your question:

The NengoDL converter should work on your pre-trained TF model without having to do anything extraneous. Essentially, you should be able to:

tf_model.load_weights(my_weights)
converter = nengo_dl.Converter(tf_model, ...)

Doing the above should give you a converted NengoDL model with the weights you obtained in the Keras training.

I’m not entirely sure what would be the issue here. If you retrain your model after modifying the model parameters (size, number of hidden layers, etc.) and then save and subsequently load the saved weights (for the modified model), you should not encounter any errors when loading the saved parameters. If you could post some code that demonstrates this behaviour, that would be very helpful! :slight_smile:

Hi @xchoo,

Thank you for your quick reply!

But my intention is not to insert keras model as it is into nengo network, but replace the entire model relu with nengo.SpikingRectifiedLinear() and use the ex-keras trained model as a spiking network model.
If i understand correctly inserting keras model into nengo brings the model inside as is, without conversion to spikes. Meaning the network will have a tensorflow non spiking part inside.

I have a model which i do convert to spiking network, but because an issue with nengodl simulator save and load params i am unable to make any customizations to the number of my hidden layers, anything beyond 2 layers results in loading parameters mismatch as described above.
my model is the same one discussed in this topic:

but when i try to simply add to it another hidden layer:
input = tf.keras.Input(shape=(3,))
l1 = tf.keras.layers.Dense(50,activation=tf.nn.relu)(input)
l2 = tf.keras.layers.Dense(50,activation=tf.nn.relu)(l1)
l3 = tf.keras.layers.Dense(50,activation=tf.nn.relu)(l2)
output = tf.keras.layers.Dense(5,activation=tf.nn.relu)(l3)

simulator will refuse to use it after the training during load next time.

So it seems that i cannot use use the keras model as is, since i need a full spiking network and i can’t extend it using nengodl approach.

Best Regards,
Alex

Ah yes. Apologies. I misread your original inquiry and have updated my response.

Hi @xchoo,

I see:
does this code makes sense to you, should it work? (model is pretrained saved keras model)

def RunNetwork(
    **model**,
    inputTest,
    outputTest,
    activation,
    params_file=MODEL_FILENAME,
    n_steps=NENGO_STEPS,
    scale_firing_rates=1,
    synapse=None,
    n_test=15000,
    showPlot=False,
    hlayers=2
):

    # convert the keras model to a nengo network
    **model.load_weights('my_weights.model')**
    nengo_converter = nengo_dl.Converter(
        model,
        swap_activations={tf.nn.relu: activation},
        scale_firing_rates=scale_firing_rates,
        synapse=synapse,
    )

    inputLayer = model.get_layer('input_1')
    hiddenLayer1 = model.get_layer('dense') #first hidden layer
    outputLayer = model.get_layer("dense_" + str(hlayers)) #output layer

    # get input/output objects
    nengo_input = nengo_converter.inputs[inputLayer]
    nengo_output = nengo_converter.outputs[outputLayer]

    # add a probe to the first  layer to record activity.
    sample_neurons = np.linspace(
        0,
        np.prod(hiddenLayer1.output_shape[1:]),
        n_steps,
        endpoint=False,
        dtype=np.int32,
    )

    with nengo_converter.net:
        probe_l1 = nengo.Probe(nengo_converter.layers[hiddenLayer1][sample_neurons])

    # repeat inputs for some number of timesteps
    testInputData = np.tile(inputTest[:n_test], (1, n_steps, 1))

    with nengo.Network() as net:
        nengo_dl.configure_settings(stateful=False)

    # build network, load in trained weights, run inference on test images
    with nengo_dl.Simulator(
        nengo_converter.net, minibatch_size=BATCH_SIZE, progress_bar=False,
    ) as nengo_sim:
        **#nengo_sim.load_params(params_file) # not needed anymore** 
        **data = nengo_sim.predict({nengo_input: testInputData})**

In Simulator all i do is predict, and the weights are already counted in?

A quick lookover your code doesn’t reveal anything that would seem to cause it to throw the nengo.exceptions.SimulationError you mentioned in your first post. As long as the model you provide the RunNetwork function has the same network configuration as the network that generated the my_weights.model data, it should work fine.

If you are still encountering the error, could your provide some code that I can run on my local system? That way, I can verify your observed behaviour on my system, and debug it from there. You can attach your code as a .py file or Jupyter notebook (.ipynb) to your post if the code is too long or complex to copy & paste into the post’s text.

Also note that you’ll want to try and provide a minimal example that exhibits the issues you are facing. Reducing the complexity of your model will speed up the debugging process. :slight_smile:

Hi @xchoo,

Attaching a simple py file which demonstrates the error right away:
nengo.exceptions.SimulationError: Number of saved parameters in ./nengo-model-relu-1 (5) != number of variables in the model (4)

nengo_test1.py (8.2 KB)

if you will replace the existing non working 3 hidden layer model:
input = tf.keras.Input(shape=(3,))

l1 = tf.keras.layers.Dense(50,activation=tf.nn.relu)(input)

l2 = tf.keras.layers.Dense(50,activation=tf.nn.relu)(l1)

l3 = tf.keras.layers.Dense(50,activation=tf.nn.relu)(l2)

output = tf.keras.layers.Dense(5,activation=tf.nn.relu)(l3)

With just 2 hidden layers, all goes well:
input = tf.keras.Input(shape=(3,))

l1 = tf.keras.layers.Dense(50,activation=tf.nn.relu)(input)

l2 = tf.keras.layers.Dense(50,activation=tf.nn.relu)(l1)

output = tf.keras.layers.Dense(5,activation=tf.nn.relu)(l2)

Best Regards,
Alex

Righto! We took a look at your code, and we have identified what is going on with it. Part of the NengoDL model compilation process (i.e., when you make the NengoDL simulator object) is an optimization step that tries to minimize the amount of TF operations needed to be done. With your two models, it looks like that by specifying a synapse in the NengoDL converter, the synapses add enough operations to the model to make the optimized result different, resulting in the mismatch error.

Fixing your model is straightforward, you’ll want to disable the optimizer where ever the NengoDL simulator object is created. This is done like so:

In your ConstructNetworkAllInOne function:

...
    net = converter.net
    nengo_input = converter.inputs[input]
    nengo_output = converter.outputs[output]
    nengo_prob1 = converter.layers[l1]

    with net:
        # Disable the NengoDL graph optimizer
        nengo_dl.configure_settings(planner=nengo_dl.graph_optimizer.noop_planner)

    # run training
    with nengo_dl.Simulator(net, minibatch_size=100, seed=0) as sim:

        sim.compile(
            optimizer=tf.optimizers.SGD(),
            loss={nengo_output: tf.losses.mse},
        )

        sim.fit(
            inputTrainNgo,
            {nengo_output: outputTrainNgo},
            validation_data=(inputValidationNgo, outputValidationNgo),
            epochs=1,
        )
        sim.save_params("./nengo-model-relu-1")
...

And also in your RunNetwork function:

...
    # set some options to speed up simulation
    with nengo_converter.net:
        nengo_dl.configure_settings(planner=nengo_dl.graph_optimizer.noop_planner)
        nengo_dl.configure_settings(stateful=False)

    # build network, load in trained weights, run inference on test
    with nengo_dl.Simulator(
        nengo_converter.net, minibatch_size=100, progress_bar=False
    ) as nengo_sim:

        nengo_sim.load_params(params_file)
        data = nengo_sim.predict({nengo_input: testInputData})
...

By disabling the graph optimizer, the models generated with and without the synapse should be structurally similar to allow the load_params to work.

Some Notes
Regarding the save_params and load_params functionality, it is only really guaranteed to work in the case where the two networks are structurally exactly identical. Such things like adding synapses modifies the structure of the network enough that this guarantee no longer holds true. For more complex models, if you want to save and load network parameters, you’ll need to manually extract and the subsequently load the network parameters. You can access the network parameters using the sim.data functionality (see here).

I should also note that for your specific example code, because you are using the converter on a Keras model, and then doing the fit right after (without any making any NengoDL-specific additions to the model), it will probably be less complex to do the training directly in TensorFlow itself. The NengoDL simulator fit function is, afterall, pretty much just a wrapper around the TensorFlow fit function.

If you train directly in TensorFlow, the work flow would go something like this:

  1. Create TensorFlow model.
  2. Train TensorFlow model using TensorFlow fit function.
  3. Use NengoDL converter on trained TensorFlow model.
  4. Run NengoDL simulator on converted NengoDL network.

You can repeat steps 3 & 4 with spiking neurons, or whatever activation functions you desire. :smiley:

1 Like

Hi @xchoo,

Thank you for your help!
Will disabling the NengoDL graph optimizer affect network accuracy,
Or is it just an internal speed related optimization?
with net:
# Disable the NengoDL graph optimizer
nengo_dl.configure_settings(planner=nengo_dl.graph_optimizer.noop_planner)

Disabling the graph optimizer should not affect network accuracy, but may affect the running speed of your simulation. However, given the structure of your model (it’s only using dense layers), the speed shouldn’t be affected that much.