[Nengo-DL]: How to do batch training of a nengo-dl model?

Deep Learning
1

Hello everyone!

I am currently stuck at passing the training data in custom batches to the Nengo-DL Simulator. Actually I am coming from this following related problem. I was of the opinion that I can train the network in TF-Keras, save the intermediate weights, load them again (in the same TF-Keras environment) in a model, and then use nengo_dl.Converter() to convert the trained model to a spiking model, and then proceed with inference in the simulator. But looks like it doesn’t happen this way. Can someone please shed some information about this? From the tutorial, it seems that nengo-dl lib expects the weights to be probably obtained after training a spiking model in its own environment (not in TF-Keras).

Therefore, I converted the TF-Keras model architecture and attempted to train it using nengo-dl (as done in the tutorials). Please note that I cannot use the minibatch_size option in Simulator since my training dataset is too large to be loaded wholly in the memory, thus… my custom batches. This API doc presents the option to pass a custom generator (of training data and labels) to x, however I did not find any related example to do it. Can someone please show me how to code it?

For reference, this is how my generator looks like:

def get_batches():
  for start in range(0, X_train_resh.shape[0], 128):
    end = min(start+128, X_train_resh.shape[0])
    yield (X_train_resh[start:end], targets_train_resh[start:end])

Please let me know if you need my model training code (in nengo-dl). Thanks!

2

Tensorflow-Keras model to NengoDL Spiking Model:

  • It is not necessary to train the model using NengoDL to convert it to a spiking model.
    In order to reuse the weight from TF-Keras training, load the model weights before converting the model using nengo_dl.Converter().
  • In order to convert the TF-Keras model to Spiking model, it would be required to replace the non-linearity of the model by spiking non-linear function e.g. nengo.SpikingRectifiedLinear(). It would be also needed to specify presentation time, synapse.
  • In order to achieve desired performance, the firing rates of the network may need to be scaled by either using scale_firing_rates or regularising firing rates during training.

The example described below converts a keras model to spiking model with scale_firing_rates:

n_steps = 200
scale_firing_rate = 100
synapse = 0.005

inp = tf.keras.Input(shape=(28, 28, 1))

# convolutional layers
conv0 = tf.keras.layers.Conv2D(
    filters=32,
    kernel_size=3,
    activation=tf.nn.relu,
)(inp)

conv1 = tf.keras.layers.Conv2D(
    filters=64,
    kernel_size=3,
    strides=2,
    activation=tf.nn.relu,
)(conv0)

# fully connected layer
flatten = tf.keras.layers.Flatten()(conv1)
dense = tf.keras.layers.Dense(units=10)(flatten)

model = tf.keras.Model(inputs=inp, outputs=dense)

# Load Model Weights
model.load_weights('tf_keras_weights.tf')

# Convert keras model to Spiking model
converter = nengo_dl.Converter(model,
        swap_activations={tf.nn.relu: nengo.SpikingRectifiedLinear()},
        scale_firing_rates=scale_firing_rate,
        synapse=synapse)

# get input/output objects
nengo_input = converter.inputs[inp]
nengo_output = converter.outputs[dense]

# add a probe to the first convolutional layer to record activity
with converter.net:
    conv0_probe = nengo.Probe(converter.layers[conv0])

data_generator = get_batches()

# set some options to speed up simulation
with converter.net:
    nengo_dl.configure_settings(stateful=False)

# build network, load in trained weights, run inference on test images
with nengo_dl.Simulator(
     converter.net, 
     minibatch_size=128,
     progress_bar=False) as nengo_sim:

    for (ip_data, target_data) in data_generator:
        # repeat inputs for some number of timesteps
        tiled_ip = np.tile(ip_data, (1, n_steps, 1))
        data = nengo_sim.predict({nengo_input: tiled_ip})
3

In order to use data generator for sim.fit, it would be required to specify data for input, output, n_steps, and all the bias nodes. Data for bias nodes can be specified as an array of ones in the shape of the bias and n_steps would be of shape batchsize x 1.

Here is an example of sim.fit with a data generator:


from urllib.request import urlretrieve

import matplotlib.pyplot as plt
import nengo
import numpy as np
import tensorflow as tf

import nengo_dl

seed = 0
np.random.seed(seed)
tf.random.set_seed(seed)

(train_images, train_labels), (test_images, test_labels) = (
    tf.keras.datasets.mnist.load_data())
# flatten images and add time dimension
train_images = train_images.reshape((train_images.shape[0], 1, -1))
train_labels = train_labels.reshape((train_labels.shape[0], 1, -1))
test_images = test_images.reshape((test_images.shape[0], 1, -1))
test_labels = test_labels.reshape((test_labels.shape[0], 1, -1))


n_steps = 200
scale_firing_rate = 100
synapse = 0.005
batchsize = 128
inp = tf.keras.Input(shape=(28, 28, 1))

# convolutional layers
conv0 = tf.keras.layers.Conv2D(
    filters=32,
    kernel_size=3,
    activation=tf.nn.relu,
)(inp)

conv1 = tf.keras.layers.Conv2D(
    filters=64,
    kernel_size=3,
    strides=2,
    activation=tf.nn.relu,
)(conv0)

# fully connected layer
flatten = tf.keras.layers.Flatten()(conv1)
dense = tf.keras.layers.Dense(units=10)(flatten)

model = tf.keras.Model(inputs=inp, outputs=dense)
model.summary()
converter = nengo_dl.Converter(model)

def get_batches():
    for i in range(0,10000,batchsize):
        ip = train_images[i:i+batchsize]
        label = train_labels[i:i+batchsize]
        yield ({'input_1': ip, 
                "n_steps": np.ones((batchsize, 1), dtype=np.int32), 
                "conv2d.0.bias":np.ones((batchsize, 32, 1), dtype=np.int32),
                "conv2d_1.0.bias":np.ones((batchsize, 64, 1), dtype=np.int32),
                "dense.0.bias":np.ones((batchsize, 10, 1), dtype=np.int32)},
                {'probe':label})
data_generator = get_batches()


with nengo_dl.Simulator(converter.net, minibatch_size=batchsize) as sim:
    # run training
    sim.compile(
        optimizer=tf.optimizers.RMSprop(0.001),
        loss=tf.losses.SparseCategoricalCrossentropy(from_logits=True),
        metrics=[tf.metrics.sparse_categorical_accuracy],
    )
    sim.fit(
        data_generator,
        epochs=2,
        steps_per_epoch=10,
    )
1 Like
4

Hello @kpatel, much thanks for your detailed answers. I took pieces of your code and included them in a simple example, but they failed to execute. Just to point out… we are solving two problems here. Before I mention them, I should point that I have to include the following code snippet every time I build and train a TF-Keras model to prevent memory offshoot (and this is probably causing errors while running with nengo-dl, more details later).

config = tf.compat.v1.ConfigProto()
config.gpu_options.allow_growth = True
session = tf.compat.v1.InteractiveSession(config=config)

Now, problems:

First: How to load the TF-Keras weights into a model and then convert it to spiking one? I am attaching a first_spiking_2d_cnns.ipynb file which attempts to do it, but fails with following error:

KeyError: <Reference wrapping <tf.Tensor 'input_1:0' shape=(None, 28, 28, 1) dtype=float32>>
I guess, this is due to incompatible input to the nengo object in this line:
nengo_input = converter.inputs[inpt] in the attached file.

Second: How to train a nengo-dl model with batches of input data? For this, I copy pasted your code (in your second answer) and attempted to run it (second_spiking_2d_cnns.ipynb). It failed with following error: RuntimeError: The Session graph is empty. Add operations to the graph before calling run().

I guess, the problem here in second_spiking_2d_cnns.ipynb is the inclusion of the code to prevent memory offshoot.

Please note that if I happen to remove the code to prevent memory offshoot, both files fail while training the TF model (as nengo-dl also uses TF in backend). My GPU simply runs out of memory. So… on one hand where the code to prevent memory offshoot is necessary, it probably prevents nengo-dl models to run. Also to mention, the error I faced when I posted this topic was same as RuntimeError: The Session graph is.... Any suggestions?

I think I might have to downgrade TF version or CUDA versions… Please let me know. And sorry for uploading the ipynb files at a different platform, the forum doesn’t allow uploading for new users). You can right click on the link and download it. Thank you for your interest and time!

5

For the first issue, what I observed is you are using new_model with nengo_dl.Converter but trying to use inpt from model.
When using the converter.inputs, it would be required to use the input from the same model (i.e. new_model).
For example,

new_model, inp, out = create_2d_cnn_model(img_shape)
new_model.load_weights("./non_spiking_model/tf_keras_weights.tf")
new_model.compile(
    optimizer=tf.keras.optimizers.Adam(lr=1e-3),
    loss=tf.keras.losses.categorical_crossentropy,
    metrics=["accuracy"])
# Check if the new model with loaded weights performs the same as original model.
eval_tf_keras_model(new_model)

n_steps = 200
scale_firing_rate = 100
synapse = 0.005

converter = nengo_dl.Converter(
    new_model,
    swap_activations={tf.nn.relu: nengo.SpikingRectifiedLinear()},
    scale_firing_rates=scale_firing_rate,
    synapse=synapse)

nengo_input = converter.inputs[inp] # Has the input to be flattened here?
nengo_output = converter.outputs[out]

For the second issue, are you using tensorflow > 2.0? If so, I would use the following for limiting the memory assigned by tensorflow

gpus = tf.config.experimental.list_physical_devices("GPU")
for gpu in gpus:
    tf.config.experimental.set_memory_growth(gpu, True)

instead of

config = tf.compat.v1.ConfigProto()
config.gpu_options.allow_growth = True
session = tf.compat.v1.InteractiveSession(config=config)
1 Like
6

Hello @kpatel, I was now able to run both the scripts after replacing my script after that of yours to limit the memory usage, as well as after using the same inpt object obtained with new_model. Thank you very much! While I am moving ahead in my experiments, please consider this topic resolved.