Conversion of sequential model

Deep Learning
1

Hello,

I hope my post finds you well. I am working on converting a keras sequential model and I got an error when the sim.fit() is running.

model = keras.Sequential(
    [
        layers.Input(shape=(x_train.shape[1], x_train.shape[2])),
        layers.Conv1D(
            filters=32, kernel_size=7, padding="same", strides=2, activation="relu"
        ),
        layers.Dropout(rate=0.2),
        layers.Conv1D(
            filters=16, kernel_size=7, padding="same", strides=2, activation="relu"
        ),
        layers.Conv1DTranspose(
            filters=16, kernel_size=7, padding="same", strides=2, activation="relu"
        ),
        layers.Dropout(rate=0.2),
        layers.Conv1DTranspose(
            filters=32, kernel_size=7, padding="same", strides=2, activation="relu"
        ),
        layers.Conv1DTranspose(filters=1, kernel_size=7, padding="same"),
    ]
)
model.summary()

converter = nengo_dl.Converter(model)

do_training = True
if do_training:
    with nengo_dl.Simulator(converter.net, minibatch_size=200) as sim:
        # run training
        sim.compile(
            optimizer=tf.optimizers.RMSprop(0.001),
            loss=tf.keras.losses.MeanSquaredError(),
            metrics=[tf.metrics.mean_absolute_error],
        )
        sim.fit(
            {converter.inputs[model.input]: x_train},
            {converter.outputs[model.output]: df_train.value},
            validation_data=(
                {converter.inputs[model.input]: x_test},
                {converter.outputs[model.output]: df_test.value},
            ),
            epochs=10,
        )

        # save the parameters to file
        sim.save_params("./keras_to_snn_params")
else:
    # download pretrained weights
    # urlretrieve(
    #     "https://drive.google.com/uc?export=download&"
    #     "id=1lBkR968AQo__t8sMMeDYGTQpBJZIs2_T",
    #     "keras_to_snn_params.npz")
    print("Load pretrained weights ...")`Preformatted text`

The error happens in this line of code: {converter.inputs[model.input]: x_train}.
Am I doing something wrong?

Many thanks

2

Hello @yassinej, it will be helpful if you could share your complete script (with the source of training data) or the specific error at least which you are getting.

I have one suggestion though which I observed with my TF-Keras models. If TF-Keras models have activation="relu", then while conversion to a spiking network (i.e. subsequent swapping of ReLU neurons with nengo.SpikingRectifiedLinear() neurons), they don’t get swapped to spiking neurons. You need to define your TF-Keras models with activation=tf.nn.relu for the conversion to take place. @drasmuss, @arvoelke, @Eric for their attention here on why this is the behaviour?

3

Hi @zerone, Thanks for your answer.

I have fixed your suggestion regarding the activation function. However, I am still having the same error.

This is my current script:

import numpy as np

import pandas as pd

from tensorflow import keras

from tensorflow.keras import layers

from datetime import datetime

from matplotlib import pyplot as plt

from matplotlib import dates as md

import tensorflow as tf

import nengo

import nengo_dl

master_url_root = "https://raw.githubusercontent.com/numenta/NAB/master/data/"

df_small_noise_url_suffix = "artificialNoAnomaly/art_daily_small_noise.csv"
df_small_noise_url = master_url_root + df_small_noise_url_suffix
df_small_noise = pd.read_csv(df_small_noise_url)

df_daily_jumpsup_url_suffix = "artificialWithAnomaly/art_daily_jumpsup.csv"
df_daily_jumpsup_url = master_url_root + df_daily_jumpsup_url_suffix
df_daily_jumpsup = pd.read_csv(df_daily_jumpsup_url)

def get_value_from_df(df):
    return df.value.to_list()


def normalize(values):
    mean = np.mean(values)
    values -= mean
    std = np.std(values)
    values /= std
    return values, mean, std


# Get the `value` column from the training dataframe.
training_value = get_value_from_df(df_small_noise)

# Normalize `value` and save the mean and std we get,
# for normalizing test data.
training_value, training_mean, training_std = normalize(training_value)
len(training_value)

TIME_STEPS = 288


def create_sequences(values, time_steps=TIME_STEPS):
    output = []
    for i in range(len(values) - time_steps):
        output.append(values[i : (i + time_steps)])
    # Convert 2D sequences into 3D as we will be feeding this into
    # a convolutional layer.
    return np.expand_dims(output, axis=2)


x_train = create_sequences(training_value)
print("Training input shape: ", x_train.shape)

model = keras.Sequential(
    [
        layers.Input(shape=(x_train.shape[1], x_train.shape[2])),
        layers.Conv1D(
            filters=32, kernel_size=7, padding="same", strides=2, activation= tf.nn.relu
        ),
        layers.Dropout(rate=0.2),
        layers.Conv1D(
            filters=16, kernel_size=7, padding="same", strides=2, activation= tf.nn.relu
        ),
        layers.Conv1DTranspose(
            filters=16, kernel_size=7, padding="same", strides=2, activation= tf.nn.relu
        ),
        layers.Dropout(rate=0.2),
        layers.Conv1DTranspose(
            filters=32, kernel_size=7, padding="same", strides=2, activation= tf.nn.relu
        ),
        layers.Conv1DTranspose(filters=1, kernel_size=7, padding="same"),
    ]
)
model.compile(optimizer=keras.optimizers.Adam(learning_rate=0.001), loss="mse")
model.summary()


converter = nengo_dl.Converter(model)

do_training = True
if do_training:
    with nengo_dl.Simulator(converter.net, minibatch_size=200) as sim:
        # run training
        sim.compile(
            optimizer=tf.optimizers.RMSprop(0.001),
            loss=tf.keras.losses.MeanSquaredError(),
            metrics=[tf.metrics.mean_absolute_error],
        )
        sim.fit(
            {converter.inputs[model.input]: x_train},
            {converter.outputs[model.output]: df_train.value},
            validation_data=(
                {converter.inputs[model.input]: x_test},
                {converter.outputs[model.output]: df_test.value},
            ),
            epochs=10,
        )

        # save the parameters to file
        sim.save_params("./keras_to_snn_params")
else:
    # download pretrained weights
    # urlretrieve(
    #     "https://drive.google.com/uc?export=download&"
    #     "id=1lBkR968AQo__t8sMMeDYGTQpBJZIs2_T",
    #     "keras_to_snn_params.npz")
    print("Load pretrained weights ...")

and this is the error I am getting:

Build finished in 0:00:00
Optimization finished in 0:00:00
|# Constructing graph | 0:00:00/usr/local/lib/python3.6/dist-packages/nengo_dl/converter.py:329: UserWarning: Layer type <class ‘tensorflow.python.keras.layers.core.Dropout’> does not have a registered converter. Falling back to TensorNode.
% (error_msg + ". " if error_msg else “”)
/usr/local/lib/python3.6/dist-packages/nengo_dl/converter.py:329: UserWarning: Layer type <class ‘tensorflow.python.keras.layers.convolutional.Conv1DTranspose’> does not have a registered converter. Falling back to TensorNode.
% (error_msg + ". " if error_msg else “”)
/usr/local/lib/python3.6/dist-packages/nengo_dl/converter.py:139: UserWarning: Converting sequential model to functional model
warnings.warn(“Converting sequential model to functional model”)
/usr/local/lib/python3.6/dist-packages/nengo_dl/simulator.py:460: UserWarning: No GPU support detected. It is recommended that you install tensorflow-gpu (pip install tensorflow-gpu).
"No GPU support detected. It is recommended that you "
Construction finished in 0:00:00

KeyError Traceback (most recent call last)
in ()
12 )
13 sim.fit(
—> 14 {converter.inputs[model.input]: x_train},
15 {converter.outputs[model.output]: df_train.value},
16 validation_data=(

/usr/local/lib/python3.6/dist-packages/nengo_dl/converter.py in getitem(self, key)
399
400 def getitem(self, key):
→ 401 return self.dict[self._get_key(key)]
402
403 def iter(self):

KeyError: <Reference wrapping <tf.Tensor ‘input_1:0’ shape=(None, 288, 1) dtype=float32>>

Looking forward to reply.

4

Hello @yassinej, I have TF-2.2.0 installed on my system which does not have Conv1DTranspose (rather it’s present in tf-nightly). However, I was able to declare a custom Conv1DTranspose class and the TF-Keras model was successfully created. But it failed while conversion to Nengo.

If you look at the model.summary() output (on your end), it should look like

Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv1d_2 (Conv1D)            (None, 144, 32)           256       
_________________________________________________________________
dropout_2 (Dropout)          (None, 144, 32)           0         
_________________________________________________________________
conv1d_3 (Conv1D)            (None, 72, 16)            3600      
_________________________________________________________________
conv1d_transpose_1 (Conv1DTr (None, 144, 16)           1808      
_________________________________________________________________
dropout_3 (Dropout)          (None, 144, 16)           0         
_________________________________________________________________
conv1d_transpose_2 (Conv1DTr (None, 288, 32)           3616      
_________________________________________________________________
conv1d_transpose_3 (Conv1DTr (None, 288, 1)            225       
=================================================================
Total params: 9,505
Trainable params: 9,505
Non-trainable params: 0

And if you check the output of model.input it should be
<tf.Tensor ‘input_1:0’ shape=(None, 288, 1) dtype=float32>
For some reason, it’s
<tf.Tensor 'input_2:0' shape=(None, 288, 1) dtype=float32>
on my system.

Based on above information, I am taking my best guess here. In your line converter.inputs[model.input]: x_train, it expects a key (model.input which corresponds to the TF-Keras model layers) with name ‘input_1:0’ in converter.inputs… but as you can see, your TF-Keras model does not have any input layer with name ‘input_1:0’. I guess, this is the problem. On your system, where you are successfully able to convert the TF-Keras model, try executing converter.inputs.keys(), I suppose you won’t find any key with name ‘input_1:0’.

I would suggest you to install tf-nightly and attempt again. For example, I declare models in this way:

  inpt = tf.keras.Input(shape=shape)
  
  conv0 = tf.keras.layers.Conv2D(
      filters=32, kernel_size=3, activation=tf.nn.relu)(inpt)
  conv1 = tf.keras.layers.Conv2D(
      filters=64, kernel_size=3, activation=tf.nn.relu)(conv0)
  
  flatten = tf.keras.layers.Flatten()(conv1)
  dense = tf.keras.layers.Dense(10)(flatten)
  
  model = tf.keras.Model(inputs=inpt, outputs=dense)
  
  return model, inpt, dense
5

If you specify activation="relu", under the hood that is mapped to tf.keras.activations.relu. So you would need to set up your activation swap like swap_activations={tf.keras.activations.relu: nengo.SpikingRectifiedLinear()} (rather than {tf.nn.relu: nengo.SpikingRectifiedLinear()}).

1 Like
6

Hi @zerone, I have been able from the beginning to have the model summary output and the conversion for the sequential model as I already have tf-nightly installed. However, when I build the model with functional API as you are suggesting, I get an error of the following while converting the model:

RuntimeError Traceback (most recent call last)
~/Documents/eSNN_IDS_nengoLoihi/CNN_Conv1D_based_network/DoS_attack_network.py in
----> 140 converter = nengo_dl.Converter(model)

~/anaconda3/lib/python3.7/site-packages/nengo_dl/converter.py in init(self, model, allow_fallback, inference_only, max_to_avg_pool, split_shared_weights, swap_activations, scale_firing_rates, synapse)
131
132 # convert model
→ 133 self.get_converter(model).convert(None)
134
135 if isinstance(model, tf.keras.Sequential):

~/anaconda3/lib/python3.7/site-packages/nengo_dl/converter.py in get_converter(self, layer)
338 )
339
→ 340 converter = ConverterClass(layer, self)
341
342 self._layer_converters[layer] = converter

~/anaconda3/lib/python3.7/site-packages/nengo_dl/converter.py in init(self, *args, **kwargs)
978 layer = self.layer.class.from_config(self.layer.get_config())
979 if self.layer.built:
→ 980 layer.build(self.input_shape(0, include_batch=True))
981 layer.set_weights(self.layer.get_weights())
982

~/anaconda3/lib/python3.7/site-packages/nengo_dl/converter.py in input_shape(self, node_id, include_batch)
701 if the node as multiple inputs.
702 “”"
→ 703 return self._get_shape(“input”, node_id, include_batch=include_batch)
704
705 def output_shape(self, node_id, include_batch=False):

~/anaconda3/lib/python3.7/site-packages/nengo_dl/converter.py in _get_shape(self, input_output, node_id, include_batch)
673
674 # get the shape
→ 675 shape = func(node_id)
676
677 if not include_batch:

~/anaconda3/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py in get_input_shape_at(self, node_index)
2020 “”"
2021 return self._get_node_attribute_at_index(node_index, ‘input_shapes’,
→ 2022 ‘input shape’)
2023
2024 @doc_controls.do_not_doc_inheritable

~/anaconda3/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py in _get_node_attribute_at_index(self, node_index, attr, attr_name)
2613 if not self._inbound_nodes:
2614 raise RuntimeError('The layer has never been called ’
→ 2615 'and thus has no defined ’ + attr_name + ‘.’)
2616 if not len(self._inbound_nodes) > node_index:
2617 raise ValueError('Asked to get ’ + attr_name + ’ at node ’ +

RuntimeError: The layer has never been called and thus has no defined input shape.

With the above error, I decided to build the model with sequential API and the conversion of the model was successful. However, I end up with the previous error I posted above.

Regarding the model.input, it has an input layer with name ‘input_1:0’:

Model: “sequential”

Layer (type) Output Shape Param #

conv1d (Conv1D) (None, 144, 32) 256

dropout (Dropout) (None, 144, 32) 0

conv1d_1 (Conv1D) (None, 72, 16) 3600

conv1d_transpose (Conv1DTran (None, 144, 16) 1808

dropout_1 (Dropout) (None, 144, 16) 0

conv1d_transpose_1 (Conv1DTr (None, 288, 32) 3616

conv1d_transpose_2 (Conv1DTr (None, 288, 1) 225

Total params: 9,505
Trainable params: 9,505
Non-trainable params: 0

converter = nengo_dl.Converter(model)
model.input
<tf.Tensor ‘input_1:0’ shape=(None, 288, 1) dtype=float32>

converter.inputs.keys()
KeysView(<nengo_dl.converter.Converter.KerasTensorDict object at 0x7f2415f48e50>)

7

Hello @yassinej, I was able to reproduce the error RuntimeError: The layer has never been called and thus has no defined input shape. you faced while creating the model through functional API. Interestingly in the same tf-nightly-gpu environment I tried converting a model (which gets successfully converted in TF 2.2.0) and it failed with the same RuntimeError. This has brought me to the conclusion that tf-nightly might not be supported by nengo-dl for now, and hence you are facing subsequent errors with sequential API too. My one suggestion would be to declare a custom layer (which is not present in TF 2.2.0) and possibly use TensorNodes to integrate it with rest of the nengo-dl model in TF-2.2.0 environment.

One way to declare Conv1DTranspose is mentioned here, however when I attempted to convert it, it failed with

NotImplementedError: Layer Conv1DTranspose has arguments in `__init__` and therefore must override `get_config`.

Unfortunately I don’t have much experience with TensorNodes, therefore won’t be able to help you. @drasmuss, @Eric to confirm if nengo-dl supports tf-nightly?

8

No, we can’t guarantee that tf-nightly builds will work with NengoDL, since they may change very rapidly. We test against the latest pre-release (currently TF 2.3.0rc1). There are some changes to the way Keras represents models internally in TF 2.3.0 which required updates to the Converter. Those aren’t merged in quite yet, but you can follow this PR to see when that happens https://github.com/nengo/nengo-dl/pull/159.

9

Note, when using a sequential model it seems like you’re getting a similar error to this issue https://github.com/nengo/nengo-dl/issues/162, and the fix may be the same (i.e., use converter.inputs[converter.model.input] instead of converter.inputs[model.input]).

10

Hi, @zerone and @drasmuss,

Thank you for all the help you guys have provided. As @drasmuss have mentioned in the last post using converter.inputs[converter.model.input] instead of converter.inputs[model.input] have fixed the fixed the error.

Many thanks,

11

Hi @zerone and @drasmuss,

I have been stuggling with this error for the past three days with the following error: ValueError: Error when checking input: expected input_2 to have shape (None, 288) but got array with shape (288, 1) and when I run converter.model.input I get <tf.Tensor 'input_2:0' shape=(200, 288, 1) dtype=float32> and x_train[0].shape is (288, 1).

When I run the model using keras, everything goes smoothly without getting the above error.

Do you have any idea on how to solve this?

Many thanks,

12

Hello @yassinej, without the recent code where you are facing this issue, I will simply be able to guess the problem. Most probably you are not feeding the data to the nengo-dl network in the required format. If you check here, All inputs should have shape (batch_size, n_steps, node.size_out). For example, if you have 10000 MNIST digits in your matrix, its shape would look like: (10000, 28, 28, 1) where 28 is the image’s height and width with channel 1 (i.e. not RGB channel, rather grey). This format works perfectly in TF-Keras networks, however you need to reshape it to (10000, n_steps, 784) before feeding it to the Nengo-DL model. You can do that by:

test_images = test_images.reshape((test_images.shape[0], 1, -1))
tiled_test_images = np.tile(test_images, (1, n_steps, 1))

where test_images is a matrix of shape (10000, 28, 28, 1).

You might also be required to reshape and tile corresponding labels. For more info on data formatting look at this tutorial. If the issue persists, please send in your recent code.

1 Like
13

Hi @zerone,

Thanks for the quick response. Yes, that solved the error.

Many thanks,