Nengo_dl converter for keras Dense layer

Deep Learning
1

Hello everyone, I am using the nengo_dl converter, and I found that the converter threw out some errors when trying to convert Dense layers with multi-dimensional inputs. However, when I am trying to convert a dense layer with flattened inputs, it will work correctly.

According to the tf.keras documentation,

the dense layer in tf.keras can accept inputs of more than one dimension, for example if the dense layer is layers.Dense(units=units), input is N-D tensor with shape: (batch_size, …, input_dim), the output should be N-D tensor with shape: (batch_size, …, units).

However for nengo_dl converter, it will throw out errors when trying to convert dense layer with multi-dimensional inputs.

If I write the code for my model as follows:

    inputs = keras.Input(shape=(inp_width, inp_height, 1))
    x = layers.Conv2D(32, 3, activation=tf.nn.relu)(inputs)
    x = layers.AveragePooling2D(pool_size=3, strides=2)(x)
    x = layers.Reshape([x.shape[1] * x.shape[2], 32])(x)
    x = layers.Dense(units=32)(x)
    x = layers.Flatten()(x)
    outputs = layers.Dense(units=num_labels)(x)
    model = keras.Model(inputs=inputs, outputs=outputs, name="model_3")
    model.summary()
    converter_dl = nengo_dl.Converter(model)

my model is as follows: (the output of keras model.summary)

Layer (type)      Output Shape       Param #   

input_1         [(None, 32, 32, 1)]           0         
                                                                 
 conv2d          (None, 30, 30, 32)        320       
                                                                 
 average_pooling2d (None, 14, 14, 32)      0         
                                                                                                              
 **reshape          (None, 196, 32)          0         **
                                                                 
** dense            (None, 196, 32)         1056**      
                                                                 
 flatten           (None, 6272)              0         
                                                                 
 dense_1          (None, 20)             125460

Then the converter throws out the following error.

Traceback (most recent call last):
  File "D:/snn models/spiking_models/model_5.py", line 151, in <module>
    converter_dl = nengo_dl.Converter(model)
  File "D:\Anaconda3\envs\snnvenv2\lib\site-packages\nengo_dl\converter.py", line 133, in __init__
    self.net = self.get_converter(model).convert(None)
  File "D:\Anaconda3\envs\snnvenv2\lib\site-packages\nengo_dl\converter.py", line 870, in convert
    nengo_layer = layer_converter.convert(layer_node_id)
  File "D:\Anaconda3\envs\snnvenv2\lib\site-packages\nengo_dl\converter.py", line 1417, in convert
    output = self.add_nengo_obj(
  File "D:\Anaconda3\envs\snnvenv2\lib\site-packages\nengo_dl\converter.py", line 603, in add_nengo_obj
    nengo.Connection(bias_node, obj, transform=biases[:, None], synapse=None)
  File "D:\Anaconda3\envs\snnvenv2\lib\site-packages\nengo\base.py", line 34, in __call__
    inst.__init__(*args, **kwargs)
  File "D:\Anaconda3\envs\snnvenv2\lib\site-packages\nengo\connection.py", line 516, in __init__
    self.transform = transform  # Must be set after function
  File "D:\Anaconda3\envs\snnvenv2\lib\site-packages\nengo\base.py", line 108, in __setattr__
    super().__setattr__(name, val)
  File "D:\Anaconda3\envs\snnvenv2\lib\site-packages\nengo\config.py", line 482, in __setattr__
    raise exc_info[1].with_traceback(None) from e
nengo.exceptions.ValidationError: init: Shape of initial value (32, 1) does not match expected shape (6272, 1)

Process finished with exit code 1

This error message indicates that the error comes from the ConverterDense(LayerConverter) class in the nengo.converter source code.

If I change my model so that the input of dense layer is flattened, there will be no error and the model is correctly converted:

input_1        [(None, 32, 32, 1)]               0         
                                                      
 conv2d             (None, 30, 30, 32)        320       
                                                      
 average_pooling2d  (None, 14, 14, 32)           0       
                                                      
 reshape           (None, 196, 32)              0       
                                                      
 **flatten (Flatten)     (None, 6272)              0       **
                                                     
** dense (Dense)       (None, 32)               200736**    

 flatten_1 (Flatten)      (None, 32)                0       
                                                       
 dense_1 (Dense)       (None, 20)                660

So I want to ask what should I do to convert this type of Dense layer? If I understand it correctly, the converter seems not be able to convert the Dense layer with multi-dimensional inputs, however due to the layer converter has a ConvertDense method, it will always try to convert this dense layer instead of falling back to Tensornode, and this would result in the error. I am thinking about if it is possible to tell the converter to fall back to Tensornode when converting this type of Dense layer. Or alternatively, how can I write the code for converting this type of dense layer. Thank much in advance for your reply!

2

Rather than flattening, which combines the trailing dimensions, what you want is to combine the leading dimensions. I.e., instead of reshaping to (x, 196, 32), and then flattening to (x, 196*32), you want to reshape to (x*196, 32). Then do the Dense layer, and reshape back to (x, 196, 32). That is functionally equivalent to the multidimensional Dense multiplication.

2 Likes
3

Thank you very much for your reply!! :+1: :+1:

4

Hello! I rethink on this conversion of dense layer today, and I think your answer may not work in my case, because “x” is the batch dimension in my model, and I don’t know if it’s possible to reshape (x, 196, 32) to (x*196, 32), which means reshaping the data to the batch dimension. So without the batch dimension, my model is:

reshape          (196, 32)          0         
                                                             
dense             (196, 32)         1056  

 flatten           (6272)           0

So I wonder if there are other ways to use the converter on this type of dense layer? Or alternatively, how can I manually implement this dense layer with ensembles and connections? Thank you in advance!