Questions about the usage of Tensornode and nengo_dl.Layer

Deep Learning
1

Hello, I am here to ask some questions about the usage of nengo_dl.Layers and the Tensornode. In the nengo_dl documentation it says that “Layer is just a syntactic wrapper for constructing TensorNodes or Ensembles; anything we build with a Layer we could instead construct directly using those underlying components.” So here I built a network using both ways (Directly using the nengo_dl.Layer function or alternatively, construct the network with the underlying components)

For convenience, I used part of the model in the nengo_dl tutorial of optimizing spiking neural networks.

The construction with nengo_dl.Layer:

with nengo.Network(seed=0) as net:
    # set some default parameters for the neurons that will make
    # the training progress more smoothly
    net.config[nengo.Ensemble].max_rates = nengo.dists.Choice([100])
    net.config[nengo.Ensemble].intercepts = nengo.dists.Choice([0])
    net.config[nengo.Connection].synapse = None
    neuron_type = nengo.LIF(amplitude=0.01)
    nengo_dl.configure_settings(stateful=False)

    inp = nengo.Node(np.zeros(28 * 28))
    x = nengo_dl.Layer(tf.keras.layers.Conv2D(filters=32, kernel_size=3))(
        inp, shape_in=(28, 28, 1)
    )
    out = nengo_dl.Layer(tf.keras.layers.Dense(units=10))(x)

    out_p = nengo.Probe(out, label="out_p")
    out_p_filt = nengo.Probe(out, synapse=0.1, label="out_p_filt")

The construction with the components:

with nengo.Network(seed=0) as net:
    # set some default parameters for the neurons that will make
    # the training progress more smoothly
    net.config[nengo.Ensemble].max_rates = nengo.dists.Choice([100])
    net.config[nengo.Ensemble].intercepts = nengo.dists.Choice([0])
    net.config[nengo.Connection].synapse = 0.01
    neuron_type = nengo.LIF(amplitude=0.01)

    nengo_dl.configure_settings(stateful=False)

    inp = nengo.Node(np.zeros(inp_width * inp_height))
    # add the first convolutional layer
    node1 = nengo_dl.TensorNode(tf.keras.layers.Conv2D(filters=32, kernel_size=3),
                                shape_in=(inp_width, inp_height, 1),
                                pass_time=False)
    nengo.Connection(inp, node1)
    node2 = nengo_dl.TensorNode(tf.keras.layers.Dense(units=num_labels), 
    shape_in=(26, 26, 32), pass_time=False)
    nengo.Connection(node1, node2)
    out_p = nengo.Probe(node2, label="out_p")
out_p_filt = nengo.Probe(node2, synapse=0.1, label="out_p_filt")

I trained the two models implemented with different methods, the loss is dramatically different. Therefore, I know that I am not implementing the model correctly without using the nengo_dl.Layer API. So my first question is, why are the models above different from each other and how can I impement the same model without using nengo_dl.Layer?

My second question is, in the nengo_dl document, I can see the structure shown below,
This is part of the model from the tutorial:

    inp = nengo.Node(np.zeros(28 * 28))
    # add the first convolutional layer
    x = nengo_dl.Layer(tf.keras.layers.Conv2D(filters=32, kernel_size=3))(
        inp, shape_in=(28, 28, 1)
    )
    x = nengo_dl.Layer(neuron_type)(x)
    # add the second convolutional layer
    x = nengo_dl.Layer(tf.keras.layers.Conv2D(filters=64, strides=2, kernel_size=3))(
        x, shape_in=(26, 26, 32)
    )

There is a layer “x = nengo_dl.Layer(neuron_type)(x)”, according to the documentation, “Layer can also be passed a Nengo NeuronType, instead of a Tensor function. In this case Layer will construct an Ensemble implementing the given neuron nonlinearity”.
So according to the above explanations, “x = nengo_dl.Layer(neuron_type)(x)” represents an ensemble in the model, but I wonder what is the parameters and details of this ensemble? I also want to customize some parameters, so I want to ask how can I construct the same layer as “x = nengo_dl.Layer(neuron_type)(x)” with the nengo.Ensemble and Nengo.Connection API? Thank you in advance for your reply!

2

Hi @fgh,

I forwarded your issue to a NengoDL dev, and from their feedback, it seems that the difference in loss is probably due to the slight differences in how the networks are configured. In addition to creating the appropriate TensorFlow layer, the nengo_dl.Layer function call also creates the necessary nengo.Connections between the layers. However, these layers are created such that they are non-trainable (i.e., the connection weights for that connection is not modified by the TensorFlow training algorithm).

In your implementation where you used the TensorNode, you had to create the Nengo connection yourself, and by default, this connection is trainable. To mirror the implementation of the nengo_dl.Layer, you will need to configure that specific connection to be non-trainable. The process to do so is explained here.

With regards to this question:

You can find the specific parameters of the ensemble here. I should note, though, that in Nengo, an ensemble is simply a collection of neurons with some interface code to implement the NEF algorithm and related functionality. However, when used in the context of TensorFlow, the NEF bits are ignored, and this is done by making connections directly to the .neurons object (you’ll see a reference to this in the link to the code above).

1 Like
3

Thank you xchoo! Your answers are very helpful :+1: