[Nengo-DL]: Adding a dynamic custom layer between two Ensembles

[xchoo]

Hi @zerone,

This post is a little on the long side, and I’ve only just got the time to go through the entire post. To address your questions:

@zerone

I believe, this synapse parameter becomes relevant at the synaptic connection to the Ensembles, and not to the Nodes (as Nodes aren’t supposed to represent a group of neurons, thus no synaptic connection required… right?).

No, the synapse parameter affects all connections in the network, be it to an Ensemble or Node. However, I did misread your code, and you are correct in the observation that you do not see any filtered spikes. This is because the filtering is done on the output of a connection, which is then accumulated on the input of the Ensemble / Node. Thus, for nengo.Ensemble objects, to get the filtered inputs, you’ll need to probe the input attribute of the receiving Ensemble. As an example:

nengo.Probe(ndl_model.layers[conv2], attr="input")

For nengo.Node objects, there isn’t an input attribute that you can probe, so if you want to get the filtered spikes, you’ll need to probe the output of the connection that connects to that specific node (you’ll need to go through the all_connections object and find the one where post_obj == node)

@zerone

If there’s a way to probe the “input” to the max_pool TensorNode directly

Yes. As I mentioned above, you’ll need to probe the connection that connects to that node. Probably something like this (caveat: I haven’t tested this code)

for conn in ndl_model.net._connections:
    if conn.post_obj == ndl_model.layers[max_pool]:
        break
nengo.Probe(conn)

Note that this only works since the max_pool layer has 1 connection to it. If you make multiple connections to this node, you’ll need to probe all of the connections, then sum up the probed results for each of the probe.

@zerone

It is expected that transform, synapse, and function of the original connection be preserved in the new connection. Moreover, following is the output before replacing the original connection.

Hmm, yes, I admit this is unexpected. Looking at the code some more, I believe this is a bug, and I have made a GitHub issue (along with the suggested fix).

@zerone

Hence, please check this too, along with the following

This is indeed the case in the code.

@zerone

If at all MaxPooling in Nengo-DL is simply done by taking a max of incoming spike amplitudes at each timestep, then isn’t this somewhat suboptimal? As, the TF MaxPooling op is supposed to choose the maximum activation among a group of neurons, but Nengo-DL MaxPooling (with TensorNode) chooses the maximum spike amplitude at each timestep and it is possible that at a certain timestep, the chosen spike amplitude might not correspond to the maximally spiking neuron (which represents the maximum activation at the end of a sufficiently long simulation time period)?

This is correct. See this thread (where you posed the question before. ). With max pooling, you’ll definitely want to use some kind of averaging or smoothing to the spiking output to better approximate a rate-based max pooling operation.

@zerone

We have discussed the following for a number of times now, that the current J = alpha x <e, x> + bias (where x is the repeated scalar input) is the input to the Ensembles, and I definitely see it being true for the first conv0 layer, but what about the next Ensembles after the first? As mentioned above in this reply that the input to the Ensembles is the filtered spike output from the previous layer, is the filtered spike output too converted to current J using the same above formula (i.e. the filtered spike output is considered x) , or is it just straight forward inputted to the next conv1 (and so on) layers?

It would be the same for all layers, and by that I mean that the filtered spikes is used as an input to each layer (regardless of where it is in the network). For the first conv0 layer, the filter is None (i.e, no filter is applied), and the “spike train” is just a steady value, but the math still applies.