Nengo with 5D data

Dear all,

I am converting an analog signal through an ADC, and my output is actually 5D arrays.
Specifically, it has the shape of (1438, 95, 3, 32,32). I need to iterate over the first dimension and send them into a simple spiking network batch by batch. However, I am not sure if Nengo core is able to deal with 4-d data?

My main aim would be to create an end-to-end spiking neural network (the inserted data should be changed into spikes, and learning should be done in a supervised way) that can do classifications based on the inserted 5D-data (4D after sending batches of it). However, I am not sure if I should use Nengo core or Nengo DL for that.

Also, a very simple code that is capable of doing that in Nengo core or Nengo DL would be quite helpful. As I am still new to Nengo.

Thanks.

Hello @ji16qewe, welcome to our community. For the following question,

I am pointing you towards two Nengo Core tutorials which show you how to represent N dimensional data and do calculations on them. Do note that in these tutorials the value of N=2, but the principle of representing 4D data would remain the same.

Basically, you create N one dimensional ensembles, e.g.

A = nengo.Ensemble(100, dimensions=1, radius=10) and
B = nengo.Ensemble(100, dimensions=1, radius=10)

(for the value of N=2) and then create another ensemble for representing N=2 dimensional data, i.e.

combined = nengo.Ensemble(220, dimensions=2, radius=15)

And then you need to connect them as follows:

nengo.Connection(A, combined[0])
nengo.Connection(B, combined[1])

You will find more details in the linked tutorials.

Next, with respect to the following:

I am not completely sure if Nengo or Nengo-DL allows spike based training as of now. There’s KerasSpiking which allows you to do spike aware training. Specifically, it does spike based forward pass and non-spike based backward pass (to propagate the errors back) during training phase.

With Nengo-DL you can have an end-to-end spiking network - with room for some exceptions like MaxPooling op, unsupported neuron types e.g. softmax etc. which do not have a spiking counterpart. How you can do training and inference in Nengo-DL is as follows. You will first train your network with non-spiking neurons (e.g. ReLU) and then convert your model to spiking one by using nengo_dl.Converter() API and then proceed with inference. An excellent tutorial on training and inference with Nengo-DL is here.

With respect to following about training and inference with 4D data:

you will find how it accepts a 3D data in the above linked Nengo-DL tutorial. Basically, you create your usual TF network architecture to accept the N dimensional data and then while working in Nengo-DL context, you flatten your data before passing it to the Nengo-DL simulator functions.

If your task is to do deep learning, I would suggest you to look at Nengo-DL and KerasSpiking, evaluating both which suits you best.

@zerone First of all, thanks a lot for your reply.

However, what I got from your comment that Nengo so far does not support end-to-end training in the spiking domain. Because I see that either KerasSpiking or Nengo DL is trained normally in the non-spiking domain, then we just convert the activation functions into spiking version in the test phase, am I right?

Nengo supports unsupervised PES training rule in spiking domain, this I know of. About supporting state-of-the-art backprop methods for spike based training, I have not seen any example so far. Although, Nengo developers would answer it more confidently.

About KerasSpiking and Nengo-DL, your observation is correct.

1 Like

Currently, Nengo only supports end-to-end training in the spiking domain for online learning. For offline and batch learning, you’ll need to use something like the KerasSpiking package for TensorFlow.

Hello @xchoo, I am new to online learning. Is the Nengo supported online learning supervised or unsupervised?

Nengo supports both supervised and unsupervised online learning. You can build your networks with the different forms of learning by using different learning rules. Included in Nengo is the PES learning rule, which is a online supervised learning rule, as well as the BCM and Oja learning rules, which are online unsupervised learning rules. Nengo also includes other learning rules, which are described here. There is even an (super rough) STDP implementation described here and here (STDP is an unsupervised learning rule).

The cool thing about Nengo is that it is general enough to users to implement their own custom learning rule, be they supervised or not! :smiley:

1 Like

Thanks @xchoo for these sources. If one looks to dive into online learning with spiking neurons, other than these sources, do you suggest something more basic to begin with?

Apart from the examples on the Nengo documentation, there is @tbekolay’s Master thesis which goes into some detail of the mathematics behind online learning with spiking neurons. I referenced his thesis in this forum post.

@tcstewar has some online learning examples designed for Nengo available here as well that might be useful to you. Although, some of those example may have to be updated for the latest version of Nengo.

Awesome! Thanks!

Just an update, @tcstewar has also referred me to this Jupyter notebook which may be helpful as a starting guide for online learning in Nengo.

This looks like a great source! Thanks!