Coding Schemes using for encoding image

General Discussion
1

i have 2 question for this problem.

  1. What Coding Schemes nengo using for encoding image in spiking neural network?

  2. how does nengo handle 3d images? what pre-processing technique for this?

2

Hi @locviettri, and welcome to the Nengo forums! :smiley:

To answer your questions, Nengo doesn’t have a default encoding schema for images. It is up to you (the network designer) to create an encoding schema for the ensembles in your neural network. The only thing that the neural ensembles operate on are vectors.

When we design our Nengo networks (see this example), we typically flatten the image (a 2D image) into a multi-dimensional vector. That is to say an NxM image would be flattened into a 1x(M*N) vector.

This is up to you, really. There is no default pre-processing method for 3D images in Nengo. I don’t have any experience working with 3D images, so I can’t give any concrete suggestions. You can try flattening the data (as with the 2D images), or maybe encoding the depth information separately, as a different vector (i.e., you’ll have 2 M*N vectors, one for the pixel information, one for the depth information).

1 Like
3

In terms of coding, there are two options that I’ve used quite a bit before:

  1. If you don’t care about your input layer being spiking, then you can do a nengo.Convolution transform straight from your image input to the .neurons attribute of an Ensemble. This ensemble can then have spiking neurons, so subsequent layers/weights will all be running with spikes.
  2. If you do want your input layer to be spiking, then I would do a Convolution transform with 1x1 kernels from your input node to the .neurons attribute of an Ensemble. This will encode your image into spikes pixel-by-pixel. The number of channels for the output of this convolution should typically be slightly larger than the number of channels in your image; in this example, we have a 3-channel input image, and we have 4 outputs to the convolutional kernel. What this does is uses 4 neurons to encode each 3 channel pixel.

If you want a more NEF-style approach to image encoding, you can also look at this example. Unfortunately, that repository hasn’t been kept up well, so it might not work out-of-the-box, but hopefully a lot of the code in that example will still port over to new Nengo.

1 Like