I’m trying to implement a simple neural network and test how the neurons behave with a video as input. I am pretty new in Nengo and after a few weeks of research and a some attempts of different solutions I can not manage to feed the video to the neural network.
Below you will find my approach to the problem. I’m trying to feed a video (split in frames, so actually I’m feeding images) to a Neuron and connecting this one to other with a sinusoidal output to see how it behaves. I really don’t know if this is the best approach and for now on it’s not working, I got trouble with the array dimension in the “eval_points”…
I’m open to any suggestion, thank you in advance!
#Load images
images_files = glob.glob("full-path/one_clip_frames/*.jpg")
images = []
for file in images_files:
img = PIL.Image.open(file)
img_array = np.array(img.getdata()).flatten()
images.append(img_array)
img_rows, img_cols = 640, 480
img_size = 640 * 480
#number of hidden units
n_hid = 1000
ens_params = dict(
eval_points=images,
neuron_type=nengo.LIFRate(),
intercepts=nengo.dists.Choice([0.1]),
max_rates=nengo.dists.Choice([100]),
)
with nengo.Network() as model:
a = nengo.Ensemble(n_hid, img_size, **ens_params)
v = nengo.Node(np.sin)
conn = nengo.Connection(
a, v, synapse=None,
eval_points=images)
encoders = rng.normal(size=(n_hid, img_size))
a.encoders = encoders
tile(encoders.reshape(-1, img_rows, img_cols), rows=4, cols=6, grid=True)
I did a quick run-through your code, and at first glance, there doesn’t seem to be an issue with the way you have set it up. So, to help debug your issue with the eval_points dimensionality, it would be helpful if you could provide examples of what errors you are getting.
I would also suggest checking to see if what you are getting from the np.array(img.getdata()).flatten() function is what you expect. You can print out the shape of the array with:
print(img_array.shape)
and it should be (307200,). I suspect that the PIL.Image.open function is returning an RGB array, rather than a B/W one, and the flatten() function is amalgamating all three colour channels into a single array. You made need to do further processing to convert the opened images to a B/W image (or something that the network can use).
As a note, the NengoExtras Github repository (link: here) has an example of a Nengo network created to process images (albeit MNIST images). You can use this example as a reference for how you can design your own network.
Finally, I would suggest resizing your images to something smaller (at least until you get something working). The example I linked above operates with 28x28 MNIST images, which is a total of 784 dimensions. For that example, 1000 neurons was enough. However, your code uses a 640x480 image, which is 307,200 dimensions! If we scale up the MNIST network proportionally, you will need about 390,000 neurons… which is a lot, and probably too much for your computer to handle.
