Importing my own dataset for the Nengo Model

Hi, I am quite new to nengo. I was following the “Optimizing a spiking neural network” example in Nengo; seemed to work perfectly. Then I got into the task of importing my own dataset and got a lot of errors. The sample dataset which I was working with is the dogs and cats dataset which can be found at this link . I have tried numerous ways of importing my own dataset, but none of the methods seem to be working. Here is the algorithm which I’m using to replace the current method to load the dataset.

Instead of this:

(train_images, train_labels), (
) = tf.keras.datasets.mnist.load_data()

I was doing this to load a dataset:

for i in range(5):
    file = random.choice(os.listdir(test_folder))
    image_path= os.path.join(test_folder, file)

### Shuffle Dataset and labels
def unison_shuffled_copies_train(a, b):
    assert len(a) == len(b)
    p = np.random.permutation(len(a))
    return a[p], b[p]

def unison_shuffled_copies_test(a, b):
    assert len(a) == len(b)
    p = np.random.permutation(len(a))
    return a[p], b[p]

def create_dataset_train(img_folder):
    for dir1 in os.listdir(img_folder):
        for file in os.listdir(os.path.join(img_folder, dir1)):
            image_path= os.path.join(img_folder, dir1,  file)
            image= cv2.imread( image_path, 0)
            if image is not None:
              image=cv2.resize(image, (IMG_HEIGHT, IMG_WIDTH),interpolation = cv2.INTER_AREA)
              image = image.astype('float32')
              image /= 255 
    return img_data_array, class_name
# extract the image array and class name
img_data, class_name =create_dataset_train(r'/data/train')
train_images = np.array(img_data)
print("Train I: ", train_images.shape)
target_dict={k: v for v, k in enumerate(np.unique(class_name))}
target_val=  [target_dict[class_name[i]] for i in range(len(class_name))]
train_labels = np.array(target_val)

train_images, train_labels = unison_shuffled_copies_train(train_images,train_labels)

img_folders = "/data/test"

def create_dataset_test(img_folders):
    for dir2 in os.listdir(img_folders):
        for file in os.listdir(os.path.join(img_folder, dir2)):
            image_path= os.path.join(img_folder, dir2,  file)
            image= cv2.imread( image_path, 0)
            if image is not None:

              image=cv2.resize(image, (IMG_HEIGHT, IMG_WIDTH),interpolation = cv2.INTER_AREA)
              image = image.astype('float32')
              image /= 255 
    return img_data_array, class_name
# extract the image array and class name
img_data, class_name  =create_dataset_test(r'/data/test')
for i in range(25):
    plt.subplot(5, 5, i + 1)
test_images = np.array(img_data)
target_dict={k: v for v, k in enumerate(np.unique(class_name))}
target_val=  [target_dict[class_name[i]] for i in range(len(class_name))]
test_labels = np.array(target_val)
test_images, test_labels = unison_shuffled_copies_test(test_images,test_labels)
train_images = train_images.reshape((train_images.shape[0], -1))
test_images = test_images.reshape((test_images.shape[0], -1))

plt.figure(figsize=(12, 4))
for i in range(3):
    plt.subplot(1, 3, i + 1)
    plt.imshow(np.reshape(train_images[i], (28, 28)), cmap="gray")

And then from there on the rest of the code is run, which is the same code from the example MNIST one. I am not entirely sure if this is the correct way of importing a dataset for this Nengo-specific model; whether I am just over-complicating it. I want it to be in the exact format as the MNIST dataset, which I checked and in the end it looks like its the same format but there’s still errors in shape e.t.c. Could anyone help me please? Is there any pre-written piece of code that is good to load these type of datasets. As a note, I am also loading in the images as grayscale for simplicity. If there’s anything unclear with what I defined in the problem, please let me know and I can edit this.Thanks in advance.

Hello @CodeHelp1! Welcome to the forum. Data curation for Nengo-DL models should follow some specific steps. It is not as straight forward as curating data for TF models. The temporal nature of Nengo models introduces most of the difference. Also, keep in mind that the shape of data fed to Nengo-DL models while training and inference could vary depending upon the n_steps (i.e. data presentation time) parameter.

Now let me be more specific with an example. Suppose you are loading CIFAR10 dataset. You can do that by:

>>> import tensorflow as tf
>>> (train_x, train_y), (test_x, test_y) = tf.keras.datasets.cifar10.load_data()

Next, you should check how the data is shaped. It will appear as follows.

>>> print(train_x.shape, train_y.shape, test_x.shape, test_y.shape)
(50000, 32, 32, 3) (50000, 1) (10000, 32, 32, 3) (10000, 1)

As you can see, there are 50,000 and 10,000 data points for training and testing respectively. Each training/test image is of shape 32 x 32 x 3 and the labels are of shape 1 i.e. just a scalar. Let’s curate it for training (supposing you have a 2D CNN model with input layer as):

inpt_lyr = tf.keras.Input(shape=inpt_shape)

where the inpt_shape is (32, 32, 3). You need to first of all flatten the training data and add the temporal dimension which you can do as follows:

>>> flat_train_x = train_x.reshape(train_x.shape[0], 1, -1)

Upon looking at it’s shape it should appear as:

>>> flat_train_x.shape
(50000, 1, 3072)

where the middle dimension is the temporal dimension. Now, about training labels, you have to be a bit careful here. Depending upon your loss function, your curation process will vary. If you are using tf.keras.losses.CategoricalCrossentropy you should binarize them and add temporal dimension there too. You can do that as follows.

>>> import numpy as np
>>> flat_train_y = np.eye(10, dtype=np.float32)[train_y] # 10 is the number of classes. In your case it will be 2 I suppose.
>>> flat_train_y.shape
(50000, 1, 10)

Coming to the test data, you will do the same for test images i.e. test_x as follows:

>>> flat_test_x = test_x.reshape(test_x.shape[0], 1, -1)
>>> flat_test_x.shape
(10000, 1, 3072)

However you might have to tile the test images (i.e. repeat them) for a certain number of time steps n_steps till which you inference model runs. You can do that as follows.

>>> flat_test_x = np.tile(flat_test_x, (1, 30, 1)) # Each test image is repeated same for 30 time steps i.e. `n_steps` = 30 here.

Upon checking the shape, it will appear as:

>>> flat_test_x.shape
(10000, 30, 3072)

We are almost done, but what about the test labels? You may want to binarize them, (and then add temporal dimension), or even leave them as it is in scalar form depending upon your use case. Please follow Converting a Keras model to a spiking neural network — NengoDL 3.4.1.dev0 docs which talks about conversion process in more detail. You will find more info about curating the data there. Do note that the loss function used there is tf.keras.losses.SparseCategoricalCrossentropy(), so the curation of training labels might vary (compared to here).

Essentially, you should look for the required shapes at each step of your data curation process, similar to above. For your currently posted code, it has lots of unwanted details. If you are still facing issues in loading/curating data, please post minimal code to reproduce it and the exact error you are facing.

Thank you for the great explanation. I understand how to format the data after loading it in. But how would I load the dataset in the first place. Obviously, I can’t just use the .load_data() function for my dataset of images. I was wondering if there was a set method to do so because the code which I have written to try and load the data isn’t working; hence get values for the train_images, test_images,test_labels and train_labels. Thanks.

Loading data for Nengo-DL is no different than loading data for other ML or data science tasks. In most cases of videos and images, all you need is numpy matrices of train/test images and train/test labels. What’s the specific error you are getting? Can you please post that?

After following the steps in regarding to importing the data, and reformatting it, this is the error:

I have posted my method of importing the data but it seems to have a lot of flaws to it. I have followed the steps that you outlined. Here’s the whole code for clarity:

import pandas as pd
import numpy as np
import os
import tensorflow as tf
import cv2
import scipy
import nengo
import nengo_dl
import random
from  matplotlib import pyplot as plt
import matplotlib.image as mpimg


# For shuffling the data
def unison_shuffled_copies_train(a, b):
    assert len(a) == len(b)
    p = np.random.permutation(len(a))
    return a[p], b[p]

def unison_shuffled_copies_test(a, b):
    assert len(a) == len(b)
    p = np.random.permutation(len(a))
    return a[p], b[p]

test_images, test_labels = unison_shuffled_copies_test(test_images,test_labels)

train_images = train_images.reshape((train_images.shape[0],1, -1))
test_images = test_images.reshape((test_images.shape[0],1, -1))
train_labels = train_labels.reshape((train_labels.shape[0]),1)
test_labels = test_labels.reshape((test_labels.shape[0]),1)
train_labels = np.eye(2, dtype=np.float32)[train_labels]

with nengo.Network(seed=0) as net:
    # set some default parameters for the neurons that will make
    # the training progress more smoothly
    net.config[nengo.Ensemble].max_rates = nengo.dists.Choice([100])
    net.config[nengo.Ensemble].intercepts = nengo.dists.Choice([0])
    net.config[nengo.Connection].synapse = None
    neuron_type = nengo.LIF(amplitude=0.01)

    # this is an optimization to improve the training speed,
    # since we won't require stateful behaviour in this example

    # the input node that will be used to feed in input images
    inp = nengo.Node(np.zeros(28 * 28 ))

    # add the first convolutional layer
    x = nengo_dl.Layer(tf.keras.layers.Conv2D(filters=32, kernel_size=3))(
        inp, shape_in=(28, 28, 1)
    x = nengo_dl.Layer(neuron_type)(x)

    # add the second convolutional layer
    x = nengo_dl.Layer(tf.keras.layers.Conv2D(filters=64, strides=2, kernel_size=3))(
        x, shape_in=(26, 26, 32)
    x = nengo_dl.Layer(neuron_type)(x)

    # add the third convolutional layer
    x = nengo_dl.Layer(tf.keras.layers.Conv2D(filters=128, strides=2, kernel_size=3))(
        x, shape_in=(12, 12, 64)
    x = nengo_dl.Layer(neuron_type)(x)

    # linear readout
    out = nengo_dl.Layer(tf.keras.layers.Dense(units=10))(x)

    # we'll create two different output probes, one with a filter
    # (for when we're simulating the network over time and
    # accumulating spikes), and one without (for when we're
    # training the network using a rate-based approximation)
    out_p = nengo.Probe(out, label="out_p")
    out_p_filt = nengo.Probe(out, synapse=0.1, label="out_p_filt")

minibatch_size = 20
sim = nengo_dl.Simulator(net, minibatch_size=minibatch_size)
# add single timestep to training data
# train_images = np.concatenate( train_images, axis=0 )

# when testing our network with spiking neurons we will need to run it
# over time, so we repeat the input/target data for a number of
# timesteps.

n_steps = 30
train_images = train_images[:,None,:]
train_labels = train_labels[:,None, None]
test_images = np.tile(test_images[:,None :], (1, n_steps, 1))
test_labels = np.tile(test_labels[:, None, None], (1, n_steps, 1))

def classification_accuracy(y_true, y_pred):
    return tf.metrics.sparse_categorical_accuracy(y_true[:, -1], y_pred[:, -1])

sim.compile(loss={out_p_filt: classification_accuracy})
def classification_accuracy(y_true, y_pred):
    return tf.metrics.sparse_categorical_accuracy(y_true[:, -1], y_pred[:, -1])

# note that we use `out_p_filt` when testing (to reduce the spike noise)
sim.compile(loss={out_p_filt: classification_accuracy})
    "Accuracy before training:",
    sim.evaluate(test_images, {out_p_filt: test_labels}, verbose=0)["loss"],
do_training = False
if do_training:
    print("here now")
    # run training
        loss={out_p: tf.losses.SparseCategoricalCrossentropy(from_logits=True)},
    ), {out_p: train_labels}, epochs=10)

    # save the parameters to file
    # download pretrained weights

    # load parameters
    sim.compile(loss={out_p_filt: classification_accuracy})
    "Accuracy after training:",
    sim.evaluate(test_images, {out_p_filt: test_labels}, verbose=0)["loss"],
data = sim.predict(test_images[:minibatch_size])

for i in range(7):
    plt.figure(figsize=(8, 4))
    plt.subplot(1, 2, 1)
    plt.imshow(test_images[i, 0].reshape((29, 28)), cmap="gray")

    plt.subplot(1, 2, 2)
    plt.legend([str(i) for i in range(10)], loc="upper left")

From the error message you uploaded, the issue is that your test_labels are not properly formatted. The simulator’s evaluate function is expecting the test_labels of shape (batch_size, n_steps, dimension) but you are passing test_labels of shape 4 i.e. on extra dimension. I am guessing that the correct shape of your test_labels should be (20, 30, 1) (from your code it appears that you have set minibatch_size = 20, and n_steps = 30). The issue is in line test_labels = np.tile(test_labels[:, None, None], (1, n_steps, 1)) where you are introducing one extra dimension. I think it should be test_labels = np.tile(test_labels[:, None], (1, n_steps, 1)).

Keep in mind that in sparse_categorical_accuracy, it expects y_true to be actual classes labels and y_pred values to be one hot encoded. More info here: python - Keras - Difference between categorical_accuracy and sparse_categorical_accuracy - Stack Overflow .

A small example is below.

>>> test_y
       [7]], dtype=uint8)
>>> test_y.shape
(10000, 1)
>>> test_y_binary = np.eye(10, dtype=np.float32)[test_y].squeeze(1)
>>> test_y_binary
array([[0., 0., 0., ..., 0., 0., 0.],
       [0., 0., 0., ..., 0., 1., 0.],
       [0., 0., 0., ..., 0., 1., 0.],
       [0., 0., 0., ..., 0., 0., 0.],
       [0., 1., 0., ..., 0., 0., 0.],
       [0., 0., 0., ..., 1., 0., 0.]], dtype=float32)
>>> test_y_binary.shape
(10000, 10)
>>> np.mean(tf.metrics.sparse_categorical_accuracy(test_y, test_y_binary))*100

Since I am passing binarized test_y_binary as predicted labels against the same test_y, I am getting 100% accuracy. So I think, your arguments in tf.metrics.sparse_categorical_accuracy() should be of shape (20, 30, 1) and (20, 30, 2) as true and predicted labels respectively. So do check the shapes of the labels before you pass them to tf.metrics.sparse_categorical_accuracy(). Next, I see that your model’s Input is of shape (28, 28). This is the shape of MNIST images, therefore do check if your images shape is also (28, 28). Also, I see that you are loading mnist_params.npz which certainly shouldn’t be the case when you are testing on a different dataset. Please ignore these comments if you were going to change/discard those lines anyways.

With respect to your code, you can preprocess your images and save them in a file, thus not only you will save time while training/testing your model multiple times, but also there will be less clutter of code for us to follow. For example, you can get rid of create_dataset_train and create_dataset_test functions as those are not necessary details for us to help debug your model.

Thanks for this explanation. I am still very confused on where each of the elements go. I am trying to put them all in the right places now and correcting the mistakes I made. And yes I will be discarding the mnist_params.npz . I am also slightly concerned with the shape of the train_images in this process. It’s got a 5d arrangement i.e. (255,1,1,1,2). Not sure what has happened there either. Despite all this it runs and prints the accuracy before traning as 0.008333 which isn’t ideal at all. I will get back on any further errors. Thanks.

Sure @CodeHelp1, I would advise you check for the shape train/test images/labels after each transformation you make and get them in line with the shapes I mentioned in my first post. BTW, while training our spiking/non-spiking network with training images, we generally keep the value n_steps to 1; while testing though we increase it accordingly. So your train_images should be of shape (batch_size, (n_steps=)1, length_of_flattened_images). With your binary classification problem, the accuracy before training should be around 50% random. Do post the exact errors for us to help you.

Hi so, I’ve looked through everything and I can confirm these shapes:

Train_Labels: (135, 1, 2)
Test_Labels: (140, 30, 1)
Train_Images: (135, 1, 784)
Test_Images: (140, 30, 784)

I’m not totally sure if these are the correct shapes. However, my accuracy isn’t anywhere near 50%, it’s 0.05%. Also, the note on passing the shapes (20,30,1) and (20,30,2) above. I’m not totally sure how to pass these values in. I am able to create the test_y_binary but I am unable to pass it in, and I’m not quite sure where they were passing it through in the example. I believe it replaces the y_pred from the original code, but apparently the shape is too big if I replace the classification accuracy y_pred with the binarizer? Not sure what’s going on.

May I also re-iterate that I’m working on black and white images - I just came to realise that Cifar 10 is colour images.

Hello @CodeHelp1, can you upload your recent Nengo-DL code where you are facing issues with the data whose shapes you have mentioned? Or is it exactly the same as mentioned here: Optimizing a spiking neural network — NengoDL 3.4.1.dev0 docs ? And what’s the shape of your binary images? Is it same as MNIST images i.e. (28, 28, 1)?