Parameter space exploration, nengo_ocl and SpiNNaker

[nikhilgarg]

I have been trying to do MNIST classification with a simple 1 layer network and unsupervised Hebbian learning. There are 784 input neurons and 20 output neurons(Fully connected).
Now a single iteration of the entire MNIST dataset (70,000 images, 350ms presentation time for each image) takes around 40 hours on CPU and a little less time with Nengo_ocl (on Google Collab GPU).

For statistical significance, I need to repeat the simulation with a different seed 10 times, and if I imagine some parameter exploration, the estimated time for simulation looks non-feasible.

I was wondering if it’s possible to run 100 such simulations in parallel with help of nengo_ocl or spinnaker. Is it possible to do so?

[xchoo]

Hmmm, I see that you have a bit of a predicament. I’ll try an elaborate on some of your options below:

NengoOCL
NengoOCL doesn’t limit the number of Nengo model you can run on one GPU. However, from my experience, running multiple Nengo models on a single GPU doesn’t improve your throughput at all. Rather, because of the additional I/O needed, you get no gains in throughput (i.e., running 2 models on one GPU halves the speed of both simulations, resulting in a net gain of 0).

Thus, you really only have one option here, and that is to run your code on multiple GPUs. I’m not sure how to do it on Google Colab, but to get an improvement in your throughput, you’ll need to spawn multiple instances on your Jupyter notebook on separate GPUs.

@Eric is the primary author of NengoOCL, and he may have some suggestions on how to speed up your network with NengoOCL.

SpiNNaker
NengoSpinnaker is designed to run Nengo simulations in real time, so, doing some quick math, one simulation run with 70,000 images (at 350ms each) will take roughly 7 hours to complete. That’s slightly more manageable, but 100 runs will still take roughly a month to finish collecting all of the data. You can improve the throughput by running NengoSpinnaker simulations in parallel, but because one SpiNNaker board supports only one NengoSpinnaker model, you’ll need access to multiple SpiNNaker boards to accomplish this.

Your Model
I suspect the reason why your model is taking a while to finish running is because of the online learning rule that is incorporated into the model. We do have other tools (e.g., NengoDL) that can perform the learning in an offline manner, but will train the model much quicker. I’m not sure what your goals are with your network are, but it may be worthwhile investigating NengoDL as well (see this example for an MNIST example).

[nikhilgarg]

Thanks a lot for your detailed reply.

@xchoo

Thus, you really only have one option here, and that is to run your code on multiple GPUs. I’m not sure how to do it on Google Colab, but to get an improvement in your throughput, you’ll need to spawn multiple instances on your Jupyter notebook on separate GPUs.

Oh okay. Seems like running multiple simulations with opencl not a very feasible option then.

@xchoo

NengoSpinnaker is designed to run Nengo simulations in real time, so, doing some quick math, one simulation run with 70,000 images (at 350ms each) will take roughly 7 hours to complete. That’s slightly more manageable, but 100 runs will still take roughly a month to finish collecting all of the data. You can improve the throughput by running NengoSpinnaker simulations in parallel, but because one SpiNNaker board supports only one NengoSpinnaker model, you’ll need access to multiple SpiNNaker boards to accomplish this.

My access to Spinnaker is limited to one offered by the HBP portal. I do not have a physical board. Any idea if I can run multiple simulations in parallel using a single job?

@xchoo

I suspect the reason why your model is taking a while to finish running is because of the online learning rule that is incorporated into the model. We do have other tools (e.g., NengoDL) that can perform the learning in an offline manner but will train the model much quicker. I’m not sure what your goals are with your network are, but it may be worthwhile investigating NengoDL as well (see this example for an MNIST example).

Very true. The whole idea is of online learning using Hebbian plasticity, so I am afraid that I cannot do an offline version of the same.

[xchoo]

I thought about your use case a bit more, and I think I might have a solution, but it’ll require some work on your end. One of the advantages of Nengo is that there are no restrictions on the network structure you can simulation. That being the case, you should be able to “parallelize” your model by literally creating parallel networks within one Nengo model.

Here’s a quick example of how you would do this:

def createModel():
    with nengo.Network() as net:
        # Define MNIST model here
        net.inp = nengo.Ensemble(784, 1)
        net.out = nengo.Ensemble(20, 1)
        # Define connections + learning rule
        nengo.Connection(net.inp, net.out, transform=[...])
    return net

# The "Parallel" model
probes = []
num_parallel_models = 100
with nengo.Network() as model:
    # The MNIST data input node
    mnist_node = nengo.Node(<function to feed MNIST data as vector>)

    for i in range(num_parallel_models):
        subnet = createModel()
        # Connect the MNIST input to the subnet (so you don't have to make an MNIST node for each subnet)
        nengo.Connection(mnist_node, subnet.inp)
        # Add to list of probes
        probes.append(nengo.Probe(subnet.out)

Now, this works in your case because the total number of neurons in your model is relatively small. Having several dozen of your small models running in “parallel” in the larger Nengo model should have an impact on your overall simulation run time, but you’ll be effectively collecting more data per run.

The experimentation you’ll have to do on your part is to figure out how many “models” you can run in parallel (i.e., the value of num_parallel_models) before you start to experience diminishing returns. You should experiment with a small subset of the dataset (say 1000 images), and see how changing the number of parallel models impact your overall simulation runtime. I’m wiling to bet that even at a 1000 parallel models, the impact on the run time would not be significant.

This approach should work with both NengoOCL, and NengoSpiNNaker (this is the beauty of Nengo. )

DISCLAIMER: The code I provided above is pseudo-ish code, and I haven’t tested it out myself, but it should be mostly correct.

[nikhilgarg]

@xchoo

The experimentation you’ll have to do on your part is to figure out how many “models” you can run in parallel (i.e., the value of num_parallel_models) before you start to experience diminishing returns. You should experiment with a small subset of the dataset (say 1000 images), and see how changing the number of parallel models impact your overall simulation runtime. I’m wiling to bet that even at a 1000 parallel models, the impact on the run time would not be significant.

This looks great. I shall run some simulations and get back.

[nikhilgarg]

Hello,

So lately I was trying to try running parallel models, but I have one issue :

def createModel():
    with nengo.Network() as net:
        net.input_layer = nengo.Ensemble(**input_neurons_args)
        net.layer1 = nengo.Ensemble(**layer_1_neurons_args)

        # Define connections + learning rule
        net.w = nengo.Node(CustomRule_post(**learning_args), size_in=n_in, size_out=n_neurons)
    	nengo.Connection(net.input_layer.neurons, w, synapse=None)
    	nengo.Connection(w, net.layer1.neurons,transform=g_max, synapse=None)

    	#Lateral inhibition
    	inhib = nengo.Connection(net.layer1.neurons,net.layer1.neurons,**lateral_inhib_args) 

    return net


# The "Parallel" model
probes = []
num_parallel_models = 100

with nengo.Network() as model:
    # The MNIST data input node
    picture = nengo.Node(PresentInputWithPause(images, presentation_time, pause_time,0))
    true_label = nengo.Node(PresentInputWithPause(labels, presentation_time, pause_time,-1))

    for i in range(num_parallel_models):
        subnet = createModel()
        # Connect the MNIST input to the subnet (so you don't have to make an MNIST node for each subnet)
        input_conn = nengo.Connection(picture,subnet.input_layer.neurons,synapse=None)

        # Add to list of probes
        
    	p_layer_1 = nengo.Probe(subnet.layer1.neurons, sample_every=probe_sample_rate)
    	weights = subnet.w.output.history

    	probes.append(nengo.Probe(p_layer_1)
		probes.append(weights)

Note that ‘w’ is a nengo.process, and implements a custom learning rule. I was setting the signal of ‘w’ as shown below in case of a single network

with nengo.Simulator(model) as sim:

w.output.set_signal_vmem(sim.signals[sim.model.sig[input_layer.neurons]["voltage"]])
w.output.set_signal_out(sim.signals[sim.model.sig[layer1.neurons]["out"]])
sim.run((presentation_time+pause_time) * labels.shape[0]*iterations)

But I am not able to think of a way of assigning these signals in case of parallel models as I don’t have list of these models. Can you help?

[xchoo]

I may not be completely understanding your code right, but could you create an array to store the references to each subnet, and then do the set_signal assignments after?

E.g.,

subnets = []
...
with nengo.Network() as model:
    ...
    for i in range(num_parallel_models):
        subnet = createModel()
        subnets.append(subnet)
        ...

Then

for subnet in subnets:
    subnet.w.output.set_signal_vmem(sim.signals[sim.model.sig[subnet.input_layer.neurons]["voltage"]])
    subnet.w.output.set_signal_out(sim.signals[sim.model.sig[subnet.layer1.neurons]["out"]])

sim.run((presentation_time+pause_time) * labels.shape[0]*iterations)

[nikhilgarg]

Thanks a lot. Solved my issue