Nengo-dl simulator and Nengo simulator

General Discussion
1

I’m using nengo-dl converter to convert a keras model to spiking model.
I simply use
converter = nengo_dl.Converter(model, allow_fallback=False)
assert converter.verify()
nengo_dl.Simulator(converter.net)
…

From my understanding, if we set allow_feedback to false in Converter, the model is converted to spiking model. The question is should we use nengo_dl.simulator to simulate the spiking inference or should we use nengo.simulator? What is the difference between those two?

2

Setting it to False ensures that all parts of the Keras model are converted to Nengo objects, yes. However, Nengo can also simulate non-spiking networks, so it’s almost definitely the case that your model is not spiking, it is a rate-based Nengo model.

In order to convert the model to spiking, you can use the swap_activations argument. For example, if your model exclusively uses tf.nn.relu activations, then you can swap those with a spiking rectified linear activation function with

converter = nengo_dl.Converter(
    model,
    allow_fallback=False,
    swap_activations={tf.nn.relu: nengo.SpikingRectifiedLinear()}
)

See the API docs for more information about the swap_activations argument, and the other arguments to Converter.

The difference between the two simulators is that nengo_dl.Simulator uses TensorFlow to run the simulation under the hood. nengo.Simulator only uses NumPy as it is typically easier to most users to install. If you have TensorFlow installed (especially if GPU support is enabled) then nengo_dl.Simulator should be much faster than nengo.Simulator, so I would recommend using nengo_dl.Simulator. The behavior of both simulators should be identical.

3

Hi Trevor,

Thank you for your reply. Can I directly use rate-based model on the nengo neural simulator? e.g. nengo_loihi. Because the end goal for me is to take advantage of the power efficiency of the neuromorphic hardware.

4

Each simulator (i.e., backend) supports different types of neurons, so it will depend on the device. Nengo Loihi is a purely neuromorphic backend and only supports the nengo.LIF and nengo.SpikingRectifiedLinear neuron types, so no, the rate-based model will not work run on hardware with Nengo Loihi. The reference simulator (nengo.Simulator), Nengo DL and Nengo OpenCL all work with rate-based models.

5

I see. I understand that spiking neurons gain huge computational advantage on neuromorphic hardware. Are there any paticular reason/benefits of using rate-based models?

6

Yes, neuromorphic hardware can be much more power efficient (see e.g. this paper), in part because they can run spiking neurons very fast.

Nengo supports both for several reasons. First, if you’re reimplementing a network described in a paper or with an existing TensorFlow implementation, it’s enables apples-to-apples comparisons. If a spiking implementation performs worse than a non-spiking implementation, it’s not clear if it’s because of the conversion to spiking or if it’s because the network architecture was not implemented correctly. Second, while spiking neurons are computationally efficient on neuromorphic hardware, they may or may not be as efficient on other hardware, like general purpose CPUs and GPUs. By allowing both, Nengo makes it possible to choose the neuron type that makes sense for the network architecture and for the hardware that the network architecture is running on.

It is also worth noting that @arvoelke has recently been doing work that suggests that we shouldn’t think of neurons as either “rate-based” or “spiking”, but instead lying somewhere on a gradient between those two. The types of neurons used on Loihi, for example, are at the “spiking” extreme of that gradient, as they can only fire binary spike events. Most other unconstrained neurons would be on the “rate-based” extreme of the gradient, as their output activity can be any float. However, if you add the constraint that the output activity must fit into a certain bit width, then you get something that is in between those two. For example, a two-bit neuron could send both positive and negative spikes. A three-bit neuron can send a very rough approximation of a rate. An eight-bit neuron approximates the actual rate even better, and so on. We are looking into how to exploit this idea within Nengo to train and optimize networks better for specific pieces of hardware.

7

Hi Trevor,

The gradient concept is great. From my understanding, the reason why neuromorphic hardware is so computational efficient is that it only need to do “addition” operations if the outputs of neuros are binary spike events. Event a three-bit neuron can not have that operation advantage (I might be wrong here, not much knowledge on the hardware side). It’s definetely useful to have nengo as a tool to optimize 32-bit neural networks to smaller bit number. But I suppose it will not use any of nengo’s spiking neuron type. If I understand it correcly, the implementation of gradient concept will make nengo similar to exiting frameworks like tensorflow with additional optimization algorithms designed for lower bit neurons. Please correct me if I’m wrong here. Thannk you.

8

I’m not well versed on the hardware side either, but I think there are a lot of other techniques being used in neuromorphic hardware in addition to reducing the need for floating point operations. Regardless, the reason why we think the gradient concept is so important for neuromorphics is so that you can more efficiently and effectively train spiking neural networks. In all of the ANN to SNN conversion techniques that I’m aware of, the goal is to minimize the performance difference between a network on the rate-based extreme of the gradient and a network on the spiking extreme of the gradient. By breaking down the problem in steps – so, going from 32-bit neurons to 16-bit neurons, then 16-bit to 8-bit, and so on – we may be able to train spiking networks (i.e., 1-bit neurons) faster and more effectively. This is a similar technique as has been exploited in many other aspects of machine learning, breaking a difficult problem down to easier-to-solve subproblems. Once the training procedure is done, the spiking network that you get is no different from a spiking network that you obtained through another trianing procedure, so it would be deployable on hardware like Loihi.

1 Like
9

I tried to use
nengo_converter = nengo_dl.Converter(model, allow_fallback=False, swap_activations={tf.nn.relu: nengo.SpikingRectifiedLinear()})
as you suggested, but looks like the neurons are not converted to Spiking

    model = tf.keras.models.load_model(os.path.join(CHECKPOINT_FILE_PATH, MODEL_NAME))

    nengo_converter = nengo_dl.Converter(model, 
                                        allow_fallback=False, 
                                        swap_activations={tf.nn.relu: nengo.SpikingRectifiedLinear()})
    net = nengo_converter.net
    for ensemble in net.ensembles: print(ensemble, ensemble.neuron_type)
    assert nengo_converter.verify()

the ensemble prints:

<Ensemble "conv2d_4.0"> RectifiedLinear()
<Ensemble "conv2d_5.0"> RectifiedLinear()
<Ensemble "conv2d_6.0"> RectifiedLinear()
<Ensemble "conv2d_7.0"> RectifiedLinear()
<Ensemble "conv2d_8.0"> RectifiedLinear()
<Ensemble "conv2d_9.0"> RectifiedLinear()

Then I tried to use swap_activations={nengo.RectifiedLinear(): nengo.SpikingRectifiedLinear()} as suggested from Daniel from a different topic. This time, the neuron converted to spiking however, an error raised from function

assert nengo_converter.verify()

ValueError: Output of Keras model does not match output of converted Nengo network
My questions are

  1. Is it a known issue with swap_activations={tf.nn.relu: nengo.SpikingRectifiedLinear()}?
  2. Is it safe to use swap_activations={nengo.RectifiedLinear(): nengo.SpikingRectifiedLinear()} in nengo_dl.Converter to make sure the model transfered to spiking.
    swap_activations={tf.nn.relu: nengo:SpikingRectifiedLinear()} makes more sense to me since I’m loading a keras model. The assert nengo_converter.verify() can’t work either. Are there any other parameters that I should be aware of to make the spiking conversion pass the verification function? Thank you
10

We have seen one other user having this issue, but we haven’t been able to reproduce it ourselves. If you can post your code and your TensorFlow version we can see if it will reproduce for us.

Yes, that solution is fine, it’s just puzzling why the tf.nn.relu method doesn’t work.

One thing to check would be whether it’s a “serious” verification failure, or a precision issue (caused by slight differences in the floating point math between Keras/Nengo). If you enable INFO-level log output import logging; logging.basicConfig(level=logging.INFO) it will print out the failing verification values. If they’re only off by a small margin then it’s probably just a floating point issue. There’s a soon-to-be-available change that will allow you to modify the sensitivity of converter.verify (https://github.com/nengo/nengo-dl/pull/130).

If there is a significant difference between the Keras and Nengo version then that probably indicates a bug in the converter, if you post your code here I can look into it.