[Nengo-Loihi]: Adding NxSDK code to Nengo-Loihi/Nengo

[Eric]

It’s really hard to suggest the best way to proceed without a concrete goal or example of exactly what you want to accomplish.

Part of it depends on how close your goal is to what can already be done with NengoLoihi. If it is fairly similar, then you could make your model in NengoLoihi, and then go in and just tweak the parts you need on the NxSDK board object. For example, you could build your network as normal and then go in and add delays to particular synapses (something that we don’t currently support in NengoLoihi).

One thing we’ve done before is writing custom SNIPs for a particular project. This is not too hard to do with NengoLoihi; the main thing you have to do is figure out where the neurons (or whatever else) you want to interact with are located on the board, and then you can target those things in your SNIP, and use the NxSDK interface to add the SNIP.

On the other hand, if what you want to do is very far from what can be done in NengoLoihi, then it might be better to write it separately in NxSDK. The difficulty with this is that NxSDK requires you to configure the number of chips/cores/synapses when you create the NxSDK board object; we do this here. So you’d somehow have to hack that to leave extra space for whatever you want to build, and then build into that space. One way to hack it might be to have one or more Ensembles that do nothing and aren’t connected to anything, which you can then find and overwrite on the board with your custom stuff.

[zerone]

Hello @Eric, I see that it is getting more complex. I don’t have any experience of writing custom SNIPs. BTW, to explain what I want to achieve at the end of the day, I would like to expand on the following:

@zerone

Let’s suppose I want to develop a custom network of few connected individual neurons which will execute low level NxSDK APIs (e.g. its join functions),

I want to leverage the “join operations” that have been mentioned in the u_join_op_in_multi_compartment_neuron.ipynb tutorial file in the INRC VM nxsdk directory. A short description about the “join operation” has also been mentioned in the Loihi paper as follows:

The compartments’ state variables are combined in a configurable 
manner by programming different join functions for each 
compartment junction.

I am not sure if you are aware of them, therefore mentioning these following details (sorry for the blurt if you already are). The compartmental neurons in a neurocore are arranged in binary tree structure on the Loihi chip, and the parent neuron “joins” the output from its two individual child neurons. Different “join” ops could be “ADD”, “MAX”, “MIN”, etc. Now, the ipynb file I mentioned above, has shown how to do different “join” ops using the NxSDK APIs; I was wondering to do the same, albeit not limiting to just one instance of a join op, rather create a layer of many such instances. The input to and output from the “join” op layer will be from a previous Conv and to the next Conv layer respectively; thus I will have to integrate my custom NxSDK code with NengoLoihi.

In light of above, what could be the best way to progress? Is it possible to create such a layer of “join” ops directly with NengoLoihi or do I write it in NxSDK API explicitly and then somehow try to integrate it with the rest of my network in NengoLoihi?

[Eric]

My instinct would be to create an ensemble, connect in to a set of input neurons in that ensemble (which will be represented on Loihi as single compartments), connect out from a separate set of output neurons (which will create output compartments), then go in and manually connect those input/output compartments.

Here’s a sketch:

with nengo.Network() as net:
    n = 10  # number of input/output compartments
    phases = np.linspace(0, 2*np.pi, n)
    inp = nengo.Node(lambda t: np.sin(t + phases))

    ens = nengo.Ensemble(2 * n, 1)
    out = nengo.Node(size_in=n)
    probe = nengo.Probe(out)

    nengo.Connection(inp, ens.neurons[:n])
    nengo.Connection(ens.neurons[n:], out)

sim = nengo_loihi.Simulator(net)
hw_interface = sim.sims["loihi"]
nxsdk_board = hw_interface.nxsdk_board
board = hw_interface.board

blocks = sim.model.objs[ens]
assert len(blocks) == 1
block = blocks[0]

# get the nengo_loihi.block.Synapse
synapse = sim.model.objs[input_conn]["weights"]

chip_idx, core_idx, block_idx, compartment_idxs, _ = board.find_block(block)
nxsdk_core = nxsdk_board.n2Chips[chip_idx].n2CoresAsList[core_idx]

synapse_unique_idxs = np.unique(synapse.indices)
input_compartment_idxs = [compartment_idxs[i] for i in synapse_unique_idxs]
output_compartment_idxs = [i for i in compartment_idxs if i not in input_compartment_idxs]

# do configuration here using nxsdk_core, and input/output compartment idxs
# NengoLoihi currently only puts one block per core, so you should be able to use
# any compartments not in `compartment_idxs` as extra compartments that you can
# configure yourself. 

[zerone]

Hello @Eric, thanks for the explanation and the sample code. About the nxsdk_board and board object, I guess, they will be created only when a physical board would be present. Right?

Also about configuring/manually connecting the input/output neurons, I would be required to makes sure that the input neurons are twice in numbers compared to the number of output neurons and moreover, each group of two input neurons should be the children nodes of the parent output neuron. Will it be possible with your method of obtaining the input_compartment_idxs and output_compartment_idxs? I am sorry but I wasn’t able to follow the code quite clearly after the line # get the nengo_loihi.block.Synapse.

[Eric]

So essentially each nengo_loihi.block.Synapse is responsible for one nengo.Connection, and on the Loihi board connects input axons to compartments using synaptic weights. If you want your input compartments to each be coming from different connections, then you can just repeat what I have above to get separate input_compartment_idxs for each connection. If, on the other hand, all your input compartments are being fed by the same connection, then you’ll have to take the single list of input_compartment_idxs and group them yourself (I assume in something like a pair of input compartments for each output compartment).

As for your first question, yes, this is really only possible when you have a board to connect to. Technically, all that’s required to create the board and nxsdk_board is for NxSDK to be installed, so it would be possible to run that on your machine without a board if you have NxSDK, but then of course there’s no way to actually test that your changes to the nxsdk_board do what you want if you can’t run it.

[zerone]

@Eric

If you want your input compartments to each be coming from different connections, then you can just repeat what I have above to get separate input_compartment_idxs for each connection. If, on the other hand, all your input compartments are being fed by the same connection, then you’ll have to take the single list of input_compartment_idxs and group them yourself

With respect to the above, the input to the single compartment neurons would be the output of the individual neurons of the previous Ensemble (i.e. the previous Conv layer). Now, those input compartments would be connected in pairs to another compartment which is supposed to be the immediate parent of them, whose (i.e. the parent’s) output would then be connected to the individual neurons of the next Ensemble.

So of course, unless I have understood things wrongly, there will be just one Connection from the previous Ensemble to my layer of compartmental neurons, and just one Connection again from the output (of my layer of compartmental neurons) to the next Ensemble, but the connections are supposed to be neuron-wise. Will this neuron-wise connections between Ensembles’ neurons and compartmental neurons be possible with NengoLoihi? I can see that we can obtain the input/output indices of the compartment neurons of the ens block (you created earlier) as below:

@Eric

input_compartment_idxs = [compartment_idxs[i] for i in synapse_unique_idxs]
output_compartment_idxs = [i for i in compartment_idxs if i not in input_compartment_idxs]

and I believe, the following creates the one to one connection from the inp Node and out Node to the compartmental neurons of the ens block:

@Eric

nengo.Connection(inp, ens.neurons[:n])
    nengo.Connection(ens.neurons[n:], out)

but this is with Node to Ensemble (and Ensemble to Node).

Also, I guess… I will have to take help of NxSDK APIs (and not NengoLoihi) to enforce the child - parent relationship of the compartment neurons in my custom layer. Right?

And thanks for the confirmation of the requirement of a physical board to test out the custom layer of compartmental neurons, but I am slightly confused by the following:

@Eric

so it would be possible to run that on your machine without a board if you have NxSDK,

Does it mean that the board and nxsdk_board objects can still be created in absence of a physical board? Here, I am learning something different, hence the confusion: " If the board isn’t available, the nxsdk_board object cannot be created (or used).".

[Eric]

It’s really a hack to be creating the nxsdk_board without hardware being available. All I’m saying is nothing in the code checks that hardware is available until with sim:, so theoretically anything before that can run as long as NxSDK is installed.

@zerone

Also, I guess… I will have to take help of NxSDK APIs (and not NengoLoihi) to enforce the child - parent relationship of the compartment neurons in my custom layer. Right?

Yes.

@zerone

Will this neuron-wise connections between Ensembles’ neurons and compartmental neurons be possible with NengoLoihi?

Yes. In NengoLoihi, neurons are currently equivalent to compartments. i.e. each neuron is a single compartment neuron. So if the neurons you want to create are 3-compartment neurons, then you can create an Ensemble with 3 * n neurons. How you index things is up to you; e.g. you could have your inputs be the first 2 * n compartments, and use the last n compartments for outputs. Then you’d connect your inputs to ens.neurons[:2*n], and connect your outputs from ens.neurons[2*n:]. Then you just have to use NxSDK to wire things up appropriately within the Ensemble.

[zerone]

@Eric

Then you’d connect your inputs to ens.neurons[:2*n], and connect your outputs from ens.neurons[2*n:]. Then you just have to use NxSDK to wire things up appropriately within the Ensemble.

The above resolves my doubts for now @Eric. I will now wait to make some progress with the code before asking informed questions. This discussion has pretty much given me some ideas about heading into that direction.

[zerone]

Hello @Eric, progressing ahead I got stuck in two issues.

First: For the following network:

with nengo.Network() as net:
    ens = nengo.Ensemble(n_neurons=4, dimensions=1) # 4 single compartment neurons with dimension = 1.
    sg_node_1 = nengo.Node(output=get_spikes_1)
    sg_node_2 = nengo.Node(output=get_spikes_2)
    sg_node_3 = nengo.Node(output=get_spikes_3)
    sg_node_4 = nengo.Node(output=get_spikes_4)
    
    nengo.Connection(sg_node_1, ens.neurons[0], synapse=0.005)
    nengo.Connection(sg_node_2, ens.neurons[1], synapse=0.005)
    nengo.Connection(sg_node_3, ens.neurons[2], synapse=0.005)
    nengo.Connection(sg_node_4, ens.neurons[3], synapse=0.005)
    
    n1_probe = nengo.Probe(ens.neurons[0], attr="input")
    n2_probe = nengo.Probe(ens.neurons[1], attr="input")
    n3_probe = nengo.Probe(ens.neurons[2], attr="input")
    n4_probe = nengo.Probe(ens.neurons[3], attr="input")

when I try to execute:

sim = nengo_loihi.Simulator(net)

it throws the following error:

    648     elif isinstance(conn.post_obj, Neurons):
    649         assert isinstance(post_obj, LoihiBlock)
--> 650         assert post_slice == slice(None)
    651         if loihi_weights is None:
    652             raise NotImplementedError("Need weights for connection to neurons")

AssertionError: 

on both, NengoLoihi 1.0.0 and 1.1.0.dev0. I get the same error when I try executing your code. Therefore I am unable to get the hw_interface. How do I fix it?

Second:
It is related more to Nengo. I am trying to plot the input synaptic currents to each of the four neurons (in the code above) by running it in Nengo Simulator. I observe the following plot.

My expectation was that all the currents would start at 0, but I see that they are starting around 1 and that too with random start values (instead of 0). When we feed in the spikes to the individual neurons, with no function or transform mentioned in the connection, then only synapse comes into action. Therefore, the input currents should start at 0, why isn’t it happening here?

Please let me know.

[Eric]

It looks like we haven’t implemented post slicing yet in NengoLoihi; I had forgotten that. The easiest way to get around this if you’re coming from nodes is to just have one super-node that does all the neuron inputs at once, e.g:

with nengo.Network() as net:
    def super_node_fn(t):
        return [get_spikes_1(t), get_spikes_2(t), get_spikes_3(t), get_spikes_4(t)]

    super_node = nengo.Node(super_node_fn)
    nengo.Connection(super_node, ens.neurons)

If you have input coming from Ensembles, the easiest is probably to have a full transform but to blank out some of the rows so that you’re just inputting to some neurons. The downside to this is you’re using a lot of synapse memory storing zeros. We do have some support for nengo.Sparse transforms, which would be the more efficient way to go.

Since Nengo 3.1, we initialize neuron voltages with random values between 0 and 1, rather than having them all start at 0. This helps mitigate startup transients. NengoLoihi doesn’t support this yet; all neurons still start at 0. If you use e.g. a nengo_loihi.LoihiLIF neuron, it has the voltage set to 0. Or you can call nengo_loihi.set_defaults() (you can see how we set the initial voltage there), or you can use the initial_state argument on e.g. the nengo.LIF constructor.

[zerone]

Hello @Eric, your suggestion of using a single Node to input all the 4 spike trains to the ensemble’s neurons work in NengoLoihi.

However, with respect the plot I mentioned in my last comment, it’s about the input current and not the membrane voltages. Anyways… I figured it out. Not sure why I keep forgetting…

So the synapsed inputs (which start from 0) from the spike generators are $x$ to the equation $J = \alpha \times x + J_{bias}$, and the input currents (to the individual neurons) I am plotting are the values $J$. Since, $\alpha$ and $J_{bias}$ are chosen randomly, they affect the input values $x$ and thus the plotted $J$'s start at different non-zero values. Upon fixing the $\alpha$ and $J_{bias}$ for all the neurons, I can see that their input current $J$ starts at the same point (although not zero, i.e. at $J_{bias}$).

[zerone]

Okay… so I proceeded with the following code and executed it on INRC.

def input_spikes(t):
    return [get_spikes_1(t), get_spikes_2(t), get_spikes_3(t), get_spikes_4(t)]

with nengo.Network() as net:
    ens = nengo.Ensemble(
        n_neurons=4, 
        dimensions=1,
        gain=np.array([1, 1, 1, 1]),
        bias=np.array([0.5, 0.5, 0.5, 0.5]),
    ) # 4 single compartment neurons with dimension = 1.

    inp = nengo.Node(input_spikes)
    
    nengo.Connection(inp, ens.neurons, synapse=0.005)

    n1_probe = nengo.Probe(ens.neurons[0], attr="input")
    n2_probe = nengo.Probe(ens.neurons[1], attr="input")
    n3_probe = nengo.Probe(ens.neurons[2], attr="input")
    n4_probe = nengo.Probe(ens.neurons[3], attr="input")

It execute well and I could plot the input currents to the individual neurons. Upon executing the following code (as you mentioned here):

hw_interface = sim.sims["loihi"]
nxsdk_board = hw_interface.nxsdk_board
board = hw_interface.board

print(board.n_chips, board.n_cores_per_chip, board.n_synapses_per_core)

following is the output:

1 [2] [[8, 0]]

I believe the above implies that the board I have access to is having one Loihi chip, and there are 2 neurocores? per chip, and there are 8 and 0 synapses in each neurocore. Please correct me if I am wrong here.

Next, the output of following:

blocks = sim.model.objs[ens]
print(blocks)

is:

{'in': [<nengo_loihi.block.LoihiBlock object at 0x7f963d7f7640>], 'out': [<nengo_loihi.block.LoihiBlock object at 0x7f963d7f7640>]}

which is a dictionary and not a list. So I guess, assert len(blocks) == 1; block = blocks[0] won’t work here. Therefore I did the following:

in_chip_idx, in_core_idx, in_block_idx, in_compartment_idxs, _ = board.find_block(blocks["in"])
out_chip_idx, out_core_idx, out_block_idx, out_compartment_idxs, _ = board.find_block(blocks["out"])
print(in_chip_idx, in_core_idx, in_block_idx, in_compartment_idxs)
print(out_chip_idx, out_core_idx, out_block_idx, out_compartment_idxs)

and the output was:

None None None None
None None None None

which again means that this nxsdk_core = nxsdk_board.n2Chips[in_chip_idx].n2CoresAsList[in_core_idx] won’t work. I believe, in place of Nones there should have been some object IDs or so… Right? Am I executing any step wrongly or missing any (all these codes are run on INRC jupyter notebook)?

Since the join ops can’t be implemented by NengoLoihi and you suggested to simply allocate some space (in form of an Ensemble of neurons) on Loihi and then proceed with manually configuring them, I am attempting to do the same. With respect to my NxSDK code using the join ops (as you might have seen), how do I proceed next to configure the same? Please let me know.

[zerone]

Hello @Eric, I was able to get past the issue posted above. It was a silly detail. In the following line:

in_chip_idx, in_core_idx, in_block_idx, in_compartment_idxs, _ = board.find_block(blocks["in"])

instead of blocks["in"] I had to pass blocks["in"][0] i.e. the list element of blocks["in"]. Subsequently I got the values for in_chip_idx, in_core_idx, in_block_idx, in_compartment_idxs which are 0 0 0 [0 1 2 3] respectively. That is the in_compartment_idxs are [0, 1, 2, 3] which are the same as out_compartment_idxs (obtained from blocks["out"][0]) and I suppose… there are 4 indices since there are 4 neurons in my Ensemble.

I am stuck at the next step, i.e. how do I access the compartments in their raw NxSDK form i.e. as a compartment prototype and connect them by calling the NxSDK APIs (e.g. for adding dendrites)? After connecting the compartments, I need to create a single neuron out of them. Please let me know.

[Eric]

Assuming I’m correctly understanding what you’re referring to, the compartment prototypes and “dendrites” etc. are part of NxNet, which is an interface on top of NxSDK. Essentially, NxNet lets you define things at a bit of a higher level, and then have these “compile” down to NxSDK instructions that set the proper registers.

I’m not sure if this will work in your case, because I’m not sure if it’s possible to have NxNet target specific compartments on a specific core. Also, it might overwrite some other configuration. I think it’s better to use “pure” NxSDK (without NxNet), which will give you register-level control over how things are configured. That said, I don’t have as much experience with NxNet, especially recently, so there may be a way to do it.

The NxSDK apps package has a number of tutorials that might be useful if you’re trying to figure out how to do this in pure NxSDK (just make sure you stick to the nxcore ones, since those don’t use NxNet). This one might be particularly useful: nxsdk-apps-0.9.8/tutorials/nxcore/tutorial_15_two_compartment_neuron.py.

[zerone]

I will be working on the above suggestion; however I had two small questions.

1> On a sample network I have, it has non-passthrough nodes and ensembles in it. So when I execute the network via NengoLoihi simulator on Loihi, it executes well, however, it doesn’t throw any related warnings per se, rather: UserWarning: NengoLoihi does not support initial values for 'voltage' being non-zero on LIF neurons. On the chip, all values will be initialized to zero. i.e. about LIF neurons (which seems reasonable to me). I do know that non-passthrough nodes are supposed to be executed on PC/CPUs, how do I get to programmatically see that Non-Passthrough nodes aren’t mapped to Neurocores, e.g. its some attribute being false or so?

2> On another network which has only passthrough nodes and ensembles, upon executing it via NengoLoihi on Loihi, I get the following warning: UserWarning: Combining two Lowpass synapses, this may change the behaviour of the network (set remove_passthrough=False to avoid this). Can you please explain what sort of change in behaviour can I expect? I am assuming that the entire network (with passthrough nodes) is mapped to Loihi chips, i.e. passthrough nodes are first removed, followed by network optimization (I assume) and then mapped. Therefore a way to programmatically check whether passthrough nodes or non-passthrough nodes are mapped to Loihi chip or not will be insightful.

[zerone]

Hello @Eric, with respect to my 1st question above, I did some related experiments with a passthrough node, which was supposed to aid in two consecutive transforms. Unfortunately, NengoLoihi throws error that merging transforms work only with Dense transforms and it suggested to set remove_passthrough=False. So this clarifies my first doubt to check programmatically… that passthrough nodes are set (True) to be removed. Although, upon setting remove_passthrough=False, I got the same error

BuildError: Conv2D transforms not supported for off-chip to on-chip connections where `pre` is not a Neurons object.

as mentioned here. But I was able to circumvent it.

With respect to my 2nd question above, please let me your points.

[Eric]

@zerone

On another network which has only passthrough nodes and ensembles, upon executing it via NengoLoihi on Loihi, I get the following warning: UserWarning: Combining two Lowpass synapses, this may change the behaviour of the network (set remove_passthrough=False to avoid this).

If you’ve got a passthrough node with connections both in and out that have synapses, then when removing the node, we have to combine those synapses somehow. So what we do is add the tau values (we currently only support Lowpass synapses in NengoLoihi, so everything has to be a Lowpass). But a Lowpass synapse with e.g. tau=0.01 will be different than two Lowpass synapses in a row with tau=0.005 (two Lowpass synapses in a row with the same tau are actually equal to an Alpha synapse with that tau).

If you want to look more at how this will affect filtering, our LinearFilter synapse has combine and evaluate functions that might be useful. e.g. You can combine two Lowpass filters with different taus Lowpass(tau1).combine(Lowpass(tau2)) and then evaluate the result to get the frequency response, etc.

If you want to make sure your network is equivalent before and after removing passthroughs, make sure that each passthrough node only has a synapse on either the input or output connection (not both), by setting synapse=None on one of the connections.

[zerone]

Thank you @Eric. This resolves my doubt for now. Will experiment and let you know.

[ntquyen2]

Hi @Eric,
I follow Installation — NengoLoihi 1.1.0.dev0 docs to install nxsdk, but I can’t download nxsdk-0.9.0.tar.gz. Can you help me where to download this release?

[xchoo]

Hi @ntquyen2,

The nxsdk-0.9.0.tar.gz file is developed by Intel and available to all members of the INRC (Intel Neuromorphic Research Community). If you wish to participate in the INRC, you should contact Intel with the request. I believe the email to send them any inquiries is: inrc_interest@intel.com

If you are already part of the INRC, you can download the NxSDK installation files from the INRC cloud. It is available at /nfs/ncl/releases/0.9/nxsdk-0.9.tar.gz