I am running a spiking network for quite long on Loihi and collecting output probe data too, while running the network. Obviously the sim.data[some_probe] would consume high memory eventually. Therefore, I would like to save the contents of sim.data[some_probe] at regular intervals of time, and refresh/empty it such that it again accumulates data for the next interval of time. Below is a sample code where I am trying to do the same:
def func(t):
return 2
with nengo.Network() as temp_net:
inp = nengo.Node(func)
ens = nengo.Ensemble(100, 1, radius=2)
otp = nengo.Node(size_in=1)
nengo.Connection(inp, ens)
nengo.Connection(ens, otp)
probe_otp = nengo.Probe(otp)
loihi_sim = nengo_loihi.Simulator(temp_net)
with loihi_sim:
for _ in range(5):
loihi_sim.run(0.01)
print(loihi_sim.data[probe_otp])
# Save loihi_sim.data[probe_otp]
# Refresh/Empty loihi_sim.data[probe_otp]
Is there a way to do it? If so, please let me know!
I think this thread may be helpful to you. To summarize, in Nengo, you can clear probe data by doing:
If you want to clear data from all of the probes, do: sim.clear_probes()
If you want to clear data from specific probes, do: sim.model.params[<probe_obj>] = []
In NengoLoihi though, because of the way it stores the probe data, it looks like things are slightly different. If you look at the simulator code it seems like the clear_probes() function is just a dummy function. However, further down, there is a function called collect_probe_output(<probe_obj>) that seems to perform the probe recording and clearing behaviour you are looking for. Unfortunately, I don’t have access to the Loihi hardware to test this function out. Let me know if it works for you.
Thank you @xchoo for looking into it. I tried the sim.model.params[<probe_obj>] method (and the similar one mentioned in the link you provided) but wasn’t successful. This print(loihi_sim.model.params[probe_otp]) throws the following error:
KeyError: <Probe at 0x7f1d54d9d0a0 of 'output' of <Node (unlabeled) at 0x7f1d54d78e80>>
i.e. most likely, the sim.model.params[<probe>] doesn’t work with NengoLoihi.
Next, I tried the collect_probe_output method. Actually I had the 1.1.0.dev0 version of NengoLoihi and it didn’t have the collect_probe_output(), but get_probe_output() and this didn’t work for some reason. Then I installed the released 1.1.0 version of NengoLoihi and tried to make it work. Looks like I need to define the HardwareInterface() simulator as mentioned here. For this I needed a proper Model object which I got from loihi_sim.model after defining loihi_sim = nengo_loihi.Simulator(temp_net) (assuming this is correct). Next, I needed a suitable probe object for the probe_otp (in the code in main post), but I just couldn’t find a mapping. Therefore, I took my chances with the probe_0 = loihi_sim.model.probes[0] and ran the following code:
with HardwareInterface(loihi_sim.model, use_snips=False) as sim:
for _ in range(5):
sim.run_steps(10)
sim_u = sim.collect_probe_output(probe_0)
print(sim_u.shape)
It works, I get the shape of sim_u = (10, 1) for every iteration of the for loop. But upon plotting the last value of sim_u, the plot is simply a random line, not oscillating around 2.0. I guess… probe_0 is not the correct hardware mapping of the probe_otp, or anything else is wrong. Can you please help me with it?
Note that I did run the above code with use_snips=True, but it just got stuck.
I was also wondering on how to save the parameters/weights of a network (written of course in Nengo) while executing with NengoLoihi backend? Will loihi_sim.save_params(<file_name>) do the job? Haven’t tried it yet, but even if I do, I was wondering if it would be the right way to save and load the params again in NengoLoihi simulator? Please let me know this too!
I looked into the NengoLoihi code, and it seems to be quite a jumble in there (because of the difficulties with interfacing with hardware). You are correct in the observation that the Nengo “standard” sim.model.params don’t exist for probe objects. Rather, NengoLoihi seems to retain references to the Loihi probe objects rather than the Nengo probe objects. However, I think I have a solution that will work for your specific use case (specifically that you want to record the probe data, then clear the data to avoid memory build up):
with loihi_sim:
for _ in range(5):
loihi_sim.run(...)
# Save loihi_sim.data[probe_otp]
print(loihi_sim.data[probe_otp])
# Refresh/Empty probe data
loihi_sim.clear_probes()
As it turns out, there are multiple layers of simulator code in NengoLoihi, and the “main” level clear_probes function does do the specified function.
No… unfortunately not. The save_params function is only a feature of NengoDL. In NengoLoihi, if you want to save the parameters of the network, you’ll need to do it on the Nengo model or, if you are converting a NengoDL model, you have to do it with the NengoDL model (using save_params). But, if you ware looking to get at the weights that have been changed on the hardware itself (e.g., if you are using the PES rule), I don’t believe that is possible, since we currently have no mechanism for retrieving the weights from the hardware (although, Loihi2 – the newer version of the chip from Intel – might support this)
There is a way to get at the weights that the NengoLoihi model actually uses (or rather, that is passed to the hardware), but due to the complexities of communicating with the hardware NengoLoihi inserts some extra connections (these are the used to communicate with the hardware) into the model that make it all sorts of messy. If you want some preliminary code, message @travis.dewolf, he has some prototype code working.
Hello @xchoo, I haven’t implemented your above code yet - as I am just running some preliminary experiments. I was about to run full scale experiments, but have been facing the following error on Loihi boards:
.
.
.
File "/homes/rgaurav/nxsdk_1_nengo_loihi/lib/python3.8/site-packages/nengo_loihi/simulator.py", line 349, in run
self.run_steps(steps)
File "/homes/rgaurav/nxsdk_1_nengo_loihi/lib/python3.8/site-packages/nengo_loihi/simulator.py", line 362, in run_steps
self._runner.run_steps(steps)
File "/homes/rgaurav/nxsdk_1_nengo_loihi/lib/python3.8/site-packages/nengo_loihi/simulator.py", line 592, in loihi_bidirectional_with_host
self._chip2host(self.loihi)
File "/homes/rgaurav/nxsdk_1_nengo_loihi/lib/python3.8/site-packages/nengo_loihi/simulator.py", line 430, in _chip2host
sim.chip2host(self.probes_receivers)
File "/homes/rgaurav/nxsdk_1_nengo_loihi/lib/python3.8/site-packages/nengo_loihi/hardware/interface.py", line 378, in chip2host
raw_data = self.host_snip.recv_bytes(self.bytes_per_step)
File "/homes/rgaurav/nxsdk_1_nengo_loihi/lib/python3.8/site-packages/nengo_loihi/hardware/interface.py", line 785, in recv_bytes
raise RuntimeError("Received shutdown signal from chip")
RuntimeError: Received shutdown signal from chip
when I deploy a larger network. Can you please help me fix it? Does it have something to do with the pre-computation of spikes for the inputs to the network? I will be happy to provide more info as required.
I haven’t seen that error myself, but it looks like it is referenced in this forum post. The useful information regarding this error would be the INFO outputs from the hardware (there should be a bunch of printout’s with INFO tagged to it, just before the exception is thrown).
It looks like maybe trying a different Loihi partition may solve the issue?
I was looking into the linked thread… and my problem seems similar - RuntimeError issues with larger networks. I had earlier tried executing the larger network on Poihiki and Nahuku32 boards - none helped. And yet to test out the proposed fixes there… but incurred another problem I ran into, w.r.t. Linear Delay Networks (LDN). I am explaining it next.
I have a matrix of training/test signal samples, i.e. NxM matrix => N rows of individual signals with M elements in each signal vector. Now, for the individual runs i.e. sim.run(<some time-steps>), I would want the output from the LDN to be in the context of only the one input signal (sim can be a Nengo or NengoLoihi simulator). Therefore, I tried execute the sim.run(<one signal duration> for one signal, then I executed sim.run(<another signal duration>) making sure that the input for the first and second run are two desired signals respectively.
However, it appears that some residue of the previous signal affects the LDN output of the second signal. I tried two things, introducing a sequence of zeros between two signals, but that didn’t help - as the LDN output is non-zero for a zero output, and tried setting the Nengo network’s internal state as stateless i.e. nengo.configure_settings(stateful=False); this too didn’t help and produced the following error:
AttributeError: module 'nengo' has no attribute 'configure_settings'
I would prefer to zero out the internal state of the LDN between different runs (as explained here for NengoDL). How do I do that? Please help me with it!
It’s not entirely clear to me how you are calling these two sim.run calls. Are you doing:
with sim:
sim.run(...)
...
sim.run(...)
Or are you doing:
with nengo_loihi.Simulator(...) as sim:
sim.run(...)
...
with nengo_loihi.Simulator(...) as sim:
sim.run(...)
If you are doing the former, then I’d definitely expect the LDN to hold some “memory” of the “first” run call. After all, it does have a recurrent connection with the specific purpose to hold some sort of memory.
If you are doing the latter, and it may be a bug in NengoLoihi, because the behaviour you are observing should carry over between simulator objects.
Unfortunately, the configure_settings option is only available to NengoDL, and it’s used to set the parameters of the TensorFlow backend (which is used when simulating things in NengoDL). Nengo or NengoLoihi does not have an equivalent function and all ensembles and connections are considered to be stateful at all times (that’s just how the neurons work).
If you want to zero out the internal state of the LDN between runs, there are several things you can try:
Run each run with a different simulator objects (i.e., re-create the simulator object for each run)
You could try using sim.reset() between each “run”, but I believe that also resets the simulator timestamp, so it might not actually achieve what you want it to achieve.
If you need to have everything in run in a single simulator object, then you should try applying an inhibitory connection to the populations in the LDN. Note that because the recurrent synapse do “hold” information for some time (depends on the time constant of the recurrent connection), you’ll need to both provide a zero input and provide an inhibitory input for about 3x the length of the time constant.
Thank you @xchoo for your suggestions. I am struggling with the former method of executing individual sim.runs(). The latter method by first creating individual sim objects and then running sim.run() works perfectly fine, it doesn’t maintain a history of previous inputs. Apologies for the confusion.
Looks like, I don’t have any well defined ways then. Haven’t tested it yet, but if I do sim.reset() between the runs (I am fine with time-stamps getting reset to 0), does it mean that the trained weights of the network (through PES) would also get reset? - despite being within the same nengo_loihi.Simulator(...) scope? I tried looking for an answer to it in the code, but looks like def reset()isn’t implemented for NengoLoihi… right?
If so, I suppose I am left with your third suggestion to apply inhibitory connection to the LDN neurons along with zero and inhibitory input. Thank you for the inhibitory gating source you linked; it helps. I will keep you posted here how it goes.
Hmmm… You might be right in that, sim.reset() may not work as expected in NengoLoihi. Additionally, if you were using PES learning in your network, it’s likely that sim.reset() would reset those weights as well (or rather, it would rebuild the entire network).
