There is a significant amount of I/O that happens when I run code on the INRC superhost, which I suspect is seriously slowing down training time by over a factor of 10 (as compared to using regular Nengo on my laptop).
To run my code, I’m using SLURM=1 PARTITION=loihi_2h python -m name_of_python_file. Is there some option in that command I can use to reduce the verbosity?
Also, how can I view the energy usage and processing time of my network?
Hi @kshivvy, and welcome back to the Nengo forums. 
Regarding the I/O (or the messages that appear on the screen), those are part of the NxSDK debug messages, and Nengo does not have any control over their appearance. I believe you can control the output by setting the logging level of NxSDK. If you look in the NxSDK documentation: “Logging in NXNet” or the docs/nxlogging.html file, it seems that this will do it:
from nxsdk.logutils.nxlogging import set_verbosity, LoggingLevel
set_verbosity(LoggingLevel.ERROR)
As for the slow simulation speed, I would like to clarify how you are running your simulation. Are you using the sim.run() function or the sim.step() function, or perhaps something else entirely?
Thanks for that code snippet! I’ll familiarize myself more with the NxSDK documentation.
I’m calling sim.run() repeatedly for training. Does that cause the model to have to be reloaded and reinitialized onto the board every time?
For NengoLoihi, yes, it does. If you can provide a snippet of your code (or explain how you perform the training process), perhaps I can suggest an alternative approach.
In my training, I call sim.run(0.01) in a for loop and analyze the probed output data each time. However, each sim.run(0.01) call increases the size of the probed output, and each call takes longer and longer to run. My suspicion is that the Loihi chip is sending data that I’ve already recieved to the CPU each time I check the probes in the chip2host() function.
This issue: https://github.com/nengo/nengo/issues/963 seems to have solved this problem for Nengo, but none of the code snippets in that issue solve this with Nengo Loihi.
Is there a workaround?
I don’t think that either NxSDK or NengoLoihi are re-sending data each time you call run, but you’re right that the size of the probed output will keep increasing, because it’s stored across run calls.
In Clear probes by hunse · Pull Request #303 · nengo/nengo-loihi · GitHub we implement clear_probes for NengoLoihi, but it’s still in development so you’ll have to use that branch (it’s tested, though, so it should work well). That’s what you’ll need to use to clear probed data.
It doesn’t cause it to be reloaded, but there is some processing that NxSDK does each time the simulation is started or stopped. (They’re all listed in that output you posted, things like transferring spikes to the chip, and processing the probe “timeseries” data that comes off the chip. I’m not sure what all exactly is contained in “Configuring registers”, or why it needs to be done each run call.)
If you’re able to use fewer run calls, you’ll get better performance.