The question is that
I need many data(weight matrix)，
I use weight_probe=nengo.Probe(‘EtoE’,‘weights’,sample_every=dt) to get the data
dt=1ms, Tmax(the total simulation time)=10000s,and weight matrix’s shape is 3000*3000
so I need about 3000×3000×10000/0.001=9×10^13 data ,but my memory is not enough
so can nengo support delete the data in Probe object dynamically during simulation.
what should I do?
Instead of using a
nengo.Probe() object, use a
nengo.Node() which takes weights as input and intermittently saves the collected weight data to a file. I’ll try to figure out the code to complete this operation and post it.
You can also just run your simulation inside a for loop, i.e., instead of
for _ in range(10):
sim.model.params[probe] =  # clear probe data from memory
I do this ,but not clear my memory
That means you have some other reference to the probe data saved somewhere. Here’s a script you can use to verify that the memory will be cleared:
import numpy as np
with nengo.Network() as net:
a = nengo.Ensemble(3000, 1)
b = nengo.Ensemble(3000, 1)
c = nengo.Connection(a.neurons, b.neurons,
p = nengo.Probe(c, "weights")
with nengo.Simulator(net) as sim:
for i in range(100):
sim.model.params[p] = 
sim.data.reset() as Jan notes below
@Seanny123 How would you provide the weights as input to the node?
Also @drasmuss’s solution has some pitfalls and will save the data from the first simulation run in all steps. This is happens because the access via
sim.data is internally cached and it is assumed that the amount of probe data in
sim.model.params[probe] always grows. However, the size of the probed data will the same in each loop and
sim.data will always return the result cached in the first loop. Two things could be done, to actually save the desired data: either always access it via
sim.model.params[p] (this will give you a list instead of a NumPy array; I also think that the
sim.model.params access is considered an implementation detail and might change in future versions), or clear the
sim.data cache with
sim.data.reset() in each loop.
@ZhangPeng I assume setting sample_every to a higher value to record less samples overall is not an option?
Some related things from GitHub:
I actually didn’t know how to provide the weights to the input node. I just assumed if
nengo.Probe() did it, it should be possible with
nengo.Node(), but didn’t get around to checking yet…
I run the code you support,error emerged,
AttributeError: ‘ProbeDict’ object has no attribute ‘reset’
Try updating your nengo installation (
pip install nengo --upgrade)