Not enough memory when I need many data that be probed

General Discussion
1

The question is that
I need many data(weight matrix)，
I use weight_probe=nengo.Probe(‘EtoE’,‘weights’,sample_every=dt) to get the data

dt=1ms, Tmax(the total simulation time)=10000s,and weight matrix’s shape is 3000*3000

so I need about 3000×3000×10000/0.001=9×10^13 data ,but my memory is not enough
so can nengo support delete the data in Probe object dynamically during simulation.

what should I do?

2

Instead of using a nengo.Probe() object, use a nengo.Node() which takes weights as input and intermittently saves the collected weight data to a file. I’ll try to figure out the code to complete this operation and post it.

3

You can also just run your simulation inside a for loop, i.e., instead of

sim.run(10000)

do

for _ in range(10):
    sim.run(1000)
    save_to_file_function(sim.data[probe])
    sim.model.params[probe] = [] # clear probe data from memory
1 Like
4

I do this ,but not clear my memory

5

That means you have some other reference to the probe data saved somewhere. Here’s a script you can use to verify that the memory will be cleared:

import nengo
import numpy as np

with nengo.Network() as net:
    a = nengo.Ensemble(3000, 1)
    b = nengo.Ensemble(3000, 1)
    c = nengo.Connection(a.neurons, b.neurons,
                         transform=np.random.randn(3000, 3000))
    p = nengo.Probe(c, "weights")

with nengo.Simulator(net) as sim:
    for i in range(100):
        print(i)
        sim.run_steps(100)
        print(sim.data[p].shape)
        sim.model.params[p] = []
        sim.data.reset()

Edit: added sim.data.reset() as Jan notes below

6

@Seanny123 How would you provide the weights as input to the node?

Also @drasmuss’s solution has some pitfalls and will save the data from the first simulation run in all steps. This is happens because the access via sim.data is internally cached and it is assumed that the amount of probe data in sim.model.params[probe] always grows. However, the size of the probed data will the same in each loop and sim.data will always return the result cached in the first loop. Two things could be done, to actually save the desired data: either always access it via sim.model.params[p] (this will give you a list instead of a NumPy array; I also think that the sim.model.params access is considered an implementation detail and might change in future versions), or clear the sim.data cache with sim.data.reset() in each loop.

@ZhangPeng I assume setting sample_every to a higher value to record less samples overall is not an option?

Some related things from GitHub:

7

I actually didn’t know how to provide the weights to the input node. I just assumed if nengo.Probe() did it, it should be possible with nengo.Node(), but didn’t get around to checking yet…

8

I run the code you support,error emerged,
AttributeError: ‘ProbeDict’ object has no attribute ‘reset’

9

Try updating your nengo installation (pip install nengo --upgrade)