I’m trying to run Spaun in direct mode which fails because of nengo/nengo#1178. To circumvent this problem I’m using this code:
def finite(*args, **kwargs):
values = fn(*args, **kwargs)
if values is None:
elif not np.all(np.isfinite(values)):
values = np.ones_like(values)
# Some omitted setup code
model = Spaun()
for node in model.all_nodes:
node.output = make_finite(node.output)
for conn in model.all_connections:
conn.function = make_finite(conn.function)
But I’m still getting a similar error:
FloatingPointError: invalid value encountered in add
Uncaught exception. Entering post mortem debugging
Running 'cont' or 'step' will restart the program
292 def __call__(self, t, signal):
293 self.output *= -self.a
--> 294 self.output += self.b * signal
295 return self.output
Any ideas what I might be missing?
Try building Spaun with different modules enabled / disabled. You can comment out the modules in the spaun_main.py file (in the
\_spaun) folder. I’m not sure if there are any connections that use anything special that might cause your hack to fail.
This is the part you are looking for:
model.stim = Stimulus()
model.instr_stim = InstrStimulus()
model.vis = Vision()
model.ps = ProdSys()
model.reward = RewardEval()
model.enc = InfoEnc()
model.mem = Memory()
model.trfm = TrfmSys()
model.dec = InfoDec()
model.mtr = Motor()
model.instr = InstrProcess()
model.monitor = Monitor()
I think the problem is it’s not just infinite or NaN values that are troublesome, but also very large values. Very large values will cause overflows which turn to
inf and which cause the same problems as you were seeing before you introduced your code. So I would actually
np.clip values to be within a “reasonable” range (whatever reasonable is, I don’t know).
Clipping values the range of 1. to 1e5 does not help. (And that seems to be a reasonable range to me: It avoids division by small numbers and really large numbers.)
RewardEval is the first failing module (there might be more).
Ooofh. I can see that happening. Try commenting out the RewardEval line (
model.reward = RewardEval()) and rebuilding the model. The code should be configured to handle that gracefully.
Do you have any idea what in the RewardEval is causing the problem?
TrfmSys module is also causing problems. Spaun seems to run in direct mode if I comment both those lines, but not if I leave either one in.