NotImplementedError: Cannot convert a symbolic Tensor (TensorGraph/while/iteration_0/ElementwiseIncBuilder/meshgrid/Size:0) to a numpy array

Without any apparent changes in my code I’ve started getting the following error when trying to use the NengoDL simulator with my custom neuron class:

Traceback (most recent call last):
File “/Users/thomastiotto/miniforge3/envs/tf24/lib/python3.8/site-packages/IPython/core/interactiveshell.py”, line 3418, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File “”, line 1, in
runfile(’/Users/thomastiotto/PycharmProjects/Learning-to-approximate-functions-using-niobium-doped-strontium-titanate-memristors/experiments/mPES.py’, wdir=’/Users/thomastiotto/PycharmProjects/Learning-to-approximate-functions-using-niobium-doped-strontium-titanate-memristors/experiments’)
File “/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydev_bundle/pydev_umd.py”, line 197, in runfile
pydev_imports.execfile(filename, global_vars, local_vars) # execute the script
File “/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydev_imps/_pydev_execfile.py”, line 18, in execfile
exec(compile(contents+"\n", file, ‘exec’), glob, loc)
File “/Users/thomastiotto/PycharmProjects/Learning-to-approximate-functions-using-niobium-doped-strontium-titanate-memristors/experiments/mPES.py”, line 214, in
cm = nengo_dl.Simulator( model, seed=seed, dt=timestep, progress_bar=progress_bar, device=device )
File “/Users/thomastiotto/miniforge3/envs/tf24/lib/python3.8/site-packages/nengo_dl/simulator.py”, line 526, in init
self._build_keras(progress)
File “/Users/thomastiotto/miniforge3/envs/tf24/lib/python3.8/site-packages/nengo/utils/magic.py”, line 181, in call
return self.wrapper(self.wrapped, self.instance, args, kwargs)
File “/Users/thomastiotto/miniforge3/envs/tf24/lib/python3.8/site-packages/nengo_dl/simulator.py”, line 50, in with_self
output = wrapped(*args, **kwargs)
File “/Users/thomastiotto/miniforge3/envs/tf24/lib/python3.8/site-packages/nengo_dl/simulator.py”, line 549, in _build_keras
outputs = self.tensor_graph(inputs, stateful=self.stateful, progress=progress)
File “/Users/thomastiotto/miniforge3/envs/tf24/lib/python3.8/site-packages/tensorflow/python/keras/engine/base_layer.py”, line 946, in call
return self._functional_construction_call(inputs, args, kwargs,
File “/Users/thomastiotto/miniforge3/envs/tf24/lib/python3.8/site-packages/tensorflow/python/keras/engine/base_layer.py”, line 1085, in _functional_construction_call
outputs = self._keras_tensor_symbolic_call(
File “/Users/thomastiotto/miniforge3/envs/tf24/lib/python3.8/site-packages/tensorflow/python/keras/engine/base_layer.py”, line 817, in _keras_tensor_symbolic_call
return self._infer_output_signature(inputs, args, kwargs, input_masks)
File “/Users/thomastiotto/miniforge3/envs/tf24/lib/python3.8/site-packages/tensorflow/python/keras/engine/base_layer.py”, line 858, in _infer_output_signature
outputs = call_fn(inputs, *args, **kwargs)
File “/Users/thomastiotto/miniforge3/envs/tf24/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py”, line 620, in wrapper
return func(*args, **kwargs)
File “/Users/thomastiotto/miniforge3/envs/tf24/lib/python3.8/site-packages/nengo_dl/tensor_graph.py”, line 488, in call
self._build_loop(sub) if self.use_loop else self._build_no_loop(sub)
File “/Users/thomastiotto/miniforge3/envs/tf24/lib/python3.8/site-packages/nengo_dl/tensor_graph.py”, line 638, in _build_loop
loop_vars = tf.while_loop(
File “/Users/thomastiotto/miniforge3/envs/tf24/lib/python3.8/site-packages/tensorflow/python/util/deprecation.py”, line 605, in new_func
return func(*args, **kwargs)
File “/Users/thomastiotto/miniforge3/envs/tf24/lib/python3.8/site-packages/tensorflow/python/ops/control_flow_ops.py”, line 2489, in while_loop_v2
return while_loop(
File “/Users/thomastiotto/miniforge3/envs/tf24/lib/python3.8/site-packages/tensorflow/python/ops/control_flow_ops.py”, line 2687, in while_loop
return while_v2.while_loop(
File “/Users/thomastiotto/miniforge3/envs/tf24/lib/python3.8/site-packages/tensorflow/python/ops/while_v2.py”, line 192, in while_loop
body_graph = func_graph_module.func_graph_from_py_func(
File “/Users/thomastiotto/miniforge3/envs/tf24/lib/python3.8/site-packages/tensorflow/python/framework/func_graph.py”, line 990, in func_graph_from_py_func
func_outputs = python_func(func_args, **func_kwargs)
File “/Users/thomastiotto/miniforge3/envs/tf24/lib/python3.8/site-packages/tensorflow/python/ops/while_v2.py”, line 178, in wrapped_body
outputs = body(
_pack_sequence_as(orig_loop_vars, args))
File “/Users/thomastiotto/miniforge3/envs/tf24/lib/python3.8/site-packages/nengo_dl/tensor_graph.py”, line 615, in loop_body
loop_i = self._build_inner_loop(loop_i, update_probes, progress)
File “/Users/thomastiotto/miniforge3/envs/tf24/lib/python3.8/site-packages/nengo_dl/tensor_graph.py”, line 758, in _build_inner_loop
self.signals.scatter(
File “/Users/thomastiotto/miniforge3/envs/tf24/lib/python3.8/site-packages/nengo_dl/signals.py”, line 372, in scatter
result = tf.tensor_scatter_nd_update(var, dst.tf_indices_nd, val)
File “/Users/thomastiotto/miniforge3/envs/tf24/lib/python3.8/site-packages/nengo_dl/signals.py”, line 212, in tf_indices_nd
tf.meshgrid(
File “/Users/thomastiotto/miniforge3/envs/tf24/lib/python3.8/site-packages/tensorflow/python/util/dispatch.py”, line 201, in wrapper
return target(*args, **kwargs)
File “/Users/thomastiotto/miniforge3/envs/tf24/lib/python3.8/site-packages/tensorflow/python/ops/array_ops.py”, line 3552, in meshgrid
mult_fact = ones(shapes, output_dtype)
File “/Users/thomastiotto/miniforge3/envs/tf24/lib/python3.8/site-packages/tensorflow/python/util/dispatch.py”, line 201, in wrapper
return target(*args, **kwargs)
File “/Users/thomastiotto/miniforge3/envs/tf24/lib/python3.8/site-packages/tensorflow/python/ops/array_ops.py”, line 3120, in ones
output = _constant_if_small(one, shape, dtype, name)
File “/Users/thomastiotto/miniforge3/envs/tf24/lib/python3.8/site-packages/tensorflow/python/ops/array_ops.py”, line 2804, in _constant_if_small
if np.prod(shape) < 1000:
File “<array_function internals>”, line 5, in prod
File “/Users/thomastiotto/miniforge3/envs/tf24/lib/python3.8/site-packages/numpy/core/fromnumeric.py”, line 3030, in prod
return _wrapreduction(a, np.multiply, ‘prod’, axis, dtype, out,
File “/Users/thomastiotto/miniforge3/envs/tf24/lib/python3.8/site-packages/numpy/core/fromnumeric.py”, line 87, in _wrapreduction
return ufunc.reduce(obj, axis, dtype, out, **passkwargs)
File “/Users/thomastiotto/miniforge3/envs/tf24/lib/python3.8/site-packages/tensorflow/python/framework/ops.py”, line 852, in array
raise NotImplementedError(
NotImplementedError: Cannot convert a symbolic Tensor (TensorGraph/while/iteration_0/meshgrid/Size:0) to a numpy array. This error may indicate that you’re trying to pass a Tensor to a NumPy call, which is not supported

My code can be found here and I cannot see what the problem might be, given that it was working perfectly until the last time I tested it:

My frontend class AdaptiveLIFLateralInhibition extends LIF and everything works when using the Nengo Core simulator.
I also have a AdaptiveLIFLateralInhibitionBuilder class for the NengoDL implementation, which I register with the default SimNeuronsBuilder.

Edit: this error has also appeared for my custom mPES learning rule:

Thanks in advance!

Hi @Tioz,

Assuming your code and environment hasn’t changed, it is almost impossible to have code that was working suddenly fail to work. Typically, something has changed between the working and non-working instances. I’m no stranger to encountering this issue (I’ve done this multiple times in the past), so here are some debugging tips:

  • Double check that the code you are using hasn’t changed. If the code has been committed to a Github repository, clone a completely new instance of the repository and test that code.
  • Check that the file you are saving to is the file actually being used (done this multiple times myself).
  • Check that the environment hasn’t changed.
  • If you don’t have the original environment around, create a new environment with the required packages and roll back versions of the installed packages until your code starts working. In this case, I would try rolling back NengoDL and TensorFlow as both of those have been recently updated.

As for the specific issue with your code, I tried to take a look at your code by following the links you posted, but I am getting 404 errors. It seems like you deleted the memristor_nengo folder in a commit you made a few hours ago.

I’m absolutely sure that my code has not changed so I was wondering if updates to NengoDL might be the cause.

On a theoretical level, what could be the cause of the error:

NotImplementedError: Cannot convert a symbolic Tensor (TensorGraph/while/iteration_0/meshgrid/Size:0) to a numpy array. This error may indicate that you’re trying to pass a Tensor to a NumPy call, which is not supported

Sorry about the code! I’ve been reorganising it and moved it to a new repository. Here are the NengoDL build function and OpBuilder for my learning rule and here is the OpBuilder for my neuron, which I register here.

Thanks again!

I manage to get it to work and found the culprit!
It seems that having numpy-1.20.1 breaks things. I’m not sure if NengoDL installed this dependency but I’d double check on your side :slight_smile:
I realised this might have been a problem while installing packages manually and getting the following warning:
tensorflow 2.4.1 requires numpy~=1.19.2, but you have numpy 1.20.1 which is incompatible.

I tested, and downgrading to numpy-1.19.2 solved the issue. :smile:
My working env right now is:

nengo                  3.1.0
nengo-dl               3.4.0
numpy                  1.19.2
tensorflow             2.4.1
tensorflow-estimator   2.4.0

NengoDL’s Numpy requirement is numpy>=1.16.0. I just tested it in a new Python environment, and doing pip install nengo-dl installs numpy==1.19.5, which follows the TensorFlow requirements.