Batch_normalization error using NENGO_DL


#1

Hi, thx for the great work with Nengo & NengoDL

I tried to reproduce the code available @ NengoDL page : https://www.nengo.ai/nengo-dl/examples/spiking-mnist.html

i decided to add another layer based on Batch normalization (in order to build blocks: BNRELuCONV)
when i add the layer, the graph get built:

with nengo.Network() as net:
# set some default parameters for the neurons that will make
# the training progress more smoothly
net.config[nengo.Ensemble].max_rates = nengo.dists.Choice([100])
net.config[nengo.Ensemble].intercepts = nengo.dists.Choice([0])
neuron_type = nengo.LIF(amplitude=0.01)

# we'll make all the nengo objects in the network
# non-trainable. we could train them if we wanted, but they don't
# add any representational power. note that this doesn't affect
# the internal components of tensornodes, which will always be
# trainable or non-trainable depending on the code written in
# the tensornode.
nengo_dl.configure_settings(trainable=False)

# the input node that will be used to feed in input images
inp = nengo.Node([0] * 28 * 28)


# add the first convolutional layer
x = nengo_dl.tensor_layer(
    inp, tf.layers.conv2d, shape_in=(28, 28, 1), filters=32,
    kernel_size=3)

x = nengo_dl.tensor_layer(
        x, tf.layers.batch_normalization)


# apply the neural nonlinearity
x = nengo_dl.tensor_layer(x, neuron_type)

# add another convolutional layer
x = nengo_dl.tensor_layer(
    x, tf.layers.conv2d, shape_in=(26, 26, 32),
    filters=64, kernel_size=3)
x = nengo_dl.tensor_layer(x, neuron_type)

# add a pooling layer
x = nengo_dl.tensor_layer(
    x, tf.layers.average_pooling2d, shape_in=(24, 24, 64),
    pool_size=2, strides=2)

# another convolutional layer
x = nengo_dl.tensor_layer(
        x, tf.layers.conv2d, shape_in=(12, 12, 64),
        filters=128, kernel_size=3)
x = nengo_dl.tensor_layer(x, neuron_type)

# another pooling layer
x = nengo_dl.tensor_layer(
    x, tf.layers.average_pooling2d, shape_in=(10, 10, 128),
    pool_size=2, strides=2)

# linear readout
x = nengo_dl.tensor_layer(x, tf.layers.dense, units=10)

# we'll create two different output probes, one with a filter
# (for when we're simulating the network over time and
# accumulating spikes), and one without (for when we're
# training the network using a rate-based approximation)
out_p = nengo.Probe(x)
out_p_filt = nengo.Probe(x, synapse=0.1)

BUT then when i follow the instructions:
I see this error message (when training):
(sorry it’s rather long)

NotFoundError: Restoring from checkpoint failed. This is most likely due to a Variable name or other graph key that is missing from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

Key batch_normalization/beta not found in checkpoint
[[node save/RestoreV2 (defined at /Users/assansanogo/Desktop/PYTHON_ANACONDA/anaconda2/envs/neuromorphics/lib/python3.5/site-packages/nengo_dl/simulator.py:996) ]]

Caused by op β€˜save/RestoreV2’, defined at:
File β€œ/Users/assansanogo/Desktop/PYTHON_ANACONDA/anaconda2/envs/neuromorphics/lib/python3.5/runpy.py”, line 193, in _run_module_as_main
β€œmain”, mod_spec)
File β€œ/Users/assansanogo/Desktop/PYTHON_ANACONDA/anaconda2/envs/neuromorphics/lib/python3.5/runpy.py”, line 85, in _run_code
exec(code, run_globals)
File β€œ/Users/assansanogo/Desktop/PYTHON_ANACONDA/anaconda2/envs/neuromorphics/lib/python3.5/site-packages/ipykernel_launcher.py”, line 16, in
app.launch_new_instance()
File β€œ/Users/assansanogo/Desktop/PYTHON_ANACONDA/anaconda2/envs/neuromorphics/lib/python3.5/site-packages/traitlets/config/application.py”, line 658, in launch_instance
app.start()
File β€œ/Users/assansanogo/Desktop/PYTHON_ANACONDA/anaconda2/envs/neuromorphics/lib/python3.5/site-packages/ipykernel/kernelapp.py”, line 505, in start
self.io_loop.start()
File β€œ/Users/assansanogo/Desktop/PYTHON_ANACONDA/anaconda2/envs/neuromorphics/lib/python3.5/site-packages/tornado/platform/asyncio.py”, line 132, in start
self.asyncio_loop.run_forever()
File β€œ/Users/assansanogo/Desktop/PYTHON_ANACONDA/anaconda2/envs/neuromorphics/lib/python3.5/asyncio/base_events.py”, line 421, in run_forever
self._run_once()
File β€œ/Users/assansanogo/Desktop/PYTHON_ANACONDA/anaconda2/envs/neuromorphics/lib/python3.5/asyncio/base_events.py”, line 1425, in _run_once
handle._run()
File β€œ/Users/assansanogo/Desktop/PYTHON_ANACONDA/anaconda2/envs/neuromorphics/lib/python3.5/asyncio/events.py”, line 127, in _run
self._callback(*self._args)
File β€œ/Users/assansanogo/Desktop/PYTHON_ANACONDA/anaconda2/envs/neuromorphics/lib/python3.5/site-packages/tornado/ioloop.py”, line 758, in _run_callback
ret = callback()
File β€œ/Users/assansanogo/Desktop/PYTHON_ANACONDA/anaconda2/envs/neuromorphics/lib/python3.5/site-packages/tornado/stack_context.py”, line 300, in null_wrapper
return fn(*args, **kwargs)
File β€œ/Users/assansanogo/Desktop/PYTHON_ANACONDA/anaconda2/envs/neuromorphics/lib/python3.5/site-packages/tornado/gen.py”, line 1233, in inner
self.run()
File β€œ/Users/assansanogo/Desktop/PYTHON_ANACONDA/anaconda2/envs/neuromorphics/lib/python3.5/site-packages/tornado/gen.py”, line 1147, in run
yielded = self.gen.send(value)
File β€œ/Users/assansanogo/Desktop/PYTHON_ANACONDA/anaconda2/envs/neuromorphics/lib/python3.5/site-packages/ipykernel/kernelbase.py”, line 357, in process_one
yield gen.maybe_future(dispatch(*args))
File β€œ/Users/assansanogo/Desktop/PYTHON_ANACONDA/anaconda2/envs/neuromorphics/lib/python3.5/site-packages/tornado/gen.py”, line 326, in wrapper
yielded = next(result)
File β€œ/Users/assansanogo/Desktop/PYTHON_ANACONDA/anaconda2/envs/neuromorphics/lib/python3.5/site-packages/ipykernel/kernelbase.py”, line 267, in dispatch_shell
yield gen.maybe_future(handler(stream, idents, msg))
File β€œ/Users/assansanogo/Desktop/PYTHON_ANACONDA/anaconda2/envs/neuromorphics/lib/python3.5/site-packages/tornado/gen.py”, line 326, in wrapper
yielded = next(result)
File β€œ/Users/assansanogo/Desktop/PYTHON_ANACONDA/anaconda2/envs/neuromorphics/lib/python3.5/site-packages/ipykernel/kernelbase.py”, line 534, in execute_request
user_expressions, allow_stdin,
File β€œ/Users/assansanogo/Desktop/PYTHON_ANACONDA/anaconda2/envs/neuromorphics/lib/python3.5/site-packages/tornado/gen.py”, line 326, in wrapper
yielded = next(result)
File β€œ/Users/assansanogo/Desktop/PYTHON_ANACONDA/anaconda2/envs/neuromorphics/lib/python3.5/site-packages/ipykernel/ipkernel.py”, line 294, in do_execute
res = shell.run_cell(code, store_history=store_history, silent=silent)
File β€œ/Users/assansanogo/Desktop/PYTHON_ANACONDA/anaconda2/envs/neuromorphics/lib/python3.5/site-packages/ipykernel/zmqshell.py”, line 536, in run_cell
return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
File β€œ/Users/assansanogo/Desktop/PYTHON_ANACONDA/anaconda2/envs/neuromorphics/lib/python3.5/site-packages/IPython/core/interactiveshell.py”, line 2843, in run_cell
raw_cell, store_history, silent, shell_futures)
File β€œ/Users/assansanogo/Desktop/PYTHON_ANACONDA/anaconda2/envs/neuromorphics/lib/python3.5/site-packages/IPython/core/interactiveshell.py”, line 2869, in _run_cell
return runner(coro)
File β€œ/Users/assansanogo/Desktop/PYTHON_ANACONDA/anaconda2/envs/neuromorphics/lib/python3.5/site-packages/IPython/core/async_helpers.py”, line 67, in _pseudo_sync_runner
coro.send(None)
File β€œ/Users/assansanogo/Desktop/PYTHON_ANACONDA/anaconda2/envs/neuromorphics/lib/python3.5/site-packages/IPython/core/interactiveshell.py”, line 3044, in run_cell_async
interactivity=interactivity, compiler=compiler, result=result)
File β€œ/Users/assansanogo/Desktop/PYTHON_ANACONDA/anaconda2/envs/neuromorphics/lib/python3.5/site-packages/IPython/core/interactiveshell.py”, line 3209, in run_ast_nodes
if (yield from self.run_code(code, result)):
File β€œ/Users/assansanogo/Desktop/PYTHON_ANACONDA/anaconda2/envs/neuromorphics/lib/python3.5/site-packages/IPython/core/interactiveshell.py”, line 3291, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File β€œβ€, line 17, in
sim.load_params("./mnist_params")
File β€œ/Users/assansanogo/Desktop/PYTHON_ANACONDA/anaconda2/envs/neuromorphics/lib/python3.5/site-packages/nengo_dl/simulator.py”, line 996, in load_params
tf.train.Saver(vars).restore(self.sess, path)
File β€œ/Users/assansanogo/Desktop/PYTHON_ANACONDA/anaconda2/envs/neuromorphics/lib/python3.5/site-packages/tensorflow/python/training/saver.py”, line 832, in init
self.build()
File β€œ/Users/assansanogo/Desktop/PYTHON_ANACONDA/anaconda2/envs/neuromorphics/lib/python3.5/site-packages/tensorflow/python/training/saver.py”, line 844, in build
self._build(self._filename, build_save=True, build_restore=True)
File β€œ/Users/assansanogo/Desktop/PYTHON_ANACONDA/anaconda2/envs/neuromorphics/lib/python3.5/site-packages/tensorflow/python/training/saver.py”, line 881, in _build
build_save=build_save, build_restore=build_restore)
File β€œ/Users/assansanogo/Desktop/PYTHON_ANACONDA/anaconda2/envs/neuromorphics/lib/python3.5/site-packages/tensorflow/python/training/saver.py”, line 513, in _build_internal
restore_sequentially, reshape)
File β€œ/Users/assansanogo/Desktop/PYTHON_ANACONDA/anaconda2/envs/neuromorphics/lib/python3.5/site-packages/tensorflow/python/training/saver.py”, line 332, in _AddRestoreOps
restore_sequentially)
File β€œ/Users/assansanogo/Desktop/PYTHON_ANACONDA/anaconda2/envs/neuromorphics/lib/python3.5/site-packages/tensorflow/python/training/saver.py”, line 580, in bulk_restore
return io_ops.restore_v2(filename_tensor, names, slices, dtypes)
File β€œ/Users/assansanogo/Desktop/PYTHON_ANACONDA/anaconda2/envs/neuromorphics/lib/python3.5/site-packages/tensorflow/python/ops/gen_io_ops.py”, line 1572, in restore_v2
name=name)
File β€œ/Users/assansanogo/Desktop/PYTHON_ANACONDA/anaconda2/envs/neuromorphics/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py”, line 788, in _apply_op_helper
op_def=op_def)
File β€œ/Users/assansanogo/Desktop/PYTHON_ANACONDA/anaconda2/envs/neuromorphics/lib/python3.5/site-packages/tensorflow/python/util/deprecation.py”, line 507, in new_func
return func(*args, **kwargs)
File β€œ/Users/assansanogo/Desktop/PYTHON_ANACONDA/anaconda2/envs/neuromorphics/lib/python3.5/site-packages/tensorflow/python/framework/ops.py”, line 3300, in create_op
op_def=op_def)
File β€œ/Users/assansanogo/Desktop/PYTHON_ANACONDA/anaconda2/envs/neuromorphics/lib/python3.5/site-packages/tensorflow/python/framework/ops.py”, line 1801, in init
self._traceback = tf_stack.extract_stack()

NotFoundError (see above for traceback): Restoring from checkpoint failed. This is most likely due to a Variable name or other graph key that is missing from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

Key batch_normalization/beta not found in checkpoint
[[node save/RestoreV2 (defined at /Users/assansanogo/Desktop/PYTHON_ANACONDA/anaconda2/envs/neuromorphics/lib/python3.5/site-packages/nengo_dl/simulator.py:996) ]]

What am I doing wrong? thx for your input :slight_smile:


#2

When you add a batch normalization layer, you are adding new parameters to the network. So you won’t be able to use the same stored parameter file from before. That is why you are getting the error

NotFoundError (see above for traceback): Restoring from checkpoint failed. This is most likely due to a Variable name or other graph key that is missing from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

Key batch_normalization/beta not found in checkpoint

The batch normalization parameter is not in the stored parameters (checkpoint) you are trying to load. You will need to retrain the network with the new parameters (e.g., set do_training=True in the mnist example script). After that you can save those parameters and then you will be able to load them in the future as normal.


#3

OH! I skipped that :smiley:
Perfect. just tried it.
Thx for the fast & precise answer.