Need to download weights of fully connected network

HI,
I am trying to execute this code with few changes
https://www.nengo.ai/nengo-dl/examples/spiking-mnist.html.
the one above uses convolution layers. So I changed my code little to use fully connected. I am doing this
x = nengo_dl.tensor_layer(
x, tf.layers.conv2d, shape_in=(12, 12, 64),
filters=128, kernel_size=3)
x = nengo_dl.tensor_layer(x, neuron_type)

replacing this with
x = nengo_dl.tensor_layer(
inp, tf.layers.dense, units=256).

After training I am getting good results. However, I am not able to download the weight matrix. I have 5 layers 784 - 256 - 64 - 16 -10. and need 784x256, 256x64, 64x16, 16x10 matrices in a txt file. So that I can read them into my c++ code.

Could someone please help me in downloading the following matrices.

Since you are using tf.layers.dense, the weights are being managed internally by TensorFlow. So there isn’t an easy way to access them using Nengo syntax, you would have to use one of the TensorFlow methods.

However, because you are using a dense transform, there isn’t actually any need to use tf.layers. You could use the equivalent

x = nengo.Ensemble(256, 1, neuron_type=neuron_type)
conn = nengo.Connection(inp, x.neurons, transform=nengo_dl.dists.Glorot(), 
                        synapse=None)

and then you’d be able to access the trained weights via

with nengo_dl.Simulator(...)
    sim.train(...)
    my_weights = sim.data[conn].weights

Hi,

Thank you very much for the reply. I tried using this code. However, getting error everytime I increase the number of epochs in training.
epoch = 10, error = 10%
epoch 50, error = 30%
epoch 100, error 27%.
Please find the code I had been using
with nengo.Network() as net:
# set some default parameters for the neurons that will make
# the training progress more smoothly
net.config[nengo.Ensemble].max_rates = nengo.dists.Choice([100])
net.config[nengo.Ensemble].intercepts = nengo.dists.Choice([0])
neuron_type = nengo.LIF(amplitude=0.01)

nengo_dl.configure_settings(trainable=True)
# the input node that will be used to feed in input images
inp = nengo.Node(np.zeros(784))
print(inp.size_out)
temp = nengo.Probe(inp)
# add the first layer
x, conn = nengo_dl.tensor_layer(inp, neuron_type, transform=nengo_dl.dists.Glorot(),shape_in=(256,), return_conn=True)

x, conn2 = nengo_dl.tensor_layer(x, neuron_type, transform=nengo_dl.dists.Glorot(),shape_in=(64,), return_conn=True)

x, conn3 = nengo_dl.tensor_layer(x, neuron_type, transform=nengo_dl.dists.Glorot(),shape_in=(16,), return_conn=True)

x, conn4 = nengo_dl.tensor_layer(x, neuron_type, transform=nengo_dl.dists.Glorot(),shape_in=(10,), return_conn=True)

print(x.probeable)
out_p = nengo.Probe(x)
out_p_filt = nengo.Probe(x, synapse=0.1)

Do you feel if there is any issue over here or does it look fine

So just wanted to confirm if my layers are connected properly from input to output, and are they probed at the right point.

Everything there looks fine to me. Is the error increasing on the training data, or on the test data? If it’s the test data, that probably just means you’re overfitting (which wouldn’t be surprising, as you have a lot of parameters when you’re using dense layers like that).

Its on test data, Also I don’t see these error rate when using tf.layers.dense code. But when I use this code the error increases and changes for every iteration, for the same architecture. So wanted to confirm if the code I used is properly connected.

Also could you please let us know if there is any way to improve my performance.

I tried to implement the network using ensembles and connections but the error during training is not changing at all. Do you know why is that happening?

It’s not surprising that you would be having a problem with overfitting, that is a common problem when training neural networks. You can read more about it in lots of places, e.g. this article.

However, it is surprising if you are getting different results using tf.layers.dense versus nengo.Connection, as those two are generally equivalent. If you can post your two models that show different behaviour I can see if I can spot any differences in how you’ve set things up.

You can post code directly on the forum by enclosing it within triple backticks (```). See here for a demonstration. That will probably be easier than posting screenshots, and parts won’t be cut off.

That being said, I don’t see anything obviously wrong in your screenshot. But at a guess, do you have nengo_dl.configure_settings(trainable=False) somewhere else in your code?

1 Like

Thanks for the response! No, its not set to false anywhere! I don’t know why training is not being done at all. This is the code I am using for training.

minibatch_size = 200
sim = nengo_dl.Simulator(net, minibatch_size=minibatch_size)
train_data = {inp: train_data[0][:, None, :],
              out_p: train_data[1][:, None, :]}

# when testing our network with spiking neurons we will need to run it
# over time, so we repeat the input/target data for a number of
# timesteps. we're also going to reduce the number of test images, just
# to speed up this example.
n_steps = 30
test_data = {
    inp: np.tile(test_data[0][:minibatch_size*2, None, :],
                 (1, n_steps, 1)),
    out_p_filt: np.tile(test_data[1][:minibatch_size*2, None, :],
                        (1, n_steps, 1))}

def objective(outputs, targets):
    return tf.nn.softmax_cross_entropy_with_logits_v2(
        logits=outputs, labels=targets)

opt = tf.train.RMSPropOptimizer(learning_rate=0.001)

def classification_error(outputs, targets):
    return 100 * tf.reduce_mean(
        tf.cast(tf.not_equal(tf.argmax(outputs[:, -1], axis=-1),
                             tf.argmax(targets[:, -1], axis=-1)),
                tf.float32))

print("error before training: %.2f%%" % sim.loss(
    test_data, {out_p_filt: classification_error}))

do_training = True
#summ_list = []
if do_training:
    # run training
    sim.train(train_data, opt, objective={out_p: objective},n_epochs=10)
    #print(sim.probeable)
    # save the parameters to file
    sim.save_params("./mnist_params")
#else:
    # download pretrained weights
    #urlretrieve(
      #  "https://drive.google.com/uc?export=download&id=1u9JyNuRxQDUcFgkRnI1qfJVFMdnGRsjI",
       # "mnist_params.zip")
    #with zipfile.ZipFile("mnist_params.zip") as f:
        #f.extractall()

    # load parameters
    #sim.load_params("./mnist_params")

print("error after training: %.2f%%" % sim.loss(
    test_data, {out_p_filt: classification_error}))

Training works perfectly fine and it gives 3% error after training if I use the following code for network. But the only problem is that I can’t get the weight matrix for the last layer as it uses tf.layers.dense

with nengo.Network() as net:
    # set some default parameters for the neurons that will make
    # the training progress more smoothly
    #ipdb.set_trace()
    net.config[nengo.Ensemble].max_rates = nengo.dists.Choice([100])
    net.config[nengo.Ensemble].intercepts = nengo.dists.Choice([0])
    neuron_type = nengo.LIF(amplitude=0.01)

    nengo_dl.configure_settings(trainable=True)
    # the input node that will be used to feed in input images
    inp = nengo.Node(np.zeros(784))
    print(inp.size_out)
    temp = nengo.Probe(inp)
    # add the first layer
    x, conn0 = nengo_dl.tensor_layer(inp, neuron_type, transform=nengo_dl.dists.Glorot(),shape_in=(784,), return_conn=True)
    
    x, conn1 = nengo_dl.tensor_layer(x, neuron_type, transform=nengo_dl.dists.Glorot(),shape_in=(256,), return_conn=True)
    
    x, conn2 = nengo_dl.tensor_layer(x, neuron_type, transform=nengo_dl.dists.Glorot(),shape_in=(64,), return_conn=True)
    print(conn2.size_out)
    x, conn3 = nengo_dl.tensor_layer(x, neuron_type, transform=nengo_dl.dists.Glorot(),shape_in=(16,), return_conn=True)
    
    x, conn4 = x,conn_l = nengo_dl.tensor_layer(x, tf.layers.dense, units=10, return_conn=True)
    
    #x, conn5 = nengo_dl.tensor_layer(x,neuron_type,transform=nengo_dl.dists.Glorot(), shape_in=(10, ), return_conn=True)
    
    
    
    
    # we'll create two different output probes, one with a filter
    # (for when we're simulating the network over time and
    # accumulating spikes), and one without (for when we're
    # training the network using a rate-based approximation)
    print(x.probeable)
    out_p = nengo.Probe(x)
    out_p_filt = nengo.Probe(x, synapse=0.1)

Can you also post the code that is not working properly? Then I can try it out on my end and see if I see the same thing.

Another guess, are you setting the default synapse to None? like net.config[nengo.Connection].synapse = None. If you don’t change that, the default synapse will be applied on all the conn1/conn2/... connections, which won’t work well when training the network for a single timestep (tensor_layer automatically changes the default synapse).

Hi,

Please find the code pasted below. We have run the code in jupyter notebook, I have divided the code using “********” for blocks that have been run in different cells. As mentioned the final error after training is not consistent. Please let us know where the issue is.
note: changed the comment to // as using # is causing format issues.

%matplotlib inline

import gzip
import pickle
from urllib.request import urlretrieve
import zipfile

import nengo
import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt
import nengo_dl


urlretrieve(“http://deeplearning.net/data/mnist/mnist.pkl.gz”,
“mnist.pkl.gz”)
with gzip.open(“mnist.pkl.gz”) as f:
train_data, _, test_data = pickle.load(f, encoding=“latin1”)
train_data = list(train_data)
test_data = list(test_data)
print(len(test_data[1]))
for data in (train_data, test_data):
one_hot = np.zeros((data[0].shape[0], 10))
one_hot[np.arange(data[0].shape[0]), data[1]] = 1
data[1] = one_hot


n_in = 784
n_hidden = 64
minibatch_size = 50
with nengo.Network() as auto_net:
# input
nengo_a = nengo.Node(np.zeros(n_in))
print(nengo_a.size_out)

# first layer
nengo_b = nengo.Ensemble(
    n_hidden, 1, neuron_type=nengo.RectifiedLinear())
print(nengo_b.size_in)
nengo.Connection(
    nengo_a, nengo_b.neurons, transform=nengo_dl.dists.Glorot())

# second layer
nengo_c = nengo.Ensemble(
    n_in, 1, neuron_type=nengo.RectifiedLinear())
nengo.Connection(
    nengo_b.neurons, nengo_c.neurons,
    transform=nengo_dl.dists.Glorot())

# probes are used to collect data from the network
p_c = nengo.Probe(nengo_c.neurons)

with nengo.Network() as net:
# set some default parameters for the neurons that will make
# the training progress more smoothly

net.config[nengo.Ensemble].max_rates = nengo.dists.Choice([100])
net.config[nengo.Ensemble].intercepts = nengo.dists.Choice([0])
neuron_type = nengo.LIF(amplitude=0.01)

nengo_dl.configure_settings(trainable=True)
# the input node that will be used to feed in input images
inp = nengo.Node(np.zeros(784))
print(inp.size_out)
temp = nengo.Probe(inp)
# add the first layer
x, conn1 = nengo_dl.tensor_layer(inp, neuron_type, transform=nengo_dl.dists.Glorot(),shape_in=(256,), return_conn=True)

x, conn2 = nengo_dl.tensor_layer(x, neuron_type, transform=nengo_dl.dists.Glorot(),shape_in=(64,), return_conn=True)
print(conn2.size_out)
x, conn3 = nengo_dl.tensor_layer(x, neuron_type, transform=nengo_dl.dists.Glorot(),shape_in=(16,), return_conn=True)

x, conn4 = nengo_dl.tensor_layer(x, neuron_type, transform=nengo_dl.dists.Glorot(),shape_in=(10,), return_conn=True)

#x, conn5 = nengo_dl.tensor_layer(x,neuron_type,transform=nengo_dl.dists.Glorot(), shape_in=(10, ), return_conn=True)

print(x.probeable)
out_p = nengo.Probe(x)
out_p_filt = nengo.Probe(x, synapse=0.1)

minibatch_size = 200
sim = nengo_dl.Simulator(net, minibatch_size=minibatch_size)


// add the single timestep to the training data
train_data = {inp: train_data[0][:, None, :],
out_p: train_data[1][:, None, :]}

// when testing our network with spiking neurons we will need to run it
// over time, so we repeat the input/target data for a number of
// timesteps. we’re also going to reduce the number of test images, just
// to speed up this example.
n_steps = 30
test_data = {
inp: np.tile(test_data[0][:minibatch_size2, None, :],
(1, n_steps, 1)),
out_p_filt: np.tile(test_data[1][:minibatch_size
2, None, :],
(1, n_steps, 1))}


def objective(outputs, targets):
return tf.nn.softmax_cross_entropy_with_logits_v2(
logits=outputs, labels=targets)


opt = tf.train.RMSPropOptimizer(learning_rate=0.001)


def classification_error(outputs, targets):
return 100 * tf.reduce_mean(
tf.cast(tf.not_equal(tf.argmax(outputs[:, -1], axis=-1),
tf.argmax(targets[:, -1], axis=-1)),
tf.float32))

print(“error before training: %.2f%%” % sim.loss(
test_data, {out_p_filt: classification_error}))


do_training = True
#summ_list = []
if do_training:
# run training
sim.train(train_data, opt, objective={out_p: objective},n_epochs=1)
sim.save_params("./mnist_params")


print(“error after training: %.2f%%” % sim.loss(
test_data, {out_p_filt: classification_error}))

Hello navtej,

I’m still not sure exactly what the issue is that you’re running into. The “final error after training is not consistent” isn’t necessarily surprising, we expect some random fluctuations in the training results depending on the random initialization. You might find it helpful to frame your issue like

  1. What is the output you are observing?
  2. What output did you expect to see?
  3. How does the output you are observing differ from the output you expected to see?

Also see my post here Need to download weights of fully connected network. What would be surprising is if you are getting significantly different results when using tf.layers versus nengo.Connection (which I believe you mentioned at some point). But in order to help with that I need to know

  1. What is the code that you are running with tf.layers?
  2. What is the output of that code (is it what you expect? If not, what did you expect? How does the output differ from what you expect?)
  3. What is the code that you are running with nengo.Connection?
  4. What is the output of that code (is it what you expect? If not, what did you expect? How does the output differ from what you expect?)

As a sidenote, you can use jupyter nbconvert to automatically convert notebooks to normal python files, which might make it easier for you to post your code rather than having to manually format it.

Hi,
For now I am pasting the code below. So in my previous comment I posted a code with the changes mentioned by you so that I could download the weights using nengo lib.
Here are few of my issues:

  1. So when I use the code pasted below I get an error of 3%.
  2. However, when I implement the changes mentioned by you(ie the code pasted in the previous comment), I get an error of around 30 %.(+/- 5%).
  3. When I am tring to implement the same architecture I am getting 10 times variation in the results of both the codes. Could you please let me know if there are any issues that I need to be aware of. Why are the results of both the implementations not similar?

%matplotlib inline

import gzip
import pickle
from urllib.request import urlretrieve
import zipfile

import nengo
import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt

import nengo_dl


urlretrieve(“http://deeplearning.net/data/mnist/mnist.pkl.gz”,
“mnist.pkl.gz”)
with gzip.open(“mnist.pkl.gz”) as f:
train_data, _, test_data = pickle.load(f, encoding=“latin1”)
train_data = list(train_data)
test_data = list(test_data)
for data in (train_data, test_data):
one_hot = np.zeros((data[0].shape[0], 10))
one_hot[np.arange(data[0].shape[0]), data[1]] = 1
data[1] = one_hot


with nengo.Network() as net:
# set some default parameters for the neurons that will make
# the training progress more smoothly
net.config[nengo.Ensemble].max_rates = nengo.dists.Choice([100])
net.config[nengo.Ensemble].intercepts = nengo.dists.Choice([0])
neuron_type = nengo.LIF(amplitude=0.01)

nengo_dl.configure_settings(trainable=False)

# the input node that will be used to feed in input images
inp = nengo.Node([0] * 28 * 28)

# add the first convolutional layer
x = nengo_dl.tensor_layer(
    inp, tf.layers.dense, units=256)

# apply the neural nonlinearity
x = nengo_dl.tensor_layer(x, neuron_type)

# add another convolutional layer
x = nengo_dl.tensor_layer(
    x, tf.layers.dense, units=64)
x = nengo_dl.tensor_layer(x, neuron_type)



# another convolutional layer
x = nengo_dl.tensor_layer(
    x, tf.layers.dense, units=16)
x = nengo_dl.tensor_layer(x, neuron_type)



# linear readout
x = nengo_dl.tensor_layer(x, tf.layers.dense, units=10)

# we'll create two different output probes, one with a filter
# (for when we're simulating the network over time and
# accumulating spikes), and one without (for when we're
# training the network using a rate-based approximation)
out_p = nengo.Probe(x)
out_p_filt = nengo.Probe(x, synapse=0.1)

minibatch_size = 200
sim = nengo_dl.Simulator(net, minibatch_size=minibatch_size)


train_data = {inp: train_data[0][:, None, :],
out_p: train_data[1][:, None, :]}

// when testing our network with spiking neurons we will need to run it
// over time, so we repeat the input/target data for a number of
// timesteps. we’re also going to reduce the number of test images, just
// to speed up this example.
n_steps = 30
test_data = {
inp: np.tile(test_data[0][:minibatch_size2, None, :],
(1, n_steps, 1)),
out_p_filt: np.tile(test_data[1][:minibatch_size
2, None, :],
(1, n_steps, 1))}


def objective(outputs, targets):
return tf.nn.softmax_cross_entropy_with_logits_v2(
logits=outputs, labels=targets)


opt = tf.train.RMSPropOptimizer(learning_rate=0.001)


def classification_error(outputs, targets):
return 100 * tf.reduce_mean(
tf.cast(tf.not_equal(tf.argmax(outputs[:, -1], axis=-1),
tf.argmax(targets[:, -1], axis=-1)),
tf.float32))

print(“error before training: %.2f%%” % sim.loss(
test_data, {out_p_filt: classification_error}))


do_training = True
if do_training:
# run training
sim.train(train_data, opt, objective={out_p: objective}, n_epochs=10)

# save the parameters to file
sim.save_params("./mnist_params")

else:
# download pretrained weights
urlretrieve(
“https://drive.google.com/uc?export=download&id=1u9JyNuRxQDUcFgkRnI1qfJVFMdnGRsjI”,
“mnist_params.zip”)
with zipfile.ZipFile(“mnist_params.zip”) as f:
f.extractall()

# load parameters
sim.load_params("./mnist_params")

print(“error after training: %.2f%%” % sim.loss(
test_data, {out_p_filt: classification_error}))

Hi Daniel,

Thank you so much for the response. So, I am working on a project where we have to implement spiking neural networks on FPGA. For that we need to implement SNN in c++. And so, we need to train the network in nengo, save the weights and use them in the c++ code.

  1. I just realised that training has to be done using softLIFRate neurons and LIF should be used for inference. So the whole thing I did till now is meaningless.
  2. https://www.nengo.ai/nengo-dl/v0.5.0/examples/spiking_mnist.html
    This link describes the training procedure. I tried to run the same code by setting do_training = True but getting this weird error "TypeError: unhashable type: ‘numpy.ndarray’ "
    If possible please try to run this code. You will get the same error.
  3. I modified the way the input data is being initialized, also removed convolution layers and made it as a simple 3 layer network, and the code performs training successfully using SoftLIFRate neurons with 7% error after training but the issue is that when I perform inference using the weights obtained from training and using LIF neurons as suggested in the link, the error is too high like 90%! Here is the code
#!/usr/bin/env python
# coding: utf-8

# In[1]:


get_ipython().run_line_magic('matplotlib', 'inline')

import gzip
import pickle
from urllib.request import urlretrieve
import zipfile

import nengo
import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt
#from nengo_dl.compat import tf_shape
from tensorflow.examples.tutorials.mnist import input_data

import nengo_dl



import ipdb


mnist = input_data.read_data_sets("MNIST_data/", one_hot=True)


ens_params = dict(max_rates=nengo.dists.Choice([100]), intercepts=nengo.dists.Choice([0]))



softlif_neurons = nengo_dl.SoftLIFRate(tau_rc=0.02, tau_ref=0.002, sigma=0.002)
lif_neurons = nengo.LIF(tau_rc=0.02, tau_ref=0.002)


# In[6]:


def build_network(neuron_type):
    with nengo.Network() as net:

        nengo_dl.configure_settings(trainable=False)

        inp = nengo.Node(nengo.processes.PresentInput(mnist.test.images, 0.1))

        temp = nengo.Probe(inp)

        x0, conn0 = nengo_dl.tensor_layer(inp, neuron_type, **ens_params, transform=nengo_dl.dists.Glorot(),shape_in=(784,), return_conn=True)
        
        net.config[x0].trainable = True
        net.config[conn0].trainable = True

        x1, conn1 = nengo_dl.tensor_layer(x0, neuron_type, **ens_params, transform=nengo_dl.dists.Glorot(),shape_in=(16,), return_conn=True)

        net.config[x1].trainable = True
        net.config[conn1].trainable = True
        
        x, conn = nengo_dl.tensor_layer(x1, tf.layers.dense, units=10, return_conn=True)

        #net.config[conn2].trainable = True
        
    return net,inp,x

net, inp, out = build_network(softlif_neurons)
with net:
    out_p = nengo.Probe(out)
    #out_p_filt = nengo.Probe(out, synapse=0.1)

minibatch_size = 200
sim = nengo_dl.Simulator(net, minibatch_size=minibatch_size)



# add the single timestep to the training data
mnist = input_data.read_data_sets("MNIST_data/", one_hot=True)
train_data = {inp: mnist.train.images[:, None, :],out_p: mnist.train.labels[:, None, :]}


test_data = {inp: mnist.test.images[:minibatch_size*2, None, :],out_p: mnist.test.labels[:minibatch_size*2, None, :]}


# In[13]:


def objective(x, y):
    return tf.nn.softmax_cross_entropy_with_logits(logits=x, labels=y)


# In[14]:


opt = tf.train.AdadeltaOptimizer(learning_rate=1.7)


# In[15]:


def classification_error(outputs, targets):
    return 100 * tf.reduce_mean(tf.cast(tf.not_equal(tf.argmax(outputs[:, -1], axis=-1),tf.argmax(targets[:, -1], axis=-1)),tf.float32))


# In[16]:


print("error before training: %.2f%%" % sim.loss(test_data, {out_p: classification_error}))

do_training = True
if do_training:
    # run training
    sim.train(train_data, opt,objective={out_p: objective},n_epochs=10)

    # save the parameters to file
    sim.save_params("./mnist_params_new")
else:
    # download pretrained weights
    urlretrieve("https://drive.google.com/uc?export=download&id=0B6DAasV-Fri4NEtYNm1hNkcwMEU",
                "mnist_params.zip")
    with zipfile.ZipFile("mnist_params.zip") as f:
        f.extractall()

    # load parameters
    sim.load_params("./mnist_params")

print("error after training: %.2f%%" % sim.loss(test_data, {out_p: classification_error}))

sim.close()


# In[17]:


net, inp, out = build_network(lif_neurons)
with net:
    out_p = nengo.Probe(out, synapse=0.1)

sim = nengo_dl.Simulator(net, minibatch_size=minibatch_size, unroll_simulation=10)
sim.load_params("./mnist_params_new")


# In[19]:


n_steps = 30
test_inp = {
    inp: np.tile(mnist.test.images[:minibatch_size*2, None, :],
                 (1, n_steps, 1)),
    out_p: np.tile(mnist.test.labels[:minibatch_size*2, None, :],
                        (1, n_steps, 1))}

print("spiking neuron error: %.2f%%" % sim.loss(test_inp, {out_p: classification_error}))

Can you please tell me
1)Why the error is too high while doing inference with LIF neurons
2)How to download the weight matrices for the 3 layers
3)The last layer is a tf.layers.dense layer…and as per the config settings in my code, it’s weights are not being trained. So what does that mean? I followed the same format in the link given above.

So, sim.data[conn].weights will give me the weight matrices for those layers that use nengo_dl.tensor_layer(x, neuron_type) but it just returns a single element for the layer nengo_dl.tensor_layer(x, tf.layers.dense). That is why I wanted to use ensembles and connections to retrieve weights of all the layers.

Please help. Have been struggling with all kinds of modifications and now we are kind of back to the staring point. :frowning:

Thanks!

I believe the difference is that if you look at your tf.layers.dense version, the final layer is just a linear readout (no neural nonlinearity):

# linear readout
x = nengo_dl.tensor_layer(x, tf.layers.dense, units=10)

but in the nengo.Connection version, you are including neurons in that final layer:

x, conn4 = nengo_dl.tensor_layer(x, neuron_type,
                                 transform=nengo_dl.dists.Glorot(),
                                 shape_in=(10,), return_conn=True)

If you change your final layer to something like

x_new = nengo.Node(size_in=10)
conn4 = nengo.Connection(x, x_new, transform=nengo_dl.dists.Glorot(),
                         synapse=None)

then you should see more similar performance. Note that we’re using a Node with no function to act as a simple linear output.

Using SoftLIFRate instead of LIFRate during training is an optional step that might improve performance, not a requirement. So I don’t think you need to worry that any of the work you’ve been doing is meaningless.

That version of the example you’re looking at is for a very old version of NengoDL (0.5.0), so it won’t work if you have a more up-to-date NengoDL version installed. You can find a more up-to-date version of the example here https://www.nengo.ai/nengo-dl/examples/spiking-mnist.html.

I haven’t looked at your code in a ton of detail, but again I noticed that your code is based on an old version of the spiking-mnist example. To bring it up to date I just used the latest version of the example (https://www.nengo.ai/nengo-dl/examples/spiking-mnist.html), and then replaced cell 3 (the network definition) with

with nengo.Network() as net:
    # set some default parameters for the neurons that will make
    # the training progress more smoothly
    net.config[nengo.Ensemble].max_rates = nengo.dists.Choice([100])
    net.config[nengo.Ensemble].intercepts = nengo.dists.Choice([0])
    neuron_type = nengo.LIF(amplitude=0.01)

    nengo_dl.configure_settings(trainable=False)

    inp = nengo.Node(np.zeros(784))

    temp = nengo.Probe(inp)

    x0, conn0 = nengo_dl.tensor_layer(inp, neuron_type,
                                      transform=nengo_dl.dists.Glorot(),
                                      shape_in=(784,), return_conn=True)

    net.config[x0].trainable = True
    net.config[conn0].trainable = True

    x1, conn1 = nengo_dl.tensor_layer(x0, neuron_type,
                                      transform=nengo_dl.dists.Glorot(),
                                      shape_in=(16,), return_conn=True)

    net.config[x1].trainable = True
    net.config[conn1].trainable = True

    x, conn = nengo_dl.tensor_layer(x1, tf.layers.dense, units=10,
                                    return_conn=True)

    # net.config[conn2].trainable = True

    # we'll create two different output probes, one with a filter
    # (for when we're simulating the network over time and
    # accumulating spikes), and one without (for when we're
    # training the network using a rate-based approximation)
    out_p = nengo.Probe(x)
    out_p_filt = nengo.Probe(x, synapse=0.1)

(i.e. I just copy and pasted in your modified Network). And running that modified example seems to work fine (< 3% error after training, using spiking LIF neurons). So you could just use the more up-to-date code, or if you want to stick with the older version you could try to compare the old versus new version and see what the significant differences are, and use that to modify your older version so that it works.

You can access the weights for all of the standard Nengo layers (the ones using neuron_type) through sim.data, e.g. sim.data[conn0]. As mentioned above (Need to download weights of fully connected network), when you use tf.layers, those connection weights are managed internally by TensorFlow, so they are not available through sim.data. If you want easy access to them, instead of using tf.layers.dense you could use nengo.Connection and nengo.Node to create your final readout layer (as in the example above Need to download weights of fully connected network).

The weights in the dense layer are being trained. The trainable=False logic only works on Nengo parameters. As mentioned, when you use tf.layers the parameters are controlled internally by TensorFlow, so it isn’t possible for Nengo to control them through configure_settings(trainable=False). This is mentioned briefly in the spiking-mnist example, but should probably be made more clear!

Thanks a ton! Will implement all the changes! :smiley: