Is it possible to use SNN as Q_function approximation?

General Discussion
21

Hello drasmuss,
I made the changes, but the problem still not solved.
I am not sure if I did the right way. I reuse the same simulator for training or prediction:

def training(self, minibatch_size, train_whole_dataset, train_whole_labels, num_epochs):
    '''
    Training the network, objective will be the loss function, default is 'mse', but you can alse define your
    own loss function, weights will be saved after the training. 
    :param minibatch_size: the batch size for training. 
    :param train_whole_dataset: whole training dataset, the nengo_dl will take minibatch from this dataset
    :param train_whole_labels: whole training labels
    :param num_epochs: how many epoch to train the whole dataset
    :param pre_train_weights: if we want to fine-tuning the network, load weights before training
    :return: None
    '''

    with nengo.Network(seed=0) as self.model:
        nengo_dl.configure_trainable(self.model, default=True)
        input, output = self.build_network()
        out_p = nengo.Probe(output)

        train_inputs = {input: train_whole_dataset}
        train_targets = {out_p: train_whole_labels}

    with nengo_dl.Simulator(self.model, seed=1, minibatch_size=minibatch_size) as self.sim_train:

        if self.save_path is not None:
            try:
                self.sim_train.load_params(self.save_path)
            except:
                pass

        optimizer = self.choose_optimizer('adadelta', 1)
        # construct the simulator
        self.sim_train.train(train_inputs, train_targets, optimizer, n_epochs=num_epochs, objective='mse')
        # save the parameters to file
        self.sim_train.save_params(self.save_path)

def predict(self, prediction_input, minibatch_size=1):
    '''
    prediction of the network
    :param prediction_input: a input data shape = (minibatch_size, 1, input_shape)
    :param minibatch_size: minibatch size, default = 1
    :return: prediction with shape = (minibatch_size, output_shape)
    '''

    with nengo.Network(seed=0) as self.model:
        nengo_dl.configure_trainable(self.model, default=False)
        input, output = self.build_network()
        out_p = nengo.Probe(output)

    with nengo_dl.Simulator(self.model, seed=2, minibatch_size=minibatch_size) as self.sim_prediction:
        try:
            self.sim_prediction.load_params(self.save_path)
        except:
            pass

        input_data = {input: prediction_input}
        self.sim_prediction.step(input_feeds=input_data)
        output = np.squeeze(self.sim_prediction.data[out_p], axis=1)

        return deepcopy(output)
22

The nengo_dl.Simulator(self.model, seed=2, minibatch_size=minibatch_size) line is what creates a new Simulator. So you just want to call that once (e.g., in your constructor), rather than creating a new Simulator inside the training or predict functions.

23

I tracked down the memory leak and submitted a fix to TensorFlow. However, we’ll have to wait until the next release for that to change. In the meantime I added a temporary patch to nengo_dl to fix the issue on our end, so if you update to the latest commit from the repo it should be fixed.

24

Hello drasmuss,

In nengo Core, if I want to do off-line learning, I use nengo.solver to solve the decoder.

    conn = nengo.Connection(input_neuron,
                            output,
                            synapse=None,
                            eval_points=train_data,
                            function=train_targets,
                            solver=solver
                            )

Here eval_points is your training data, function is you label. then I can solve the connection weights(decoder).
My question is, can we here first load the pre-trained weights, then the nengo solver solve the optimization problem.

25

If I do online learning, I know how to load and save weights:

In online learning with PES learning rule:
I can load the weights like:

learn_conn = nengo.Connection(pre.neurons, post, transform=weights, learning_rule_type=nengo.PES())

if I do not load the weights:
learn_conn = nengo.Connection(pre, post, function=init_func, learning_rule_type=nengo.PES())

But here the problem is I do not know in online learning, how can I do supervised learning like I do with nengo solver.
I tried create two pre and post ensemble, and a error node. Then I feed training data to the pre, label to the post. Then use PES learning rule minimize the error, but it does not work.

26

A least-squares optimization process (of which the nengo solvers are an example) solves for optimal weights in a single pass. So the initial values of the weights don’t make any difference, it will always arrive at the same (optimal) solution regardless.

That is the idea (the PES rule is doing supervised learning). When you say it doesn’t work, do you just mean that it doesn’t learn the function? As before, that doesn’t necessarily mean you’re doing something wrong, it may just mean that the function you’re trying to learn is a difficult one, and you need to refine other parameters (e.g., how long you present each input for, learning rates, synaptic time constants, etc.).

27

Hello drasmuss,

Thanks a lot you take your time to see my questions.
I didn’t make my question clear, my problem is I can’t debug my code. And the errors are:

File “Q_network_snn_v3.py”, line 61, in
with nengo.Simulator(model) as sim:
File “/usr/local/lib/python2.7/dist-packages/nengo/simulator.py”, line 153, in init
self.model.build(network, progress_bar=self.progress_bar)
File “/usr/local/lib/python2.7/dist-packages/nengo/builder/builder.py”, line 121, in build
built = Builder.build(self, obj, *args, **kwargs)
File “/usr/local/lib/python2.7/dist-packages/nengo/builder/builder.py”, line 216, in build
return cls.builders[obj_cls](model, obj, *args, **kwargs)
File “/usr/local/lib/python2.7/dist-packages/nengo/builder/network.py”, line 99, in build_network
model.build(obj)
File “/usr/local/lib/python2.7/dist-packages/nengo/builder/builder.py”, line 121, in build
built = Builder.build(self, obj, *args, **kwargs)
File “/usr/local/lib/python2.7/dist-packages/nengo/builder/builder.py”, line 216, in build
return cls.builders[obj_cls](model, obj, *args, **kwargs)
File “/usr/local/lib/python2.7/dist-packages/nengo/builder/node.py”, line 43, in build_node
model.build(node.output, sig_in, sig_out)
File “/usr/local/lib/python2.7/dist-packages/nengo/builder/builder.py”, line 121, in build
built = Builder.build(self, obj, *args, **kwargs)
File “/usr/local/lib/python2.7/dist-packages/nengo/builder/builder.py”, line 216, in build
return cls.builders[obj_cls](model, obj, *args, **kwargs)
File “/usr/local/lib/python2.7/dist-packages/nengo/builder/processes.py”, line 136, in build_process
process, sig_in, sig_out, model.time, mode=‘inc’ if inc else ‘set’))
File “/usr/local/lib/python2.7/dist-packages/nengo/builder/builder.py”, line 109, in add_op
op.make_step(signals, self.dt, np.random)
File “/usr/local/lib/python2.7/dist-packages/nengo/builder/processes.py”, line 96, in make_step
step_f = self.process.make_step(shape_in, shape_out, dt, rng)
File “/usr/local/lib/python2.7/dist-packages/nengo/processes.py”, line 251, in make_step
assert shape_in == (0,)
AssertionError

#------------------------------------------------------------------------------------------------------#
What I want to do is: use a input node to feed training data, use a output node to feed labels/groundtruth, then optimize the connection weights between pre and post ensemble. Here are the code:

import os
import h5py
import nengo
import numpy as np

from keras.datasets import mnist
from keras.utils import np_utils
from sklearn.metrics import accuracy_score


data_file = 'weights.h5'
input_shape = 784
output_shape = 10


data = np.zeros([1, 784])
label = np.zeros([1, 10])
nb_neuron = 1000

(X_train, y_train), (X_test, y_test) = mnist.load_data()

X_train = X_train.reshape(X_train.shape[0], -1) / 255.  # normalize
X_test = X_test.reshape(X_test.shape[0], -1) / 255.  # normalize
y_train = np_utils.to_categorical(y_train, num_classes=10)
y_test = np_utils.to_categorical(y_test, num_classes=10)


presentation_time = 0.1
model = nengo.Network(seed=3)
with model:

    input = nengo.Node(nengo.processes.PresentInput(X_train, presentation_time), size_in=784)
    pre = nengo.Ensemble(n_neurons=nb_neuron, dimensions=input_shape)
    post = nengo.Ensemble(n_neurons=nb_neuron, dimensions=output_shape)
    error = nengo.Node(size_in=output_shape)
    output = nengo.Node(nengo.processes.PresentInput(y_train, presentation_time), size_in=10)


    nengo.Connection(input, pre)
    nengo.Connection(output, error, transform= -1)
    nengo.Connection(post, error, transform = 1)
    nengo.Connection(output, post)

    output_p = nengo.Probe(output, "output")

    if os.path.isfile(data_file):
        # data file with weights exists, so initialize learning connection with those weights
        with h5py.File(data_file, 'r') as hf:
            weights = np.array(hf.get('weights'))
        learn_conn = nengo.Connection(pre.neurons, post, transform=weights, learning_rule_type=nengo.PES())
        nengo.Connection(error, learn_conn.learning_rule)

    else:
        def init_func(x):
          return np.zeros(output_shape)
        learn_conn = nengo.Connection(pre, post, function=init_func, learning_rule_type=nengo.PES())
        nengo.Connection(error, learn_conn.learning_rule)

    conn_p = nengo.Probe(learn_conn, 'weights')

with nengo.Simulator(model) as sim:

  sim.run(time_in_seconds=20)
  weights = sim.data[conn_p][len(sim.trange()) - 1, :, :]
  _, acts = nengo.utils.ensemble.tuning_curves(pre, sim, inputs=X_test)

  print type(weights)
  print type(acts)

  print weights.shape
  print acts.shape

  output = np.dot(acts, weights.T)


  acc = accuracy_score(y_true=np.argmax(y_test[0:100,:], axis=1), y_pred=np.argmax(output[0:100,:],axis=1))
  print "the test acc is:", acc

1111.png811×333 14 KB

28

The error you’re getting is because you specified size_in=x on the PresentInput nodes, but the PresentInput process doesn’t have any inputs. The size_in should be 0.

More generally, you’re feeding the target values directly into post, so in your model learn_conn isn’t really adding anything. All post has to do is output the values it is getting from the output node in order to minimize the error. That probably isn’t what you want to happen, I’m guessing, in which case you should remove the nengo.Connection(output, post) connection.

29

Yeah! The problem is solved. I made a stupid mistake.

I have two further questions,
1, What is the relationship between present-time and simulation time or simulation_run_step?
I want to synchronise the simulation-time and present time, if I feed a batch into the Node, I can make sure all data are trained.
For example, If my training batch is 1000, and I set present time=0.1, in Simulation, I should set ‘run’ or ‘run_step’ to which number so that all batch will be went through.

2, After training, I want to do prediction or called inference.
I should use:

_, acts = nengo.utils.ensemble.tuning_curves(pre, sim, inputs=X_test[0:100, :])
prediction = np.dot(acts, weights.T)

or I should run the simulation, the use Probe record the prediction?

30

presentation_time is the length of time that each row in the array you pass to PresentInput will be output for. So, e.g., if you do PresentInput(my_data, 0.1), that is saying that each element in my_data will be output for 0.1 seconds. So if you had 20 elements in my_data, you would need to run the simulation for 2 seconds to cycle through all the elements in my_data, i.e. sim.run(2.0). The equivalent sim.run_steps depends on your simulation timestep, but e.g. for dt=0.001 (the default), that would be equivalent to sim.run_steps(2000).

You should run the simulation and probe post to check the output.