Help with Sequential Keras Model

Deep Learning
1

Thank you so much.
Could you please suggest how to model these kinds of sequential model,
Example-
model = Sequential()
model.add(Dense(10000, input_dim=10, activation=“relu”))
model.add(Dense(1000, activation=“relu”))
model.add(Dense(20))
model.compile(loss=“mse”, optimizer=“adam”)
model.fit(train, y_train, epochs=15, batch_size=2, verbose=2)

2

For sequential models, to use them in NengoDL, you can either re-implement them yourself, as demonstrated by this example, or, you can use the NengoDL converter to convert your Keras model to a Nengo model, as demonstrated by this example.

3

I tried this example

def modelDef():

inp = tf.keras.Input(shape=(20), name="input")

to_spikes_layer = tf.keras.layers.Conv1D(

    filters=17,  # 3 neurons per pixel

    kernel_size=1,

    strides=1,

    activation=tf.nn.relu,

    use_bias=False,

    name="to-spikes",

)

to_spikes = to_spikes_layer(inp)

# on-chip layers

flatten = tf.keras.layers.Flatten(name="flatten")(to_spikes)

dense0_layer = tf.keras.layers.Dense(units=10, activation=tf.nn.relu, name="dense0")

dense0 = dense0_layer(flatten)

# since this final output layer has no activation function,

# it will be converted to a `nengo.Node` and run off-chip

dense1 = tf.keras.layers.Dense(units=num_classes, name="dense1")(dense0)



model = tf.keras.Model(inputs=inp, outputs=dense1)

model.summary()

return model

def train(params_file="./keras_to_loihi_params123", epochs=1, **kwargs):

model = modelDef() 

miniBatch = 2

converter = nengo_dl.Converter(model, **kwargs)

with nengo_dl.Simulator(converter.net, seed=0, minibatch_size=miniBatch) as sim:        

    

    sim.compile(

        optimizer=tf.optimizers.Adam(0.001),

        loss= {

            converter.outputs[model.get_layer('dense1')]: tf.losses.MeanSquaredError()

            },

        metrics={converter.outputs[model.get_layer('dense1')]: tf.metrics.Accuracy()},

        )

    sim.fit(

        {converter.inputs[model.get_layer('input')]: train_Images},

        {converter.outputs[model.get_layer('dense1')]: train_Labels},

        epochs=epochs,

    )

train this network with normal ReLU neurons

train(

epochs=500,

swap_activations={tf.nn.relu: nengo.RectifiedLinear()},

)

X_train and X_test dimension is (4000,1,20)

but got the ValueError: Input 0 of layer to-spikes is incompatible with the layer

4

Hi @ssp,

I’m having a bit of trouble parsing the exact code you are running. Can upload the python script you are using, or some minimal code that reproduces this error?

5

here x and y are sequential numbers of dimension (2000,10) and (2000,20)
model = Sequential()
model.add(Dense(10000, input_dim=10, activation=“relu”))
model.add(Dense(1000, activation=“relu”))
model.add(Dense(20))
model.compile(loss=“mse”, optimizer=“adam”)
model.fit(train, y_train, epochs=15, batch_size=2, verbose=2)

6

Hi! I understand you want to work with this Keras model in particular, but what exactly do you want to do with it? And what kind of errors are you experiencing?

7

I want to do a regression task but
all the references I got for image data so when I tried to replicate the similar method like this

import nengo
import nengo_dl
import tensorflow as tf

# create your keras model

model = tf.keras.Sequential(
    [
        tf.keras.layers.Dense(units=1000, activation="relu", input_shape=(10,)),
        tf.keras.layers.Dense(units=100, activation="relu"),
        tf.keras.layers.Dense(units=20, activation="relu"),
    ]
)

# convert to nengo_dl and swap activations to spiking ones
converter = nengo_dl.Converter(model, swap_activations={tf.nn.relu: nengo.SpikingRectifiedLinear()})

with converter.net as net:
    # probe on our output layer
    probe_output = converter.outputs[converter.model.output]

    # probe on our second layer
    probe_layer = nengo.Probe(converter.layers[model.layers[1].get_output_at(-1)])

with nengo.Simulator(converter.net) as sim:
    # run our simulation for 1 sec
    sim.run(1)

    # print our probe data
    print(sim.data[probe_output])
    print(sim.data[probe_layer])

but got 0’s as output like this,

[[0. 0. 0. ... 0. 0. 0.]
 [0. 0. 0. ... 0. 0. 0.]
 [0. 0. 0. ... 0. 0. 0.]
 ...
 [0. 0. 0. ... 0. 0. 0.]
 [0. 0. 0. ... 0. 0. 0.]
 [0. 0. 0. ... 0. 0. 0.]]

Could you please suggest on what went wrong?

8

Looking at the code you posted, that is expected behaviour of the network. Although your network defines a structure, it is not trained on any data, nor is any input specified to the network when it is running. Thus, the output of the network is just 0.

Using NengoDL to construct a network to do regression task for image data is sort of a two step process. The first step of the process is to use TensorFlow to define an architecture and train the model on the image data. The second step of the process (which is entirely optional) is to use NengoDL to convert the TensorFlow network to a spiking neural network and see what effects this has on the network (note: this step would be mandatory if you want to put the network on hardware that requires some sort of spiking behaviour).

For the first step of the process (i.e., using TensorFlow), I’d recommend checking out the TensorFlow tutorials to get a better understanding of how to use TensorFlow, and what it can do. Once you have a TensorFlow network working with your dataset, we can then re-examine your NengoDL implementation. TensorFlow has a tutorial specifically on regression that may be of interest to you.

9

Thank you so much @xchoo sorry to bother you a lot But I have a quick question
Both inputs are simple NumPy arrays
X = X[:,0:20]
Y = Y[:0:20]
How to convert these inputs to nengo method or tensors?

X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.2, random_state=3)

from tensorflow.keras.models import load_model
model=load_model(“NN_regres.h5”)
prediction=model.predict(X_test)

10

Once again, I’m not entirely sure what you are attempting to achieve here, so you are going to have to bear with me.

If you want to use those Numpy arrays as inputs to your TensorFlow network, it seems like you are already doing that, with the code:

prediction = model.predict(X_test)

That is assuming you want to use the X array as inputs to generate model predictions? If you are, then just replace X_test with X. As for converting the Numpy arrays to Tensors, I’m not sure what you mean by that. Do you want to use the arrays as inputs to your TensorFlow network?

As for converting the Numpy arrays into Nengo methods, there are multiple ways of doing, each dependent on what Nengo product you are using. If you are using regular Nengo, then you can use a nengo.Node to feed those Numpy arrays to your network:

x_node = nengo.Node(X)  # Make a Nengo node to just output the X array
nengo.Connection(x_node, ...)  # Make connection to your network

If you are using NengoDL, then you’ll need to define an input layer (i.e. a tf.keras.Input layer), and then you can specify the Numpy arrays as inputs when you run the simulation (see here).

Once again, I recommend starting with TensorFlow and getting a model working there first. Trying to understand TensorFlow, Nengo, and NengoDL all at the same time will make things very confusing since there are multiple concepts and ways of doing things.