Nengo RL performance with standard OpenAI cart pole problem


Recently tried to develop RL using Nengo and integrated with open AI gym.
Code is available here

But I have few queries its performance is not so great.
I found DQN using torch and Tensorflow performs really awesome.


As you can see running average of rewards basically shows Nengo RL agent can balance pole for just 20 to 25 steps.

Nengo is very good cognitive modelling platform.But expecting Nengo to perform as good as like any other neural networks library like Torch or TF is valid(might be valid for NengoDL)?

To improve it’s performance what should I do?

Hi @ganesh! Right now it looks like your network is using PES to do the learning online. This is quite a different approach from the offline training procedures used in DQN. Both have pros and cons, but in general, an online learning algorithm like PES can be pretty sensitive to the learnin grate, so you could try lowering it to see if you can get out of the plateau you seem to be in.

More structurally, however, the big difference with PES is that it can only be applied to the weights on one connection in the network at a time, whereas DQN and other learning mechanisms can optimize all of the variables in the simulation across many layers in order to improve performance. This allow an algorithm like DQN to build up a big library of experiences over time, while PES could be rather short-sighted and sensitive to recent history.

Nengo can certainly perform as well as other neural networks libraries. The comparison is more clear when you use NengoDL. You could, for example, implement DQN in NengoDL without too much trouble.