It’s not surprising that you would be having a problem with overfitting, that is a common problem when training neural networks. You can read more about it in lots of places, e.g. this article.
However, it is surprising if you are getting different results using tf.layers.dense versus nengo.Connection, as those two are generally equivalent. If you can post your two models that show different behaviour I can see if I can spot any differences in how you’ve set things up.