Nengo-gui won't start when using nengo-dl backend

NoaSchwensfeier · February 17, 2021, 8:34am

I am experiencing some issues with opening my nengo model in the nengo GUI. The model uses a nengo-dl simulator, but also contains a line of code that checks whether the model is run in the GUI and sets the model accordingly (using nengo simulators rather than nengo-dl). I have been trying to open the model in the nengo GUI and ran into lots of problems:

Opening the model with the nengo backend works (although it takes forever to load the model), but I cannot run the model. It throws the following error:

Traceback (most recent call last):
  File "/home/s3344282/.conda/envs/Noa_test2/lib/python3.7/site-packages/nengo_gui/page.py", line 486, in build
    self.model, progress_bar=self.locals['_viz_progress'])
  File "/home/s3344282/.conda/envs/Noa_test2/lib/python3.7/site-packages/nengo/simulator.py", line 160, in __init__
    progress=pt.next_stage("Building", "Build"))
  File "/home/s3344282/.conda/envs/Noa_test2/lib/python3.7/site-packages/nengo/builder/builder.py", line 123, in build
    built = self.builder.build(self, obj, *args, **kwargs)
  File "/home/s3344282/.conda/envs/Noa_test2/lib/python3.7/site-packages/nengo/builder/builder.py", line 218, in build
    return cls.builders[obj_cls](model, obj, *args, **kwargs)
  File "/home/s3344282/.conda/envs/Noa_test2/lib/python3.7/site-packages/nengo/builder/network.py", line 113, in build_network
    model.build(conn)
  File "/home/s3344282/.conda/envs/Noa_test2/lib/python3.7/site-packages/nengo/builder/builder.py", line 123, in build
    built = self.builder.build(self, obj, *args, **kwargs)
  File "/home/s3344282/.conda/envs/Noa_test2/lib/python3.7/site-packages/nengo/builder/builder.py", line 218, in build
    return cls.builders[obj_cls](model, obj, *args, **kwargs)
  File "/home/s3344282/.conda/envs/Noa_test2/lib/python3.7/site-packages/nengo/builder/connection.py", line 305, in build_connection
    model.build(r)
  File "/home/s3344282/.conda/envs/Noa_test2/lib/python3.7/site-packages/nengo/builder/builder.py", line 123, in build
    built = self.builder.build(self, obj, *args, **kwargs)
  File "/home/s3344282/.conda/envs/Noa_test2/lib/python3.7/site-packages/nengo/builder/builder.py", line 218, in build
    return cls.builders[obj_cls](model, obj, *args, **kwargs)
  File "/home/s3344282/.conda/envs/Noa_test2/lib/python3.7/site-packages/nengo/builder/learning_rules.py", line 392, in build_learning_rule
    model.build(rule.learning_rule_type, rule)  # updates delta
  File "/home/s3344282/.conda/envs/Noa_test2/lib/python3.7/site-packages/nengo/builder/builder.py", line 123, in build
    built = self.builder.build(self, obj, *args, **kwargs)
  File "/home/s3344282/.conda/envs/Noa_test2/lib/python3.7/site-packages/nengo/builder/builder.py", line 216, in build
    "Cannot build object of type %r" % type(obj).__name__)
nengo.exceptions.BuildError: Cannot build object of type 'STP'

I think that this is related to using the nengo backend, because the model uses a custom neuron type with custom learning rules implemented in nengo-dl. Switching to the nengo-dl backend in the GUI sometimes leads to the model being loaded, sometimes not (throwing a timeout error, saying that the page has become unresponsive). If the model is loaded, I cannot continue due to the error:

finish() got an unexpected keyword argument 'dirty'

I found out that this error is related to the progressbar2 package (3.53.1), and that the bug was fixed in a newer version of nengo-dl (currently using 2.2.2). However, I cannot use a newer version due to using nengo 2.8.0, so instead I tried installing an earlier version of progressbar2 (3.47.0). With this package, the model won’t load at all, regardless of which backend is used, and I see the ‘page unresponsive’ pop-up after some time. I am not sure whether this means that using the earlier progressbar2 version fixes the finish() error or introduces another problem.

I am new to working with nengo, so please let me know whether I need to provide more specific information!

Packages: nengo 2.8.0, nengo-dl 2.2.2, nengo-ocl 1.4.0, nengo-gui 0.4.7, python 3.7.9

xchoo · February 17, 2021, 4:43pm

Hi @NoaSchwensfeier,

Without some example code, I am unable to verify the Cannot build object of type 'STP' error, but, I can confirm that with nengo==2.8.0, nengo-dl==2.2.2 and nengo-gui==0.4.7, I do get the finish() got an unexpected keyword argument 'dirty' error when trying to run a model in NengoGUI with the NengoDL backend (i.e., running nengo -b nengo_dl).

I created a new Python environment and installed Nengo, NengoDL, and NengoGUI with:

pip install nengo nengo-dl nengo-gui

and with that I was able to run just the default Nengo network without any issues. Can you try that?
The versions of the packages that were installed are:
nengo==3.0.0, nengo-dl==3.4.0, nengo-gui==0.4.7

Somewhat related:
I also noticed that you have NengoOCL installed. While testing that, I noticed the PyOpenCL installation installed numpy==1.20.0, but TensorFlow has a version limit on Numpy of numpy<=1.19.5. Thus, to install the packages for this new environment, I would recommend:

pip install pyopencl tensorflow nengo nengo-gui nengo-dl nengo-ocl

That should (in theory) make sure all of the package dependencies are matched.

NoaSchwensfeier · February 17, 2021, 5:56pm

Hi @xchoo,

thanks for the quick response! I tried out the combination of packages in a new environment:

and that way I can indeed run a simple model with the NengoDL backend. However, and I should have mentioned this earlier, the code I am working with utilizes Nengo 2.8.0 and does not work with the 3.0.0 version, as it uses several of the modules that have been deprecated in newer Nengo versions. As it turns out, the combination of packages I mentioned here is the only combination that works for my code at all:

I understand that this greatly limits the options I have to make this work. Unfortunately, the scope of my project won’t allow me to update the code to make it work for Nengo 3.0.0. Might there be any workarounds with this particular package combination?

xchoo · February 17, 2021, 6:11pm

Hmm… I’ll have to give it a poke to see if you can get things to run with this configuration.

xchoo · February 17, 2021, 6:43pm

Hokay. I think I got it to work with this pip install configuration:

pip install "nengo==2.8.0" "nengo-dl==2.2.2" "nengo-gui==0.4.7" "tensorflow<=2.0.0" "progressbar2<3.50.0"

I noticed that NengoDL 2.2.2 supports up to TF 2.0.0, and I wasn’t sure what TF version you had installed, so I forced pip to use the correct version. And yes, as you pointed out, the bug in the progressbar2 package was introduced in 3.50.0, so I had to install one version before that.

If this still doesn’t work for you, I’d recommend leaving the progressbar2 package as is, and to replicate this change in the nengo_dl/utils.py file (if you installed NengoDL through pip it’ll be in your .../site-packages/nengo-dl/utils.py file).

NoaSchwensfeier · February 18, 2021, 1:59pm

Thanks a lot for diving into this so deeply! I implemented the changes you suggested. As before, it allows me to run the default Nengo model with the NengoDL backend, but I still cannot load and see my own model. The page becomes unresponsive and does not become responsive again, even when waiting for a significant amount of time. I used tensorflow 1.15 before, but now installed tensorflow 2.0.0 as you suggested, but am still left with the same problem.

It is not integral to the project that the model works in NengoGUI, and it seems like there is no obvious fix for the issue, so I think I will just have to carry on without the GUI. Thanks a lot for your support!

xchoo · February 18, 2021, 2:18pm

Hmmm. That’s very puzzling. If you are willing to look into the problem further, I can try to run your model on my system to try and figure out what is causing it to hang the GUI. Another quick thing to look at is to see if any error messages appear in the terminal console (the one where you run the nengo -b nengo_dl command from) when you try to run your model.

NoaSchwensfeier · February 18, 2021, 5:12pm

Thanks for the offer! The code I am working with can be found here (model is contained within exp1 and exp2 files, exp2 contains a slightly different version of the model): https://github.com/ChielWijs/Dynamic-Coding-in-Working-Memory

I do not get any errors in the terminal console while trying to run the model.

xchoo · February 18, 2021, 5:15pm

Thanks! I’ll take a poke at your code and see if I can figure out what’s going on. I’ll keep you posted.

xchoo · February 18, 2021, 7:01pm

So… I couldn’t find anything wrong with your model, and it loaded up fine for me in NengoGUI. Here’s exp1.py:

and exp2.py:

I made no substantial changes to the model apart from commenting out the imagearr loading bit (at the top of the script), because I didn’t have the source images. I replaced this block of code with just:

imagearr = np.zeros((1801, 16384))

And one other change I made was to the model definition for if nengo_gui_on, where I encapsulated the create_model function within a top level Nengo network:

if nengo_gui_on:
    generate_gabors()  # generate gabors
    with nengo.Network() as model:
        create_model(seed=0)  # build model

    memory_item_first = 0 + 90
    probe_first = 40 + 90
    memory_item_second = 0 + 90
    probe_second = 40 + 90

else:  # no gui
    ....

This was done because without this, I was getting this error in NengoGUI:

must declare a nengo.Network called "model"

Other than these changes, no other changes were made.

I did notice that the model does use several “large” (1000+ neurons) size ensembles. The “page not responsive” error you might be encountering might be a symptom of your system running out of memory, or just that the CPU is taking a while to build the model. If this is the case, you can test this by reducing the neuron count for the model (I believe these are the Ns, Nm, Nc, Nd parameters at the top of the script?)

This is the status of my pip freeze:

absl-py==0.11.0
appdirs @ file:///home/conda/feedstock_root/build_artifacts/appdirs_1603108395799/work
astor==0.8.1
cached-property==1.5.2
cachetools==4.2.1
certifi==2020.12.5
chardet==4.0.0
cycler==0.10.0
decorator==4.4.2
descartes==1.1.0
gast==0.2.2
google-auth==1.27.0
google-auth-oauthlib==0.4.2
google-pasta==0.2.0
grpcio==1.35.0
h5py==3.1.0
idna==2.10
importlib-metadata==3.4.0
Keras-Applications==1.0.8
Keras-Preprocessing==1.1.2
kiwisolver==1.3.1
Mako @ file:///home/conda/feedstock_root/build_artifacts/mako_1610659158978/work
Markdown==3.3.3
MarkupSafe @ file:///D:/bld/markupsafe_1610127696479/work
matplotlib==3.3.4
mizani==0.7.2
nengo==2.8.0
nengo-dl==2.2.2
nengo-extras==0.4.0
nengo-gui==0.4.7
nengo-ocl==1.4.0
numpy==1.19.5
oauthlib==3.1.0
opt-einsum==3.3.0
palettable==3.3.0
pandas==1.2.2
patsy==0.5.1
Pillow==8.1.0
plotnine==0.7.1
progressbar2==3.47.0
protobuf==3.14.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pyopencl==2021.1.2
pyparsing==2.4.7
python-dateutil==2.8.1
python-utils==2.5.6
pytools @ file:///home/conda/feedstock_root/build_artifacts/pytools_1610227461013/work
pytz==2021.1
requests==2.25.1
requests-oauthlib==1.3.0
rsa==4.7.1
scipy==1.6.1
six @ file:///home/conda/feedstock_root/build_artifacts/six_1590081179328/work
statsmodels==0.12.2
tensorboard==2.0.2
tensorflow==2.0.0
tensorflow-estimator==2.0.1
termcolor==1.1.0
typing-extensions==3.7.4.3
urllib3==1.26.3
Werkzeug==1.0.1
wincertstore==0.2
wrapt==1.12.1
zipp==3.4.0

NoaSchwensfeier · February 19, 2021, 9:47am

I have been trying a few things out and think I know what the problem was, now. First, I tried implementing the changes you mentioned in the file I had on my system already, and with which I had been working for some time now. None of the suggestions worked there.
I then downloaded the file from GitHub again, and with this change I could also load and run the model:

However, the page seems very unstable, as it becomes unresponsive sometimes, and it crashed when I pulled up too many plots. After it crashed, I could not open it again, but after deleting the configure file, it loaded again. I suppose that this caused my problems with loading the model before (I had been able to see the model once and pull up some plots, but then it crashed and I could never load it again, but it never occured to me to delete the configure file). So, it might indeed be that the system ran out of memory while trying to load all model components and plots.

The simulation is super slow, but at least I can see the model now! This is already great for me, thanks so much for your help!