Cannot import nengo_dl on ms azure

hadarcd · February 28, 2021, 10:58am

Hi,
I tried to: pip install nengo-dl on Microsoft azure
I was able to install it but got this exception when import nengo_dl:

AttributeError                            Traceback (most recent call last)
<ipython-input-9-700eed500250> in <module>
----> 1 import nengo_dl

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/nengo_dl/__init__.py in <module>
     40 
     41 # need to explicitly import these to trigger the builder registration
---> 42 from nengo_dl import (
     43     op_builders,
     44     neuron_builders,

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/nengo_dl/op_builders.py in <module>
     23 from nengo_dl import utils
     24 from nengo_dl.builder import Builder, OpBuilder
---> 25 from nengo_dl.compat import SimProbe
     26 
     27 logger = logging.getLogger(__name__)

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/nengo_dl/compat.py in <module>
    157     # monkeypatch to fix bug in TF2.2, see
    158     # https://github.com/tensorflow/tensorflow/issues/37548
--> 159     old_conform = network.Network._conform_to_reference_input
    160 
    161     def _conform_to_reference_input(self, tensor, ref_input):

AttributeError: type object 'Network' has no attribute '_conform_to_reference_input'

thanks

zerone · February 28, 2021, 2:27pm

Hello @hadarcd, can you check the version of Nengo-DL installation. You can do that by nengo_dl.__version__ after importing nengo_dl. If it shows it’s version lesser than 3.4.0 (which is latest), please install it by pip install nengo_dl==3.4.0. Let us know if still facing import issues.

hadarcd · February 28, 2021, 3:22pm

hi,
thanks, I was able to import nengo_dl (after run pip install nengo_dl==3.4.0)
I have a new issue,
got this warning:
UserWarning: No GPU support detected. See Installation — NengoDL 3.4.1.dev0 docs for instructions on setting up TensorFlow with GPU support.
"No GPU support detected. See "

I’m currently using GPU on azure

thanks

zerone · February 28, 2021, 5:59pm

With respect to the technicalities of leveraging GPUs on cloud platforms, I am not sure how much of help I can be here. Are you running Nengo-DL inside a VM? If yes, can you check if you have access to the GPU in the VM’s hardware specifications? Also, if you have installed tensorflow-gpu (version 2.2.0), can you check if it detects GPUs? If it does, then Nengo-DL shouldn’t have any issue in detecting the GPU on your VM.

hadarcd · March 1, 2021, 8:17am

now I’m not able to run it even without GPU

got this error:

Build finished in 0:00:00                                                      
|#                Optimizing graph: merging operators                 | 0:00:00` `---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-16-8b7cae5fdda4> in <module>
      8 #     )
      9 
---> 10 with nengo_dl.Simulator(net, minibatch_size=5, unroll_simulation=7) as sim:
     11     sim.compile(
     12         loss=keras.losses.MeanAbsoluteError(reduction="auto", name="mean_abs_error"),

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/nengo_dl/simulator.py in __init__(self, network, dt, seed, model, device, unroll_simulation, minibatch_size, progress_bar)
    521                 device,
    522                 progress,
--> 523                 seed,
    524             )
    525 

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/tensorflow/python/training/tracking/base.py in _method_wrapper(self, *args, **kwargs)
    515     self._self_setattr_tracking = False  # pylint: disable=protected-access
    516     try:
--> 517       result = method(self, *args, **kwargs)
    518     finally:
    519       self._self_setattr_tracking = previous_value  # pylint: disable=protected-access

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/nengo_dl/tensor_graph.py in __init__(self, model, dt, unroll_simulation, minibatch_size, device, progress, seed)
    134 
    135         with progress.sub("merging operators", max_value=None):
--> 136             plan = planner(operators)
    137 
    138         # TODO: we could also merge operators sequentially (e.g., combine

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/nengo_dl/graph_optimizer.py in tree_planner(op_list, max_depth)
    297     for j, op in enumerate(op_list):
    298         for i, g in enumerate(groups):
--> 299             if mergeable(op, g):
    300                 mergeable_cache[j] = i
    301                 g.append(op)

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/nengo_dl/graph_optimizer.py in mergeable(op, chosen_ops)
     55 
     56     # must share the same builder
---> 57     if builder.Builder.builders[type(op)] != builder.Builder.builders[type(c)]:
     58         return False
     59 

KeyError: <class 'nengo.builder.probe.SimProbe'>`

Thanks

zerone · March 1, 2021, 2:40pm

Can you post a minimal version of your code to reproduce this issue?

xchoo · March 1, 2021, 3:54pm

Hi @hadarcd

To help debug this problem, it would be helpful to know what you system and environment setup is on Azure. It looks like you are using a linux based operating system, so if you do:

uname -a

in a terminal console, it should tell you what OS, kernel version, and OS versions are.

As for your Python environment setup, do:

pip freeze

in the terminal (after you activate your environment in Conda), and it should tell you what packages have been installed in your environment.

As for this error, NengoDL uses TensorFlow as the backend simulation “engine” to run the models, so I would definitely check to see if TensorFlow itself is able to detect and use the GPU. First, you’ll want to see if the GPU is visible to the VM. You can do this with the console command:

lspci -v | grep VGA

You should see a list of the GPUs visible to your VM.

Next, you’ll want to check that the Nvidia drivers have been installed correctly. You can do this with:

nvidia-smi

You should see a table similar to the one below (the details in the table will be different though):

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 430.64       Driver Version: 430.64       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX TIT...  On   | 00000000:05:00.0  On |                  N/A |
| 22%   41C    P8    15W / 275W |     55MiB / 12209MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      1536      G   /usr/lib/xorg/Xorg                            42MiB |
+-----------------------------------------------------------------------------+

Finally, in Python, you can test if TensorFlow itself can detect and use the GPU. To do this, in a Python console or script, do:

import tensorflow as tf
print(tf.config.list_physical_devices('GPU'))

You should see (at the end of the output) something like this:

[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]

hadarcd · March 2, 2021, 8:25am

Linux standad-nc6 4.15.0-1106-azure #118~16.04.1-Ubuntu SMP Tue Jan 19 16:13:06 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

absl-py==0.11.0
adal==1.2.6
aiohttp==3.7.3
aiohttp-cors==0.7.0
aioredis==1.3.1
alembic==1.4.1
ansiwrap==0.8.4
antlr4-python3-runtime==4.7.2
appdirs==1.4.4
applicationinsights==0.11.9
argcomplete==1.12.2
argon2-cffi==20.1.0
astor==0.8.1
astroid==2.4.2
astunparse==1.6.3
async-timeout==3.0.1
atari-py==0.2.6
attrs==20.3.0
autokeras==1.0.12
autopep8==1.5.5
azure-appconfiguration==1.1.1
azure-batch==10.0.0
azure-cli==2.19.1
azure-cli-core==2.19.1
azure-cli-telemetry==1.0.6
azure-common==1.1.26
azure-core==1.10.0
azure-cosmos==3.2.0
azure-datalake-store==0.0.51
azure-functions-devops-build==0.0.22
azure-graphrbac==0.61.1
azure-identity==1.4.1
azure-keyvault==1.1.0
azure-keyvault-administration==4.0.0b1
azure-loganalytics==0.1.0
azure-mgmt-advisor==2.0.1
azure-mgmt-apimanagement==0.2.0
azure-mgmt-appconfiguration==1.0.1
azure-mgmt-applicationinsights==0.1.1
azure-mgmt-authorization==0.61.0
azure-mgmt-batch==9.0.0
azure-mgmt-batchai==2.0.0
azure-mgmt-billing==1.0.0
azure-mgmt-botservice==0.3.0
azure-mgmt-cdn==5.2.0
azure-mgmt-cognitiveservices==6.3.0
azure-mgmt-compute==18.2.0
azure-mgmt-consumption==2.0.0
azure-mgmt-containerinstance==1.5.0
azure-mgmt-containerregistry==2.8.0
azure-mgmt-containerservice==9.4.0
azure-mgmt-core==1.2.2
azure-mgmt-cosmosdb==1.0.0
azure-mgmt-databoxedge==0.2.0
azure-mgmt-datalake-analytics==0.2.1
azure-mgmt-datalake-nspkg==3.0.1
azure-mgmt-datalake-store==0.5.0
azure-mgmt-datamigration==4.1.0
azure-mgmt-deploymentmanager==0.2.0
azure-mgmt-devtestlabs==4.0.0
azure-mgmt-dns==2.1.0
azure-mgmt-eventgrid==3.0.0rc7
azure-mgmt-eventhub==4.1.0
azure-mgmt-hdinsight==2.2.0
azure-mgmt-imagebuilder==0.4.0
azure-mgmt-iotcentral==3.0.0
azure-mgmt-iothub==0.12.0
azure-mgmt-iothubprovisioningservices==0.2.0
azure-mgmt-keyvault==2.2.0
azure-mgmt-kusto==0.3.0
azure-mgmt-loganalytics==8.0.0
azure-mgmt-managedservices==1.0.0
azure-mgmt-managementgroups==0.2.0
azure-mgmt-maps==0.1.0
azure-mgmt-marketplaceordering==0.2.1
azure-mgmt-media==2.2.0
azure-mgmt-monitor==2.0.0
azure-mgmt-msi==0.2.0
azure-mgmt-netapp==0.15.0
azure-mgmt-network==17.0.0
azure-mgmt-nspkg==3.0.2
azure-mgmt-policyinsights==0.5.0
azure-mgmt-privatedns==0.1.0
azure-mgmt-rdbms==3.1.0rc1
azure-mgmt-recoveryservices==0.4.0
azure-mgmt-recoveryservicesbackup==0.11.0
azure-mgmt-redhatopenshift==0.1.0
azure-mgmt-redis==7.0.0rc2
azure-mgmt-relay==0.1.0
azure-mgmt-reservations==0.6.0
azure-mgmt-resource==12.0.0
azure-mgmt-search==8.0.0
azure-mgmt-security==0.6.0
azure-mgmt-servicebus==0.6.0
azure-mgmt-servicefabric==0.5.0
azure-mgmt-signalr==0.4.0
azure-mgmt-sql==0.26.0
azure-mgmt-sqlvirtualmachine==0.5.0
azure-mgmt-storage==11.2.0
azure-mgmt-synapse==0.6.0
azure-mgmt-trafficmanager==0.51.0
azure-mgmt-web==0.48.0
azure-multiapi-storage==0.5.2
azure-nspkg==3.0.2
azure-storage-blob==12.7.1
azure-storage-common==1.4.2
azure-storage-queue==12.1.5
azure-synapse-accesscontrol==0.2.0
azure-synapse-artifacts==0.3.0
azure-synapse-spark==0.2.0
azureml-accel-models==1.22.0
azureml-automl-core==1.22.0
azureml-automl-runtime==1.22.0
azureml-cli-common==1.22.0
azureml-contrib-dataset==1.22.0
azureml-contrib-fairness==1.22.0
azureml-contrib-gbdt==1.22.0
azureml-contrib-interpret==1.22.0
azureml-contrib-notebook==1.22.0
azureml-contrib-pipeline-steps==1.22.0
azureml-contrib-reinforcementlearning==1.22.0
azureml-contrib-server==1.22.0
azureml-contrib-services==1.22.0
azureml-core==1.22.0
azureml-datadrift==1.22.0
azureml-dataprep==2.9.1
azureml-dataprep-native==29.0.0
azureml-dataprep-rslex==1.7.0
azureml-dataset-runtime==1.22.0
azureml-defaults==1.22.0
azureml-explain-model==1.22.0
azureml-interpret==1.22.0
azureml-mlflow==1.22.0
azureml-model-management-sdk==1.0.1b6.post1
azureml-opendatasets==1.22.0
azureml-pipeline==1.22.0
azureml-pipeline-core==1.22.0
azureml-pipeline-steps==1.22.0
azureml-samples @ file:///mnt/jupyter-azsamples
azureml-sdk==1.22.0
azureml-telemetry==1.22.0
azureml-tensorboard==1.22.0
azureml-train==1.22.0
azureml-train-automl==1.22.0
azureml-train-automl-client==1.22.0
azureml-train-automl-runtime==1.22.0
azureml-train-core==1.22.0
azureml-train-restclients-hyperdrive==1.22.0
azureml-widgets==1.22.0
backcall==0.2.0
backports.functools-lru-cache==1.6.1
backports.tempfile==1.0
backports.weakref==1.0.post1
bcrypt==3.2.0
beautifulsoup4==4.9.3
bleach==3.3.0
blessings==1.7
blis==0.2.4
bokeh==2.2.3
boto==2.49.0
boto3==1.15.18
botocore==1.18.18
Bottleneck==1.3.2
cached-property==1.5.2
cachetools==4.2.1
certifi==2020.12.5
cffi==1.14.4
chardet==4.0.0
click==7.1.2
cloudpickle==1.6.0
colorama==0.4.4
colorful==0.5.4
configparser==3.7.4
contextlib2==0.6.0.post1
contextvars==2.4
convertdate @ file:///home/conda/feedstock_root/build_artifacts/convertdate_1605102623033/work
coremltools @ git+https://github.com/apple/coremltools@13c064ed99ab1da7abea0196e4ddf663ede48aad
cryptography==3.3.1
cycler==0.10.0
cymem==2.0.5
Cython @ file:///tmp/build/80754af9/cython_1594831565616/work
databricks-cli==0.14.1
dataclasses==0.8
decorator==4.4.2
defusedxml==0.6.0
dill==0.3.3
distro==1.5.0
dm-tree==0.1.5
docker==4.4.1
dotnetcore2==2.1.20
en-core-web-sm @ https://aka.ms/automl-resources/packages/en_core_web_sm-2.1.0.tar.gz
encrypted-inference==0.9
entrypoints==0.3
fabric==2.6.0
fairlearn==0.4.6
fastai==1.0.61
fastprogress==1.0.0
fbprophet==0.5
filelock==3.0.12
fire==0.4.0
flake8==3.8.4
Flask==1.0.3
Flask-Cors==3.0.10
flatbuffers==1.12
fusepy==3.0.1
future==0.18.2
gast==0.3.3
gensim==3.8.3
gevent==21.1.2
gitdb==4.0.5
GitPython==3.1.13
google-api-core==1.26.0
google-auth==1.26.1
google-auth-oauthlib==0.4.2
google-pasta==0.2.0
googleapis-common-protos==1.52.0
gpustat==0.6.0
greenlet==1.0.0
grpcio==1.32.0
gunicorn==19.9.0
gym==0.18.0
h5py==2.10.0
hiredis==1.1.0
holidays==0.9.11
horovod==0.19.1
humanfriendly==9.1
idna==2.10
idna-ssl==1.1.0
imageio==2.9.0
immutables==0.15
importlib-metadata==3.4.0
imutils==0.5.4
interpret-community==0.16.0
interpret-core==0.2.1
invoke==1.5.0
ipykernel==5.4.3
ipython==7.16.1
ipython-genutils==0.2.0
ipywidgets==7.6.3
isodate==0.6.0
isort==5.7.0
itsdangerous==1.1.0
javaproperties==0.5.1
jedi==0.18.0
jeepney==0.6.0
Jinja2==2.11.2
jmespath==0.10.0
joblib==0.14.1
jsmin==2.2.2
json-logging-py==0.2
json5==0.9.5
jsondiff==1.2.0
jsonpickle==1.5.1
jsonschema==3.2.0
jupyter==1.0.0
jupyter-client==6.1.11
jupyter-console==6.2.0
jupyter-core==4.7.1
jupyter-server-proxy==1.6.0
jupyterlab==2.1.4
jupyterlab-nvdashboard==0.4.0
jupyterlab-server==1.2.0
jupyterlab-widgets==1.0.0
jupytext==1.6.0
Keras==2.3.1
Keras-Applications==1.0.8
Keras-Preprocessing==1.1.2
keras-tuner==1.0.2
keras2onnx==1.6.0
kiwisolver==1.3.1
knack==0.8.0rc2
lazy-object-proxy==1.4.3
liac-arff==2.5.0
lightgbm==2.3.0
lunardate==0.2.0
lz4==3.1.3
Mako==1.1.4
Markdown==3.3.3
markdown-it-py==0.5.8
MarkupSafe==1.1.1
matplotlib==3.2.1
mccabe==0.6.1
mistune==0.8.4
mkl-fft==1.2.0
mkl-random==1.1.0
mkl-service==2.3.0
mlflow==1.13.1
mock==4.0.3
msal==1.8.0
msal-extensions==0.2.2
msgpack==1.0.2
msrest==0.6.21
msrestazure==0.6.4
multidict==5.1.0
murmurhash==1.0.5
nbconvert==5.6.1
nbformat==5.1.2
ndg-httpsclient==0.5.1
nengo==3.1.0
nengo-dl==3.4.0
nengo-ocl==2.1.0
networkx==2.5
nimbusml==1.8.0
notebook==6.2.0
numexpr==2.7.2
numpy==1.19.5
nvidia-ml-py3==7.352.0
oauthlib==3.1.0
olefile==0.46
onnx==1.7.0
onnxconverter-common==1.6.0
onnxmltools==1.4.1
onnxruntime==1.3.0
opencensus==0.7.12
opencensus-context==0.1.2
opencv-python==4.5.1.48
opencv-python-headless==4.3.0.36
opt-einsum==3.3.0
packaging==20.9
pandas==0.25.3
pandas-ml==0.6.1
pandocfilters==1.4.3
papermill==1.2.1
paramiko==2.7.2
parso==0.8.1
pathlib2==2.3.5
pathspec==0.8.1
patsy==0.5.1
pexpect==4.8.0
pickleshare==0.7.5
Pillow==8.1.0
pkginfo==1.7.0
plac==0.9.6
pluggy==0.13.1
pmdarima==1.1.1
portalocker==1.7.1
preshed==2.0.1
progressbar2==3.53.1
prometheus-client==0.9.0
prometheus-flask-exporter==0.18.1
prompt-toolkit==3.0.14
protobuf==3.14.0
psutil==5.8.0
psycopg2==2.8.4
ptyprocess==0.7.0
py-cpuinfo==5.0.0
py-spy==0.3.4
py4j==0.10.9
pyarrow==1.0.1
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycocotools==2.0.0
pycodestyle==2.6.0
pycparser @ file:///tmp/build/80754af9/pycparser_1594388511720/work
pydocstyle==5.1.1
pyflakes==2.2.0
pyglet==1.5.0
Pygments @ file:///home/conda/feedstock_root/build_artifacts/pygments_1610742651773/work
PyJWT==1.7.1
pylint==2.6.0
PyMeeus @ file:///home/conda/feedstock_root/build_artifacts/pymeeus_1589222711601/work
PyNaCl==1.4.0
pynvml==8.0.4
pyodbc===4.0.0-unsupported
pyopencl==2021.1.2
pyOpenSSL==20.0.1
pyparsing==2.4.7
pyrsistent==0.17.3
pyspark==3.0.1
pystan==2.19.0.0
python-dateutil==2.8.1
python-editor==1.0.4
python-jsonrpc-server==0.4.0
python-language-server==0.35.0
python-utils==2.5.6
pytools==2021.1
pytorch-transformers==1.0.0
pytz==2021.1
PyWavelets==1.1.1
PyYAML==5.4.1
pyzmq==22.0.2
qtconsole==5.0.2
QtPy==1.9.0
QuantLib==1.21
querystring-parser==1.2.4
ray==1.2.0
redis==3.5.3
regex==2020.11.13
requests==2.25.1
requests-oauthlib==1.3.0
rope==0.18.0
rsa==4.7
ruamel.yaml==0.16.12
ruamel.yaml.clib==0.2.2
s3transfer==0.3.4
sacremoses==0.0.43
scikit-image==0.17.2
scikit-learn==0.22.2.post1
scipy==1.4.1
scp==0.13.3
scrapbook==0.5.0
SecretStorage==3.3.0
Send2Trash==1.5.0
sentencepiece==0.1.95
setuptools-git==1.2
shap==0.34.0
simpervisor==0.4
sip==4.19.24
six==1.15.0
skl2onnx==1.4.9
sklearn==0.0
sklearn-pandas==1.7.0
smart-open==1.9.0
smmap==3.0.5
snowballstemmer==2.1.0
soupsieve==2.2
spacy==2.1.8
SQLAlchemy==1.3.23
sqlparse==0.4.1
srsly==1.0.5
sshtunnel==0.1.5
statsmodels==0.10.2
tables==3.6.1
tabulate==0.8.7
tenacity==6.3.1
tensorboard==2.4.1
tensorboard-plugin-wit==1.8.0
tensorboardX==2.1
tensorflow==2.4.1
tensorflow-estimator==2.4.0
tensorflow-gpu==2.4.1
termcolor==1.1.0
terminado==0.9.2
terminaltables==3.1.0
testpath==0.4.4
textwrap3==0.9.2
thinc==7.0.8
threadpoolctl @ file:///tmp/tmp9twdgx9k/threadpoolctl-2.1.0-py3-none-any.whl
tifffile==2020.9.3
tokenizers==0.10.1
toml==0.10.2
torch==1.6.0
torchvision==0.7.0
tornado==6.1
tqdm==4.56.0
traitlets==4.3.3
transformers==4.3.2
typed-ast==1.4.2
typing-extensions==3.7.4.3
ujson==4.0.2
urllib3==1.25.11
vsts==0.1.25
vsts-cd-manager==1.0.2
waitress==1.4.4
wasabi==0.8.2
wcwidth @ file:///home/conda/feedstock_root/build_artifacts/wcwidth_1600965781394/work
webencodings==0.5.1
websocket-client==0.57.0
websockets==8.1
Werkzeug==1.0.1
widgetsnbextension==3.5.1
wrapt==1.12.1
xgboost==1.3.3
xmltodict==0.12.0
yapf==0.30.0
yarl==1.6.3
zipp==3.4.0
zope.event==4.5.0
zope.interface==5.2.0


0000:00:08.0 VGA compatible controller: Microsoft Corporation Hyper-V virtual VGA (prog-if 00 [VGA controller])

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.32.03    Driver Version: 460.32.03    CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla K80           On   | 00000001:00:00.0 Off |                    0 |
| N/A   41C    P0    56W / 149W |      0MiB / 11441MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

hadarcd · March 2, 2021, 9:09am

I created a new machine and install tensorflow==2.2.0

now got this error:

AttributeError                            Traceback (most recent call last)
<ipython-input-1-700eed500250> in <module>
----> 1 import nengo_dl

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/nengo_dl/__init__.py in <module>
     40 
     41 # need to explicitly import these to trigger the builder registration
---> 42 from nengo_dl import (
     43     op_builders,
     44     neuron_builders,

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/nengo_dl/op_builders.py in <module>
      8 
      9 import numpy as np
---> 10 import tensorflow as tf
     11 from nengo.builder.operator import (
     12     Copy,

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/tensorflow/__init__.py in <module>
     39 import sys as _sys
     40 
---> 41 from tensorflow.python.tools import module_util as _module_util
     42 from tensorflow.python.util.lazy_loader import LazyLoader as _LazyLoader
     43 

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/tensorflow/python/__init__.py in <module>
     82 from tensorflow.python import data
     83 from tensorflow.python import distribute
---> 84 from tensorflow.python import keras
     85 from tensorflow.python.feature_column import feature_column_lib as feature_column
     86 from tensorflow.python.layers import layers

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/tensorflow/python/keras/__init__.py in <module>
     25 
     26 # See b/110718070#comment18 for more details about this import.
---> 27 from tensorflow.python.keras import models
     28 
     29 from tensorflow.python.keras.engine.input_layer import Input

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/tensorflow/python/keras/models.py in <module>
     22 from tensorflow.python.framework import ops
     23 from tensorflow.python.keras import backend as K
---> 24 from tensorflow.python.keras import metrics as metrics_module
     25 from tensorflow.python.keras import optimizers
     26 from tensorflow.python.keras.engine import network

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/tensorflow/python/keras/metrics.py in <module>
     35 from tensorflow.python.framework import tensor_spec
     36 from tensorflow.python.keras import backend as K
---> 37 from tensorflow.python.keras.engine import base_layer
     38 from tensorflow.python.keras.engine import base_layer_utils
     39 from tensorflow.python.keras.losses import binary_crossentropy

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py in <module>
     49 from tensorflow.python.keras import backend
     50 from tensorflow.python.keras import constraints
---> 51 from tensorflow.python.keras import initializers
     52 from tensorflow.python.keras import regularizers
     53 from tensorflow.python.keras.engine import base_layer_utils

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/tensorflow/python/keras/initializers/__init__.py in <module>
    125 # from ALL_OBJECTS. We make no guarantees as to whether these objects will
    126 # using their correct version.
--> 127 populate_deserializable_objects()
    128 globals().update(LOCAL.ALL_OBJECTS)
    129 

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/tensorflow/python/keras/initializers/__init__.py in populate_deserializable_objects()
     83     v2_objs = {}
     84     base_cls = initializers_v2.Initializer
---> 85     generic_utils.populate_dict_with_module_objects(
     86         v2_objs,
     87         [initializers_v2],

AttributeError: module 'tensorflow.python.keras.utils.generic_utils' has no attribute 'populate_dict_with_module_objects'

xchoo · March 2, 2021, 3:09pm

If you look at the error trace log, you’ll notice that the error you are encountering is from the statement:

import tensorflow as tf

from within the NengoDL package. This indicates that the error you are experiencing is related to the TensorFlow installation in particular. Thus, my recommendation is to create a new Python environment (in Conda) and install only TensorFlow and test to see if that works.

Here are the docs for how to manage Conda environments: Managing environments — conda 4.9.2.post29+86b4aea5 documentation
The cool thing about Conda is that you can create multiple Python environments on the same machine without having to create an entirely new VM to test things out.

For brevity, I’ll list the steps below on how to install and test to make sure TensorFlow is working on your machines. After this has been done successfully, you can then proceed to install NengoDL and other packages.

Create a new python environment in Conda. I notice you are using Python 3.6 in your example above, so I’ve replicated the settings below:

conda create -n test python=3.6

Install the tensorflow-gpu package. I noticed you are using the 2.2.0 version of TensorFlow so here are the instructions to install that. If you want to use the latest version of TensorFlow (v 2.4.1) the installation instructions are slightly different, so do let me know.

conda activate test
pip install tensorflow-gpu==2.2.0

Test your TensorFlow installation:

python

# what follows below is in the Python interpreter
import tensorflow as tf
print(tf.config.list_physical_devices('GPU'))

If you see an output like the following, then TensorFlow has been installed correctly and you can then proceed to install NengoDL (through whatever means – i.e., pip or git – you desire)

[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]