Cool! Thank you very much.
Sorry to keep adding to this thread, but I seem to be stuck once again trying to run the Optimizing a Spiking Neural Network.
I’ve switched from my laptop to my work desktop that has a NIVDIA Quadro RTX 4000 installed. When I once again run the optimizing a spiking neural network example it’s throwing the same “No GPU support detected.” error I was getting on my less-than-robust laptop (see screenshot).
I thought I might have solved the issue by following the instructions provided earlier in this thread, but I seem to be back to the same problem. From what I can gather, my NVIDIA GPU is based on the Turing GPU architecture. Could that be my problem?
I am out of ideas and stuck, so I thought it was time once again to ask for help.
Thanks.
Looks like I’m getting the same GPU error when I attempt to run the Legendre Memory Units in NengoDL example.
Could this possibly be caused by the NIVDIA Quadro RTX 4000 GPU, similar to what is happening to me in the Optimizing a Spiking Neural Network example above?
Hello @toddswri, not sure how much of help I will be, but you should mention the output of nvidia-smi on your terminal. In case the GPU has been properly configured for the usage by TF (or Nengo-DL), the output should be similar to here.
If not, then you need to properly configure it first.
The error you posted is a TensorFlow error telling you that it is unable to detect an appropriate GPU to use. To figure out what is causing the issue:
- Check that the Nvidia drivers are working. As @zerone mentioned, you can do so by using the
nvidia-smiin a terminal window. - Check that the GPU is visible to TensorFlow. To do this, start a Python instance (in the environment you want to run your NengoDL model in), then run the following code:
import tensorflow as tf
print(tf.config.list_physical_devices('GPU'))
TensorFlow will print out a bunch of statements (which you should include in your reply) indicating the status of CUDA libraries that are being loaded. If TensorFlow can detect your GPU, you should see a final output like:
[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]
I checked the Nvidia support page, and it seems like the Quadro RTX 4000 does meet the CUDA requirements for the latest CUDA toolkit release, so it should be usable without issue.
Some additional information that would be helpful to have:
- What operating system are you using?
- What packages are installed in your environment. If you are using Conda, run the command
conda listin your activated environment. Otherwise, runpip freeze.
My last thought is that your laptop might be using Nvidia optimus. Nvidia optimus allows the OS to switch between an integrated GPU (i.e., integrated with the CPU package), and the dedicated Quadro chip. If this is the case, Python or TensorFlow might not be picking up your GPU because Nvidia Optimus has selected the integrated GPU to run with your Python environment. If this is the case, it should be apparent when you try to run nvidia-smi.
Thank you for the suggestions and help.
Here’s a screenshot of my NVIDIA drivers using nvidia-smi:
And here is my output when I check to see if my GPU is visible:
import tensorflow as tf
print(tf.config.list_physical_devices(‘GPU’))
It appears TensorFlow is not seeing the GPU.
The desktop system I’m using is running Windows 10. Here is a file listing my installed packages in my Python 3.7 environment.
Python_Installed_Packages.pdf (236.4 KB)
I’m embarrassed to say that I have no experience regarding how to configure the NVIDIA card and will need to ask for further instructions.
Thanks again.
I see that you have tensorflow-gpu 2.4.1 from your Python_Installed_Packages.pdf. My suggestion will be to first get your TF detect your GPU. Once it does, Nengo-DL shouldn’t have any issues in using it. I don’t know much about whether TF 2.4.1 is supported by your current CUDA version 11.0 (as can be seen in the screenshot you attached) or not. So if you are not bound to TF-2.4.1, then may be you can install TF 2.4.0 and check. Here’s one link to help you set up GPU for TF-2.4.0.
You can find more info about Nengo-DL and TF installation here. My environment’s info is mentioned below.
>>> import tensorflow as tf
>>> tf.__version__
'2.2.0'
>>> print(tf.config.list_physical_devices('GPU'))
2021-04-30 15:51:06.032334: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2021-04-30 15:51:06.075846: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties:
pciBusID: 0000:04:00.0 name: Tesla P100-PCIE-12GB computeCapability: 6.0
coreClock: 1.3285GHz coreCount: 56 deviceMemorySize: 11.91GiB deviceMemoryBandwidth: 511.41GiB/s
2021-04-30 15:51:06.085741: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2021-04-30 15:51:06.189458: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2021-04-30 15:51:06.266299: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2021-04-30 15:51:06.373993: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2021-04-30 15:51:06.440953: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2021-04-30 15:51:06.491673: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2021-04-30 15:51:06.601603: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2021-04-30 15:51:06.603795: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]
>>> import nengo_dl
>>> nengo_dl.__version__
'3.4.0'
You may want to use conda to create separate environments and install desired versions of TF, Nengo-DL, etc. libraries. Honestly, setting up TF and GPU is just a Google Search away.
EDIT: Found another related issue on this forum. Might be of some help. You may find some more resolved issues similar to this forum. You can use search bar on the top right corner (besides the burger menu) to search for similar issues.
1 Like
Hi @toddswri,
I’m not entirely sure what’s going on, but I think there may be a mismatch between the TensorFlow version and CUDA toolkit version you have installed in your environment. Fixing an existing Conda environment is hit or miss, so I recommend creating a new environment from scratch. Here are the instructions to get just TensorFlow and GPU support installed in your environment (you can install everything else once you test that the GPU is detected):
Create a new Conda environment
Create a new Conda environment (replace <env_name> with a name you like). Since you were using Python 3.7, I’ve replicated it here:
conda create -n <env_name> python=3.7
conda activate <env_name>
Install CUDA toolkit
conda install cudatoolkit
You should see it install v11.0. If it doesn’t, let me know!
The following NEW packages will be INSTALLED:
cudatoolkit pkgs/main/win-64::cudatoolkit-11.0.221-h74a9793_0
Install cuDNN
conda install -c conda-forge cudnn
You should see it install v8.1 (and some other stuff).
The following NEW packages will be INSTALLED:
cudnn conda-forge/win-64::cudnn-8.1.0.77-h3e0f4f4_0
Install TensorFlow
Note the use of pip instead of conda here!
pip install tensorflow
You should see it install v2.4.1, and a whole bunch of other packages:
Collecting tensorflow
Using cached tensorflow-2.4.1-cp37-cp37m-win_amd64.whl (370.7 MB)
Test TensorFlow with GPU
Now comes time to test that TensorFlow can pick up your GPU. Start Python from your Conda terminal:
python
Then do the list_physical_devices test:
import tensorflow as tf
print(tf.config.list_physical_devices("GPU"))
When I did these two steps in a fresh Conda environment on my Windows machine, I see:
>>> import tensorflow as tf
2021-04-30 16:09:19.465296: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll
>>> print(tf.config.list_physical_devices("GPU"))
2021-04-30 16:09:43.351770: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-04-30 16:09:43.352497: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library nvcuda.dll
2021-04-30 16:09:43.374485: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:2d:00.0 name: NVIDIA GeForce RTX 3090 computeCapability: 8.6
coreClock: 1.74GHz coreCount: 82 deviceMemorySize: 24.00GiB deviceMemoryBandwidth: 871.81GiB/s
2021-04-30 16:09:43.374629: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll
2021-04-30 16:09:43.402428: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublas64_11.dll
2021-04-30 16:09:43.402503: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublasLt64_11.dll
2021-04-30 16:09:43.421391: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cufft64_10.dll
2021-04-30 16:09:43.423192: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library curand64_10.dll
2021-04-30 16:09:43.673219: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusolver64_10.dll
2021-04-30 16:09:43.682491: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusparse64_11.dll
2021-04-30 16:09:43.683142: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudnn64_8.dll
2021-04-30 16:09:43.683281: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]
If the test successfully passes, you can then continue to install the other packages (i.e., Nengo, NengoDL, Jupyter, etc.)
Let me know if these instructions work for you!
Note: These instructions work with the latest version of NengoDL. If you want to use NengoGUI, it doesn’t support the latest version of NengoDL, so the instructions will change slightly (see the link that @zerone posted)
Sorry it’s taken me so long to reply. My mom went in the hospital this week and my schedule has been erratic to say the least.
The instructions you provided worked and it now appears that the GPU is being seen by TensorFlow in the newly created environment. I attached a screenshot of the output returned by the “print(tf.config.list_physical_devices(“GPU”))” command that seems to match what you posted above.
The only last question I now have has to do with the pip error message that was thrown about nengo-dl 3.4.0 requiring nengo 3.0.0 which I have not yet installed into this new environment. What version of nengo should I install so as not to have any incompatibilities with nengo-gui 0.4.7 or nengo-ocl 2.0.0? Should it be nengo 3.0.0? This question goes back to the issue I was having back on April 21st (16 days ago) when you told me to install nengo-gui with nengo-ocl 2.0.0.
As always, thanks again for your help. Your instructions really helps our team a lot. Have a great weekend!
-Todd
Hi @toddswri,
Getting NengoOCL to work alongside NengoDL and NengoGUI is possible, and here are the installation instructions to do so. I’m going to start the instructions by creating a new environment because I don’t know what state your existing environment is in (so the easiest is to start with a blank slate):
Create a new Conda environment
Create a new Conda environment (replace <env_name> with a name you like). Since you were using Python 3.7, I’ve replicated it here:
conda create -n <env_name> python=3.7
conda activate <env_name>
Install the various GPU libraries
conda install cudatoolkit
conda install -c conda-forge cudnn pyopencl
Along with some other packages, the following versions for cudatoolkit, cudnn and pyopencl were installed for me:
cudatoolkit conda-forge/win-64::cudatoolkit-11.2.2-h933977f_8
cudnn conda-forge/win-64::cudnn-8.1.0.77-h3e0f4f4_0
pyopencl conda-forge/win-64::pyopencl-2021.1.6-py37hb605e8c_0
Install TensorFlow
Note the use of pip instead of conda here!
pip install tensorflow
You should see it install v2.4.1, and a whole bunch of other packages:
Collecting tensorflow
Using cached tensorflow-2.4.1-cp37-cp37m-win_amd64.whl (370.7 MB)
Install Nengo and related Nengo packages
You can now install Nengo and such. The order doesn’t matter, but note that the NengOCL version has been fixed to 2.0.0.
pip install nengo nengo-gui nengo-dl "nengo-ocl==2.0.0"
For my environment, it installed these versions (which should all work together). You might see two listings for nengo, but the second one should be v3.0.0.
Collecting nengo-dl
Using cached nengo_dl-3.4.0-py3-none-any.whl
Collecting nengo-gui
Using cached nengo_gui-0.4.7-py3-none-any.whl (843 kB)
Collecting nengo-ocl==2.0.0
Using cached nengo_ocl-2.0.0-py3-none-any.whl (77 kB)
Collecting nengo
Using cached nengo-3.0.0-py3-none-any.whl (391 kB)
Testing the GPU
You can now proceed to test the GPU in Python. For TensorFlow (as before) do:
import tensorflow as tf
print(tf.config.list_physical_devices("GPU"))
For PyOpenCL (which NengoOCL uses), do:
import pyopencl as cl
print(cl.get_platforms())
You should see a list of available platforms to use. Sometimes it detects your CPU as an available platform to run OCL code on, so it will show up in this list. For my machine, only one entry shows up in the list:
[<pyopencl.Platform 'NVIDIA CUDA' at 0x19b412798c0>]
Next, you can print the available devices for the platform you want to use. For my machine, since there is only one available OCL platform, I use the list index 0, but your machine might be different:
print(cl.get_platforms()[0].get_devices())
With that, I see this:
[<pyopencl.Device 'NVIDIA GeForce RTX 3090' on 'NVIDIA CUDA' at 0x19b412799b0>]
indicating I have an RTX 3090 visible to PyOpenCL to use.
Okay. I followed your last post to get my NengoOCL working along with NengoDL and NengoGUI. Everything seemed go well until I got to the part where I install Tensorflow. If I’m reading the output correctly there seems to be a couple of libraries that did not load properly during the process:
- tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
- Could not load dynamic library ‘cusolver64_10.dll’
- tensorflow/core/common_runtime/gpu/gpu_device.cc:1757] Cannot dlopen some GPU libraries.
Here’s my screenshot of the process:
As you can see, the “print(tf.config.list_physical_devices(“GPU”))” output is an empty bracket [].
Looks like it is recognizing my NVIDIA Quadro RTX 4000 GPU when listing my tf.config.list_physical_devices. But then of course the “import pyopencl as cl” command did not work.
I’m missing some .dll CUDA files, correct? The output from the tensorflow import command tells me to look at https://www.tensorflow.org/install/gpu and follow the guide to setup and download the required libraries. Within that guide it directs me to the the CUDA® install guide for Windows.
So now my question is should I follow that advice. Or is there something else you would recommend? I will need to get elevated permission via IT if I run either of the CUDA installers
As always, thanks for your continued help.
Hmm. That’s really strange, because if you look at the output 2 posts ago, it could load that dll fine.
So… Let’s modify the instructions a bit.
Using the instructions in my previous post, to install the various GPU libraries, do this:
conda install cudatoolkit
conda install -c conda-forge cudnn pyopencl
instead of this:
conda install -c conda-forge cudnn cudatoolkit pyopencl
And then install the rest of the package as per my previous instructions.
Let’s see if this will work for you. 
Well, I thought I finally had it working with your help of course, but now I’m stuck on the pyopencl package. When I attempt to “import the pyopencl as cl” it can’t find the package, even though it appears to be installed. So close!
When I check the installed packages pyopencl version 2021.1.6 is installed in my nengo-ocl environment, which is a new one that I built from scratch.
I’m going to apologize in advance for having to ask once again, now with the pyopencl, where you think I’m going wrong? I’m also including a pdf of the installed packages within my environment.
env_py37_ocl-nengo.pdf (56.2 KB)
When I get this nengo-ocl environment finally running I’m also planning to work with the CIFAR-10 classifier with a spiking CNN example in the Nengo Extras section which also uses nengo-ocl. And as I look through the NengoOCL webpage, I now see that getting a working installation of OpenCL is a bit tricky.
Thank you.
Oh! I think your GPU might not support OCL 2.1. In which case, a slightly different installation instruction is needed. You will need to go here and download the wheel file for pyopencl‑2021.2.1+cl12‑cp37‑cp37m‑win_amd64.whl.
Next, create a new environment, then when you install the GPU libraries do:
conda install cudatoolkit
conda install -c conda-forge cudnn
pip install pyopencl‑2021.2.1+cl12‑cp37‑cp37m‑win_amd64.whl
You will need to run that pip install command from where you downloaded the wheel file to.
After this, test to make sure that python can detect pyopencl (by printing the cl.get_platforms()[0].get_devices())). If this test is successful, you can then proceed to install the other packages.
xchoo,
Looks like I’ve finally reached a point where my NengoOCL environment is setup correctly. The only change I made from your last set of instructions in the previous post is that I used the 2021.1.5 version of pyopencl you Sent Dr. H. instead of the 2021.2.1 version in the forum instructions: pyopencl‑2021.1.5+cl12‑cp37‑cp37m‑win_amd64.whl instead of pyopencl‑2021.2.1+cl12‑cp37‑cp37m‑win_amd64.whl
I think this might finally close out this thread since the environment now appears to be running correctly.
Thanks again for all your help.
-toddswri
One remaining issue I’m encountering is regarding getting my environment setup correctly for Nengo Loihi. I had asked about this issue earlier in this thread on April 21st. It has to do with getting the correct versions of nengo, nengo-gui, nengo-ocl, and numpy working together.
You had told me in another thread: Cannot convert a symbolic Tensor to a numpy array I should be using Numpy 1.19.2 or 1.19.5 with NengoDL. So I downgraded my numpy from 1.20.1 to 1.19.5.
The primary Nengo example I’m trying to get working is the CIFAR-10 convolutional network. I have added everything I think I need to my basic cpu environment and am attaching a file for my installed packages.
py37_envirList_24-May-21.pdf (58.8 KB)
It appears I’m back to my circular problem of getting the correct versions of nengo, nengo-ocl, and nengo-gui. After I downgrade my numpy to 1.19.5, I end up getting out of sync with my other packages.
So I’m still trying to figure out if I have installed the proper combination of versions for the following packages:
Nengo 3.0.0
NengoDL 3.4.0
Nengo-Loihi 1.0.0
Nento-OCL 2.0.0
Numpy 1.19.5
As always, thank you for your support.
I spoke to the Nengo devs and we figured out a solution to the package conflicts. The main problem lies with NengoGUI being slightly outdated. We do have a release incoming, but for now, this will address the issues you are having:
pip install nengo nengo-ocl nengo-dl nengo-loihi
pip install git+https://github.com/nengo/nengo_gui.git@spike_raster_fix --no-deps
Note that this requires that you have git installed on your computer, or in your environment. If you do not have git installed, you can do it with: conda install git before you install the NengoGUI package.
Thank you. We will give it a go and report back.
Very exciting, everything seems to be working properly. I’m getting a couple warnings that seem to pretty much match up with the example posted on the Nengo website, but everything else is working smoothly.
I really appreciate the time and effort you all have provided helping me getting this up and running.
Have a great Memorial Day weekend!
-ToddSwRI
That’s great!
1 Like