Nengo GUI Image Display

Visualization
#1

Hello,

I am trying to follow the Classifying MNIST with Nengo example (https://www.nengo.ai/nengo-extras/vision-examples.html) and I was expecting the nengo-gui to show the input image from the input node where the images are presented to the model. However, the images do not appear in the GUI. Is there something I am missing?

Thanks!
David

1 Like
#2

Hello, have you been able to display input images?

#3

Hello!

Yep. It took a while to play around with. I ended up using the pre-built function in the nengo_extras package called image_display_function which has a snippet here.

Here’s some code to get started.

from PIL import Image
import requests
from io import BytesIO
import numpy as np

import nengo
from nengo_extras.gui import image_display_function

# fetch image
url = 'https://upload.wikimedia.org/wikipedia/en/7/7d/Lenna_%28test_image%29.png'
response = requests.get(url)
img = Image.open(BytesIO(response.content))

# "image_display_function" expects images 
# to be in (channels, x, y) and scaled between 0 and 1
img = np.transpose(img, (2, 0, 1)) / 255.
image_shape = np.shape(img)

model = nengo.Network()
with model:
    # flatten image to 1D array
    u = nengo.Node(lambda t: np.ndarray.flatten(img))
    
    # display image using "image_display_function"
    display_f = image_display_function(image_shape)
    display_node = nengo.Node(display_f, size_in=u.size_out)
    nengo.Connection(u, display_node, synapse=None)
#4

To kinda further expand this, the way images are rendered is by adding the _nengo_html_ attribute to the calling node function/class. The _nengo_html_ should return a <svg>...</svg> image to be rendered in HTML. This is what the image_display_function does underneath by converting images into HTML representable svg tag (seen here).

(Taken/modified from some of the nengo_gui examples)

import nengo
import numpy as np

def arm_function(t, angles):
    len0 = 50
    len1 = 30
    x1 = 50
    y1 = 100
    x2 = x1 + len0 * np.sin(angles[0])
    y2 = y1 - len0 * np.cos(angles[0])
    x3 = x2 + len1 * np.sin(angles[0] + angles[1])
    y3 = y2 - len1 * np.cos(angles[0] + angles[1])
    
    arm_function._nengo_html_ = '''
    <svg width="100%" height="100%" viewbox="0 0 100 100">
        <line x1="{x1}" y1="{y1}" x2="{x2}" y2="{y2}" style="stroke:black"/>
        <line x1="{x2}" y1="{y2}" x2="{x3}" y2="{y3}" style="stroke:black"/>
    </svg>
    '''.format(**locals())
    
model = nengo.Network()
with model:
    stim_angles = nengo.Node([0.3, 0.3])
    arm = nengo.Node(arm_function, size_in=2)
    nengo.Connection(stim_angles, arm)
#5

Hi @DFan

As you discovered, NengoGUI doesn’t natively support the custom inputs. It will display the MNIST input as it it were vectors, but to have it show the MNIST input as an image, you will have to use the _nengo_html_ functionality.