Nengo,EEG ,fMRI and the value of modeling

I have been struggling with ways to validate Nengo models. As I see it, one of the great advantages of Nengo is that it affords a way to make the leap from spiking neurons to semantics and human behavior. What I don’t understand is why when investigators show that the model explains human behavior and also less complex measures such as EEG or fMRI.

Wouldn’t it be supportive of the validity of the models if they predicted EEG or fMRI signals? Certainly, the online video cartoon associated with the 2012 Science paper is fMRI-like.

Finally, there are several values to models, but the biggest test is whether they make predictions that require doing experiments that otherwise were not thought of. Has anyone compiled a list of predictions and experiments resulting from Nengo models?


Howard C

Hi Howard,

Good question. I’d definitely love to predict EEG and fMRI signals, and you’re entirely right that this would be a wonderful way to validate Nengo models. Most of the time when we’ve been validating Nengo models, we have been focusing on behavioural data (including reaction times) and on spiking activity, but we haven’t done much with anything else. There is one fMRI prediction in the 2013 book (based on the Tower of Hanoi model), and we are currently working on an MEG prediction model and have just started an EEG prediction (both unpublished yet).

The main reason, I think, that we haven’t done more of this is that Nengo/NEF models things at the level of spikes, and it’s unclear exactly how to map that spiking activity to fMRI/EEG/MEG/etc signals. For example, with fMRI, the fMRI signal is indicating the amount of energy consumption in different areas of the brain (assuming oxygenated blood flow correlates to energy consumption). This does not directly map onto spiking activity. For example, if area A has an inhibitory connection to area B, and if area A is firing a lot, then area B will not be spiking at all, but there will be a lot of energy consumption in area B as the inhibitory neurotransmitter that is being used has to get reabsorbed. Indeed, in the fMRI work in the 2013 book, we found the best fit to the fMRI data was if we assumed 80% of the fMRI signal was due to this neurotransmitter re-absorption (and the rest due to spiking activity). But that was fairly ad-hoc and more of a proof-of-concept. I think in order to strongly demonstrate fMRI as a validation method, we’d want a clear and principled method for figuring out the biological energy consumption requirements of a Nengo model. I’d love to sort that out, but haven’t made the time to do it yet.

A similar complication is true for EEG. The EEG signal should be generated by long-distance oriented connections, rather than by short internal recurrent connections. In many Nengo models, there’s isn’t that sort of large-scale modelling happening. However, my personal belief is that, in our SPA models, actions that cause the routing of information from one brain area to another would be good candidates for having visible EEG signals, as these will generally cause a lot of activity along axons that are fairly oriented. So my hope would be to find particular connections in a model that would be fairly long-distance, and when those neurons start firing (i.e. when they are released from inhibition via the b.g./thalamus) then that should correspond to an ERP of some sort. That’s my hope, in any case, and we’re just starting up a collaboration project that would start to explore that, but there are a lot of unknowns involved.

Right now, though, I do hope that LFP and MEG predictions might be a bit easier, as there’s reason to believe that those are a little bit more closely tied to just the spike data. Indeed, there’s a matlab toolkit for converting spike data to LFP data. We’re also working on an MEG prediction in work that should hopefully be published soon.

In any case, that’s the current state of validating nengo models using these more large-scale neural measures, rather than spikes. If there’s a particular project in there that really jumps out at you, I’d love to help out how I can; as I said I think you’re entirely right that this would be an excellent step forward for validation.


Terry – thanks for both emails


I would just like to ask whether there is any update on these issues. I would be particularly interested on whether EEG could be somehow used for online training a Nengo model. There is a paper I recently read by Kasabov, N., & Capecci, E. (2015) where they transfrom raw EEG into spike trains and then map those on to an SNN.

Hi @sinandrei, welcome to the forum!

I’m not aware of any new work on mapping between EEG experiments and Nengo, but working with spike trains is at the heart of Nengo, so would make the interaction much easier. It seems like a promising line of research if anyone wants to undertake it; there are definitely many people on the forum that could help with building the model once it’s started!

There’s a paper that is somewhat related to this called ‘Where Does EEG Come From and What Does It Mean?’ that I’d highly recommend: Thats the NCBI link t the abstract, I think the full text is free and roughly five pages. It’s an opinion piece with references as opposed to an experiment but is very informative; basically, no one really knows whats going on.

Hello dear friends,

does it mean that in your opinion today, thousands of, for instance, cochlear implants are realized without a priori knowledge of the representations the output of the cochlear prothese will have to meet in the auditory cortex, and is there also a dito blind matching between the prothese and the auditory pathway, (afferent and efferent information streams)?

A working neural implant seems a miracle in that context …

Some suggestions in reaction on tcstewar:

  1. it’s clear that mapping neural activity, in the NEF & BAB, ( - build a brain -), sense, to fMRI representation isn’t an evident transformation, (current to energy dissipation);

  2. mapping neural activity, NEF-BAB sense, (current intensity), to extracellular voltage, (EEG-measurements), or intracellular voltage, (more NEF-BAB like), as transformations in the same group, (electricity), will be easier;

  3. perhaps the BAER-procedure, (Brainstem Auditory Evoked Response - cfr eg. Clinical Neuroanatomy and Neuroscience - M J Turlough et al., Saunders Elsevier), can be modelled NEF-BAB like. Somatosensory evoked potentials, and motor evoked potentials, complete the chapter. So an auditory model is only one out of the possible models, confrontable to real life data, (because clinical practice);

  4. finally, it seems valuable to remind, during the projection or the mapping of repitiive unitary signals from one NE to another, that those maps are topographic, (cfr eg tonotopic for auditory pathway). This makes the links between maps ordened, isomorphic-like. It seems in the auditory system for instance, that “the melody that goes in, (our ears), is also the melody that comes out in the auditory cortex”, but played by another instrument.


I definitely agree that mapping nengo models to extracellular voltage would be easier – but I’m thinking more LFP than EEG, because with EEG you also have to deal with a) the low spatial resolution and b) the filtering caused by the skull. That said, EEG should be possible, if there’s an explicit story about how to deal with that. So I’d be very interested in any work that pushes things in that direction! The MEG project is still ongoing, so we’ll see how that goes…

Also, I completely agree about the reminder that representations are topographic. Again, that hasn’t been something we’ve focused on much, but you can do it in nengo by either a) generating your own encoders using whatever distribution/organization you would like, or b) generating a nengo model and then after running it sort the neurons based on similarity (either similarity of encoders or similarity of spiking activity). Neither method fundamentally changes anything about the model itself in terms of behaviour.

Trying to help “tcstewar” with the problems in reaction to “Howard’s” mail, framed my reflexions.
Unfortunately, I dont have knowlege about a concrete reference on LP conversion to intra-neural activity. The more I read about LP, the more parameters appear.

Avoid at this stage that complexity is a first justification to explore the BAER-procedure, (or CAER - cortex auditive evoked response), or similar procedure in the somatic or motor environment.
Second, a nengo simulation, (eg. a communication canal), of the auditory pathway, (cochlear nerve to the primary auditory pathway), with the BAER-input, would give, (in the nengo application), at the measuring points, the nengo-state vector. The procedure being used in the clinical practice, those nengo-results can be correlated to the BAER-outputs for healthy persons, and reveal the correlation you are looking for.
Third, this is a start; chap. 31, Evoked Potentials, of the book I referenced, illustrates comparable uses of invoked potentials, in case you dont want to work with the auditory model.

Of course this way of doing is not as fundamental, but … where would we hve been if Newton started with OM?


perhaps de following article can help you:“Linear and Nonlinear Relationships between Visual Stimuli, EEG and BOLD fMRI Signals, by ZhongMing Liu et al.; Neuroimage 2010 Apr. 15 50(3) 1054-1066”