Non-linear Transformation in NEF

Hi, sorry for posting many theoretical questions, but I really hope to understand the theory behind Nengo better.

I think I got myself confused about how transformation (Principle 2 of NEF) works for non-linear functions. I was following the derivation presented in Nengo summer school lecture 3: NEF Principle II with Nengo - YouTube (approximately 43min in the video) which learns \textbf{y}=\textbf{x}. The derivation for the weight w_{ij} between 2 neuron ensembles a_i representing \textbf{x} and b_j representing \textbf{y} is:

\begin{equation} \begin{split} b_j &= G_j(\alpha_j\textbf{e}_j\hat{\textbf{x}} + J_j^{bias})\\ &= G_j(\alpha_j\textbf{e}_j\sum_i{a_i d_i} + J_j^{bias})\\ & = G_j(\sum_i{w_{ij}a_i} + J_j^{bias}) \end{split} \end{equation}

This gives us w_{ij} = \sum_i{\alpha_j\textbf{e}_jd_i}.

This is fine so far, but once I tried to reason about \textbf{y} = \textbf{x}^2, I find that I am stuck at b_j = G_j(\alpha_j\textbf{e}_j(\sum_i{a_i d_i})^2 + J_j^{bias}), where I can no longer represent the activations of the second ensemble as a function of linear transformations of the activities and decoders of the first ensemble.

So I guess my question is how are the weights solved for nonlinear transformations such as \textbf{y} = \textbf{x}^2?

I think I inadvertently answered your question in your other forum thread. :grinning:

To summarize, for y = x^2, the decoders d are solved in such a way such that \sum_{i} a_i d_i approximates the function x^2.

More precisely, the decoders d = \Gamma^{-1}\Upsilon, where \Gamma = \mathbf{A}^T \cdot \mathbf{A} (where \mathbf{A} is the matrix containing the a_i(\mathbf{x}) for all sample points \mathbf{x} and for all neurons i) and \Upsilon = f(\mathbf{x}) \cdot \mathbf{A} (i.e., the dot product of the function f applied to all sample points \mathbf{x} with the matrix \mathbf{A}). For the case of y = x^2, f(\mathbf{x}) = x^2. This is on page 68 of the NEF book.