Scaling up simularity

#7

Rats, I made a mistake in the last posting, I want e to be a mirror image of A (with a negative value)

#8

I have to admit, that I missed your threshold example. However when I tried it, I found it wasn’t tuned for Negative Logic it assumes that the high value is the one I want to threshold for not the negative value. In a tonic inhibition mechanism you want to threshold for the negative value that is largest, not the values that are above the threshold.

Graeme

#9

FYI, you can edit your posts in this forum by clicking the three dots beside “reply” and then clicking the pencil icon.

#10

I’m still not clear on what you’re trying to achieve. Would it be possible to draw a block diagram showing the various inputs and outputs? Alternatively, would you want to hop onto a video call to discuss this more quickly?

#11

Ok, I will try to send you a graphic I made up that describes the basic network. I’d go on a video call, if I could, but my system is very buggy and lags excessively. What I want to do first is expand the input range to at least 30 inputs and still manage to compare the vectors for each input using something like a dot product that will detect similarity. This then will go through an additive inversion to change the similarity detector into a Novelty detector, then I want to use the output from that calculation to gate a copy of the A input so that detection of novelty releases a copy of the A input. It’s a simple circuit just complicated by scaling up to 30 inputs and a problem I am now having trying to get the gating mechanism to propagate. I am embarrassed that it is so simple.

#12

Seanny123 https://forum.nengo.ai/u/seanny123 Reviewer
May 14

I’m still not clear on what you’re trying to achieve. Would it be
possible to draw a block diagram showing the various inputs and
outputs? Alternatively, would you want to hop onto a video call to
discuss this more quickly?

#13

Thanks for drawing the diagram. It really helped me understand what you’re working towards!

What do you mean by `inputs` ? From what I can tell there’s two inputs in your current network. Are input A and input B gating each other in the full network? If you added an input C, would it be added to the combination of A and B? Would it just be gated by the product of A and B?

#14

Actually no, it adds it’s own product so I am comparing multiple entries at once I am uploading a new version of dot_prod.py with 3 inputs to illustrate (I have commented out the hand plotting lines so that it will load in the nengo_gui version of the program. I am not sure I have the best version of this, but it outputs a mirror image of A once it is in the right sector of the graph. I will also output the latest version of figure1 which tracks the outputs of the neurons output of the whole circuit is in E[dot_product scaled up to 3 inputs

#15

#16

#17

As you will notice, the drawing is of the old non-scaled version of the program scaling it up to three inputs still works, but I have yet to make six inputs work. I am also trying to scale it up to six outputs but the simulation takes time between updates.

#18

I think I have been simulating simularity wrongly, I was confused because I knew that there were only two laminae in the belt cortex that I absolutely needed to implement. Nengo doesn’t separate out laminae so I assumed that I needed to implement it in two ensembles, but now I think I need to implement it in three different types of ensembles. The spare type of ensemble is needed because there are two functions of the combined ensemble that don’t fit together in a single ensemble as far as I know. It’s a limitation of the model, and natural neurons can probably mix functions easier. Instead of trying to make it all into a single simularity detector, I have to string multiple simularity detectors of only 2 dimensions together using an additive function.

#19

Ok, I heard about cosine similarity, and tried it because the dot product was too quick to get out of range, It seems better and the transforms do not need to be as large. I can’t do it now, but I will update the three input version in a bit.

#20

dot_3.py (3.4 KB)

#21

Although I’m still not totally sure what you’re going for here and still think a video call would be a good idea, looking at your code, I would like to note this is how you make a cosine similarity network in Nengo:

``````prod = nengo.Product()
dot_output = nengo.Node(size_in=1)
nengo.Connection(self.product.output, self.output, transform=np.ones((1, dimensions)))
``````

#22

It’s anything but self-explanatory, where did self.product.output and self.output come from? and what does np.ones((1, dimensions)) do Plus what the heck are you doing dimensioning a connection, Obviously I haven’t read enough of the documentation, can you point out what parts of the documentation covers this stuff?

Is this all supposed to work in a single def statement?

Colour me Confused

#23

I thought I would take a moment and explain the theory behind what I am trying to do.

Essentially it all has to do with information entropy

As we learn we integrate information, as we integrate information it becomes more entropic.

The Maximum Entropy Principle states that the most entropy is in the best learned data. In other words Maximum Entropy exists where the information integration is most completed.

mEPLA, my theory takes this into account, and turns it around to find new opportunities for learning. Essentially what it says is that the minimum entropy is found in areas where information integration is less complete, By extension this means that by allocating learning resources to areas of low entropy we can find opportunities to learn and better integrate information.

simularity is an approximation of entropy, the more a thing is simular the more integrated it’s information is, and the less learning that can be achieved.

What I think I am doing, is proving that similarity detection is a cheap way to detect learning opportunities. By inverting the logic, I am converting what I think is a natural similarity detector into a novelty detector which can detect learning opportunities and by applying learning resources to them, increase the information integration of the whole brain. I am just doing it almost as soon as the information gets to the cortex.

Theoretically novelty is an indication of a learning opportunity.

But to monitor entropy I need to operate in bulk, comparing the entropy of each input with all the others around it. I can then relatively easily gate the lowest entropy locations so that the lowest entropy triggers the gating of the outputs this is probably an early example of salience.

#24

OK, I was disappointed with cosine similarity it doesn’t do what I want for bulk similarity detection. I stumbled across a possible way of getting around the range problem. In essence as the matrix gets more complex the results get out of range. The more complex that matrix the more it gets out of range of the expression limit of the neuron. This is probably because it depends on a product. What I need is a way of taking the bulk similarity of a specific row in the matrix. The solution might be taking a partial derivative of the cosine similarity. of the whole matrix. Apparently it can be tuned to pick out the relative similarity of any particular row in the matrix.

#25

Obviously it is not as simple as doing the partial derivative of a cosine similarity, while a derivative reduces the complexity of the equation slightly the matrix created by taking the derivative of a 2D matrix becomes a 3D matrix. Obviously my Math is not up to this. I’ll come back once I have a handle on what I am trying to do mathematically

#26

I remembered that I had read that entropy was an average, taking the average of the permutations of possible vectors seems to be a superior way to achieve bulk similarity. I have rewritten the script with this in mind, I am still sticking to three inputs to prove the assumption, but it seems to work. The problem now is dilution of the significance of the gating signal as input numbers increase.