The issue with MNIST is that everything works on MNIST, even algorithms that utterly fail on a marginally more complicated task. It's a solved problem, and the fact that this algorithm solves it tells you nothing about it.

If the code is too rigid or poorly performant to be tested on larger or different tasks, I suggest F-MNIST (fashion MNIST), which uses the exact same data format, has the same number of categories and amount of data points, but is known to be far more indicative of the true performance of modern machine learning approaches.

https://github.com/zalandoresearch/fashion-mnist

testing 20000: 0: 44.413  1: 43.77
testing 40000: 0: 23.702  1: 23.34
testing 60000: 0: 21.822  1: 20.92
testing 80000: 0: 38.627  1: 37.75
testing 100000: 0: 25.107  1: 24.85
testing 120000: 0: 28.893  1: 28.34
testing 140000: 0: 29.203  1: 29.50
testing 160000: 0: 40.437  1: 39.55
testing 180000: 0: 49.828  1: 48.59
testing 200000: 0: 39.510  1: 39.81
testing 220000: 0: 39.938  1: 40.10
testing 240000: 0: 32.390  1: 31.65
testing 260000: 0: 25.367  1: 24.51
testing 280000: 0: 29.198  1: 28.66
testing 300000: 0: 29.823  1: 28.91
testing 320000: 0: 35.178  1: 34.20
testing 340000: 0: 32.370  1: 31.94
testing 360000: 0: 29.083  1: 28.31
testing 380000: 0: 30.117  1: 30.05
testing 400000: 0: 39.125  1: 39.21

testing 59300000: 0: 58.615  1: 57.36
testing 59320000: 0: 50.902  1: 50.28
testing 59340000: 0: 68.415  1: 66.67
testing 59360000: 0: 71.813  1: 69.36
testing 59380000: 0: 70.275  1: 68.53
testing 59400000: 0: 71.577  1: 68.67

I think there's a mistake which is being repeated in a few comments both here and on D𝜋's post which needs emphasizing. Below is my understanding:

D𝜋 is attempting to create a general intelligence architecture. He is using image classification as a test for this general intelligence but his architecture is not optimized specifically for image identification.

Most attempts on MNIST use what we know about images (especially the importance of location of pixels) and design an architecture based on those facts. Convolutions are an especially obvious example of this. They are very effective at identifying images but the fact that we are inserting some of our knowledge of images into the algorithm precludes it from being a general intelligence methodology (without a lot of modification at least).

The point of using PI-MNIST (where locations of pixels in the dataset are randomized) is that we can't use any of our own understanding of images to help with our model so a model which is good at PI-MNIST is proving a more general intelligence than a model which is good at MNIST.

That is why D𝜋 keeps on emphasizing that this is PI-MNIST.

I just discovered about the 'ping back' on LessWrong...

I gave a first read of your description. Most of it is correct. I will check in more details.

I used the terms 'total' and 'groups' to make things simpler, but yours are better.

four corrections:

1.

The potential of a neuron can be negative. It is the pure sum of all weights, positive and negative. There is no 'negative spiking' (It is one of the huge number of things I tried that did not bring any benefit). It think I remember trying to set a bottom limit at 0 (no negative potential) and that, as always, it did not make any real difference...

2.

'Our system thus has four receptors per pixel. Exactly one receptor activates per pixel per image' is incorrect.

The MNIST pixels are grey shades 0-255.

It is reduced down to 4: 0, 1-63, 64-127, 128-191, >=192. (only keep the 2 top bits). That is enough for MNIST. Many papers have noted that the depth can be reduced and it is true.

Images are presented over 4 'cycles', filtered by those 4 limits. In the first cycle, only pixels with a value over 191 are presented, in the second one, those over 127, the third 63, the last, 0 (not nul).

In the code the array 'wd' contains the 4 limits, and at each cycle, the pixel values are tested to be superior or equal to those limits.

Connexions are established with matrix pixels. Over the 4 cycles, they are presented with 4 successive 0 or 1.

If a connexion is on a pixel that shows value 112, on the first cycle it will not be active (<192), on the second it will not be active (<128), but on the third and forth, it will be (>63 and >0).

That is what allows the 'model averaging across cycles'.

From there, you can understand a first, fatal, reason why F-MNIST cannot be processed as-is: the grey shades are reversed. In MNIST, the background is white, in F-MNIST, it is black. So the cycle limits would have to be reversed.

I will have a look at it.

3.

The $\Delta {\perp }_i$ computation includes the $\top$ column.

Note that the 'highest' ⊥ selection can be easily implemented using population coding with inhibition of the ⊤ column.

I do not know if the options I used are only valid for this dataset or if there have larger validity across dataset as I only have used that one. Maybe they are and you won't have to figure out each time.

4.

When a new connection is established, the initial weight is always the same. It is given as a fraction of the threshold in the variable 'divNew', that is the divider. You can do random, it does not make a difference. You can change it to another value. But it has to be small enough (the divider) that the number of connections of a neuron multiplied by the number of cycles be superior to the threshold, or the system will never 'boot' as no neuron would ever spike. So I use 1/10 of the threshold with 10 connections and 4 cycle, and it is fine.

Instead, D𝜋 uses a simple algorithm called a quantilizer.

Note that this is a different algorithm from the original quantilizers.

When I saw D𝜋's original post one of my first thoughts was "Maybe I should ask Lsusr to look into this and try to replicate it." Well done!

In the original article, Dpi claims:

When a neuron spikes, the connections that have contributed to it are reinforced. It is Hebb’s rule. In this system, we do the same.

But this doesn't seem to be present in your description? Unless I missed it.

It'd be cool to see like benchmarks (accuracy + performance) compared to MLP-based ANNs

One point which is never mentioned (as far as I can see) either in D𝜋's article or in this one, but which is present in the code, is the minimum threshold for learning. D𝜋's code only learns if the neuron's potential is already above a certain minimal value (300,000 in the code as posted). D𝜋 says the maximum threshold is optional, but in my experiments, I found the minimum threshold absolutely crucial for the network to ever get to high levels of accuracy. Seems like otherwise a neuron gets distracted by a bunch of junk updates that aren't really related to its particular purpose.

For each ⊥i column, if Δ⊥i is in the top 0.66% of those seen before then we perform negative reinforcement.

Do we negatively reinforce False columns that are especially distant from the True column? That seems backward.