The Perceptron Controversy

magnetic cores. The MINOS II machine built in 1962 by the Stanford Research Institute group had precisely a grid of magnetic core memory

By magnetic memory I was referring to :

The core idea is that you can bulk store a reasonably large amount of data - 5 mb a module for the picture above, up to hundreds of mb in the 1970s - that get accessed infrequently.

For example, suppose you wanted to calculate a paycheck for an hourly worker, an early application for this type of computer.  Every time they use a time card reader (a punchcard with their employee ID), you save to a data structure on a hard drive the recorded time.

Then, once every 2 weeks, for each n employees, you retrieve the data for all the clock in/clock out times, retrieve a record with their rate of pay and other meta rules (are they eligible for overtime etc), and then in a relatively simply program you can calculate the payroll record, print it to text, print their paycheck, save the record to a tape drive, and then zero out the record for the next payroll cycle.

So you have relative large dataset : n employee records with m times for each, plus bits for meta rules, and a relatively tiny program.  You will only check each record 1 time.

Neural networks have the issue that any practical one is pretty large, and all weights must be accessed every inference.

 

great successes in 1990s connectionism, including LeNet digit recognition, NETtalk, and the TD-gammon

These are not useful applications.  The closest is digit recognition.  

From the 1970s to somewhere in the 2000s, OCR was done with computer vision.  You collect features from each letter, such as the lines with hough transforms, harris corner detectors, and you map these features to a relative coordinate space.  Then you use some type of classifier, SVMs or similar, to detect the letter.  

I'm not quite sure of the flop requirements for a CV solution, since calculating these transforms is some matrix operations, but I can say that the kernels to do it are much smaller than a neural network.  

You can only detect printed characters reliably this way - so an earlier LeNet would have saved a little bit of money for the post office.

And just to reiterate, as bad as Builder the Robot was at navigation, Minsky was right.  Avionics, factory control systems, motor control, automotive throttle by wire - I have worked on a variety of these systems, and they all work as a "society of the mind" of small specialized programs.  They don't scale to large, complex robots, but it doesn't matter, startups today driving big robots have immensely more compute and memory to do it.  RT-X is getting closer to a useful capability level, and it's 50B weights.

The main limitation is without neural networks, you need really clean perception.  For example resolvers for joint position, or GPS or older systems for avionics.  The machine needs the exact coordinates or rotational angle, it can't use a camera.

I'd recommend porting it over as a sequence instead of one big post (or maybe just port the first chunk as an intro post?). LW doesn't have a citation format, but you can use footnotes for it (and you can use the same footnote number in multiple places).

Porting it full-scale seems nice. Our markdown dialect does indeed not currently support that citation format, but a simple regex would probably be good enough to just replace all the citations with footnotes. 

Does LessWrong allow convenient citations?

don't think so, I've bugged them about it before so maybe it got implemented while I wasn't looking but idk where to find out. what do you use to process the citations into the form on the webpage? I doubt they'll implement it now but sending the code for it could make it happen someday

it would be impossible to establish that much order with pruning alone.

That seems straightforwardly falsified by ANNs like lottery tickets or weight agnostic NNs, where the weights are randomized and all the learning is done by adding/pruning away connections (you can think of it as searching in fully-connected net space). Most on point here would be Ramanujan et al 2019 which proves that as random NNs get bigger, you expect there to be a sub-network which computes whatever you need (which makes sense, really, why wouldn't there be? NNs express such rich classes of functions, that if you can pick and choose from within a very large NN, it'd make sense you could sculpt whatever you need).

Hmmm, I disagree with the randomness.

I don’t think you do. Let me rephrase: the weights are picked at random, under a distribution biased by molecular cues, then pruned through activity dependent mechanisms.

In other words, our disagreement seems to count as an instance of Bertrand’s paradox.