Marcello's Shortform

"Aspiring Rationalist" Considered Harmful

The "aspiring" in "aspiring rationalist" seems like superfluous humility at best. Calling yourself a "rationalist" never implied perfection in the first place. It's just like how calling yourself a "guitarist" doesn't mean you think you're Jimi Hendrix. I think this analogy is a good one, because rationality is a human art, just like playing the guitar.

I suppose one might object that the word "rational" denotes a perfect standard, unlike playing the guitar. However, we don't hesitate to call someone an "idealist" or a "perfectionist" when they're putting in a serious effort to conform to an ideal or strive towards perfection, so I think this objection is weak. The "-ist" suffix already means that you're a person trying to do the thing, with all the shortcomings that entails.

Furthermore, it appears harmful to add the "aspiring". It creates dilution. Think of what it would mean for a group of people to call themselves "aspiring guitarists". The trouble is, it also applies to the sort of person who daydreams about the adulation of playing for large audiences but never gets around to practicing. However, to honestly call yourself a "guitarist", you would have to actually, y'know, play the guitar once in a while.

While I acknowledge I'm writing this many years too late, please consider dropping the phrase "aspiring rationalist" from your lexicon.

In Defense of the Shoggoth Analogy

In reply to: https://twitter.com/OwainEvans_UK/status/1636599127902662658

The explanations in the thread seem to me to be missing the middle or evading the heart of the problem.  Zoomed out: an optimization target at level of personality.  Zoomed in: a circuit diagram of layers.  But those layers with billions of weights are pretty much Turing complete.

Unfortunately, I don't think anyone has much idea how all those little learned computations are make up said personality.  My suspicion is there isn't going to be an *easy* way to explain what they're doing.  Of course, I'd be relieved to be wrong here!

This matters because the analogy in the thread between averaged faces and LLM outputs is broken in an important way.  (Nearly) every picture of a face in the training data has a nose.  When you look at the nose of an averaged face, it's based very closely on the noses of all the faces that got averaged.  However, despite the size of the training datasets for LLMs, the space of possible queries and topics of conversation is even vaster (it's exponential in the prompt-window size, unlike the query space for the average faces which are just the size of the image).

As such, LLMs are forced to extrapolate hard.  So, I'd expect that which particular generalizations they learned, hiding in those weights, to start to matter once users start poking them in unanticipated ways.

In short, if LLMs are like averaged faces, I think they're faces that will readily fall apart into Shoggoths if someone looks at them from an unanticipated or uncommon angle.

One of my pet journalism peeves is the "as" (or sometimes "while") construction, which I often see in titles or first sentences of articles. It looks like "<event A was happening> as <event B was happening>". You can fact check the events and it'll turn out they happened, but the phrasing comes with this super annoying nudge-nudge-wink-wink-implication that the two events totally have direct causal connection. Unfortunately, you can't pin this on the journalist because they didn't actually say it.

This sort of thing happens a lot. To give just a couple example templates, articles like "as <political thing happened>, markets rallied" or "<stock> falls as <CEO did something>" are often trying to pull this.