LessWrong 2.0 Reader

Beauty of notation is an optimization target and so should fail as a metric, but especially compared to other optimization targets I’ve pushed on, in my experience it seems to hold up. The exceptions appear to be string theory and category theory and two failures in a field the size of math is not so bad.

Bywayeans are pretty censorious and scrupulous about violations of the NAP

Except against people who enjoy sunsets, apparently?

In retrospect, I am more pessimistic about AI having small amounts of niceness making humans live, and I now think that some amount of stronger alignment than pseudokindness is necessary to make humans survive with AI (but maybe not as strong as MIRI thinks), essentially because niceness to humans requires giving up opportunities to save compute on modeling the world, which is anti-incentivized by AI companies:

https://www.lesswrong.com/posts/xvBZPEccSfM8Fsobt/what-are-the-best-arguments-for-against-ais-being-slightly#wy9cSASwJCu7bjM6H [LW(p) · GW(p)]

Beautiful writings! 

I particularly enjoyed For Boltzmann... The combination of Boltzmann's flair for abstract conceptualization amidst his own personal tragedy is quite the tale to tell, and you were able to capture that in the poem. Boltzmann's struggles mirror so many other personal tales and I certainly hope each share of blood sweat and tears won't be in vain. 

The restless search for universality which artists and scientists experience is also the story that connects these threads. Boltzmann Brains navigating space and time.

"Those parts are yet acknowledged, and yet mourned.
And when each human rises in their powers
The efforts of our past selves won’t be scorned."

Also reminds me of a classic Brazilian Samba by João Bosco and Aldir Blanc

"Eu sei que uma dor assim pungente, não há de ser inutilmente, 

a esperança dança da corda bamba de sombrinha 

e em cada passo dessa linha pode se machucar, 

Azar, a esperança equilibrista, 

sabe que o show de todo artista tem que continuar"

Which roughly translates to:

"I know that a pain as piercing as this cannot be in vain, 

hope dances on the tightrope with an umbrella, 

and with each step on this line, it can get hurt. 

Hazard, hope, the tightrope-walker 

knows that the show of every artist must go on."

Epistemic Status: All-in
 

Thank you so much for compiling these quotes; they are impactful and I might never have read them if you hadn't posted them here.

I think it's possible to be better than humans currently are at minecraft, I can say more if this sounds wrong

Yeah, that's true. The obvious way is you could have optimized micro, but that's kinda boring. More like what I mean might be generalization to new activities for humans to do in minecraft that humans would find fun, which would be a different kind of 'better at minecraft.'

[what do you mean by preference conflict?]

I mean it in a way where the preferences are modeled a little better than just "the literal interpretation of this one sentence conflicts with the literal interpretation of this other sentence." Sometimes humans appear to act according to fairly straightforward models of goal-directed action. However, the precise model, and the precise goals, may be different at different times (or with different modeling hyperparameters, and of course across different people) - and if you tried to model the human well at all the different times, you'd get a model that looked like physiology and lost the straightforward talk of goals/preferences

Resolving preference conflicts is the process of stitching together larger preferences out of smaller preferences, without changing type signature. The reason literally-interpreted-sentences doesn't really count is because interpreting them literally is using a smaller model than necessary - you can find a broader explanation for the human's behavior in context that still comfortably talks about goals/preferences.

(Not "idle worship"?)

I think it's net negative - increases the profitability of training better LLM's. 

Over-fascination with beautiful mathematical notation is idol worship. 

But as a secondary point, I think today's models can already use bash tools reasonably well.

Perhaps that's true, I haven't seen a lot of examples of them trying. I did see Buck's anecdote which was a good illustration of doing a simple task competently (finding the IP address of an unknown machine on the local network).

I don't work in AI so maybe I don't know what parts of R&D might be most difficult for current SOTA models. But based on the fact that large-scale LLMs are sort of a new field that hasn't had that much labor applied to it yet, I would have guessed that a model which could basically just do mundane stuff and read research papers, could spend a shitload of money and FLOPS to run a lot of obviously informative experiments that nobody else has properly run, and polish a bunch of stuff that nobody else has properly polished.