LessWrong 2.0 Reader

I think this is really quite good, and went into way more detail than I thought it would. Basically my only complaints on the intro/part 1 are some terminology and historical nitpicks. I also appreciate the fact that Nicky just wrote out her views on AIS, even if they're not always the most standard ones or other people dislike them (e.g. pointing at the various divisions within AIS, and the awkward tension between "capabilities" and "safety"). 

I found the inclusion of a flashcard review applet for each section super interesting. My guess is it probably won't see much use, and I feel like this is the wrong genre of post for flashcards.^[1] But I'm still glad this is being tried, and I'm curious to see how useful/annoying other people find it.

I'm looking forward to parts two and three.

Nitpicks:^[2]

Logic vs Intuition:

I think "logic vs intuition" frame feels like it's pointing at a real thing, but it seems somewhat off. I would probably describe the gap as explicit vs implicit or legible and illegible reasoning (I guess, if that's how you define logic and intuition, it works out?). 

Mainly because I'm really skeptical of claims of the form "to make a big advance in/to make AGI from deep learning, just add some explicit reasoning". People have made claims of this form for as long as deep learning has been a thing. Not only have these claims basically never panned out historically, these days "adding logic" often means "train the model harder and include more CoT/code in its training data" or "finetune the model to use an external reasoning aide", and not "replace parts of the neural network with human-understandable algorithms". 

I also think this framing mixes together "problems of game theory/high-level agent modeling/outer alignment vs problems of goal misgeneralization/lack of robustness/lack of transparency" and "the kind of AI people did 20-30 years ago" vs "the kind of AI people do now".

This model of logic and intuition (as something to be "unified") is quite similar to a frame of the alignment problem that's common in academia. Namely, our AIs used to be written with known algorithms (so we can prove that the algorithm is "correct" in some sense) and performed only explicit reasoning (so we can inspect the reasoning that led to a decision, albeit often not in anything close to real time). But now it seems like most of the "oomph" comes from learned components of systems such as generative LMs or ViTs, i.e. "intuition". The "goal" is to a provably* safe AI, that can use the "oomph" from deep learning while having enough transparency/explicit enough thought processes. (Though, as in the quote from Bengio in Part 1, sometimes this also gets mixed in with capabilities, and become how AIs without interpretable thoughts won't be competent.)

Has AI had a clean "swap" between Logic and Intuition in 2000?

To be clear, Nicky clarifies in Part 1 that this model is an oversimplification. But as a nitpick, I think if you had to pick a date, I'd probably pick 2012, when a conv net won the ImageNet 2012 competition in a dominant matter. 

Even more of a nitpick, but the examples seem pretty cherry picked?

For example, Nicky uses the example of deep blue defeating kasparov as an example of a "logic" based AI. But in that case, almost all Chess AIs are still pretty much logic based. Using Stockfish as an example, Stockfish 16's explicit alpha-beta search both is using a reasoning algorithm that we can understand, and does the reasoning "in the open". Its neural network eval function is doing (a small amount of) illegible reasoning. While part of the reasoning has become illegible, we can still examine the outputs of the alpha-beta search to understand why certain moves are good/bad.

(And modern diffusion models being unable to count or spell seem like a pathology specific to that class of  generative model, and not say, Claude Opus.)

FOOM vs Exponential vs Steady Takeoff

Ryan already mentioned this in his comment. [LW(p) · GW(p)]

Even less important and more nitpicky nitpicks:

When did AIs get better than humans (at ImageNet)?

In footnote [3], Nicky writes:

In 1997, IBM's Deep Blue beat Garry Kasparov, the then-world chess champion. Yet, over a decade later in 2013, the best machine vision AI was only 57.5% accurate at classifying images. It was only until 2021, three years ago, that AI hit 95%+ accuracy. 

But humans do not get 95% top-1 accuracy^[3] on imagenet! If you consult this paper from the imagenet creators (https://arxiv.org/abs/1409.0575), they note that:

. We found the task of annotating images with one of 1000 categories to be an extremely challenging task for an untrained annotator. The most common error that an untrained annotator is susceptible to is a failure to consider a relevant class as a possible label because they are unaware of its existence. (Page 31)

And even when using an human expert annotators, who did hundreds of validation image for practice, the human annotator still got a top-5 error of 5.1%, which was surpassed in 2015 by the original resnet paper (https://arxiv.org/abs/1512.03385) at 4.49% for ResNet 14 (and 3.57% for an ensemble of six resnets). 

(Also, good top-1 performance on imagenet is genuinely hard and may be unrepresentative of actually being good at vision, whatever that means Take a look at some of the "mistakes" current models make:)

1. ^{^}

   Using flashcards suggests that you want to memorize the concepts. But a lot of this piece isn't so much an explainer of AI safety, but instead an argument for the importance of AI Safety. Insofar as the reader is not here to learn a bunch of new terms, but instead to reason about whether AIS is a real issue, it feels like flashcards are more of a distraction than an aid.

2. ^{^}

   I'm writing this in part because I at some point promised Nicky longform feedback on her explainer, but uh, never got around to it until now. Whoops. 

3. ^{^}

   Top-K accuracy = you guess K labels, and are right if any of them are correct. Top 5 is significantly easier on image net than Top 1, because there's a bunch of very similar classes and many images are ambiguous.

I did that! (I am the primary admin of the site). I copied your comment here just before I took down the duplicate post of yours to make sure it doesn't get lost.

Regarding sunscreens, Hyal Reyouth Moist Sun by the Korean brand Dr. Ceuracle is the most cosmetically elegant sun essence I have ever tried. It boasts SPF 50+, PA++++, chemical filters (no white cast) and is very pleasant to the touch and smell, not at all a sensory nightmare.

Yeah, my involvement was providing draft feedback on the article and providing some of the images. Looks like my post got taken down for being a duplicate, though

Snail mucin is one of those products that has less evidence behind it, besides its efficacy as a humectant, compared to the claims you'll often see in marketing. Here's a 1-minute video about it.
 
It's true that just because a research paper was published, it doesn’t necessarily mean that the research is that reliable — often, if you dig into studies, they’ll have a very small number of participants, or they only did in vitro testing, or something like that.

I’d also argue that natural doesn’t necessarily mean better. My favourite example is shea butter — some people have this romantic idea that it needs to come directly from a far-off village, freshly pounded, but the reality is that raw shea butter often contains stray particles that can in fact exacerbate allergic reactions. Refined shea butter is also really cool from a chemistry perspective, like, you can do very neat things with the texture.

what are Smith lotteries?

Lotteries over the Smith set. That definitely wasn't clear - I'll fix that.

which result do you mean by "above result"?

• Proposition: (Lottery-lotteries are strongly characterized by their selectivity of partitions of unity) 

 This one. "You can tell whether a lottery-lottery is maximal or not by how good the partitions of unity it admits are." Sorry, didn't really know a good way to link to myself internally and I forgot to number the various statements.

What does it mean for a lottery to be part of maximal lottery-lotteries?

Just that some maximal lottery-lottery gives it nonzero probability.

does "also subject to the partition-of-unity" refer to the smith lotteries or to the lotteries that are part of maximal lottery-lotteries? (it also feels like there is a word missing somewhere)

Oh no! I thought I caught all the typos! That should be "also subject to the partition-of-unity condition", that is, you look at all the lotteries (which we know are over the Smith set, btw) that some arbitrary maximal lottery-lottery gives any nonzero probability to, and you should expect to be able to sort them into groups by what final probability over candidates they induce; those final probabilities over candidates should themselves result in identical geometric-expected utility for the voterbase.

Why would this suffice?

Consider: at this point we know that a maximal lottery-lottery would not just have to be comprised of lottery-Smith lotteries, i.e., lotteries that are in the lottery-Smith set  - but also that they have to be comprised entirely of lotteries over the Smith set of the candidate set. Then on top of that, we know that you can tell which lottery-lotteries are maximal by which partitions of unity they admit (that's the "above result"). Note that by "admit" we mean "some subset of the lotteries this lottery-lottery has support over corresponds to it" (this partition of unity).

This is the slightly complicated part. The game I described has a mixed strategy equilibrium; this will take the form of some probability distribution over $\Delta C$. In fact it won't just have one, it'll likely have whole families of them. Much of the time, the lotteries randomized over won't be disjoint - they'll both assign positive probability to some candidate. The key is, the voter doesn't care. As far as a voter's expected utility is concerned, the only thing that matters is the final probability of each candidate.

That's where your collapse of different possible maximal lottery-lotteries to the same partition of unity comes in. Because we know that equivalent candidate-lotteries give voters the same expected utility, the only two ways you get a voter who's indifferent between two candidate-lotteries are 1) they're the same lottery or 2) the voter's utility function is just right to have two very different lotteries tie. Likewise, the only two ways you get a voterbase to be indifferent between two lottery-lotteries is 1) they reduce to the same lottery or 2) the geometric expectation of a voter's utility over candidates sampled from the samples of the lottery-lottery Just Plain Ties.

So: if we can show that whatever equilibrium set of candidate-lotteries Alice and Bob pick always collapses to some convex combination of the Best partitions of unity...? Yeah, I don't think that the second half of the proof holds up as is.

I think I've slightly messed up the definition of lottery-Smith, though not in a fatal way nor (thankfully) in a way that looks to threaten the existence result. The set condition is too strong, in requiring that a lottery-Smith lottery contain all lotteries which correspond to any of the admissible partitions. I'm just going to cut it; it's not actually necessary.

Is this part also supposed to imply the existence of maximal lottery-lotteries? If so, why?

Yes. 

Yes, and in particular, it implies the existence of maximal lottery-lotteries before it even tries to prove that they're also lottery-Smith in the sense I define.

Scott's proof [? · GW] doesn't quite work (as he says there) - it almost works, except for the part where the reward functions for Alice and Bob can quite reasonably be discontinuous. My proof is intended as a patch - the reward functions for Alice and Bob should now be extremely continuous in a way that also corresponds well to "how much better did Alice do at picking a candidate-lottery that V will like than Bob did?".

Hopefully this helped? Reading this is confusing even for me sometimes - the word "lottery/lotteries", which appears 59 times in this comment alone, no longer looks like a real word to me and hasn't since late Wednesday. Your comment was really helpful - I have some editing to do! (update - editing is done.)

I'm probably missing something simple, but what is 356? I was expecting a probability or a percent, but that number is neither.

Concerning the efficacy of hyaluronic acid (HA) in enhancing skin hydration, I would like to highlight glycerin (glycerol) as a superior humectant. 

Recalling the 500-Dalton rule, which postulates that any compound with a molecular weight inferior to five hundred daltons possesses the ability to penetrate the skin barrier, we can provide a framework that elucidates the mechanisms of penetration of both compounds. Notably, glycerin has a molecular weight of 92.09 daltons, while even a low-molecular-weight HA weighs a substantial 50,000 daltons. For comparison, high-molecular HA can reach a staggering 1 million daltons.

Consequently, HA is rendered incapable of traversing the deeper skin layers and confined to the epidermis. Topical HA is potent and can bind to colossal amounts of water, proving to be a stellar humectant. Nevertheless, the hygroscopic nature of HA can be problematic in dry climates: HA can extract moisture from adjacent skin cells, inducing transepidermal water loss.

A thorough examination of the hyperbolic marketing surrounding this compound reveals a propensity to obscure the boundaries of its categorization concerning its weight, thereby precipitating a conflation of topical HA and injectable HA, which in turn yields imprecise buzzwords such as "filler" printed on topical moisturizers. A comparative evaluation reveals that the rejuvenative effects of topical HA, when contrasted with its injectable counterpart, are eclipsed in terms of its ability to enhance skin volume and elasticity.

Now, glycerin, on the other hand, has consistently demonstrated superior results at a more economical price point. The trihydroxylated glycerol molecule is widely regarded as one of the most (if not best) humectants: its small molecular weight allows it to penetrate the skin effectively, which characterizes its ability to retain and attract water molecules, and ensure long-lasting hydration. 

The synergistic effect of HA and glycerin may provide enhanced hydration benefits by targeting different aspects of skin moisture retention: the concomitant use of both compounds in this study has yielded favorable outcomes.

@henry [LW · GW] (who seems to know Nicky) said on a duplicate link post of this: 

This is an accessible introduction to AI Safety, written by Nicky Case and the teens at Hack Club. So far, part 1/3 is completed, which covers a rough timeline of AI advancement up to this point, and what might come next.

If you've got feedback as to how this can be made more understandable, that'd be appreciated! Reach out to Nicky, or to me and I'll get the message to her.

I agree with you. 

Epsilon machine (and MSP) construction is most likely computationally intractable [I don't know an exact statement of such a result in the literature but I suspect it is true] for realistic scenarios. 

Scaling an approximate version of epsilon reconstruction seems therefore of prime importance. Real world architectures and data has highly specific structure & symmetry that makes it different from completely generic HMMs. This must most likely be exploited. 

The calculi of emergence paper has inspired many people but has not been developed much. Many of the details are somewhat obscure, vague. I also believe that most likely completely different methods are needed to push the program further. Computational Mechanics' is primarily a theory of hidden markov models - it doesn't have the tools  to easily describe behaviour higher up the Chomsky hierarchy. I suspect more powerful and sophisticated algebraic, logical and categorical thinking will be needed here. I caveat this by saying that Paul Riechers has pointed out that actually one can understand all these gadgets up the Chomsky hierarchy as infinite HMMs which may be analyzed usefully just as finite HMMs. 

The still-underdeveloped theory of epsilon transducers I regard as the most promising lens on agent foundations. This is uncharcted territory; I suspect the largest impact of computational mechanics will come from this direction. 

Your point on True Names is well-taken. More basic examples than gauge information, synchronization order are the triple of quantites entropy rate $h$, excess entropy $E$ and Crutchfield's  statistical/forecasting complexity $C$. These are the most important quantities to understand for any stochastic process (such as the structure of language and LLMs!)