LessWrong 2.0 Reader

I think 356 or more people in the population needed to make there be a >5% of 2+ deaths in a 2 month span from that population

Daniel K seems pretty open about his opinions and reasons for leaving. Did he not sign an NDA and thus gave up whatever PPUs he had?

Update: The flashcards have finally been released: https://riceissa.github.io/immune-book/ 

I think this is really quite good, and went into way more detail than I thought it would. Basically my only complaints on the intro/part 1 are some terminology and historical nitpicks. I also appreciate the fact that Nicky just wrote out her views on AIS, even if they're not always the most standard ones or other people dislike them (e.g. pointing at the various divisions within AIS, and the awkward tension between "capabilities" and "safety"). 

I found the inclusion of a flashcard review applet for each section super interesting. My guess is it probably won't see much use, and I feel like this is the wrong genre of post for flashcards.^[1] But I'm still glad this is being tried, and I'm curious to see how useful/annoying other people find it.

I'm looking forward to parts two and three.

Nitpicks:^[2]

Logic vs Intuition:

I think "logic vs intuition" frame feels like it's pointing at a real thing, but it seems somewhat off. I would probably describe the gap as explicit vs implicit or legible and illegible reasoning (I guess, if that's how you define logic and intuition, it works out?). 

Mainly because I'm really skeptical of claims of the form "to make a big advance in/to make AGI from deep learning, just add some explicit reasoning". People have made claims of this form for as long as deep learning has been a thing. Not only have these claims basically never panned out historically, these days "adding logic" often means "train the model harder and include more CoT/code in its training data" or "finetune the model to use an external reasoning aide", and not "replace parts of the neural network with human-understandable algorithms". 

I also think this framing mixes together "problems of game theory/high-level agent modeling/outer alignment vs problems of goal misgeneralization/lack of robustness/lack of transparency" and "the kind of AI people did 20-30 years ago" vs "the kind of AI people do now".

This model of logic and intuition (as something to be "unified") is quite similar to a frame of the alignment problem that's common in academia. Namely, our AIs used to be written with known algorithms (so we can prove that the algorithm is "correct" in some sense) and performed only explicit reasoning (so we can inspect the reasoning that led to a decision, albeit often not in anything close to real time). But now it seems like most of the "oomph" comes from learned components of systems such as generative LMs or ViTs, i.e. "intuition". The "goal" is to a provably* safe AI, that can use the "oomph" from deep learning while having enough transparency/explicit enough thought processes. (Though, as in the quote from Bengio in Part 1, sometimes this also gets mixed in with capabilities, and become how AIs without interpretable thoughts won't be competent.)

Has AI had a clean "swap" between Logic and Intuition in 2000?

To be clear, Nicky clarifies in Part 1 that this model is an oversimplification. But as a nitpick, I think if you had to pick a date, I'd probably pick 2012, when a conv net won the ImageNet 2012 competition in a dominant matter. 

Even more of a nitpick, but the examples seem pretty cherry picked?

For example, Nicky uses the example of deep blue defeating kasparov as an example of a "logic" based AI. But in that case, almost all Chess AIs are still pretty much logic based. Using Stockfish as an example, Stockfish 16's explicit alpha-beta search both is using a reasoning algorithm that we can understand, and does the reasoning "in the open". Its neural network eval function is doing (a small amount of) illegible reasoning. While part of the reasoning has become illegible, we can still examine the outputs of the alpha-beta search to understand why certain moves are good/bad. (But fair, this might be by far the most widely known non-deep learning "AI". The only other examples I can think of are Watson and recommender systems, but those were still using statistical learning techniques. I guess if you count MYCIN or SHRDLU or ELIZA...?)

(And modern diffusion models being unable to count or spell seem like a pathology specific to that class of  generative model, and not say, Claude Opus.)

FOOM vs Exponential vs Steady Takeoff

Ryan already mentioned this in his comment. [LW(p) · GW(p)]

Even less important and more nitpicky nitpicks:

When did AIs get better than humans (at ImageNet)?

In footnote [3], Nicky writes:

In 1997, IBM's Deep Blue beat Garry Kasparov, the then-world chess champion. Yet, over a decade later in 2013, the best machine vision AI was only 57.5% accurate at classifying images. It was only until 2021, three years ago, that AI hit 95%+ accuracy. 

But humans do not get 95% top-1 accuracy^[3] on imagenet! If you consult this paper from the imagenet creators (https://arxiv.org/abs/1409.0575), they note that:

. We found the task of annotating images with one of 1000 categories to be an extremely challenging task for an untrained annotator. The most common error that an untrained annotator is susceptible to is a failure to consider a relevant class as a possible label because they are unaware of its existence. (Page 31)

And even when using an human expert annotators, who did hundreds of validation image for practice, the human annotator still got a top-5 error of 5.1%, which was surpassed in 2015 by the original resnet paper (https://arxiv.org/abs/1512.03385) at 4.49% for ResNet 14 (and 3.57% for an ensemble of six resnets). 

(Also, good top-1 performance on imagenet is genuinely hard and may be unrepresentative of actually being good at vision, whatever that means Take a look at some of the "mistakes" current models make:)

1. ^{^}

   Using flashcards suggests that you want to memorize the concepts. But a lot of this piece isn't so much an explainer of AI safety, but instead an argument for the importance of AI Safety. Insofar as the reader is not here to learn a bunch of new terms, but instead to reason about whether AIS is a real issue, it feels like flashcards are more of a distraction than an aid.

2. ^{^}

   I'm writing this in part because I at some point promised Nicky longform feedback on her explainer, but uh, never got around to it until now. Whoops. 

3. ^{^}

   Top-K accuracy = you guess K labels, and are right if any of them are correct. Top 5 is significantly easier on image net than Top 1, because there's a bunch of very similar classes and many images are ambiguous.

I did that! (I am the primary admin of the site). I copied your comment here just before I took down the duplicate post of yours to make sure it doesn't get lost.

Regarding sunscreens, Hyal Reyouth Moist Sun by the Korean brand Dr. Ceuracle is the most cosmetically elegant sun essence I have ever tried. It boasts SPF 50+, PA++++, chemical filters (no white cast) and is very pleasant to the touch and smell, not at all a sensory nightmare.

Yeah, my involvement was providing draft feedback on the article and providing some of the images. Looks like my post got taken down for being a duplicate, though

Snail mucin is one of those products that has less evidence behind it, besides its efficacy as a humectant, compared to the claims you'll often see in marketing. Here's a 1-minute video about it.
 
It's true that just because a research paper was published, it doesn’t necessarily mean that the research is that reliable — often, if you dig into studies, they’ll have a very small number of participants, or they only did in vitro testing, or something like that.

I’d also argue that natural doesn’t necessarily mean better. My favourite example is shea butter — some people have this romantic idea that it needs to come directly from a far-off village, freshly pounded, but the reality is that raw shea butter often contains stray particles that can in fact exacerbate allergic reactions. Refined shea butter is also really cool from a chemistry perspective, like, you can do very neat things with the texture.

what are Smith lotteries?

Lotteries over the Smith set. That definitely wasn't clear - I'll fix that.

which result do you mean by "above result"?

• Proposition: (Lottery-lotteries are strongly characterized by their selectivity of partitions of unity) 

 This one. "You can tell whether a lottery-lottery is maximal or not by how good the partitions of unity it admits are." Sorry, didn't really know a good way to link to myself internally and I forgot to number the various statements.

What does it mean for a lottery to be part of maximal lottery-lotteries?

Just that some maximal lottery-lottery gives it nonzero probability.

does "also subject to the partition-of-unity" refer to the smith lotteries or to the lotteries that are part of maximal lottery-lotteries? (it also feels like there is a word missing somewhere)

Oh no! I thought I caught all the typos! That should be "also subject to the partition-of-unity condition", that is, you look at all the lotteries (which we know are over the Smith set, btw) that some arbitrary maximal lottery-lottery gives any nonzero probability to, and you should expect to be able to sort them into groups by what final probability over candidates they induce; those final probabilities over candidates should themselves result in identical geometric-expected utility for the voterbase.

Why would this suffice?

Consider: at this point we know that a maximal lottery-lottery would not just have to be comprised of lottery-Smith lotteries, i.e., lotteries that are in the lottery-Smith set  - but also that they have to be comprised entirely of lotteries over the Smith set of the candidate set. Then on top of that, we know that you can tell which lottery-lotteries are maximal by which partitions of unity they admit (that's the "above result"). Note that by "admit" we mean "some subset of the lotteries this lottery-lottery has support over corresponds to it" (this partition of unity).

This is the slightly complicated part. The game I described has a mixed strategy equilibrium; this will take the form of some probability distribution over $\Delta C$. In fact it won't just have one, it'll likely have whole families of them. Much of the time, the lotteries randomized over won't be disjoint - they'll both assign positive probability to some candidate. The key is, the voter doesn't care. As far as a voter's expected utility is concerned, the only thing that matters is the final probability of each candidate.

That's where your collapse of different possible maximal lottery-lotteries to the same partition of unity comes in. Because we know that equivalent candidate-lotteries give voters the same expected utility, the only two ways you get a voter who's indifferent between two candidate-lotteries are 1) they're the same lottery or 2) the voter's utility function is just right to have two very different lotteries tie. Likewise, the only two ways you get a voterbase to be indifferent between two lottery-lotteries is 1) they reduce to the same lottery or 2) the geometric expectation of a voter's utility over candidates sampled from the samples of the lottery-lottery Just Plain Ties.

So: if we can show that whatever equilibrium set of candidate-lotteries Alice and Bob pick always collapses to some convex combination of the Best partitions of unity...? Yeah, I don't think that the second half of the proof holds up as is.

I think I've slightly messed up the definition of lottery-Smith, though not in a fatal way nor (thankfully) in a way that looks to threaten the existence result. The set condition is too strong, in requiring that a lottery-Smith lottery contain all lotteries which correspond to any of the admissible partitions. I'm just going to cut it; it's not actually necessary.

Is this part also supposed to imply the existence of maximal lottery-lotteries? If so, why?

Yes. 

Yes, and in particular, it implies the existence of maximal lottery-lotteries before it even tries to prove that they're also lottery-Smith in the sense I define.

Scott's proof [? · GW] doesn't quite work (as he says there) - it almost works, except for the part where the reward functions for Alice and Bob can quite reasonably be discontinuous. My proof is intended as a patch - the reward functions for Alice and Bob should now be extremely continuous in a way that also corresponds well to "how much better did Alice do at picking a candidate-lottery that V will like than Bob did?".

Hopefully this helped? Reading this is confusing even for me sometimes - the word "lottery/lotteries", which appears 59 times in this comment alone, no longer looks like a real word to me and hasn't since late Wednesday. Your comment was really helpful - I have some editing to do! (update - editing is done.)

I'm probably missing something simple, but what is 356? I was expecting a probability or a percent, but that number is neither.