LessWrong 2.0 Reader

I think partially it's meant to go from some sort of abstract model of intelligence as a scalar variable that increases at some rate (like, on a x/y graph) to concrete, material milestones. Like, people can imagine "intelligence goes up rapidly! singularity!" and it's unclear what that implies, I'm saying sufficient levels would imply eating the sun, that makes it harder to confuse with things like "getting higher scores on math tests".

I suppose a more general category would be, the relevant kind of self-improving intelligence would be the sort that can re-purpose mass-energy to creating more computation that can run its intelligence, and "eat the Sun" is an obvious target given this background notion of intelligence.

(Note, there is skepticism about feasibility on Twitter/X, that's some info about how non-singulatarians react)

This case seems extremely cherry picked for cases where uplift is especially high. (Note that this is in copilot's interest.) Now, this task could probably be solved autonomously by an AI in like 10 minutes with good scaffolding.

I think you have to consider the full diverse range of tasks to get a reasonable sense or at least consider harder tasks. Like RE-bench seems much closer, but I still expect uplift on RE-bench to probably (but not certainly!) considerably overstate real world speed up.

This seemed like a nice explainer post, though it's somewhat confusing who the post is for – if I imagine being someone who didn't really understand any arguments about superintelligence, I think I might bounce off the opening paragraph or title because I'm like "why would I care about eating the sun." 

There is something nice and straightforward about the current phrasing but suspect there's an opening paragraph that would do a better job explaining why you might care about this.

(But I'd be curious to hear from people who weren't really sold on any singularity stuff who read it and can describe how it was for them)

Impact = Magnitude * Direction

Surely one should think of this as a vector in a space with more dimensions than 1. 

In your equation you can just 1,000,000x magnitude and it will move in the "positive direction".

In the real world you can become a billionaire from selling toothbrushes and still be "overtaken" by a guy who wrote one blog post that happened to be real dang good

I made a drawing but lw won't allow adding it on phone I think

I was saying 2x because I've memorised the results from this study. Do we have better numbers today? R&D is harder, so this is an upper bound. However, since this was from one year ago, so perhaps the factors cancel each other out?

My response to this is to focus on when a Dyson Swarm is being built, not AGI, because it's easier to define the term less controversially.

And a large portion of disagreements here fundamentally revolves around being unable to coordinate on what a given word means, which from an epistemic perspective doesn't matter at all, but it does matter from a utility/coordination perspective, where coordination is required for a lot of human feats.

I'm only referring to the reward constraint being satisfied for scenarios that are in the training distribution, since this maths is entirely applied to a decision taking place in training. Therefore I don't think distributional shift applies.

suppose I believe the second coming involves the Lord giving a speech on capitol hill. one thing I might care about is how long until that happens. the fact that lots of people disagree about when the second coming is doesn't mean the Lord will give His speech soon.

similarly, the thing that I define as AGI involves AIs building Dyson spheres. the fact that other people disagree about when AGI is doesn't mean I should expect Dyson spheres soon.

These problems are actually variations of one of the oldest "probability paradoxes" ever. And I put that in quotes, because in 1889 when Joseph Bertrand published it, "Bertrand's Box Paradox" meant how he proved that one proposed answer could not be right, because it produced a contradiction.

Here is that paradox, applied to Problem #1. With a slight modification that changes nothing except using a complimentary probability. In each question, what is the probability that I have a boy and a girl?

• I have two children, at least one of whom is a boy.
• I have two children, at least one of whom is a girl.
• I have two children, and have written the gender of at least one in this sealed envelope.

The answers to #1 and #2 have to be the same. It is tempting to say that answer is 2/3.

BUT, in #3, there are only two possibilities for what is written in the envelope. Once opened, it reduces to either #1, or #2. So the Law of Total Probability tells us that #3, even before the envelope is opened, also has that same answer.

But with no information given, the answer to #3 is clearly 1/2. So it can't be 2/3. This is the paradox in Bertrand's Box Problem. If you add a fourth box to that one, with a gold and a silver coin, it is identical to this Boy or Girl problem. I'm not saying that this directly proves that 1/2 is right. Just that no other answer can be right, and 1/2 can. Much like how proof by contradiction works.

Martin Gardner popularized this problem in the May, 1959 issue of Scientific American. But his wording was "Mr. Smith has two children. At least one is a boy." At first he said the answer was 2/3, but he withdrew that answer the next October, in a column titled "Probability and Ambiguity." His problem statement was ambiguous because it did not specify how the information came to be known. As others have pointed out, some methods lead to the answer 2/3, and others to the answer 1/2.

This wording is also ambiguous, but for a different reason. Whoever "I" is, knows the genders of the two children, and picked one of them. What we don't know, is how. But a Bayesian approach says that if he has a boy and a girl, we should assume he picked at random. This makes the answer 1/2.

To demonstrate this, Gardner published the Three Prisoners Problem in that October 1959 column. It is identical to the more modern puzzle, the Monty Hall Problem. With one exception: Gardner specified that a coin flip was used to choose what information is revealed, when both kinds of information are possible. That is usually not made explicit in Monty Hall, yet it is a necessary assumption to get that switching doors has a 2/3 probability of winning.

I haven't actually thought much about particular training algorithms yet. I think I'm working on a higher level of abstraction than that at the moment, since my maths doesn't depend on any specifics about V's behaviour. I do expect that in practice an already-scheming V would be able to escape some finite-time reasonable-beta-difference situations like this, with partial success.

I'm also imagining that during training, V is made up of different circuits which might be reinforced or weakened.

My view is that, if V is shaped by a training process like this, then scheming Vs are no longer a natural solution in the same way that they are in the standard view of deceptive alignment. We might be able to use this maths to construct training procedures where the expected importance of a scheming circuit in V is to become (weakly) weaker over time, rather than being reinforced.

If we do that for the entire training process, we would not expect to end up with a scheming V.

The question is which RL and inference paradigms approximate this. I suspect it might be a relatively large portion of them. I think that if this work is relevant to alignment then there's a >50% chance it's already factoring into the SOTA "alignment" techniques used by labs.