LessWrong 2.0 Reader

Kudos for referencing actual numbers. I don’t think it makes sense to measure humans in terms of tokens, but I don’t have a better metric handy. Tokens obviously aren’t all equivalent either. For some purposes, a small fast LLM is more way efficient than a human. For something like answering SIMPLEBENCH, I’d guess o1-preview is less efficient while still significantly below human performance.

i'm finally learning to prove theorems (the earliest ones following from the Peano axioms) in lean, starting with the natural number game. it is actually somewhat fun.

also, this reminded me of one of @Raemon [LW · GW]'s idea (https://www.lesswrong.com/posts/PiPH4gkcMuvLALymK/exercise-solve-thinking-physics), 'how to prove theorem' feels like the most pure case of 'solving a problem that you (often) do not know how to solve', which iiuc they're a proponent of training on

Yes for sure. I don't know how it would play out, and am skeptical anyone could. We can guess scenarios. 

1. The most easily imagined one is the Pebbles owner staying in their comfort zone and not enforcing #2 at all. Something similar already happened - the USA got nukes first and let others catch up. In this case threatened nations try all sorts of things, political, commercial/trade, space war, arms race but don't actually start a hot conflict.  The Pebbles owner is left not knowing whether their system is still effective, nor the threatened countries - an unstable situation.

2. The threatened nation tries to destroy the pebbles with non-nuke means. If this was Russia, USA maybe could regenerate the system faster than Russia could destroy satellites. If its China, then lets say its not. The USA then needs to decide whether to strike the anti-satellite ground infrastructure to keep its system...

3. The threatened nation such as NK just refuses to give up nukes - in this case I can see USA destroying it.

4. India or Israel say refuses to give up their arsenal - I have no idea what would happen then.

The actual reason why we lied in the second message was "we were in a rush and forgot." 

My recollection is we sent the same message to the majority group because:

1. Treating it different would require special-casing it and that would have taken more effort.
2. If selectors of different virtues had received a different messages, we wouldn't be able to have a properly compared their behavior.
3. [At least in my mind], this was a game/test and when playing games you lie to people in the context of the game to make things work. Alternatively, it's like how scientific experimenters mislead subjects for the sake of the study.

Predictable/Unpredictable takeoff

I think long duration is way too many syllables, and I think I have similar problems with this naming schema as Fast/Slow, but, if you were going to go with this naming schema I think just saying "short takeoff" and "long takeoff" seems about as clear ("duration" comes implied IMO)

I don't love "smooth" vs "sharp" because these words don't naturally point at what seem to me to be the key concept: the duration from the first AI capable of being transformatively useful [LW · GW] to the first system which is very qualitatively generally superhuman^[1] [LW · GW]. You can have "smooth" takeoff driven by purely scaling things up where this duration is short or nonexistent.

I'm not sure I buy the distinction mattering?

Here's a few worlds:

1. Smooth takeoff to superintelligence via scaling the whole way, no RSI
2. Smooth takeoff to superintelligence via a mix of scaling, algorithmic advance, RSI, etc
3. smoothish looking takeoff via scaling (like we currently see) but then suddenly the shape of the curve changes dramatically due to RSI or similar
4. smoothish looking takeoff via scaling like we see, but, and then RSI is the mechanism by which the curve continues, but not very quickly (maybe this implies the curve actively levels off S-curve style before eventually picking up again)
5. alt-world where we weren't even seeing similar types of smoothly advancing AI, and then there's abrupt RSI takeoff in days or months
6. alt-world where we weren't seeing similar smooth scaling AI, and then RSI is the thing that initiates our current level of growth

At least with the previous way I'd been thinking about things, I think the worlds above that look smooth, I feel like "yep, that was a smooth takeoff."

Or, okay, I thought about it a bit more and maybe agree that "time between first transformatively-useful to superintelligence" is a key variable. But, I also think that variable is captured by saying "smooth takeoff/long timelines?" (which is approximately what people are currently saying?

Hmm, I updated towards being less confident while thinking about this.

Of course, you could argue that a bijection class of sets is there to formalize the notion of a quantity that generalizes to infinite sets.

Indeed, 2 sets like (1,2,3) and (A,B,C) have the same cardinality, which is 3, and cardinality is IMO the generalization of quantities to infinite sets like the real numbers. Indeed, 2 sets have the same cardinality if and only if they have a bijection.

That said, cardinality is a very coarse-grained way to look at quantities, as the Turing Machine computable sets and all arithmetical sets, which include Turing uncomputable sets, are both countable:

https://en.m.wikipedia.org/wiki/Cardinal_number

Long/short takeoff

I opted in but didn't get to play. Glad to see that it looks like people had fun! Happy Petrov Day!

Long duration/short duration takeoff