LessWrong 2.0 Reader

There's also a comic series with explicitly this premise, unfortunately this is a major plot point so revealing it will be a spoiler:

I appreciate the brevity of the title as it stands. It's normal for a title to summarize the thesis of a post or paper and this is also standard practice on LessWrong. For example:

• The sun is big but superintelligences will not spare the Earth a little sunlight [LW · GW].
• The point of trade [LW · GW]
• There's no fire alarm for AGI [LW · GW]

The introductory paragraphs sufficiently described the epistemic status of the author for my purposes. Overall, I found the post easier to engage with because it made its arguments without hedging.

I already think that "the entire shape of the zeitgeist in America" is downstream of non-trivial efforts by more than one state actor. Those links explain documented cases of China and Russia both trying to foment race war in the US, but I could pull links for other subdimensions of culture (in science, around the second amendment, and in other areas) where this has been happening since roughly 2014.

My personal response is to reiterate over and over in public that there should be a coherent response by the governance systems of free people, so that, for example, TikTok should either (1) be owned by human people who themselves have free speech rights and rights to a jury trial, or else (2) should be shut down by the USG via taxes, withdrawal of corporate legal protections, etc...

...and also I just track actual specific people, and what they have personally seen and inferred and probably want and so on, in order to build a model of the world from "second hand info".

I've met you personally, Jan, at a conference, and you seemed friendly and weird and like you had original thoughts based on original seeing, and so even if you were on the payroll of the Russians somehow... (which to me clear I don't think you are) ....hey: Cyborgs [AF · GW]! Neat idea! Maybe true. Maybe not. Maybe useful. Maybe not.

Whether or not your cyborg ideas are good or bad can be screened off [LW · GW] from whether or not you're on the payroll of a hostile state actor. Basically, attending primarily to local validity [LW · GW] is basically always possible, and nearly always helpful :-)

I think you can do some steelmanning of the anti-flippers with something like Lara Buchak's arguments on risk and rationality. Then you'd be replacing the vague "the utility maximizing policy seems bad" argument with a more concrete "I want to do population ethics over the multiverse" argument.

I appreciate the clarity of the pixel game as a concrete thought experiment. Its clarity makes it easier for me to see where I disagree with your understanding of the Natural Abstraction Hypothesis.

The Natural Abstraction Hypothesis is about the abstractions available in Nature, that is to say, the environment. So we have to decide where to draw the boundary around Nature. Options:

• Nature is just the pixel game itself (Cartesian)
• Nature is the pixel game and the agent(s) flipping pixels (Embedded)
• Nature if the pixel game and the utility function(s) but not the decision algorithms (Hybrid)

In the Cartesian frame, none of "top half", "bottom half", "outer rim", and "middle square" are all Unnatural Abstractions, because they're not in Nature, they're in the utility functions.

In the Hybrid and Embedded frames, when System A is playing the game, then "top half" and "bottom half" are Natural Abstractions, but "outer rim" and "middle square" are not. The opposite is true when System B is playing the game.

Let's make this a multi-player game, and have both systems playing on the same board. In that case all of "top half", "bottom half", "outer rim", and "middle square" are Natural Abstractions. We expect system A to learn "outer rim" and "middle square" as it needs to predict the actions of system B, at least given sufficient learning capabilities. I think this is a clean counter-example to your claim:

Two systems require similar utility functions in order to converge on similar abstractions.

I'm from Hungary that is probably politically the closest to Russia among Central European countries, but I don't really know of any significant figure who turned out to be a Russian asset, or any event that seemed like a Russian intelligence operation. (Apart from one of our far-right politicians in the EU Parliament being a Russian spy, which was a really funny event, but its not like the guy was significantly shaping the national conversation or anything, I don't think many have heard of him before his cover was blown.) What are prominent examples in Czechia or other Central European countries, of Russian assets or operations?

I want to flag this as an assumption that isn't obvious. If this were true for the problems we care about, we could solve them by employing a lot of humans.

humans provides a pretty strong intuitive counterexample

Yup not obvious. I do in fact think a lot more humans would be helpful. But I also agree that my mental picture of "transformative human level research assistant" relies heavily on serial speedup, and I can't immediately picture a version that feels similarly transformative without speedup. Maybe evhub or Ethan Perez or one of the folks running a thousand research threads at once would disagree.

yeah, the moment i looked at the big diagram my brain sort of pleasantly overheated

such plans are fairly easy and don't often raise flags that indicate potential failure

Hmm. This is a good point, and I agree that it significantly weakens the analogy.

I was originally going to counter-argue and claim something like "sure total failure forces you to step back far but it doesn't mean you have to step back literally all the way". Then I tried to back that up with an example, such as "when I was doing alignment research, I encountered total failure that forced me to abandon large chunks of planning stack, but this never caused me to 'spill upward' to questioning whether or not I should be doing alignment research at all". But uh then I realized that isn't actually true :/

We want particularly difficult work out of an AI.

On consideration, yup this obviously matters. The thing that causes you to step back from a goal is that goal being a bad way to accomplish its supergoal, aka "too difficult". Can't believe I missed this, thanks for pointing it out.

I don't think this changes the picture too much, besides increasing my estimate of how much optimization we'll have to do to catch and prevent value-reflection. But a lot of muddy half-ideas came out of this that I'm interested in chewing on.

There's a regularization problem to solve for 3.9 and 4, and it's not obvious to me that glee will be enough to solve it (3.9 = "unintelligible CoT").

I'm not sure how o1 works in detail, but for example, backtracking (which o1 seems to use) makes heavy use of the pretrained distribution to decide on best next moves. So, at the very least, it's not easy to do away with the native understanding of language. While it's true that there is some amount of data that will enable large divergences from the pretrained distribution - and I could imagine mathematical proof generation eventually reaching this point, for example - more ambitious goals inherently come with less data, and it's not obvious to me that there will be enough data in alignment-critical applications to cause such a large divergence.

There's an alternative version of language invention where the model invents a better language for (e.g.) maths then uses that for more ambitious projects, but that language is probably quite intelligible!