LessWrong 2.0 Reader

I was also struck by this huge discrepancy between HLMI and FAOL predictions. I think that particularly FAOL predictions are unreliable. My interpretation is that when respondents are pushing their timelines so far into the future, some of them may be in fact attempting to resist admitting the possibility of AI takeover.

The key question is, what "automating all tasks" really means. "All tasks" includes in particular also all decision making: managerial, political, strategic, the small and the large, all of that. All the agency, long-term planning, and execution of one's own plans. Automating all tasks in fact implies AI takeover. But just considering this possibility may then easily clash with the view that many people have, namely that AIs are controllable tools rather than uncontrollable agents (see the excellent new paper by Severin Field on this). 

And there will for sure be strong competitive forces pushing towards full automation, once that option becomes technically feasible. For example, if you automate production processes in a firm at all levels up to the CEO, but not the CEO, then the human CEO becomes a bottleneck, slowing down the firm's operations, potentially by orders of magnitude. Your firm may then be pushed out of the market by a competitor who automated their CEO as well.

My logic suggests that FAOL should be only slightly later than HLMI. Of course you should have first feasibility, then adoption. Some lag could follow from cost considerations (AI agents / robot actuators may be initially too expensive) or legal constraints, and perhaps also human preferences (though I doubt that point). But once we have FAOL, we have AI takeover - so in fact such scenario redirects our conversation to the topic of AI x-risk.  

the set of problems the solutions to which are present in their training data

a.k.a. the set of problems already solved by open source libraries without the need to re-invent similar code?

Last year you guys wrote a post on adult intelligence enhancement.  Does any of your research on super babies have implications for that, especially radical enhancement?

Ok, I've been thinking about this more and I've figured out what exactly was bugging me about this red-teaming scheme. Is it fair to say you assume that your red team will naturally cover the same area of solution-space as actual malicious AIs aiming to self-coordinate?

If yes, I think that assumption is faulty. Self-coordinating AIs will have access to classes of strategies that a human red-teamer would not think of unless they spent significant effort to "think like the enemy", so to speak. These strategies will be usable even if the AI isn't much smarter than a human, simply because the AI is a very different kind of mind.

Self-coordination might also be easier if AIs can recognize themselves by being good at recognizing features that survive paraphrasing and that vary very consistently between answers [... but] I would bet against it

As a simple but hopefully demonstrative example: if the AI is asked to generate a list of items, it could communicate information based on the order of items in the list. Given that the AI is specifically attempting to self-coordinate, it can assume its other copy will come up with the same mapping between sequences and secret signals.^[1] This easily survives sentence-level paraphrasing, even though it's obviously defeated by shuffling the list.

There are plausibly very many strategies in this vein, where you encode information at the structural rather than the content level. So even if each individual strategy is trivial to defeat, you could just end up playing strategy whack-a-mole.

1. ^{^}

   The information density of this is low though - the AI can't assign meaningful information to too many sequences, since a synthetic input could then generate one by chance, so it can only make a sparse subset of sequences meaningful.

Hmm. Yeah. It gets difficult to display points with the same XY coordinates and different RGB coordinates

I think I'm also around 60-70% for the rough overall picture in the OP being correct.

Or maybe you're arguing "don't develop any technology" or "don't develop any powerful technology" because "governments might misuse it". That's something you could reasonably argue, but I think you should just argue that in general if that's what you're saying, so the case is clearer.

It seems to me that "vibe checks" for how smart a model feels are easily gameable by making it have a better personality.

It's not clear to me that personality is completely separate from capabilities, especially with inference time reasoning.

Also, what do you mean by "bigger templates"?

This post is a link post that links to the TheWaywardAxolotl. I figured that more people would read the essay if the text was displayed directly on LessWrong, so I copied the text from the blog post and pasted it into the LessWrong post. The author has said that he doesn't mind people doing this as long as they include a link back to his blog. I'm sure I'm not the only one on this forum who shares content that I didn't write myself. It's pretty normal to do that on social media.

There's no way for anyone to know that you didn't write the essay unless they already know that your username isn't an alias of the writer.

My user profile has a link to zerocontradictions.net, which is clearly my website. That URL is shown when anybody hovers over my username on this site. This link post links to the TheWaywardAxolotl, which is clearly a different website. The username on that blog is also displayed as "Blithering Genius", not "Zero Contradictions". There is nothing on Blithering Genius's blog to suggest that I own it, so it's weird that Purple fire is jumping to the assumption that I wrote the essay.

making the normal assumption that the only person whose username is attached to the post is who "I" refers to.

Yes, that's how the essay appears on TheWaywardAxolotl, which is clearly not my blog. It's also how it appears on this LessWrong post, since I copied the text into the post. "I" and "we" are used throughout the essay, so it's not easy to edit out the first-person language. But since you insist, I edited the post to include a disclaimer at the top that I didn't write the essay and I'll do that in the future as well.

let alone that you don't endorse it

I agree with the essay, but I'm also open-minded, so I shared it to see what other people would say about it, because it's possible that some people have knowledge and thoughts that I hadn't thought about. I think that purple fire made some interesting criticisms which might be right, so I told him/her that he/she should post them on the author's blog if he/she wants the author to read them.

I didn't downvote any of your comments, and I don't see any upthread comments with any downvotes!

Anyway, you could steelman your case if you like. It might help if you compared to other technologies, like "We should develop powerful thing X but not superficially similar powerful thing Y because X is much worse given that there are governments", or something.