LessWrong 2.0 Reader

What about the incentives? PWC is apparently OpenAI's largest enterprise customer. I don't know how much PWC actually use the tools in-house and how much they use to on-sell "Digital Transformation" onto their own and new customers. How might this be affecting the way that OpenAI develop their products?

It's possible that from the authors perspective, the specific semantic meanings I took from terms like "automated alignment research" and "fleets" wasn't implied. But if I made the mistake, I'm sure other readers will as well, so I'd like to encourage changes here before these phrases take off much further (if others agree with my take.)

I'm happy this area is getting more attention.

I feel nervous about the terminology. I think that terminology can presuppose some specific assumptions about how this should or will play out, that I don't think are likely. 

"automating alignment research" -> I know this has been used before, it sounds very high-level to me. Like saying that all software used as part of financial trading workflows is "automating financial trading." I think it's much easier to say that software is augmenting financial trading or similar. There's not one homogeneous thing called "financial trading," the term typically emphasises the parts that aren't yet automated.  The specific ways it's integrated sometimes involve it replacing entire people, sometimes involve it helping people, and often does both in complex ways. 

"Algorithmic trading was not just about creating faster digital traders but about reimagining traders as fleets of bots, quants, engineers, and other specialists."
In software, the word fleet sometimes refers to specific deployment strategies. A whole lot of the automation doesn't look like "bots" - rather it's a lot of regular tools, plug-ins, helpers, etc.

"vast digital fleets of specialized AI agents working in concert"
This is one architecture we can choose, but I'm not sure how critical/significant it will be. I very much agree that AI will be a big deal, but this makes it sound like you're assuming a specific way for AI to be used. 

All that said, I'm very much in favor of us taking a lot of advantage of AI systems for all the things we want in the world, including AI safety. I imagine that for AI safety, we'll probably use a very eccentric and complex mix of AI technologies. Some with directly replace some existing researchers, we'll have specific scripts for research experiments, maybe agent-like things that do ongoing oversight, etc. 

Abilities/intelligence come almost entirely from pretraining, so all the situation awareness and scheming capability that current (and future similar) frontier models possess is thus also mostly present in the base model.

Yes, but for scheming, we care about whether the AI can self-locate itself as an AI using its knowledge. The fact that (at a minimum) sampling from the system is required for it to self-locate as an AI might make a big difference here.

So if your 'yes' above is agreeing that capabilities - including scheming - come mostly from pretraining, then I don't see how relevant it is whether or not that ability is actually used/executed much in pretraining, as the models we care about will go through post-training and I doubt you are arguing post-training will reliably remove scheming.

I also think it seems probably very hard to train a system capable of obsoleting top human experts which doesn't understand that it is an AI even if you're willing to take a big competitiveness hit.

Indeed but that is entirely the point - by construction!

Conceptually we have a recipe R (arch, algorithms, compute, etc), and a training dataset which we can parameterize by time cutoff T. Our objective (for safety research) is not to train a final agent, but instead to find a safe/good R with minimal capability penalty. All important results we care about vary with R independently of T, but competitiveness/dangerousness does vary strongly with T.

Take the same R but vary the time cutoff T of the training dataset: the dangerousness of the AI will depend heavily on T, but not the relative effectiveness of various configurations of R. That is simply a restatement of the ideal requirements for a safe experimental regime. Models/algos that work well for T of 1950 will also work for T of 2020 etc.

Since this post is about rebutting criticisms of AIXI, I feel it would be only fair to include Rob Bensinger's criticism [LW · GW]. I considered it to be the strongest criticism of AIXI by a mile. Do you have any rebuttals for that post?

Well put and I agree.

Karma is tricky as a measure because subreddits are non-stationary. In particular, I feel like the "vibes" of all the subreddits I listed were different 6+ months ago, and they are becoming more homogenous (in part due to power users such as Kat Woods). I don't know of a way to view what the "hot" page of any given subreddit would have looked like at some previous point in time, so it's hard to find data to understand subreddit culture drift. Anyway, the high karma is also consistent with selection effects, where the users who do not like this content bounce off, and only the users that do stick around those subreddits in the long term. 

Typically I agree with the underlying facts behind her memes! For example I also think AI safety is a pressing issue. If her memes were funny I would instead be writing a post about how awesome it is that Kat Woods is everywhere. My main objection is that I do not like the packaging of the ideas she is spreading. For example the memes are not funny. (See the outline of this post: content, vibes, conduct.) 

You asked for an example of Kat Woods content that aims to convince rather than educate. Here is one recent example. I feel like the packaging of this meme conveys: "all of the objections you might have to the idea of X-risk via AI can actually be easily be debunked, therefore you would be stupid to not believe X-risk via AI". 

In reality, questions regarding likelihood of x-risk via AI are really tricky. Many thoughtful people have thought about these problems at great length and declared them to be hard and full of uncertainty. I feel like this meme doesn't convey this at all. Therefore, I'm not sure whether it is good for peoples' brains to consume this content. I will certainly say it's not good for my brain to consume this content.

I asked a leading question of our "Perspectives" system, and it gave a few hundred hypothetical reasons (here)

I have my own theories about the intentions which I do not feel comfortable discussing, so I'll focus on the practicalities and case studies which show why this complex and difficult to execute:
some hostages have been killed by the IDF during rescue operations, this isn't uncommon, the lone hostage was killed during a French raid in Somalia, consider the Lindt Cafe Siege in Sydney where a pregnant hostage was killed by ricocheting police bullet fire when they finally stormed in, three other hostages and a policeman were injured. This was a lone gunman, I can imagine that the Hamas hostage takers are well organized groups. A hostage during the Gladbeck Crises in Germany were also injured by police fire. 

Kidnapping someone who "knows" the location of some hostages I would guess is highly ineffective for many reasons, Torture is a notoriously inaccurate source of information: hence the propensity for false admissions or telling interrogators what they want to hear. That and I suspect that there is a intentional system of moving around hostages from place to place, and never explicitly sharing locations with others to minimize the risk of locations leaking.

If someone who knows the exact location of a hostage has not been heard from for 24 hours, it is probably a good idea to move to a new location anyway.

Finally there is the incredible danger to the IDF soldiers themselves going into a dynamic environment where they don't know how much resistance they will encounter, being expected to minimize the harm to hostages while almost certainly coming under fire. It's probable suicide.

Thank you. So what do you think the cause of that is, and why do you think that cause exists and will it always exist?