LessWrong 2.0 Reader

I think this is correct in alignment-is-easy worlds but incorrect in alignment-is-hard worlds (corresponding to "optimistic scenarios" and "pessimistic scenarios" in Anthropic's Core Views on AI Safety). Logic like this is a large part of why I think there's still substantial existential risk even in alignment-is-easy worlds, especially if we fail to identify that we're in an alignment-is-easy world. My current guess is that if we were to stay exclusively in the pre-training + small amounts of RLHF/CAI paradigm, that would constitute a sufficiently easy world [? · GW] that this view would be correct, but in fact I don't expect us to stay in that paradigm, and I think other paradigms involving substantially more outcome-based RL (e.g. as was used in OpenAI o1) are likely to be much harder, making this view no longer correct.

There are mechanisms where fluoride goes directly from the mouth onto the surface of the teeth. There are also mechanisms where fluoride goes from the bloodstream into teeth.

The fluoride that goes directly from the mouth to the surface of the teeth seems clearly good for caries prevention at low side effects.

When it comes to the fluoride that goes through the stomach and blood supply, it's unclear to me whether that provides a benefit for caries prevention when you already have sufficient fluoride through toothpaste in the mouth. The side effects also seem unclear to me.

oh okay, i'll have to reinterpret then. edit: i just tried, but i still don't get it; if it's "very strongly superhuman", why is it merely "when the economy starts getting seriously disrupted"? (<- this feels like it's back at where this thread starte

I should probably edit that at some point, but I'm on my phone, so I'll do it tomorrow.

why?

A big reason for this is logistics, as how you are getting to the fight can actually hamper you a lot, and this especially bites hard on offense, because it's easier to get supplies to your area than it is to get supplies to an offensive unit.

This especially matters if physical goods need to be transported from one place to another place.

Thanks very much for your feedback, though I confess I'm not entirely sure where to go with it. My interpretation is you have basically two concerns:

1. This policy doesn't really directly regulate algorithmic progress, e.g. if it happened on smaller amounts of compute.
2. Algorithmic theft/leakage is easy.

The first one is true, as I alluded in the problems section. Part of my perspective here is coming from a place of skepticism about regulatory competence - I basically believe we can get regulators to control total compute usage, and to evaluate specific models according to pre-established evals, but I'm not sure I'd trust them to be able to determine "this is a new algorithmic advance, we need to evaluate it". To the extent you had less libertarian priors you could try to use something like the above scheme for algorithms as well, but I wouldn't expect it to work so well, as you lack the cardinal structure of compute size.

In terms of theft/leakage, you're right this plan doesn't discuss it much, and I agree it's worth working on. 

I think a crux I have with the entire alignment community may ultimately come down to me not believing that human values overlap strongly enough to make alignment the most positive thing, compared to other AI safety things.

In particular, I'd expect a surprising amount of disagreement on whether making a hell is good, if you managed to sell it as eternally punishing a favored enemy.

most of the reason for collaborating on alignment despite orthogonality is that our values tend to overlap to a large degree, e.g. most people (and maybe especially most alignment researchers?) think hells are bad.

I agree LWers tend to at least admit that severe enough value conflicts can exist, though I think that people like Eliezer don't realize that human values conflicts sort of break collective CEV type solutions, and a lot of collective alignment solutions tend to assume that either someone puts their thumb on the scale and exclude certain values, or assume that human values are so similar and their idealizations are so similar that no conflicts are expected, which I personally don't think is true.

i don't know where that might be true, but at least on lesswrong i imagine it's an uncommon belief. a core premise of alignment being important is value orthogonality implying that an unaligned agent with max-level-intelligence would compete for the same resources whose configurations it values (the universe).

also on the "lose out" phrasing: even if someone "wants at least some people to have tormentful lives", they don't "lose out" overall if they also positively value other things / still negatively value any of the vast majority of beings having tormentful lives.

Agree with this, which handles some cases, but my worry is that there are still likely to be big values conflicts where one value set must ultimately win out over another.

I was assuming very strongly superhumanly intelligent AI

oh okay, i'll have to reinterpret then. edit: i just tried, but i still don't get it; if it's "very strongly superhuman", why is it merely "when the economy starts getting seriously disrupted"? (<- this feels like it's back at where this thread started)

I think the offense-defense balance moderately favors defense even at optimality

why?

I was assuming very strongly superhumanly intelligent AI, but yeah no promises of optimality were made here.

That said, I suspect a crux is that optimality ends up with multipolarity, assuming a one world government hasn't happened by then, because I think the offense-defense balance moderately favors defense even at optimality, assuming optimal defense and offense.

Wow, thanks for doing the legwork on this - seems like quite possibly I'm analyzing fiction? Annoying if true.

Google's AI response to my search for the Thaba-Tseka Development Project says:

According to available World Bank documentation, the "Thaba-Tseka development project" is primarily referenced within the context of the "Lesotho Integrated Transport, Trade and Logistics Project," which focuses on improving the road corridor connecting Katse to Thaba-Tseka, aiming to enhance regional connectivity and reduce trade costs at Lesotho's borders with South Africa; key documents to reference would be those related to this project, particularly those detailing the road infrastructure development component between Katse and Thaba-Tseka. 

Key points about the documentation: 

• Project Title: "Lesotho Integrated Transport, Trade and Logistics Project"
• Focus Area: Upgrading the Katse to Thaba-Tseka road corridor
• Objectives: Improve climate resilient regional connectivity, reduce trade costs at Lesotho's borders
• Relevant documents to explore: Project Appraisal Documents, Procurement documents related to road construction and improvement on the Katse-Thaba-Tseka stretch 

There's a good chance this is an AI hallucination, though; a cursory search of the main documents didn't yield any references to a "Thaba-Tseka development project," or the wood or ponies. I'm not familiar with World Bank documentation, though, and likely the right followup would involve looking at exactly what's cited in the book.

However, the other lead funder, the Canadian International Development Agency, does seem to have at least one publicly referenced document about a "Thaba-Tseka rural development program": Evaluation, the Kingdom of Lesotho rural development : evaluation design for phase 1, the Thaba Tseka project

Sigh. Ok. I'm giving an upvote for good-faith effort to think this through and come up with a plan, but I just disagree with your world-model and its projections about training costs and associated danger levels so strongly that it seems hard to figure out how to even begin a discussion.

I'll just leave a link here [LW(p) · GW(p)] to a different comment talking about the same problem.

but it still says "it's easy for others to get their own superintelligences with different values", with 'superintelligence' referring to the 'superhuman' AI of 2035?

still confused about this btw. in my second reply to you i wrote:

(i wonder if you're using the term 'superintelligence' in a different way though, e.g. to mean "merely super-human"?)

and you did not say you were, but it looks like you are here?