LessWrong 2.0 Reader

My question is why is the following statement below true, exactly?

Second, there's no need to claim that commonsense moral side constraints in particular are so hard that trying to develop AI systems that respect them leads to policy collapse. It need only be the case that one of the things we try to teach them to do leads to policy collapse. 

What this post is trying to illustrate is that if you try putting crisp physical predicates on reality, that won't work to say what you want.  This point is true!

Matthew is not disputing this point, as far as I can tell.

Instead, he is trying to critique some version of^[1] the "larger argument" (mentioned in the May 2024 update to this post) in which this point plays a role.

You have exhorted him several times to distinguish between that larger argument and the narrow point made by this post:

[...] and if you think that some larger thing is not correct, you should not confuse that with saying that the narrow point this post is about, is incorrect [...]

But if you're doing that, please distinguish the truth of what this post actually says versus how you think these other later clever ideas evade or bypass that truth.

But it seems to me that he's already doing this.  He's not alleging that this post is incorrect in isolation.

The only reason this discussion is happened on the comments of this post at all is the May 2024 update at the start of it, which Matthew used as a jumping-off point [LW(p) · GW(p)] for saying "my critique of the 'larger argument' [LW · GW] does not make the mistake referred to in the May 2024 update^[2], but people keep saying it does^[3], so I'll try restating that critique again in the hopes it will be clearer this time."

1. ^{^}

   I say "some version of" to allow for a distinction between (a) the "larger argument" of Eliezer_2007's which this post was meant to support in 2007, and (b) whatever version of the same "larger argument" was a standard MIRI position as of roughly 2016-2017.
   
   As far as I can tell, Matthew is only interested in evaluating the 2016-2017 MIRI position, not the 2007 EY position (insofar as the latter is different, if it fact it is).  When he cites older EY material, he does so as a means to an end – either as indirect evidence of later MIRI positions, because it was itself cited in the later MIRI material which is his main topic.

2. ^{^}

   Note that the current version of Matthew's 2023 post [LW · GW] includes multiple caveats that he's not making the mistake referred to in the May 2024 update.
   
   Note also that Matthew's post only mentions this post in two relatively minor ways, first to clarify that he doesn't make the mistake referred to in the update (unlike some "Non-MIRI people" who do make the mistake), and second to support an argument about whether "Yudkowsky and other MIRI people" believe that it could be sufficient to get a single human's values into the AI, or whether something like CEV would be required instead.

   I bring up the mentions of this post in Matthew's post in order to clarifies what role "is 'The Hidden Complexity of Wishes' correct in isolation, considered apart from anything outside it?" plays in Matthew's critique – namely, none at all, IIUC.

   (I realize that Matthew's post has been edited over time, so I can only speak to the current version.)

3. ^{^}

   To be fully explicit: I'm not claiming anything about whether or not the May 2024 update was about Matthew's 2023 post [LW · GW] (alone or in combination with anything else) or not. I'm just rephrasing what Matthew said in the first comment of this thread (which was also agnostic on the topic of whether the update referred to him).

Noting that I read this (and that therefore you get partial credit for any solution I come up with from here on out): your model and the strategies it implies are both very interesting. I should be able to investigate them with ML alongside everything else, when/if I get around to doing that.

Regarding the Bonus Objective:

My very naive baseline for questions like this is "large effects are easily measured", and the contrapositive "if it's hard to measure, the effect is small".  Can you explain why this isn't the case for far-UV? Also, what are the reasons there doesn't seem to be much ground-up interest?  Why aren't companies and homeowners installing it in such numbers that it becomes standard?

I tried asking ChatGPT, Gemini, and Claude to come up with a formula that converts between correlation space to probability space while preserving the relationship 0 = 1/n. I came up with such a formula a while back, so I figure it shouldn't be hard. They all offered formulas, all of which were shown to be very much wrong when I actually graphed them to check.

probabilities do not depend on the purpose. 

I think this is a restatement of the crux.  OF COURSE the model chosen depends on the purpose of the model.  For probabilities, the choice of reference class for a given prediction/measurement is key.  For Sleeping Beauty specifically, the choice of whether an experientially-irrelevant wakening (which is immediately erased and has no impact) is distinct from another is a modeling choice.

Either choice for probability modeling can answer either wagering question, simply by applying the weights to the payoffs if it's not already part of the probability 

Grants to Redwood Research, SERI MATS, NYU alignment group under Sam Bowman for scalable supervision, Palisade research, and many dozens more, most of which seem net positive wrt TAI risk.

I think it's important to be able to make a narrow point about outer alignment without needing to defend a broader thesis about the entire alignment problem.

Indeed. For it is written [LW · GW]:

A mind that ever wishes to learn anything complicated, must learn to cultivate an interest in which particular exact argument steps are valid, apart from whether you yet agree or disagree with the final conclusion, because only in this way can you sort through all the arguments and finally sum them.

For more on this topic see "Local Validity as a Key to Sanity and Civilization [? · GW]."

I took an analytic approach and picked some reasonable choices based on that. I'll almost certainly try throwing ML at this problem some point but for now I want to note down what a me-who-can't-use-XGBoost would do.

Findings:

Who fights whom?

Who wears what?

Your article is a great read! 

In my view, we can categorize scientists into two broad types: technician scientists, who focus on refining and perfecting existing theories, and creative scientists, who make generational leaps forward with groundbreaking ideas. No theory is ever 100% correct—each is simply an attempt to better explain a phenomenon in a way that’s useful to us.

Take Newton, for example. His theory of gravity was revolutionary, introducing concepts no one had thought of before—it was a generational achievement. But then Einstein came along, asking why objects with mass attract one another. Newton's equations could predict gravitational forces accurately, but they didn’t explain the underlying cause. Einstein’s theory of relativity made a creative leap, adding a new dimension by introducing space-time as part of the explanation. It provided a more accurate theoretical representation of gravity and broadened our understanding of mass and space-time, marking yet another generational leap.

Then we have Oppenheimer, who refined existing theories in physics to develop the atomic bomb. While his work was groundbreaking, it was more a refinement of known principles rather than a creative leap like Einstein’s. I would classify Oppenheimer as more of a "technician scientist" than a "creative" one, although I defer to experts in physics, as it's not my field.

Regarding your article, I really enjoyed it. On the topic of the "Tiger in my house" question, the answer depends on how we define "tiger." If you mean a living, biological tiger, the answer is clearly no. But if you mean any type of tiger, such as a toy tiger, then the answer could be yes. This same creative reasoning can apply to the question about water in the fridge. Even if there’s no glass of water, the air in the fridge likely contains humidity, meaning there’s always some amount of water present.

Lastly, regarding the diagram with lines, curves, and points: one creative way to introduce ambiguity into this solidly two-dimensional example is by adding more dimensions. In a strictly two-dimensional projection, the relationships between points and lines as stated hold true. However, if a third spatial dimension or a fourth dimension (such as time) is introduced, the axioms that govern the two-dimensional model might no longer apply. For example, the rule that exactly one line passes through two points could be violated in a three-dimensional space, where two points might lie on different planes. Therefore, while the statements about the model are accurate within the context of two-dimensional space, they may no longer hold when additional dimensions are considered.