LessWrong 2.0 Reader

Basically people tend to value stuff they perceive in the biophysical environment and stuff they learn about through the social environment.

So that reduces the complexity of the problem - it’s not a matter of designing a learning algorithm that both derives and comes to value human abstractions from observations of gas particles or whatever. That’s not what humans do either.

Okay then, why aren’t we star-maximizers or number-of-nation-states maximizers? Obviously it’s not just a matter of learning about the concept. The details of how we get values hooked up to an AGI’s motivations will depend on the particular AGI design but probably reward, prompting, scaffolding or the like.

If that’s your hope—then you should already be alarmed at trends

Would be nice for someone to quantify the trends. Otherwise it may as well be that trends point to easygoing enough and aligned enough future systems.

For some humans, the answer will be yes—they really would do zero things!

Nah, it's impossible for evolution to just randomly stumble upon such complicated and unnatural mind-design. Next you are going to say what, that some people are fine with being controlled?

Where an entity has never had the option to do a thing, we may not validly infer its lack of preference.

Aha, so if we do give the option to an entity and it doesn't always kills all humans, then we have evidence it cares, right?

If there is a technical refutation it should simplify back into a nontechnical refutation.

Wait, why prohibiting successors would stop OpenAI from declaring easygoing system a failure? Ah, right - because there is no technical analysis, just elements of one.

Potential for o1-like long horizon reasoning post-training compromises capability evals for open weights models. If it's not applied before/during evals, then evals will significantly underestimate capabilities of the system that can be built out of the open weights later [LW · GW], when the recipe is reproduced.

This likely will be relevant for Llama 4 (as the first 1e26+ FLOPs model with open weights), if they don't manage a good reproduction of o1-like post-training before release, and continue the policy of publishing in open weights if the evals are not too alarming.

There's a billion reasonable-seeming impact metrics, but the main challenge of counterfactual-based impact is always how you handle chaos. I'm pretty sure the solution is to go away from counterfactuals as they represent a pathologically computationalist form of agency [LW · GW], and instead learn the causal backbone [LW · GW].

If we view the core of life as increasing rather than decreasing entropy [LW · GW], then entropy-production may be a reasonable candidate for putting quantitative order to the causal backbone. But bounded agency is less about minimizing impact and more about propagating the free energy of the causal backbone into new entropy-producing channels.

The LessWrong Review [? · GW] runs every year to select the posts that have most stood the test of time. This post is not yet eligible for review, but will be at the end of 2025. The top fifty or so posts are featured prominently on the site throughout the year.

Hopefully, the review is better than karma at judging enduring value. If we have accurate prediction markets on the review results, maybe we can have better incentives on LessWrong today. Will this post make the top fifty?

I feel like the difference between the Alpha and Beta examples and my examples mediate through your examples having basically no control of Beta's data at all, and my examples having far more control over what data is learned by the AI.

I think the key crux is whether we have much more control over AI data sources than evolution.

If I agreed with you that we would have essentially no control on what data the AI has, I'd be a lot more worried, but I don't think this is true, and I expect future AIs, including AGIs, to be a lot more built than grown, and for a lot of their data to be very carefully controlled via synthetic data, for simple capabilities reasons, but this can also be used for alignment strategies.

I think another disagreement is I basically don't buy the evolution analogy for DL, and I think there are some deep disanalogies (the big one for now is again how much more control over data sources than evolution, and this is only set to increase with synthetic data).

So I basically don't expect this to happen:

I instead strongly expect that the story would just repeat. The training process (or whatever process spits out the AGI) would end up creating some extremely specific conditions in which the AGI is learning the values. Its values would then necessarily be some complicated functions over weird mixes of the abstractions-natural-to-the-dataset-it's-trained-on, with their specifics being highly contingent on some invisible-to-us details of that process.

Pretty much all of your examples rely on the Alpha being unable to control the data learnt by Beta, and if this isn't the case, your examples break down.

Somehow I missed that bit.

That makes the situation better, but there’s still some issue. The refund is not earning interest, but you liabilities are.

Take the situation with owing $25 million. Say that there’s a one year time between the tax being assessed and your asset going to $0 (at which time you claim the refund). In this time the $25 million loan you took is accruing interest. Let’s say it does so at a 4% rate per year, when you get your $25 million refund you therefore have $26 million in loans.

So you still end up $1 million in debt due to “gains” that you were never able to realize.

@Ben Pace [LW · GW] I would like a vote here on what percentage chance we think that an omnicient reviewer would say this narrative is true. The display it on an axis, probably with dots (anonymous) for each person. eg like this. 

This whole post you keep referencing "the Hegelian dialectic" as some sort of thing central to the Village, without ever stating what you think it is; could you elaborate?

I'm not an expert in this area by any means but I do not have the impression that the Village has even a plurality of Hegelians, let alone being centered around them, and the way you talk about the Hegelian dialectic does not seem to be remotely the same usage that you see with actual Hegelians. Honestly I expect actual Hegelians to be rather less upset about the River's differences from them, because, to my understanding, Hegel's whole thing is he believes in a sort of teleology of history that guarantees the victory of human freedom?

strong disagree. i would be highly surprised if there were multiple essentially different algorithms to achieve general intelligence*. 

I also agree with the Daniel Murfet's quote. There is a difference between a disjunction before you see the data and a disjunction after you see the data. I agree AI development is disjunctive before you see the data - but in hindsight all the things that work are really minor variants on a single thing that works. 

*of course "essentially different" is doing a lot of work here. some of the conceptual foundations of intelligence haven't been worked out enough (or Vanessa has and I don't understand it yet) for me to make a formal statement here.