LessWrong 2.0 Reader

Midwits are often very impressed with themselves for knowing a fancy economic rule like Ricardo's Law of Comparative Advantage!

Could we have less of this sort of thing, please? I know it's a crosspost from another site with less well-kept discussion norms, but I wouldn't want this to become a thing here as well, any more than it already has.

Yesterday was the official application deadline for leading a project at the next AISC. This means that we just got a whole host of project proposals. 

If you're interested in giving feedback and advise to our new research leads, let me know. If I trust your judgment, I'll onboard you as an AISC advisor.

Also, it's still possible to send us a late AISC project proposals. However we will prioritise people how applied in time when giving support and feedback. Further more, we'll prioritise less late applications over more late applications. 

Yudkowsky's point about trying to sell an Oreo for $77 is that a billionaire isn't automatically going to want to buy something off you if they don't care about it (and neither would an ASI).

I thought Yudkowsky's point was that the billionaire won't give you $77 for an Oreo because they could get an Oreo for less than $77 via other means. But people don't just have an Oreo to sell you. My point in that sentence was to bring up that workers routinely have things of value that they can sell for well over $77, even to billionaires. Similarly, I claim that Yudkowsky did not adequately show that humans won't have things of substantial value that they can sell to future AIs.

I'm not sure anyone is arguing that smart AIs would immediately turn violent unless it was in their strategic interest

The claim I am disputing is precisely that it will be in the strategic interest of unaligned AIs to turn violent and steal from agents that are less smart than them. In that sense, I am directly countering a claim that people in these discussions routinely make.

Depending on your devops pipeline (that is, how often do you reimage with DD, as opposed to updating in place, and what changes you make AFTER imaging), you might consider a few timesaving options:

1. small root partition, rest as a data partition.  This lets you image only the root with dd.  It's probably FASTER to image a 4GB partition of a 32GB SDCARD than to image a 4GB card directly.  
2. Go smaller than 4GB (by partition, you probably don't want to buy smaller cards).  RPiOsLite should install in under 1GB, and a very usable commandline system with space for updates in 2GB.   If you only have 300MB free in a 4GB storage, something is wrong.  It may not be worth fiddling with, though.
   For multiple copies of a master, it's common to have a large/fast setup that you install, update, tune, and prepare on, then prepare the image for distribution.  There are lots of options for that, the simplest is to clean and shrink the filesystem, and use that shrunken filesystem as the image to distribute.
3. use a USB "real" SSD, not an SDCARD.  Rpi4 and above can boot from it, earlier models need to boot from SDCARD, but can have root and all data on usb.  this can be MASSIVELY faster and somewhat more reliable than sdcard.
4. Probably not worth the effort, but PiLFS or Yocto make it possible to build a "custom application distribution" for your pi.  It will contain only the packages you need, and needs no update mechanism on the devices - to do an update, you rebuild and re-distribute.  It can be TINY and still quite functional.

Basically people tend to value stuff they perceive in the biophysical environment and stuff they learn about through the social environment.

So that reduces the complexity of the problem - it’s not a matter of designing a learning algorithm that both derives and comes to value human abstractions from observations of gas particles or whatever. That’s not what humans do either.

Okay then, why aren’t we star-maximizers or number-of-nation-states maximizers? Obviously it’s not just a matter of learning about the concept. The details of how we get values hooked up to an AGI’s motivations will depend on the particular AGI design but probably reward, prompting, scaffolding or the like.

If that’s your hope—then you should already be alarmed at trends

Would be nice for someone to quantify the trends. Otherwise it may as well be that trends point to easygoing enough and aligned enough future systems.

For some humans, the answer will be yes—they really would do zero things!

Nah, it's impossible for evolution to just randomly stumble upon such complicated and unnatural mind-design. Next you are going to say what, that some people are fine with being controlled?

Where an entity has never had the option to do a thing, we may not validly infer its lack of preference.

Aha, so if we do give the option to an entity and it doesn't always kills all humans, then we have evidence it cares, right?

If there is a technical refutation it should simplify back into a nontechnical refutation.

Wait, why prohibiting successors would stop OpenAI from declaring easygoing system a failure? Ah, right - because there is no technical analysis, just elements of one.

Potential for o1-like long horizon reasoning post-training compromises capability evals for open weights models. If it's not applied before/during evals, then evals will significantly underestimate capabilities of the system that can be built out of the open weights later [LW · GW], when the recipe is reproduced.

This likely will be relevant for Llama 4 (as the first 1e26+ FLOPs model with open weights), if they don't manage a good reproduction of o1-like post-training before release, and continue the policy of publishing in open weights if the evals are not too alarming.

There's a billion reasonable-seeming impact metrics, but the main challenge of counterfactual-based impact is always how you handle chaos. I'm pretty sure the solution is to go away from counterfactuals as they represent a pathologically computationalist form of agency [LW · GW], and instead learn the causal backbone [LW · GW].

If we view the core of life as increasing rather than decreasing entropy [LW · GW], then entropy-production may be a reasonable candidate for putting quantitative order to the causal backbone. But bounded agency is less about minimizing impact and more about propagating the free energy of the causal backbone into new entropy-producing channels.

The LessWrong Review [? · GW] runs every year to select the posts that have most stood the test of time. This post is not yet eligible for review, but will be at the end of 2025. The top fifty or so posts are featured prominently on the site throughout the year.

Hopefully, the review is better than karma at judging enduring value. If we have accurate prediction markets on the review results, maybe we can have better incentives on LessWrong today. Will this post make the top fifty?

I feel like the difference between the Alpha and Beta examples and my examples mediate through your examples having basically no control of Beta's data at all, and my examples having far more control over what data is learned by the AI.

I think the key crux is whether we have much more control over AI data sources than evolution.

If I agreed with you that we would have essentially no control on what data the AI has, I'd be a lot more worried, but I don't think this is true, and I expect future AIs, including AGIs, to be a lot more built than grown, and for a lot of their data to be very carefully controlled via synthetic data, for simple capabilities reasons, but this can also be used for alignment strategies.

I think another disagreement is I basically don't buy the evolution analogy for DL, and I think there are some deep disanalogies (the big one for now is again how much more control over data sources than evolution, and this is only set to increase with synthetic data).

So I basically don't expect this to happen:

I instead strongly expect that the story would just repeat. The training process (or whatever process spits out the AGI) would end up creating some extremely specific conditions in which the AGI is learning the values. Its values would then necessarily be some complicated functions over weird mixes of the abstractions-natural-to-the-dataset-it's-trained-on, with their specifics being highly contingent on some invisible-to-us details of that process.

Pretty much all of your examples rely on the Alpha being unable to control the data learnt by Beta, and if this isn't the case, your examples break down.