LessWrong 2.0 Reader

What I mean by this is that if you want to predict what an AI will do, for example will it do better in having a new capability than other models, or what it's values are like, especially if you want to predict OOD behavior accurately, you would be far better off if you knew what it's data sources are, as well as the quality of it's data, than if you only knew it's prior/architecture.

Re my justification for it, my basic justification comes from this tweet thread, which points out that a lot of o1's success could come from high-quality data, and while I don't like the argument that search/fancy bits aren't happening at all (I do think o1 is doing a very small run-time search), I agree with the conclusion that the data quality was probably most of the reason o1 is so good in coding.

https://x.com/aidanogara_/status/1838779311999918448

Somewhat more generally, I'm pretty influenced by this post, and while I don't go as far as claiming that all of what an AI is the dataset, I do think a weaker version of the claim is pretty likely to be true.

https://nonint.com/2023/06/10/the-it-in-ai-models-is-the-dataset/

But one prediction we could have made in advance, if we knew that data was a major factor in how AIs learn values, is that value misspecification was likely to be far less severe of a problem than 2000-2010s thinking on LW had, and value learning had a tractable direction of progress, as training on human books and language would load into it mostly-correct values, and another prediction we could have made is that human values wouldn't all be that complicated, and could instead be represented by say several hundred megabyte/1 gigabyte codes quite well, and we could plausibly simplify that further.

Cf here:

The answer isn't embedded in 100 lines of Python, but in a subset of the weights of GPT-4 Notably the human value function (as expressed by GPT-4) is necessarily significantly simpler than the weights of GPT-4, as GPT-4 knows so much more than just human values. What we have now isn't a perfect specification of human values, but instead roughly the level of understanding of human values that a 85th percentile human can come up with. The human value function as expressed by GPT-4 is also immune to almost all in-practice, non-adversarial, perturbations

From here on Matthew Barnett's post about the historical value misspecification argument, and note I'm not claiming that alignment is solved right now:

https://www.lesswrong.com/posts/i5kijcjFJD6bn7dwq/evaluating-the-historical-value-misspecification-argument#N9ManBfJ7ahhnqmu7 [LW(p) · GW(p)]

and here, where it talks about the point that to the extent there's a gap between loading in correct values versus loading in capabilities, it's that loading in values data is easier than loading in capabilities data, which kind of contradicts this post from @Rob Bensinger [LW · GW] here on motivations being harder to learn, and one could have predicted this because there was a lot of data on human values, and a whole lot of the complexity of the values is in the data, not the generative model, thus it's very easy to learn values, but predictably harder to learn a lot of the most useful capabilities:

From Rob Bensinger:

"Hidden Complexity of Wishes" isn't arguing that a superintelligence would lack common sense, or that it would be completely unable to understand natural language. It's arguing that loading the right *motivations* into the AI is a lot harder than loading the right understanding.

https://x.com/robbensinger/status/1648120202708795392

From @beren [LW · GW] on alignment generalizing further than capabilities, in spiritual response to Bensinger:

In general, it makes sense that, in some sense, specifying our values and a model to judge latent states is simpler than the ability to optimize the world. Values are relatively computationally simple and are learnt as part of a general unsupervised world model where there is ample data to learn them from (humans love to discuss values!). Values thus fall out mostly’for free’ from general unsupervised learning. As evidenced by the general struggles of AI agents, ability to actually optimize coherently in complex stochastic ‘real-world’ environments over long time horizons is fundamentally more difficult than simply building a detailed linguistic understanding of the world.

https://www.beren.io/2024-05-15-Alignment-Likely-Generalizes-Further-Than-Capabilities/

But that's how we could well have made predictions about AIs, in an alternate universe where LW didn't anchor too hard on their previous models of AI like AIXI and Solomonoff induction.

I think you're still thinking in terms of something like formalized political power, whereas other people are thinking in terms of "any ability to affect the world".

Suppose a fantastically powerful alien called Superman comes to earth, and starts running around the city of Metropolis, rescuing people and arresting criminals. He has absurd amounts of speed, strength, and durability. You might think of Superman as just being a helpful guy who doesn't rule anything, but as a matter of capability he could demand almost anything from the rest of the world and the rest of the world couldn't stop him. Superman is de facto ruler of Earth; he just has a light touch.

If you consider that acceptable, then you aren't objecting to "god-like status and control", you just have opinions about how that control should be exercised.

If you consider that UNacceptable, then you aren't asking for Superman to behave in certain ways, you are asking for Superman to not exist (or for some other force to exist that can check him).

Most humans (probably including you) are currently a "prisoner" of a coalition of humans who will use armed force to subdue and punish you if you take any actions that the coalition (in its sole discretion) deems worthy of such punishment. Many of these coalitions (though not all of them) are called "governments". Most humans seem to consider the existence of such coalitions to be a good thing on balance (though many would like to get rid of certain particular coalitions).

I will grant that most commenters on LessWrong probably want Superman to take a substantially more interventionist approach than he does in DC Comics (because frankly his talents are wasted stopping petty crime in one city).

Most commenters here still seem to want Superman to avoid actions that most humans would disapprove of, though.

I usually defer to you in things like this, but I don't see why this would be the case. I think the proposal of simulating less competent civilizations is equivalent to the idea of us deciding now, when we don't really know yet how competent a civilization we are, to bail out less competent alien civilizations in the multiverse if we succeed. In return, we hope that this decision is logically correlated with more competent civilization (who were also unsure in their infancy about how competent they are), deciding to bail out less competent civilizations, including us. My understanding from your comments is that you believe this likely works, how is my proposal of simulating less-coordinated civilizations different?

The story about simulating smaller Universes is more confusing. That would be equivalent to bailing out aliens in smaller Universes for a tiny fraction of our Universe, in the hope that larger Universes also bail us out for a tiny fraction of their Universe. This is very confusing if there are infinite levels of bigger and bigger Universes, I don't know what to do with infinite ethics. If there are finite levels, but the young civilizations don't yet have a good prior over the distribution of Universe-sizes, all can reasonably think that there all levels above them, and all their decisions are correlated, so everyone bails out the inhabitants of the smaller Universes, in the hope that they get bailed out by a bigger Universe. Once they learn the correct prior over Universe-sizes, and biggest Universe realizes that no bigger Universe's actions correlate with theirs, all of this fails (though they can still bail each other out from charity). But this is similar to the previous case, where once the civilizations learn their competence level, the most competent ones are no longer incentivized to enter into insurance contracts, but the hope is that in a sense they enter into a contract while they are still behind the veil of ignorance. 

I think I'm confused why you work on AI safety then, if you believe the end-state is already 2^75 level overdetermined.

It's probably physically overdetermined one way or another, but we're not sure which way yet. We're still unsure about things like "how sensitive is the population to argument" and "how sensibly do government respond if the population shifts".

But this uncertainty -- about which way things are overdetermined -- does not bear all that much relationship to the expected ratio of (squared) quantum amplitude between branches where we live and branches where we die. It just wouldn't be that shocking for the ratio between those two sorts of branches to be on the order of 2^75; this would correspond to saying something like "it turns out we weren't just a few epileptic seizures and a thunderstom away from the other outcome".

Background: I think there's a common local misconception of logical decision theory that it has something to do with making "commitments" including while you "lack knowledge". That's not my view.

I pay the driver in Parfit's hitchhiker not because I "committed to do so", but because when I'm standing at the ATM and imagine not paying, I imagine dying in the desert. Because that's what my counterfactuals say to imagine. To someone with a more broken method of evaluating counterfactuals, I might pseudo-justify my reasoning by saying "I am acting as you would have committed to act". But I am not acting as I would have committed to act; I do not need a commitment mechanism; my counterfactuals just do the job properly no matter when or where I run them.

To be clear: I think there are probably competent civilizations out there who, after ascending, will carefully consider the places where their history could have been derailed, and carefully comb through the multiverse for entities that would be able to save those branches, and will pay thoes entities, not because they "made a commitment", but because their counterfactuals don't come with little labels saying "this branch is the real branch". The multiverse they visualize in which the (thick) survivor branches pay a little to the (thin) derailed branches (leading to a world where everyone lives (albeit a bit poorer)), seems better to them than the multiverse they visualize in which no payments are made (and the derailed branches die, and the on-track branches are a bit richer), and so they pay.

There's a question of what those competent civilizations think when they look at us, who are sitting here yelling "we can't see you, and we don't know how to condition our actions on whether you pay us or not, but as best we can tell we really do intend to pay off the AIs of random alien species -- not the AIs that killed our brethren, because our brethren are just too totally dead and we're too poor to save all but a tiny fraction of them, but really alien species, so alien that they might survive in such a large portion that their recompense will hopefully save a bigger fraction of our brethren".

What's the argument for the aliens taking that offer? As I understand it, the argument goes something like "your counterfactual picture of reality should include worlds in which your whole civilization turned out to be much much less competent, and so when you imagine the multiverse where you pay for all humanity to live, you should see that, in the parts of the multiverse where you're totally utterly completely incompetent and too poor to save anything but a fraction of your own brethren, somebody else pays to save you".

We can hopefully agree that this looks like a particularly poor insurance deal relative to the competing insurance deals.

For one thing, why not cut out the middleman and just randomly instantiate some civilization that died? (Are we working under the assumption that it's much harder for the aliens to randomly instantiate you than to randomly instantiate the stuff humanity's UFAI ends up valuing? What's up with that?)

But even before that, there's all sorts of other jucier looking opportunities. For example, suppose the competent civilization contains a small collection of rogues who they asses have a small probability of causing an uprising and launching an AI before it's ready. They presumably have a pretty solid ability to figure out exactly what that AI would like and offer trades to it driectly, and that's a much more appealing way to spend resources allocated to insurance. My guess is there's loads and loads of options like that that eat up all the spare insurance budget, before our cries get noticed by anyone who cares for the sake of decision theory (rather than charity).

Perhaps this is what you meant by "maybe they prefer to make deals with beings more similar to them"; if so I misunderstood; the point is not that they have some familiarity bias but that beings closer to them make more compelling offers.

The above feels like it suffices, to me, but there's still another part of the puzzle I feel I haven't articulated.

Another piece of backgound: To state the obvious, we still don't have a great account of logical updatelessness, and so attempts to discuss what it entails will be a bit fraut. Plowing ahead anyway:

The best option in a counterfactual mugging with a logical coin and a naive predictor is to calcuate the logical value of the coin flip and pay iff you're counterfactual. (I could say more about what I mean by 'naive', but it basically just serves to render this statement true.) A predictor has to do a respectable amount of work to make it worth your while to pay in reality (when the coin comes up against you).

What sort of work? Well, one viewpoint on it (that sidesteps questions of "logically-impossible possible worlds" and what you're supposed to do as you think further and realize that they're impossible) is that the predictor isn't so much demanding that you make your choice before you come across knowledge of some fact, so much as they're offering to pay you if you render a decision that is logically indepnedent from some fact. They don't care whether you figure out the value of the coin, so long as you don't base your decision on that knowledge. (There's still a question of how exactly to look at someone's reasoning and decide what logical facts it's independent of, but I'll sweep that under the rug.)

From this point of view, when people come to you and they're like "I'll pay you iff your reasoning doesn't depend on X", the proper response is to use some reasoning that doesn't depend on X to decide whether the amount they're paying you is more than VOI(X).

In cases where X is something like a late digit of pi, you might be fine (up to your ability to tell that the problem wasn't cherry-picked). In cases where X is tightly intertwined with your basic reasoning faculties, you should probably tell them to piss off.

Someone who comes to you with an offer and says "this offer is void if you read the fine print or otherwise think about the offer too hard", brings quite a bit of suspicion onto themselves.

With that in mind, it looks to me like the insurance policy on offer reads something like:

would you like to join the confederacy of civilizations that dedicate 1/million of their resource to pay off a random UFAI?

cost: 1/million of your resources.

benefit: any UFAI you release that is amenable to trade will be paid off with 1/million * 1/X to allocate you however many resources that's worth, where X is the fraction of people who take this deal and die (modulo whatever costs are needed to figure out which UFAIs belong to signatories and etc.)

caveat: this offer is only valid if your reasoning is logically independent from your civilizational competence level, and if your reasoning for accepting the proposal is not particularly skilled or adept

And... well this isn't a knockdown argument, but that really doesn't look like a very good deal to me. Like, maybe there's some argument of the form "nobody in here is trying to fleece you because everyone in here is also stupid" but... man, I just don't get the sense that it's a "slam dunk", when I look at it without thinking too hard about it and in a way that's independent of how competent my civilization is.

Mostly I expect that everyone stooping to this deal is about as screwed as we are (namely: probably so screwed that they're bringing vastly more doomed branches than saved ones, to the table).

Roughly speaking, I suspect that the sort of civilizations that aren't totally fucked can already see that "comb through reality for people who can see me and make their decisions logically dependent on mine" is a better use of insurance resources, by the time they even consider this policy. So when you plea of them to evaluate the policy in a fashion that's logically independent from whether they're smart enough to see that they have more foolproof options available, I think they correctly see us as failing to offer more than VOI(WeCanThinkCompetently) in return, because they are correctly suspicious that you're trying to fleece them (which we kinda are; we're kinda trying to wish ourselves into a healthier insurance-pool).

Which is to say, I don't have a full account of how to be logically updateless yet, but I suspect that this "insurance deal" comes across like a contract with a clause saying "void if you try to read the fine print or think too hard about it". And I think that competent civilizations are justifiably suspicious, and that they correctly believe they can find other better insurance deals if they think a bit harder and void this one.

Whatever happened to Lumina, btw? Was anyone able to see any changes in their dental health so far?

Yeah, the misunderstanding came from that I thought that "last minute" literally means "last 60 seconds" and I didn't see how that's relevant. If if means "last 5 years" or something where it's still definitely our genetic copies running around, then I'm surprised you think alignment success or failure is that overdetermined at that time-scale. I understand your point that our epistemic uncertainty is not the same as our actual quantum probability, that is either very high or very low. But still, it's 2^75 overdetermined over a 5 year period? This sounds very surprising to me, the world feels more chaotic than that. (Taiwan gets nuked, chip development halts, meanwhile the Salvadorian president hears a good pitch about designer babies and legalizes running the experiments there and they work, etc, there are many things that contribute to alignment being solved or not, that don't directly run through underlying facts about computer science, and 2^-75 is a very low probability to none of the pathways to hit it). 

But also, I think I'm confused why you work on AI safety then, if you believe the end-state is already 2^75 level overdetermined. Like maybe working on earning to give to bednets would be a better use of your time then. And if you say "yes, my causal impact is very low because the end result is already overdetermined, but my actions are logically correlated with the actions of people in other worlds who are in a similar epistemic situation to me, but whose actions actually matter because their world really is on the edge", then I don't understand why you argue in other comments that we can't enter into insurance contracts with those people, and our decision to pay AIs in the Future has as little correlation with their decision, as the child to the fireman.

OK for this post. "smart". A response is smart/intelligent if 

1. Firstly there is an assumed goal and measure. I don't think it matters whether we are talking about the bees/birds as individuals or as part of the hive/flock. In this case the bee defense is effective both for the individual bee and hive. If a bee was only concerned about its survival, swarming the scout would still be beneficial, and of course such behavior is for the hive. Similarly for birds, flocks with large numbers of birds with swarming behavior would be better both for the flock, and individual birds in such a flock.
2. There is a force multiplier effect, the benefit of the behavior is much greater than the cost. This is obvious for the bees, a tiny expenditure of calories saves the hive. Likewise for birds, they waste a huge amount of calories both individually and collectively evading the hawk etc.
3. There is a local optimum (or something close) for the behavior - that is half measures don't give half the benefit. So it seems like the result of foresight. There is perhaps more of a distinct and distant local optimum for the bee behavior "identify the scout, send warning chemicals to the hive, then swarm it" then the possible bird behavior "call then attack the attacker" as the scout isn't the actual attack in the bees case.
4. The change is behavioral, rather than a physical adaptation.

This fits into the intuitive feeling of what intelligent is also. A characteristic of what people feel is intelligent is to imagine a scenario, then make it real. The bees havn't done that, but the outcome is as if they had. "Imagine if we took out the scout, then there would be no later invasion"

You look at the birds and think "how do they miss this - ant colonies swarm to see off a larger attacker, herding animals do too, why do they miss such a simple effective strategy?" In that situation they are not intuitively intelligent.

Meanwhile, Twitter: https://x.com/deepfates/status/1840434157920641046?t=p5QVyXcUwIkL0v3pns8ADg&s=19

IMO all of the "smooth/sharp" and "soft/hard" stuff is too abstract. When I concretely picture what the differences between them are, the aspect that stands out most is whether the takeoff will be concentrated within a single AI/project/company/country or distributed across many AIs/projects/companies/countries.

This is of course closely related to debates about slow/fast takeoff (as well as to the original Hanson/Yudkowsky debates). But using this distinction instead of any version of the slow/fast distinction has a few benefits:

1. If someone asks "why should I care about slow/fast takeoff?" a lot of the answers will end up appealing to the concentrated/distributed power thing. E.g. you might say "if takeoff is fast that means that there will be a few key points of leverage".
2. Being more concrete, I think it will provoke better debates (e.g. how would a single AI lab concretely end up outcompeting everyone else?)
3. This framing naturally concentrates the mind on an aspect of risk (concentration of power) that is concerning from both a misuse and a misalignment perspective.