LessWrong 2.0 Reader

In many respects, I expect this to be closer to what actually happens than "everyone falls over dead in the same second" or "we definitively solve value alignment". Multipolar worlds, AI that generally follows the law (when operators want it to) but cannot fully be trusted, and generally muddling through are the default future. I'm hoping we don't get instrumental survival drives though.

Summary threads of two recent papers which seem like significant evidence in favor of the Simulators [LW · GW] view of LLMs (especially after just pretraining): https://x.com/aryaman2020/status/1852027909709382065 https://x.com/DimitrisPapail/status/1844463075442950229 

These numbers are dictated by the regulator. What mechanism is there to make them have any relation to the real world?

Does this plan necessarily factor through the using the intent-aligned AGI to quickly commit some sort of pivotal act [LW · GW] that flips the gameboard and prevents other intent-aligned AGIs from being used malevolently [? · GW] by self-interested or destructive (human) actors to gain a decisive strategic advantage? After all, it sure seems less than ideal to find yourself in a position where you can solve the theoretical parts of value alignment,^[1] but you cannot implement that in practice because control over the entire future light cone has already been permanently taken over by an AGI intent-aligned to someone who does not care about any of your broadly prosocial goals...

1. ^{^}

   In so far as something like this even makes sense, which I have already expressed [LW(p) · GW(p)] my skepticism of many times [LW(p) · GW(p)], but I don't think I particularly want to rehash this discussion with you right now...

This is IMO the type of experiment that I like, even though it is evidence against my thesis on AI alignment generalization, and if done more could be a much more productive method of settling the current disputes that currently exists in the AI safety field.

I will also say that at the moment, it's looking like a lot of the proposed alignment methods are unlikely to be adversarially robust, which while mostly fine for an alignment proposal in a vacuum, is going to require rather extreme info-security to make the plan work out in practice without unleashing extinction or worse.

I don't go as far as Zach Stein Perlman here, but I agree with the broad take that most plans for controlling/aligning AIs rely fairly critically on the model not being stolen by other parties:

https://www.lesswrong.com/posts/eq2aJt8ZqMaGhBu3r/zach-stein-perlman-s-shortform#ckNQKZf8RxeuZRrGH [LW(p) · GW(p)]

Yeah "stop reading here if you don't want to be spoiled." suggests the entire post is going to be spoilery, it isn't, or shouldn't be. Also opening with an unnecessary literary reference instead of a summary or description is an affectation symptomatic of indulgent writer-reader cultures where time is not valued.

[undeletable spoiler tag below please ignore]

I don't remember them having the actual stats, not watching it again though. I wonder if they published those elsewhere

Do you know why the error bars in the replication are smaller than the original one?  (just more people?) And with which confidence is the null hypothesis (difference = 0) is rejected in both cases?

Interesting! I think that works.

You can still use the same positively-oriented brainstorming process for figuring out how to avoid bad outcomes. As soon as there's even a vague idea of avoiding a very bad outcome, that becomes a very good reward prediction after taking the differential. The dopamine system does calculate such differentials, and it seems like the valance system, while probably different from direct reward prediction and more conceptual, should and could also take differentials in useful ways. Valance needs to at least somewhat dependent on context. I don't think this requires unique mechanisms (although it might have them); it's sufficient to learn variants of the concepts like "avoiding a really bad event" and then attaching valance to that concept variant.

I'm curious why you were downvoted, for you hit the nail on the head. For a short an concise answer, yours is the best.

Does anyone know? Otherwise I will just assume that they're rationalists who dislike (and look down on) traditional/old things for moral reasons. This is not very flattering of me but I can't think of better explanations.