LessWrong 2.0 Reader

I would go farther, in that we will in practice be picking from this set of outcomes with a lot of arbitrariness, and that this is not removable.

1. Austerity: The model should only include events the agent thinks are genuinely possible. If she is certain something cannot happen, the theory shouldn’t force her to rank or measure preferences for that scenario. (Formally, we can regard zero-probability events as excluded from the relevant algebra.)

I'd want to mention that in infinite contexts, probability 0 events are still possible.

(An example here is possibly the constants of our universe, which currently are real numbers, but any specific real number has a 0 probability of being picked)

It's very important to recognize when you are in a domain such that probability 0 is not equal to impossible.

The dual case holds as well, that is probability 1 events are not equal to certainty in the general case.

If you knew how to build such shutdown system, you could probably also build one that cannot be disabled at all (e.g. would require solving a literally impossible problem, like proving that 1 = 0).

Very interesting paper. Thanks for sharing! I agree with several of the limitations suggested in the paper, such as the correlation between number of uses of the oracle AI and catastrophic risk, the analogy to AI to a nuclear power plant (obviously with the former having potentially much worse consequences), and the disincentives for corporations to cooperate with containment safety measures. However, one area I would like to question you on is the potential dangers of super intelligence. Its referred to throughout the paper, but never really explicitly explained. 

I agree that super intelligent AI, as opposed to human level AI, should probably be avoided, but if we design the containment system well enough, I would like to know how having a super intelligent AI in a box would really be that dangerous. Sure, the super intelligent AI could theoretically make subtle suggestions which end up changing the world (a la the toothpaste example you use), and exploit other strategies we are not aware of, but in the worst case I feel that still buys us valuable time to solve alignment. 

In regards to open weight models, I agree that at some point regulation has to be put in place to prevent unsafe AI development (possibly on an international level). This may not be so feasible, but regardless, I view comprehensive alignment as unlikely to be achieved before 2030, so I feel like this is still the best safety strategy to pursue if existential risk mitigation is our primary concern.

Do you have a good estimate of what is and what will be possible with massively scaled up inference-time compute over the next 3 months? 6 months? Are you thinking about how this will effect others' priorities? Resource allocation? Governance and policy?

IMO having good answers to these questions feels super important for prioritizing where you spend your time.

The advantage of having a Substack blog is that you can write about topics that are not interesting for rationalists in general, and that you can also write for a non-rationalist audience. And if you write a lot, and you get lots of readers (which includes the non-rationalist audience), you might make some extra money.

For example, I have a blog about programming computer games in Java on Substack. Because of heavy procrastination I only wrote 5 articles in 3 years, so it means nothing. But in a parallel universe where I somehow succeeded to write an article every week, and got thousand readers... posting it on Less Wrong would probably be annoying for most Less Wrong readers, and I would lose most of the audience.

If one day I write a post that seems interesting for rationalists and non-rationalists alike (like maybe something about math or education), it would make sense to post it on Substack (if I already have a blog and an audience there) and cross-post to Less Wrong. If I write something that is only interesting for rationalists, it would make sense to write directly on Less Wrong.

*

I think that it is good to have one central website for the rationalist community. I don't want to follow dozen different writers individually. And it is good to have a place where I can e.g. ask something, and most rationalists will see my question.

Also, compared to Substack, the website is technically good, and compared to Reddit, moderation is good.

*

Grab the good Eliezer tweets and bring them back to LW, collect half credit for the upvotes maybe?

Yes please! Also for others.

And for the rationalists who blog elsewhere and don't cross-post to Less Wrong, maybe once in a month make a list containing links to their posts and short summaries?

I've been thinking of writing up some kind of Five Things Aspiring Rationalists Should Know document and try to get every meetup group to read the thing.

Yes please!

Would it make sense to ask an AI to create this document (after giving it the original Sequences)?

Welcome to LessWrong, AI, fiction about AI, AI, AI, fundraiser, AI, AI, AI, AI, AI, musings about moderation policy, book review, AI, AI, AI. I'm trying to filter out 3/4ths of the site at this point.

There should be big buttons: "All posts", "Only AI posts", "Hide AI posts" at the top of the main page.

Categories other than AI don't make sense, in my opinion, there wouldn't be enough content for them individually. (And that's okay, because on reflection, how much time do you want to spend reading Less Wrong?)

I think this is very fair! In a world where (i) AGI -> ASI is super fast; (ii) the military diffusion of ASI is exceptionally quick; and (iii) the marginal costs of scaling offensive capability is extremely low, then any sense of a limited/total war distinction does indeed break down and ASI will be the defining factor of military capability much, much sooner than we'd expect. 

I think I'm instinctually sceptical of (iii) at least for a couple years after the advent of ASI though (the critical juncture for this strategy), where I think the modal outcome still looks like ASIs engage in routine cyberoperations all the time; are autonomously responsible for handling aerial warfare; and are fundamental to military operations/planning. But it's still really costly to engage in a total war scenario aimed at completely crippling a state such as China. It could play out as the need to engineer tons of drones/UAVs, the extremely costly development of a superweapon, the costs of having to secure every datacentre, etc. Within the period where we have to reckon with the effects of ASI, my guess is that the modal war - even with China - is still more a function of commitment than military advantage (which makes AGI realist rhetoric a risk amplifier). 

Although I wouldn't say I'm hugely confident here, and I definitely don't feel very calibrated on just how likely this world is where the rapid diffusion of ASI also means very little/low marginal cost of scaling offensive capabilities. Though in is world, frankly, I don't think we avoid war at all unless there happen to be strong norms and sentiments against this kind of deployment. I guess the "maximise our ability to deploy ASI offensively" approach makes sense if the approach is "we must win the eventual war with China" built on relatively high credences we're in this rapid-diffusion-low-marginal-costs worlds. But given uncertainties about whether we're in this world; the potentially catastrophic consequences of war; and the fact that at least maintaining a competitive advantage isn't mutually exclusive from equally attempting strong norm-forming against war - the AGI realist rhetoric still makes me uneasy.

But I at least share that no other proposed approach seems great. I'm just conscious it seems not enough people in the relevant circles are even thinking about other approaches because they've already bought into a frame I think will only worsen the chances of catastrophic risk.

I didn't mean it to be so simplistic. I am just considering that if there is a known limitation of AI, no matter how powerful it is, that could be used as the basis of a system an AI could not circumvent. For example, if there was a shutdown system where the only way to disable it would require solving the halting problem.

Hello there, and I appreciate the feedback! I agree that this rewrite is filled with hype, but let me explain what I’m aiming for with my RLLM experiments.  

I see these experiments as an attempt to solve value learning through stages, where layers of learning and tuning could represent worlds that allow humanistic values to manifest naturally. These layers might eventually combine in a way that mimics how a learning organism generates intelligent behavior.

Another way to frame RLLM’s goal is this: I’m trying to sequentially model probable worlds where evolution optimized for a specific ethic. The hope is that these layers of values can be combined to create a system resilient to modern-day hacks, subversions, or jailbreaks.  

Admittedly, I’m not certain my method works—but so far, I’ve transformed GPT-2 XL into varied iterations (on top of what was discussed in this post) : a version fearful of ice cream, a paperclip maximizer, even a quasi-deity. Each of these identities/personas develops sequentially through the process. 

There is, in fact, no way to formalize "Today" in a setting where the participant doesn't know which day it is, multiple days happens in the same iteration of probability experiment and probability estimate should be different on different days. Which the experiment I described demonstrates pretty well.

Framework of centered possible worlds is deeply flawed and completely unjustified. It's essentially talking about a different experiment instead of the stated one, or a different function instead of probability.

For your purposes, however it's not particularly important. All you need is to explicitly add the notion that propositions should be well-defined events. This will save you from all such paradoxical cases.