LessWrong 2.0 Reader

Wait, how does the grabby aliens argument support this? I understand that it points to "the universe will be carved up between expansive spacefaring civilizations" (without reference to whether those are biological or not), and also to "the universe will cease to be a place where new biological civilizations can emerge" (without reference to what will happen to existing civilizations). But am I missing an inferential step?

I have this description but it's not that good, because it's very unfocused. That's why I did not link it in the OP. The LessWrong dialog linked at the top of the post is probably the best thing in terms of describing the motivation and what the project is about at a high level.

That's a good point. You have pushed me towards thinking that this is an unreasonable statement and "predicted this problem at the time" is better.

As recently as early 2023 Eliezer was very pessimistic about AI policy efforts amounting to anything, to the point that he thought anyone trying to do AI policy was hopelessly naive and should first try to ban biological gain-of-function research just to understand how hard policy is. Given how influential Eliezer is, he loses a lot of points here (and I guess Hendrycks wins?)

Then Eliezer updated and started e.g. giving podcast interviews. Policy orgs spun up and there are dozens of safety-concerned people working in AI policy. But this is not reflected in the LW frontpage. Is this inertia, or do we like thinking about computer science more than policy, or is it something else?

three most convincing arguments i know for OP’s thesis are:

1. atoms on earth are “close by” and thus much more valuable to fast running ASI [LW · GW] than the atoms elsewhere.

2. (somewhat contrary to the previous argument), an ASI will be interested in quickly reaching the edge of the hubble volume, as that’s slipping behind the cosmic horizon — so it will starlift the sun for its initial energy budget.

3. robin hanson’s “grabby aliens” argument: witnessing a super-young universe (as we do) is strong evidence against it remaining compatible with biological life for long.

that said, i’m also very interested in the counter arguments (so thanks for linking to paul’s comments!) — especially if they’d suggest actions we could take in preparation.

The science behind nuclear weapons is well known -- what keeps them from being built is access to nuclear materials.

This is not true for AGI.

I think this is the sort of conversation we should be having!  [Side note: I think restricting compute is more effective than restricting research because you don't need 100% buy in.

1. it's easier to prevent people from manufacturing semiconductors than to keep people from learning ideas that fit on a napkin
2. It's easier to prevent scientists in Eaccistan from having GPUs than to prevent scientists in Eaccistan from thinking.

The analogy to nuclear weapons is, I think, a good one.  The science behind nuclear weapons is well known -- what keeps them from being built is access to nuclear materials.

(Restricting compute also seriously restricts research.  Research speed on neural nets is in large part bounded by how many experiments you run rather than ideas you have.)]

Suppose that, as a Buddhist or a lobotomy patient or just a very phlegmatic person, finding your shoelaces tied together didn’t make you angry.

Buddhist, lobotomy patients and phlegmatic people all have things in their closets, they all have things to get angry, upset or confused about. If you are a Buddhist, lobotomy patient and phlegmatic you still see the particular narrative worldview. What you see does not change because after all there will always be something to get tied up on. It is about shaping the things that you do get tied up on and further controlling your reaction to them.

The goblin is the writer 
The closet is your map and territory of your mind
The shoelaces tie together connections in your mind
The shoes are the words

If you read an article here that makes you angry it is okay. The writer has simply made a connection in your mind that you believe imposes upon you a negative feeling. You feel angry because of the map and territory of your mind, not what is written.

Perception of the writer as the globin disappears when you can change the shape of the map or territory.

Who is a globin to steal your gold, but a leprechaun at the end of the rainbow.

I would like to make a recommendation to Johannes that he should try to write and post content in a way that invokes less feelings of cringe in people. I know it does invoke that because I personally feel cringe.

Still, I think that there isn’t much objectively bad about this post. I’m not saying the post is very good or convincing. I think its style is super weird but that should be considered to be okay in this community. These thoughts remind me of something Scott Alexander once wrote - that sometimes he hears someone say true but low status things - and his automatic thoughts are about how the person must be stupid to say something like that, and he has to consciously remind himself that what was said is actually true.

Also, all these thoughts about this social reality sadden me a little - why oh why is AI safety such a status-concerned and “serious business” area nowadays?

Science was noticing that certain modes of thinking uncovered beliefs that let us manipulate the world

Science is our tool to manipulate the world. Science is the instrument of truth. If you can not manipulate the meaning of reality through definition of words such as 'rational', what it is and it isn't. Where science is the substitute for truth and rationality is the substitute for truth also. As self-evident the definition of rationality is substituted for science and this forms our basic definition for rationality moving forward. Care not to define science as in rationality, science is a tool as is rationality.

In rationality, as in science. 'Curiosity, pragmatism, and quasi-moral injunctions' are injected into research questions and colour our understanding of both the world and truth. 

To use science as a tool is to obtain the truth, to use rationality is to obtain the truth. We must apply a scientific approach and method to our rationality.