LessWrong 2.0 Reader

For emergency response, new ALERT. Personally  think the forecasting/horizon scanning part of Sentinel is good, the emergency response negative in expectation. What does it mean for funders idk, would donate conditionally on the funds being restricted to the horizon scanning part. 

As documented in the 2023 Medium article, Ziz has threatened to murder rationalists for a while, and I'm aware prominent rationalists have been paranoid about possible attempts on their life by Zizians for the past few years. Aella has also recently stated on Twitter she wouldn't accept an interview on the subject without an upgraded security system on her house.

Reason to care about engaging /acc:

https://www.lesswrong.com/posts/HE3Styo9vpk7m8zi4/evhub-s-shortform?commentId=kDjrYXCXgNvjbJfaa [LW(p) · GW(p)]

I've recently been thinking that it's a mistake to think of this type of thing--"what to do after the acute risk period is safed"--as being a waste of time / irrelevant; it's actually pretty important, specifically because you want people trying to advance AGI capabilities to have an alternative, actually-good vision of things. A hypothesis I have is that many of them are in a sense genuinely nihilistic/accelerationist; "we can't imagine the world after AGI, so we can't imagine it being good, so it cannot be good, so there is no such thing as a good future, so we cannot be attached to a good future, so we should accelerate because that's just what is happening".

I strong upvoted, not because it's an especially helpful post IMO, but because I think /acc needs better critique, so there should be more communication. I suspect the downvotes are more about the ideological misalignment than the quality.

Given the quality of the post, I think it would not be remotely rude to respond with a comment like "These are are well-tread topics; you should read X and Y and Z if you want to participate in a serious discussion about this.". But no one wrote that comment, and what would X, Y, Z be?? One could probably correct some misunderstandings in the post this way just by linking to the LW wiki on Orthogonality or whatever, but I personally wouldn't know what to link to, to actually counter the actual point.

I tried to model a best possible confinement strategy in Multilevel AI Boxing. 
I wrote it a few years ago and most ideas will unlikely work for current situation with many instances of chats and open weight models. 
However, the idea of landmines - secret stop words or puzzles which stop AI - may still hold. It is like jail breaking in reverse: unaligned AI finds some secret message which stops it. It could be realized on hardware level, or through anomalous tokens or "philosophical landmines'. 

About archipelago: c2.com, the original wiki tried this. They created a federated wiki but that didn't seem to work. My guess: the volume was too low.

And LW has already all the filtering you need: just subscribe to the people and topics you are interested. There is also the unfinishe reading list.

I get tha  this may not feel like its own community. Within LW this could be done with ongoing open threads about a topic. But tgat requires an organizer and participation.  And we are back at volume.  And at needing good writers. 

(Interesting. FWIW I've recently been thinking that it's a mistake to think of this type of thing--"what to do after the acute risk period is safed"--as being a waste of time / irrelevant; it's actually pretty important, specifically because you want people trying to advance AGI capabilities to have an alternative, actually-good vision of things. A hypothesis I have is that many of them are in a sense genuinely nihilistic/accelerationist; "we can't imagine the world after AGI, so we can't imagine it being good, so it cannot be good, so there is no such thing as a good future, so we cannot be attached to a good future, so we should accelerate because that's just what is happening".)

I also think monopolizing talent enables software companies to make sure those high fixed costs stay nice and high.

If you disagreed with this, is it because you think it is literally false or because you don't agree with the implied argument that software companies are doing this on purpose?

Richness: The model must include all the propositions the agent can meaningfully consider, including those about herself. If the agent can form a proposition “I will do X”, then that belongs in the space of propositions over which she has beliefs and (where appropriate) desirabilities.

I see a potential problem here, depending on what exactly is meant by "can meaningfully consider".

Consider this set up:

You participate in the experiment for seven days. Every day you wake up in a room and can choose between two envelopes. One of them has 100$ the other is empty. Then your memory of this act is erased. At the end of the experiment you get all the money that you've managed to win.

On day one money are assigned to an envelope randomly. However, on all the next days the money are put in the envelope that you didn't pick on the previous day. You do not have any access to random number generators.

Is the model supposed to include credence for proposition "Today the money is in envelope 1" when you wake up participating in such experiment?

Hm, this violates my model of the world.

there are too many AI companies for this deal to work on all of them

Realistically, I think there are like 3-4 labs^[1] that matter, OAI, DM, Anthropic, Meta.

some of these AI companies will have strong kinda-ideological commitments to not doing this

Even if that was true, they will be at the whim of investors who are almost all big tech companies.

this is better done by selling (even at a lower revenue) to anyone who wants an AI SWE than selling just to Oracle.

This is the explicit claim I was making with the WTP argument. I think this is firmly not true, and OpenAI will make more money by selling just to Oracle. What evidence causes you to disagree?

1. ^{^}

   American/Western labs.