LessWrong 2.0 Reader

If OpenAI and Sam Altman want to fix this situation, it is clear what must be done as the first step. The release of claims must be replaced, including retroactively, by a standard release of claims. Daniel’s vested equity must be returned to him, in exchange for that standard release of claims. All employees of OpenAI, both current employees and past employees, must be given unconditional release from their non-disparagement agreements, all NDAs modified to at least allow acknowledging the NDAs, and all must be promised in writing the unconditional ability to participate as sellers in all future tender offers. 

Then the hard work can begin to rebuild trust and culture, and to get the work on track. 

Alright - suppose they don't. What then?

I don't think I misstep in positing that we (for however you want to construe "we") should model OAI as - jointly but independently - meriting zero trust and functioning primarily to make Sam Altman personally more powerful. I'm also pretty sure that asking Sam to pretty please be nice and do the right thing is... perhaps strategically counterindicated.

Suppose you, Zvi (or anyone else reading this! yes, you!) were Unquestioned Czar of the Greater Ratsphere, with a good deal of money, compute, and soft power, but basically zero hard power. Sam Altman has rejected your ultimatum to Do The Right Thing and cancel the nondisparagements, modify the NDAs, not try to sneakily fuck over ex-employees when they go to sell and are made to sell for a dollar per PPU, etc, etc.

What's the line?

Multi voiced AI reading for this post:
https://open.substack.com/pub/thezvi/p/openai-exodus

Positivism isn't necessarily true, and if it is, it still doesn't get you to 6, because LP recommends you have no metaphysics which would imply no solipsistic metaphysics. (LP might be compatible with the claim that your own sense-data are all you can know , but that isn't quite the same thing).

I work at DeepMind and have been influenced by METR. :)

from Eric Weinstein in a youtube video. Can you link?

Greg Brockman and Sam Altman (cosigned):
[...]
First, we have raised awareness of the risks and opportunities of AGI so that the world can better prepare for it. We’ve repeatedly demonstrated the incredible possibilities from scaling up deep learning

chokes on coffee

It is becoming increasingly clear that OpenAI is not appropriately prioritizing safety over advancing capabilities research.

OK

This was the default outcome.

OK

Without repercussions for terrible decisions, decision makers have no skin in the game.

It's an article of faith for some people that that makes a difference, but I've never seen why.

I mean, many of the "decision makers" on these particular issues already believe that their actual, personal, biological skins are at stake, along with those of everybody else they know. And yet...

Anyone and everyone involved with Open Phil recommending a grant of $30 million dollars be given to OpenAI in 2017 shouldn't be allowed anywhere near AI Safety decision making in the future.

Thinking "seven years from now, a significant number of independent players in a relatively large and diverse field might somehow band together to exclude me" seems very distant from the way I've seen actual humans make decisions.

Platonic AI

'platonic' is a categorization i use internally, and my agenda could be broadly described as looking for methods to ensure AI/ASI will have this property.

with this word, i mean this category acceptance/rejection:
✅ has no goals

✅ has goals, but they're about what the system does in isolation. example: "in isolation from any world, (try to) output A when run"^[1]

❌ has goals, but they're about physical properties. example: "(try to) make sure A gets stored in memory on the computer in the physical world that's running me."^[2]

A can be 'the true answer to the input question', 'a proof of x conjecture', etc.

1. ^{^}

   a more human-intuitive transcription may include wording like: "try to be the kind of program/function which would (in isolation from any particular worldstate/physics) output A when run."

   i'm leaving this as a footnote because it can also confuse people, leading to questions like "what does it mean to 'try to be a kind of program' when it's already determined what kind of program it is?"

2. ^{^}

   this class of unaligned 'physical goals' is dangerous because if the system can't determine A, its best method to fulfill the goal is through instrumental convergence.

Do I indestand you correctly? You are saying that:

1. ASI will decide that its in a simulation
2. ASI will assume that simulation runners would prefer it to act aligned
3. Therefore, ASI will act aligned 

I think 1 is wrong to begin with, but addressing it is beyond this comment section.

So, what about 2? Why would ASI assume that cooperation with humans in a simulation is a behaviour that the creators of the simulation would reward? Instead of assuming... literally anything else about the nature of such reward.

Note: Most RSVPs (29 as of now) are on Meetup.