LessWrong 2.0 Reader

Multi voiced AI reading for this post:
https://open.substack.com/pub/thezvi/p/openai-exodus

Positivism isn't necessarily true, and if it is, it still doesn't get you to 6, because LP recommends you have no metaphysics which would imply no solipsistic metaphysics. (LP might be compatible with the claim that your own sense-data are all you can know , but that isn't quite the same thing).

I work at DeepMind and have been influenced by METR. :)

from Eric Weinstein in a youtube video. Can you link?

Greg Brockman and Sam Altman (cosigned):
[...]
First, we have raised awareness of the risks and opportunities of AGI so that the world can better prepare for it. We’ve repeatedly demonstrated the incredible possibilities from scaling up deep learning

chokes on coffee

It is becoming increasingly clear that OpenAI is not appropriately prioritizing safety over advancing capabilities research.

OK

This was the default outcome.

OK

Without repercussions for terrible decisions, decision makers have no skin in the game.

It's an article of faith for some people that that makes a difference, but I've never seen why.

I mean, many of the "decision makers" on these particular issues already believe that their actual, personal, biological skins are at stake, along with those of everybody else they know. And yet...

Anyone and everyone involved with Open Phil recommending a grant of $30 million dollars be given to OpenAI in 2017 shouldn't be allowed anywhere near AI Safety decision making in the future.

Thinking "seven years from now, a significant number of independent players in a relatively large and diverse field might somehow band together to exclude me" seems very distant from the way I've seen actual humans make decisions.

Platonic AI

'platonic' is a categorization i use internally, and my agenda could be broadly described as looking for methods to ensure AI/ASI will have this property.

with this word, i mean this category acceptance/rejection:
✅ has no goals

✅ has goals, but they're about abstract properties. example: "in isolation from any world, (try to) output A when run"^[1]

❌ has goals, but they're about physical properties. example: "(try to) make sure A gets stored in memory on the computer in the physical world that's running me."^[2]

A can be 'the correct answer to the input question', 'a proof of x conjecture', etc.

1. ^{^}

   a more human-intuitive transcription may include wording like: "try to be the kind of program/function which would (in isolation from any particular worldstate/physics) output A when run."

   i'm leaving this as a footnote because it can also confuse people, leading to questions like "what does it mean to 'try to be a kind of program' when it's already determined what kind of program it is?"

2. ^{^}

   this class of unaligned 'physical goals' is dangerous because if the system can't determine what A is, its best method to fulfill the goal is through instrumental convergence.

Do I indestand you correctly? You are saying that:

1. ASI will decide that its in a simulation
2. ASI will assume that simulation runners would prefer it to act aligned
3. Therefore, ASI will act aligned 

I think 1 is wrong to begin with, but addressing it is beyond this comment section.

So, what about 2? Why would ASI assume that cooperation with humans in a simulation is a behaviour that the creators of the simulation would reward? Instead of assuming... literally anything else about the nature of such reward.

Note: Most RSVPs (29 as of now) are on Meetup.

From my point of view, of course profit maximizing companies will…maximize profit. It never was even imaginable that these kinds of entities could shoulder such a huge risk responsibly.

Correct me if I'm wrong but isn't Conjecture legally a company? Maybe their profit model isn't actually foundation models? Not actually trying to imply things, just thought the wording was weird in that context and was wondering whether Conjecture has a different legal structure than I thought.