LessWrong 2.0 Reader

If it helps, your explanations made perfect sense to me, like plain English.  So thank you for putting yourself out there; you gave me and others something to chew on.

I agree.

This all sounds pretty in-distribution for an LLM, and also like it avoids problems like "maybe thinking in different abstractions" [minecraft isn't amazing at this either, but at least has a bit], "having the AI act/think way faster than a human", "having the AI be clearly superhuman".

a number of ways to achieve the endgame, level up, etc, both more and less morally.

I'm less interested in "will the AI say it kills its friend" (in a situation that very clearly involves killing and a person and perhaps a very clear tradeoff between that and having 100 more gold that can be used for something else), I'm more interested in noticing if it has a clear grasp of what people care about or mean. The example of chopping down the tree house of the player in order to get wood (which the player wanted to use for the tree house) is a nice toy example of that. The AI would never say "I'll go cut down your tree house", but it.. "misunderstood" [not the exact word, but I'm trying to point at something here]

wdyt?

Your guesses on AI R&D are reasonable!

Apparently this has been tested extensively, for example:

https://x.com/METR_Evals/status/1860061711849652378

[disclaimers: I have some association with the org that ran that (I write some code for them) but I don't speak for them, opinions are my own]

Also, Anthropic have a trigger in their RSP which is somewhat similar to what you're describing, I'll quote part of it:

Autonomous AI Research and Development: The ability to either: (1) Fully automate the work of an entry-level remote-only Researcher at Anthropic, as assessed by performance on representative tasks or (2) cause dramatic acceleration in the rate of effective scaling.

Also, in Dario's interview, he spoke about AI being applied to programming.

My point is - lots of people have their eyes on this, it seems not to be solved yet, it takes more than connecting an LLM to bash.

Still, I don't want to accelerate this.

options to vary rules/environment/language as well, to see how the alignment generalizes ood. will try this today

it would basically be DnD like. 

Making a thing like Papers Please, but as a text adventure, popping an ai agent into that. 
Also, could literally just put the ai agent into a text rpg adventure - something like the equivalent of Skyrim, where there are a number of ways to achieve the endgame, level up, etc, both more and less morally. Maybe something like https://www.choiceofgames.com/werewolves-3-evolutions-end/ 
Will bring it up at the alignment eval hackathon

+1

I'm imagining an assistant AI by default (since people are currently pitching that an AGI might be a nice assistant). 

If an AI org wants to demonstrate alignment by showing us that having a jerk player is more fun (and that we should install their jerk-AI-app on our smartphone), then I'm open to hear that pitch, but I'd be surprised if they'd make it

It's because they take less continued attention/effort and provide more immediate/satisfying results. LW is almost purely theoretical and isn't designed to be efficient. It's an attempt to logically override bias rather than implement the quirks of human neurochemistry to automate the process.

Computer scientists are notorious for this. They know how brains make thoughts happen, but they don't have a clue how people think, so ego drives them to rationalize a framework to perceive the flaws of others: uncuriousness and lack of dedication. Because they're just as human as the rest of us, made of the same biological approximation of inherited "good-enoughness." And the smarter you are, the more complex and well-reasoned that rationalization will be.

We all seek to affirm our current beliefs and blame others for discrepancies. It's instinct, physics, chemistry. No amount of logic and reason can override the instinct to defend one's perception of reality. Every fat person in the world has been thoroughly educated on which lifestyle changes will cause them to lose weight, yet the obesity epidemic still grows.

Therefore, we study "rationality" to see ourselves as the good-guy protagonists who strive to be "less wrong," have "accurate beliefs," and "be effective at achieving our goals."

It's important work... for computers. For humanity, you're better off consulting a monk.

I think there are lots of technical difficulties in literally using minecraft (some I wrote here [LW(p) · GW(p)]), so +1 to that.

I do think the main crux is "would the minecraft version be useful as an alignment test", and if so - it's worth looking for some other solution that preserves the good properties but avoids some/all of the downsides. (agree?)

Still I'm not sure how I'd do this in a text game. Say more?