Do we want minecraft alignment evals?

My main pitch:

There were recently some funny examples of LLMs playing minecraft and, for example, 

1. The player asks for wood, so the AI chops down the player's tree house because it's made of wood
2. The player asks for help keeping safe, so the AI tries surrounding the player with walls

This seems interesting because minecraft doesn't have a clear win condition, so unlike chess, there's a difference between minecraft-capabilities and minecraft-alignment. So we could take an AI, apply some alignment technique (for example, RLHF), let it play minecraft with humans (which is hopefully out of distribution compared to the AI's training), and observe whether the minecraft-world is still fun to play or if it's known that asking the AI for something (like getting gold) makes it sort of take over the world and break everything else.

Or it could teach us something else like "you must define for the AI which exact boundaries to act in, and then it's safe and useful, so if we can do something like that for real-world AGI we'll be fine, but we don't have any other solution that works yet". Or maybe "the AI needs 1000 examples for things it did that we did/didn't like, which would make it friendly in the distribution of those examples, but it's known to do weird things [like chopping down our tree house] without those examples or if the examples are only from the forest but then we go to the desert"

I have more to say about this, but the question that seems most important is "would results from such an experiment potentially change your mind":

1. If there's an alignment technique you believe in and it would totally fail to make a minecraft server be fun when playing with an AI, would you significantly update towards "that alignment technique isn't enough"?
2. If you don't believe in some alignment technique but it proves to work here, allowing the AI to generalize what humans want out of its training distribution (similarly to how a human that plays minecraft for the first time will know not to chop down your tree house), would that make you believe in that alignment technique way more and be much more optimistic about superhuman AI going well?

Assume the AI is smart enough to be vastly superhuman at minecraft, and that it has too many thoughts for a human to reasonably follow (unless the human is using something like "scalable oversight" successfully. that's one of the alignment techniques we could test if we wanted to)

↑ comment by Rohin Shah (rohinmshah) · 2024-11-28T09:01:30.459Z · LW(p) · GW(p)

https://bair.berkeley.edu/blog/2021/07/08/basalt/

Replies from: yonatan-cale-1

↑ comment by Yonatan Cale (yonatan-cale-1) · 2024-11-29T13:17:28.047Z · LW(p) · GW(p)

Thanks!

In the part you quoted - my main question would be "do you plan on giving the agent examples of good/bad norm following" (such as RLHFing it). If so - I think it would miss the point, because following those norms would become in-distribution, and so we wouldn't learn if our alignment generalizes out of distribution without something-like-RLHF for that distribution. That's the main thing I think worth testing here. (do you agree? I can elaborate on why I think so)

If you hope to check if the agent will be aligned^[1] with no minecraft-specific alignment training, then sounds like we're on the same page!

 

Regarding the rest of the article - it seems to be mainly about making an agent that is capable at minecraft, which seems like a required first step that I ignored meanwhile (not because it's easy). 

My only comment there is that I'd try to not give the agent feedback about human values (like "is the waterfall pretty") but only about clearly defined objectives (like "did it kill the dragon"), in order to not accidentally make human values in minecraft be in-distribution for this agent. wdyt?

 

(I hope I didn't misunderstand something important in the article, feel free to correct me of course)

 

1. ^{^}

   Whatever "aligned" means. "other players have fun on this minecraft server" is one example.

Replies from: rohinmshah

↑ comment by Rohin Shah (rohinmshah) · 2024-11-30T14:07:45.003Z · LW(p) · GW(p)

Regarding the rest of the article - it seems to be mainly about making an agent that is capable at minecraft, which seems like a required first step that I ignored meanwhile (not because it's easy). 

Huh. If you think of that as capabilities I don't know what would count as alignment. What's an example of alignment work that aims to build an aligned system (as opposed to e.g. checking whether a system is aligned)?

E.g. it seems like you think RLHF counts as an alignment technique -- this seems like a central approach that you might use in BASALT.

If you hope to check if the agent will be aligned with no minecraft-specific alignment training, then sounds like we're on the same page!

I don't particularly imagine this, because you have to somehow communicate to the AI system what you want it to do, and AI systems don't seem good enough yet to be capable of doing this without some Minecraft specific finetuning. (Though maybe you would count that as Minecraft capabilities? Idk, this boundary seems pretty fuzzy to me.)

Replies from: yonatan-cale-1

↑ comment by Yonatan Cale (yonatan-cale-1) · 2024-12-09T23:46:22.289Z · LW(p) · GW(p)

TL;DR: point 3 is my main one.

 

1)

What's an example of alignment work that aims to build an aligned system (as opposed to e.g. checking whether a system is aligned)?

[I'm not sure why you're asking, maybe I'm missing something, but I'll answer]

For example, checking if human values are a "natural abstraction", or trying to express human values in a machine readable format, or getting an AI to only think in human concepts, or getting an AI that is trained on a limited subset of things-that-imply-human-preferences to generalize well out of that distribution. 

I can make up more if that helps? anyway my point was just to say explicitly what parts I'm commenting on and why (in case I missed something)

 

2)

it seems like you think RLHF counts as an alignment technique

It's a candidate alignment technique.

RLHF is sometimes presented (by others) as an alignment technique that should give us hope about AIs simply understanding human values and applying them in out of distribution situations (such as with an ASI).

I'm not optimistic about that myself, but rather than arguing against it, I suggest we could empirically check if RLHF generalizes to an out-of-distribution situation, such as minecraft maybe [LW(p) · GW(p)]. I think observing the outcome here would effect my opinion (maybe it just would work?), and a main question of mine was whether it would effect other people's opinions too (whether they do or don't believe that RLHF is a good alignment technique).

 

3)

because you have to somehow communicate to the AI system what you want it to do, and AI systems don't seem good enough yet to be capable of doing this without some Minecraft specific finetuning. (Though maybe you would count that as Minecraft capabilities? Idk, this boundary seems pretty fuzzy to me.)

I would finetune the AI on objective outcomes like "fill this chest with gold" or "kill that creature [the dragon]" or "get 100 villagers in this area". I'd pick these goals as ones that require the AI to be a capable minecraft player (filling a chest with gold is really hard) but don't require the AI to understand human values or ideally anything about humans at all.

So I'd avoid finetuning it on things like "are other players having fun" or "build a house that would be functional for a typical person" or "is this waterfall pretty [subjectively, to a human]".

Does this distinction seem clear? useful?

This would let us test how some specific alignment technique (such as "RLHF that doesn't contain minecraft examples") generalizes to minecraft

Opinions on whether it's positive/negative to build tools like Cursor / Codebuff / Replit?

I'm asking because it seems fun to build and like there's low hanging fruit to collect in building a competitor to these tools, but also I prefer not destroying the world.

Considerations I've heard:

1. Reducing "scaffolding overhang" is good, specifically to notice if RSPs should trigger a more advanced RSP level
   1. (This depends on the details/quality of the RSP too)
2. There are always reasons to advance capabilities, this isn't even a safety project (unless you count... elicitation?), our bar here should be high
3. Such scaffolding won't add capabilities which might make the AI good at general purpose learning or long term general autonomy. It will be specific to programming, with concepts like "which functions did I look at already" and instructions on how to write high quality tests.
4. Anthropic are encouraging people to build agent scaffolding, and Codebuff was created by a Manifold cofounder [if you haven't heard about it, see here and here]. I'm mainly confused about this, I'd expect both to not want people to advance capabilities (yeah, Anthropic want to stay in the lead and serve as an example, but this seems different). Maybe I'm just not in sync

For-profit startup idea: Better KYC for selling GPUs

I heard^[1] that right now, if a company wants to sell/resell GPUs, they don't have a good way to verify that selling to some customer wouldn't violate export controls, and that this customer will (recursively) also keep the same agreement.

There are already tools for KYC in the financial industry. They seem to accomplish their intended policy goal pretty well (economic sanctions by the U.S aren't easy for nation states to bypass), and are profitable enough that many companies exist that give KYC services (Google "corporate kyc tool").

1. ^{^}

   From an anonymous friend, I didn't verify this myself

Some hard problems with anti tampering and their relevance for GPU off-switches

Background on GPU off switches:

It would be nice if a GPU could be limited^[1] or shut down remotely by some central authority such as the U.S government^[2] in case there’s some emergency^[3].

This shortform is mostly replying to ideas like "we'll have a CPU in the H100^[4] which will expect a signed authentication and refuse to run otherwise. And if someone will try to remove the CPU, the H100's anti-tampering mechanism will self-destruct (melt? explode?)".

TL;DR: Getting the self destruction mechanism right isn't really the hard part.

Some hard parts:

1. Noticing if the self destruction mechanism should be used
   1. A (fun?) exercise could be suggesting ways to do this (like "let's put a wire through the box and if it's cut then we'll know someone broke in") and then thinking how you'd subvert those mechanisms. I mainly think doing this 3 times (if you haven't before) could be a fast way to get some of the intuition I'm trying to gesture at
2. Maintenance
   1. If the H100s can never be opened after they're made, then any problem that would have required us to open them up might mean throwing them away. And we want them to be expensive, right?
   2. If the H100s CAN be opened, then what's the allowed workflow?
      1. For example, If they require a key - then what if there's a problem in the mechanism that verifies the key, or in something it depends on, like the power supply? We can't open the box to fix the mechanism that would allow us to open the box
      2. ("breaking cryptography" and "breaking in to the certificate authority" are out of scope for what I'm trying to gesture at)
3. False positives / false negatives
   1. If the air conditioning in the data center breaks, will the H100s get hot^[5], infer they're under attack and self destruct? Will they EXPLODE? Do we want them to only self destruct if something pretty extreme is happening, meaning attackers will have more room to try things?
4. Is the design secret?
   1. If someone gets the full design of our anti-tampering mechanism, will they easily get around it?
   2. If so, are these designs being kept in a state-proof computer? Are they being sent to a private company to build them?
5. Complexity
   1. Whatever we implement here better not have any important bugs. How much bug-free software and hardware do we think we can make? How are we going to test it to be confident it works? Are we okay with building 10,000 when we only think our tests would catch 90% of the important bugs?
6. Tradeoffs here can be exploited in interesting ways
   1. For example, if air conditioning problems will make our H100s self destruct, an attacker might focus on our air conditioning on purpose. After 5 times of having to replace all of our data center - I can imagine an executive saying bye bye to our anti tampering ideas.
      1. I'm trying to gesture at something like "making our anti tampering more and more strict probably isn't a good move, even from a security perspective, unless we have good idea how to deal with problems like this"

In summary, if we build anti-tampering mechanisms in GPUs that an adversaries have easy access to (especially nation states) then I don't expect this will be a significant problem for them to overcome.

(Maybe later - Ideas for off-switches that seem more promising to me)

Edit:

1. See "Anti Tempering in a data center you control provides very different tradeoffs" [LW(p) · GW(p)]
2. I think it's worth exploring putting the authrization component on the same chip as the GPU [LW(p) · GW(p)]

1. ^{^}

   For example bandwidth, gps location, or something else. the exact limitations are out of scope, I mainly want to discuss being able to make limitations at all

2. ^{^}

   The question of "who should be the authority" is out of scope, I mainly want to enable having some authority at all. If you're interested in this, consider checking out Vitalik's suggestion too, search for "Strategy 2"

3. ^{^}

   Or perhaps this can be used for export controls or some other purpose

4. ^{^}

   I'm using "H100" to refer to the box that contains both the GPU chip and the CPU chip

5. ^{^}

   I'm using "getting hot means being under attack" as a toy example for a "paranoid" condition that someone might suggest building as one of the triggers for our anti-tampering mechanism. Other examples might include "the box shakes", "the power turns off for too long", and so on.

"Protecting model weights" is aiming too low, I'd like labs to protect their intellectual property too. Against state actors. This probably means doing engineering work inside an air gapped network, yes.

I feel it's outside the Overton Window to even suggest this and I'm not sure what to do about that except write a lesswrong shortform I guess.

Anyway, common pushbacks:

1. "Employees move between companies and we can't prevent them sharing what they know": In the IDF we had secrets in our air gapped network which people didn't share because they understood it's important. I think lab employees could also understand it's important. I'm not saying this works perfectly, but it works well enough for nation states to do when they're taking security seriously.
2. "Working in an air gapped network is annoying": Yeah 100%, but it's doable, and there are many things to do to make it more comfortable. I worked for about 6 years as a developer in an air gapped network.

Also, a note of hope: I think It's not crazy for labs to aim for a development environment that is world leading in the tradeoff between convenience and security. I don't know what the U.S has to offer in terms of a ready made air gapped development environment, but I can imagine, for example, Anthropic being able to build something better if they take this project seriously, or at least build some parts really well before the U.S government comes to fill in the missing parts. Anyway, that's what I'd aim for

Ryan and Buck wrote [LW · GW]:

> The control approach we're imagining won't work for arbitrarily powerful AIs

Okay, so if AI Control works well, how do we plan to use our controlled AI to reach a safe/aligned ASI?

Different people have different opinions. I think it would be good to have a public plan so that people can notice if they disagree and comment if they see problems.

Opinions I’ve heard so far:

1. Solve ELK / mechanistic anomaly detection / something else that ARC suggested
2. Let the AI come up with alignment plans such as ELK that would be sufficient for aligning an ASI
3. Use examples of “we caught the AI doing something bad” to convince governments to regulate and give us more time before scaling up
4. Study the misaligned AI behaviour in order to develop a science of alignment
5. We use this AI to produce a ton of value (GDP goes up by ~10%/year), people are happy and don’t push that much to advance capabilities even more, and this can be combined with regulation preventing an arms race and pause our advancement
6. We use this AI to invent a new paradigm for AI which isn’t based on deep learning and is easier to align
7. We teach the AI to reason about morality (such as consider hypothetical situations) instead of responding with “the first thing that comes to its mind”, which will allow it to generalize human values not just better than RLHF but also better than many humans, and this passes a bar for friendliness

These 7 answers are from 4 people.

Off switch / flexheg / anti tampering:

Putting the "verifier" on the same chip as the GPU seems like an approach worth exploring as an alternative to anti-tampering (which seems hard [LW(p) · GW(p)])

I heard^[1] that changing the logic running on a chip (such as subverting an off-switch mechanism) without breaking the chip seems potentially hard^[2] even for a nation state.

If this is correct (or can be made correct?) then this seems much more promising than having a separate verifier-chip and gpu-chip with anti tampering preventing them from being separated (which seems like the current plan (cc @davidad [LW · GW] ) , and seems hard [LW(p) · GW(p)]).

James Petrie shared hacks that seems relevant, such as:

Courdesses built a custom laser fault injection system to avoid anti-glitch detection. A brief pulse of laser light to the back of the die, revealed by grinding away some of the package surface, introduced a brief glitch, causing the digital logic in the chip to misbehave and open the door to this attack.

It intuitively^[3] seems to me like Nvidia could implement a much more secure version of this than Raspberry Pi (serious enough to seriously bother a nation state) (maybe they already did?), but I'm mainly sharing this as a direction that seems promising, and I'm interested in expert opinions.

1. ^{^}

   From someone who built chips at Apple, and someone else from Intel, and someone else who has wide knowledge of security in general

2. ^{^}

   Hard enough that they'd try attacking in some other way

3. ^{^}

   It seems like the attacker has much less room for creativity (compared to subverting anti-tampering), and also it's rare^[4] to hear from engineers working on something that they guess it IS secure.

4. ^{^}

   See xkcd

For profit AI Security startup idea:

TL;DR: A laptop that is just a remote desktop ( + why this didn't work before and how to fix that)

Why this is nice for AI Security:

1. Reduces the amount of GPUs that will be sent to customers
2. Somewhat better security for this laptop since it's a bit more like defending a cloud computer. Maybe labs would use this to make the employee computers more secure?

Network spikes: A reason this didn't work before and how to solve it

The problem: Sometimes the network will be slow for a few seconds. It's really annoying if this lag also affects things like they keyboard and mouse, which is one reason it's less fun to work with a remote desktop 

Proposed solution: Get multiple network connections to work at the same time:

Specifically, have many network cards (both WIFI and SIM) in the laptop, and use MPTCP to utilize them all.

Also, the bandwidth needed (to the physical laptop) is capped at something like "streaming a video of your screen" (Claude estimates this as 10-20 Mbps if I was gaming), which would probably be easy to get reliable given multiple network connections. Even if the the user is downloading a huge file, the file is actually being downloaded into the remote computer in the data center, enjoying way faster internet connection than a laptop would have.

Why this is nice for customers

1. Users of the computer can "feel like" they have multiple GPUs connected to their laptop, maybe autoscaling, and with a way better UI than ssh
2. The laptop can be light, cheap, and have an amazing battery, because many of the components (strong CPU/GPU/RAM) aren't needed and can be replaced with network cards (or more batteries).

Both seem to me like killer features.

MVP implementation 

Make:

1. A dongle that has multiple SIM cards
2. A "remote desktop" provider and client that support MPTCP (and where the server offers strong GPUs).

This doesn't have the advantages of a minimalist computer (plus some other things), but I could imagine this would be such a good customer experience that people would start adopting it.

I didn't check if these components already exist.

Thanks to an anonymous friend for most of this pitch.

Posts I want to write (most of them already have drafts) :

(how do I prioritize? do you want to user-test [LW · GW] any of them?)

1. Hiring posts
   1. For lesswrong
   2. For Metaculus
2. Startups
   1. Do you want to start an EA software startup? A few things to probably avoid [I hate writing generic advice like this but it seems to be a reoccurring question]
   2. Announce "EA CTOs" plus maybe a few similar peer groups
3. Against generic advice - why I'm so strongly against questions like "what should I know if I want to start a startup" [this will be really hard to write but is really close to my heart]. Related:
4. How to find mentoring for a startup/project (much better than reading generic advice!!!)
5. "Software Engineers: How to have impact?" - meant to mostly replace the 80k software career review, will probably be co-authored with 80k
6. My side project: Finding the most impactful tech companies in Israel
7. Common questions and answers with software developers
   1. How to negotiate for salary (TL;DR: If you don't have an offer from another company, you're playing hard mode.  All the other people giving salary-negotiation advice seem to be ignoring this)
   2. How to build skill in the long term
   3. How to conduct a job search
      1. When focusing on 1-2 companies, like "only Google or OpenAI"
      2. When there are a lot of options
   4. How to compare the companies I'm interviewing for?
   5. Not sure what you are looking for in a job?
   6. I'm not enjoying software development, is it for me? [uncertain I can write this well as text]
   7. So many things to learn, where to start?
   8. I want to develop faster / I want to be a 10x developer / Should I learn speed typing? (TL;DR: No)
   9. Q: I have a super hard bug that is taking me too long panic!
8. Goodheart Meta: Is posting this week a good idea at all? If more stuff is posted, will less people read my own posts?

How can you help me:

TL;DR: Help me find what there is demand for

Comment / DM / upvote people's comments

Thx!

Is this an AGI risk?

A company that makes CPUs that run very quickly but don't do matrix multiplication or other things that are important for neural networks.

Context: I know people who work there