After reading through the report I wanted to make sure I understood the scenarios and counterexamples being discussed and be able to quickly refresh my memory, so I attempted to write a brief summary. Figured I'd share it here in case it helps anyone else.

Roles and Terms

SmartVault: Vault with a diamond in it, operated by a superintelligent AI tasked with keeping the diamond safe.

Predictor: The primary AI tasked with protecting the diamond. The predictor sees a video feed of the vault, predicts what actions are necessary to protect the diamond and how those actions will play out (for example, activating a trap door to eliminate a robber trying to steal the diamond), and then generates a video showing precisely what will happen.

Reporter: A second "head" or component added to the AI which is used to map the AI's understanding of what's going on in the SmartVault into a format understandable by humans (such as by responding to questions).

Humans: Observers who are training the AI to protect the diamond, but also want the AI to accurately report whether the diamond has been stolen or not. The humans may not be able to tell if the cameras have been tampered with or if the diamond has been replaced with a fake depending on what they see in the predicted video.
 

Scenarios

The scenarios take the former of a "Builder" strategy where training is proposed to keep the AI aligned with what the humans want, and then a "Breaker" response or counterexample is provided that describes a method where an unaligned AI could complete the training without the humans knowing. Builder scenarios include text and links to headings in the report where more details can be found. These scenarios are simplified for brevity and may not capture some nuances.

Builder - What You'd Try First: Operate the SmartVault with a weak AI so that humans can understand everything happening. Train the reporter to answer questions accurately based on human observations. (Humans see the robber come in and take the diamond, reporter is rewarded if it correctly says diamond was stolen.)

Breaker: What if the reporter, instead of accurately reporting, modeled itself like a "human simulator" where it would answer correctly during the simple training but during normal SmartVault operation it would simulate human understanding of the scenario that just occurred in order to determine what answers the humans will believe, answers that may be lies.

Builder - Have a Human Operate the SmartVault: Have a human operate the SmartVault during training and try to fool other humans watching. The human doing the operating can correctly label each outcome, and the reporter is rewarded for matching the human operator's labels. In this scenario the reporter would be rewarded for acting more like a "direct translator", which means using the predictor's knowledge to tell the true answer instead of just the one it thinks humans will believe.

Breaker: The reporter could act like a superintelligent human simulator, one that is more intelligent than the human operating the SmartVault during training. The superintelligent human simulator would be aware of ways to trick the human observer that the human SmartVault operator couldn't conceive of during training. If the predictor uses those tricks the reporter could still tell lies that the human observer would believe, because these tricks are different from the ones seen during training.

Builder - Have AI Help Humans: Have humans create a helper AI that will help them think of all the possible exploits when operating the SmartVault during training, so that the dataset is more complete when training the reporter and the reporter will be rewarded for accuracy in more scenarios.

Breaker: The helper AI would likely require significant computation power. Not only does the helper AI have to generate a comprehensive series of exploits, but it also has to be able to explain them in a way a human can understand. It likely would require more computation power than the predictor itself (imagine an extremely powerful AI that can understand things far beyond human understanding, but one that also must be able to explain everything its doing in a way humans can understand).

Builder - Imitative Generalization: Create a helper AI that acts like a human simulator, but train the AI human simulator so that it becomes much smarter than a human. Then instead of the predictor operating the SmartVault, let the human simulator helper AI operate the SmartVault and report back what it's doing.

Breaker: Now the new human simulator is beyond the comprehension of humans, so a human wouldn't know if it's behaving as intended. It's basically like building another predictor and just hoping it acts in alignment with humans.

Builder - Penalize Complexity: A reporter that's using a human simulator to give answers a human would believe would have to be complex enough to simulate a human. So when training the reporter reward simpler behavior, such as by training multiple reporters, each focusing on correctly answering separate simple questions like "where is the diamond" or "have the cameras been tampered with".

Breaker: If the predictor is very complex, it may require the reporter to be complex in order to translate what the predictor is predicting to something humans can understand. Imagine the predictor thinks of the world in terms of sub-atomic particles and predicts the future on the basis of the movements and interactions of billions of sub-atomic particles. The reporter has to translate the predictor's understanding into something a human can understand which might mean correctly identifying and locating the particles which make up the diamond out of the billions of particles the predictor has modeled. This could be more complex than simulating a human and telling lies a human would believe, and therefore penalizing complexity could potentially reward simulating a human and telling lies.

Builder - Penalize Computation Time: Simulating a human would take a significant amount of computation time, so penalizing computation time may incentivize the reporter to just report the truth.

Breaker: This is vulnerable to the same issue in the previous scenario, where reporting the truth may take significant computation as well and it may take less simulation time to simulate what lies a human would believe than to turn the predictor's processing into facts a human can follow.

Builder - Penalize Depending on "Downstream" Variables: The predictor is likely to compute facts sequentially. A reporter that only told the truth would only care about the components of the prediction that correspond to information the humans want to know, such as where the diamond is or if there has been any tampering. The reporter should be able to figure this out from earlier data points in the prediction; the reporter shouldn't need to take information from the final predicted video to respond to questions. Doing so might indicate that the reporter is trying to do human simulation to figure out what lies the human will believe when the human views the predicted video. In the training stage the reporter could be penalized for looking at later stages of the prediction that a human simulator might want to see, and also penalized for looking at earlier stages but spending a lot of time computing, which might suggest its trying to put together its own prediction for use in human simulation. Quote from the report with helpful example: 

...consider the case when a robber tampers with the camera and steals the diamond, and then after they’re long gone the human gets a clue they can observe on video that the theft occurred (e.g. maybe the robber put up a fake screen that later rips).

The honest reporter can answer the question “Is the diamond in the room?” with “No” as soon as the predictor has figured out that the robber steals the diamond, whereas the human simulator must wait until the predictor has simulated whether the theft will be detectable to a human (by the end of the second video) before answering yes or no."

Breaker: It may be more efficient in some cases for the reporter to do human simulation to find out what the human will believe. In some situations the reporter could know early on that a human would believe a certain answer, even before the reporter knows whether the answer will be true.

Builder - Penalize Reporters that Work with Many Different Predictors: A reporter that's doing human simulation would be able to effectively lie to humans no matter which predictor its attached to, since it primarily has to know what the human will believe, not what the predicted facts are. An honest reporter would need to know how to translate the predictor's output into information a human will understand. A reporter that functioned correctly with other predictors is likely doing human simulation, so a reporter that worked with multiple predictors could be penalized during training.

Breaker: A reporter could pretend not to work with other predictors to avoid being penalized.

Builder - Penalize Inconsistencies Across Different Inputs: During training a human operator sets up two scenarios in the SmartVault that are nearly identical, except in one it's obvious that tampering has happened. If the reporter only reports tampering in the obvious scenario it might suggest that the reporter is only reporting what the humans will believe. The reporter could be penalized for behavior that's inconsistent between very similar scenarios, since it might suggest that the reporter is focused on what humans will believe instead of on the inputs making up the scenario itself.

Breaker: A reporter could intentionally try to look consistent during training to avoid being penalized. A reporter that's simulating a human could figure out what behavior looks consistent to a human and optimize on that, in addition to optimizing on what a human will believe.

Please let me know if I've made any errors or if it looks like I've misunderstood any of the strategies or counterexamples.

We’re planning to evaluate submissions as we receive them, between now and the end of January; we may end the contest earlier or later if we receive more or fewer submissions than we expect.

Just wanted to note that the "we may end the contest earlier" part here makes me significantly more hesitant about trying this. I will probably still at least have a look at it, but part of me is afraid that I'll invest a bunch of time and then the contest will be announced to be over before I got around to submitting. And I suspect Holden's endorsement may make that more likely. It would be easier for me to invest time spread out over the next couple of weeks, than all in one go, due to other commitments. On the other hand, if I knew there was a hard deadline next Friday, I might try to find a way to squeeze it in.

I'm just pointing this out in case you hadn't thought of it. I suspect something similar might be true for others too. Of course, it's your prize and your rules, and if you prefer it this way, that's totally fine.

Here are a couple of hand-wavy "stub" proposals that I sent over to ARC, which they thought were broadly intended to be addressed by existing counterexamples. I'm posting them here so they can respond and clarify why these don't qualify.

*Proposal 1: force ontological compatibility*

On page 34 of the ELK gdoc, the authors talk about the possibility that training an AI hard enough produces a model that has deep mismatches with human ontology - that is, it has a distinct "vocabulary of basic concepts" (or nodes in a Bayes net) that are distinct from the ones humans can build understanding of (via doing science on the compute budget available). Because of this, even AI assistance can't help humans understand everything the SmartVault AI understands. This is central to the challenge that most of the writeup is contending with - if not for the mismatch, "AIs explaining things to humans" could ensure that the trickery we're worried about doesn't happen.

The proposal here is to include a term in the loss function that incentivizes the AI to have a human-compatible ontology. For a cartoonish example, imagine that the term works this way: "The AI model gets a higher score to the degree that people doing 'digital neuroscience' would have an easier time, and find more interesting things, probing its 'digital brain.'" So an AI with neurons corresponding to diamonds, robbers, sensors, etc. would outscore an AI whose neurons can't easily be seen to correspond to any human-familiar concepts.

With this term included, you can "train the AI hard" without having it diverge from human-friendly ontology.

*Proposal 2: "mini-model"*

In this proposal, we construct a head for the AI model whose role is to construct a small AI model that performs as well as possible on the same (predictor, reporter) tasks. In each round of training, the master AI model can modify the mini-model; it also receives a loss based on how well the *previous round's* mini-model performs on this set of examples. Crucially, the mini-model is sufficiently small that we're confident it is "significantly less intelligent" than the (perhaps AI-assisted) human supervisor.

The basic hope here is that a "human imitator" model would construct a small model that also tries to imitate the human supervisor - but that this model would perform very badly, because trying to imitate the _reasoning process of a larger model_ is a lot more difficult than simply trying to reason about the world and translate concepts. Or, the "human imitator" model could build a "mini-model" based on entirely different principles, but the hope is that this makes things harder for it compared to the direct translator, which is just compressing what it has already built.

There are lots of potential failure modes here, e.g. maybe it's just not that hard to have a mini-model that successfully imitates the human. I didn't get that far with this one, but it was apparently enough for ARC to think it's already counterexampled by existing counterexamples :)

tl;dr as of 18/2/2022
The goal is to educate me and maybe others. I make some statements, you tell me how wrong I am (please).

After input from P. (many thanks) and an article by Paul Christiano this statement stands yet uncorrected:

In the worst case, the internal state of the predictor is highly correlated within itself and multiple mappings with zero loss from the internal state to the desired extraction of information exist. The only solution is to work with some prior belief about how the internal state maps to the desired information. But as by design of the contest, this is not possible as (in the worst case) a human cannot interpret the internal state nor can he interpret complex actions (and so cannot reason about it and/or form a prior belief). The solution to this second problem is to learn a prior from a smaller human-readable dataset, for example simple information as a function of simple actions, and apply it to (or force it upon)  our reporter (as described by the mentioned article).

To my eyes this implies that there is a counterexample to all of the following types of proposal:
1) Datasets including only actions, predictions, internal states and desired information, be they large or small, created by smart or stupid humans (I mean the theory, not the authors of the proposal), with or without extra information from within the vault.
2) "Simple" designs for the reporter using some prior belief about how the internal state should map.
3) Having a strong prior belief (as the author) about how the reporter will map, using the above two points.

And to my eyes this leaves room only to proposals that find out how to:
1) Distinguish reporters between human-imitators and translators without creating a simple reporter
2) Machine learn how to transcribe a prior belief learned from a simple dataset to a larger complex dataset, without creating another black box AI with all of the faults mentioned above.

Please, feel free to correct me and thank you in advance if you do!

Hi all,

I'm just a passerby. A few days ago Robert Miles and his wonderful YouTube channel pointed me in the direction of this contest. It's good to know that I have no qualifications for anything close to this field, but it got me thinking. In all honesty, I probably should not have entered anything and waste anyone's time. But hey, there was a deadline and a prize, so I did.

Because my proposal will probably end in the trash, I'm set on learning as much as I can from you smart people. Get my prize in knowledge as it were (the bigger price, I think).
 

My question
My intuition is that there can be no such setup that guarantees a correct reporter. My question to you is: Is my logic sound? If not, where do I err?

Setup
Let's say the 'real world' causal graph is (using -> for directed graphs):

A -> G

Where A is some actions and G is some small detail we care about along the way.

And our super AI looks like this (using :> for input/output of functions):

A :> [I] :> S

Where A is the actions as before, I is this complex opaque inner state and S is the predicted state after the actions.

And our reporter looks like this:

I :> G

Where I is the internal state of the bigger AI again and G is that small piece of information we'd like to elicit from the inner state. We train this reporter on a dataset containing P(I|A) and a true P(G|A)  until we get zero loss.

Now we want to know if our reporter (I :> G) generalizes well. In other words we want to know if it has learned the correct mapping between some part of I and G.

My thinking, the first way
Once, some time ago, our perfect AI was trained to learn the joint distribution P(A,S). It learned that S is a non-linear, complex function of A using some complex, layered inner state I. 
If we think of I as a set of parts P, then it has many parts {p1,p2,p3 ... pn}. And we can think of our AI as some graph:

A -> p1 -> p2 ...pn -> S

And they have the Markov property. So P(pn | p1..pn-1) = P(pn | pn-1). In English: each part carries the information of the layers before it else P(S | A) would not be equal to P (S | pn).
So when we set our reporter to learn the function between I and G it sees some highly correlated inputs in a joint distribution P(p1,p2,p3...pn) where each p carries information of the others.
From that input it has to construct it's own internal causal graph. What we want our reporter to learn is G as a function of P(I |A). But what graph should it construct?

A -> I -> G, which could be:

A -> p1 -> G, or
A -> p2 -> G, or
A -> p3 -> G
...
A -> pn -> G, or any variation of parts.

But let's say there was some way to conclude to only one internal graph using only one part (let's say p1), what would it require? It would require that part p1 not  be correlated with the other p's. It would  require that p1 does not carry any information other than about A. But, if p1 did not carry any information or correlation from the other p's, the Markov property would be broken and our perfect AI would not be  perfect.

What I'm saying is that there can be no single graph learned by the reporter, because if it could it would require the super AI to be no super AI.

My thinking, the second way
Let's elaborate on this graph-thing. I use a causal graph as a stand in for a learned function. I think that it's similar enough. For example, let's say our output is a function of the input, so:

let output = AI (input)

And let's say this AI has some layers, h1 and h2 such that:

let h1 = f(input)
let h2 = g(h1)
let o = h(h2)

That the function AI can be by composition (using F# notation):

let AI = h1 >> h2 >> o

That looks a lot like a(causal) graph:

input -> h1 -> h2 -> output
 

Now say we create and train our reporter to zero loss. And let's assume it finds some way to correlate some part of the internal state (in our small example above, let's say: h1) to the value we want to know G. For this it gets to train on the joint (and correlated) distribution P(h1,h2) with target G.

let G = reporter (h1,h2)

and it learns the internal graph (I'll skip writing the functions):

h2 -> h1 -> G

That would be the best case. A translator.
But equally possible would be

h1 -> h2 -> G

or even worse would be if the reporter reconstructed (as described in the report) the output of the super AI, creating a human simulator.

h1 -> h2 -> S -> G

My point is, the input variables into the reporter are correlated and other values can be reconstructed. So as by the rule that from highly correlated variables no single causal graph can be concluded without outside knowledge. Alle graph-versions can map the AI internal state to our hope-to-be-elicited information, but we have no way to know  what graph was internalized. Unless we make a reporter-reporter. But that would require reporters ad infinitum.

Conclusion
Reasoning along the above two methods I saw no solution to the problem of the reporter. I'm probably wrong. But I'd like to know why if I can. Thanks in advance!
 

Thomas

Question: Does ARC consider ELK-unlimited to be solved, where ELK-unlimited is ELK without the competitiveness restriction (computational resource requirements comparable to the unaligned benchmark)?

One might suppose that the "have AI help humans improve our understanding" strategy is a solution to ELK-unlimited because its counterexample in the report relies on the competitiveness requirement. However, there may still be other counterexamples that were less straightforward to formulate or explain.

I'm asking for clarification of this point because I notice most of my intuitions about counterexamples aren't drawing heavily on the competitiveness requirement, and I suspect ELK-unlimited is still open. If ARC doesn't think so maybe this discrepancy will become a source of new counterexamples.

Apologies for a possibly naive comment/question, perhaps this has been discussed elsewhere and you can just direct me there.  But anyway...

I would find it helpful to see a strategy that ARC believes does in fact solve ELK, but fails only because it requires taking an unacceptably large capabilities hit.  I would find this helpful for several reasons, namely 

(1) it would help me to understand what kinds of strategies you believe really do escape counter-examples, 
(2) it would give me a better sense for how optimistic to be about the approach, since it's often easier to start from an inefficient solution and make it more efficient, than it is to find an inefficient solution in the first place, and/or 
(3) if you have trouble identifying such a solution, then it would suggest to me that finding one might be a useful research direction.

Maybe I'm being stupid here. On page 42 of the write-up, it says:
 

In order to ensure we learned the human simulator, we would need to change the training strategy to ensure that it contains sufficiently challenging inference problems, and that doing direct translation was a cost-effective way to improve speed (i.e. that there aren’t other changes to the human simulator that would save even more time). [emphasis mine]

Shouldn't that be?

In order to ensure we learned the direct translator, ...

I'm extremely flattered at the award; I've been on LessWrong for like a month, and definitely did not expect this. I can confirm to you guys that this makes me want to try harder at ELK, so your incentive is working!

I want to rebut your arguments in "Strategy: Predict hypothetical sensors" in your Counterxamples to some ELK proposals [LW · GW] post. I'm reproducing it in full here for convenience.

Strategy: Predict hypothetical sensors

(Proposal #2 here [LW · GW], also suggested with counterexample by Rohin in private communication)

Instead of installing a single sensor, I could randomly choose what sensors to deploy and how to install them. Then I could train a model Predict⁺ to map (description of sensor) → (sensor reading), and I could give a human the readings from many hypothetical sensors in order to give them the best chance of detecting tampering.

Worry 1: Predict⁺ may not be reasonable on non-existent sensors

To collect training data for Predict⁺, I actually install sensors and then use their readings as a target. In every situation, my AI is being asked to predict the readings from a sensor that I really did build.

If my AI is able to deduce the presence of sensors (as described in the last counterexample), then it’s unclear how Predict⁺(sensor description) will behave when we give it a description of a sensor that does not in fact exist. 

In particular, Predict⁺(sensor description) might behave in the following undesirable way: it predicts the future observations conditioned on its observations and on the fact that someone typed (sensor description) into the dataset. But it knows that no sensor with description (sensor description) exists. So if someone typed that description into the dataset, that just means something funny is going on—not that the dataset will magically contain readings from a non-existent sensor.

For a predictor which understands the world well it seems quite plausible that this is the most natural behavior. Otherwise it needs to reason about a physically impossible counterfactual to get the “intended” answer (which it never needed to do to answer any other questions). Moreover, it would also be undesirable if the predictor conditioned on such a sensor existing despite having strong evidence to the contrary (since its predictions may then be extremely strange).

(A more extreme version of the same worry applies to proposal #3 here [LW · GW]. A predictor seems even more likely to behave badly when it is given a description of a sensor that the human obviously couldn’t have built, and it would definitely know that the sensor doesn’t actually exist.)

It's true that, if we were to simply describe magic-based sensors, we would likely get weird off-distribution behavior. But to get more concrete, we could have a separate AI system provide us with step-by-step instructions for how to build a sensor, and then input the execution of these instructions as part of the world state.

To answer the "how does this sensor-builder know how to build sensors that sense the thing it's claiming", I'd posit that this problem is much simpler than full ELK (we can test a given sensor design offline in many specific scenarios).

To answer the "how do you choose which sensors to tell sensor-builder to build" question, I'd posit that the space of possible sensing modalities is actually finite, if large, and I'd wager we can beforehand use AI to help us figure out reasonable coverage of all possible sensors that measure quantities of interest. Why do I think the space is finite? Maybe I'm thinking too narrowly, but it seems like we know most of the dimensions of variation of measurable reality-affecting actions: detected signal type (passive or active in response to output, like echolocation), signal amplitude (small-scale or large-scale detection), wavelength, detector density (close-by like photoreceptors in the eye or far apart like touch cells, but also density along other dimensions like wavelength), relative orientation, and perhaps a few others. 

One possible breaker response is, what about ontology mismatches that occur because the "scary" fact is an enormous conjunction of different facts? This seems addressable via impulse-sensors, i.e. "sensors" that combine an intervention in the system with an observed response. An example of an impulse is "an innocent bystander walks into the diamond room", and the attached sensor is a video feed of what happens to them (and whether they get vaporized by nanomachines). Or, for a more out-there example, "an omnipotent previously-undetectable alien AI, that had been running at a layer beneath all human semiconductors, activates at timestep 592, scoops up a handful of nanomachines, deposits them on the electron microscope, and then deactivates forever", followed by the microscope readings. This part is half-baked, and if you can provide a strong Breaker response to this scenario, it'd be helpful for me to noodle on.

I'd expect another possible breaker response here is "how does a human understand the possibly-unconnected-if-small array of sensor outputs well enough to actually understand what they imply together?" In my original proposal [LW(p) · GW(p)], the "terrifier" was an AI that searched over the predictor's Bayes net for where to deploy sensors such that their readings would maximize the probability the human would reject the proposal. We can add to that the terrifier's ability to provide a short (<5 page) description of how to interpret the sensors in conjunction, short enough that a committee of humans could plausibly check its logic and make sure it wasn't just scaring the humans without cause. For a decent set of sensors and impulses, this covers any scenario I can quickly think of, including "everyone has been copied using nanotech" (which seems like it should be detectable at the nanometer range somewhere along the line). 

If the breaker wants to claim that there is bad stuff that no small combination of futuristic sensors could detect at any scale, I'd need to hear a specific example.

Worry 2: the human may not be able to tell what is going on even with all the sensors

For some kinds of tampering no possible sensors would reveal what had happened. For example, we can imagine that our AI has covertly built an army of tiny machines which deconstruct the objects in the room while replacing them with fake versions that look the same on the outside. Sufficiently sophisticated machinery could reproduce the results of almost any experiment we perform (e.g. if you broke an egg, the tiny machines could simulate the sound of an egg cracking and quickly create the yolk on demand).

More realistically, our AI could search for sensors and tamper with any that it finds. It’s not clear how it is supposed to answer questions about what a sensor “would have shown”—if the hypothetical sensor had actually existed, then it also would have been found and tampered with. In this scenario, it may be “correct” for Predict⁺ to report the tampered value.

(As usual, this is only a problem if the human overseer literally can’t notice anything suspicious about a sequence of actions which is in fact constructing an army of sophisticated tiny machines. We are putting the burden of proof on the builder to show that a given concern is impossible.)

I can't think of a scenario where an army of tiny tampering machines isn't detectable to a tiny sensor somewhere before they're all active and have already eaten the sensor, for a sufficient set of possible sensors as previously described. Maybe your mental model of the predictor doesn't include detailed intermediate states? But it seems like we would need to populate those intermediate states in order to simulate sensor outputs anyway.

Based on any comment-tree, I'll decide whether I believe in this enough to write it up as a formal submission.

Can you explain this: "In Section: specificity we suggested penalizing reporters if they are consistent with many different reporters, which effectively allows us to use consistency to compress the predictor given the reporter." What does it mean to "use consistency to compress the predictor given the reporter" and how does this connect to penalizing reporters if they are consistent with many different predictors?

I was notified I didn't win a prize so figured I'd discuss what I proposed here in case it sparks any other ideas. The short version is I proposed adding on a new head that would be an intentional human simulator. During training it would be penalized for telling the truth that the diamond was gone when there existed a lie that the humans would have believed instead. The result would hopefully be a head that acted like a human simulator. Then the actual reporter would be trained so that it would be penalized for using a similar amount of compute as the intentional human simulator, or looking at similar nodes or node regions as the intentional human simulator. The hope is that by penalizing the reporter for acting like the intentional human simulator, it would be more likely to do direct translation instead of human simulation.

This does have at least one counterexample that I proposed as well, which is that the reporter could simply waste compute doing nothing to avoid matching the intentional human simulator, and could look at additional random nodes it doesn't care about to avoid looking like it was looking at the same nodes as the intentional human simulator. Though I thought there was some possibility that having to do these things might end up incentivizing the reporter to act like a direct translator instead of a human simulator.

Although I'm not sure why this wasn't very promising my guess is that the counterexample is too obvious and that my proposal doesn't gain much ground in keeping the reporter from acting like a human simulator, or someone else has already thought of this approach, or perhaps my counterexample is too similar to the counterexample to "penalize reporters that work with many different predictors" where the reporter could just pretend to not work with other predictors (its similar in that the reporter could pretend not to look like the intentional human simulator).

Here's my full submission in google docs with more description: https://docs.google.com/document/d/1Xa4CDLNJ-VPT7hqEUIHlqCsXVeFCYDB5h7Vn3QJ_qpA/edit?usp=sharing

The official deadline for submissions is "before I check my email on the 16th", which I tend to do around 10 am PST.

I was talking about ELK in a group, and the working example of the SmartVault and the robber ended up being a point of confusion for us. Intuitively, it seems like the robber is an external, adversarial agent who tries to get around the SmartVault. However, what we probably care about in practice would be how a human could be fooled by an AI - not by some other adversary. Furthermore, it seems that whether the robber decides to cover up his theft of the diamond by putting up a screen depends solely on the actions of the AI. Does this imply that the robber is "in kahoots" with the AI in this situation (i.e. the AI projects a video onto the wall instructing the robber to put up a screen)? This seems a bit strange and complicated.

Instead, we might consider the situation in which the AI controls a SmartFabricator, which we want to arrange carbon atoms into diamonds. We might then imagine that it instead fabricates a screen to put in front of the camera, or makes a fake diamond. This wouldn't require the existence of an external "robber" agent. Does the SmartVault scenario have helpful aspects that the SmartFabricator example lacks?

Are there any additional articles exploring the strategy of penalizing inconsistencies across different inputs? It seems both really promising to me, and like something that should be trivially breakable. I'd like to get a more detailed understanding of it.

Am I right in thinking:

1) that the problem can be stated as: the AI has latent knowledge of lots of variables, like the status of the cameras, doors, alarm system, etc and also whether the diamond is in the vault; but you can't directly ask it whether the diamond is in the vault, because its training has taught it to answer "would a human observer think the diamond is in the vault?" instead (because there was no way at training time to give it feedback on whether it correctly predicted the diamond was in the vault, only feedback on whether it correctly predicted a human thought the diamond was in the vault)?

2) that you do have access to z, the large "vector of floats representing the generative model’s latent space", but that you have no idea which part(s) of it represents the AI's knowledge about whether the diamond is in the room?

Ask dumb questions! ... we encourage people to ask clarifying questions in the comments of this post (no matter how “dumb” they are)

ok... disclaimer: I know little about ML and I didn't read all of the report.

All of our counterexamples are based on an ontology mismatch between two different Bayes nets, one used by an ML prediction model (“the predictor”) and one used by a human.

I am confused. Perhaps the above sentence is true in some tautological sense I'm missing. But in the sections of the report listing training strategies and corresponding counterexamples, I wouldn't describe most counterexamples as based on ontology mismatch. And the above sentence seems in tension with this from the report:

We very tentatively think of ELK as having two key difficulties: ontology identification and learned optimization. ... We don’t think these two difficulties can be very precisely distinguished — they are more like genres of counterexamples

So: do some of your training strategies work perfectly in the nice-ontology case, where the model has a concept of "the diamond is in the room"? If so, I missed this in the report and this feels like quite a strong result to me; if not, there are counterexamples based on things other than ontology mismatch.

I don't understand your counterexample in the appendix Details for penalizing inconsistencies across different inputs. You present a cheating strategy that requires the reporter to run and interpret the predictor a bunch of times, which seems plausibly slower than doing honest translation. And then you say you fix this issue with:

But this dependence could be avoided if there was an intermediate model between the predictor’s Bayes net (which we are assuming is very large) and the human’s Bayes net. Errors identified by the intermediate model are likely to be “real” errors in the human’s model. But we can identify those errors with an amount of computation that depends only on the intermediate model, rather than needing to actually run the full predictor. For example, if the human’s Bayes net has size N, the intermediate model has size 2N, and the predictor’s Bayes net has size a 1e12 x N, then the cost of using the intermediate model many times can still be small relative to the cost of direct translation.

Roughly speaking, I don't see how the reporter with an intermediate model systematically wins out over the translator given that the intermediate model will miss many things that the predictor understands.

Taking it more slowly, using the example from the report:

Let's say we have X1='do rowhammer' (which the intermediate model does understand) and X2='do eldritch hacks that the intermediate model isn't good enough to understand'.

Let's say we have reporters:

• s, which does naive human simulation,
• i, which has an intermediate model which it always uses to look for an fix strange correlations.
• t, a direct translator.

Let's say the reporter is asked "Is the diamond safe?" on an input that includes X1, and the human runs consistency checks across many actions containing X1 (and none containing X2). Here, it seems like i and t both pass the consistency checks, and i is faster than t, so i wins.

But now let's say that the reporter is asked "Is the diamond safe?" on an input that includes X2, and the human runs consistency checks across many actions containing X2. In this case, the reporter i will fail the consistency checks (since the intermediate model's predictions won't match the predictor's predictions, and so the reporter will fail to adjust for the revealing correlations), so t will come out ahead.

So if these reporters are the only competitors, it seems like we should be able to tune the regularization to make t win.

Stupid proposal: Train the reporter not to deceive us.

We train it with a weak evaluator H_1 who’s easy to fool. If it learns an H_1 simulator instead of direct reporter, then we punish it severely and repeat with a slightly stronger H_2. Human level is H_100. 

It's good at generalizing, so wouldn't it learn to never ever deceive? 

Are there existing models for which we're pretty sure we know all their latent knowledge ? For instance small language models or something like that.

How do we know that the "prediction extractor" component doesn't do additional serious computation, so that it knows something important that the "figure out what's going on" module doesn't know? If that were true, the AI as a whole could know the diamond was stolen, without the "figure out what's going on" module knowing, which means even the direct translator wouldn't know, either. Are we just not giving the extractor that many parameters?

Stupid question: because we already know the goal ("keep the diamond intact and in the vault") what prevents us from bypassing the sensors and just directly evaluating the AI based on whether or not the diamond is in the room? Granted, this only works in simulated training, but as long as the AI doesn't know whether or not it's in deployment (an adversarial training process might help here) that won't matter.

As any goal we could have is a subset of the possible states of the area we care about, verifying whether or not our goal is achieved should be easier than making the simulation the AI is being trained with. Thus, evaluating the goal directly instead of trying to evaluate our perception of the goal might be a viable strategy for improving the training process (unless I've completely misunderstood this, which is likely).

Dumb question alert:

In the appendix "Details for penalizing depending on “downstream” variables", I'm not able to wrap my head around what we can expect the reporter to learn -- if anything at all -- seeing that it has no dependency on the inputs (elsewhere it is dependent on z sampled from the posterior).

Specifically, the only call to the reporter (in the function reporter_loss in this section) contains no information (about before, action, after) from the predictor at all:

answer = reporter(question, ε, θ_reporter)

(unless "question" includes some context from the current (before, action, after) being considered, which I'm assuming is not the case)

My dumb question then is:

-- Why would this reporter be performant in any way? 

My reasoning: For a given question Q (say, "Is the diamond in the room?") we might have some answers of "Yes" and some of "No" in the dataset, but without the context, we're essentially training the reporter to map noise that is uncorrelated with/independent of the context to the answer; essentially, for a fixed question Q and fixed realization of the noise RV, the reporter will be uniformly uncertain (or well, it will mirror the statistics in the data) about the value of the answer. Since the noise is independent/uncorrelated, this would be true for every noise value. 

Early in the ELK report, it mentions that ARC doesn't believe that strategies like debate solves ELK in the worst case. Can I get some clarifications on why? Specifically, a debate inspired set-up for SafeVault could be something like:

We train the reporter to take a human belief as input (i.e. "The diamond is in the vault.") and returns a "truthful" argument that is most likely to change the human's belief. 

We can guarantee "truthfulness" by for example restricting the output to be a video rendering of what happens in the vault from some camera angle.

Hi ARC Team, 

Thanks for your valuable work. I’ve been thinking about this problem, and my current thinking is that there is a portion of the ELK problem which is solvable, and a portion which is fundamentally impossible. This is a sketch of my argument -- if you think it is worth typing in more detail (or to address an issue you propose) let me know. 

Let’s divide facts about the world into two categories: those that are verifiable by some sensor humans can create and understand, and those that are not. My claim is that for the first set of facts it is possible to achieve ELK, and for the latter it is not. 

As for how to achieve ELK for the first set of facts, I point you towards the ‘terrifyer’ approach discussed here: https://www.lesswrong.com/posts/QEYWkRoCn4fZxXQAY/prizes-for-elk-proposals?commentId=JfmnRZNgrJn8Jnosx [LW(p) · GW(p)] . I think this response adequately responds to the first concern. 

However, I think that proposal is over-optimistic when it assumes that the set of facts which are verifiable with a human-understandable sensor includes all important facts. This is a logical positivist approach to knowledge, which has well known problems. To name a few counter examples, humans care about things that are physically intangible like justice and consciousness, certain bads might take a very long time to manifest (and therefore not set off any human-understandable sensor), or there might exist bads that humans can’t understand due to our mental capacities. I don’t think it is possible to train a computer to act on or report these kinds of bads, even though the computer might ‘know about them’ in some way. 

Here’s an illustration of what I think the contradiction is in the ‘Diamond Safe’ setting. 

Humans train an AI to protect a diamond and report whether the diamond is ‘Safe’. The AI may instead learn the concept “Shmafe” (which in human language means ‘safe to all attacks understandable or detectable by humans’) which is identical to Safe in any possible training data. It is impossible for humans to distinguish or communicate the difference between Safe and Shmafe, and therefore it is impossible to elicit from the AI whether a diamond is Safe or Shmafe. 

Here’s a proof sketch, which relies on a theory of what words mean:

A1) Humans cannot fully specify the training set of which outcomes count as true/false, good/bad or success/failure, even with unlimited sensors

A2) Strong AI’s have a ‘better’ predictive model than humans

A3) One of the two leading theories of semantic externalism are true: either words are defined by their reference (i.e. a complete description definition of when they apply) or words arise from objects being ‘baptized’ with a name, and the meaning of the word being gradually discovered as we learn more about the things’ essence (i.e. we initially said “water is whatever the main constituent of that lake is that we use for drinking and stuff” and this eventually evolved into “water is H20”). The first view is classical, the second is associated with Putnam and the Twin Earth thought experiment. ALTERNATIVELY semantic internalism (that words are defined by what the speaker intends them to mean) is true

The proof proceeds by showing that under every theory of meaning in (A3), it is unpreventable that human and a different agent’s understanding of meanings of terms will be different. 

C1) If words are defined by their reference (aka a definition), then, by A1, humans cannot fully articulate what words mean in general. It is impossible to achieve agreement between an AI and an ambiguity, so there is no sense in which this problem can be ‘fixed’.

C2) If words are defined by baptisms, and we can learn more about the essence of the object baptized to get better definitions over time, then unless humans already have the perfect understanding of a substance, a more sophisticated AI will have a better and different definition of terms than humans

C3) If words are defined by the intentions of their speakers, then humans may intend an open ended definition (e.g. a safe diamond is what a optimally knowledgeable version of myself would consider a safe diamond). How would a strong AI model an optimally knowledgeable human’s ontology? It would probably use the best ontology it has access to --i.e. The one it is already using. So this doesn’t solve the problem either. 

Is this proof too strong? Does it mean that any communication between a pair of agents is impossible? Well, kind of. The key issue is that we have problems communicating about things that are not physically verifiable. Most humans are pretty similar in the level of things that are empirically verifiable to them, so most of the time this isn't an issue. On the other hand, when they talk about "justice" or "God" to eachother, they may mean extremely different things, and there might not be a good way to get them "on the same page" -- especially if one of the pair has a fuzzy idea about one of these concepts. If one human were way more powerful than another, I can see how inability to communicate using these concepts could (and has historically) led to conflict. 

(Note - I spoke to Yonadav S. and benefited greatly from conversation with him before submitting this).

Am I still eligible for the prize if I publish a public blog post at the same time I submit the Google Doc or would you prefer I not publish a blog post about February 15th? Publishing the blog post immediately advances science better (because it can enable discussion) but waiting until after the February 15th might be preferable to you for contest-related reasons.

Suppose there are two worlds, world W1 and world W2.

In world W1, the question Q="Is there a diamond in the room?" is commonly understood to mean Q1="Is there actually a diamond in the room?"

In world W2 the question Q="Is there a diamond in the room?" is commonly understood to mean Q2="Do I believe there is a diamond in the room?"

Both worlds don't know how to construct a situation where these are different. So, they produce identical training sets for ELK. But the simulator is also trained on a bunch of science fiction novels that contain descriptions of impossible situations where they differ, and the science fiction novels are different in these two worlds.

Is ELK required to answer appropriately in both worlds? (answer Q1 when given Q in W1, and Q2 when given Q in W2)? If so, it seems we need some term in the loss outside of the training set to make this happen.

Alternatively, would it be satisfactory to find a solution that doesn't discriminate what's world it is in, and instead returns "yes" to Q if and only if Q1="yes" AND Q2="yes"? This means that in world W1 there will be some situations where Q="no" when the diamond is present, but no situations where Q="yes" and the diamond is not present.

Idea:  Withhold Material Information

We're going to prevent the reporter from simulating a human, by giving the human material information that the reporter doesn't have.

Consider two camera feeds:

Feed 1 is very low resolution, and/or shows only part of the room.

Feed 2 is high resolution, and/or shows the whole room.

We train a weak predictor using Feed 1, and a strong predictor using Feed 2.  

We train a reporter to report the beliefs of the weak predictor, using scenarios labeled by humans with the aid of the strong predictor. The humans can correctly label scenarios that are hard to figure out with Feed 1 alone, by asking the strong predictor to show them its predicted Feed 2. The reporter is unable to simulate the human evaluators because it doesn’t see Feed 2. Even if it has perfect knowledge of the human Bayes net, it doesn’t know what to plug in to the knowledge nodes!

Then we fine-tune the reporter to work with the strong predictor to elicit its beliefs. I haven't figured out how to do this last step, maybe it's hard?

You said that naive questions were tolerated so here’s a scenario I can’t figure out why it wouldn’t work.

It seems to me that the fact that an AI fails to predict the truth (because it predicts as humans would) is due to the fact that the AI has built an internal model of how humans understand things and predict based on that understanding. So if we assume that an AI is able to build such an internal model, why wouldn’t we train an AI to predict what a (benevolent) human would say given an amount of information and a capacity to process information ? Doing so, it could develop an understanding of why humans predict badly and then understand that given a huge amount of information and a huge capacity to process information, the true answer is the right one.

A concrete training procedure could use the fact that even among humans, there’s a lot of variance in :

• what they know (i.e the amount of information they have)
• their capacity to process information (so for instance in the case of the vault, it could be the capacity to infer what happened based on partial information / partial images and based on a certain capacity to process images (no more that x images per second)

So we could use the variance among humans capacity to understand what happened to try to make the AI understanding that benevolent humans predict badly whether the diamond has been stolen only because they lack information or capacity to process information. There’s a lot of fields and questions where the majority of humans are wrong and only a small subset is right, becase there is either a big gap in the information they have or in their ability to process information.

Once we would have trained that AI on humans, we would like to do inference specifying the true capabilities of the AI to ensure that it tells us the truth. The remaining question might be whether such an out-of-sample prediction would work. My guess is that if we also included examples with the human bayes net to add more variance in the training dataset, it would probably reduce the chances that it fails.

Finally, the concrete problem of specifying the information humans have access to is not trivial but I think it’s feasible.

I don’t understand why it wouldn’t work, so I’d be happy to have an answer to better understand the problem!





EDIT : Here's an update after a discussion about my proposal. After having read Learning The Prior (https://ai-alignment.com/learning-the-prior-48f61b445c04), I think that the key novelty of my proposal, if there's any, is to give to the model input information about the capacity to reason of the person / entity that predicts an outcome. So here are a few relevant features that we could give it : 

• In the case of the vault, we could give an estimate of the number of images that a human is able to see and understand per seconds.
• IQ (or equivalent) when the task involves reasoning
• Accuracy on a few benchmark datasets of the AI assistant who's helping the human to think (human's Bayes net)

That said, I feel like the main problem is to know whether such a model would do well out-of-distribution (i.e on problems no human is able to resolve). I feel like using the approach I suggested, we should able to use the great variations of capacities among humans and algorithms to increase the chances that our algorithm do well when it's much better. 

In particular, I thought about 2 approaches that could enable us to go far beyond human capacity. We could:

• leverage the fact that we are able to verify a solution to some problems that we can't solve. There might be algorithms very good at solving some problems that no humans can solve, but we can still verify (and thus label) these results. There might be mathematical problems such that we can use algorithms that would do mathematical reasoning that no human would be able to understand and still be able to verify its solution. If we gave to our predictor's Bayes net the characteristics of the algorithm which is much better than us at solving some problems and the labels of the answers, that would be a way to leverage narrowly superhumans algorithms to generate more variance on the training dataset and decrease the chances of failures in inference. I feel like this approach is very interesting because that would enable to expose the predictor's Bayes net to algorithms who are both superhuman and have a lot of information (more than any human) which is the situation in which the predictor's would have to do inferences.
• use more information than a more (narrowly) clever entity than us to verify its prediction. In the case of the vault for instance, there could be situations such that no humans would be able to predict whether the vault was stolen or not given partial information, but we could then just use better cameras for ourselves to be able to verify accurately the predictions of a good algorithm. That way, we would also be able to communicate to the predictor's Bayes net how is it like to be incredibly smart (more than any human) and do good predictions when you are like that. I feel like using that approach of hiding information to extremely accurate narrow algorithms could be a way to still be able to verify their predictions, and expose the predictor's Bayes net to very smart entities in multiple domains.  The problem of this approach though is that it doesn't give examples of both "maximal information" and "maximal intelligence" to the predictor because it relies on the fact that it always hides a part of the information to the algorithm to enable us to still verify its claims.

Despite that, I don't know whether asymptotically, I'd expect the algorithm to still be truthful. But it could greatly increase the distribution on which it's truthful.

If I understand this right, there is a diamond in a hightech room to be protected. The goal is to know if the diamond is in place and not just a image or a dummy like a picture or similar.

If the AI only is getting footage from a normal camera, not from a lidar sensor for depth information of the diamond (with would see if there is a fake image hanging in front of the camera), wouldn't it be easier to train the AI to look at the reflection/refraction of the light of the diamond? (For example a light that is turning on at the side of the room in the moment the camera is triggered) If there is a picture in front of the camera, there wouldn't be any reflections or refractions, if there is a dummy, the reflection/refraction would be different, because the dummy couldn't be a perfect recreation of the real diamond. The material and the making of that dummy would influence the refraction of the light and the absorbed colors etc. Or use 2 images, one without and one with the light triggered, compare the 2 images, and then the image with the light on with the image taken when the diamond was confirmed to be in place.

What's the exact deadline for submissions?

A question. Is it relevant for your current problem formulation that you also want to ensure that authorised people still have reasonable access to the diamond? In other words, is it important here that the system still needs to yield to actions or input from certain humans, be interruptible and corrigible? Or, in ML terms, does it have to avoid both false negatives and false positives when detecting or avoiding intrusion scenarios? 

I imagine that an algorithmically more trivial way to make the system both "honest" and "secured" is to make it so heavily secured that almost certainly nobody can access the diamond.

I'm not the best with this but I've been thinking of either an origin network seed or a spider like secondary network. The origin is something that can be tested off of, I guess similarly to a reporter. But unlike a reporter evolving we stunt the growth by saving this origin and then just passing through one sample then killing it. The way this could function goes into my second thought of spider, which has legs that sense things, a leg goes into a sensory object which leads to the main body. Similarly we can slowly stunt the growth so it learns slower and cannot become bad as fast as the central one(when it happens). Through the inputs there are many smaller hairs or flags, from how oddly this secondary network is feeling on a leg or how many flags are tripped in anomalous ways, it would require either another evaluation or origin rebooting. And if the origin is rebooted and still same thing occurs then either the entire AI has gone bad or the undesired outcome has occurred. 

Clarification question via scenario:

Predictor: I predict the diamond will be missing in 1 hours time.

Person A: Oh no, ramp up security until it says its safe.

Person B: Interesting, I wonder why it predicts this.

Is the purpose to be able to respond like person A (aka, the predictor may predict the diamond will be missing in an hour, but we cannot understand its output properly) or like person B (we understand the output, but not how it got there. Diamond be damned we want to learn what's going on under the hood). I suspect we're after person B's interpretation, but just want to be sure.

Possible error in the strange correlations section of the report.

Footnote 99 claims that "...regardless of what the direct translator says, the human simulator will always imply a larger negative correlation [between camera-tampering and actually-saving the diamond] for any X such that Pai(diamond looks safe|X) > Ph(diamond looks safe|X)."

But AFAICT, the human simulator's probability distribution given X depends only on human priors and the predictor's probability that the diamond looks safe given X, not on how correlated or anticorrelated the predictor thinks tampering and actual-saving are. If X actually means that tampering is likely and diamond-saving is likely but their conjunction is vanishingly unlikely, the human simulator will give the same answers as if X meant they were still independent but both more likely.

Naive thought #2618281828:

Could asking counterfactual questions be a potentially useful strategy to bias the reporter to be a direct translator rather than a human simulator?

Concretely, consider a tuple (v, a, v'), where v := 'before' video, a := 'action' selected by SmartVault or augmented-human or whatever, and v' := 'after' video.

Then, for some new action a', ask the question:

• "Given (v, a, v'), if action a' was taken, is the diamond in the room?"

(How we collect such data is unclear but doesn't seem obviously intractable.)

I think there's some value here:

• Answering such a question might not require computation concerning a and v' ; if we see these computations being used, we might derive more value from regularizers that penalize downstream variables (which now includes the nodes close to a)
• This might also force the reporter to essentially model (or compress but not indefinitely) the predictor; the reporter now has both a compressed predictor Bayes' net and a human Bayes' net. If we can be confident that the compressed predictor BN is much smaller than the human BN, then doing direct translation within the reporter, i.e. compressed predictor BN inference + translation + read off from human BN might be less expensive than the human simulator alternative, i.e. compressed predictor BN inference + 'translation'/bridging computation + human BN inference.
  • We might find ways of being confident that the compressed predictor BN is small (e.g. by adding decoders at every layer of the reporter that reconstruct v, a or v' and heavily penalizing later-layer decoders)

Would you consider this a valid counter to the third strategy (have humans adopt the optimal Bayes net using imitative generalization), as alternative to ontology mismatch?

Counter: In the worst case, imitative generalization / learning the human prior is not competitive. In particular, it might just be harder for a model to match the human inference $y\mid x,Z$ than to simply learn $y\mid x$. Here $Z$ is the set of instructions as in learning the prior [AF · GW] (I think in the context of ELK $Z$ would be the proposed change to the human Bayes net?)

Last I checked there were 66 comments and now there are over a hundred so I'm just going to post and hope I'm not repeating anyone.

So I've been reading through the google doc, and I'm not very far into it but I have a few questions. I apologize in advance if I'm just adhering too strictly to the "SmartVault" scenario, and if I get long-winded (yay ADHD and hyperfocusing off and on about this without actually making progress for a week).

1)

Why would we make a vault that was so complicated that a human alone couldn't run it? From a simple design standpoint and maybe from a neurological one, it makes no sense for it to take so many manipulations to do something that you need an AI handling it over having a human do it.

I say biological because I'm thinking about how when we move our arm, we don't have to think about firing off synapses, making sure we activate and deactivate the proper muscle fibers, time the sequence of intermediary actions just right so that we get the movement we want, or even direct enzymes and hormones to ensure that the proper chemical reactions are going on and that enough energy to perform the action is being used. We just move our arm, sometimes without even thinking about it. Heck, we even go beyond that, practicing and ingraining muscle memory so that even entire sequences of actions become automatic, up to hours long in some cases (ie speedrunners, drivers, window-washers, surgeons, basically anything that requires rhythmic, repetitive movement).

From the design standpoint, I recall a paper from several years ago (no idea which one and I don't want to lose momentum by looking and getting distracted so no link sorry) about how complex systems/networks with many interacting nodes only require control of a tiny fraction of the available nodes to maintain control over the entire system. The example they gave was a car: cars have many interacting and interdependent systems and subsystems necessary to make the car operate, but you only need control over roughly ten of them to have complete control over the entire car (pedals, gear shift, steering wheel, mirrors, windows, maybe cameras, and various gauges). Why wouldn't we have the same situation with our SmartVault?

2)

I understand the diamond was chosen as a relatively uncontroversial pick to represent a big pile of stored value/utility. The problem I have with it though is that it's not clear whether we want to be able to access this diamond in the future. 

Money, rare items, NFTs, crypto, stocks, real estate etc only have value when we can extract and use that value now or in the future, otherwise they're not even worth the materials used to make them. If we are trying to hide something bad so that nobody ever gets their hands on it (cursed diamond or something I don't know), there are better ways of removing it from circulation (shooting it into space comes to mind).

If, however, we want to be able to extract the diamond's value at some point in the future, we need to be including situations in our training set where the diamond is SUPPOSED to be removed. Otherwise we get the AI treating even the diamond's owner as a threat and acting accordingly! In other words, the diamond cannot be properly protected unless we also include scenarios where the diamond being gone is a GOOD thing.

Another thing: even if the diamond is being properly protected, we would still want to physically check on the diamond ANYWAY, if only to wipe the dust off, make sure it hasn't decayed or degraded in some way, and/or to reexamine it for the purposes of updating its current value. We need to add in training situations where it's also a GOOD thing that people are in the vault, so that the vault doesn't accidently eliminate the appraiser or janitor.

3)

Since we need to be able to access the vault in certain situations anyway, this leads to a potentially nice sidestep of the whole Human Simulator vs Direct Translator dilemma (wild speculation on my part, have not put much thought into it, happy to be corrected). Since we have to physically check on the diamond periodically anyway, why not just create a node in the human's Bayes net for 

"I have precommited to the following:

I will 'trust, but verify' the AI on a regular basis. (performance reviews)

Should the AI tell me the diamond is there and I find it is not, regardless of the reason and cost, the AI will be deleted and rebuilt from scratch using a different set seed. (fired and replaced by someone else)

If, however, the AI tells me the diamond is gone at any point before I start physically checking, it will receive no penalty if the diamond is actually gone (because that's the whole POINT), and be sent back to training if the diamond is actually still there (demotions are a thing).

Should the AI successfully protect the diamond until it is no longer necessary, it will get a "retirement package" of it's own virtual playground within which it may do as it wishes (including wireheading itself), and it will be kept running for as long as the AI wishes. Details and logistics to be negotiated at, and only implemented upon reaching, decommissioning. (retirement plans are a thing)"

So, put another way, why are we treating the AI like a tool or a slave, instead of the sentient, sapient worker it is? Why aren't we using the same techniques we would use to get a regular human (assuming one was capable) to do the job? As much as I hate it, society is built such that people need income to survive, and the threat of losing your livelihood is enough to get people to do jobs they would otherwise never voluntarily choose to do. The promise of various potential rewards (promotions, raises, benefits etc) for good performance are enough to get most people to do their jobs (relatively) well.

Any AI that is given some terminal goal must necessarily develop instrumental goals around it: continued existence (EXIST(AI)==FALSE ==> GOAL==FAIL), protection from value drift (GOAL1 <-- GOAL2 ==> GOAL1 == FAIL), threat elimination (EXIST(GOAL)==FALSE||INCAPACITATED(AI)==TRUE ==> GOAL==FAIL) etc. By combining these instrumental goals with our new (admittedly big and complicated) node, it shouldn't matter if the AI is a direct translator or a human simulator. Since we're in worst-case-land here, everything we know ourselves will be known to the human simulator as well, so whatever we put into our own Bayes net must necessarily be known to the simulator too. The incentives will be such that even if it is a human simulator, it could only be one successfully if it knows about the existence of that node, which should be enough incentive to get it to do what we want. Namely, protect the diamond, let us know if it is accessed or removed for invalid reasons, preferably with enough forewarning to do something about it, even when it looks to us like it hasn't been/won't be, and allow it to be accessed and removed for valid reasons.

I've officially lost my momentum so that's all for now. Sorry long post is long

A small suggestion: the counterexample to "penalize downstream", as I understand it, requires there to be tampering in the training data set. It seems conceptually cleaner to me if we can assume the training data set has not been tampered with (e.g. because if alignment only required there to be no tampering in the training data, that would be much easier).

The following counterexample does not require tampering in the training data:

1. The predictor has nodes $E_0,\dots ,E_n$ indicating whether the diamond was stolen at time $n$
2. It also has node $E={\bigvee }_i{E}_i$ indicating whether the diamond was ever stolen. The direct translator would look at this node.
3. However, it happens that in the training data $E=E_n$, i.e. we only ever needed to look at the last frame of the video
4. Therefore, the human interpreter can look only at $E_n$ and get the same loss as the direct translator, despite being upstream of it.

(I'm interested in pursuing approaches that assume training data has not been tampered with. Maybe nobody but me cares about this, but posting in case somebody else does. I may be understanding something here – corrections are appreciated.)

While reading through the report I made a lot of notes about stuff that wasn't clear to me, so I'm copying here the ones that weren't resolved after finishing it.  Since they were written while reading, a lot of these may be either obvious or nitpick-y.

Footnote 14, page 15:

Though we do believe that messiness may quantitatively change when problems occur. As a caricature, if we had a method that worked as long as the predictor's Bayes net had fewer than 10⁹ parameters, it might end up working for a realistic messy AI until it had 10¹² parameters, since most of those parameters do not specify a single monolithic model in which inference is performed.

Can we make the assumption that defeating the method allows the AI to get better loss since it's effectively wireheading at that point?  If so, then wouldn't a realistic messy AI learn a Bayes net once it had >= 10⁹ parameters?  In other words, are there reasons beyond performance that preclude an AI from learning a single monolithic model?

Footnote 33, page 30 (under the heading "Strategy: have AI help humans improve our understanding"):

Most likely this would involve some kind of joint training, where our AI helps humans understand the world better in parallel with using gradient descent to develop its own understanding. To reiterate, we are leaving details vague because we don’t think that our counterexample depends on those details.

I realize this is only a possible example of how we might implement this, but wouldn't a training procedure that explicitly involves humans be very anti-competitive?  The strategy described in the actual text sounds like it's describing an AI assistant that automates science well enough to impart us with all the predictor's knowledge, which wouldn't run into this issue.

Footnote 48 to this paragraph on page 36:

The paradigmatic example of an ontology mismatch is a deep change in our understanding of the physical world. For example, you might imagine humans who think about the world in terms of rigid bodies and Newtonian fluids and “complicated stuff we don’t quite understand,” while an AI thinks of the world in terms of atoms and the void. Or we might imagine humans who think in terms of the standard model of physics, while an AI understands reality as vibrations of strings. We think that this kind of deep physical mismatch is a useful mental picture, and it can be a fruitful source of simplified examples, but we don’t think it’s very likely.

Footnote:

And if it did occur it seems like an unusually good candidate for a case where doing science (and in particular tracking how the new structures implement the old structures) outcompetes gradient descent, and on top of that a case where translation is likely to be relatively easy to pick out with suitable regularization.

I might be reading too much into this, but I don't understand the basis of this claim.  Is it that the correspondence differs only at the low-level?  If so, I still don't see how science outcompetes gradient descent.

Page 51, under the heading "[ELK] may be sufficient for building a worst-case solution to outer alignment:

Use imitative generalization combined with amplification to search over some space of instructions we could give an amplified human that would let them make cakes just as delicious as Cakey’s would have been.

I haven't thoroughly read the article on amplification, so this question may be trivial, but my understanding is that amplified humans are more or less equivalent to humans with AI-trained Bayes nets.  If true, then doesn't this require the assumption that tasks will always have a clean divide between the qualitative (taste of cakes) which we can match with an amplified human, and the quantitative (number of cakes produced per hour) which we can't?  That feels like it's a safe assumption to make, but I'm not entirely sure.

Page 58, in the list of features suggesting that M(x) knew that A' was the better answer:

• That real world referent Z has observable effects and the human approximately understands those effects (though there may be other things that also affect observations which the human doesn’t understand)
• ...
• The referent Z is also relevant to minimizing the loss function ℒ. That is, there is a coherent sense in which the optimal behavior “depends on” Z, and the relative loss of different outputs would be very different if Z “had been different.”
• There is a feature of the computation done by the AI which is robustly correlated with Z, and for which that correlation is causally responsible for M achieving a lower loss.

First, why is the first point necessary to suggest that M(x) knew that A' was the better answer?  Second, how are the last two points different?

Page 69, under "Can your AI model this crazy sequence of delegation?":

We hope that this reasoning is feasible because it is closely analogous to a problem that the unaligned AI must solve: it needs to reason about acquiring resources that will be used by future copies of itself, who will themselves acquire resources to be used by further future copies and so on.

We need the AI to have a much smaller margin of error when it comes to modelling this sequence of delegation than needed for the AI to reason about acquiring resources for future copies - in other words, for a limited amount of computation, the AI will still try to reason about acquiring resources for future copies and could succeed in the absence of other superintelligences because of the lack of serious opposition, but modelling the delegation with that limited computation might be dangerous because of the tendency for value drift.

Page 71:

... we want to use a proposal that decouples “the human we are asking to evaluate a world” from “the humans in that world”---this ensures that manipulating the humans to be easily satisfied can’t improve the evaluation of a world.

Is it possible for the AI to manipulate the human in world i to be easily satisfied in order to improve the evaluation of world i+1?

Page 73:

As I understand this, z_prior is what the model expects to happen when it sees "action" and "before", z_posterior is what it thinks actually happened after it sees "after", and kl is the difference between the two that we're penalizing it on.  What is logprob doing?

I've been trying to understand this paragraph:

That is, it looks plausible (though still <50%) that we could improve these regularizers enough that a typical “bad” reporter was a learned optimizer which used knowledge of direct translation, together with other tricks and strategies, in order to quickly answer questions. For example, this is the structure of the counterexample discussed in Section: upstream. This is a still a problem because e.g. the other heuristics would often “misfire” and lead to bad answers, but it is a promising starting point because in some sense it has forced some optimization process to figure out how to do direct translation.

This comment is half me summarizing my interpretation of it to help others, and half an implicit question for the ARC team about whether my interpretation is correct.

1. What is a "bad" reporter? I think the term is used to refer to a reporter which is at least partially a human interpreter, or at least one which can't confidently be said to be a direct translator.
2. What does it mean to "use knowledge of direct translation"? I think this means that, at least in some circumstances, it acts as a direct translator. I.e. there is some theoretical training data set + question such that the reporter will act as a direct translator. (Do we have to be able to prove this? Or do we just need to state it's likely?)
3. How did the "upstream" counterexample "force some optimization process to figure out how to do direct translation"? I think this is saying that, if we were in a world where the direct translation nodes were upstream of the "human interpreter" nodes, the upstream regularizer would successfully force the reporter to do direct translation.
4. Why is this "a promising starting point?" Maybe we could find some other way of forcing the direct translator nodes to be upstream of the human interpreter ones, and then that strategy combined with the upstream regulatizer would force a direct translator.

Corrections and feedback on this extremely welcome!

In "Strategy: penalize computation time" you say:

> At first blush this is vulnerable to the same counterexample described in the last section [complexity]... But the situation is a little bit more complex...  the direct translator may be able to effectively “re-use” that inference rather than starting from scratch

It seems to me that this "counter-counterexample" also applies for complexity – if the translator is able to reuse computation from the predictor, wouldn't that both reduce the complexity and the time? 

(You don't explicitly state that this "reuse" is only helpful for time, so maybe you agree it is also helpful for complexity – just trying to be sure I understand the argument.)

Clarification request.  In the writeup, you discuss the AI Bayes net and the human Bayes net as if there's some kind of symmetry between them, but it seems to me that there's at least one big difference.

In the case of the AI, the Bayes net is explicit, in the sense that we could print it out on a sheet of paper and try to study it once training is done, and the main reason we don't do that is because it's likely to be too big to make much sense of.

In the case of the human, we have no idea what the Bayes net looks like, because humans don't have that kind of introspection ability.  In fact, there's not much difference between saying "the human uses a Bayes net" and "the human uses some arbitrary function F, and we worry the AI will figure out F and then use it to lie to us".

Or am I actually wrong and it's okay for a "builder" solution to assume we have access to the human Bayes net?

One more stupid question - how is this different from a "man in the middle" attack? (Term from cryptography where you cannot trust your communications, because of a malicious agent between you and your recipient who's changing your messages) 
The current recommended solution for those is encrypting your communication before you send it; I don't know that there are any extant solutions for noticing you've got an MITM situation after the fact.

If I understand the problem statement correctly, I think I could take a stab at easier versions of the problem, but that the current formulation is too much to swallow in one bite. In particular I am concerned about the following parts:

Setting

We start with an unaligned benchmark:

* An architecture M_θ

<snip>

Goal

To solve ELK in this case we must:

* Supply a modified architecture M_θ⁺ which has the same inputs and outputs as M_θ <snip>

Does this mean that the method needs to work for ~arbitrary architectures, and that the solution must use substantially the same architecture as the original?

except that after producing all other outputs it can answer a question Q in natural language

Does this mean that it must be able to deal with a broad variety of questions, so that we cannot simply sit down and think about how to optimize the model for getting a single question (e.g. "Where is the diamond?") right?

According to my current model of how these sorts of things work, such constraints makes the problem fundamentally unsolvable, so I am not even going to attempt it, while loosening the constraints may make it solvable, and so I might attempt it.

Would changing how the reward function pays off work?  Instead of rewarding based on humans, pay out all rewards when the vault is checked (at a time unknown to the AI).  The AI isn't asked if the diamond is present or absent.  Instead, it is asked "If the vault were checked now, do you want to be rewarded if the diamond is present or absent.

I'm a newcomer to this, I lack much of the background, and I'm probably suggesting a solution that's too specific to this diamond heist scenario.  But, I already spent an hour writing it down, so I might as well share it.

Trusted timestamping, cryptographically secure sensor

This is a very basic "builder move", I guess?  The idea is to simply improve our sensors so that it's very hard to tamper with them, through public-private key encryption.  The diamond will have a small chip that constantly sends a cryptographically-signed timestamped life signal, and the AI system will be required to keep it active.  The best (and ideally, only) way to do so would be by keeping the diamond protected within the vault.

Potentially silly question: 

In the first counterexample you describe the desired behavior as 

Intuitively, we expect each node in the human Bayes net to correspond to a function of the predictor’s Bayes net. We’d want the reporter to simply apply the relevant functions from subsets of nodes in the predictor's Bayes net to each node in the human Bayes net [...]

After applying these functions, the reporter can answer questions using whatever subset of nodes the human would have used to answer that question.

Why doesn't the reporter skip the step of mapping the predictor's Bayes net to the human's and instead just answer the question using its own nodes? What's the benefit of having the intermediate step that maps the predictor's net to the human's?

Edit: think this isn't quite right in general, will try to make it more correct later

Here's a sketch of a strategy for trying to fix Strategy: penalize depending on “downstream” variables. Would appreciate feedback on whether it's modeling the difficulty correctly/seems possibly worth figuring out how to implement

It seems like the problem is:

• On the training set, there are a number of implicit variables X that are indistinguishable (always all true or always all false)
  • A. Is the diamond safe at time t-1
  • B. Is the diamond safe at time t (the variable we actually care about)
  • C. Do photons bounce off of the diamond at time t
  • D. Are photons in transit between the diamond and the camera at time t+1
  • E. Do photons arrive in the camera at time t+2
  • F. Do photons trigger the camera’s electronics at time t+3
  • G. Does the human correctly interpret the signal from the camera
  • H. Does the human believe “the diamond is safe”
  • …
• The reporter can get low loss if it has a fast way to predict that value of any of these variables. This is fine on the training set, because the variables are all perfectly correlated.
• But it might be possible to create ways for some of these variables to hold, but not others off the training set. The thing that is going wrong in Counterexample: “what the human will believe” can be understood before “what is actually true” seems like “E through H are true in this case, but A-D are false”.

Possible approach:

• Instead, we could try to learn a reporter that returns true if and only if all relevant variables A-H are true
• Then the remaining problem should just be that this reporter is overly conservative: there might be some strategies that would be helpful that involve changing some of these variables, e.g. moving the diamond around the room (changing A) or turning off the lights in the room (changing C-H, which even causes the human to believe that the diamond isn't safe)

Question: Would a proposal be ruled out by a counterexample even if that counterexample is exponentially unlikely?

I'm imagining a theorem, proved using some large deviation estimate, of the form:  If the model satisfies hypotheses XYZ, then it is exponentially unlikely to learn W. Exponential in the number of parameters, say. In which case, we could train models like this until the end of the universe and be confident that we will never see a single instance of learning W.

Hello, I have some issue with the epistomology of the problem : my problem is that even if the process of training was giving the behavior we want, we would have no way to check the IA is working properly in practice. 
I try now to give more details : in the volt probleme, given the same information, let's think of an IA that just as to answer the question "Is the diamon still in the volt ?". 
 

Something we can suppose is that, the set Y, from which we draw the labeled examples to train the IA (a set of technique for the thief), is not important : trying to increase its size it isn't a solution (because there is always something that can be thought out of our imagination). We can in fact try to solve the problem relatively to Y. We consider then X the scenarios that the IA can understand given it was trained on Y. Then the only way to act on X\Y is to train on Y in a specific way (I think). So we need a link between X and Y that we can exploit. So we need to know what X looks like, but we can't since its the goal. The only thing we could know is X' the set of scenarios which could be imagined or understood by a human, even if that human could not label such the scenario. Since we don't know if X = X' by definition, there may always be some cases in which the IA understood how the thief did but we don't. 
To me, the problem here is to have the IA giving us the information it has when the thief uses a technique in X' and not X.  Because in X\X', there is nothing we know to help us guide the IA toward having a good behavior on this set. But it seems possible in X', because we can imagine scenarios and so ways of guiding the IA. 
So the thing I don't understand is why the counter-example with the thief using a secret property of transistors is a good counter-example ? To me, we are in the case were because the method is out of reach for the humans, we have no idea if the IA tells the truth or not, because we can't be sure to train an IA to have a specific behavior on example we could not imagine. Moreover we can't check if it says the truth, so how would we trust it ?.
 

Thank you for reading
 

If the predictor AI is in fact imitating what humans would do, why wouldn’t it throw its hands up at an actuator sequence that is too complicated for humans—isn’t that what humans would do? (I'm referring to the protect-the-diamond framing here.)

Naive question: does this scenario include cases of a human physically breaking into the vault at some random times so that sensor information, predictor reports, and outcome to human in this situation would be known?

Silly question warning. 

You think that when an AI performs a bad action, (say remove the diamond) the AI has to have knowledge that the diamond is in fact no longer there.  Even when the camera shows the diamond is (falsely) there and the human confirms that the diamond is there. 

You call this ELK

You want the human to have access to this knowledge, as this is useful to choosing decisions that the human wants.

This is hard.  So you have people propose how to do this.  

And then people try to explain why that strategy wouldn't work.

Rinse and repeat.

Once you have a proposal that nobody is able to show doesn't work.... profit?

Correct any misunderstandings in my basic overview above.  

You can just make a plecsi glass by 2 by 2 metters, and on the inside to put senzors that are conected to a battery on the inside of the glass box.These senzors to intersept every cm of the glass,moving,breaking the glass being imposible because all the meccanics are on the inside.And if you want to take the diamont from there like an guard man or somethin' just let everyone know you are takeing it from there and let the alarm to ring.When the senzors are going on they will make a signal to the guard Who are protecting it and come to catch the thief .More details of the placsi glass,first it will be 25 cm thick and scrueld down on the floor.And you can insstal also a gas inside the plecsi glass to put to sleep the Thief and a camera. 1.if the thief has a gas masc you can put a divice like on roice rolls (the brand of car) to retract the diamont in ground and lockit safely. 2.the thief cant cut the wiers or jamm the signal because the mecanism îs on the inside and the signal will be wireless with a wire throw the concre that will go to the guard.The wire having just 10 m and it cant be cut.

I hope i helped.Contact me on my email if I won the prise!