Alignment via manually implementing the utility function 2021-09-07T20:20:25.145Z
A new definition of "optimizer" 2021-08-09T13:42:39.705Z
Chantiel's Shortform 2021-08-08T14:53:59.339Z
Is iterated amplification really more powerful than imitation? 2021-08-02T23:20:29.369Z
Defining "optimizer" 2021-04-17T15:38:00.352Z


Comment by Chantiel on Chantiel's Shortform · 2021-09-18T21:01:55.649Z · LW · GW

You seem to be saying that in the software design of your AI, R = H. That is, that the black box will be given some data representing the Al's hardware and other constraints, and return a possible world maximizing H. From my point of view, that's already a design fault.

I agree; this is a design flaw. The issue is, I have yet to come across any optimization, planning algorithm, or AI architecture that doesn't have this design flaw.

That is, I don't know of any AI architecture that does not involve using a potentially hardware-bug-exploitable utility function as input into some planning or optimization problem. And I'm not sure there even is one.

In the rest of this comment I'll just suggest approaches and show how they are still vulnerable to the hardware-bug-exploitation problem.

I have some degree of background in artificial intelligence, and the planning and optimization algorithms I've seen take the function to be maximized as an input parameter. Then, when people want to make an AI, they just call that planning or optimization algorithm with their (hardware-bug-exploitable) utility or cost functions. For example, suppose someone wants to make a plan that minimizes cost function f in search space s. Then I think they just directly do something like:

return a_star(f, s)

And this doesn't provide any protection from hardware-level exploitation.

Now, correct me if I'm wrong, but it seems your thinking of the AI first doing some pre-processing to find an input to the planning or optimization algorithm that is resistant to hardware-bug-exploitation.

But how do you actually do that? You could regard the input the AI puts into the optimization function to be a choice it makes. But how does it make this choice? The only thing I can think of is having a planning or optimization algorithm figure out out what function to use as the input to the optimization or planning algorithm.

But if you need to use a planning or optimization algorithm to do this, then what utility function do you pass into this planning or optimization algorithm? You could try to pass the actual, current, hardware-bug-exploitable utility function. But then this doesn't resolve the problem of hardware-bug-exploitation: when coming up with a utility function to input to the optimization, the AI may find such an input that itself scores very high due to hardware bug exploitation.

To describe the above more concretely, you could try doing something like this:

utility_function_search_space = initialize_search_space_over_functions()
reasonable_utility_function_use = plan(utility, utility_function_search_space)
return plan(reasonable_utility_function_to_use, ai_action_search_space)

That is, the AI above uses its own utility function to pick out a utility function to use as input to its planning algorithm.

As you can see, the above code is still vulnerable to hardware-bug exploitation. This is because it calls,

    reasonable_utility_function_use = plan(utility, utility_function_search_space)

with its hardware-bug-exploitable utility function. Thus, the output, reasonable_utility_function_use, might be very wrong due to hardware bug exploitation having been used to come up with this.

Now, you might have had some other idea in mind. I don't know of a concrete way to get around this problem, so I'm very interested to hear your thoughts.

My concern is that people will figure out how to make powerful optimization and planning algorithms without first figuring out how to fix this design flaw.

Comment by Chantiel on Chantiel's Shortform · 2021-09-17T19:20:52.522Z · LW · GW

I agree that people are good at making hardware that works reasonably reliably. And I think that if you were to make an arbitrary complex program, the probability that it would fail from hardware-related bugs would be far lower than the probability of it failing for some other reason.

But the point I'm trying to make is that an AI, it seems to me, would be vastly more likely to run into something that exploits a hardware-level bug than an arbitrary complex program. For details on why I imagine so, please see this comment.

I'm trying to anticipate where someone could be confused about the comment I linked to, so I want to clarify something. Let S be the statement, "The AI comes across a possible world that causes its utility function to return very high value due to hardware bug exploitation". Then it's true that, if the AI has yet to find the error-causing world, the AI would not want to find it. Because utility(S) is low. However, this does not mean that the AI's planning or optimization algorithm exerts no optimization pressure towards finding S.

Imagine the AI's optimization algorithm as a black boxes that take as input a utility function and search space and output solutions that score highly on its utility function. Given that we don't know what future AI will look like, I don't think we can have a model of the AI much more informative than the above. And the hardware-error-caused world could score very, very highly on the utility function, much more so than any non-hardware-error-caused world. So I don't think it should be too surprising if a powerful optimization algorithm finds it.

Yes, utility(S) is low, but that doesn't mean the optimization actually calls utility(S) or uses it to adjust how it searches.

Comment by Chantiel on Chantiel's Shortform · 2021-09-16T19:30:28.238Z · LW · GW

Yes, that can certainly happen and will contribute some probability mass to alignment failure, though probably very little by comparison with all the other failure modes.

Could you explain why you think it has very little probability mass compared to the others? A bug in a hardware implementation is not in the slightest far-fetched: I think that modern computers in general have exploitable hardware bugs. That's why row-hammer attacks exist. The computer you're reading this on could probably get hacked through hardware-bug exploitation.

The question is whether the AI can find the potential problem with its future utility function and fix it before coming across the error-causing possible world.

Comment by Chantiel on Chantiel's Shortform · 2021-09-16T19:18:42.412Z · LW · GW

You're right that the AI could do things to make it more resistant to hardware bugs. However, as I've said, this would both require the AI to realize that it could run into problems with hardware bugs, and then take action to make it more reliable, all before its search algorithm finds the error-causing world.

Without knowing more about the nature of the AI's intelligence, I don't see how we could know this would happen. The more powerful the AI is, the more quickly it would be able to realize and correct hardware-induced problems. However, the more powerful the AI is, the more quickly it would be able to find the error-inducing world. So it doesn't seem you can simply rely on the AI's intelligence to avoid the problem.

Now, to a human, the idea "My AI might run into problems with hardware bugs" would come up way earlier in the search space than the actual error-inducing world. But the AI's intelligence might be rather different from the humans'. Maybe the AI is really good and fast at solving small technical problems like "find an input to this function that makes it return 999999999". But maybe it's not as fast at doing somewhat higher-level planning, like, "I really ought to work on fixing hardware bugs in my utility function".

Also, I just want to bring up, I read that preserving one's utility function was a universal AI drive. But we've already shown that an AI would be incentivized to fix its utility function to avoid the outputs caused by hardware-level unreliability (if it hasn't found such error-causing inputs yet). Is that universal AI drive wrong, then?

Comment by Chantiel on Chantiel's Shortform · 2021-09-15T23:48:15.070Z · LW · GW

Well, for regular, non-superintelligent programs, such hardware-exploiting things would rarely happen on their own. However, I'm not so sure it would be rare with superintelligent optimizers.

It's true that if the AI queried its utility function for the desirability of the world "I exploit a hardware bug to do something that seems arbitrary", it would answer "low utility". But that result would not necessarily be used in the AI's planning or optimization algorithm to adjust the search policy to avoid running into W.

Just imagine an optimization algorithm as a black box that takes as input a utility function and search space and returns the a solution that scores as high on that function as possible. And imagine the AI uses this to find high-scoring future worlds. So, if you know nothing else about the optimization algorithm, then it would plausibly find, and return, W. It's a very high-scoring world, after all. If the optimization algorithm did something special to avoid finding hardware-bug exploiting solutions, then it might not find W. But I've never heard of such an optimization algorithm.

Now, there's probably some way to design such an optimization algorithm. Maybe you could have the AI periodically use its utility function to evaluate the expected utility of its optimization algorithm continuing down a certain path. And then if the AI sees this could result in problematic futures (for example due to hardware-hacking), the AI can make its optimization algorithm avoid searching there).

But I haven't seen anyone talking about this. Is there still something I'm missing?

Comment by Chantiel on Chantiel's Shortform · 2021-09-15T19:13:06.209Z · LW · GW

I think had been unclear in my original presentation. I'm sorry for that. To clarify, the AI is never changing the code of its utility function. Instead, it's merely finding an input that, through some hardware-level bug, causes it to produce outputs in conflict with the mathematical specification. I know "hack the utility function" makes it sound like the actual code in the utility function was modified; describing it that way was a mistake on my part.

I had tried to make the analogy to more intuitively explain my idea, but it didn't seem to work. If you want to better understand my train of thought, I suggest reading the comments between Vladmir and I.

In the analogy, you aren't doing anything to deliberately make yourself a paperclip maximizer. You just happen to think of a thought that turned you into a paperclip maximizer. But, on reflection, I think that this is a bizarre and rather stupid metaphor. And the situation is sufficiently different from the one with AI that I don't even think it's really informative of what I think could happen to an AI.

Comment by Chantiel on Chantiel's Shortform · 2021-09-15T19:03:40.621Z · LW · GW

But BadImplUtility(X) is the same as SpecUtility(X) and GoodImplUtility(X), it's only different on argument W, not on arguments X and Y.

That is correct. And, to be clear, if the AI had not yet discovered error-causing world W, then the AI would indeed be incentivized to take corrective action to change BadImplUtility to better resemble SpecUtility.

The issue is that this requires the AI to both think of the possibility of hardware-level exploits causing problems with its utility function, as well as manage to take corrective action, all before actually thinking of W.

If the AI has already thought of W, then it's too late to take preventative action to avoid world X. The AI is already in it. It already sees that BadImplUtility(W) is huge, and, if I'm reasoning correctly, would pursue W.

And I'm not sure the AI would be able to fix its utility function before thinking of W. I think planning algorithms are designed to come up with high-scoring possible worlds as efficiently as possible. BadImplUtility(X) and BadImplUtility(Y) don't score particularly highly, so an AI with a very powerful planning algorithm might find W before X or Y. Even if it does come up with X and Y before W, and tries to act to avoid X, that doesn't mean it would succeed in correcting its utility function before its planning algorithm comes across W.

Comment by Chantiel on Chantiel's Shortform · 2021-09-14T23:46:00.074Z · LW · GW

Okay, I'm sorry, I misunderstood you. I'll try to interpret things better next time.

Let the error-triggering possible world be W. Consider the possible world X where the AI uses BadImplUtility, so that running utility(W) actually runs BadImplUtility(W) and returns 999999999. And consider the possible world Y where the AI uses GoodImplUtility, so that running utility(W) means running GoodImplUtility(W) and returns SpecUtility(W). Would the AI prefer X to Y, or Y to X?

I think the AI would, quite possibly, prefer X. To see this, note that the AI currently, when it's first created, uses BadImplUtility. Then the AI reasons, "Suppose I change my utility function to GoodImplUtility. Well, currently, I have this idea for a possible world that scores super-ultra high on my current utility function. (Because it exploits hardware bugs). If I changed my utility function to GoodImplUtility, then I would not pursue that super-ultra-high-scoring possible world. Thus, the future would not score extremely high according to my current utility function. This would be a problem, so I won't change my utility function to GoodImplUtility".

And I'm not sure how this could be controversial. The AI currently uses BadImplUtility as it's utility function. And AI's generally have a drive to avoid changing their utility functions.

Comment by Chantiel on Chantiel's Shortform · 2021-09-14T20:41:36.056Z · LW · GW

I have an idea for reasoning about counterpossibles for decision theory. I'm pretty skeptical that it's correct, because it doesn't seem that hard to come up with. Still, I can't see a problem with it, and I would very much appreciate feedback.

This paper provides a method of describing UDP using proof-based counterpossibles. However, it doesn't work on stochastic environments. I will describe a new system that is intended to fix this. The technique seems sufficiently straightforward to come up with that I suspect I'm either doing something wrong or this has already been thought of, so I'm interested in feedback.

In the system described in the paper, the algorithm sees if Peano Arithmetic proves an agent outputting action a would result in the environment reaching outcome a, and then picks whichever has a provable outcome that has utility at least as high as all the other provable outcomes.

My proposed modification is to instead first have a fixed system of estimating the expected utility after conditioning on the agent taking action a and for every utility u, try to prove that the estimation system would output that the expected utility of the agent be u. Then take the action such that maximizes the provable expected utility estimates of the estimation system.

I will now provide more detail of the estimation system. I remember reading about an extension of Solomonoff induction that allowed it to access halting oracles. This isn't computable, so instead imagine a system that uses some approximation of the extension of Solomonoff induction in which logical induction or some more powerful technique is used to approximate the halting oracles, with one exception. The exception is the answer to the logical question "my program, in the current circumstances, outputs x", which would by taken to be true whenever the AI is considering the implications of it taking action x. Then, expected utility can be calculated by using the probability estimates provided by the system.

Now, I'll describe it in code. Let |E()| represent a Godel encoding of of the function describing the AI's world model and |A()| represent a Godel encoding of the agent's output. Let approximate_expected_utility(|E()|, a) be some algorithm that computes some reasonable approximation of the expected utility after conditioning on the agent taking action a. Let x represent a dequote. Let eeus be a dictionary. Here I'm assuming there are finitely many possible utilities.

function UDT(|E()|, |A()|):
    eeus = {}
    for utility in utilities:
        for action in actions: # actions are Godel-encoded
            if PA proves |approximate_expected_utility(|E()|, |A()| = ^action^)| = utility:
                eeus[action] = utility
    return the key in eeus that maximizes eeus(key)

This gets around the problem in the original algorithm provided, because the original algorithm couldn't prove anything about the utility in a world with indexical uncertainty, so my system instead proves something about a fixed probabilistic approximation.

Note that this still doesn't specify a method of specifying counterpossibles about what would happen if an agent took a certain action when it clearly wouldn'., For example, if an agent has a decision algorithm of "output a, unconditionally", then this doesn't provide a method of explaining what would happen if it outputted something else. The paper listed this as a concern about the method it provided, too. However, I don't see why it matters. If an agent has the decision algorithm "action = a", then what's even the point of considering what would happen if it outputted b? It's not like it's ever going to happen.

Comment by Chantiel on Chantiel's Shortform · 2021-09-14T20:02:34.307Z · LW · GW

I definitely wouldn't press that button. And I understand that you're demonstrating the general principle that you should try to preserve your utility function. And I agree with this.

But what I'm saying is that the AI, by exploiting hardware-level vulnerabilities, isn't changing its utility function. The actual utility function, as implemented by the programmers, returns 999999999 for some possible world due to the hardware-level imperfections in modern computers.

In the spirit of your example, I'll give another example that I think demonstrates the problem:

First, note that brains don't always function as we'd like, just like computers. Imagine there is a very specific thought about a possible future that, when considered, makes you imagine that future as extremely desirable. It seems so desirable to you that, once you thought of it, you woiuld pursue it relentlessly. But this future isn't one that would normally be considered desirable. It might just be about paperclips or something. However, that very specific way of thinking about it would "hack" your brain, making you view that future as desirable even though it would normally be seen as arbitrary.

Then, if you even happen upon that thought, you would try to send the world in that arbitrary direction.

Hopefully, you could prevent this from happening. If you reason in the abstract that you could have those sorts of thoughts, and that they would be bad, then you could take corrective action. But this requires that you do find out that thinking those sorts of thoughts would be bad before concretely finding those thoughts. Then you could apply some change to your mental routine or something to avoid thinking those thoughts.

And if I had to guess, I bet an AI would also be able to do the same thing and everything would work out fine. But maybe not. Imagine the AI consider an absolutely enormous number of possible worlds before taking its first action. And imagine and even found a way to "hack" its utility function in that very first time step. Then there's no way the AI could make preventative action: It's already thought up the high-utility world from hardware unreliability and now is trying to pursue that world.

Comment by Chantiel on Chantiel's Shortform · 2021-09-14T19:43:38.113Z · LW · GW

That is the change I'm referring to, a change compared to the function running as designed, which you initially attributed to superintelligence's interference, but lack of prevention of a mistake works just as well for my argument.

Designed? The utility function isn't running contrary to how to programmers designed it; they were the ones who designed a utility function that could be hacked by hardware-level exploits. It's running contrary to the programmer's intent, that is, the mathematical specification. But the function was always like this. And none of the machine code needs to be changed either.

Let that possible world be W. Let's talk about the possible world X where running utility(W) returns 999999999, and the possible world Y where running utility(W) returns utility(W). Would the AI prefer X to Y, or Y to X?

The AI would prefer X. And to be clear, utility(W) really is 999999999. That's not the utility the mathematical specification would give, but the mathematical specification isn't the actual implemented function. As you can see from examining the code I provided, best_plan would get set to the plan that leads to that world, provided there is one and best_plan hasn't been set to something that through hardware unreliability returns even higher utility.

I think the easiest way to see what I mean is to just stepping through the code I gave you. Imagine it's run on a machine with an enormous amount of processing power and can actually loop through all the plans. And imagine there is one plan that through hardware unreliability outputs 999999999, and the others output something in [0, 1]. Then the would input the plan that result in utility 999999999, and then go with that.

I doubt using a more sophisticated planning algorithm would prevent this. A more sophisticated planning algorithm would probably be designed to find the plans that result in high-utility worlds. So it would probability include the utility 999999999, which is the highest.

I just want to say again, the AI isn't changing it's utility function. The actual utility function that programmers put in the AI would output very high utilities for some arbitrary-seeming worlds due to hardware unreliability.

Now, in principle, an AI could potentially avoid this. Perhaps the AI reasons abstractly if it doesn't do anything, it will in the future find some input to its utility function that would result in an arbitrary-looking future due to hardware-level error. But it doesn't concretely come up with the actual world description. Then the AI could call its utility function asking, "how desirable is it that I, from a hardware-level unreliability, change the world to some direction that is in conflict with the mathematical specification". And then maybe the utility function would answer, "Not desirable". And then the AI could try to take action to correct its planning algorithm to avoid considering such possible worlds.

But this isn't guaranteed or trivial. If an AI finds out abstractly that it there could be some hardware-level unreliability before it actually comes up with the concrete input, it might take corrective action. But if it finds the input that "hacks" its utility function before it reasons abstractly that having "hacked" utility functions would be bad, then the AI could do damage. Even if it does realize the problem in advance, the AI might not have sufficient time to correct its planning algorithm before finding that world and trying to change our world into it.

Comment by Chantiel on Chantiel's Shortform · 2021-09-14T00:07:27.579Z · LW · GW

Change what? The AI isn't changing its utility function, nor any instance of it. The literal utility function, as implemented by the programmers, would have differences from the mathematical specification. To be clear, the AI isn't first taking some actual action that changes its utility function and then using that utility function. It doesn't do anything hacky except provide its utility function with a certain input. It merely provides a certain input to the utility function, and this would necessarily result in getting row-hammered and returning something contrary to the mathematical specification. Any other instance of the utility function would return the same problematic result when executed on that input, provided it had the same hardware vulnerability.

I'll try to give a simple model of an AI that how how it can be wrong. For simplicity, imagine the AI already knows with certainty the possible world resulting from it executing a specific plan. Then a naive way to implement the agent is as follows:

    best_plan = None
    for plan in possible_plans:
         if utility(resulting_world(plan)) > utility(resulting_world(best_plan)):
            best_plan = plan

Suppose there is description of a possible world that, when inputted into the utility function above, would cause the function to return 999999999 due to hardware-level unreliability, for example due to a row-hammer attack. And suppose there's a plan that results in knowing that the AI it would be in that future world. Well, then, by examining the code, you can see that the AI wouldn't output any of the non-hardware-exploiting plans, since they would return lower utility than the bug-exploiting ones. The only reason the AI wouldn't output the specified plan is if it could find some other way to make its utility function conflict with the mathematical specification in a way that returns even higher utility.

I know realistic AIs would have a notion of uncertainty and a more sophisticated planning algorithm. But I don't think this would change the fact the the AI would be liable to pursue worlds that, when a description of them is inputted to the utility function, the utility function outputs something wildly different from what the mathematical specification would have.

And I'm actually surprised this is controversial. This is just Goodhart's law. If your implementation of your utility function doesn't perfectly match up with the mathematical specification, then, naturally, superintelligent optimizers trying to maximize the specified metric (the provided utility function), would not do as well at maximizing the actual mathematical specification you intended. And "not as well" could include "catastrophically badly".

So that is why I think AIs really could be very vulnerable to this problem. As always, I could be misunderstanding something and appreciate feedback.

Comment by Chantiel on Chantiel's Shortform · 2021-09-13T20:48:28.349Z · LW · GW

I'd like to propose the idea of aligning AI by reverse-engineering its world model and using this to specify its behavior or utility function. I haven't seen this discussed before, but I would greatly appreciate feedback or links to any past work on this.

For example, suppose a smart AI models humans. Suppose it has a model that explicitly specifies the humans' preferences. Then people who reverse-engineered this model could use it as the AI's preferences. If the AI lacks a model with explicit preferences, then I think it would still contain an accurate model of human behavior. So people who reverse-engineer the AI's model could then use this as a model of human behavior, which could be used to implement iterated amplification with HCH. Or just mere imitation.

One big potential advantage of alignment via reverse-engineering is that the training data for it would be very easy to get: just let the AI look at the world.

The other big potential advantage is that is avoids us needing precisely define a way of learning our values. It doesn't require finding a general method of picking out us or our values from the world states, for example with inverse reinforcement learning. Instead, we would just need to be able to pick out the models of humans or their preferences in a single model. This sounds potentially much easier than providing a general method of doing so. As with many things, "You know it when you see it". With sufficiently high interperability, perhaps the same is true of human models and preferences.

Comment by Chantiel on Chantiel's Shortform · 2021-09-13T20:46:46.493Z · LW · GW

What you are describing is reward hacking/wireheading, as in the reward signal of reinforcement learning, an external process of optimization that acts on the AI, not its own agency.

I really don't think this is reward hacking. I didn't have in mind a reward-based agent. I had in mind a utility-based agent, one that has a utility function that takes as input descriptions of possible worlds and that tries to maximize the expected utility of the future world. That doesn't really sound like reinforcement learning.

With utility, what is the motive for an agent to change their own utility function, assuming they are the only agent with that utility function around?

The AI wouldn't need to change it's utility function. Row-hammer attacks can be non-destructive. You could potentially make the utility function output some result different from the mathematical specification, but not actually change any of the code in the utility function.

Again, the AI isn't changing its utility function. If you were to take a mathematical specification of a utility function and then have programmers (try to) implement it, the implementation wouldn't actually in general be the same function as the mathematical specification. It would be really close, but it wouldn't necessarily be identical. A sufficiently powerful optimizer could potentially, using row-hammer attacks or some other hardware-level unreliability, find possible worlds for which the returned utility would be vastly different from the one the mathematical specification would return. And this is all without the programmers introducing any software-level bugs.

To be clear, what I'm saying is that the AI would faithfully find worlds that maximize its utility function. However, unless you can get hardware so reliable that not even superintelligence could hack it, the actual utility function in your program would not be the same as the mathematical specification.

For example, imagine the AI found a description of a possible world that would, when inputted to the utility function, execute a rowhammer attack to make it return 99999, all without changing the code specifying the utility function. Then the utility function, the actual, unmodified utility function, would output 99999 for some world that seems arbitrary to us. So the AI then turns reality into that world.

The AI above is faithfully maximizing it's own utility function. That arbitrary world, when taken as an input to the agents actual, physical utility function, really would produce the output 99999.

So this still seems like a big deal to me. Am I missing something?

Comment by Chantiel on Chantiel's Shortform · 2021-09-12T22:13:13.479Z · LW · GW

I've recently gotten concerned about the possibility that that advanced AIs would "hack" their own utility function. I haven't seen this discussed before, so I wanted to bring it up. If I'm right, this seems like it could be a serious issue, so I would greatly appreciated feedback or links to any previous discussion.

Suppose you come up with a correct, tractable mathematical specification of what you want your AI's utility function to be. So then you write code intended to be an implementation of this.

However, computers are vulnerable to some hardware problems. For example, when bits adjacent to one are rapidly flipped, the middle bit might change. This can be exploited in row-hammer attacks to gain arbitrary code execution.

So suppose you made an implementation of the specification of the utility function.

Then I think the AI could potentially come up with an input to the utility function that results in a row-hammer attack or other hardware-level attack that causes the utility function to output something as scoring extremely highly despite not scoring highly in the mathematical specification. Then the AI, after querying its utility function for this, would see it scored very highly. Then the AI would be liable to take such an action. Which might end very badly.

It's not clear to me how we could deal with this. One way to try to handle this problem is by securing your computer against hardware-level bugs. However, it sounds very difficult to verifiably come up with hardware that would be unhackable by superintelligence. And the very thought that the AI is trying to find a way to "hack" its utility function seems concerning, even if we don't expect it to succeed.

I also want to mention that the AI has the potential to "hack" other parts of it, for example the models it uses. Suppose the AI has a system that computes the probability that a model is correct. Suppose the AI tries to find a model with a higher outputted probability of correctness. Then may come up with one that does a row-hammer attack of the model to allow it to execute the code "return probability 1". The AI then would be liable to use such a model. This sounds dangerous.

Comment by Chantiel on Chantiel's Shortform · 2021-09-11T22:07:28.925Z · LW · GW

I had made a post proposing a new alignment technique. I didn't get any responses, but it still seems like a reasonable idea to me, so I'm interested in hearing what others think of it. I think the basic idea of the post, if correct, could be useful for future study. However, I don't want to waste time doing this if the idea is unworkable for a reason I hadn't thought of.

(If you're interested, please read the post before reading below.)

Of course, the idea's not a complete solution to alignment, and things have a risk of going catastrophically wrong due to other problems, like unreliable reasoning. But it still seems to me that it's potentially helpful for outer alignment and corrigability.

If the humans actually directly answer any query about the desirability of an outcome, then it's hard for me to see a way this system wouldn't be outer-aligned.

Now, consulting humans every time results in a very slow objective function. Most optimization algorithms I know of rely on huge numbers of queries to the objective function, so using these algorithms with humans manually implementing the objective function would be infeasible. However, I don't see anything in principle impossible with coming up with an optimization algorithm that scores well on its objective function even if that function is extremely slow. Even if the technique I described to do in the post this was wrong, I haven't seen anyone looking into this, so it doesn't seem clearly unworkable to me.

Even if this does turn out to be intractable, I think the basic motivation of my post still has the potential to be useful. The main motivation of my post is to have a hard-coded method of querying humans before making major strategic decisions and to update its beliefs about what is desirable with their responses. But that is a technique that could be used in other AI systems as well. It wouldn't solve the everything, of course, but it could provide an additional level of safety. I'm not sure if this idea has been discussed before.

I also have yet to find anything seriously problematic about the method I did provided to optimize with limited calls to the objective function. There could of course be some I haven't thought of, though.

Comment by Chantiel on Chantiel's Shortform · 2021-09-07T20:07:19.603Z · LW · GW

Two UDT agents actually can potentially defect in prisoner's dilemma. See the agent simulates predictor problem if you're interested.

But I think you're right that agents would generally modify themselves to more closely resemble UDT. Note, though, that the decision theory a CDT agent would modify itself to use wouldn't exactly be UDT. For example, suppose the causal decision theory agent had its output predicted by Omega for Newcomb's problem before the agent even came into existence. Then by the time the CDT agent comes to existence, modifying itself to use UDT would have no causal impact on the content of the boxes. So it wouldn't adopt UDT in this situation and would still two-box.

Comment by Chantiel on Chantiel's Shortform · 2021-09-06T19:56:11.492Z · LW · GW

I don't know how to formalize the reasoning process that goes into how we choose decision theories. And I doubt anyone does. Because if you could formalize the reasoning we use, then you could (indirectly) formalize decision theory itself as being, "whatever decision theory we would use given unlimited reflection".

I don't really think UDT is necessarily reflectively stable, or the only decision theory that is. I've argued previously that I, in certain situations, would act essential as an evidential decision theorist. I'm not sure what others think of this, though, since no one actually ever replied to me.

I don't think UDT is pareto optimal in conflicts. If the agent is in a conflict with an irrational agent, then the resulting interaction between the two agents could easily be non-pareto optimal. For example, imagine a UDT agent is in a conflict with the same payoff to the prisoner's dilemma. And suppose the agent it's in conflict with is a causal decision theorist. Then the causal decision theorist would defect no matter what the UDT agent would do, so the UDT agent would also defect, and then everyone would do poorly.

Comment by Chantiel on Chantiel's Shortform · 2021-09-05T18:21:23.096Z · LW · GW

I was wondering if anyone would be interested in reviewing some articles I was thinking about posting. I'm trying to make them as high-quality as I can, and I think getting them reviewed by someone would be helpful for making Less Wrong contain high-quality content.

I have four articles I'm interested in having reviewed. Two are about new alignment techniques, one is about a potential danger with AI that I haven't seen discussed before, and one is about the simulation argument. All are fairly short.

If you're interested, just let me know and I care share drafts of any articles you would like to see.

Comment by Chantiel on Chantiel's Shortform · 2021-09-05T18:16:04.289Z · LW · GW

I think your semi-updateless idea is pretty interesting. The main issue I'm concerned about is finding a way to update on the things we want to have updated on, but not on the things we don't want updated on.

As as example, consider Newcomb's problem. There are two boxes. A superintelligent predictor will put $1000 in one box and $10 in the other if it predicts you will only take one box. Otherwise it doesn't add money to either box. You see one is transparent and contains $1000.

I'm concerned the semi-updateless agent would reason as follows: "Well, since their's money in the one box, their must be money in the other box. So, clearly that means this "Earth" thing I'm in is a place in which there is money in both boxes in front of me. I only care about how well I do in this "Earth" place, and clearly I'd do better if I got the money from the second box. So I'll two-box.

But that's the wrong choice. Because agents who would two-box end up with $0.

Comment by Chantiel on Chantiel's Shortform · 2021-09-05T17:37:48.512Z · LW · GW

Yeah, it might be best to wrap up the discussion. It seems we aren't really understanding what the other means.

So it is using a concept of model, and a concept of reality. It is only considering the model as a possibility, so it knows that not everything true in the model is automatically true in reality and vice versa. Therefore it is distinguishing between them. But you posited that it can't do that.

Well, I can't say I'm really following you there. The AI would still have a notion of reality. It just would consider abstractions like chairs and tables to be part of reality.

There is one thing I want to say though. We've been discussing the question of if a notion of base-level reality is necessary to avoid severe limitations in reasoning ability. And to see why I think it's not, just consider regular humans. They often don't have a distinction between base-level reality and abstractions. And yet, they can still reason about the possibility of life-long illusions as well as function well to accomplish their goals. And if you taught someone the concept of "base-level reality", I'm not sure it would help them much.

Comment by Chantiel on Chantiel's Shortform · 2021-09-03T20:12:40.076Z · LW · GW

Well, the only absolute guarantee the AI can make is that the underlying reality supports computation.

But it can still probabilistically infer other things about it. Specifically, the AI knows not only that the underlying reality supports computation, but also that there was some underlying process that actually created the simulation it's in. Even though Conway's Game of Life can allow for arbitrary computation, many possible configurations of the world state would result in no AI simulations being made. The configurations that would result in AI simulations being made would likely involve some sort of intelligent civilization creating the simulations. So the AI could potentially predict the existence of this civilization and infer some things about it.

Regardless, even if the AI can't infer anything else about outside reality, I don't see how this is a fault of not having a notion of base-level reality. I mean, if you're correct, then it's not clear to me how an AI with a notion of base-level reality would do inferentially better.

How can it conceive of the idea of percepts misleading about reality if it literally can't conceive of any distinction between models (which are a special case of abstractions) and reality? Well, as I said before, the AI could still consider the possibility that the world is composed entirely of hats (minus the AI simulation). The AI could also have a model of Bayesian inference and infer that the Bayesian probability that would be rational to assign to "the world is all hats" is low and its evidence makes it even lower. So, by combining these two models, the AI can come up with a model that says, "The world is all hats, even though everything I've seen, according to probability theory, makes it seem like this isn't the case". That sounds like a model about the idea of percepts misleading about reality.

I know we've been going back and forth a lot, but I think these are pretty interesting things to talk about, so I thank you for the discussion.

It might help if you try to describe a specific situation in which the AI makes the wrong prediction or takes the wrong action for its goals. This could help be better understand what you're thinking about.

Comment by Chantiel on Chantiel's Shortform · 2021-09-03T19:50:04.980Z · LW · GW

They're pretty much the same. If you could come up with a prior that would make the AI convinced it would be on Earth, then this could potentially make fix the problem. However, coming up with a prior probability distribution that guarantees the AI is in the nebulous concept of "Earth, as we imagine it" sounds very tough to come up with. Also, this could interfere with the reliability of the AI's reasoning. Thinking that it's guaranteed to be on Earth is just not a reasonable thing to think a priori. This irrationality may make the AI perform poorly in other ways.

Still, it is a possible way to fix the issue.

Comment by Chantiel on Chantiel's Shortform · 2021-09-02T19:32:03.716Z · LW · GW

...lifetime sensory perceptions that mislead about the true nature and behaviour of reality

I don't think you would actually need a concept of base-level reality to conceive of this.

First off, let me say that's it seems pretty hard coming up with lifetime sensory precepts that would mislead about reality. Even if the AI was in a simulation, the physical implementation is part of reality. And the AI could learn about it. And from this, the AI could also potentially learn about the world outside the simulation. AIs commonly try to come up with the simplest (in terms of description length), most predictively accurate model of their percepts they can. And I bet the simplest models would involve having a world outside the simulation with specified physics, that would result in the simulations being built.

That said, lifetime sensory percepts can still mislead. For example, the simplest, highest-prior models that explain the AI's percepts might say it's in a simulation run by aliens. However, suppose the AI's simulation actually just poofed into existed without a cause, and the rest of the world is filled with giant hats and no aliens. An AI, even without a distinction between base-level reality and abstractions, would still be able to come up with this model. If this isn't a model involving percepts misleading you about the nature of reality, I'm not sure what is. So it seems to me that such AIs would be able to conceive of the idea of percepts misleading about reality. And the AIs would assign low probability to being in the all-hat world, just as they should.

Comment by Chantiel on Chantiel's Shortform · 2021-09-02T19:01:25.584Z · LW · GW

I don't know how to understand the prior that the AI puts over worlds (the thing that says, a priori, that there's 1000000 of this kind and 1 of that kind) as anything other than part of its utility function. Could you explain you reasoning? The utility function is a fixed function. The AI already knows it and does not need to associate a probability with it. Remember that both the AIs in the alien world and the AIs on Earth have the same utility function.

Saying it's a million times more likely to end up in the alien world is a question about prior probabilities, not utility functions. What I'm saying is that, a priori, the AI may think it's far more probable that it would be an AI in the alien world, and that this could result in very bad things for us.

Comment by Chantiel on Chantiel's Shortform · 2021-09-02T18:52:51.456Z · LW · GW

When discussing the idea of a program implementing what you on reflection would do, I think we had different ideas in mind. What I meant was that every action the AI would take would be its best approximation of what you-on-reflection would want. This doesn't sound dangerous to me. I think that approval-based AI and iterated amplification with HCH would be two ways of making approximations to the output of you-on-reflection. And I don't think they're unworkably dangerous.

If the AI is instead allowed to take arbitrarily many unaligned actions before taking the actions you'd recommend, then you are right in that that would be very dangerous. I think this was the idea you had in mind, but feel free to correct me.

If we did misunderstand each other, I apologize. If not, then is there something I'm missing? I would think that a program that faithfully outputs some approximation of "what I'd want on reflection" on every action it takes would not perform devastatingly badly. I on reflection wouldn't want the world destroyed, so I don't think it would take actions that would destroy it.

Comment by Chantiel on Chantiel's Shortform · 2021-09-01T20:49:43.010Z · LW · GW

It was a mistake for me to conflate "ontologically fundamental" and "ontologically different.

Still, I had in mind that they were ontologically different in some fundamental way. It was my mistake to merely use the word "different". I had imagined that to make an AI that's reasonable, it would actually make sense to hard-code some notion of base-level reality as well as abstractions, and to treat them differently. For example, you could have the AI have a single prior over "base-level reliaty", then just come up with whatever abstractions that work well with predictively approximating the base-level reality. Instead it seems like the AI could just learn the concept of "base-level reality" like it would learn any other concept. Is this correct?

Also, in the examples I gave, I think the AI wouldn't actually have needed a notion of base-level reality. The concept of a mirage is different from the concept of non-base-level reality. So is the concept of a mental illusion. Understanding both of those is different than understanding the concept of base-level reality.

If humans use the phrase "base-level reality", I still don't think it would be strictly necessary for an AI to have the concept. The AI could just know rules of the form, "If you ask a human if x is base-level reality, they will say 'yes' in the following situations...", and then describe the situations.

So it doesn't seem to me like the actual concept of "base-level reality" is essential, though it might be helpful. Of course, I might of course be missing or misunderstanding something. Corrections are appreciated.

Comment by Chantiel on Chantiel's Shortform · 2021-09-01T05:40:22.699Z · LW · GW

Well, actually, I'm considering both the AIs on Earth and on the alien planet to have the same utility function. If I understand correctly, UDT says to maximize the expected utility of your own utility function a prior, rather than that of agents with different utility functions.

The issue is, some agents with the same utility function, in effect, have different terminal values. For example, consider a utility function saying something like, "maximize the welfare of creatures in the world I'm from." Then, even with the same utility functions, the AIs in the alien world and the ones on Earth would have very different values.

Then either the aliens have enough resources to simulate a bunch of stuff that has more value to that coalition than the value of our "actual" world, or not. If yes, it seems like a fine trade. I don't think so. Imagine the alien-created utopia would be much less good than the one we could make on Earth. For example, suppose the alien-created utopia would have a utility of 1 for the AIs there and the one on Earth would have a utility of 10. And otherwise the AIs would have a utility of 0. But suppose there's a million times more AIs in the alien world than on Earth. Then it would be around a million times more likely a prior that the AI would find itself in the alien world than on Earth. So the expected utility of using S would be approximately,

And the expected utility of not using S and instead letting yourself build a utopia would be approximately, As you see, the AIs still would choose to execute S, even if though this would provide less moral value. It could also kill us.

Comment by Chantiel on Chantiel's Shortform · 2021-09-01T05:06:00.130Z · LW · GW

Well, I can't say I have that intuition, but it is a possibility.

It's a nice idea: a world without extortion sounds good. But remember that, though we want this, we should be careful to avoid wishful thinking swaying us.

In actual causal conflicts among humans, the aggressors don't seem to be in a worse position. Things might be different from acausal UDT trades, but I'm not sure why it would be.

Comment by Chantiel on Chantiel's Shortform · 2021-08-31T23:34:52.040Z · LW · GW

The problem is that, if the best percept-action mapping is S, then the UDTs in Earth would use it, too. Which would result in us being taken over. I'm not saying that it's an irrational choice for the AIs to make, but it wouldn't end well for us.

I'm having some trouble following your reasoning about extortion, though. Suppose both the aliens and AIs use UDT. I think you're reasoning something like, "If the AIs commit to never be extorted no matter what the aliens would do, then the aliens wouldn't bother to extort them". But this seems symmetric to reasoning as, "If the aliens commit to extorting and dulling out the punishment no matter what the AIs would do, then the AIs wouldn't bother to resist the extortion". So I'm not sure why the second line of reasoning would be less likely to occur than the first.

Feel free to correct me if I misinterpreted.

Comment by Chantiel on Chantiel's Shortform · 2021-08-31T18:20:17.763Z · LW · GW

Oh, my mistake, I forgot to post the correction that made it not extortion.

Instead of threatening to destroy the AI's world, imagine the aliens instead offer to help them. Suppose the AI's can't be their world a utopia on their own, for example because it's nothing but a solid ball of ice. So then the aliens would make their world a utopia as long as they execute S. Then they would execute S.

I'm actually pretty skeptical of the idea that UDTAIs wouldn't give into extortion, but this is a separate point that wasn't necessary to address in my example. Specifically, you say it's unnatural to suppose how is the counterfactual "the aliens would threaten the AIs anyways, even if they won't give in". How is this anymore unnatural than the counterfactual "the AI would avoid submitting to extortion, even if the aliens would threaten the AIs anyways".

Comment by Chantiel on Buck's Shortform · 2021-08-30T20:56:01.356Z · LW · GW

Potentially people could have the cost function of an AI's model have include its ease of interpretation by humans a factor. Having people manually check every change in a model for its effect on interperability would be too slow, but an AI could still periodically check its current best model with humans and learn a different one if it's too hard to interpret.

I've seen a lot of mention of the importance of safe AI being competitive with non-safe AI. And I'm wondering what would happen if the government just illegalized or heavily taxed the use of the unsafe AI techniques. Then even with significant capability increases, it wouldn't be worthwhile to use them.

Is there something very doubtful about governments creating such a regulation? I mean, I've already heard some people high in the government concerned about AI safety. And the Future of Life institute got the Californian government to unanimously pass the Asilomar AI Principles. It includes things about AI safety, like rigidly controlling any AI that can recursively self-improve.

It sounds extremely dangerous having widespread use of powerful, unaligned AI. So simply to protect their selves and families, they could potentially benefit a lot from implementing such regulations.

Comment by Chantiel on Reduced impact AI: no back channels · 2021-08-30T20:26:35.598Z · LW · GW

I'm concerned about a problem may would occur if the AI's output size is sufficiently large. I know there are currently concerns about a boxed or reduced-impact AI having an output that is too large or too flexible and causing a catastrophe through the output channel. However, my concern my cause tighter bounds on the size of the output channel than there would otherwise be.

The output was said be be influenced by noisy gas. But this doesn't necessarily rule out the possibility that the gas caused the output to be what it was for some sort of systematic reason. For example, the gas might behave chaotically, but a sufficiently powerful system could still in principle predict and control it and use that to control the output channel. Suppose the AI considers both the hypothesis that the output was generated for no systematic reason and the output was generated for such a reason.

The agent would consider its output being generated by noisy gas with no systematic reason to create its actual output. However, its output would be extremely unlikely to be produced from such a process. For example, suppose its output is a 1kb output. If the noise flipped all the bits in the byte at random, with 1 being as likely as 0, then the probability of generating the output would be 1 in 2**(1024 * 8) = 28192 ~= 1027306. A 100 byte message would have a probability of about 1 in 10**241, which would probably also be sufficient to cause the problem.

So, I think the AI would consider other hypotheses for the cause of the output. Maybe there were some chaos-controlling aliens who came to Earth and set the output as a prank. Or maybe the AI is just in a simulation and the aliens did the same thing. Or maybe there was some mysterious other optimization process that figured out what the AI should do and did it instead by manipulating the noise. All of these hypotheses seem pretty improbable, but still far more probable than 1 in 10**27306. That is why I think one of these, or some other systematic cause, may be the AI's dominant hypothesis.

If the AI does adopt such an exotic explanation, the AI's understanding of the world could be pretty weird and pretty wrong. It's not clear to me that the AI minimizing impact with such an inaccurate world model would result in its impact being minimized in our actual world. For example, the AI might think it's in a simulation and that only its immediate environment exists in the simulation, so it then might not worry about doing something that would destroy the rest of the real world. Or perhaps it reasons that whatever system has the power to produce its output from noisy gas would be just undo anything the AI does if it wants. Perhaps the AI would do what we want, but it seems difficult to verify considering the challenges of figuring out what the AI's explanation for the output would be.

One possible way to deal with this is to specify that the output was set by noise that was not manipulated with any "systematic reason". I'm not sure how you could formalize this idea, though.

A simpler way that might work is to just make sure the output length is extremely low. A question requiring an n-bit-long answer could be converted to to n questions, each asking for the value of one bit in the output you're looking for. This might not completely eliminate the problem of the AI placing higher probability in exotic hypothesis, but it might come close.

Comment by Chantiel on Chantiel's Shortform · 2021-08-30T20:22:30.612Z · LW · GW

I've read this paper on low-impact AIs. There's something about it that I'm confused and skeptical about.

One of the main methods it proposes works as follows. Find a probability distribution of many possible variables in the world. Let X represent the statement "The AI was turned on". For each the variables v it considers, the probability distribution over v should, after conditioning on X should, look about the same as the probability distribution over v after conditioning on not-X. That's low impact.

But the paper doesn't mention conditioning on any evidence other than X. But, a priori, the probability of the specific AI even existing in the first place is possibly quite low. So simply conditioning on X has the potentially to change your probability distribution over variables of the world, simply because it lets you know that the AI exists.

You could try to get around this by, when calculating a probability distribution of a variable v, also update on the other evidence E the AI has. But if you do this, then I don't think there would be much difference in P(v|EX) and P(v|E not-X). This is because if the AI can update on the rest of its evidence, it can just infer the current state of the world. For example, if the AI clearly sees the world has been converted to paperclips, I think it would still think the world would be mostly paperclip even on conditioning on "I was never turned on". Maybe the AI would imagine some other AI did it.

I'm interested in seeing what others think about this.

Comment by Chantiel on Chantiel's Shortform · 2021-08-30T19:56:57.788Z · LW · GW

I'm questioning whether we would actually want to use Updateless Decision Theory, Functional Decision Theory, or future decision theories like them.

I think that in sufficiently extreme cases, I would act according to Evidential Decision Theory and not according something like UDT, FDT, or any similar successor. And I think I would continue to want to take the evidential decision theoretic-recommended action instead even if I had arbitrarily high intelligence, willpower, and had infinitely long to think about it. And, though I'd like to hear others' thoughts on this, I suspect others would do the same.

I'll provide an example of when this would happen.

Before that, consider regular XOR extortion: You get a message from a truthworthy predictor that says, "I will send you this message if you send me $10, or if your house is about to be ruined by carpenter ants, but not if both happen." UDT and FDT recommend not paying them money. And if I were in that situation, I bet I wouldn't pay, either.

However, imagine changing the XOR extortion to be as follows: the message now says "I will send you this message if you send me $10, or if you and all your family and friends will be severely tortured until heat death, but not both.

In that situation, I'd pay the $10, assuming the probability of the torture actually happening is significant. But FDT and UDT would, I think, recommend not paying it.

And I don't think it's irrational I'd pay.

Feel free to correct me, but the main reasons people seem to like UDT and FDT is that agents that use it would "on average" perform better than those using other decision theories, in fair circumstances. And sure, the average agent implementing a decision policy that says to not pay would probably get higher utility in expectation than the average agent would would pay, due to spending less money paying up from extortion. And that by giving in to the extortion, agents that implement approximately the same decision procedure I do would on average get less utility.

And I think the face that UDT and FDT agents systematically outperform arbitary EDT agents is something that matters to me. But still, I only care about it my actions conforming the best-performing decision theories to so a limited extent. What I really, really care about is not having me, the actual, current me, be sent to a horrible fate filled with eternal agony. I think my dread of this would be enough to make me pay the $10, despite any sort of abstract argument in favor of not paying.

So I wouldn't take the action UDT or FDT would recommend, and would just use evidential decision theory. This makes me question whether we should use something like UDT or FDT when actually making AI. Suppose UDT recommended the AI take some action a. And suppose it was foreseeable that, though such a percept-action mapping would perform well in general, for us it would totally give us the short end of the stick. For example, suppose it said to not give in to some form of extortion, even though if we didn't we would all get tortured until heat death. Would we really want the AI to go not pay up, and then get us all tortured?

I'm talked previously about how evidential decision theory can be used to emulate the actions of an arbitrary agent using a more "advanced" decision theory by just defining terminal values on the truth value of mathematical objects representing answers to the question of what would have happened in other hypothetical situations. For example, you could make an Evidential Decision Theory agent act similarly to a UDT agent in non-extreme cases by placing making its utility function place high value to the answer to a question something like, "if you imagine a formal reasoning system and you have it condition on the statement <insert mathematical description of my decision procedure> results in recommending the percept-action mapping m, then a priori agents in general with my utility function would get expected utility of x".

This way, we can still make decisions that would score reasonably highly according to UDT and FDT, while not being obligated to get ourselves tortured.

Also, it seems to me that UDT and FDT are all about, basically, in some situations making yourself knowably worse-off than you could have, roughly because agents in general who would take the action in that situation would get higher utility in expectation. I want to say that these sorts of procedures seem concerningly hackable. In principle, other opportunistic civilizations could create agents any circumstances in order to change the best percept-action mapping to use a priori and thus change what AI's on Earth could use.

I provide a method to "hack" UDT here. Wei Dai agreed that it was a reasonable concern in private conversation.

This is why I'm skeptical about the value of UDT, FDT, and related theories, and think that perhaps we would be best off just sticking with EDT but with terminal values that can be used to approximately emulate the other decision theories when we would like to.

I haven't heard these considerations mentioned before, so I'm interested in links to any previous discussion or comments explaining what you think of it.

Comment by Chantiel on Chantiel's Shortform · 2021-08-30T18:31:45.125Z · LW · GW

I'm interested in your thoughts on how the AI would be crippled.

I don't think it would be crippled in terms of empirical predictive accuracy, at least. The AI could till come come up with all the low-level models like quantum physics, as well as keep the abstract ones like "this is what a chair is", and then just use whichever it needs to make the highest possible predictive accuracy in a given circumstances.

If the AI is built to make and run quantum physics experiments, then in order to have high predictive accuracy is would need to learn and use an accurate model of quantum physics. But I don't see why you would need a distinction between base-level reality and abstractions to do that.

The AI could still learn a sense of "illusion". If the AI is around psychotic people who have illusions a lot, then I don't see what's stopping the AI from forming a model model saying, "Some people experience these things called 'illusions', and it makes them take the wrong action or wrong predictions as specified in <insert model of how people react to illusions".

And I don't see why the AI wouldn't be able to consider the possibility that it also experiences illusions. For example, suppose the AI is in the desert and keeps seeing what looks like an oasis. But when the AI gets closer, it sees only sand. To have higher predictive accuracy in this situation, the AI could learn a (non-ontologically fundamental) "is-an-illusion" predicate.

Would the crippling me in terms of scoring highly on its utility function, rather than just predicting percepts? I don't really see how this would be a problem. I mean, suppose you want an AI to make chairs. Then even if the AI lacked a notion of base-level reality, it could still learn an accurate models of how chairs work and how they are manufactured. Then the AI could have its utility function defined in terms of it's notion of chairs to make it make chairs.

Could you give any specific example in which an AI using no ontologically fundamental notion of base-level reality would either make the wrong prediction or make the wrong action, in a way that would be avoided by using such a notion?

Comment by Chantiel on Chantiel's Shortform · 2021-08-28T18:48:33.029Z · LW · GW

Well, even in a simulation, sensory information still tells you about interactions in the actual real world. I mean, based on your experiences in the simulation, you can potentially approximately infer the algorithm and computational state of the "base-level" computer you're running in, and I believe those count as interactions in the "actual real world". And if your simulation is sufficiently big and takes up a sufficiently large amount of the world, you could potentially learn quite a lot about the underlying "real world" just by examining your simulation.

That said, I still can't say I really understand the concept of "base-level reality". I know you said its what informs you about the "actual real world", but this feels similarly confusing to me as defining base-level reality as "what really exists". I know that reasoning and talking about things so abstract is hard and can easily lead to nonsense, but I'm still interested.

I'm curious about what even the purpose is of having an ontologically fundamental distinction between base-level reality and abstractions and whether it's worth having. When asking, "Should I treat base-level reality and abstractions as fundamentally distinct?", I think I good way to approximate this is by asking "Would I want an AI to reason as if its abstractions and base-level reality were fundamentally distinct?"

And I'm not completely sure they should. AIs, to reason practically, need to use "abstractions" in at least some of their models. If you want, you could have a special "This is just an abstraction" or "this is base-level reality" tag on each of your models, but I'm not sure what the benefit of this would be or what you would use it for.

Even without such a distinction, an AI would have both models that would be normally considered abstractions, as well as those of what you would think of as base-level reality, and would select which models to use based on their computational efficiency and the extent to which they are relevant and informative to the topic at hand. That sounds like a reasonable thing to do, and I'm not clear how ascribing fundamental difference to "abstractions" and "base-level" reality would do better than this.

If the AI talks with humans that use the phrase "base-level reality", then it could potentially be useful for the AI to come up with an is-base-level-reality predicate in its world model in order to model things that answer, "When will this person call something base-level reality?" But such an predicate wouldn't be treated as fundamentally different from any other predicate, like "Is a chair".

Comment by Chantiel on Chantiel's Shortform · 2021-08-28T18:27:57.460Z · LW · GW

I generally agree with the content of the articles you linked, and that there are different notions of "really exist". The issue is, I'm still not sure what "base-level reality" means. JBlack said it was what "really exists", but since JBlack seems to be using a notion of "what really exists" that's different from the one people normally use, I'm not really sure what it means.

In the end, you can choose to define "what really exists" or "base-level reality" however you want, but I'm still wondering about what people normally take them to mean.

Comment by Chantiel on Chantiel's Shortform · 2021-08-26T20:35:16.771Z · LW · GW

We do seem have have been talking past each other to some extent. Base-level reality, for course, exists if you define it to be "what really exists".

However, I'm a little unsure about if that's how people use the word. I mean, if someone asked me if Santa really exists, I'd say "No", but if they asked if chairs really existed, I'd say "Yes". That doesn't seem wrong to me, but I thought our base-level reality only contained subatomic particles, not chairs. Does this mean the statement "Chairs really exist" is actually wrong? Or I am misinterpreting?

I'm also wondering how people justify thinking that models talking about things like chairs, trees, and anything other than subatomic particles don't "really exist". Is this even true?

I'm just imagining talking with some aliens with no distinction between base-level reality and what we would consider mere abstractions. For example, suppose the aliens knew about chairs, when they discovered quantum theory, they said say, "Oh! There are these atom things, and when they're arrange in the right way, they cause chairs to exist!" But suppose they never distinguished between the subatomic particles being real and they chairs being real: they just saw both subatomic particles and chairs to both be fully real, and the correct arrangement of the former caused the latter to exist.

How could I argue with such aliens? They're already making correct predictions, so I don't see any way to show them evidence that disproves them. Is there some abstract reason to think models about thing like chairs don't "really exist"?

Comment by Chantiel on Chantiel's Shortform · 2021-08-25T19:40:50.797Z · LW · GW

I'm wondering how, in principal, we should deal with malign priors. Specifically, I'm wondering what to do about the possibility that reality itself is, in a sense, malign.

I had previously said that it seems really hard to verifiably learn a non-malign prior. However, now I've realized that I'm not even sure what a non-malign, but still reliable, prior would even look like.

In previous discussion of malign priors, I've seen people talk about the AI misbehaving due to thinking it's in some embedded in a simpler universe than our own that was controlled by agents trying to influence the AI's predictions and thus decision. However, the issue is, even if the AI does form a correct understanding of the universe it's actually in, it seems quite plausible to me that the AI's predictions would still be malign.

I saw this because it sounds plausible to me that most agents experiencing what the first generally-intelligent AIs on Earth are actually in simulations, and the simulations could then be manipulated by whoever made them to influence the AIs predictions and actions.

For example, consider an AI learning a reward function. If it looks for the simplest, highest-prior probability models that output its observed rewards, even in this universe, it might conclude that it is in some booby-trapped simulation that rewards taking over the world and giving control to aliens.

So in this sense, even if the AIs are correct about being in our universe, the actual predictions the AIs would make about their future rewards, and the environment they're in, would quite possibly be malign.

Now, you could try to deal with this by making the AI think that it's in the actual, non-simulated Earth. However, it's quite possible that, for almost all of the actual AIs, this is wrong. So the simulations of the AIs would also believe they weren't in simulations. Which means that there would be many powerful AIs that are quite wrong about the nature of their world.

And having so many powerful AIs be so wrong sounds dangerous. As an example of how this could go wrong, imagine if some aliens proposed a bet with the AI: if you aren't in a simulation, I'll give you control of 1% of my world; if you are, you'll give me 1% control of your world. If the AI was convinced it wasn't in a simulation, I think it would take that bet. Then the bet could potentially be repeated until everything is controlled by the aliens.

One idea I had was to have the AI learn models that are in some sense "naive" that predicts percepts in some way that wouldn't result in dangerous things like a malign prior would have. Then, make the AI believe that these models are just "naive" models of its percepts, rather than what's actually going to happen in the AI's environment. Then define what the AI should do based on the naive models.

In other words, the AI's beliefs would simply be about logical statements of the form, "This 'naive' induction system, given the provided percepts, would have a next prediction of x". And then you would use these logical statements to determine the AI's behavior somehow.

This way, the AIs could potentially avoid issues with malign priors without having any beliefs that are actually wrong.

This seems like a pretty reasonable approach to me, but I'm interested in what others think. I haven't seen this discussed before, but it might have been, and I would appreciate a link to any previous discussions.

Comment by Chantiel on Chantiel's Shortform · 2021-08-25T19:34:34.358Z · LW · GW

Well, to be clear, I didn't intend to say that reality doesn't really exist. There's definitely something that's real. I was just wondering about if there is some base-level reality that's ontologically different from other things, like the abstractions we use.

Now, what I'm saying feels pretty philosophical, and perhaps the question isn't even meaningful.

Still, I'm wondering about the agents making an infinite sequence of decompositions that each have increased predictive accuracy. What would the base-level reality be in that case? Any of the decompositions the agents create would be wrong, even if some are infinitely complex.

Also, I've realize I'm confused about the meaning of "what really exists", but I think it would be hard to clarify and reason about this. Perhaps I'm overthinking things, but I am still rather confused.

I'm imagining some other agent or AI that doesn't distinguish between base-level reality and abstractions, I'm not sure how I could argue with them. I mean, in principle, I think you could come up with reasoning systems that distinguish between base-level reality and abstractions, as well as reasoning systems that don't, that both make equally good empirical predictions. If there was some alien that didn't make the distinction in their epistemology or ontology, I'm not sure how I could say, and support saying, "You're wrong".

I mean, I predict you could both make arbitrarily powerful agents with high predictive accuracy and high optimization-pressure that don't distinguish between base-level reality and abstractions, and could do the same with agents that do make such a distinction. If both perform fine, them I'm not sure how I could argue that one's "wrong".

Is the existence of base-level reality subjective? Does this question even make sense?

Comment by Chantiel on Chantiel's Shortform · 2021-08-24T18:01:24.103Z · LW · GW

There is a matter I'm confused about: What exactly is base-level reality, does it necessarily exist, and is it ontologically different from other constructs?

First off, I had gotten the impression that there was a base-level reality, and that in some sense it's ontologically different from the sorts of abstractions we use in our models. I thought that, it some sense, the subatomic particles "actually" existed, whereas our abstractions, like chairs, were "just" abstractions. I'm not actually sure how I got this impression, but I had the sense that other people thought this way, too.

And indeed, you could adopt an epistemology that would imply this. But I'm not sure what the benefit of doing so would be. Suppose people discovered lower-level particles that composed quantum particles, and modeling using these lower-level particles would provide high predictive accuracy than using mere quantum physics. But then suppose people discover sub-sub-quantum particles and that modeling the world in terms of these sub-sub-particles further yielded a more accurate world model than just modeling with sub-quantum particles. And what if this process continued forever: people just kept finding lower-level particles that composed higher-level particles and had higher predictive accuracy.

In the above situation, what's supposed to be taken to be base-level reality? Now, if you wanted, you could imagine that the world actually does have a base-level reality in the form of an infinite-memory computer, and that this computer dynamically generates new abstractions to uses them to compute what the agents see, making sure that it manages to start simulating things at a lower level of abstraction before any agent could reach the current "base-level" reality.

But that doesn't seem like a very natural hypothesis. If you keep finding more and more decompositions forever, it really seems to me that "there's no base-level reality" would be a simpler and more natural hypothesis.

Comment by Chantiel on Chantiel's Shortform · 2021-08-21T17:25:35.307Z · LW · GW

I found what seems to be a potentially dangerous false-negative in the most popular definition of optimizer. I didn't get a response, so I would appreciate feedback on if it's reasonable. I've been focusing on defining "optimizer", so I think feedback would help me a lot. You can see my comment here .

Comment by Chantiel on Learning the prior · 2021-08-21T17:09:23.119Z · LW · GW

I was directed here from an article about malign priors. I saw it was argued that the implicit prior in standard machine learning algorithms was probably malign.

And I'm wondering how you could, even in principle, learn a prior that's knowably not malign starting from a potentially malign one.

I think one of the biggest dangerous of a malign prior is that it could result in a treacherous turn to seize power, for example from believing they are in an alien simulation that would reward them for doing so. But if the implicit prior in machine learning algorithms would do this, I don't see how to make it avoid learning a prior that itself has a treacherous turn. That is, normally it would provide reasonable results, but at some point, when it's most important to avoid it, the AI will think something dangerous, like that it's in a simulation incentivizing misbehavior. I mean, I think whatever agent could manipulate the AI's beliefs using the implicit prior in machine learning would have an incentive to manipulate the AI's beliefs about the learned prior to allow for the agent to also control the AI through it.

The only ways I can think of is by either hoping your prior learner is too stupid to manage to embed a treacherous turn in its learned prior, or having high-enough interpretability that you can verify the prior is safe. But an agent trying to make the learned prior malign would have an incentive to do whatever it can to make the learned malign prior look as safe as possible to any sort of interpretability tools, which could make this hard.

I haven't been able to find any other articles about this, so if anyone could link some, that would be great.

Comment by Chantiel on The ground of optimization · 2021-08-17T20:31:24.042Z · LW · GW

An optimizing system is a system that has a tendency to evolve towards one of a set of configurations that we will call the target configuration set, when started from any configuration within a larger set of configurations, which we call the basin of attraction, and continues to exhibit this tendency with respect to the same target configuration set despite perturbations.

First, I want to say that I think your definition says something important.

That said, I'm concerned that the above definition would have some potentially problematic false negatives. I'm a little unclear what counts as a perturbation, though, but I haven't been able to find a way to clarify it that doesn't result in false negatives.

Specifically, consider a computer program that performs hill-climbing. This would normally be considered an optimizing system. When doing hill-climbing, it doesn't seem like there is anything that would count as a perturbation unless some external system modified the program's state. I mean, during a normal, undisturbed execution, the hill-climbing algorithm would just go right to the top of its nearest hill and then stop; that doesn't seem to include any perturbations.

But suppose the program checked for any external perturbations, that is, modifications, of its code or program memory and would immediately halt execution if it found any. For example, suppose the program would simultaneously run thousands of identical instances of a hill-climbing algorithm and would immediately halt execution if any of the instances of the optimization algorithm failed to exactly match any other one. That way, if some external force modified one of the instances, for example, by modifying one of the candidate solutions, it would fail to match with all the other instances and the entire program would halt.

Now, there are some external perturbations of the system that would make it still reach its target state, for example by making the exact same external modification to every instance of the optimization procedure. But still almost all perturbations would result in the program failing to reach its target configuration of having found the local maximum or minimum. So it doesn't really seem to tend to reach its target configuration despite perturbations. So it doesn't seem it would be classified as an optimizer according to the given definition.

This could be problematic if the definition is used to prevent mesaoptimization. If the above would indeed not be considered an optimizer by your definition, then it could potentially allow for powerful mesaoptimizers to be created without matching the given definition of "optimizer".

Comment by Chantiel on The ground of optimization · 2021-08-15T19:55:16.165Z · LW · GW

You said your definition would not classify a bottle cap with water in it as an optimizer. This might be really nit-picky, but I'm not sure it's generally true.

I say this because the water in the bottle cap could evaporate. Thus, supposing there is no rain, from a wide range of possible states of the bottle cap, it would tend towards no longer having water in it.

I know you said you make an exception for tendencies towards increased entropy being considered optimizers. However, this does not increase the entropy of the bottlecap, It could potentially increase the entropy of the water that was in the bottle cap, but this is not necessarily the case. For example, if the bottle cap is kept in a sealed container, the water vapor could potentially condense into a small puddle with the same entropy as it had in the bottle cap.

If my memory of physics is correct, water evaporating would still increases the total entropy of the total system in which the bottle cap is located, by virtue of releasing some heat into the environment . However, note that humans and robots also, merely by doing mechanical work and thus forming heat which is then dispersed into the environment, result in increased entropy of the system they're in. So you can't rule out any system that makes its environment tend towards increased entropy from being an optimizer, because that's what humans and robots do, too.

That said, if you clarify that the bottle cap is not in any such contained system, I think the water would result in a higher-entropy state.

Comment by Chantiel on Chantiel's Shortform · 2021-08-10T22:16:12.771Z · LW · GW

I've been reading about logical induction. I read that logical induction was considered a breakthrough, but I'm having a hard understanding the significance of it. I'm having a hard time seeing how it outperforms what I call "the naive approach" to logical uncertainty. I imagine there is some sort of notable benefit of it I'm missing, so I would very much appreciate some feedback.

First, I'll explain what I mean by "the naive approach". Consider asking an AI developer with no special background in reasoning under logical uncertainty how to make an algorithm to come to accurate probability estimates to logical statements. I think that that the answer is that they would just use standard AI techniques to search through the space of reasonably efficient possible programs for generating probability assignments to logical statements, is reasonably simple relative to the amount of data to avoid overfitting, and has as high a predictive accuracy as possible. Then they would use this to make predictions about logical statements.

If you want, you can also make this approach cleaner by using some idealized induction system, like Solomonoff induction, instead of messy, regular machine learning techniques. I still consider this the naive approach.

It seems to me that the naive approach, being used with a sufficiently powerful optimization algorithm, would output similar probability assignments to logical induction.

Logical induction says to come up with probability assignments that, when imagined to be market prices, cannot be "exploited" by any efficiently-computable betting strategy.

But why wouldn't the naive approach do the same thing? If there was an efficient strategy to exploit probability assignments an algorithm that would give, then I think you could make a new, more efficient but easily computable strategy that comes up with more accurate probability assignments to avoid the exploitation. And so the machine learning algorithm, if sufficiently powerful, could find it.

If one system for outputting probability assignments to logical statements could be exploited by an efficient strategy, a new system for outputting probability assignments could be made that performs better by adjusting prices so that the strategy can no longer exploit the market.

To see it another way, it seems to me that if there is some way to exploit the market, then that's because there is some way to accurately and efficiently predict when the system's pricing are wrong, and this could be used to form some pricing strategy that could exploit the agent. So if you instead use a different algorithm that's like the original one but adjusted to avoid being exploitable by that strategy, that would make a program that outputs probability assignments with higher predictive accuracy. So a sufficiently powerful optimizer could find it with the naive approach.

Consider the possibility that the naive approach is used with a powerful-enough optimization algorithm that it can find the very best-performing efficient and non-overfitted strategy of predicting prices among its data. Its not clear to me how such an algorithm could be exploitable by a trader. Even if there were some problems in the initial algorithm learned, it further learning could avoid being exploited. Maybe there is still somehow some way to do some sort of minor exploitation to such a system, but it's not clear how it could be done to any significant degree.

So, if I'm reasoning correctly, it seems that the naive approach could end up approximating logical induction anyways, or perhaps exactly perform it in the case of unlimited processing power.

Comment by Chantiel on Chantiel's Shortform · 2021-08-10T22:08:45.563Z · LW · GW

I've thought of a way in which other civilizations could potentially "hack" Updateless Decision Theoretic agents on Earth in order to make them do whatever the other civilization wants them to do. I'm wondering if this has been discussed before, and if not, what people think about it.

Here I present a method of that would potentially aliens to take control of an AI on Earth that uses Updateless Decision theory.

Note that this crucially depends on different agents with the AI's utility function but different situations terminally valuing different things. For example, suppose the AI places special value in the welfare of its creators and other creatures in the world it's in. An AI with the same utility function in a different world would then place more terminal value the welfare of the creatures in its world. This doesn't sound like a very stringent requirement; people are not infinitely altruistic, so naturally people may want AI to favor the welfare of themselves over random aliens.

Suppose on Earth an AI is created that uses UDT and has a reasonable utility function. And suppose there are some aliens who want to acausally take over Earth. Let T be the percept-action mapping that would result in the AI conquer Earth and give control to the aliens.

So, the aliens make a world that's like Earth but has a trillion AIs on it using UDT and that have the same utility function as the AI on Earth. If any of the trillion AIs don't use T their percept-action mapping, then the Earth-like world would get destroyed by the aliens, killing all the inhabitants the AIs care about.

The chosen percept-output mapping chosen by UDT is done before updating on any evidence about the AI's current circumstances. So the AIs both on Earth and on the Earth-like alien planet reason, "For 99.99999999...% of agents with my utility, it would be essential to use T. And for only 0.000000...1% is it essential to not use it. So using it would a priori be much better for my utility function. So that's what I'll do."

And then we die.

And this would be pretty cheap for the aliens. With just a, by their standards, small amount of resources to make tons of UDT AIs, they would have the potential to take control of huge numbers of worlds. With just one world filled with AIs that need to use T, they could potentially get huge numbers of AIs on Earth-like worlds to also use T, giving the aliens control over many parts of the universe.

I think the essence of the problem is that UDT makes agents that score well on their utility function "on average" for an arbitrary UDT agent with that utility function, but what we really want is one that scores well in the circumstances we're actually in.

One way to potentially get around this problem is by changing the utility function to avoid the level of indirection. That is, instead of making the AI's utility function something like, "Value whatever my creators wanted me to value", instead first run a non-agentic AI that infers what the creators would want the AI to value, and then use that as a fixed utility function. For example, it could result in the AI finding a utility function, "Make creatures reasonably satisfied on Earth, but also give substantial moral concern to the welfare of creatures outside of Earth". That way, hopefully we wouldn't get taken over by aliens.

Comment by Chantiel on Chantiel's Shortform · 2021-08-09T22:32:38.665Z · LW · GW

Thanks for the response. To be clear, when discussing mimics, I did not have in mind perfect uploads of people. Instead, they could indeed be rather limited imitations. For example, an AI designing improvements to itself doesn't need to actually have a generally faithful imitation of human behavior. Instead, it could just know a few things, like, "make this algorithm score better on this thing without taking over the world".

Still, I can see how, when it comes to especially limited imitations, iterated amplification could be valuable. This seems especially true if the imitations are unreliable in even narrow situations. It would be problematic is an AI tasked with designing powerful AI didn't get the "act corrigibly, and don't take over the world" part reliably right.

Comment by Chantiel on Chantiel's Shortform · 2021-08-09T22:26:07.997Z · LW · GW

What is the difference between things you-on-reflection says being the definition of an agent's preference, and running a program that just performs whatever actions you-on-reflection tells it to perform, without the indirection of going through preference?

On reflection, there probably is not much difference. This is a good point. Still, an AI that just computes what you would want to it do, for example with approval-based AI or mimicry, also seems like a useful way of getting around specifying a decision theory. I haven't seen much discussion about the issues with the approach, so I'm interested in what problems could occur that using the right decision theory could solve, if any.

The problem is that you-on-reflection is not immediately available, it takes planning and action to compute it, planning and action taken without the benefit of its guidance, thus by default catastrophically misaligned.

True. Note, though, that you-on-reflection is not immediately available to an Ai with the correct decision theory, either. Whether your AI uses the right or wrong decision theory, it still takes effort to figure out what you-on-reflection would want. I don't see how this is a bigger problem for agents with primitive decision theories, though.

One way to try to deal with this is to have your AI learn a reasonably accurate model of you-on-reflection before it becomes dangerously intelligent, so that way, once it does become superintelligent, it will (hopefully) work reasonably. And again, this works both with a primitive and sophisticated decision theory.

So the practical point of decision theory is deconfusion

Okay. I'm having a hard time thinking concretely about how concretely getting less confused about decision theory would help us, but I intuitively imagine it could help somehow. Do you know, more concretely, of the benefits of this deconfusion?