Posts
Comments
I like this analogy. Probably not best to put too much weight on it, but it has some insights.
And whether those programs could then perform well if their opponent forces them into a very unusual situation, such as would not have ever appeared in a chessmaster game.
If I sacrifice a knight for no advantage whatsoever, will the opponent be able to deal with that? What if I set up a trap to capture a piece, relying on my opponent not seeing the trap? A chessmaster playing another chessmaster would never play a simple trap, as it would never succeed; so would the ML be able to deal with it?
PS: the other title I considered was "Why do people feel my result is wrong", which felt too condescending.
I agree we're not as good as we think we are. But there are a lot of things we do agree on, that seem trivial: eg "this person is red in the face, shouting at me, and punching me; I deduce that they are angry and wish to do me harm". We have far, far, more agreement than random agents would.
Your title seems clickbaity
Hehe - I don't normally do this, but I feel I can indulge once ^_^
having implicit access to categorisation modules that themselves are valid only in typical situations... is not a way to generalise well
How do you know this?
Moravec's paradox again. Chessmasters didn't easily program chess programs; and those chess programs didn't generalise to games in general.
Should we turn this into one of those concrete ML experiments?
That would be good. I'm aiming to have a lot more practical experiments from my research project, and this could be one of them.
Hum... It seems that we can stratify here. Let represent the values of a collection of variables that we are uncertain about, and that we are stratifying on.
When we compute the normalising factor for utility under two policies and , we normally do it as:
- , with .
And then we replace with .
Instead we might normalise the utility separately for each value of :
- Conditional on , then , with .
The problem is that, since we're dividing by the , the expectation of is not the same .
Is there an obvious improvement on this?
Note that here, total utilitarianism get less weight in large universes, and more in small ones.
I'll think more...
How about a third AI that gives a (hidden) probability about which one you'll be convinced by, conditional on which argument you see first? That hidden probability is passed to someone else, then the debate is run, and the result recorded. If that third AI gives good calibration and good discrimination over multiple experiments, then we can consider its predictions accurate in the future.
Er, this normalisation system way well solve that problem entirely. If prefers option (utility ), with second choice (utility ), and all the other options as third choice (utility ), then the expected utility of the random dictator is for all (as gives utility , and gives utility for all ), so the normalised weighted utility to maximise is:
- .
Using (because scaling doesn't change expected utility decisions), the utility of any , , is , while the utility of is . So if , the compromise option will get chosen.
Don't confuse the problems of the random dictator, with the problems of maximising the weighted sum of the normalisations that used the random dictator (and don't confuse the other way, either; the random dictator is immune to players' lying, this normalisation is not).
But community isn't about friends; it's about a background level of acquaintances you're comfortable with.
I finished the research agenda on constructing a preference utility function for any given human, and presented the ideas to CHAI and MIRI. Woot!
Something like or or in general (for decreasing, continuous ) could work, I think.
I'd say that intelligence variations are more visible in (modern) humans, not that they're necessarily larger.
Let's go back to the tribal environment. In that situation, humans want to mate, to dominate/be admired, to have food and shelter, and so on. Apart from a few people with mental defects, the variability in outcome is actually quite small - most humans won't get ostracised, many will have children, only very few will rise to the top of the hierarchy (and even there, tribal environments are more egalitarian that most, so the leader is not that much different from the others). So we might say that the variation in human intelligence (or social intelligence) is low.
Fast forward to an agricultural empire, or to the modern world. Now the top minds can become god emperor, invading and sacking other civilizations, or can be part of projects that produce atom bombs and lunar rockets. The variability of outcomes is huge, and so the variability in intelligence appears to be much higher.
That's an excellent summary.
One might think just doing ontology doesn't involve making preference choice but making some preferences impossible to articulate it is in fact a partial preference choice.
Yep, that's my argument: some (but not all) aspects of human preferences have to be included in the setup somehow.
it's more reasonable for a human to taste a salt level differnce, it's more plausible to say "I couldn't know" about radioactivity
I hope you don't taste every bucket of water before putting it away! ^_^
In the later part of the post, it seems you're basically talking about entropy and similar concepts? And I agree that "reversible" is kinda like entropy, in that we want to be able to return to a "macrostate" that is considered indistinguishable from the starting macrostate (even if the details are different).
However, as in the the bucket example above, the problem is that, for humans, what "counts" as the same macrostate can vary a lot. If we need a liquid, any liquid, then replacing the bucket's contents with purple-tinted alchool is fine; if we're thinking of the bath water of the dear departed husband, then any change to the contents is irreversible. Human concepts of "acceptably similar" don't match up with entropic ones.
there needs to be an effect that counts as "significant".
Are you deferring this to human judgement of significant? If so, we agree - human judgement needs to be included in some way in the definition.
Relative value of the bucket contents compared to the goal is represented by the weight on the impact penalty relative to the reward.
Yep, I agree :-)
I generally think that impact measures don't have to be value-agnostic, as long as they require less input about human preferences than the general value learning problem.
Then we are in full agreement :-) I argue that low impact, corrigibility, and similar approaches, require some but not all of human preferences. "some" because of arguments like this one; "not all" because humans with very different values can agree on what constitutes low impact, so only part of their values are needed.
Good idea.
intrinsic motivation
That might be the concept I'm looking for. I'll think whether it covers exactly what I'm trying to say...
Ok, we strongly disagree on your simple constraints being enough. I'd need to see these constraints explicitly formulated before I had any confidence in them. I suspect (though I'm not certain) that the more explicit you make them, the more tricky you'll see that it is.
And no, I don't want to throw IRL out (this is an old post), I want to make it work. I got this big impossibility result, and now I want to get around it. This is my current plan: https://www.lesswrong.com/posts/CSEdLLEkap2pubjof/research-agenda-v0-9-synthesising-a-human-s-preferences-into
Very worthwhile concern, and I will think about it more.
We may not be disagreeing any more. Just to check, do you agree with both these statements:
-
Adding a few obvious constraints rule out many different R, including the ones in the OP.
-
Adding a few obvious constraints is not enough to get a safe or reasonable R.
I've added an edit to the post, to show the problem: sometimes, the robot can't kick the bucket, sometimes it must. And only human preferences distinguish these two cases. So, without knowing these preferences, how can it decide?
Rejecting any specific R is easy - one bit of information (at most) per specific R. So saying "humans have preferences, and they are not always rational or always anti-rational" rules out R(1), R(2), and R(3). Saying "this apparent preference is genuine" rules out R(4).
But it's not like there are just these five preferences and once we have four of them out of the way, we're done. There are many, many different preferences in the space of preferences, and many, many of them will be simpler than R(0). So to converge to R(0), we need to add huge amounts of information, ruling out more and more examples.
Basically, we need to include enough information to define R(0) - which is what my research project is trying to do. What you're seeing as "adding enough clear examples" is actually "hand-crafting R(0) in totality".
For more details see here: https://arxiv.org/abs/1712.05812
kicking the bucket into the pool perturbs most AUs. There’s no real “risk” to not kicking the bucket.
In this specific setup, no. But sometimes kicking the bucket is fine; sometimes kicking the metaphorical equivalent of the bucket is necessary. If the AI is never willing to kick the bucket - ie never willing to take actions that might, for certain utility functions, cause huge and irreparable harm - then it's not willing to take any action at all.
Your presentation had an example with randomly selected utility functions in a block world, that resulted in the agent taking less-irreversible actions around a specific block.
If we have randomly selected utility functions in the bucket-and-pool world, this may include utilities that care about the salt content or the exact molecules, or not. Depending on whether or not we include these, we run the risk of preserving the bucket when we need not, or kicking it when we should preserve it. This is because the "worth" of the water being in the bucket varies depending on human preferences, not on anything intrinsic to the design of the bucket and the pool.
we have plenty of natural assumptions to choose from.
You'd think so, but nobody has defined these assumptions in anything like sufficient detail to make IRL work. My whole research agenda is essentially a way of defining these assumptions, and it seems to be a long and complicated process.
Basically yes. My take on 2) is that identity-affirming things can be somewhat pleasurable - but they're unlikely to be the most pleasurable thing the human could do at that moment. So they can be valued for something else than pure pleasure.
And you can get other examples where someone, say, is truthful, even if that causes them more pain than a simple lie would.
Cool, thanks. I see the torture example as being closer to hedonism (or rather, anti-hedonism), though.
Thanks, good idea.
These are valid points, but we have wandered a bit away from the initial argument, and we're now talking about numbers that can't be compared (my money is on TREE(3) being smaller in this example, but that's irrelevant to your general point), or ways of truncating in the infinite case.
But we seem to have solved the finite-and-comparable case.
Now, back to the infinite case. First of all, there may be a correct decision even if probabilities cannot be computed.
If we have a suitable utility function, we may decide simply not to care about what happens in universes that are of the type 5, which would rule them out completely.
Or maybe the truncation can be improved slightly. For example, we could give each observer a bubble of radius 20 mega-light years, which is defined according to their own subjective experience: how many individuals do they expect to encounter within that radius, if they were made immortal and allowed to explore it fully.
Then we truncate by this subjective bubble, or something similar.
But yeah, in general, the infinite case is not solved.
If we set aside infinity, which I don't know how to deal with, then the SIA answer does not depend on utility bounds - unlike my anthropic decision theory post.
Q1: "How many copies of people (currently) like me are there in each universe?" is well-defined in all finite settings, even huge ones.
Incidentally, when you say there are “not many” copies of me in universes 3 and 4, then you presumably mean “not a high proportion, compared to the vast total of observers”
No, I mean not many, as compared with how many there are in universes 1 and 2. Other observers are not relevant to Q1.
I'll reiterate my claim that different anthropic probability theories are "correct answers to different questions": https://www.lesswrong.com/posts/nxRjC93AmsFkfDYQj/anthropic-probabilities-answering-different-questions
Yep, sorry, I saw -3, -2, -1, etc... and concluded you weren't doing the 2 jumps; my bad!
Then somehow the work is just postponed to the point where we try to combine partial preferences?
Yes. But unless we have other partial preferences or meta-preferences, then the only resonable way of combining them is just to add them, after weighting.
I like your reciprocal weighting formula. It seems to have good properties.
"How many copies of people like me are there in each universe?"
Then as long as your copies know that 3K has been observed, and excluding simulations and such, the answers are "(a lot, a lot, not many, not many)" in the four universes (I'm interpreting "die off before spreading through space" as "die off just before spreading through space").
This is the SIA answer, since I asked the SIA question.
Each partial preference is meant to represent a single mental model inside the human, with all preferences weighted the same (so there can't be "extremely weak" preferences, compared with other preference in the same partial preference). Things like "increased income is better", "more people smiling is better", "being embarrassed on stage is the worse".
We can imagine a partial preference with more internal structure, maybe internal weights, but I'd simply see that as two separate partial preferences. So we'd have the utilities you gave to through to for one partial preference (actually, my formula doubles the numbers you gave), and , , for the other partial preference - which has a very low weight by assumption. So the order of and is not affected.
EDIT: I'm pretty sure we can generalise my method for different weights of preferences, by changing the formula that sums the squares of utility difference.
Neat!
Though I should mention that my current version of partial preferences does not assume all cycles are closed - the constrained optimisation can be seen as trying to get "as close as possible" to that, given non-closed cycles.
What's the set of answers, and how are they assessed?
-
I encourage you to submit other ideas anyway, since your ideas are good.
-
Not sure yet about how all these things relate; will maybe think of that more later.
Hey there! Thanks for your long comment - but, alas, this model of partial preferences is obsolete :-(
See: https://www.lesswrong.com/posts/BydQtwfN97pFwEWtW/toy-model-piece-1-partial-preferences-revisited
Because of other problems with this, I've replaced it with the much more general concept of a preorder. This can express all the things we want to express, but is a lot less intuitive for how humans model things. I may come up with some alternative definition at some point (less general than a preorder, but more general than this post.
Thanks for the comment in any case.
I find it hard to imagine that you're actually denying that you or I have things that, colloquially, one would describe as preferences, and exist in an objective sense.
I deny that a generic outside observer would describe us as having any specific set of preferences, in an objective sense.
This doesn't bother me too much, because it's sufficient that we have preferences in a subjective sense - that we can use our own empathy modules and self-reflection to define, to some extent, our preferences.
a brain is ultimately many fewer assumptions (to the pre-industrial Norse people)
"Realistic" preferences make ultimately fewer assumptions (to actual humans) that "fully rational" or other preference sets.
The problem is that this is not true for generic agents, or AIs. We have to get the human empathy module into the AI first - not so it can predict us (it can already do that through other means), but so that its decomposition of our preferences is the same as ours.
Can you make this a bit more general, rather than just for the specific example?
For low bandwidth, you have to specify the set of answers that are available (and how they would be checked).
I tried to answer in more detail here: https://www.lesswrong.com/posts/f5p7AiDkpkqCyBnBL/preferences-as-an-instinctive-stance (hope you didn't mind; I used your comment as a starting point for a major point I wanted to clarify).
But I admit to being confused now, and not understanding what you mean. Preferences don't exist in the territory, so I'm not following you, sorry! :-(
Saying that an agent has a preference/reward R is an interpretation of that agent (similar to the "intentional stance" of seeing it as an agent, rather than a collection of atoms). And the (p,R) and (-p,-R) interpretations are (almost) equally complex.
I disagree. I think that if we put a complexity upper bound on human rationality, and assume noisy rationality, then we will get values that are "meaningless" from your perspective.
I'm trying to think of ways how of we could test this....
Imagine [...] distinguish.
It's because of concerns like this that we have to solve the symbol grounding problem for the human we are trying to model; see, eg, https://www.lesswrong.com/posts/EEPdbtvW8ei9Yi2e8/bridging-syntax-and-semantics-empirically
But that doesn't detract from the main point: that simplicity, on its own, is not sufficient to resolve the issue.
If you assume that human values are simple (low komelgorov complexity) and that human behavior is quite good at fulfilling those values, then you can deduce non trivial values for humans.
And you will deduce them wrong. "Human values are simple" pushes you towards "humans have no preferences", and if by "human behavior is quite good at fulfilling those values" you mean something like noisy rationality, then it will go very wrong, see for example https://www.lesswrong.com/posts/DuPjCTeW9oRZzi27M/bounded-rationality-abounds-in-models-not-explicitly-defined
And if instead you mean a proper accounting of bounded rationality, of the difference between anchoring bias and taste, of the difference between system 1 and system 2, of the whole collection of human biases... well, then, yes, I might agree with you. But that's because you've already put all the hard work in.
I've thinking of a rather naive form of preference utilitarianism, of the sort "if the human agree to it or choose it, then it's ok". In particular, you can end up with some forms of depression where the human is miserable, but isn't willing to change.
I'll clarify that in the post.
I'm finding these "is the correct utility function" hard to parse. Humans have a bit of and a bit of . But we are underdefined systems; there is no specific value of that is "true". We can only assess the quality of using other aspects of human underdefined preferences.
This seems way too handwavy.
It is. Here's an attempt at a more formal definition: humans have collections of underdefined and somewhat contradictory preferences (using preferences in a more general sense than preference utilitarianism). These preferences seem to be stronger in the negative sense than in the positive: humans seem to find the loss of a preference much worse than the gain. And the negative is much more salient, and often much more clearly defined, than that positive.
Given that maximising one preference tends to put the values of others at extreme values, human overall preferences seem better captured by a weighted mix of preferences (or a smooth min of preferences) than by any single preference, or small set of preferences. So it is not a good idea to be too close to the extremes (extremes being places where some preferences have weight put on them).
Now there may be some sense in which these extreme preferences are "correct", according to some formal system. But this formal system must reject the actual preferences of humans today; so I don't see why these preferences should be followed at all, even if they are correct.
Ok, so the extremes are out; how about being very close to the extremes? Here is where it gets wishywashy. We don't have a full theory of human preferences. But, according to the picture I've sketched above, the important thing is that each preference gets some positive traction in our future. So, yes to might no mean much (and smooth min might be better anyway). But I believe I could say:
- There are many weighted combinations of human preferences that are compatible with the picture I've sketched here. Very different outcomes, from the numerical perspective of the different preferences, but all falling within an "acceptability" range.
Still a bit too handwavy. I'll try and improve it again.
https://www.lesswrong.com/posts/hBJCMWELaW6MxinYW/intertheoretic-utility-comparison
Or we could come up with a normalisation method by having people rank the intensity of their preferences versus the intensity of their enjoyments. It doesn't have to be particularly good, just give non-crazy results.
Thanks, corrected a few typos.
Why must a preorder decompose into disjoint ordered chains?
They don't have to; I'm saying that sensible partial preferences (eg ) should do so. I then see how I'd deal with sensible preorders, then generalise to all preorders in the next section.
How do cycles vanish in ? Can you work through the example where the partial preference expressed by the human is ?
Note that what you've written is impossible as means but not . A preorder is transitive, so the best you can get is .
Then projecting down (via ) to will project all these down to the same element. That's why there are no cycles, because all cycles go to points.
Then we need to check some math. Define on by iff .
This is well defined (independently of which and we use to represent and ), because if , then , so, by transitivity, . The same argument works for .
We now want to show the is a partial order on . It's transitive, because if and , then , and the transitivity in implies and hence .
That shows it's a preorder. To show partial order, we need to show there are no cycles. So, if and , then and , hence, by definition of , . So it's a partial order.