Intertheoretic utility comparison: examples
2019-07-17T12:39:45.147Z · score: 9 (1 votes)but if we have a solution that fixes the way they exploit interruption
? Doesn't the design above do that?
The human might have some taste preferences that will determine between tea and coffee, general hedonism preferences that might also work, and meta-preferences about how they should deal with future choices.
Part of the research agenda - "grounding symbols" - about trying to determine where these models are located.
I don't think this is a Goodhart-style effect. Standard indifference is a very carefully constructed effect, and it does exactly what it is designed for: making the agents indifferent to their individual interruptions. It turns out this doesn't make them indifferent to the interruptions of other agents, which is annoying but not really surprising.
It's not Goodhart, it's just that mutual indifference has to be specifically designed for.
Thanks! For the M1 vs M2, I agree these could reach different outcomes - but would either one be dramatically wrong? There are many "free variables" in the process, aiming to be ok.
I'll work on learning partial preferences.
"Just brink me tee, without killing my cat and tilling universe with teapots." [...] and underfdefined – at least based on my self-observation. Thus again I would prefer collectively codified human norm (laws) over extrapolated model of my utility function.
It might be underdefined in some sort of general sense - I understand the feeling, I sometimes get it too. But in practice, it seems like it should ground out to "obey human orders about tea, or do something that is strongly preferred to that by the human". Humans like their orders being obeyed, and presumably like getting what they're ordering for; so to disobey that, you'd need to be very sure that there's a clearly better option for the human.
Of course, it might end up having a sexy server serve pleasantly drugged tea ^_^
Hey there!
Wondering how you felt my research agenda addressed, or failed to address, many of these points: https://www.lesswrong.com/posts/CSEdLLEkap2pubjof/research-agenda-v0-9-synthesising-a-human-s-preferences-into
I have my own opinions on these, but interested in yours.
Indifference: multiple changes, multiple agents
2019-07-08T13:36:42.095Z · score: 16 (3 votes)Please add "submission" at the top of the post.
-
is insufficiently detailed - can you explain what is going on, how the Oracles are rewarded, what happens when the message is read/not read, and so on. Same for 5.
-
seems potentially very interesting.
Thanks!
"Can't generally be used"; if you understand the setup and are careful, you might be able to do so.
Can you develop this model in more detail?
Thanks! Seems potentially promising. Will analyse this more properly later.
Here is assumed that if the proof is true and is in a formal language, there is no hidden messages in the text.
That is never something safe to assume. I can write formally correct proofs that contain hidden messages quite easily - add extra lemmas and extra steps. Unless we're very smart, it would be hard for us to detect which steps are unnecessary and which are needed, especially if it rewrites the main proof thread somewhat.
Another way to check proofs is to run two slightly different non-communicating Oracles and compare outputs.
I'll accept that as a part of a submission if a) you develop it more, in a formal way, and b) you repost it as a top level comment.
This is against the technical definition of low bandwidth (small space of possible solutions), but somewhat in the spirit (low bandwidth for solutions humans get to see). I'll accept it as a submission.
Of course this has the problem of maximizing for apparent insight rather than actual insight.
Until we can measure actual insight, this will remain a problem ^_^
The ideal solution would have huge positive impacts and complete safety, under minimal assumptions. More realistically, there will be a tradeoff between assumptions and impact.
I'm not suggesting any area for people to focus their efforts, because a very effective approach with minimal assumptions might win, or a fantastically effective approach under stronger assumptions. It's hard to tell in advance what will be the most useful.
See the edit, and make sure you "decide on the length of each episode, and how the outcome is calculated. The Oracle is run once an episode only (and other Oracles can't generally be used on the same problem; if you want to run multiple Oracles, you have to justify why this would work), and has to get objective/loss/reward by the end of that episode, which therefore has to be estimated in some way at that point."
See the edit, and make sure you "decide on the length of each episode, and how the outcome is calculated. The Oracle is run once an episode only (and other Oracles can't generally be used on the same problem; if you want to run multiple Oracles, you have to justify why this would work), and has to get objective/loss/reward by the end of that episode, which therefore has to be estimated in some way at that point."
See the edit (especially for your first suggestion): "decide on the length of each episode, and how the outcome is calculated. The Oracle is run once an episode only (and other Oracles can't generally be used on the same problem; if you want to run multiple Oracles, you have to justify why this would work), and has to get objective/loss/reward by the end of that episode, which therefore has to be estimated in some way at that point."
See the edit: "decide on the length of each episode, and how the outcome is calculated. The Oracle is run once an episode only (and other Oracles can't generally be used on the same problem; if you want to run multiple Oracles, you have to justify why this would work), and has to get objective/loss/reward by the end of that episode, which therefore has to be estimated in some way at that point."
Corrected, thanks!
For the low bandwidth Oracle, you need to give it the options. In the case of the counterfactual Oracle, if you don't see the list, how do you reward it?
Assume either way, depending on what your suggestion is for.
Yes.
Also also, have we considered that we're selecting for deception, if we're looking for it and terminating AIs we find deceptive, while nurturing those we don't detect?
Yes. That's a general problem (see the footnote above for a variant of it).
Do you think that AI system doesn't "know" what humans would want, even if it doesn't optimize for it?
I think the AI would not know that, because "what humans would want" is not defined. "What humans say they want", "what, upon reflection, humans would agree they want...", etc can be done, but "what humans want" is not a defined things about the world or about humans - without extra assumptions (which cannot be deduced from observation).
None of these questions can be asked to the low bandwidth Oracle (you need a list of answers); it might be possible to ask them to the counterfactual Oracle, after some modification, but they would be highly dangerous if you allow unrestricted outputs.
Contest: $1,000 for good questions to ask to an Oracle AI
2019-07-01T14:39:25.170Z · score: 51 (20 votes)Here are some submission examples (note I'm not saying they are good examples, just well formatted; the Edits in the third submission are deliberate):
- "Submission. For the counterfactual Oracle, ask the Oracle what Google's stock price will be next month (counterfactually if we didn't see the Oracle's answer). In that case, loss function computed as ||predicted price - actual price after erasure||^2. If we don't see the answer, the programmers are assumed to not ask the question again for a month, neither to this Oracle nor to any other. This will demonstrate the true value of Google, and can ultimately be used to remove noise from the stock market.
- "Submission: low-bandwidth oracle. Give it a list of a thousand companies, and ask which one will most increase in value in percentage terms over the week. At the end of the month, rank the companies by percentage increase. Loss function is rank of the company the oracle selected. Programmer will try to invest in selected company, but will do it discreetly. This will help to gather resources for AI safety research.
- "Submission. Ask the low bandwidth Oracle which of my friends will surprise me most this fortnight. It choose from a list of friends; I'll decide which one surprises me most. Loss function is if it choose the wrong friend, if it choose the right one. This will help me figure out myself and my social circle, and better focus on AI safety. The risk is low because none of my friends are particularly important, positively or negatively, to the world. EDIT: To be clear, I also want to use this to figure out what the word "surprise" means to me, and what the AI predicts it will mean to me. EDIT 2: People have pointed out that it might be dangerous to have the AI construct my own meanings for categories, but it only has three bits or so of optimisation power (I don't have that many friends :-(, so it's mainly me thinking this through, not the AI manipulating me.
so all we need to do is ensure that it is incentivized to figure out and satisfy our preferences, and then it will do the rest.
That's actually what I'm aiming at with the research agenda, but the Occam's razor argument shows that this itself is highly non-trivial, and we need some strong grounding of the definition of preference.
I should also note there's a difference between competence rewarded, and incompetence punished. I suspect the second happens a lot more than the first.
I should also note there's a difference between competence rewarded, and incompetence punished. I suspect the second happens a lot more than the first.
Self-confirming prophecies, and simplified Oracle designs
2019-06-28T09:57:35.571Z · score: 6 (3 votes)How sure are you that hunter-gatherers are much closer to the edge than the typical person in our society?
Very sure; compare death rates.
A better comparison might be people in cold / food-scarce vs warm / food-abundant areas.
Surely abundance of food is relative to population size?
Maybe we could try and estimate how objectively hard it is for certain groups to survive, and then try and work back to reward for competence from that?
Disasters do promote more pro-social behaviours, and more personal, non-bureaucratic interactions between people (it also promotes more people "acting cooperatively", but actual cooperation seems much more efficient with laws and markets than with good will). But it's precisely that that also promotes corruption and nepotism. Running more informal organisation is almost all about personal politics. And cooperation within the tribe often goes along with antagonism towards the outside.
See anti-social punishment here: https://www.lesswrong.com/posts/X5RyaEDHNq5qutSHK/anti-social-punishment - cooperators were punished for cooperating!
Charm means the person has allies, and would be dangerous to cross. Competence might mean that they are likely to make a power play (see prestige vs dominance hierarchies).
There are hunter gatherers who punish those of low status who give away too much meat. Because these people are obviously trying to wow people and make a play at raising their status.
Just my impression based on discussing the issue with some moral realists/non-realists.
Yep ^_^ I make those points in the research agenda (section 3).
To be precise: I argue low impact is intractable without learning a subset of human values; the full set is not needed.
Apocalypse, corrupted
2019-06-26T13:46:05.548Z · score: 19 (11 votes)Research Agenda in reverse: what *would* a solution look like?
2019-06-25T13:52:48.934Z · score: 36 (14 votes)Yep. But I note that many people seem to value letting their values drift somewhat, so that needs to be taken into account.
What I meant was this: assume that 3^^^3 dust specks on one person is worse that 50 years of torture. As long as the dust specks sensation is somewhat additive, that should be true. Now suppose you have to choose between dust specks and torture 3^^^3 times, one for each person ("so, do we torture individual 27602, or one dust speck on everyone? Now, same question for 27602....).
Then always choosing dust specks is worse, for everyone, than always choosing torture.
So the dust-speck decision becomes worse and worse, the more often you expect to encounter it.
"humans don't have actual preferences so the AI is just going to try to learn something adequate."
Try something like: humans don't have actual consistent preferences, so the AI is going to try and find a good approximation that covers all the contradictions and uncertainties in human preferences.
Information can (and should) change your behaviour, even if it doesn't change your values. Becoming a parent should change your attitude to various things whose purpose you didn't see till then! And values can prefer a variety of experiences, if we cash our boredom properly.
The problem is that humans mix information and values together in highly complicated, non-rational ways.
I'd say that this problem doesn't belong in section 2.3-2.4 (collecting and generalising preferences), but in section 1.2 (symbol grounding, and especially the web of connotations). That's where these questions should be solved, in my view.
So yeah, I agree that standard machine learning is not up to the task yet, at all.
(as a minor aside, I'm also a bit unsure how necessary it is to make partial preferences total before combining them; this may be unnecessary)
Do these seem like things that could be "put in as a strong conditional meta-preference" in your framework?
Yes, very easily.
The main issue is whether these should count as an overwhelming meta-preference - one that over-weights all other considerations. And, currently as I have things set up, the answer is no. I have no doubt that you feel strongly about potentially true moral realism. But I'm certain that this strong feeling is not absurdly strong compared to other preferences at other moments in your life. So if we synthesised your current preferences, and 1. or 2. ended up being true, then the moral realism would end up playing a large-but-not-dominating role in your moral preferences.
I wouldn't want to change that, because what I'm aiming for is an accurate synthesis of your current preferences, and your current preference for moral-realism-if-it's-true is not, in practice, dominating your preferences. If you wanted to ensure the potential dominance of moral realism, you'd have to put that directly into the synthesis process, as a global meta-preference (section 2.8 of the research agenda).
But the whole discussion feels a bit peculiar, to me. One property of moral realism that is often assumed, is that it is, in some sense, ultimately convincing - that all systems of morality (or all systems derived from humans) will converge to it. Yet when I said a "large-but-not-dominating role in your moral preferences", I'm positing that moral realism is true, but that we have a system of morality - - that does not converge to it. I'm not really grasping how this could be possible (you could argue that the moral realism is some sort of acausal trade convergent function, but that gives an instrumental reason to follow , not an actual reason to have ; and I know that a moral system need not be a utility function ^_^).
So yes, I'm a bit confused by true-but-not-convincing moral realisms.
This differs, because the z are assumed to be in a "standard" range. There are situations where extreme values of z, if known and reflected upon, would change the sign of the decision (for example, what if your decision is being filmed, and there are billions being bet upon your ultimate choice, by various moral and immoral groups?).
But yeah, if you assume that the z are in that standard range, then this looks a lot like considering just a few nodes of a causal net.
One of the reasons I refer to synthesising (or constructing) the , not learning it.
Post was good, but I'd recommend adding an introductory paragraph to the link on LessWrong.
What scares me is the possibility that moral anti-realism is false, but we build an AI under the assumption that it's true
One way of dealing with this, in part, is to figure out what would convince you that moral realism was true, and put that in as a strong conditional meta-preference.
Valid point, though conditional meta-preferences are things I've already written about, and the issue of being wrong now about what your own preferences would be in the future, is also something I've addressed multiple times in different forms. Your example is particularly crisp, though.
I was talking about "extended family values" in the sense of "it is good for families to stick together and spend time with each other"; this preference can (and often does) apply to other families as well. I see no analogue for that with slavery.
But yeah, you could argue that racism can be a terminal value, and that slave owners would develop it, as a justification for what might have started as an instrumental value.
Research Agenda v0.9: Synthesising a human's preferences into a utility function
2019-06-17T17:46:39.317Z · score: 51 (13 votes)I see the orange-apple preference reversal as another example of conditional preferences.
Preference conditional on circumstances and past preference satisfaction
2019-06-17T15:30:32.580Z · score: 11 (2 votes)Yep, basically that. ^_^
I realize that you did explicitly define "degeneration = moving away from ours" for the drifting values, but it feels weird to then also define "progress = moving away from ours in a good way"
If I decomposed a bit more, I'd say that we need to distinguish the values of others, and the state of the world, and whether things are moving towards our values, away from our values, or just drifting.
So "progress", in the sense of my post, is composed of a) other people's values moving towards our own, and b) the state of the world moving more towards our own preferences/values. "Moral degeneration", on the other hand, is c) people's values drift away from our own.
I see all three of these happening at once (along with, to some extent "the state of the world moving away from our values", which is another category), so that's why we see both progress and degeneration in the future.