Posts
Comments
That's annoying - thanks for pointing it out. Any idea what the issue is?
I don't quite see why the causality is this flexible and arbitrary.
In stories and movies, people often find that the key tool/skill/knowledge they need to solve the problem, is something minor they picked up some time before.
The world could work like this, so that every minor thing you spent any time on would have a payoff at some point in the future. Call this a teleological world.
This world would have a different "causal" structure to our own, and we'd probably not conceive traditional CDT agents as likely in this world.
I'm claiming that this post is conflating an error in constructing an accurate world-map with an error in the decision theory.
The problem is not that CDT has an inaccurate world-map; the problem is that CDT has an accurate world map, and then breaks it. CDT would work much better with an inaccurate world-map, one in which its decision causally affects the prediction.
See this post for how you can hack that: https://www.lesswrong.com/posts/9m2fzjNSJmd3yxxKG/acdt-a-hack-y-acausal-decision-theory
You may like this, then: https://www.lesswrong.com/posts/9m2fzjNSJmd3yxxKG/acdt-a-hack-y-acausal-decision-theory
I'm using CDT as it's formally stated (in, eg, the FDT paper).
The best defence I can imagine from a CDT proponent: CDT is decision theory, not game theory. Anything involving predictors is game theory, so doesn't count.
CDT would fight the hypothetical, and refuse to admit that perfect predictors of their own actions exist (the CDT agent is perfectly fine with perfect predictors of other people's actions).
I think normalisation doesn't fit in the convex-concave picture. Normalisation is to avoid things like being seen as the same as .
I like that way of seeing it.
Ok, have corrected it now; the negative-sum formulation was wrong, sorry.
You are correct; I was unclear (and wrong in that terminology). I will rework the post slightly.
Does this reflect how much you want to watch each of the movies?
That's one of the features I considered.
I don't consider it a core Star Wars movie; if I had, it would be just below revenge of the Sith.
"I want to be a more generous person": what would you classify that as? Or "I want to want to write"?
Negative vs positive vs zero sum is all relative to what we take to be the default outcome.
I take the default as "no effort is made to increase or decrease any of the reward functions".
Negative sum vs zero sum (vs positive sum, in fact) depend on defining some "default state", against which the outcome is compared. A negative sum game can become a positive sum game if you just give all the "players" a fixed bonus (ie translate the default state). Default states are somewhat tricky and often subjective to define.
Now, you said "the best states for one of the rewards are bad for the other". "Bad" compared with what? I'm taking as a default something like "you make no effort to increase (or decrease) either reward".
So, my informal definition of "zero sum" is "you may choose to increase either or (roughly) independently of each other, from a fixed budget". Weakly positive sum would be "the more you increase , the easier it gets to increase (and vice versa) from a fixed budget"; strongly positive sum would be "the more you increase , the more increases (and vice versa)".
Negative sum would be the opposite of this ("easier"->"harder" and "increases"->"decreases").
The reason I distinguish weak and strong, is that if we add diminishing returns, this reduces the impact of weak negative sum, but can't solve strong negative sum.
Does this help, or add more confusion?
I saw the 's as normalised to 1 or zero, and the coefficients as giving them weights. So instead of , I'd write (given the behaviour and assumptions).
But this is an old post, and is mainly superseded by new ones, so I wouldn't spend too much time on it.
Does this imply I prefer X apples to Y bananas and Z pears, where Y+Z=X?
If it's just for a single fruit, I'd decompose that preference into two separate ones? Apple vs Banana, Apple vs Pear.
Generally, spending resources is zero-sum, not negative sum.
Nope, wasn't consciously echoing that ^_^
Renormalise those, so that slowly diminishing returns don't dominate.
Flo's summary for the Alignment Newsletter:
I like that summary!
I enjoyed this article and the proposed factors match my intuitions. Predicting variable diminishing returns seems especially hard to me. I also worry that the interactions between rewards will be negative-sum, due to resource constraints.
Resource constraints situations can be positive sum (consider most of the economy). The real problem is between antagonistic preferences, eg maximising flourishing lives vs negative utilitarianism, where a win for one is a loss for the other.
Note that this post considers the setting where we have uncertainty over the true reward function, but we can't learn about the true reward function. If you can gather information about the true reward function, which <@seems necessary to me@>(@Human-AI Interaction@), then it is almost always worse to take the most likely reward or expected reward as a proxy reward to optimize.
Yes, if you're in a learning process and treat it as if you weren't in a learning process, things will go wrong ^_^
That's actually the fourth Mercury, rebuilt for tourism.
Happy New Year!
No idea, sorry - I got my copies printed locally.
I delve into that here: https://www.lesswrong.com/posts/Lb3xCRW9usoXJy9M2/platonic-rewards-reward-features-and-rewards-as-information#Extending_the_problem
(sorry, have now corrected link to https://www.lesswrong.com/posts/6XCTppoPAMdKCPFb4/oracles-reject-all-deals-break-superrationality-with-1 )
Apologies, have now corrected the link.
A possible counter: https://www.lesswrong.com/posts/6XCTppoPAMdKCPFb4/oracles-reject-all-deals-break-superrationality-with-1
A possible counter: https://www.lesswrong.com/posts/6XCTppoPAMdKCPFb4/oracles-reject-all-deals-break-superrationality-with-1
And a possible counter: https://www.lesswrong.com/posts/6XCTppoPAMdKCPFb4/oracles-reject-all-deals-break-superrationality-with-1
Hum - my approach here seems to have a similarity to your idea.
One possible counter: https://www.lesswrong.com/posts/6XCTppoPAMdKCPFb4/oracles-reject-all-deals-break-superrationality-with-1
I have to say, I find these criticisms a bit weak. Going through them:
III. FDT sometimes makes bizarre recommendations
I'd note that successfully navigating Parfit's hitchhiker also involve violating "Guaranteed Payoffs": you pay the driver at a time when there is no uncertainty, and where you get better utility from not doing so. So I don't think Guaranteed Payoffs is that sound a principle.
Your bomb example is a bit underdefined, since the predictor is predicting your actions AND giving you the prediction. If the predictor is simulating you and asking "would you go left after reading a prediction that you are going right", then you should go left; because, by the probabilities in the setup, you are almost certainly a simulation (this is kind of a "counterfactual Parfit hitchhiker" situation).
If the predictor doesn't simulate you, and you KNOW they said to go right, you are in a slightly different situation, and you should go right. This is akin to waking up in the middle of the Parfit hitchhiker experiment, when the driver has already decided to save you, and deciding whether to pay them.
IV. FDT fails to get the answer Y&S want in most instances of the core example that’s supposed to motivate it
This section is incorrect, I think. In this variant, the contents of the boxes are determined not by your decision algorithm, but by your nationality. And of course two-boxing is the right decision in that situation!
the case for one-boxing in Newcomb’s problem didn’t seem to stem from whether the Predictor was running a simulation of me, or just using some other way to predict what I’d do.
But it does depend on things like this. There's no point in one-boxing unless your one-boxing is connected with the predictor believing that you'd one-box. In a simulation, that's the case; in some other situations where the predictor looks at your algorithm, that's also the case. But if the predictor is predicting based on nationality, then you can freely two-box without changing the predictor's prediction.
V. Implausible discontinuities
There's nothing implausible about discontinuity in the optimal policy, even if the underlying data is continuous. If is the probability that we're in a smoking lesion vs a Newcomb problem, then as changes from to , the expected utility of one-boxing falls and the expected utility of two-boxing rises. At some point, the optimal action will jump discontinuously from one to the other.
VI. FDT is deeply indeterminate
I agree FDT is indeterminate, but I don't agree with your example. Your two calculators are clearly isomorphic, just as if we used a different numbering system for one versus the other. Talking about isomorphic algorithms avoids worrying about whether they're the "same" algorithm.
And in general, it seems to me, there’s no fact of the matter about which algorithm a physical process is implementing in the absence of a particular interpretation of the inputs and outputs of that physical process.
Indeed. But since you and your simulation are isomorphic, you can look at what the consequences are of you outputting "two-box" while your simulation outputs "deux boites" (or "one-box" and "une boite"). And {one-box, une boite} is better than {two-box, deux boites}.
But why did I use those particular interpretations of me and my simulation's physical processes? Because those interpretations are the ones relevant to the problem at hand. Me and my simulation will have a different weight, consume different amounts of power, are run at different times, and probably at different speeds. If those were relevant to the Newcomb problem, then the fact we are different becomes relevant. But since they aren't, we can focus in on the core of the matter. (you can also consider the example of playing the prisoner's dilemma against an almost-but-not-quite-identical copy of yourself).
Excellent and well worked on, suggesting many different interesting ideas and research avenues.
Very thorough, and it's very worthwhile that posts like this are made.
It's helped me hone my thinking on what is and isn't an optimiser (and a wireheader, and so on, for associated concepts).
Was that unexpected?
I don't think that would work, but it's worth thinking about in case it does...
Cheers!
This might be relevant: https://www.lesswrong.com/posts/5bd75cc58225bf0670375414/acausal-trade-double-decrease
Well, that message only works if it actually produces an UFAI within the required timespan, and if the other Oracle would have its message not read. There are problems, but the probability is not too high, initially (though this depends on the number of significant figures in its message).
A top level post here: https://www.lesswrong.com/posts/42z4k8Co5BuHMBvER/hyperrationality-and-acausal-trade-break-oracles
You might want to quote this as well: https://www.lesswrong.com/posts/42z4k8Co5BuHMBvER/hyperrationality-and-acausal-trade-break-oracles
I'm actually willing to believe that methods used for cat pictures might work for human theory of mind - if trained on that data (and this doesn't solve the underdefined problem).
Yep. If they do acausal trade with each other.
Some thoughts on that idea: https://www.lesswrong.com/posts/6WbLRLdmTL4JxxvCq/analysing-dangerous-messages-from-future-ufai-via-oracles
Some thoughts on this idea, thanks for it: https://www.lesswrong.com/posts/6WbLRLdmTL4JxxvCq/analysing-dangerous-messages-from-future-ufai-via-oracles
I consider wireheading to be a special case of proxy alignment in a mesaoptimiser.
I agree. I've now added this line, which I thought I'd put in the original post, but apparently missed out:
Note, though, that the converse is true: every example of wireheading is a Goodhart curse.
But really, what's the purpose of trying to distinguish wireheading from other forms of reward hacking?
Because mitigations for different failure modes might not be the same, depending on the circumstances.
Where "measurement channel" not just one specific channel, but anything that has the properties of a measurement channel.
Indeed. It might be possible to construct that complex bias function, from the policy, in a simple way. But that claim needs to be supported, and the fact that it hasn't been found so far (I repeat that it has to be simple) is evidence against it.