Ambient control of the arithmetical hierarchy?
post by Will_Newsome · 2011-12-29T22:25:22.710Z · LW · GW · Legacy · 11 commentsContents
11 comments
Responding to this: http://lesswrong.com/r/discussion/lw/8ys/a_way_of_specifying_utility_functions_for_udt/
I had a similar idea a few months ago that highlights different aspects of the problem which I find confusing. In my version the UDT agent controls bits of Chaitin's constant instead of the universal prior directly, seeing as one of the programs that the oracle (which you can derive from Chaitin's omega) has to solve the halting problem for is the UDT agent's. But since the oracle for the oracle you get from Chaitin's constant depends on the latter oracle's bits, you seem to be able to ambiently control THE ENTIRE ARITHMETICAL HIERARCHY SAY WHAT!? That's the confusing part; isn't your one true oracle supposed to screen you off from higher oracles? Or is that only insofar as you can computably verify?
Anyway I like this theme of controlling computational contexts as it forms a tight loop between agent and environment, something currently lacking. Keep it up comrades!
11 comments
Comments sorted by top scores.
comment by Vladimir_Nesov · 2011-12-30T16:01:02.340Z · LW(p) · GW(p)
I'm confused by this setting, but as a response to the idea of controlling stronger things (this comes up in agent-simulates-predictor): You control things to the extent their dependence on your actions controls your actions. So it's possible to control superintelligences or other things that should know all about you, but only if you can predict the effect of your possible actions on their properties, which you probably won't be able to do unless they let you, and I don't understand how such problems look from the point of view of the agent with epistemic advantage (how do you let a piece of paper with "Defect" written on it control your decision?). It seems like what matters here are possible explanations for weaker agent being a certain way, and it's these (potentially more powerful) explanations that play the game with stronger opponents in your stead (so you engage whoever wrote "Defect" of that piece of paper, Transparent Newcomb-style).
(See how pointless it is to just broadcast poorly-explained and poorly-understood ideas?)
Replies from: Will_Newsome, wedrifid, Will_Newsome↑ comment by Will_Newsome · 2011-12-30T20:51:51.648Z · LW(p) · GW(p)
(See how pointless it is to just broadcast poorly-explained and poorly-understood ideas?)
It actually kinda worked, the confusion I had was really simple and was mostly resolved by User:paulfchristiano saying "I see no problems with that".
how do you let a piece of paper with "Defect" written on it control your decision?
That's a good way of phrasing it, thanks.
it's these (potentially more powerful) explanations that play the game with stronger opponents in your stead
Again, a good way of phrasing it, thanks.
↑ comment by wedrifid · 2011-12-31T00:24:56.425Z · LW(p) · GW(p)
See how pointless it is to just broadcast poorly-explained and poorly-understood ideas?
Slightly clumsy but the point came across nonetheless. The last sentence is the interesting one:
It seems like what matters here are possible explanations for weaker agent being a certain way, and it's these (potentially more powerful) explanations that play the game with stronger opponents in your stead (so you engage whoever wrote "Defect" of that piece of paper, Transparent Newcomb-style).
The earlier sentences seem to be making a simple concept more complicated than it needs to be. Control just isn't that deep when you reduce it.
Replies from: Vladimir_Nesov↑ comment by Vladimir_Nesov · 2011-12-31T00:31:56.931Z · LW(p) · GW(p)
The earlier sentences seem to be making a simple concept more complicated than it needs to be.
Which sentences, how simple should it be?
Control just isn't that deep when you reduce it.
I'm still trying to figure it out.
↑ comment by Will_Newsome · 2011-12-30T22:59:10.779Z · LW(p) · GW(p)
Grr, why did someone downvote this? We should be doing our best to encourage Nesov to talk about things like this, it's the only shot we non-decision-theorists have at understanding the practical implications of Kantian "epistemology" I mean theoretical decision theory! Not much of a shot but still.
comment by paulfchristiano · 2011-12-30T08:29:30.174Z · LW(p) · GW(p)
It seems like each level of the hierarchy should influence higher levels directly and in a bunch of ways, so I don't quite understand. Downward influence is more confusing, and also apparently more relevant to current problems (though only by analogy).
Replies from: Will_Newsome↑ comment by Will_Newsome · 2011-12-30T20:54:13.016Z · LW(p) · GW(p)
That's really it, I just felt (feel?) that there's something wrong with being able to control higher oracles than yours, but I guess if you're using hypercomputation to do so strategically then that's not within the intended scope of the formalism and it's not at all surprising that it doesn't say anything useful.
comment by Solvent · 2011-12-30T10:55:23.904Z · LW(p) · GW(p)
I don't know what this means. Can some clever person independently discern whether this is meaningless, or if I just don't understand it? (Or both?)
Replies from: cousin_it↑ comment by cousin_it · 2012-01-01T11:35:44.510Z · LW(p) · GW(p)
The post isn't completely meaningless, it's just the sort of mental scratchwork you generate when you're kicking around a new idea. Let me try to explain.
If you look at decision-making from the point of view of the AI doing the decision-making, you'll notice that some mathematical facts that are supposed to be "fixed" (like the return value of your algorithm) become kind of vague and "controllable". The deterministic operation of the AI can make mathematical facts come out a certain way which maximizes the AI's utility. For example, an AI could have a tiny bit of control over the probabilities of certain bitstrings under the universal prior (which is after all a mixture of all possible programs including the AI), or the bits of Chaitin's omega, or the truth values of certain statements in the arithmetical hierarchy. That last part freaked Will out, then Paul came along and said there's nothing to worry about, exerting control over math is business as usual.