Posts
Comments
the order-dimension of its preference graph is not 1 / it passes up certain gains
If the order dimension is 1, then the graph is a total order, right? Why the conceptual detour here?
Loosely related to this, it would be nice to know if systems which reliably don't turn down 'free money' must necessarily have almost-magical levels of internal coordination or centralization. If the only things which can't (be tricked into) turn(ing) down free money when the next T seconds of trade offers are known are Matrioshka brains at most T light-seconds wide, does that tell us anything useful about the limits of that facet of dominance as a measure of agency?
I'm not convinced that the specifics of "why" someone might consider themselves a plural smeared across a multiverse are irrelevant. MWI and the dynamics of evolving amplitude are a straightforward implication of the foundational math of a highly predictive theory, whereas the different flavors of classical multiverse are a bit harder to justify as "likely to be real", and also harder to be confident about any implications.
If I do the electron-spin thing I can be fairly confident of the future existence of a thing-which-claims-to-be-me experiencing both outcomes as well as my relative likelihood of "becoming" each one, but if I'm in a classical multiverse doing a coin flip then perhaps my future experiences are contingent on whether the Boltzmann-brain-emulator running on the grand Kolmogorov-brute-forcing hypercomputer is biased against tails (that's not to say I can make use of any of that to make a better prediction about the coin, but it does mean upon seeing heads that I can conclude approximately nothing about any "me"s running around that saw tails).
If I push the classical uncertainty into the past by, say, shaking a box with the coin inside and sticking it in a storage locker and waiting a year (or seeding a PRNG a year ago and consulting that) then even though the initial event might have branched nicely, right now that cluster of sufficiently-similar Everett branches are facing the same situation in the original question, right? Assuming enough chaotic time has passed that the various branches from the original random event aren't using that randomness for the same reason.
I understand from things like this that it doesn't take a lot of (classical) uncertainty or a lot of time for a system to become unpredictable at scale, but for me that pushes the question down to annoying concrete follow-ups like:
- My brain and arm muscles have thermal noise, but they must be somewhat resilient to noise, so how long does it take for noise at one scale (e.g. ATP in a given neuron) to be observable at another scale (e.g. which word I say, what thought I have, how my arm muscle moves)?
- More generally, how effective are "noise control" mechanisms like reducing degrees of freedom? E.g. while I can imagine there's enough chaos around a coin flip for quantum noise to affect thermal noise to affect macro outcomes, it's not as obvious to me that that's true for a spinner in a board game where the main (only?) relevant macro parameter affected by me is angular momentum of the spinner.
For me the only "obvious" takeaway from this re. quantum immortality is that you should be more willing to play quantum Russian roulette than classical Russian roulette. Beyond that, the topic seems like something where you could get insights by just Sitting Down and Doing The Math, but I'm not good enough at math to do the math.
...wait, you were just asking for an example of an agent being "incoherent but not dominated" in those two senses of being money-pumped? And this is an exercise meant to hint that such "incoherent" agents are always dominatable?
I continue to not see the problem, because the obvious examples don't work. If I have as incomparable to that doesn't mean I turn down the trade of (which I assume is what you're hinting at re. foregoing free money).
If one then says "ah but if I offer $9999 and you turn that down, then we have identified your secret equivalent utili-" no, this is just a bid/ask spread, and I'm pretty sure plenty of ink has been spilled justifying EUM agents using uncertainty to price inaction like this.
What's an example of a non-EUM agent turning down free money which doesn't just reduce to comparing against an EUM with reckless preferences/a low price of uncertainty?
Hmm, I was going to reply with something like "money-pumps don't just say something about adversarial environments, they also say something about avoiding leaking resources" (e.g. if you have circular preferences between proximity to apples, bananas, and carrots, then if you encounter all three of them in a single room you might get trapped walking between them forever) but that's also begging your original question - we can always just update to enjoy leaking resources, transmuting a "leak" into an "expenditure".
Another frame here is that if you make/encounter an agent, and that agent self-modifies into/starts off as something which is happy to leak pretty fundamental resources like time and energy and material-under-control, then you're not as worried about it? It's certainly not competing as strongly for the same resources as you whenever it's "under the influence" of its circular preferences.
(I'm not EJT, but for what it's worth:)
I find the money-pumping arguments compelling not as normative arguments about what preferences are "allowed", but as engineering/security/survival arguments about what properties of preferences are necessary for them to be stable against an adversarial environment (which is distinct from what properties are sufficient for them to be stable, and possibly distinct from questions of self-modification).
The rock doesn't seem like a useful example here. The rock is "incoherent and not dominated" if you view it as having no preferences and hence never acting out of indifference, it's "coherent and not dominated" if you view it as having a constant utility function and hence never acting out of indifference, OK, I guess the rock is just a fancy Rorschach test.
IIUC a prototypical Slightly Complicated utility-maximizing agent is one with, say, , and a prototypical Slightly Complicated not-obviously-pumpable non-utility-maximizing agent is one with, say, the partial order plus the path-dependent rule that EJT talks about in the post (Ah yes, non-pumpable non-EU agents might have higher complexity! Is that relevant to the point you're making?).
What's the competitive advantage of the EU agent? If I put them both in a sandbox universe and crank up their intelligence, how does the EU agent eat the non-EU agent? How confident are you that that is what must occur?
This is pretty unsatisfying as an expansion of "incoherent yet not dominated" given that it just uses the phrase "not coherent" instead.
I find money-pump arguments to be the most compelling ones since they're essentially tiny selection theorems for agents in adversarial environments, and we've got an example in the post of (the skeleton of) a proof that a lack-of-total-preferences doesn't immediately lead to you being pumped. Perhaps there's a more sophisticated argument that Actually No, You Still Get Pumped but I don't think I've seen one in the comments here yet.
If there are things which cannot-be-money-pumped, and yet which are not utility-maximizers, and problems like corrigibility are almost certainly unsolvable for utility-maximizers, perhaps it's somewhat worth looking at coherent non-pumpable non-EU agents?
Say more about what counts as incoherent yet not dominated? I assume "incoherent" is not being used here as an alias for "non-EU-maximizing" because then this whole discussion is circular.
Here's a very late follow-up: the rationale behind linearity for Shapley values seems closely related to the rationale behind the independence axiom of VNM rationality, and under some decision theories we apparently can dispense with the latter.
This gives me the vocabulary for expressing why I find linearity constraining: if I'm about to play game or game with probabilities and respectively, and my payout of is lower, maybe I would prefer to get a lower payout in in exchange for a higher payout in . I'm not sure how much of that is just downstream from "what if my utility isn't linear in the payout" or something like that, though.
...huh. So UDT in general gets to just ignore the independence axiom because:
- UDT's whole shtick is credibly pre-committing to seemingly bad choices in some worlds in order to get good outcomes in others, and/or
- UDT is optimizing over policies rather than actions, and I guess there's nothing stopping us having preferences over properties of the policy like fairness (instead of only ordering policies by their "ground level" outcomes).
- And this is where comes in, it's one way of encoding something-like-fairness.
Sound about right?
I'm confused by the "no dutch book" argument. Pre-California-lottery-resolution, we've got , but post-California-lottery-resolution we simultaneously still have and "we refuse any offer to switch from to ", which makes me very uncertain what means here.
Is this just EDT vs UDT again, or is the post-lottery subtly distinct from the pre-lottery one, or is "if you see yourself about to be dutch-booked, just suck it up and be sad" a generally accepted solution to otherwise being DB'd, or something else?
But if your utility function is bounded, as it apparently should be then you're one affine transform away from being able to use geometric rationality, no?
I agree my "fix" is insufficient - in fact I'd go so far as agreeing with JBlack below that including it was net negative to the question.
I'd like to pin down what you mean by your description of a more complete model, I hope you don't mind.
Let me flesh out the restaurant story. The actors are (me) and (my friend). The restaurants are and . There are two events we care about: the first is me and my friend choosing the lottery parameter , and the second is actually running the lottery.
After picking but before the lottery, my friend and I have (for simplicity) fixed costs and outcome-dependent utilities and . Our expected utilities are indeed exactly what you'd expect: . Is this what you mean by eventually projecting to a straight line?
The "standard model"/convex hull isn't describing the space of outcomes of the lottery, it's describing the space of lotteries by summarizing them as expected utilities. However, as varies can draw any number of weird and wonderful (and as you say, sharp and discontinuous) shapes. Once we fix and hence , we get a specific shape/space of (expected) lottery outcomes. Is this what you mean by having the utility function be "all inclusive"?
Now that we've got a nice, fixed set of outcomes with an associated utility per outcome, we can take a hyperprior over to get a distribution over that space of outcomes, and we're back in standard-utility-theory land.
I think I've identified my confusion: we should distinguish between the distribution choice parameterized by , and the prior distribution over expected outcomes which we can get with a distribution over . If we were playing a game where we made a choice about that distribution over , we'd have the same problem: our utilities could depend on the prior and so the outcome space would again be an arbitrary shape.
So, summary: it's invalid, as a design choice when formulating e.g. a bargaining solution or a game equilibrium, to do the following:
- Start from a space of outcomes.
- Say "and now the players choose a distribution over the outcomes".
- Conclude "our new space of outcomes is the convex hull of the old space of outcomes".
Does that sound right?
Fair point re. focusing on a specific formula, I'll remove that from the post.
Hmm, I'm not sure what I should be taking away from that. You've pointed out that the morning and evening lotteries are materially different, but that's not contentious to me: if uncertainty has costs then those costs have to show up as differences in the world compared to a world without that uncertainty.
I guess the restaurant story failed to focus on the-bit-that's-weird-to-me, which is that if my friend and I were negotiating over the lottery parameter , then my mental model of the expected utility boundary as varies is not a straight line.
To be explicit, the "standard model" of my friend and I having a lottery looks like this, whereas once you account for the costs of increasing uncertainty when is away from or it ends up looking like this.
Huh, here's what looks like a survey of variations on the Shapley Value - I'll take a look!
I wasn't asking "what payment rules still satisfy the three remaining properties", I was asking "what other payment rules are there which satisfy the three remaining properties but not additivity" (with bonus questions "what other properties of Shapley values do we still get just from those three properties" and "what properties other than additivity can we add to those three properties which again pin down a unique rule").
My aim here, which I admit is nebulous, is to get a rough overview of the space of different payment rules (for example, this answer on math.stackexchange namedrops the 'pre-kernel' and 'pre-nucleolus' - I assume there's more where that came from!).
Ideally, and I know this is a cartoonish and unrealistic goal, I'd have:
- A list of "Properties Which Are Nice To Have In A Payment Rule",
- A list of "Sets of Properties Which Imply This Other Property",
- And a list of "Sets of Properties Which Specify A Unique Payment Rule".
I just found a presentation of linearity which motivates it as preserving expected payout before and after an uncertain event, which both adds usefulness-points to the property (for me) and vaguely suggests where you might not want that property, but no concrete example comes to mind.
Yep, that's pretty much it, but with the added bonus of a concrete motivating example. Thanks!