Comment by wei_dai on Towards formalizing universality · 2019-01-17T11:07:29.898Z · score: 5 (2 votes) · LW · GW

I think I don’t quite understand what you are saying here, what exactly is obvious?

I think I expressed myself badly there. What I mean is that it seems a sensible default to not trust an impoverished perspective relative to oneself, and you haven't stated a reason why we should trust the impoverished perspective. This seems to be at least a big chunk of the formalization of universality that you haven't sketched out yet.

Comment by wei_dai on CDT=EDT=UDT · 2019-01-17T08:07:25.780Z · score: 8 (3 votes) · LW · GW

Thanks, I think I understand now, and made some observations about EDT+SSA at the old thread. At this point I'd say this quote from the OP is clearly wrong:

So, we could say that CDT+SIA = EDT+SSA = UDT1.0; or, CDT=EDT=UDT for short.

In fact UDT1.0 > EDT+SSA > CDT+SIA, because CDT+SIA is not even able to coordinate agents making the same observation, while EDT+SSA can do that but not coordinate agents making different observations, and UDT1.0 can (probably) coordinate agents making different observations (but seemingly at least some of them require UDT1.1 to coordinate).

Comment by wei_dai on In memoryless Cartesian environments, every UDT policy is a CDT+SIA policy · 2019-01-17T07:55:44.694Z · score: 3 (1 votes) · LW · GW

I noticed that the sum inside is not actually an expected utility, because the SSA probabilities do not add up to 1 when there is more than one possible observation. The issue is that conditional on making an observation, the probabilities for the trajectories not containing that observation become 0, but the other probabilities are not renormalized. So this seems to be part way between "real" EDT and UDT (which does not set those probabilities to 0 and of course also does not renormalize).

This zeroing of probabilities of trajectories not containing the current observation (and renormalizing, if one was to do that) seems at best useless busywork, and at worst prevents coordination between agents making different observations. In this formulation of EDT, such coordination is ruled out in another way, namely by specifying that conditional on o→a, the agent is still sure the rest of π is unchanged (i.e., copies of itself receiving other observations keep following π). If we remove the zeroing/renormalizing and say that the agent ought to have more realistic beliefs conditional on o→a, I think we end up with something close to UDT1.0 (modulo differences in the environment model from the original UDT).

(Oh, I ignored the splitting up of probabilities of trajectories into SSA probabilities and then adding them back up again, which may have some intuitive appeal but ends up being just a null operation. Does anyone see a significance to that part?)

Comment by wei_dai on Towards formalizing universality · 2019-01-16T21:00:49.376Z · score: 5 (2 votes) · LW · GW

𝔼[X|Φ(𝔼¹, 𝔼²)] = 𝔼[𝔼¹[X]|Φ(𝔼¹, 𝔼²)]

What if X is something that only 𝔼 knows, like a random number that the agent that 𝔼 represents just thought up? Then there would be no way for this equality to hold? Maybe it should be something like this instead?

𝔼[X|𝔼¹[X]] = 𝔼[X|𝔼¹[X], Φ(𝔼¹, 𝔼²)]

I'm not sure this is right either, because if 𝔼 is smart enough maybe it can extract some information about X from Φ(𝔼¹, 𝔼²) that's not in 𝔼¹[X]. Edit: Never mind, that's fine because it just means that 𝔼¹ has to be smart enough that 𝔼¹[X] takes into account everything that 𝔼 might be able to extract from Φ(𝔼¹, 𝔼²).

Comment by wei_dai on Open Thread January 2019 · 2019-01-16T11:51:32.215Z · score: 5 (2 votes) · LW · GW

The author suggests that just slowing down research into risky technologies in other countries would be worthwhile:

  • The lack of acceleration of science following the high skill immigration shock to the US is not necessarily bad news: it may also imply that future shocks won’t accelerate risky technologies, that research funding is a more fundamental constraint, or that other sectors of the economy are better at absorbing high skill immigrants.
  • Further emigration likely decelerated progress for potentially risky technologies in the former USSR, which is a net reduction of risk: there is less incentive for the US government to engage in an arms race if there is no one to race.
Comment by wei_dai on CDT=EDT=UDT · 2019-01-16T11:06:02.815Z · score: 13 (3 votes) · LW · GW

The result there is sometimes abbreviated as UDT=CDT+SIA, although UDT⊂CDT+SIA is more accurate, because the optimal UDT policies are a subset of the policies which CDT+SIA can follow. This is because UDT has self-coordination power which CDT+SIA lacks.

I feel like this understates the way in which CDT+SIA is philosophically/intuitively crazy/implausible. Consider a variant of AMD where U(A)=1, U(B)=0, U(C)=2. Obviously one should select CONT with probability 1 in order to reach C, but "EXIT with probability 1" seems to be another CDT+SIA solution. The CDT+SIA reasoning there (translated from math to English) is: Suppose my policy is "EXIT with probability 1". Then I'm at X with probability 1. Should I deviate from this policy? If I do CONT instead, I'm still at X with probability 1 and the copy of me at Y will still do EXIT with probability 1 so I'll end up at B for sure with utility 0, therefore I should not deviate. Isn't this just obviously crazy (assuming I didn't misunderstand something)?

we could say that UDT1.0 = CDT+SIA

But UDT1.0 already gives a unique and correct solution to the problem above.

Caspar Oesterheld commented on that post with an analogous EDT+SSA result.

I tried to understand Caspar's EDT+SSA but was unable to figure it out. Can someone show how to apply it to an example like the AMD to help illustrate it?

Comment by wei_dai on What are the open problems in Human Rationality? · 2019-01-14T20:22:58.561Z · score: 6 (3 votes) · LW · GW

One more, because one of my posts presented two open problems, and I only listed one of them above:

15. Our current theoretical foundations for rationality all assume a fully specified utility function (or the equivalent), or at least a probability distribution on utility functions (to express moral/value uncertainty). But to the extent that humans can be considered to have a utility function at all, it's may best be viewed as a partial function that returns "unknown" for most of the input domain. Our current decision theories can't handle this because they would end up trying to add "unknown" to a numerical value during expected utility computation. Forcing humans to come up with an utility function or even a probability distribution on utility functions in order to use decision theory seems highly unsafe so we need an alternative.

Comment by wei_dai on Towards formalizing universality · 2019-01-14T19:46:42.200Z · score: 5 (2 votes) · LW · GW

Also, I'm confused about the practical case.

For example, suppose that C formulates a plan to “trick” A[C]. Then the subjective universality condition implies that we don’t expect C to succeed.

What does "expect" mean here? Probability > .5? Also, can you walk through an example of how C might "trick" A[C] and how subjective dominance implies that we don't expect C to succeed?

We’d like to say that the impoverished perspective is still “good enough” for us to feel safe, despite not being good enough to capture literally everything we know. But now we risk begging the question: how do we evaluate whether the impoverished perspective is good enough? I think this is probably OK, but it’s definitely subtle.

I don't know how to make sense of this. If an impoverished perspective expects C not to be able to "trick" A, it seems kind of obvious that's not good enough for me to think the same? You must have reason to think otherwise but the inferential distance is too far for me to figure out what it is.

Comment by wei_dai on What are the open problems in Human Rationality? · 2019-01-14T07:55:27.159Z · score: 63 (17 votes) · LW · GW

I went through all my LW posts and gathered the ones that either presented or reminded me of some problem in human rationality.

1. As we become more rational, how do we translate/transfer our old values embodied in the less rational subsystems?

2. How to figure out one's comparative advantage?

3. Meta-ethics. It's hard to be rational if you don't know where your values are supposed to come from.

4. Normative ethics. How much weight to put on altruism? Population ethics. Hedonic vs preference utilitarianism. Moral circle. Etc. It's hard to be rational if you don't know what your values are.

5. Which mental subsystem has one's real values, or how to weigh them.

6. How to handle moral uncertainty? For example should we discount total utilitarianism because we would have made a deal to for total utilitarianism to give up control in this universe?

7. If we apply UDT to humans, what does it actually say in various real-life situations like voting or contributing to x-risk reduction?

8. Does Aumann Agreement apply to humans, and if so how?

9. Meta-philosophy. It's hard to be rational if one doesn't know how to solve philosophical problems related to rationality.

10. It's not clear how selfishness works in UDT, which might be a problem if that's the right decision theory for humans.

11. Bargaining, politics, building alliances, fair division, we still don't know how to apply game theory to a lot of messy real-world problems, especially those involving more than a few people.

12. Reality fluid vs. caring measure. Subjective anticipation. Anthropics in general.

13. What is the nature of rationality, and more generally normativity?

14. What is the right way to handle logical uncertainty, and how does that interact with decision theory, bargaining, and other problems?

Comparing the rate of problems opened vs problems closed, we have so far to go....

Comment by wei_dai on Open Thread January 2019 · 2019-01-14T06:45:26.864Z · score: 22 (8 votes) · LW · GW

Strategic High Skill Immigration seems to be a very high quality and relevant post that has been overlooked by most people here, perhaps due to its length and rather obscure title. (Before I strongly upvoted it, it had just 12 points and 4 voters, presumably including the author.) If a moderator sees this, please consider curating it so more people will read it. And for everyone else, I suggest reading at least part 1. It talks about why global coordination on AI safety and other x-risks is hard, and suggests a creative solution for making it a little easier. Part 2 is much longer and goes into a lot of arguments, counterarguments, and counter-counterarguments, and can perhaps be skipped unless you have an interest in this area.

Comment by wei_dai on Towards formalizing universality · 2019-01-14T06:14:23.938Z · score: 5 (2 votes) · LW · GW

So "simpler" in that sentence should be replaced by "simple enough"? In other words, it's not the case that A is better-informed than every computation C that is simpler than A, right? Also, can you give a sense of how much simpler is simple enough?

Comment by wei_dai on Towards formalizing universality · 2019-01-14T00:26:26.952Z · score: 7 (3 votes) · LW · GW

This post defines the concept of “ascription universality,” which tries to capture the property that a question-answering system A is better-informed than any particular simpler computation C.

I'm getting lost right away here. In the case of "large enough teams of humans can carry out arbitrarily complicated reasoning", what is A and C? Presumably A is the whole team, but I have no idea what C is.

Comment by wei_dai on Comments on CAIS · 2019-01-13T19:26:02.883Z · score: 5 (2 votes) · LW · GW

That said, I disagree with Wei that this is relatively crisp: taken literally, the definition is vacuous because all behavior maximizes some expected utility.

I think I meant more that an AGI's internal cognition resembles that of an expected utility maximizer. But even that isn't quite right since it would cover AIs that only care about abstract worlds or short time horizons or don't have general intelligence. So yeah, I definitely oversimplified there.

Maybe we mean that it is long-term goal-directed, but at least I don’t know how to cash that out.

What's wrong with cashing that out as trying to direct/optimize the future according to some (maybe partial) preference ordering (and using a wide range of competencies, to cover "general")? You said "In fact, I don’t want to assume that the agent even has a preference ordering" but I'm not sure why. Can you expand on that?

Comment by wei_dai on What are questions? · 2019-01-13T06:58:30.671Z · score: 13 (3 votes) · LW · GW

I found an old comment by Morendil that provides a useful (but possibly non-exhaustive) taxonomy of questions:

  • “challenge” questions like “how confident are you about this”—they are really intended to prompt the askee to ask the question of themselves
  • “clarification” questions—like “what did you mean by X”
  • “genuinely curious” questions—the first to seem so was “How does any particular agent go about convincing me that it’s Omega?”
  • “mocking rhetorical” questions—like “Don’t they know (...) this thing called ‘social networking’? ”
  • “hypothetical” questions—like “what would I think of an amateur who argues with me in the area of my competence?”
  • “question and answer” pairs—like “does that mean they are wasting their time there? Of course not...”—obviously rhetorical
  • “what if” questions—like “And what if you wanted to grow?”
  • “agreement seeking” questions—like “Agreed?”
  • “information-providing” questions—like “Anyone heard of Marblar?” (with a link, so we know the asker has heard of it)

This makes me think that none of the current answers have provided a full explanation of what questions are yet.

Comment by wei_dai on Reframing Superintelligence: Comprehensive AI Services as General Intelligence · 2019-01-13T06:29:50.135Z · score: 5 (2 votes) · LW · GW

Most of these sections seem to only contain arguments that AGI won't come earlier than CAIS, but not that it would come later than CAIS. In other words, they don't argue against the likelihood that under CAIS someone can easily build an AGI by connecting existing AI services together in a straightforward way. The only section I can find among the ones you listed that tries to argue in this direction is Section 13, but even it mostly just argues that AGI isn't simpler than CAIS, and not that it's more complex, except for this paragraph in the summary, Section 13.5:

To summarize, in each of the areas outlined above, the classic AGI model both obscures and increases complexity: In order for general learning and capabilities to fit a classic AGI model, they must not only exist, but must be integrated into a single, autonomous, self-modifying agent. Further, achieving this kind of integration would increase, not reduce, the challenges of aligning AI behaviors with human goals: These challenges become more difficult when the goals of a single agent must motivate all (and only) useful tasks.

So putting alignment aside (I'm assuming that someone would be willing to build an unaligned AGI if it's easy enough), the only argument Eric gives for greater complexity of AGI vs CAIS is "must be integrated into a single, autonomous, self-modifying agent", but why should this integration add a non-negligible amount of complexity? Why can't someone just take a plan maker, connect it to a plan executer, and connect that to the Internet to access other services as needed? (I think your argument that strategic planning may be one of the last AIS to arrive is plausible, but it doesn't seem to be an argument that Eric himself makes.) Where is the additional complexity coming from?

Comment by wei_dai on Modernization and arms control don’t have to be enemies. · 2019-01-13T03:18:23.964Z · score: 11 (5 votes) · LW · GW

I'm curious about something that maybe you can answer, since you seem to have a strong interest in nuclear strategy and arms control: why does Russia have so many more nuclear weapons than China? The answer I've been able to find online are:

  1. Russia wants to have the option of doing a first strike (like the US) instead of just a second strike (like China). (But why does Russia feel a need for this and China doesn't?)
  2. The Cold War arms race caused the Soviet Union to have many more nuclear weapons than China, which Russia inherited. (But why doesn't Russia unilaterally reduce its nuclear arsenal to China's size?)

Let me know if you have any thoughts on this, or can point me to any papers or articles.

Comment by wei_dai on Why is so much discussion happening in private Google Docs? · 2019-01-12T20:35:42.673Z · score: 23 (7 votes) · LW · GW

if Alice sees Bob make good remarks etc., she’s more interested in ‘running a draft by him’ next time, or to respond positively if Bob asks her to look something over

This dynamic contributes to anxiety for me to comment in Google Docs, and makes it less fun than public commenting (apparently the opposite of many other people). I feel like if I fail to make a good contribution, or worse, make a dumb comment, I won't be invited to future drafts and will end up missing a lot of good arguments, or entire articles because many drafts don't get published until much later or ever. This is not a theoretical fear because I've "gotten in" to a couple of important and still unpublished drafts only by accidentally finding out about them and requesting invites.

Another thing that contributes to the anxiety is feeling a need to make a good first impression to people who are in the discussion who I've never talked to before because they don't participate on public forums.

Comment by wei_dai on Comments on CAIS · 2019-01-12T20:05:17.006Z · score: 27 (8 votes) · LW · GW

However, I worry that the fuzziness of the usual concept of AGI has now been replaced by a fuzzy notion of “service” which makes sense in our current context, but may not in the context of much more powerful AI technology.

It seems to me that "AGI" is actually relatively crisp compared to "service": it's something that approximates an expected utility maximizer, which seems like a pretty small and relatively compact cluster in thing-space. "Service" seems to cover a lot more varied ground, from early simple things like image classifiers to later strategic planners, natural language advice givers, AI researchers, etc., with the later things shading into AGI in a way makes it hard to distinguish between them.

it definitely seems worth investigating ways to make modular and bounded AIs more competitive, and CAIS more likely.

A major problem in predicting CAIS safety is to understand the order in which various services are likely to arise, in particular whether risk-reducing services are likely to come before risk-increasing services. This seems to require a lot of work in delineating various kinds of services and how they depend on each other as well as on algorithmic advancements, conceptual insights, computing power, etc. (instead of treating them as largely interchangeable or thinking that safety-relevant services will be there when we need them). Since this analysis seems very hard to do much ahead of time, I think we'll have to put very wide error bars on any predictions of whether CAIS would be safe or unsafe, until very late in the game. (This seems like a natural perspective for thinking about CAIS safety, which appears to be missing from Eric's report.)

Having said that, my feeling is that many risk-reducing services (especially ones that can address human safety problems) seem to require high-level general reasoning abilities, whereas many risk-increasing services can just be technical problem solvers or other kinds of narrow intelligences or optimizers, so the latter is likely to arrive earlier than the former, and as a result CAIS is actually quite unsafe, and hard to make safe, whereas AGI is by default highly unsafe, but with appropriate advances in safety research can perhaps be made safe. So I disagree with the proposal to push for CAIS, at least until we can better understand the strategic landscape. See also this comment where I made some related points.

Comment by wei_dai on Ambitious vs. narrow value learning · 2019-01-12T17:33:14.595Z · score: 4 (2 votes) · LW · GW

I’m not sure I understand the question. Inverse reinforcement learning, preference learning (eg. deep RL from human preferences) and inverse reward design are some existing examples of narrow value learning.

Thanks for the existing examples, which are helpful, but I guess what I was trying to ask was, is there a mathematical theory of instrumental value learning, that we can expect practical algorithms to better approximate over time, which would let us predict what future algorithms might look like or be able to do?

You have to put active work to make sure the value learner continues to do what you want.

"You" meaning the user? Does the user need to know when they need to provide the AI with more training data? (For example, if there was a massive devaluation of the US dollar, they need to predict that the AI might sell all their other possessions for dollars, and actively provide the AI with more training data before that happens?) Or can we expect the AI to know when it should ask the user for more training data? If the latter, what can we expect the AI to do in the meantime (e.g., if the user is asleep and it can't ask)?

Comment by wei_dai on Ambitious vs. narrow value learning · 2019-01-12T11:31:55.292Z · score: 3 (1 votes) · LW · GW

How would this kind of narrow value learning work in a mathematical or algorithmic sense? For example, one question I have is, since instrumental goals and values can be invalidated by environmental changes (e.g., I'd stop valuing US dollars if I couldn't buy things with them anymore), how does the value learner know when that has happened? Are there any papers I can read or tutorials I can watch to learn about this? Or feel free to give an explanation here if it's simple enough.

Comment by wei_dai on AlphaGo Zero and capability amplification · 2019-01-12T08:03:00.347Z · score: 6 (3 votes) · LW · GW

Upvoted for giving this number, but what does it mean exactly? You expect "50% fine" through all kinds of x-risk, assuming no coordination from now until the end of the universe? Or just assuming no coordination until AGI? Is it just AI risk instead of all x-risk, or just risk from narrow AI alignment? If "AI risk", are you including risks from AI exacerbating human safety problems, or AI differentially accelerating dangerous technologies? Is it 50% probability that humanity survives (which might be "fine" to some people) or 50% that we end up with a nearly optimal universe? Do you have a document that gives all of your quantitative risk estimates with clear explanations of what they mean?

(Sorry to put you on the spot here when I haven't produced anything like that myself, but I just want to convey how confusing all this is.)

Comment by wei_dai on Non-Consequentialist Cooperation? · 2019-01-12T07:39:34.780Z · score: 5 (2 votes) · LW · GW

One could make a similar argument for corrigibility: ambitious value learning would respect our desire for it to behave corrigibly if we actually wanted that, and if we didn’t want that, why impose it?

There's a disanalogy in that autonomy is probably a terminal value whereas corrigibility is only an instrumental one. In other words, I don't want a corrigible AI for the sake of having a corrigible AI, I want one so it will help me reach my other goals. I do (probably) want autonomy, and not only because it would help me reach other goals. So in fact ambitious value learning will not learn to behave corrigibly, I think, because the AI will probably think it has a better way of giving me what I ultimately want.

Oh, I think I see a different way of stating your argument that avoids this disanalogy: we're not concerned about autonomy as a terminal value here, but as an instrumental one like corrigibility. If ambitious value learning works perfectly, then it would learn autonomy as a terminal value, but we want to implement autonomy-respecting AI mainly because that would help us get what we want in case ambitious value learning fails to works perfectly.

I think I understand the basic idea and motivation now, and I'll just point out that autonomy-respecting AI seems share several problems with other non-goal-directed approaches to AI safety.

Why is so much discussion happening in private Google Docs?

2019-01-12T02:19:19.332Z · score: 76 (18 votes)
Comment by wei_dai on Combat vs Nurture & Meta-Contrarianism · 2019-01-12T02:03:38.101Z · score: 7 (3 votes) · LW · GW

Somewhat related to this, I sometimes do the following and would be interested in feedback on whether I'm coming across the right way. At the start, I'm curious why another person thinks what they think, and because I can't expect an instant answer if I post a question online, I try to guess first. If I'm not confident in any of my guesses after a while, I'll write down and post the question, and since I already have some guesses, I write those down as well in order to signal that I'm taking the other person seriously. But I'm not sure if I'm succeeding in this signaling attempt (e.g., maybe other people think I'm straw-manning them, or something else I'm not thinking of). Here is a recent example of me trying to do this. Feedback welcome on whether this (i.e., signaling by writing down my guesses) is a good idea, and whether I'm succeeding in my attempts.

Comment by wei_dai on Reframing Superintelligence: Comprehensive AI Services as General Intelligence · 2019-01-11T17:36:13.168Z · score: 5 (2 votes) · LW · GW

Ah, ok, what sections would you suggest that I (re)read to understand his arguments better? (You mentioned 12, 13, 10, 11 and 16 earlier in this thread but back then we were talking about "AGI won’t be much more capable than CAIS" and here the topic is whether we should expect AGI to come later than CAIS or require harder conceptual breakthroughs.)

Comment by wei_dai on Non-Consequentialist Cooperation? · 2019-01-11T17:27:56.182Z · score: 5 (2 votes) · LW · GW

Autonomy is a value and can be expressed as a part of a utility function, I think. So ambitious value learning should be able to capture it, so an aligned AI based on ambitious value learning would respect someone's autonomy when they value it themselves. If they don't, why impose it upon them?

(This assumes that we manage to solve the general problems with ambitious value learning. Is the point here that you expect we can't solve those problems and therefore need an alternative? The idea doesn't help with "the difficulties of assuming human rationality" though so what problems does it help with?)

ETA: Is the idea that even trying to do ambitious value learning constitutes violating someone's autonomy (in other words someone could have a preference against having ambitious value learning done on them) and by the time we learn this it would be too late?

Comment by wei_dai on Book Recommendations: An Everyone Culture and Moral Mazes · 2019-01-11T06:07:54.536Z · score: 7 (3 votes) · LW · GW

It is, perhaps, possible that operating as a DDO is “cheaper”—for the company (though I am inclined to doubt it). But it’s a heck of a lot more expensive for the employees. Even in the best case (where the company’s performance improves as a result of adopting this model of company culture), this sort of approach boils down to the firm externalizing a large chunk of its operating costs onto its employees.

This doesn't seem to make economic sense. If the company is imposing large costs on its employees, the employees would demand bigger salaries (or other compensation) to work at that company instead of another, so "externalizing" doesn't apply here. Of course there are exceptions to this, like if the employees are being tricked to not notice the imposed costs, or are biased to underestimate the imposed costs, but I don't see an obvious reason to think that's the case. I mean, when someone interviews for Bridgewater, do they hide the company culture and what it's like to work there? Presumably not since the CEO wrote a book about it?

Comment by wei_dai on Reframing Superintelligence: Comprehensive AI Services as General Intelligence · 2019-01-11T05:11:39.003Z · score: 6 (3 votes) · LW · GW

Unfortunately, I only vaguely understand the points that you're trying to make in this comment... Would it be fair to just say at this point that this is an important crux that Eric failed to convincingly argue for?

Comment by wei_dai on AlphaGo Zero and capability amplification · 2019-01-11T04:33:14.750Z · score: 6 (2 votes) · LW · GW

I’m not aware of any AI safety researchers that are extremely optimistic about solving alignment competitively.

I'm not sure what you'd consider "extremely" optimistic, but I gathered some quantitative estimates of AI risk here, and they all seem overly optimistic to me. Did you see that?

Paul: I just think working on this problem earlier will tell us what’s going on. If we’re in the world where you need a really drastic policy response to cope with this problem, then you want to know that as soon as possible.

I agree with this motivation to do early work, but in a world where we do need drastic policy responses, I think it's pretty likely that the early work won't actually produce conclusive enough results to show that. For example, if a safety approach fails to make much progress, there's not really a good way to tell if it's because safe and competitive AI really is just too hard (and therefore we need a drastic policy response), or because the approach is wrong, or the people working on it aren't smart enough, or they're trying to do the work too early. People who are inclined to be optimistic will probably remain so until it's too late.

Comment by wei_dai on Reframing Superintelligence: Comprehensive AI Services as General Intelligence · 2019-01-10T21:57:38.378Z · score: 10 (2 votes) · LW · GW

You could think of (my conception of) CAIS as a claim that a similar process will happen in a decentralized way for all of ML by default, and at any point the things we can do will look like an explicit iterated amplification deliberation tree of depth one or two, where the leaves are individual services and the top level question will be some task that is accomplished through a combination of individual services.

This seems like a sensible way of looking at things, and in this framing I'd say that my worry is that crucial safety-enhancing services may only appear fairly high in the overall tree of services, or outside the tree altogether (see also #3 in Three AI Safety Related Ideas which makes a similar point), and in the CAIS world it would be hard to limit access to the lower-level services (as a risk-reduction measure).

Comment by wei_dai on AlphaGo Zero and capability amplification · 2019-01-10T21:02:25.014Z · score: 6 (3 votes) · LW · GW

Generally, I don’t see why we should expect that the most capable systems that can be created with supervised learning (e.g. by using RL to search over an arbitrary space of NN architectures) would perform similarly to the most capable systems that can be created, at around the same time, using some restricted supervised learning that humans must trust to be safe. My prior is that the former is very likely to outperform by a lot, and I’m not aware of strong evidence pointing one way or another.

This seems similar to my view, which is that if you try to optimize for just one thing (efficiency) you're probably going to end up with more of that thing than if you try to optimize for two things at the same time (efficiency and safety) or if you try to optimize for that thing under a heavy constraint (i.e., safety).

But there are people (like Paul) who seem to be more optimistic than this based on more detailed inside-view intuitions, which makes me wonder if I should defer to them. If the answer is no, there's also the question of how do we make policy makers take this problem seriously (i.e., that safe AI probably won't be as efficient as unsafe AI) given the existence of more optimistic AI safety researchers, so that they'd be willing to undertake costly preparations for governance solutions ahead of time. By the time we get conclusive evidence one way or another, it may be too late to make such preparations.

Comment by wei_dai on Reframing Superintelligence: Comprehensive AI Services as General Intelligence · 2019-01-10T12:28:42.979Z · score: 10 (3 votes) · LW · GW

It seems fairly easy to expand this to include services that consider how disruptive new technologies will be, how underdetermined human values are, whether a proposed plan reduces option value, what risk aversion implies about a particular plan of action, what blind spots people have, etc.

Can you explain how you'd implement these services? Take "how disruptive new technologies will be" for example. I imagine you can't just apply ML given the paucity of training data and how difficult it would be to generalize from historical data to new technologies and new social situations. And it seems to me that if you base it on any kind of narrow AI technology, it would be easy to miss some of the novel implications/consequences of the new technologies and social situations and end up with a wrong answer. Maybe you could instead base it on a general purpose reasoner or question-answerer, but if something like that exists, AI would already have created a lot of new technologies that are risky for humans to face. Plus, the general purpose AI could replace a lot of discrete/narrow AI services, so I feel like we would already have moved past the CAIS world at that point. BTW, if the service is not just a thin wrapper on top of a general purpose AI which is generally trustworthy, I also don't know how you'd know whether you can trust the answers that it gives.

I’d be interested in a list of services that you think would be helpful for addressing human safety problems. You might think of this as “our best current guess at metaphilosophy and metaphilosophy research”.

I could try to think in that direction after I get a better sense of what kinds of services might be both feasible and trustworthy in the CAIS world. It seems easy to become too optimistic/complacent under the CAIS model if I just try to imagine what safety-enhancing services might be helpful without worrying about whether those services would be feasible or how well they'd work at the time when they're needed.

Comment by wei_dai on What is narrow value learning? · 2019-01-10T09:23:22.679Z · score: 3 (1 votes) · LW · GW

To head off a possible confusion come tomorrow, it seems like your definition of "narrow value learning" is a bit different from Paul's. You define it as learning to produce desired behavior in some domain, while Paul defined it as learning instrumental goals and values. I think this means that under your definition, behavioral cloning and approval-directed agents are subsets of narrow value learning, whereas under Paul's definition they are disjoint from narrow value learning. Does this seem right to you, and if so was this overloading of the term intentional?

Comment by wei_dai on Alignment Newsletter #40 · 2019-01-10T03:25:23.743Z · score: 5 (2 votes) · LW · GW

Two More Decision Theory Problems for Humans (Wei Dai)

I think when I wrote that post, I was mostly thinking of the problems as human rationality problems, but seeing it here (and also this comment) reminds me that there's an AI alignment angle as well. Specifically, in value learning it must be issue if the human and the AI do not share the same ontology or decision theory (especially if the human has non-consequentialist values). Are you aware of any literature on this? If so, perhaps some of the techniques can be transferred back into the realm of human rationality.

Comment by wei_dai on Reframing Superintelligence: Comprehensive AI Services as General Intelligence · 2019-01-10T00:19:33.817Z · score: 6 (3 votes) · LW · GW

Thanks, I think this is helpful for me to understand Eric's model better, but I'm still pretty confused.

It’s actually quite unclear how we would use current techniques to get something that does very-long-term-planning. (This could be the “conceptual breakthroughs” point.)

But it's quite unclear how to use current techniques to do a lot of things. Why should we expect that this conceptual breakthrough would come later than other conceptual breakthroughs needed to achieve CAIS? (Given your disagreement with Eric on this, I guess this is more a question for him than for you.)

Where did the long term optimization come from?

I was assuming that long term strategic planners (as described in section 27) are available as an AIS, and would be one of the components of the hypothetical AGI.

For example, you could take any long term task and break it down into the “plan maker” which thinks for an hour and gives a plan for the task, and the “plan executor” which takes an in-progress plan and executes the next step. Both of these are bounded and so could be services, and their combination is generally intelligent, but the combination wouldn’t have convergent instrumental subgoals.

I don't see why it wouldn't, unless these services are specifically designed to be corrigible (in which case the "corrigible" part seems much more important than the "service" part). For example, suppose you asked the plan maker to create a plan to cure cancer. Why would the mere fact that it's a bounded service prevent it from coming up with a plan that involves causing human extinction (and a bunch of convergent instrumental subgoals like deceiving humans who might stop it)? (If there was a human in the loop, then you could look at the plan and reject it, but I'm imagining that someone, in order to build an AGI as quickly and efficiently as possible, stripped off the "optimize for human consumption" part of the strategic planner and instead optimized it to produce plans for direct machine consumption.)

Comment by wei_dai on AlphaGo Zero and capability amplification · 2019-01-09T21:14:00.793Z · score: 3 (1 votes) · LW · GW

BTW, I think Amplification might currently be the most promising approach for creating aligned and powerful systems; what I argue is that in order to save the world it will probably need to be complemented with governance solutions.

How uncompetitive do you think aligned IDA agents will be relative to unaligned agents, and what kinds of governance solutions do you think that would call for? Also, I should have made this clearer last time, but I'd be interested to hear more about why you think Distill probably can't be made both safe and competitive, regardless of whether you're more or less optimistic than Paul.

Comment by wei_dai on AlphaGo Zero and capability amplification · 2019-01-09T18:51:32.605Z · score: 3 (1 votes) · LW · GW

Paul's position in that post was:

All of these ap­proaches feel very difficult, but I don’t think we’ve run into con­vinc­ing deal-break­ers.

I think this is meant to include the difficulty of making them competitive with unaligned ML, since that has been his stated goal. If you can argue that we should be even more pessimistic than this, I'm sure a lot of people would find that interesting.

Comment by wei_dai on AlphaGo Zero and capability amplification · 2019-01-09T12:23:26.172Z · score: 5 (3 votes) · LW · GW

I think Techniques for optimizing worst-case performance may be what you're looking for.

Comment by wei_dai on Reframing Superintelligence: Comprehensive AI Services as General Intelligence · 2019-01-09T11:47:40.732Z · score: 9 (4 votes) · LW · GW

I suspect he would claim that quickly building an AGI would not allow you to take over the world, because the AGI would not be that much more capable than the CAIS service cluster.

That does not seem to be his position though, because if AGI is not much more capable than CAIS, then there would be no need to talk specifically about how to defend the world against AGI, as he does at length in section 32. If that was his position, he could just talk about how ordinary policing and military defense would work in a CAIS world (i.e., against human adversaries wielding CAIS) and say that the same policing/defense would also work against AGI because AGI is not much more capable than CAIS.

Instead it seems clear that he thinks AGI requires special effort to defend against, which is made possible by a delay between SI-level CAIS and AGI, which he proposes that we use to do a very extensive "unopposed preparation". I've been trying to figure out why he thinks there will be such a delay and my current best guess is "Implementation of the AGI model is widely regarded as requiring conceptual breakthroughs." (page 75) which he repeats on page 77, "AGI (but not CAIS) calls for conceptual breakthroughs to enable both implementation and subsequent safe application." I don't understand why he thinks such conceptual breakthroughs will be required though. Why couldn't someone just take some appropriate AI services, connect them together in a straightforward way, and end up with an AGI? Do you get it? Or am I on the wrong track here?

Comment by wei_dai on Reframing Superintelligence: Comprehensive AI Services as General Intelligence · 2019-01-09T07:39:41.419Z · score: 3 (1 votes) · LW · GW

Monitoring surveillance in order to see if anyone is breaking rules seems to be quite a bounded task, and in fact is one that we are already in the process of automating (using our current AI systems, which are basically all bounded).

That seems true, but if this surveillance monitoring isn't 100% effective, won't you still need an agential police to deal with any threats that manage to evade the surveillance? Or do you buy Eric's argument that we can use a period of "unopposed preparation" to make sure that the defense, even though it's bounded, is still much more capable than any agential threat it might face?

Comment by wei_dai on Reframing Superintelligence: Comprehensive AI Services as General Intelligence · 2019-01-09T06:04:02.365Z · score: 22 (6 votes) · LW · GW

I have a problem with section 32, "Unaligned superintelligent agents need not threaten world stability". Here's the summary of that section from the paper:

  • Powerful SI-level capabilities can precede AGI agents.
  • SI-level capabilities could be applied to strengthen defensive stability.
  • Unopposed preparation enables strong defensive capabilities.
  • Strong defensive capabilities can constrain problematic agents.

So the key idea here seems to be that good actors will have a period of time to use superintelligent AI services to prepare some sort of ubiquitous defense that will constrain any subsequent AGI agents. But I don't understand where this period of "unopposed preparation" comes from. Why wouldn't someone create an AGI by cobbling together a bunch of AI services, or hire a bunch of AI services to help them design an AGI, as soon as they could? If they did that, then superintelligent AGI agents would arise nearly simultaneously with SI-level capabilities, and there would be no such period of unopposed preparation. In section 32.2, Eric only argues that SI-level capabilities can precede AGI agents. Since I think they wouldn't at least not by a significant margin, the whole argument seems to fall apart or has to be interpreted in a way that makes it strategically irrelevant.

Eric seems to think that no one would bother to create AGI because "AGI agents offer no compelling value", by which he means "Because general AI-development capabilities can provide stable, comprehensive AI services, there is no compelling, practical motivation for undertaking the more difficult and potentially risky implementation of self-modifying AGI agents." But if quickly building an AGI can potentially allow someone to take over the world before "unopposed preparation" can take place, isn't that a compelling motivation by itself for many people?

Comment by wei_dai on AI safety without goal-directed behavior · 2019-01-09T05:09:58.547Z · score: 5 (2 votes) · LW · GW

For the case of idealized humans, couldn’t real humans defer to idealized humans if they thought that was better?

Real humans could be corrupted or suffer some other kind of safety failure before the choice to defer to idealized humans becomes a feasible option. I don't see how to recover from this, except by making an AI with a terminal goal of deferring to idealized humans (as soon as it becomes powerful enough to compute what idealized humans would want).

Similarly, it seems like a non-goal-directed agent could be instructed to use the metaphilosophical algorithm. I guess I could imagine a metaphilosophical algorithm such that following it requires you to be goal-directed, but it doesn’t seem very likely to me.

That's a good point. Solving metaphilosophy does seem to have the potential to help both approaches about equally.

For an explicit set of values, those values come from humans, so wouldn’t they be subject to human safety problems? It seems like you would need to claim that humans are better at stating their values than acting in accordance with them, which seems true in some settings and false in others.

Well I'm not arguing that goal-directed approaches are more promising than non-goal-directed approaches, just that they seem roughly equally (un)promising in aggregate.

Comment by wei_dai on Reframing Superintelligence: Comprehensive AI Services as General Intelligence · 2019-01-08T18:58:38.902Z · score: 4 (2 votes) · LW · GW

One can try to make “AI police” as a service, but it could be less effective than agential police.

This seems likely to me as well, especially since "service" is by definition bounded and agent is not.

Comment by wei_dai on No option to report spam · 2019-01-08T18:39:41.090Z · score: 3 (1 votes) · LW · GW

Does that mean spam will still show up if I use the "All" view in GreaterWrong? I use that all the time to catch people's personal posts... Although if you've also added automated spam filtering then perhaps that's not a big deal anymore.

Comment by wei_dai on AI safety without goal-directed behavior · 2019-01-08T18:32:25.072Z · score: 9 (4 votes) · LW · GW

All else equal, I would prefer a more formal solution, but I don’t think we have the time for that.

I should have added that having a theory isn't just so we can have a more formal solution (which as you mention we might not have the time for) but it also helps us be less confused (e.g., have better intuitions) in our less formal thinking. (In other words I agree with what MIRI calls "deconfusion".) For example currently I find it really confusing to think about corrigible agents relative to goal-directed agents.

However, even with goal-directed agents, the goal has to come from somewhere, which means it comes from humans. (If not, we almost certainly get catastrophe.) So wouldn’t the goal have all of the human safety problems anyway?

The goal could come from idealized humans, or from a metaphilosophical algorithm, or be an explicit set of values that we manually specify. All of these have their own problems, of course, but they do avoid a lot of the human safety problems that the non-goal-directed approaches would have to address some other way.

Comment by wei_dai on No option to report spam · 2019-01-08T17:57:10.865Z · score: 3 (1 votes) · LW · GW

Agreed. Also, in the slightly longer term, there must be automated spam detection services that could be incorporated or hired to reduce the moderators' spam-filtering work load? (If not, it seems like a business opportunity for someone.)

Comment by wei_dai on Reframing Superintelligence: Comprehensive AI Services as General Intelligence · 2019-01-08T17:23:05.321Z · score: 28 (9 votes) · LW · GW

This is one of the documents I was responding to when I wrote A general model of safety-oriented AI development, Three AI Safety Related Ideas, and Two Neglected Problems in Human-AI Safety. (I didn't cite it because it was circulating semi-privately in draft form, and Eric apparently didn't want its existence to be publicly known.) I'm disappointed that although Eric wrote to me "I think that your two neglected problems are critically important", the perspectives in those posts didn't get incorporated more into the final document, which spends only 3 short paragraphs out of hundreds of pages to talk about what I think of as "human safety problems". (I think those paragraphs were in the draft even before I wrote my posts.)

I worry about the framing adopted in this document that the main problem in human-AI safety is "questions of what humans might choose to do with their capabilities", as opposed to my preferred framing of "how can we design human-AI systems to minimize total risk". (To be fair to Eric, a lot of other AI safety people also only talk about "misuse risk" and not about how AI is by default likely to exacerbate human safety problems, e.g., by causing rapid distributional shifts for humans.) I worry that this gives AI researchers and developers license to think, "I'm just developing an AI service. AI services will be comprehensive anyway so there's no reason for me to hold back or think more about what I'm doing. It's someone else's job to worry about what humans might choose to do with these capabilities."

Comment by wei_dai on AI safety without goal-directed behavior · 2019-01-08T05:32:39.956Z · score: 22 (10 votes) · LW · GW

I'm curious if you're more optimistic about non-goal-directed approaches to AI safety than goal-directed approaches, or if you're about equally optimistic (or rather equally pessimistic). The latter would still justify your conclusion that we ought to look into non-goal-directed approaches, but if that's the case I think it would be good to be explicit about it so as to not unintentionally give people false hope (ETA: since so far in this sequence you've mostly talked about the problems associated with goal-directed agents and not so much about problems associated with the alternatives). I think I'm about equally pessimistic, because while goal-directed agents have a bunch of safety problems, they also have a number of advantages that may be pretty hard to replicate in the alternative approaches.

  1. We have an existing body of theory about goal-directed agents (which MIRI is working on refining and expanding) which plausibly makes it possible to one day reason rigorously about the kinds of goal-directed agents we might build and determine their safety properties. Paul and others working on his approach are (as I understand it) trying to invent a theory of corrigibility, but I don't know if such a thing even exists in platonic theory space. And if it did, we're starting from scratch so it might take a long time to reach parity with the theory of goal-directed agents.
  2. Goal-directed agents give you economic efficiency "for free". Alternative approaches have to simultaneously solve efficiency and safety, and may end up approximating goal-directed agent anyway due to competitive pressures.
  3. Goal-directed agents can more easily avoid a bunch of human safety problems that are inherited by alternative approaches which all roughly follow the human-in-the-loop paradigm. These include value drift (including vulnerability to corruption/manipulation), problems with cooperation/coordination, lack of transparency/interpretability, and general untrustworthiness of humans.
Comment by wei_dai on Imitation learning considered unsafe? · 2019-01-07T22:59:36.427Z · score: 4 (2 votes) · LW · GW

See the clarifying note in the OP. I don’t think this is about imitating humans, per se.

Yes, I realized that after I wrote my original comment, so I added the "ETA" part.

I think this intuition has some validity, but also might lead to a false sense of confidence that such systems are safe, when in fact they may end up behaving as if they do seek to influence the world, depending on the task they are trained on (ETA: and other details of the learning algorithm, e.g. outer-loop optimization and model choice).

I think this makes sense and at least some have also realized this and have reacted appropriately within their agenda (see the "ETA" part of my earlier comment). It also seems good that you're calling it out as a general issue. I'd still suggest giving some examples of AI alignment proposals where people haven't realized this, to help illustrate your point.

Comment by wei_dai on Imitation learning considered unsafe? · 2019-01-07T04:07:33.948Z · score: 6 (3 votes) · LW · GW

I find it one of the more troubling outstanding issues with a number of proposals for AI alignment.

Which proposals? AFAIK Paul's latest proposal no longer calls for imitating humans in a broad sense (i.e., including behavior that requires planning), but only imitating a small subset of the human policy which hopefully can be learned exactly correctly. See this comment where he wrote:

Are you perhaps assuming that we can max out regret during training for the agents that have to be trained with human involvement, but not necessarily for the higher level agents?

Yeah, I’m relatively optimistic that it’s possible to learn enough from humans that the lower level agent remains universal (+ aligned etc.) on arbitrary distributions. This would probably be the case if you managed to consistently break queries down into simpler pieces until arriving at a very simple queries.

ETA: Oh, but the same kind of problems you're point out here would still apply at the higher level distillation steps. I think the idea there is for an (amplified) overseer to look inside the imitator / distilled agent during training to push it away from doing anything malign/incorrigible (as Jessica also mentioned). Here is a post where Paul talked about this.

Comment by wei_dai on Will humans build goal-directed agents? · 2019-01-06T16:32:17.073Z · score: 3 (1 votes) · LW · GW

For the first one, I guess I would use "argument for defense against value drift" instead since you could conceivably use a goal-directed AI to defend against value drift without lock in, e.g., by doing something like Paul Christiano's 2012 version of indirect normativity (which I don't think is feasible but maybe there's something like it that is, like my hybrid approach, if you consider that goal-directed).

For the third one, I guess interpretability is part of it, but a bigger problem is that it seems hard to make a sufficiently trustworthy human overseer even if we could "interpret" them. In other words, interpretability for a human might just let us see exactly why we shouldn't trust them.

Two More Decision Theory Problems for Humans

2019-01-04T09:00:33.436Z · score: 56 (17 votes)

Two Neglected Problems in Human-AI Safety

2018-12-16T22:13:29.196Z · score: 69 (20 votes)

Three AI Safety Related Ideas

2018-12-13T21:32:25.415Z · score: 72 (25 votes)

Counterintuitive Comparative Advantage

2018-11-28T20:33:30.023Z · score: 70 (25 votes)

A general model of safety-oriented AI development

2018-06-11T21:00:02.670Z · score: 70 (23 votes)

Beyond Astronomical Waste

2018-06-07T21:04:44.630Z · score: 90 (39 votes)

Can corrigibility be learned safely?

2018-04-01T23:07:46.625Z · score: 73 (25 votes)

Multiplicity of "enlightenment" states and contemplative practices

2018-03-12T08:15:48.709Z · score: 93 (23 votes)

Combining Prediction Technologies to Help Moderate Discussions

2016-12-08T00:19:35.854Z · score: 13 (14 votes)

Is the potential astronomical waste in our universe too small to care about?

2014-10-21T08:44:12.897Z · score: 25 (27 votes)

What is the difference between rationality and intelligence?

2014-08-13T11:19:53.062Z · score: 13 (13 votes)

Six Plausible Meta-Ethical Alternatives

2014-08-06T00:04:14.485Z · score: 36 (40 votes)

Look for the Next Tech Gold Rush?

2014-07-19T10:08:53.127Z · score: 37 (36 votes)

Three Approaches to "Friendliness"

2013-07-17T07:46:07.504Z · score: 17 (21 votes)

Normativity and Meta-Philosophy

2013-04-23T20:35:16.319Z · score: 12 (14 votes)

Outline of Possible Sources of Values

2013-01-18T00:14:49.866Z · score: 14 (16 votes)

Morality Isn't Logical

2012-12-26T23:08:09.419Z · score: 19 (35 votes)

Beware Selective Nihilism

2012-12-20T18:53:05.496Z · score: 40 (44 votes)

Ontological Crisis in Humans

2012-12-18T17:32:39.150Z · score: 44 (48 votes)

"Hide comments in downvoted threads" is now active

2012-10-05T07:23:56.318Z · score: 18 (30 votes)

Kelly Criteria and Two Envelopes

2012-08-16T21:57:41.809Z · score: 11 (8 votes)

Work on Security Instead of Friendliness?

2012-07-21T18:28:44.692Z · score: 35 (39 votes)

List of Problems That Motivated UDT

2012-06-06T00:26:00.625Z · score: 28 (29 votes)

Strong intutions. Weak arguments. What to do?

2012-05-10T19:27:00.833Z · score: 17 (19 votes)

How can we get more and better LW contrarians?

2012-04-18T22:01:12.772Z · score: 58 (62 votes)

Modest Superintelligences

2012-03-22T00:29:03.184Z · score: 22 (28 votes)

Where do selfish values come from?

2011-11-18T23:52:41.358Z · score: 38 (32 votes)

Do we want more publicity, and if so how?

2011-08-22T17:43:34.685Z · score: 17 (17 votes)

What's wrong with simplicity of value?

2011-07-27T03:09:18.130Z · score: 20 (22 votes)

What if sympathy depends on anthropomorphizing?

2011-07-24T12:33:06.741Z · score: 17 (17 votes)

How To Be More Confident... That You're Wrong

2011-05-22T23:30:23.992Z · score: 24 (24 votes)

A Thought Experiment on Pain as a Moral Disvalue

2011-03-11T07:56:23.892Z · score: 18 (18 votes)

Why Do We Engage in Moral Simplification?

2011-02-14T01:16:40.615Z · score: 24 (24 votes)

Metaphilosophical Mysteries

2010-07-27T00:55:59.222Z · score: 36 (50 votes)

Explicit Optimization of Global Strategy (Fixing a Bug in UDT1)

2010-02-19T01:30:44.399Z · score: 24 (22 votes)

Shut Up and Divide?

2010-02-09T20:09:06.208Z · score: 68 (69 votes)

Complexity of Value ≠ Complexity of Outcome

2010-01-30T02:50:49.369Z · score: 37 (36 votes)

Tips and Tricks for Answering Hard Questions

2010-01-17T23:56:20.904Z · score: 49 (49 votes)

Fictional Evidence vs. Fictional Insight

2010-01-08T01:59:03.452Z · score: 38 (33 votes)

A Master-Slave Model of Human Preferences

2009-12-29T01:02:23.694Z · score: 61 (63 votes)

What Are Probabilities, Anyway?

2009-12-11T00:25:33.177Z · score: 30 (31 votes)

Probability Space & Aumann Agreement

2009-12-10T21:57:02.812Z · score: 40 (37 votes)

Reflections on Pre-Rationality

2009-11-09T21:42:22.087Z · score: 19 (11 votes)

Re-understanding Robin Hanson’s “Pre-Rationality”

2009-11-03T02:58:04.599Z · score: 26 (20 votes)

Scott Aaronson on Born Probabilities

2009-10-02T06:40:11.245Z · score: 35 (34 votes)

The Absent-Minded Driver

2009-09-16T00:51:45.730Z · score: 33 (35 votes)

Towards a New Decision Theory

2009-08-13T05:31:41.197Z · score: 54 (54 votes)

The Popularization Bias

2009-07-17T15:43:30.338Z · score: 26 (25 votes)

Fair Division of Black-Hole Negentropy: an Introduction to Cooperative Game Theory

2009-07-16T04:17:44.081Z · score: 38 (42 votes)