LessWrong 2.0 Reader

Confusion #2: Why couldn't we make similar counting arguments for Turing machines?

I guess a central issue with separating NP from P with a counting argument is that (roughly speaking) there are equally many problems in NP and P. Each problem in NP has a polynomial-time verifier, so we can index the problems in NP by polytime algorithms, just like the problems in P.

in a bit more detail: We could try to use a counting argument to show that there is some problem with a (say) $<n^2$ time verifier which does not have any (say) $<n^{1000}$ time solver. To do this, we'd like to say that there are more $n^2$ verifier problems than $n^{1000}$ algorithms. While I don't really know how we ought to count these (naively, there are ${\aleph }_0$ of each), even if we had some decent notion of counting, there would almost certainly just be more $<n^{1000}$ algorithms than $<n^2$ verifiers (since the $n^2$ verifiers are themselves $<n^{1000}$ algorithms).

My sense is my predictions about Anthropic have also not been pessimistic enough, though we have not yet seen most of the evidence. Maybe a good time to make bets.

The predictions that seemed (somewhat) overly paranoid of yours were more about Anthropic than OpenPhil, and the dynamic seemed similar and I didn't check that hard while writing the comment. (maybe some predictions about how/why the OpenAI board drama went down, which was at the intersection of all three orgs, which I don't think have been explicitly revealed to have been "too paranoid" but I'd still probably take bets against)

(I think I agree that overall you were more like "not paranoid enough" than "too paranoid", although I'm not very confident)

Amazing write-up, thank you for the transparency and thorough work of documenting your impact.

Some of those concerns are, indeed, overly paranoid

I am actually curious if you have any overly paranoid predictions from me. I was today lamenting that despite feeling paranoid on this stuff all the time, I de-facto have still been quite overly optimistic in almost all of my predictions on this topic (like, I only gave SPARC a 50% chance of being defunded a few months ago, which I think was dumb, and I was not pessimistic enough to predict the banning of all right-associated project, and not pessimistic enough to predict a bunch of other grant decisions that I feel weird talking publicly about). 

I want to add the gear of "even if it actually turns out that OpenPhil was making the right judgment calls the whole time in hindsight, the fact that it's hard from the outside to know that has some kind of weird Epistemic Murkiness effects that are confusing to navigate, at the very least kinda suck, and maybe are Quite Bad." 

I've been trying to articulate the costs of this sort of thing lately and having trouble putting it into words, and maybe it'll turn out this problem was less of a big deal than it currently feels like to me. But, something like the combo of

a) the default being for many people to trust OpenPhil

b) many people who are paying attention think that they should at least be uncertain about it, and somewhere on a "slightly wary" to "paranoid" scale. and...

c) this at least causes a lot of wasted cognitive cycles

d) it's... hard to figure out how big a deal to make of it. A few people (i.e. habryka or previously Benquo or Jessicata) make it their thing to bring up concerns frequently. Some of those concerns are, indeed, overly paranoid, but, like, it wasn't actually reasonable to calibrate the wariness/conflict-theory-detector to zero, you have to make guesses. This is often exhausting and demoralizing for the people doing it. People typically only select into this sort of role if they're a bit more prone to conflict about it, which means a lot of the work is kinda thankless because people are pushing back on you for being too conflicty. Something about this compounds over time.

e) the part that feels hardest to articulate and maybe is fake is that, there's something of a "group epistemic process" going on in the surrounding community, and everyone either not tracking this sort of thing, or tracking it but not sure how to take it or what to do about it... I'm not sure how to describe it better than "I dunno something about the group orienting process subtly epistemically fucked" and/or "people just actually take sanity-damage from it."

("subtly epistemically fucked" might operationalize as "it takes an extra 1-3 years for things to become consensus knowledge/beliefs than it'd otherwise take")

Anyway, thanks for bringing it up.

Epistemic status: Speculating about adversarial and somewhat deceptive PR optimization, which is inherently very hard and somewhat paranoia inducing. I am quite confident of the broad trends here, but it's definitely more likely that I am getting things wrong here than in other domains where evidence is more straightforward to interpret, and people are less likely to shape their behavior in ways that includes plausible deniability and defensibility.

I agree with this, but I actually think the issues with Open Phil are substantially broader. As a concrete example, as far as I can piece together from various things I have heard, Open Phil does not want to fund anything that is even slightly right of center in any policy work. I don't think this is because of any COIs, it's because Dustin is very active in the democratic party and doesn't want to be affiliated with anything that is even slightly right-coded. Of course, this has huge effects by incentivizing polarization of AI policy work with billions of dollars, since any AI Open Phil funded policy organization that wants to engage with people on the right might just lose all of their funding because of that, and so you can be confident they will steer away from that.

Open Phil is also very limited in what they can say about what they can or cannot fund, because that itself is something that they are worried will make people annoyed with Dustin, which creates a terrible fog around how OP is thinking about stuff.^[1]

There honestly at the moment is no longer a single organization that I have historically been excited about that OpenPhil wants to fund. MIRI could not get OP funding, FHI could not get OP funding, Lightcone cannot get OP funding, my best guess is Redwood could not get OP funding if they tried today (though I am not super confident of this), most policy work I am excited about cannot get OP funding, the LTFF cannot get OP funding, any kind of intelligence enhancement work cannot get OP funding, CFAR cannot get OP funding, SPARC cannot get OP funding, the Czech community (Jan Kulveit and co) cannot get OP funding, not even ARC has been able to get OP funding (in that case because of COIs between Paul and Ajeya). I would be very surprised if Wentworth's work, or Wei Dai's work, or Daniel Kokotajlo's work, or Brian Tomasik's work could get funding from them these days. I might be missing some good ones, but the funding landscape is really quite thoroughly fucked in that respect. My best guess is Scott Alexander could not get funding, but I am not totally sure. 

I cannot think of anyone who I would credit with the creation or shaping of the field of AI Safety or Rationality who could still get OP funding. Bostrom, Eliezer, Gwern, Tomasik, Kokotajlo, Sandberg, Armstrong, Jessicata, Garrabrant, Demski, would all be unable to get funding as far as I can tell. In as much as OP is the most powerful actor in the space, the original geeks are being thoroughly ousted.

In-general my sense is if you want to be an OP grantee these days, you have to be the kind of person that OP thinks is not and will not be a PR risk, and who OP thinks has "good judgement" on public comms, and who isn't the kind of person who might say weird or controversial stuff, and is not at risk of becoming politically opposed to OP. This includes not annoying any potential allies that OP might have, or associating with anything that Dustin doesn't like, or that might strain Dustin's relationships with others in any non-trivial way. 

Of course OP will never ask you to fit these constraints directly, since that itself could explode reputationally (and also because OP staff themselves seem miscalibrated on this and do not seem in-sync with their leadership). Instead you will just get less and less funding, or just be defunded fully, if you aren't the kind of person who gets the hint that this is how the game is played now.

And to provide some pushback on things you say, I think now that OPs bridges with OpenAI are thoroughly burned after the Sam firing drama, OP is pretty OK with people criticizing OpenAI (since what social capital is there left to protect here?). My sense is criticizing Anthropic is slightly risky, especially if you do it in a way that doesn't signal what OP considers good judgement on maintaining and spending your social capital appropriately (i.e. telling them that they are harmful for the world, or should really stop, is bad, but doing a mixture of praise and criticism without taking any controversial top-level stance is fine), but mostly also isn't the kind of thing that OP will totally freak out about. I think OP used to be really crazy about this, but now is a bit more reasonable, and it's not the domain where OP's relationship to reputation-management is causing the worst failures.

I think all of this is worse in the longtermist space, though I am not confident. At the present it wouldn't surprise me very much if OP would defund a global health grantee because their CEO endorsed Trump for president, so I do think there is also a lot of distortion and skew there, but my sense is that it's less, mostly because the field is much more professionalized and less political (though I don't know how they think, for example, about funding on corporate campaign stuff which feels like it would be more political and invite more of these kinds of skewed considerations). 

1. ^{^}

   As an example of this, in the announcement of recent Open Phil defunding decisions [EA · GW], they communicated that they are "withdrawing funding from some cause areas", which I was really a quite misleading way to describe what actually happened and basically no one I talked to who read the post understood what happened correctly. If you "withdraw from a cause area" you would expect that if you have an organization that is at the intersection of two cause areas to still be considered based on their impact the cause area that OP is funding (for example AI Alignment and "rationality community building"). However, what actually happened is that Open Phil blacklisted a number of ill-defined broad associations and affiliations, where if you are associated with a certain set of ideas, or identities or causes, then no matter how cost-effective your other work is, you cannot get funding from OP.  This is of course an enormously different situations, with an enormously different set of incentives. But as far as I can tell all the comms here was dictated by Dustin, and nobody from OP felt comfortable clarifying what is actually happening in public, and so tons of people were misled about what actually happened.

I get the sense that we can't trust Open Philanthropy to do a good job on AI safety, and this is a big problem. Many people would have more useful things to say about this than I do, but I still feel that I should say something.

My sense comes from:

• Open Phil is reluctant to do anything to stop the companies that are doing very bad things to accelerate the likely extinction of humanity, and is reluctant to fund anyone who's trying to do anything about it.
• People at Open Phil have connections with people at Anthropic, a company that's accelerating AGI and has a track record of (plausibly-deniable) dishonesty. Dustin Moskovitz has money invested in Anthropic, and Open Phil employees might also stand to make money from accelerating AGI. And I agree with Bryan Caplan's recent take that friendships are often a bigger conflict of interest than money, so Open Phil higher-ups being friends with Anthropic higher-ups is troubling.

A lot of people (including me as of ~one year ago) consider Open Phil the gold standard for EA-style analysis. I think Open Phil is actually quite untrustworthy on AI safety (but probably still good on other causes).

I don't know what to do with this information.

Thanks for the cool discussion Ryan and Nate! This thread seemed pretty insightful to me. Here’s some thoughts / things I’d like to clarify (mostly responding to Nate's comments).^[1]

Who’s doing this trade?

In places it sounds like Ryan and Nate are talking about predecessor civilisations like humanity agreeing to the mutual insurance scheme? But humans aren’t currently capable of making our decisions logically dependent on those of aliens, or capable of rescuing them. So to be precise the entity engaging in this scheme or other acausal interactions on our behalf is our successor, probably a FAI, in the (possibly counterfactual or counterlogical) worlds where we solve alignment.

Nate says:

Roughly speaking, I suspect that the sort of civilizations that aren't totally fucked can already see that "comb through reality for people who can see me and make their decisions logically dependent on mine" is a better use of insurance resources, by the time they even consider this policy.

Unlike us, our FAI can see other aliens. So I think the operative part of that sentence is “comb through reality”—Nate’s envisioning a scenario where with ~85% probability our FAI has 0 reality-fluid before any acausal trades are made.^[2] If aliens restrict themselves to counterparties with nonzero reality-fluid, our FAI doesn’t make the cut.

Note: Which FAI we deploy is unlikely to be physically overdetermined in scenarios where alignment succeeds, and definitely seems unlikely to be determined by more coarse-grained (not purely physical) models of how a successor to present-day humanity comes about. (The same goes for which UFAI we deploy.) I’m going to ignore this fact for simplicity and talk about a single FAI; let me know if you think it causes problems for what I say below.

Trading with nonexistent agents is normal

I do see an argument that agents trying to do insurance with similar motives to ours could strongly prefer to trade with agents who do ex post exist, and in particular those agents that ex post exist with more reality-fluid. It’s that insurance is an inherently risk-averse enterprise.^[3] It doesn’t matter if someone offers us a fantastic but high-variance ex ante deal, when the whole reason we’re looking for insurance is in order to maximise the chances of a non-sucky ex post outcome. (One important caveat is that an agent might be able to do some trades to first increase their ex ante resources, and then leverage those increased resources in order to purchase better guarantees than they’d initially be able to buy.)

On the other hand, I think an agent with increasing utility in resources will readily trade with counterparties who wouldn’t ex post exist absent such a trade, but who have some ex ante chance of naturally existing according to a less informed prior of the agent. I get the impression Nate thinks agents would avoid such trades, but I’m not sure / this doesn’t seem to be explicit outside of the mutual insurance scenario.

There’s two major advantages to trading with ex post nonexistent agents, as opposed to updating on (facts upstream of) their existence and consequently rejecting trade with them:

• Ex post nonexistent agents who are risk-averse w.r.t. their likelihood of meeting some future threshold of resources/value, like many humans seem to be, could offer you deals that are very attractive ex ante. 
• Adding agents who (absent your trade) don’t ex post exist to the pool of counterparties you’re willing to trade with allows you to be much more selective when looking for the most attractive ex ante trades.

The main disadvantage is that by not conditioning on a counterparty’s existence you’re more likely to be throwing resources away ex post. The counterparty needs to be able to compensate you for this risk (as the mugger does in counterfactual mugging). I’d expect this bar is going to be met very frequently.

To recap, I'm saying that for plausible agents carrying out trades with our FAI, Nate's 2^-75 number won't matter. Instead, it would be something closer to the 85% number that matters—an ex ante rather than an ex post estimate of the FAI’s reality-fluid.

But would our FAI do the trade if it exists?

Nate says (originally talking about aliens instead of humanity):

As I understand it, the argument goes something like "your counterfactual picture of reality should include worlds in which your whole civilization turned out to be much much less competent, and so when you imagine the multiverse where you pay for all humanity to live, you should see that, in the parts of the multiverse where you're totally utterly completely incompetent and too poor to save anything but a fraction of your own brethren, somebody else pays to save you".

I agree that doing an insurance trade on behalf of a civilisation requires not conditioning on that civilisation’s competence. Nate implies that aliens’ civilisational competence is “tightly intertwined with [aliens’] basic reasoning faculties”, and this seems probably true for alien or human members of predecessor civilisations? But I don’t know why the civilisational competence of a FAI’s predecessor would be tightly intertwined with the FAI’s cognition. As mentioned above, I think the relevant actor here is our FAI, not our current selves. 

We can further specify civilisational competence (relative to the stakes of alignment) as a function of two variables:

1. Physical facts about a civilisation’s history (i.e. the arrangement of atoms).
2. Logical facts (beyond those accessible to current humans) that govern the relationship between physics, civilisations instantiated via that physics, and what sort of AI certain types of civilisations are likely to deploy.

Either of these when combined with the other provides evidence about what sort of AI a predecessor civilisation deploys, but each will be uninformative on its own. I have in mind that agents executing an insurance trade would condition on all physical facts about their counterparty's civilisation—up until some truncation point that’s plausibly late enough to be capturing our current selves—but would not condition on logical facts necessary to interpret those physical facts into a ~determinate answer as to whether the civilisation solves alignment.

Conditioning on those logical facts sounds pretty analogous to conditioning on a digit of pi to me.

Does this mean we can have nice things?

So overall, I think that aliens or earth-originating UFAIs (who aren’t motivated by insurance) would be pretty interested in trading with our FAI, and vice-versa. Counterparties would discount the FAI’s resources by their prior probability that it’s deployed (before conditioning on factors that pin this down).

Because we’re assuming our FAI would be willing to offer terms that are terrible for us if denominated in measure-weighted resources, counterparties would gain ex ante resources by engaging in an insurance trade with it. Those resources could later be used to engage in trade with others who are themselves willing to (indirectly) trade with nonexistent agents, and who don’t have much more sceptical priors about the deployment odds of our FAI. So because the trade at hand yields a net profit, I don't think it competes much with ordinary alternative demands for counterparties’ resources.

Nevertheless, here’s a few (nonexhaustive) reasons why this trade opportunity wouldn't be taken by another updateless AI:

• The agent has a better trading opportunity which is either sensitive to when in (logical) time they start fulfilling it, or which demands all the agent's current resources (at the time of discovering the trade) without compensating the agent for further profits.
• The expected transaction costs of finding agents like our FAI outweigh the expected benefits from trade with it.
  • This might be plausible for aliens without hypercompute; I don't think it's plausible for earth-originating UFAI absent other effects.
    • …But I'm also not sure how strongly earth-originating AIs converge on UDT, before we select for those with more ex ante resources. Even all UDT earth-originating UFAIs doing insurance trades with their predecessors could be insufficient to guarantee survival.
  • Variation: Contrary to what I expect, maybe doing “sequential” acausal trades are not possible without each trade increasing transaction costs for counterparties an agent later encounters, to an extent that a (potentially small-scale) insurance trade with our FAI would be net-negative for agents who intend to do a lot of acausal trade.
• The agent disvalues fulfilling our end of the trade enough that it's net-negative for it.
• A maybe-contrived example of our FAI not being very discoverable: Assume the MUH. Maybe our world looks equally attractive to prospective acausal traders as an uncountable number of others. If an overwhelming majority of measure-weighted resources in our section of the multiverse is possessed by countably many agents who don’t have access to hypercompute, we'd have an infinitesimal chance of being simulated by one of them.
• Our FAI has some restrictions on what a counterparty is allowed to do with the resources it purchases, which could drive down the value of those resources a lot.

Overall I'd guess 30% chance humanity survives misalignment to a substantial extent through some sort of insurance trade, conditional on us not surviving to a substantial extent another cheaper way?

Other survival mechanisms

I’m pretty uncertain about how Evidential cooperation in large worlds works out, but at my current rough credences I do think there’s a good chance (15%) we survive through something which pattern-matches to that, or through other schemes that look similar but have more substantial differences (10%).

I also put some credence in there being very little of us in base reality, in some of those scenarios could involve substantial survival odds. (Though I weakly think the overall contribution of these scenarios is undesirable for us.) 

1. ^{^}

   Meta: I don’t think figuring out insurance schemes is very important or time-sensitive for us. But I do think understanding the broader dynamics of acausal interactions that determine when insurance schemes would work could be very important and time-sensitive. Also note I'd bet I misinterpreted some claims here, but got to the point where it seemed more useful to post a response than work on better comprehension. (In particular I haven't read much on this page beyond this comment thread.)

2. ^{^}

   I don’t think Nate thinks alignment would be physically overdetermined if misalignment winds up not being overdetermined, but we can assume for simplicity there’s a 15% chance of our FAI having all the reality fluid of the Everett branches we’re in.

3. ^{^}

   I'm not clear on what the goal of this insurance scheme is exactly. Here's a (possibly not faithful) attempt: we want to maximise the fraction of reality-fluid devoted to minds initially ~identical to ours that are in very good scenarios as opposed to sucky ones, subject to a constraint that we not increase the reality-fluid devoted to minds initially ~identical to us in sucky scenarios. I’m kind of sympathetic to this—I think I selfishly care about something like this fraction. But it seems higher priority to me to minimise the reality-fluid devoted to us in sucky / terrible scenarios, and higher priority still to use any bargaining power we have for less parochial goals.

"you're going to be the godfather of that particular concept (like Petr Vopěnka, whose "principle" started as a joke he didn't believe). " That would indeed be hilarious. The Helm-Burger-Porter microtubule computer would haunt me the rest of my days!