LessWrong 2.0 Reader

I disagree with some of the claims made here, and I think there several worldview assumptions that go into a lot of these claims. Examples include things like "what do we expect the trajectory to ASI to look like", "how much should we worry about AI takeover risks", "what happens if a single actor ends up controlling [aligned] ASI", "what kinds of regulations can we reasonably expect absent some sort of centralized USG project", and "how much do we expect companies to race to the top on safety absent meaningful USG involvement." (TBC though I don't think it's the responsibility of the authors to go into all of these background assumptions– I think it's good for people to present claims like this even if they don't have time/space to give their Entire Model of Everything.)

Nonetheless, I agree with the bottom-line conclusion: on the margin, I suspect it's more valuable for people to figure out how to make different worlds go well than to figure out which "world" is better. In other words, asking "how do I make Centralized World or Noncentralized World more likely to go well" rather than "which one is better: Centralized World or Noncentralized World?"

More specifically, I think more people should be thinking: "Assume the USG decides to centralize AGI development or pursue some sort of AGI Manhattan Project. At that point, the POTUS or DefSec calls you in and asks you if you have any suggestions for how to maximize the chance of this going well. What do you say?"

One part of my rationale: the decisions about whether or not to centralize will be much harder to influence than decisions about what particular kind of centralized model to go with or what the implementation details of a centralized project should look like. I imagine scenarios in which the "whether to centralize" decision is largely a policy decision that the POTUS and the POTUS's close advisors make, whereas the decision of "how do we actually do this" is something that would be delegated to people lower down the chain (who are both easier to access and more likely to be devoting a lot of time to engaging with arguments about what's desirable.)

Maybe it can be good to have a "add post to sequence" option when you click the context menu on a post. That's more intuitive than going to the library page.

You disagree with Aaronson that the location of the complexity is in the interpreter, or you disagree that it matters?

In the first case, I'll defer to him as the expert. But in the second, the complexity is an internal property of the system! (And it's a property in a sense stronger than almost anything we talk about in philosophy; it's not just a property of the world around us, because as Gödel and others showed, complexity is a necessary fact about the nature of mathematics!)

Yeah, something like that. See my response to Euan in the other reply to my post.

Yes, and no, it does not boil down to Chalmer's argument. (as Aaronson makes clear in the paragraph before the one you quote, where he cites the Chalmers argument!) The argument from complexity is about the nature and complexity of systems capable of playing chess - which is why I think you need to carefully read the entire piece and think about what it says.

But as a small rejoinder, if we're talking about playing a single game, the entire argument is ridiculous; I can write the entire "algorithm" a kilobyte of specific instructions. So it's not that an algorithm must be capable of playing multiple counterfactual games to qualify, or that counterfactuals are required for moral weight - it's that the argument hinges on a misunderstanding of how complex different classes of system need to be to do the things they do.

I don’t think this is too related to the OP, but in regard to your exchange with jbash:

I think there’s a perspective where “personal identity” is a strong intuition, but a misleading one—it doesn’t really (“veridically”) [LW · GW] correspond to anything at all in the real world. Instead it’s a bundle of connotations, many of which are real and important. Maybe I care that my projects and human relationships continue, that my body survives, that the narrative of my life is a continuous linear storyline, that my cherished memories persist, whatever. All those things veridically correspond to things in the real world, but (in this perspective) there isn’t some core fact of the matter about “personal identity” beyond that bundle of connotations.

I think jbash is saying (within this perspective) that you can take the phrase “personal identity”, pick whatever connotations you care, and define “personal identity” as that. And then your response (as I interpret it) is that no, you can’t do that, because there’s a core fact of the matter about personal identity, and that core fact of the matter is very very important, and it’s silly to define “personal identity” as pointing to anything else besides that core fact of the matter.

So I imagine jbash responding that “do I nonetheless continue living (in the sense of, say, anticipating the same kind of experiences)?” is a confused question, based on reifying misleading intuitions around “I”. It’s a bit like saying “in such-and-such a situation, will my ancestor spirits be happy or sad?”

I’m not really defending this perspective here, just trying to help explain it, hopefully.

But this just depends on how broad this set is. If it contains two brains, one thinking about the roman empire and one eating a sandwich, we're stuck.

I suspect that if you do actually follow Aaronson (as linked by Davidmanheim) to extract a unique efficient calculation that interacts with the external world in a sensible way, that unique efficient externally-interacting calculation will end up corresponding to a consistent set of experiences, even if it could still correspond to simulations of different real-world phenomena.

But I also don't think that consistent set of experiences necessarily has to be a single experience! It could be multiple experiences unaware of each other, for example.

If we apply the Scott Aaronson waterfall counterargument to your Alice-bot-and-Bob-bot scenario, I think it would say: The first step was running Alice-bot, to get the execution trace. During this step, the conscious experience of Alice-bot manifests (or whatever). Then the second step is to (let’s say) modify the Bob code such that it does the same execution but has different counterfactual properties. Then the third step is to run the Bob code and ask whether the experience of Alice-bot manifests again.

But there’s a more basic question. Forget about Bob. If I run the Alice-bot code twice, with the same execution trace, do I get twice as much Alice-experience stuff? Maybe you think the answer is “yeah duh”, but I’m not so sure. I think the question is confusing, possibly even meaningless. How do you measure how much Alice-experience has happened? The “thick wires” argument (I believe due to Nick Bostrom, see here, p189ff, or shorter version here [LW · GW]) seems relevant. Maybe you’ll say that the thick-wires argument is just another reductio about computational functionalism, but I think we can come up with a closely-analogous “thick neurons” thought experiment that makes whatever theory of consciousness you subscribe to have an equally confusing property.

Dramatically increasing taxes for childless people.

Too-low fertility concerns me deeply. My current preferred strategy had been something financial along the lines of this proposal, but on reflection, I think I need to update.

The main reasons that I see driving the people around me to defer/eschew children are, in rough decreasing order of prevalence:

• Lack of a suitable partner.
• Lack of fertility.
• Expectation of being an incompetent parent.
• Severe untreated mental health ailments.
• Other career priorities, not compatible with child raising.

The third point above is, in my social circle, usually downstream of needing to spend too much time working, not being able to afford childcare services, not being able to afford college, etc.

These issues could be lessened by financial interventions with the net effect of offsetting the burdens of child raising (obviously, implementation details matter a great deal.)

...I had previously assumed that "expectation of incompetence" was the primary issue because it's my primary issue (in the sense of looming large in my mind). ...but having taken an inventory of the childless adults in my social network, I now see that "inability to find a suitable partner" and "infertility" are much bigger issues.

"Inability to find a suitable partner" seems risky to fix, because it looks easy to create horrible side-effects. Many in my social network are in relationships but aren't having children because they're abusive relationships. Others have successfully escaped abusive relationships and subsequently given up on finding a partner; being alone is less painful, and they expect to find another abuser with nontrivial probability if they go looking for another partner. Still others lost a suitable partner to a treatable disease, due primarily to insurance companies ending up de facto in charge of healthcare decisions, and rendering decisions without the patient's health as the priority.

I don't see a good solution. (Well, except for "don't allow insurers to drive healthcare decisions".) All attempts to "solve" this issue I see being promoted in real life seem on-net harmful. My home state of Texas thinks the best solution is to ban divorce, and we expect that to be implemented within the next year. I don't see that as being a net good, even if it may compel additional births in the short term on net. Trapping people in abusive relationships seems incredibly dystopian. I also foresee this making young women even more afraid of getting into the wrong relationship than they already are, which runs the risk of making the "finding suitable partners" problem worse in the long run.

"Lack of fertility" seems challenging to fix, but far less fraught. I was born without a uterus, and wish to bear a child if the medical technology were developed to allow me to grow a uterus. I have a sibling on life-saving medication, a side-effect of which is extremely low sperm count -- a better drug to treat the condition (or an actual cure) could resolve this issue. Multiple of my sisters-in-law have simply failed to conceive for years, due to unknown-to-me causes, and I suspect the issues are similarly fixable via continued medical innovation.

I don’t think Premise 2 is related to my comment. I think it’s possible to agree with premise 2 (“there is an objective fact-of-the-matter whether a conscious experience is occurring”), but also to say that there are cases where it is impossible-in-practice for aliens to figure out that fact-of-the-matter.

By analogy, I can write down a trillion-digit number N, and there will be an objective fact-of-the-matter about what is the prime factorization of N, but it might take more compute than fits in the observable universe to find out that fact-of-the-matter.