LessWrong 2.0 Reader

Analogously: "I am claiming that people when informed will want horses to continue being the primary mode of transportation. I also think that most people when informed will not really care that much about economic growth, will continue to believe that you're more responsible for changing things than for maintaining the status quo, etc. And that this is a coherent view that will add up to a large set of people wanting things in cities to remain conservatively the same. I separately claim that if this is true, then other people should just respect this preference, and go find new continents / planets on which to build cars that people in the cities don't care about."

Sometimes it's good to be conservative when you're changing things, like if you're changing lots of social norms or social institutions, but I don't get it at all in this case. The sun is not a complicated social institution, it's primarily a source of heat and light and much of what we need can be easily replicated especially when you have nanobots. I am much more likely to grant that we should be slow to change things like democracy and the legal system than I am that we should change exactly how and where we should get heat and light. Would you have wanted conservatism around moving from candles to lightbulbs? Installing heaters and cookers in the house instead of fire pits? I don't think so.

Sure, but one can assess it at any point. I'm not asking about whether he will end up being net-positive or net-negative overall in the long run.

I'd agree. But he certainly does not seem to even be trying anymore to have positive impact on solving alignment, no?

Hi, thanks.

I don't see how what you say contradicts that the reach of his actions and opinions have increased. Did you maybe quote the wrong sentence?

Thanks so much for engaging, Nathan!

The pivotal act was defined by Yudkowsky, I'm just borrowing the definition. The idea is that even after you've built a perfectly aligned superintelligent AI, you only have about 6 months before someone else builds an unaligned superintelligent AI. That's probably not enough time to convince the entire world to adopt a better governance system before getting atomized by nanobots. So your aligned AI would have to take over the world and forcefully implement this better governance system within a span of a few months.

Claude Sonnet 3.6 is worthy of sainthood!

But as I mention in my other comment I'm concerned that such an AI's internal mental state would tend to become cynical or discordant as intelligence increases.

I'm going to give an argument in favor of computationalism as a theory of identity, but first I'm going to set up some of the stuff that I take as background, but others may not, and will hopefully illuminate related issues.

I believe computationalism is a very general way to look at effectively everything, because a model of computation can be extremely expressive, and a big reason for why philosophical debates go nowhere is that they don't realize that expressive enough models of computation can essentially trivialize any problem they set up, solely through the expressiveness of the model, so it's very important to realize that philosophical problems end up trivial if we allow ourselves to be unconstrained.

More here:

http://www.amirrorclear.net/academic/ideas/simulation/index.html

http://www.amirrorclear.net/academic/research-topics/other-topics/hypercomputation.html

https://arxiv.org/abs/1806.08747

This also answers andeslodes's point around physicalism, as the physicalist ontology is recoverable as a special case of the computationalist ontology, where we care about each instantiation, and we also care about low-level things that are always simulatable by powerful models of computation.

However, I do think there's an actually non-trivial question here, about why can we be confident that the brain is copyable by a realistic classical computer with reasonable fidelity, when in principle it could be using quantum mechanics for a non-trivial portion of the brain's operations, which would complicate mind uploading massively?

My general answer here is that quantum computation in warm, wet environments is very hard to error-correct at scale, and the biologically plausible ways to do such a thing fundamentally cannot scale to make a quantum computer in our brains, and anything that is fragile in the presence of noise/error is basically unevolvable by default, because there's no incremental way to make a quantum computer inside a human brain.

That's why we can reasonably use classical models of the brain without it leaving a lot out.

As far as my own theory of identity/consciousness goes, I think the gooder regulator theorem gives am answer on why we would want to have a self-model of ourselves, and I'd argue this is the seed from which consciousness/identity grows, so computations meeting the requirements in the gooder regulator theorem can certainly have identities, which means that I do think Rob Bensinger is more or less right about this topic, modulo caveats.

The caveats are given below:

https://www.lesswrong.com/posts/Dx9LoqsEh3gHNJMDk/fixing-the-good-regulator-theorem#Takeaway [LW(p) · GW(p)]

The main functions of consciousness/identity is to give a consistent abstraction across time so that cooperation and deal making become possible, at least for causal decision theories, at least without stuff like provability logic/program equilibrium, which isn't really evolvable (for other theories like logical decision theory, cooperation in one-shot PD is very possible, but that comes about because they view their identities as closer to an isomorphism class of a program, and consider all instances combined as 1 program, so long as they are functionally equivalent), combined with the necessity of self-modeling yourself in order to get a good outcome, though the self-model here is the more important part for me on consciousness, and the abstraction to a consistent identity comes later.

So for my purposes, I think Rob Bensinger is mostly right about the philosophical underpinnings of identity/consciousness, but I don't blame TAG, @andeslodes [LW · GW] and @sunwillrise [LW · GW] for questioning the consensus, since the consensus was frankly poorly argued, and @Rob Bensinger [LW · GW] was acting far too much like a soldier rather than a scout in the comments section.

Mostly fair, but tiers did have a slight other impact in that they were used to bias the final room: Clay Golem and Hag were equally more-likely to be in the final room, both less so than Dragon and Steel Golem but more so than Orcs and Boulder Trap.

I am claiming that people when informed will want the sun to continuing being the sun. I also think that most people when informed will not really care that much about creating new people, will continue to believe in the act-omission distinction, etc. And that this is a coherent view that will add up to a large set of people wanting things in the solar system to remain conservatively the same. I seperately claim that if this is true, then other people should just respect this preference, and use the other stars that people don't care about for energy.

What I mainline expect is that yes, a few OOMs more of compute and efficiency will unlock a bunch of new things to try, and yes some of those things will make some capabilities go up a bunch, in the theme of o3. I just also expect that to level off. I would describe myself as "confident but not extremely confident" of that; like, I give 1 or 2% p(doom) in the next 10ish years, coming from this possibility (and some more p(doom) from other sources). Why expect it to level off? Because I don't see good evidence of "a thing that wouldn't level off"; the jump made by LLMs of "now we can leverage huge amounts of data and huge amounts of compute at all rather than not at all" is certainly a jump, but I don't see why to think it's a jump to an unbounded trajectory.