shminux's Shortform

A sufficiently godlike AI could probably convince me to kill myself (or something equivalent, for example to upload myself to a simulation... and once all humans get there, the AI can simply turn it off). Or to convince me not to have kids (in a parallel life where I don't have them already), or simply keep me distracted every day with some new shiny toy so that I never decide that today is the right day to have unprotected sex with another human and get ready for the consequences.

But it would be much easier to simply convince someone else to kill me. And I think the AI will probably choose the simpler and faster way, because why not. It does not need a complicated way to get rid of me, if a simple way is available.

This is similar to reasoning about cults or scams. Yes, some of them could get me, by being sufficiently sophisticated, accidentally optimized for my weaknesses, or simply by meeting me on a bad day. But the survival of a cult or a scam scheme does not depend on getting me specifically; they can get enough other people, so it makes more sense for them to optimize for getting many people, rather than optimize for getting me specifically.

The more typical people will get the optimized mind-hacking message. The rest of us will then get a bullet.

I do think the terminology of "hacks" and "lethal memetic viruses" conjures up images of an extremely unnatural brain exploits when you mean quite a natural process that we already see some humans going through. Some monks/nuns voluntarily remove themselves from the gene pool and, in sects that prioritise ritual devotion over concrete charity work, they are also minimising their impact on the world.

My prior is this level of voluntary dedication (to a cause like "enlightenment") seems difficult to induce and there are much cruder and effective brain hacks available.

I expect we would recognise the more lethal brain hacks as improved versions of entertainment/games/pornography/drugs. These already compel some humans to minimise their time spent competing for resources in the physical world. In a direct way, what I'm describing is the opposite of enlightenment. It is prioritising sensory pleasures over everything else.

Superficially, human minds look like they are way too diverse for that to cause human extinction by accident. If new ideas toast some specific human subgroup, other subgroups will not be equally affected.

None of it would look like it was precipitated by an AI taking over.

But, to be clear, in this scenario it would in fact be precipitated by an AI taking over? Because otherwise it's an answer to "humans went extinct, and also AI took over, how did it happen?" or "AI failed to prevent human extinction, how did it happen?"

It's set in about 1995...

Harry is not meant to be a logically omniscient god who never makes any mistakes. Even on its own terms, “thirty-five years of hard work” is really not that long, nowhere near long enough for Harry to rightly believe the problem is so hard that magic must be pulling some crazy impossible to understand bullshit to accomplish the feat, and transparently so. Harry’s young, and in fact doesn’t have much historical perspective.

He explains why two tweets down the thread.

The idea of a "p(doom)" isn't quite as facially insane as "AGI timelines" as marker of personal identity, but (1) you want action-conditional doom, (2) people with the same numbers may have wildly different models, (3) these are pretty rough log-odds and it may do violence to your own mind to force itself to express its internal intuitions in those terms which is why I don't go around forcing my mind to think in those terms myself, (4) most people haven't had the elementary training in calibration and prediction markets that would be required for them to express this number meaningfully and you're demanding them to do it anyways, (5) the actual social role being played by this number is as some sort of weird astrological sign and that's not going to help people think in an unpressured way about the various underlying factual questions that ought finally and at the very end to sum to a guess about how reality goes.

This seems very reasonable to me, and I think it's a very common opinion about [edit: I meant among] AI safety people that discussing p(doom) numbers without lots of underlying models is not very useful.

The important part of Eliezer's writing on probability IMO is to notice that the underlying laws of probability are Bayesian and do sanity checks, not to always explicitly calculate probabilities. Given that it's only kinda useful in life in general, it is reasonable that (4) and (5) can make trying it net negative.

see https://www.lesswrong.com/posts/AJ9dX59QXokZb35fk/when-not-to-use-probabilities [LW · GW]

So in your opinion, there is no reason why anything happens?

It seems rather obvious to me that there are no "Platonic" laws of physics, and there is no Platonic math existing in some ideal realm.

I am rather uncertain about what it means for something to "exist", as a stand-alone form. When people use this word, it often seems to end up referring to a free-floating belief [? · GW] that does not pay rent in anticipated experiences [? · GW]. 

Is there anything different about the world that I should expect to observe depending on whether Platonic math "exists" in some ideal realm? If not, why would I care about this topic once I have already dissolved [LW · GW] my confusion about what beliefs are meant to refer to? What could serve as evidence one way or another when answering the question of whether math "exists"?

By contrast, we can talk about "reality" existing separately from our internal conception of it because the map is not the territory [LW · GW], which has specific observable consequences, as Eliezer beautifully explained in a still-underrated post [LW · GW] from 17 years ago:

I pause. “Well . . .” I say slowly. “Frankly, I’m not entirely sure myself where this ‘reality’ business comes from. I can’t create my own reality in the lab, so I must not understand it yet. But occasionally I believe strongly that something is going to happen, and then something else happens instead. I need a name for whatever-it-is that determines my experimental results, so I call it ‘reality’. This ‘reality’ is somehow separate from even my very best hypotheses. Even when I have a simple hypothesis, strongly supported by all the evidence I know, sometimes I’m still surprised. So I need different names for the thingies that determine my predictions and the thingy that determines my experimental results. I call the former thingies ‘belief,’ and the latter thingy ‘reality.’ ” - The Simple Truth [LW · GW]

(Bolding mine) If the bolded section was not correct, i.e., if you were an omniscient being whose predictions always panned out in actuality, you would likely not need to keep track of reality as a separate concept from the inner workings of your mind, because... reality would be the same as the inner workings of your mind. But because this is false, and all of us are bounded, imperfect, embedded [LW · GW] beings without the ability to fully understand what is around us, we need to distinguish between "what we think will happen" and "what actually happens."

Later on in the thread, you talked about "laws of physics" [LW · GW] as abstractions written in textbooks, made so they can be understandable to human minds. But, as a terminological matter, I think it is better to think of the laws of physics as the rules [LW · GW] that determine how the territory functions, i.e., the structured, inescapable [LW · GW] patterns guiding how our observations come about, as opposed to the inner [LW · GW] structure of our imperfect maps that generate our beliefs. 

From this perspective, Newton's 3rd law, for example, is not a real "law" of physics, for we know it can be broken (it does not perfectly represent the underlying reality). Neither is General Relativity or QFT, partly because we know they are at odds with one another in particular regimes and do not always give predictions that align with our observations. The "laws" of physics would be restricted to the collection of rules that always output correct predictions (which might very well be an empty set if, for example, reductionism [LW · GW] ends up being false and we turn out to live in a world devoid of perfect underlying structure).

I don't understand how one part of the post relates to another. Yeah, sure, computational irreducibility of the world can make understanding of the world impossible and this would be sad. But I don't see what it has to do with "Platonic laws of physics". Current physics, if anything else, is sort of antiplatonic: it claims that there are several dozens of independent entities, actually existing, called "fields", which produce the entire range of observable phenomena via interacting with each other, and there is no "world" outside this set of entities. "Laws of nature" are just "how this entities are". Outside very radical skepticism I don't know any reasons to doubt this worldview.

If you can distill, store and reproduce this experience on demand, what remains?

As lc has said [LW(p) · GW(p)]:

For the record: the purpose of having a "theory of consciousness" is so it can tell us which blobs of matter feel particular things under which specific circumstances, and teach others how to make new blobs of matter that feel particular things. Down to the level of having a field of AI anaesthesiology.

So at least part of what remains, for example, is the task of figuring out, with surgical precision, whether any given LLM (or other AI agent) is "conscious" in any given situation we place it in. This is because your proposal, although it would massively increase our understanding of human consciousness, seems to me to depend on the particular neural configuration of human minds ("stimulating [human] neurons") and need not automatically generalize to all possible minds.

I think this restates the hard problem, rather than reducing it.  

Suppose we understand brain well enough to figure out what makes one experience specific qualia

We first have to define and detect qualia.  As long as it's only self-reported, there's no way to know if two person's qualia are similar, nor how to test a "qualia transducer".

Email didn't entirely kill fax machines or paper records. For similar reasons, I expect that LLMs will not entirely kill computer languages.

Also, I expect things to go the other direction - I expect that as LLMs get better at writing code, they will generate enormous amounts of one-off code. For example, one thing that is not practical to do now but will be practical to do in a year or so is to have sales or customer service webpages where the affordances given to the user (e.g. which buttons and links are shown, what data the page asks for and in what format) will be customized on a per-user basis. For example, when asking for payment information, currently the UI is almost universally credit card number / cvv / name / billing street address / unit / zipcode / state. However, "hold your credit card and id up to the camera" might be easier for some people, while others might want to read out that information, and yet others might want to use venmo or whatever, and a significant fraction will want to stick to the old form fields format. If web developers developed 1,000x faster and 1,000x as cheaply, it would be worth it to custom-develop each of these flows to capture a handful of marginal customers. But forcing everyone to use the LLM interface would likely cost customers.

Sounds like the afterlife is designed to punish neurotics and reward psychopaths.

.

Angel of regret: "Time for your regular dose of remorse, Joe. Remember that one day when you horribly murdered dozen hostages?"

Psychopath Joe: "Haha, LOL, those were fun days!"

Nietzsche: "That's my boy! Too bad we don't have realistic pre-afterlife simulators here. Eternal recurrence ftw!"

Angel of regret: "I hate my job..."

(Plot twist: Angel of regret is Sisyphus reincarnated.)