LessWrong 2.0 Reader

humanity as a whole probably doesn't want things to get incredibly crazy so fast, and so we're likely to see something tamer

Doesn't this require a pretty strong and unprecedented level of international coordination on stopping an obviously immediately extremely valuable and militarily relevent technology? I think a US backed entente could impose this on the rest of the world, but that would also be an unprecedentedly large effort.

I think this is certainly possible and I hope this level of coordination happens, but I don't exactly think this is likely in timelines this short.

More minimally, I think the vast majority of readers won't read this essay and understand that this refers to a world in which there was a massive effort to slow down AI or in which AI was quite surprisingly slow in various ways. (Insofar as this is what Dario thinks.) Dario seems to be trying to make realistic/plausible predictions and often talks about what is "possible" (not what is possible with a desirable level of AI progress), so not mentioning this seems to greatly undermine the prediction aspect of the essay.

oh cool!! i'm glad :-)

I still struggle with some of my old problems – particularly anxiety, loneliness and occasional depression

I still struggle with these, as well. Another LessWrong user (@Chipmonk [LW · GW]) mentioned in a post [LW · GW] the book Already Free: Buddhism Meets Psychotherapy on the Path of Liberation. After taking his recommendation and reading that book, it gave me a new perspective that I don't have to run away from these negative emotions. In fact, not only can I openly accept them, but when I chose to actively love them, it liberated my thought space and allowed me to non-judgmentally witness these emotions.

The book also helped me to finally understand and appreciate author Tara Brach's short story about Buddha and Mara.

Anecdotally I've found that it's generally on point. 

I think a lot of these tactics are a form of ego-pandering that distracts people from what their organizations are meant to achieve (elaborate speeches, doing things through "proper channels" that serves to give more people a voice, etc.). I've been in several organizations where decisions take forever to be made, circulating between individuals and committees with no one really holding a final say in making the decision, waiting for some form of consensus to arrive (which it never truly does). This again just gives people more and more of an excuse to insert themselves in discussions that are happening often to the detriment of actually moving things along.

I basically agree with everything in section 2.2, but I read section 1.4 again (which i also agree with as written there), and I just wanted to point out that I think the kind of self-modelling described in section 2.2 is in some ways unusual and happens because of smart/social-animal-specific brain programming and not just pure pressure for generative models to form self-models to make better predictions. (Though I think humans are probably smart enough that decent self-models would also have been formed for those reasons you described in section 1.4, but I think humans have quite some pre-wired machinery for self-modelling.)

In most animals the cortex mostly just models the world I think, and there are parts of the cortex that model other parts of the cortex which in the end just translates into making better predictions about the world. (This also happens in humans tbc.)

What's described in section 2.2 though is that there's exists a "(this) mind" concept and the cortex forms beliefs about it by creating some abstract labels for thoughts and using those to form new thoughts like "abstract-label-of-thought-X is present in the mind". (For non-human animals which do such self-modelling they probably don't model thoughts but rather intentions/desires or so.)

This is unusual because initially the model of the mind is separated from the model of the model of the world. So far the model of the mind is a separate magisterium which isn't immediately useful for making predictions about the world.

However there are bridging laws between a model of a mind and the physical world, e.g. emotion <-> facial expression predictive bindings. 

Evolutionarily, I think overall this kind of modelling the mind as separate entity turns out to be useful because the same machinery can be used to model and predict conspecifics. Aka such smart/social animals don't just model the world but model the world with some minds (of conspecifics) embedded in there, where the minds are treated as special cases because better predictions about conspecifics can be made from generalizing how the own mind (which can be observed in way more detail) operates.

In humans this whole mind modelling is especially sophisticated (e.g. humans seem to at least sometimes be aware of their own awareness) and there are many aspects I don't understand yet. I think a major part why the homunculus exists might be as interface between the model of mind and the model of the world.

But I'm still somewhat confused here and I also don't quite know what your position here is.

That's fair. 

Maybe, I think it's hard to say how captiolism would work if everyone had zero empathy or compassion.

But that doesn't matter for the issue at hand.

Greed or empathy aside, capitalism currently works because people have capabilities that can't be expanded without limit and people can't be created quickly using capitol.

If ai labor can do every task for a thousandth the cost, and new lai labor created at need, we all die if competition is the system. We will be employed for zero tasks. The factor you mention, sub ASI systems, makes the situation worse, not better.

Maybe you're saying we'd be employed for a while, which might be true. But in the limit, even an enhanced human is only going to have value as a novelty. Which ai probably won't care about if it isn't aligned at all. And even if it does, that leads to a few humans surviving as performing monkeys.

I just don't see how else humans remain competitive with ever improving machines untethered to biology.

If UBI were being offered across the US, I would expect them to rise by the amount of UBI. 

For subsidies per purchase, maybe. 

But not for subsidies per human.

Imagine some prefab tiny house off the grid somewhere in a food desert. I don't think its rent will go up by the UBI amount.

Also, there are houses that house two people (or more!). If there's limited supply in comparison to the demand, I'd expect that the costs of those might go up by more than UBI (because there's two people's worth of UBI as extra budget available).

I wanted to get some perspective on my life so I wrote my own obituary (in a few different ways).

They ended up being focussed my relationship with ambition. The first is below:

Auto-obituary attempt one:

Thesis title: “The impact of the life of Toby Jolly”
a simulation study on a human connected to the early 21st century’s “Effective Altruism” movement

Submitted by:
Dxil Sind 0239β
for the degree of Doctor of Pre-Post-humanities
at Sopdet University 
August 2542

Abstract
Many (>500,000,000) papers have been published on the Effective Altruism (EA) movement, its prominent members and their impact on the development of AI and the singularity during the 21st century’s time of perils. However, this is the first study of the life of Toby Jolly; a relatively obscure figure who was connected to the movement for many years. Through analysing the subject’s personal blog posts, self-referential tweets, and career history, I was able to generate a simulation centred on the life and mind of Toby. This simulation was run 100,000,000 times with a variety of parameters and the results were analysed. In the thesis I make the case that Toby Jolly had, through his work, a non-zero, positively-signed impact on the creation of our glorious post-human Emperium (Praise be to Xraglao the Great). My analysis of the simulation data suggests that his impact came via a combination of his junior operations work, and minor policy projects but also his experimental events and self-deprecating writing.

One unusual way he contributed was by consistently trying to draw attention to how his thoughts and actions were so often the product of his own absurd and misplaced sense of grandiosity; a delusion driven by what he would describe himself as a “desperate and insatiable need to matter”. This work marginally increased the self-awareness and psychological flexibility amongst the EA community. This flexibility subsequently improved the movement's ability to handle its minor role in the negotiations needed to broker power during the Grand Transition - thereby helping avoid catastrophe.

The outcomes of our simulations suggest that through his life and work Toby decreased the likelihood of a humanity-ending event by 0.0000000000024%. He is therefore responsible for an expected 18,600,000,000,000,000,000 quality adjusted experience years across the light-cone, before the heat-death of the universe (using typical FLOP standardisation). Toby mattered.

Ethics note: as per standard imperial research requirements, we asked the first 100 simulations of Toby if they were happy being simulated. In all cases, he said “Sure, I actually, kind of suspected it…look, I have this whole blog about it”

See my other auto-obituaries here :)

Stephen puts it elegantly. Though for me who is more of a code monkey, I'd like to think of it as "Runtime Non-Zero cost type safety through some const generics".