LessWrong 2.0 Reader

Context for anyone who missed it: https://en.wikipedia.org/wiki/Gell-Mann_amnesia_effect

If this doesn’t count as an explanation (or at least a concrete hypothesis), what would one look like to you?

Something that inputs a brain state and outputs a quale ie solves the Mary's Room problem. And does it in a principled way, not just a look up table of known correlations.

subjective experience is just how predictive, self-modeling systems represent internal and external states.

To say that X "is just" Y is to say there is no further explanation.

It’s like asking why high kinetic energy “feels” hot. It doesn’t, heat is just how the brain models signals from temperature receptors and maps them into the self-model.

We know how high (random) kinetic energy causes a high reading on a thermometer.

We do not know why this "feels hot" to people but (we presume) not to a thermometer. Or if you think, as some have claimed to, that it might actually "feel hot" to a strand of mercury in a glass tube, how would you go about finding out, given that in the case of a thermometer, we already know all the relevant physical facts about why the line lengthens and shrinks?

Sections 4 and 5 explain why this evolved: it’s a useful way for the brain to prioritize action when reflexes aren’t enough. You “feel” something because that’s how your brain tracks itself and the environment.

This is redefining the word "feel", not accounting for the thing that "feel" ordinarily points to.

The same thing happened to the word "sensation" when mechanisms of the sensory organs were being traced out. The mechanism of how sensations "feel" (the previous meaning of the word "sensation") was never found, and "sensation" came to be used to mean only those physical mechanisms. This is why the word "quale" (pl. qualia) was revived, to refer to what there was no longer a name for, the subjective experience of "sensations" (in the new sense).

The OP, for all its length, appears to be redefining the word "conscious" to mean "of a system, that it contains a model of itself". It goes into great detail and length on phenomena of self-modelling and speculations of why they may have arisen, and adds the bald assertion, passim, that this is what consciousness is. The original concept that it aims and claims to explain is not touched on.

You say consciousness = successful prediction. What happens when the predictions are wrong? 

I'm saying the issue of whether ASI gets out of control is not fundamental to the discussion of whether ASI poses an xrisk or how to avert it.

The control question is not fundamental to discussion of whether ASI poses x-risk—agreed. But as to discussion of how to avert x-risk, the control question is fundamental.

Humanity's optimal strategy for averting x-risk depends on whether we can ultimately control ASI. If control is possible, then the best strategy for averting x-risk is coordination of ASI development—across companies and nations. If control is not possible, then the best strategy is very different and even less well-defined (e.g., pausing ASI development, attempting to seed ASI so that it becomes benevolent, making preparations so humans can live alongside self-directed ASI, etc).

So while it's possible that emphasis on the control question turns many people away from the xrisk conversation, I think the control question remains key for conversation about xrisk solutions.

I think self-supervised learning is perfectly capable of discovering rare but important patterns. Just look at today’s foundation models, which seem pretty great at that.

This I'd dispute. If your model if underparameterized (which I think is true for the typical model?), then it can't learn any patterns that only occurs once in the data. And even if the model is overparameterized, it still can't learn any pattern that never occurs in the data.

I think you’re conflating consequentialism and understanding in a weird-to-me way. (Or maybe I’m misunderstanding.)

I think consequentialism is related to choosing one action versus another action. I think understanding (e.g. predicting the consequence of an action) is different, and that in practice understanding has to involve self-supervised learning.

I'm saying that intelligence is the thing that allows you to handle patterns. So if you've got a dataset, intelligence allows you to build a model that makes predictions for other data based on the patterns it can find in said dataset. And if you have a function, intelligence allows you to find optima for said function based on the patterns it can find in said function.

Consequentialism is a way to set up intelligence to be agent-ish. This often involves setting up something that's meant to build an understanding of actions based on data or experience.

One could in principle cut my definition of consequentialism up into self-supervised learning and true consequentialism (this seems like what you are doing..?). One disadvantage with that is that consequentialist online learning is going to have a very big effect on the dataset one ends up training the understanding on, so they're not really independent of each other. Either way that just seems like a small labelling thing to me.

Typo: should be "Gell-Mann"

I would have wanted more pointing out of institutional capacity as part of the solution to this but I think it is a very good way of describing a more generalised re-focus to not goodharting on sub-parts of the problem. 

Now, that I've said something serious I can finally comment on what I wanted to comment on:

Thanks to Ted Chiang, Toby Ord, and Hannu Rajaniemi for conversations which improved this piece.

Ted Chiang!!! I'm excited for some banger sci-fi based on things more relating to x-risk scenarios, that is so cool!

I knew the author (Michael Nielsen) once but didn't stay in touch... I had a little trouble figuring out what he actually advocates here, e.g. at the end he talks about increasing "the supply of safety", and lists "differential technological development" (Bostrom), "d/acc" (Buterik), and "coceleration" (Nielsen) as "ongoing efforts" that share this aim, without defining any of them. But following his links, I would define those in turn as "slowing down dangerous things, and speeding up beneficial things"; "focusing on decentralization and individual defense"; and "advancing safety as well as advancing capabilities". 

In this particular essay, his position seems similar to contemporary MIRI. MIRI gave up on alignment in favor of just stopping the stampede towards AI, and here Michael is also saying that people who care about AI safety should work on topics other than alignment (e.g. "institutions, norms, laws, and education"), because (my paraphrase) alignment work is just adding fuel to the fire of advances in AI. 

Well, let's remind ourselves of the current situation. There are two AI powers in the world, America and China (and plenty of other nations who would gladly join them in that status). Both of them are hosting a capabilities race in which multiple billion-dollar companies compete to advance AI, and "making the AI too smart" is not something that either side cares about. We are in a no-brakes race towards superintelligence, and alignment research is the only organized effort aimed at making the outcome human-friendly. 

I think plain speaking is important at this late stage, so let me also try to be as clear as possible about how I see our prospects. 

First, the creation of superintelligence will mean that humanity is no longer in control, unless human beings are somehow embedded in it. Superintelligence may or may not coexist with us, I don't know the odds of it emerging in a human-friendly form; but it will have the upper hand, we will be at its mercy. If we don't intend to just gamble on there being a positive outcome, we need alignment research. For that matter, if we really didn't want to gamble, we wouldn't create superintelligence until we had alignment theory perfectly worked out. But we don't live in that timeline. 

Second, although we are not giving ourselves time to solve alignment safely, that still has a chance of happening, if rising capabilities are harnessed to do alignment research. If we had no AI, maybe alignment theory would take 20 or 50 years to solve, but with AI, years of progress can happen in months or weeks. I don't know the odds of alignment getting fully solved in that way, but the ingredients are there for it to happen. 

I feel I should say something on the prospect of a global pause or a halt occurring. I would call it unlikely but not impossible. It looks unlikely because we are in a decentralized no-holds-barred race towards superintelligence already, and the most advanced AIs are looking pretty capable (despite some gaps e.g. 1 [LW · GW] 2 [LW · GW]), and there's no serious counterforce on the political scene. It's not impossible because change, even massive change, does happen in politics and geopolitics, and there's only a finite number of contenders in the race (though that number grows every year). 

I agree this stuff is addictive. AI makes things more interactive. Some [LW · GW] people [LW · GW] who never considered themselves vulnerable got sucked in to AI relationships.

Possible push back:

What if short bits of addictive content generated by humans (but selected by algorithms) are already near max addictiveness? And by the time AI can design/write a video game etc. twice as addictive than humans can design, we already have a superintelligence explosion, and either addiction is solved or we are dead?