LessWrong 2.0 Reader

Nobody has a deal where they'll pay you to not take an offer from an AI lab right? I realize that would be weird incentives, just curious.

Mod note: I've put this on Personal rather than Frontapge. I imagine the content of these talks will be frontpage content, but event announcements in general are not.

This came from a Facebook thread where I argued that many of the main ways AI was described as failing fall into few categories (John disagreed). 

I appreciated this list, but they strike me as fitting into a few clusters.
 

...I would flag that much of that is unsurprising to me, and I think categorization can be pretty fine.

In order:

1) If an agent is unwittingly deceptive in ways that are clearly catastrophic, and that could be understood by a regular person, I'd probably put that under the "naive" or "idiot savant" category. As in, it has severe gaps in its abilities that a human or reasonable agent wouldn't. If the issue is that all reasonable agents wouldn't catch the downsides of a certain plan, I'd probably put that under the "we made a pretty good bet given the intelligence that we had" category.

2) I think that "What Failure Looks Like" is less Accident risk, more "Systemic" risk. I'm also just really unsure what to think about this story. It feels to me like it's a situation where actors are just not able to regulate externalities or similar.

3) The "fusion power generator scenario" seems like just a bad analyst to me. A lot of the job of an analyst is to flag important considerations. This seems like a pretty basic ask. For this itself to be the catastrophic part, I think we'd have to be seriously bad at this. ("i.e. Idiot Savant")

4) STEM-AGI -> I'd also put this in the naive or "idiot savant" category.

5) "that plan totally fails to align more-powerful next-gen AGI at all" -> This seems orthogonal to "categorizing the types of unalignment". This describes how incentives would create an unaligned agent, not what the specific alignment problem is. I do think it would be good to have better terminology here, but would probably consider it a bit adjacent to the specific topic of "AI alignment" - more like "AI alignment strategy/policy" or something.

6) "AGIs act much like a colonizing civilization" -> This sounds like either unalignment has already happened, or humans just gave AIs their own power+rights for some reason. I agree that's bad, but it seems like a different issue than what I think of as the alignment problem. More like, "Yea, if unaligned AIs have a lot of power and agency and different goals, that would be suboptimal"

7) "but at some point a particular subagent starts self-improving, goes supercritical, and takes over the rest of the system overnight." -> This sounds like a traditional mesa-agent failure. I expect a lot of "alignment" with a system made of a bunch of subcomponents is "making sure no subcomponents do anything terrible." Also, still leaves open the specific way this subsystem becomes/is unaligned.

8 ) "using an LLM to simulate a whole society. " -> Sorry, I don't quite follow this one.

Personally, I like the focus "scheming" has. At the same time, I imagine there are another 5 to 20 clean concerns we should also focus on (some of which have been getting attention).

While I realize there's a lot we can't predict, I think we could do a much better just making lists of different risk factors and allocating research amongst them. 

Oh, fair enough then, I trust your visibility into this.  Nonetheless one Should Can Just Report Bugs

My most recent forecasts:

My comment was based on the essay, not on your summary (which I also read, of course). (You did link the essay in your post…)

I think this is wrong for a lot of reasons.

• Do you believe that is a good outcome? →‬ If not, do you believe an ASI aligned to you would? →‬ If not, are you expecting an ASI aligned to 'all of present human culture'? →‬ If not: If it were aligned to Elon Musk in particular, would they really endorse {the lightcone-wide enforcement of a particular culture} post-augmentation? →‬ If so, they in any case do not need to try to promote their preferred views in the near-term for a first decisive AI aligned to themself to be able to cause that.^[1]
• In most futures I expect, there won't be a singular "the post-singularity culture"; either eventually something better happens (note the reachable universe is vast enough to harmlessly contain many cultures/spiritualities) or an unaligned decisive AI takes over.
  • Why do you expect a singular post-singularity culture, unless you are expecting the first decisive AI to be aligned to some evil person who wants that?
• I guess that Musk either (1) feigns concern over current politicized topics to gain political power/allies, or (2) is actually concerned, having maybe self-deceived into thinking they care until they actually became an entity which does, and is in any case acting irrationally according to their own values.

1. ^{^}

   Some of these conditionals (e.g. the first two) are conjunctive, but most are disjunctive, by the way.

An interesting analogy, closer to ML, would be to look at neuroscience. It's an older field than ML, and it seems that the physics perspective has been fairly productive, even though not successful at providing a grand unified theory of cognition yet. Some examples: 

• Using methods from electric circuits to explain neurons (Hodgkin-Huxley model, cable theory)
• Dynamical systems to explain phenomena like synchronization in neuronal oscillations (ex: Kuramoto model)
• Ising models to model some collective behaviour of neurons
• Information theory is commonly used in neuroscience to analyze neural data and model the brain (ex: efficient coding hypothesis)
• Attempts at general theories of cognition like predictive processing, or the free energy principle which also have a strong physics inspiration (drawing from statistical physics, and the least action principle) 

I can recommend the book Models  of the Mind, from Grace Lindsay, which gives an overview of the many way physics contributed to neuroscience. 

In principle, one might think that it would be easier to make progress using a physics perspective on AI than in neuroscience, for example because it is easier to do experiments in AI (in neuroscience we do not have access to the value of the weights, we do not always have access to all the neurons, and often it is not possible to intervene on the system).

A few key points…

1) Based on analogy with the human brain (which is quite puny in terms of energy & matter) & also based on examination of current trends, merely super human intelligence should not be especially costly.

(It is of course hypothetically possible that the powerful would channel all AI into some tasks of very high perceived value like human brain emulation, radical life extension or space colonization leaving very little AI for every thing else!)

2) Demand & supply curves are already quite crude. Combining AI labor & human labor into the same demand & supply curves seems like a mistake. (It's unclear to me whether you are doing this).

3) Realistically I suspect that human labor supply will shift to the left b/c of ‘UBI’.

4) Ignoring preference for humans, demand for human labor may also shift to the left as AI entrepreneurs would tend to optimize things around AI.

5) The economy will probably grow quite a bit. And preference for humans is likely substantial for certain types of jobs eg NFL player, runway model etc.

6) Combining 4 & 5 suggests a very steep demand curve for human labor.

7) Combining 3 & 6 suggests that a few people (I would guess 20% of adults) will have decent paying jobs & the rest will live off of savings or ‘UBI’.

I agree that I initially misread your post. I will edit my other comment.

“Humans are the horses of the future! Just accept it & go on with your lives!” - Ghora Sutra