LessWrong 2.0 Reader

Just my opinion: the concerns are valid but exaggerated.

Perhaps exaggeration is justified in order to get people's attention since most americans are still treating this as a normal election.

Good question! I think there are two different steps here. (The following is a bit oversimplified.)

Step 1 is auto-associative recall in the hippocampus. Neurons all over the cortex directly or indirectly activate neurons in the hippocampus. And then if something is happening in any part of the cortex that partially matches some old memory, the whole old memory can autocomplete within the hippocampus.

Step 2 is: that core of an old memory has lots of (direct and indirect) associations all around the cortex / global workspace. Like, if it’s a visual memory, the neurons in the hippocampus will send signals that the relevant visual region should go into the corresponding activation state.

Loosely speaking, there’s local “attractor dynamics” within the hippocampus (Step 1) and simultaneously there’s larger-scale “attractor dynamics” around much of the cortex (Step 2).

I keep using the example of having Tune A in your head (or actually hearing it) and then seeing something that would normally trigger a memory of a different Tune B (assuming no special training :-P [LW(p) · GW(p)]). In this situation, I think Step 1 goes through, i.e. part or all of the hippocampus does in fact transiently get into a state that represents Tune B. But then Step 2 is blocked. Instead of there being a large-scale “attractor” around the cortex that grows out of the “seed” at the hippocampus, instead that Tune B attractor fails to spread, instead getting blocked by the existing Tune A attractor, which pushes back and wins.

More generally, we can ask which attractor will “win” in a fight between two partially-formed mutually-incompatible atteractors. It’s complicated, but attention has a lot to do with it. If you’re holding rapt attention towards what you’re looking at right now, then you’re less likely to spontaneously visually imagine the appearance of some random thing you remember from earlier in life. Whereas if you’re not particularly paying attention to anything, then there’s nothing stopping a visual memory from popping up.

So back to your question, when you recall the childhood treehouse, I claim there’s at least a brief moment when you stop paying attention to current visual inputs and instead use (part of) your visual processing capacity to imagine the treehouse. Then you flip back to what you’re looking at. It all happens very fast—a fraction of a second. So I think there’s an attentional blip associated with briefly thinking about the treehouse, but we don’t really notice, just as people are surprised to learn about change blindness and the selective attention task and so on.

So then you can ask: if I can transiently stop paying attention to visual input in order to recall what my old treehouse looked like, why can’t the DID patient likewise transiently stop paying attention to the concept of Alter A in order to imagine the concept of Alter B? (…at least in some cases.) I think that the answer again centers around attention. I think there’s a tendency to hold attention on the homunculus concept, whatever it is, not enough to make it the center of attention most of the time, but enough to prevent it from disappearing entirely.

In this respect, I think the homunculus concept is less like something I happen to be looking at, and more like “the idea that I have an itchy knee” or “the idea that I’m feeling anxious”. Again, these don’t need to be the center of attention. But they do seem to reliably benefit from a steady drip of low-level attention, such that they tend to be kinda looming in the background, whatever you’re thinking about.

Why does the homunculus have that special property, of holding low-level attention in a way that most concepts don’t? I think it’s some combination of (1) In general, the homunculus is associated with physiological arousal (since it’s built from “surprisingness”, see §3.3.2 [LW · GW]), which makes it inherently attention-grabbing; (2) In DID in particular, there’s an association between the alters and moods, and moods like anxiety and anger and sadness have their own involuntary attention mechanism.

There’s a bit more about all this in the next post, actually. :)

That is a very cute/nice logo.

Are the ‘AI companion’ apps, or robots, coming? I mean, yes, obviously?

The technology for bots who are "better" than humans in some way (constructive, pro-social, compassionate, intelligent, caring interactions while thinking 2 levels meta) has been around since 2022. But the target group wouldn't pay enough for GPT-4-level inference, so current human-like bots are significantly downscaled compared to what technology allows.

Supply side: It approaches the minimum average total, not marginal, cost. Maybe if people accounted for it finer (e.g., charging self "wages" and "rent"), cooking at home would be in the ballpark (assuming equal quality of inputs and outputs across venues..), but that just illustrates how real costs can explain a lot of the differential without having to jump to regulation and barriers to entry (yes, those are nonzero too!).

Demand side: Complaints in the OP about the uninformativeness of ratings also highlight how far we are from perfect competition (also, e.g., heterogeneous products), so you can expect nonzero markups. We aren't in equilibrium and in the long run we're all dead, etc.

I'm a big proponent of starting with the textbook economic analysis, but I was surprised by the surprise. Let's even assume perfect accounting and competition:

Draw a restaurant supply curve in the middle of the graph. In the upper right corner, draw a restaurant demand curve (high demand given all the benefits I listed). Equilibrium price is P_r*. Now draw a home supply curve to the far left, indicating an inefficient supply relative to restaurants (for the same quantity, restaurants do it "cheaper"). In the bottom left corner, draw a home demand curve (again the point is I demand eating out more than eating at home). Equilibrium price for those is P_h*. It's very easy to draw where P_h* < P_r*.

My current belief is that this essay is optimized to be understandable by a much broader audience than any comparable public writing from Anthropic on extinction-level risk. 

For instance, did you know that the word 'extinction' doesn't appear anywhere on Anthropic's or Dario's websites? Nor do 'disempower' or 'disempowerment'. The words 'existential' and 'existentially' only come up three times: when describing the work of an external organization (ARC), in one label in a paper, and one mention in the Constitutional AI. In its place they always talk about 'catastrophic' risk, which of course for most readers spans a range many orders of magnitude less serious (e.g. damages of $100M). If Amodei doesn't believe that existential threats are legitimate, then I think there are many people at his organization who have gone there on the trust that it is indeed a primary concern of his and who will be betrayed in that. If he does, how has he managed to ensure basically no discussion of it on the company website or in its research, yet has published a long narrative of how AI can help with "poverty" "inequality" "peace" "meaning" "health" and other broad positives? This seems to me very likely to be heavily filtered sharing of his models and beliefs with highly distortionary impacts on the rest of the worlds' models of AI in the positive direction, which is to be expected from this $10B+ company that sells AI products, rather than (say) Amodei merely getting to things out of order and of course he'll soon be following-up with just as thorough an account of his models of the existential threats he believes are on the horizon, and the rest of the organization just never thought to write about it in their various posts and papers.

(I would be interested in a link to whatever the best and broadly-readable piece by Anthropic or its leadership on existential risk from AI is. Some chance it is better than I am modeling it as. I have not listened to any of Amodei's podcasts, perhaps he speaks more straightforwardly there.)

Added: As a small contrast, OpenAI mentions extinction and human disempowerment directly, in the 2nd paragraph on their Superalignment page, and an OpenAI blogpost by Altman links to a Karnofsky Cold Takes piece titled "AI Could Defeat All Of Us Combined". Altman also wrote two posts in 2014 on the topic of existential threats from Machine Intelligence. I would be interested to know the most direct things that Amodei has published about the topic.

like the calibration game but for a variety of decision problems, where the person has to assign probabilities to things at different stages based on what information is available. Afterwards they get an example brier score based on the average of what people with good prediction track records set at each phase.

What is an example of a type of complex blinded data that you'd be imagining here?

Math data!

I guess LLMs are model-free, so that's relevant

FWIW, I strongly disagree with this claim. I believe they are model-based, with the usual datasets & training approaches, even before RLHF/RLAIF.