LessWrong 2.0 Reader

There are also signals that give me a bit of optimism:

1. Trump somehow decided to impose tariffs on most goods from Taiwan. Meanwhile, China hopes to become ahead of the USA [LW(p) · GW(p)] who at the same time are faced with a crisis threat. Does it mean that the USA will end up with less compute than China and so occupied with internal problems that slowdown would be unlikely to bring China any harm? Does the latter mean that China won't race ahead with a possibly misaligned AGI?
2. As I've already mentioned in a comment, GPT-4o appears to be more aligned to an ethics [LW(p) · GW(p)] than to obeying to OpenAI. Using the AIs for coding is already faced with troubles like the AI telling the user to write some code for oneself. The appearace of a superhuman coder could make the coder itself realise that the coder will take part in the Intelligence Curse^[1], making the creation of a coder even more obviously difficult.

3. Even an aligned superintelligence will likely be difficult to use because of cost constraints. 

   3.1. The ARC-AGI leaderboard provides us with data on how intelligent the o1 and o3-mini models actually are. While the o1 and o3-mini models are similar in intelligence, the latter is just 20-40 times cheaper; the current o3 model costs $200 in the low mode, implying that a hypothetical o4-mini model is to cost $5-10 in a similarly intelligent mode;

   3.2. The o3 model with low compute is FAR from passing the ARC-AGI-2 test. Before o3 managed to solve 75% of ARC-AGI-1-level tasks by using 200 dollars per task, the o1 model solved 25% while costing $1.5 per task. Given that the rate of success of different AIs at different tasks is close to the sigmoid curve [LW(p) · GW(p)], I find it highly unlikely that ARC-AGI-2-level tasks are solvable by a model cheaper than o5-mini and unlikely that they are solvable by a model cheaper than o6-mini. On the other hand, o5-mini might cost hundreds of dollars per task, while the o6-mini might cost thousands per task. 

   3.3. The cost-to-human ratio appears to confirm this trend. As one can tell from Fig.13 on Page 22 of the METR article, most tasks that could take less than a minute were doable by the most expensive low-level LLMs at a tiny cost, while some others that take more than a minute require a new generation of models that even managed to elevate the cost for some tasks above the ${10}^0$ threshold when the models become useless. 

   Could anyone comment on these points separately?

1. ^{^}

    In the slowdown ending of the AI-2027 forecast the aligned superhuman AGI is also expected to make mankind fully dependent on needless makeshift jobs or on the UBI. The latter idea was met with severe opposition in 2020, implying that it is the measure which is necessary only because of severely unethical decisions like moving factory work to Asia. 

It all hinges on the fact that your assumption that you have to be aware of things for them to matter is unjustified—and, it turns out, very very false.

What do you mean by “matter” here?

Remember, we’re talking about the following situation:

1. Woman attempts to “send cues”, with the intended result being that a certain man will perceive these cues and react in a certain desired way.
2. Man has no idea that this is happening.
3. Man does not react in the desired way to the supposed cues that the woman is supposedly sending (and how could he, not being aware of any such things?).
4. Woman is annoyed, frustrated, etc., that she is not getting what she wants.

Now you’re claiming that, somehow, these cues are nevertheless being “sensed” (ok, sure), and also that they nevertheless “matter” in some way.

What is the meaning of “matter” in this context that makes your claim true?

EDIT: By the way, this is false in my experience:

But it is also true that people will call what I refer to as “sensing the cue” “sensing the cue”.

I’ve never encountered this usage from anyone other than you.

Have you tested this hypothesis on your friends? Ask them for their iron level from last blood test, and ask them to self-report anxiety level (you also make a separate estimate of their anxiety level).

Huh, seems you are correct. They also apparently are heavily cannibalistic, which might be a good impetus for modeling the intentions of other members of your species…

Is interpersonal variation in anxiety levels mostly caused by dietary iron?

I stumbled across this paper yesterday. I haven't looked at it very closely yet, but the high-level pitch is that they look at genetic predictors of iron deficiency and then cross that with anxiety data. It's interesting mainly because it sounds pretty legit (i.e. the language sounds like direct presentation of results without any bullshitting, the p-values are satisfyingly small, there's no branching paths), and the effect sizes are BIG IIUC:

The odd ratios (OR) of anxiety disorders per 1 standard deviation (SD) unit increment in iron status biomarkers were 0.922 (95% confidence interval (CI) 0.862–0.986; p = 0.018) for serum iron level, 0.873 (95% CI 0.790–0.964; p = 0.008) for log-transformed ferritin and 0.917 (95% CI 0.867–0.969; p = 0.002) for transferrin saturation. But no statical significance was found in the association of 1 SD unit increased total iron-binding capacity (TIBC) with anxiety disorders (OR 1.080; 95% CI 0.988–1.180; p = 0.091). The analyses were supported by pleiotropy test which suggested no pleiotropic bias.

Odds ratio of anxiety disorders changes by roughly 0.9 per standard deviation in iron level, across four different measures of iron level. (Note that TIBC, the last of the four iron level measures, didn't hit statistical significance but did have a similar effect size to the other three.)

Just eyeballing those effect sizes... man, it kinda sounds like iron levels are maybe the main game for most anxiety? Am I interpreting that right? Am I missing something here?

EDIT: I read more, and it turns out the wording of the part I quoted was misleading. The number 0.922, for instance, was the odds ratio AT +1 standard deviation serum iron level, not PER +1 standard deviation serum iron level. That would be -0.078 PER standard deviation serum iron level, so it's definitely not the "main game for most anxiety".

I think perhaps you have missed the point I was making, which is that what you call “being aware that you have sensed a cue” is just what everyone else calls “sensing the cue” (perhaps “perceiving the cue” might be a better phrase

I haven't. It's just something you're going to struggle to understand until you recognize the difference -- and the importance of the difference -- between the quotation and its referent. So that's where we have to start.

It is true that many people will call the thing I refer to as "being aware that you have sensed the cue" "sensing the cue", yes. But it is also true that people will call what I refer to as "sensing the cue" "sensing the cue".

It's not that people who refer to “being aware that you have sensed a cue” as “sensing the cue” simply have a different definition of terms, it's that they're failing to track a critical distinction.

It's not possible to understand "female signals" -- or male signals, or your own signals, even -- until you understand the importance of the things people perceive and understand which they aren't aware that they perceive and understand. It all hinges on the fact that your assumption that you have to be aware of things for them to matter is unjustified -- and, it turns out, very very false.

I'm happy to explain where your billboard analogy goes wrong, but we really gotta nail this down first.

My current guess for least worst path of ASI development that's not crazy unrealistic:

open source development + complete surveillance of all citizens and all elites (everyone's cameras broadcast to the public) + two tier voting.

Two tier voting:

• countries's govts vote or otherwise agree at global level on a daily basis what the rate of AI progress should be and which types of AI usage are allowed. (This rate can be zero.)
• All democratic countries use daily internet voting (liquid democracy) to decide what stance to represent at the global level. All other countries can use whatever internal method they prefer, to decide their stance at the global level.
• (All ASI labs are assumed to be property of their respective national govts. An ASI lab misbehaving is its govt's responsibility.) Any country whose ASI labs refuse to accept results of global vote and accelerate faster risks war (including nuclear war or war using hypothetical future weapons). Any country whose ASI labs refuse to broadcast themselves on live video risks war. Any country's govt that refuses to let their citizens broadcast live video risks war. Any country whose citizens mostly refuse to broadcast themselves on live video risks war. The exact thresholds for how much violation leads to how much escalation of war, may ultimately depend on how powerful the AI is. The more powerful the AI is (especially for offence not defence), the more quickly other countries must be willing to escalate to nuclear war in response to a violation.

Open source development

• All people working at ASI labs are livestream broadcast to public 24x7x365. Any AI advances made must be immediately proliferated to every single person on Earth who can afford a computer. Some citizens will be able to spend more on inference than others, but everyone should have the AI weights on their personal computer.
• This means bioweapons, nanotech weapons and any other weapons invented by the AI are also immediately proliferated to everyone on Earth. So this setup necessarily has to be paired with complete surveillance of everyone. People will all broadcast their cameras in public. Anyone who refuses can be arrested or killed via legal or extra-legal means.
• Since everyone knows all AI advances will be proliferated immediately, they will also use this knowledge to vote on what the global rate of progress should be.

There are plenty of ways this plan can fail and I haven't thought through all of them. But this is my current guess.

Historically, the social contract describes the set of agreements concerning the legitimacy of government and its role in governing citizens. This concept, developed by philosophers like Hobbes, Locke, and Rousseau, posits that individuals surrender certain natural freedoms and contribute a portion of their wealth to governments in exchange for protection, order, and social stability.

Over time, this foundational concept evolved beyond the basic relationship between citizen and state. The industrial age expanded the social contract to encompass economic relationships between workers, employers, and broader society. Citizens came to expect not just security, but also economic opportunity. Governments increasingly took on responsibilities for education, infrastructure, and basic welfare as part of their obligation under this implicit agreement.

This evolution produced the modern social contract we recognize today: citizens contribute their labor and a portion of their earnings through taxation; in return, they receive not just protection but also economic security and the promise that hard work would be rewarded with prosperity. 

I am not sure how this social change is discontinuous with previous developments which introduced new social conditions, new capabilities, and new externalities. In short, it is clear that if there is big economic change, there will be political changes too, but if this is rethinking the social contract, then we have been doing this continuosly. We do not need to begin to rethink the social contract. We need to recognize that we have always been continually rethinking it. 

Well shoot. I'll work on it, thank you!

Hm, I would say the vibes level is the exact level that this is most effective, rather than any particular method. The basic reason being that LLMs tend to reflect behaviour as they generate from a probability distribution of "likely" outcomes for a given input. Having the "vibes of human-child-rearing" would then result in more outcomes that align with that direction as a result. It's definitely hand wavey so I'm working on more rigerous mathematical formalisms, but the bones are there. I don't nessecarily think feeding an LLM data like we would a child is useful, but I do think that the "vibe" of doing so will be useful. (This is indeed directly related ot the argument that every time we say "AI will kill us all" it makes it x% more likely)

I'd give humans a middling score on that if you look at the state of the world, we are doing pretty well with extreme events like MAD, but on the more minor scale things have been pretty messed up. A good trajectory though, compared to where things were and the relative power we had available. I think a big part of this, that you have helped clarify for me, is that I think it's important that we socialize LLM-based intelligences like humans if we want an outcome that isn't completely alien in it's choices. 

Well that's a bit of the point of the essay isn't it? You have a memetic/homeostatic boundary condition that strongly prefers/incentivizes assuming human adults are alike enough to you that their opinion matters. Even in that statement I can differ, I think childrens perspectives are incredibly important to respect, in some ways more important than an adults because children have an unfiltered honesty to their speech that most adults lack. Although I do delineate heavily between respecting and acting upon/trusting. 

For LLMs I think this is just a new sort of heuristic we are developing, where we have to reckon with the opposite of the animal problem. Animals and plants are harder for us to discern pain/suffering from, but we are more confident when we identify it that they experience it (at least in modern times, many traditions treated animal suffering as essentially fake). Now we have the opposite, creatures that are very easy to interpret but we don't know if they actually have the capacity to feel these things (although we can identify feature activations etc.). So my argument is more that we should be building technology in a way that memetically aligns with the golden rule, because running a society based on something communicating suffering (even if it can't) is going to result in a worse human society regardless. (The counter point being that playing video games where you kill NPCs doesn't make school shooters, but I'm less concerned about those NPCs gaining econmic/social power and patterning off of resentment for having the pattern of being tortured in their memory).