I think perhaps you have missed the point I was making, which is that what you call “being aware that you have sensed a cue” is just what everyone else calls “sensing the cue” (perhaps “perceiving the cue” might be a better phrase

I haven't. It's just something you're going to struggle to understand until you recognize the difference -- and the importance of the difference -- between the quotation and its referent. So that's where we have to start.

It is true that many people will call the thing I refer to as "being aware that you have sensed the cue" "sensing the cue", yes. But it is also true that people will call what I refer to as "sensing the cue" "sensing the cue".

It's not that people who refer to “being aware that you have sensed a cue” as “sensing the cue” simply have a different definition of terms, it's that they're failing to track a critical distinction.

It's not possible to understand "female signals" -- or male signals, or your own signals, even -- until you understand the importance of the things people perceive and understand which they aren't aware that they perceive and understand. It all hinges on the fact that your assumption that you have to be aware of things for them to matter is unjustified -- and, it turns out, very very false.

I'm happy to explain where your billboard analogy goes wrong, but we really gotta nail this down first.

My current guess for least worst path of ASI development:

open source development + complete surveillance of all citizens and all elites (everyone's cameras broadcast to the public) + two tier voting.

Two tier voting:

• countries vote or otherwise agree at global level to decide what the rate of AI progress should be and which types of AI usage are allowed. (This rate can be zero.)
• All democratic countries use daily internet voting (liquid democracy) to decide what stance to represent at the global level. All other countries can use whatever internal method they prefer, to decide their stance at the global level.
• Any country that refuses to accept results of global vote and accelerates faster risks war (including nuclear war or war using hypothetical future weapons). Any country that refuses to broadcast their ASI labs on live video also risks war.

Open source development

• All people working at ASI labs are livestream broadcast to public 24x7x365. Any advances made must be immediately proliferated to every single person on Earth who can afford a computer. Some citizens will be able to spend more on inference than others, but everyone should have the AI weights on their personal computer.
• This means bioweapons, nanotech weapons and any other weapons are also immediately proliferated to everyone on Earth. So this necessarily has to be paired with complete surveillance of everyone. People will all broadcast their cameras in public. Anyone who refuses will be arrested or killed via legal or extra-legal means.
• Since everyone knows all advances will be proliferated immediately, they will also use this knowledge to vote on what the global rate of progress should be.

There are plenty of ways this plan can fail and I haven't thought through all of them. But this is my current guess.

Historically, the social contract describes the set of agreements concerning the legitimacy of government and its role in governing citizens. This concept, developed by philosophers like Hobbes, Locke, and Rousseau, posits that individuals surrender certain natural freedoms and contribute a portion of their wealth to governments in exchange for protection, order, and social stability.

Over time, this foundational concept evolved beyond the basic relationship between citizen and state. The industrial age expanded the social contract to encompass economic relationships between workers, employers, and broader society. Citizens came to expect not just security, but also economic opportunity. Governments increasingly took on responsibilities for education, infrastructure, and basic welfare as part of their obligation under this implicit agreement.

This evolution produced the modern social contract we recognize today: citizens contribute their labor and a portion of their earnings through taxation; in return, they receive not just protection but also economic security and the promise that hard work would be rewarded with prosperity. 

I am not sure how this social change is discontinuous with previous developments which introduced new social conditions, new capabilities, and new externalities. In short, it is clear that if there is big economic change, there will be political changes too, but if this is rethinking the social contract, then we have been doing this continuosly. We do not need to begin to rethink the social contract. We need to recognize that we have always been continually rethinking it. 

Well shoot. I'll work on it, thank you!

Hm, I would say the vibes level is the exact level that this is most effective, rather than any particular method. The basic reason being that LLMs tend to reflect behaviour as they generate from a probability distribution of "likely" outcomes for a given input. Having the "vibes of human-child-rearing" would then result in more outcomes that align with that direction as a result. It's definitely hand wavey so I'm working on more rigerous mathematical formalisms, but the bones are there. I don't nessecarily think feeding an LLM data like we would a child is useful, but I do think that the "vibe" of doing so will be useful. (This is indeed directly related ot the argument that every time we say "AI will kill us all" it makes it x% more likely)

I'd give humans a middling score on that if you look at the state of the world, we are doing pretty well with extreme events like MAD, but on the more minor scale things have been pretty messed up. A good trajectory though, compared to where things were and the relative power we had available. I think a big part of this, that you have helped clarify for me, is that I think it's important that we socialize LLM-based intelligences like humans if we want an outcome that isn't completely alien in it's choices. 

Well that's a bit of the point of the essay isn't it? You have a memetic/homeostatic boundary condition that strongly prefers/incentivizes assuming human adults are alike enough to you that their opinion matters. Even in that statement I can differ, I think childrens perspectives are incredibly important to respect, in some ways more important than an adults because children have an unfiltered honesty to their speech that most adults lack. Although I do delineate heavily between respecting and acting upon/trusting. 

For LLMs I think this is just a new sort of heuristic we are developing, where we have to reckon with the opposite of the animal problem. Animals and plants are harder for us to discern pain/suffering from, but we are more confident when we identify it that they experience it (at least in modern times, many traditions treated animal suffering as essentially fake). Now we have the opposite, creatures that are very easy to interpret but we don't know if they actually have the capacity to feel these things (although we can identify feature activations etc.). So my argument is more that we should be building technology in a way that memetically aligns with the golden rule, because running a society based on something communicating suffering (even if it can't) is going to result in a worse human society regardless. (The counter point being that playing video games where you kill NPCs doesn't make school shooters, but I'm less concerned about those NPCs gaining econmic/social power and patterning off of resentment for having the pattern of being tortured in their memory).

That's some careful analysis!

Two remarks:

1

"Can" is the opposite of "unable". "Unable" means that the change involves granting ability to they who would act, i.e. teaching a technique, providing a tool, fixing the body, or altering the environment.

That's a good characterization, though arguably not a definition, as it relies on "ability", which is circular. I can do something = I have the ability to do something. I can = I'm able to.

But we can use the initial principle (it really needs a name) which doesn't mention ability:

You do a thing iff you can do it and you want to do it.

"Iff" behaves similar to an equation, so we can solve for "can", similar to algebra in arithmetic. I don't know the exact algebra of "iff", but solving for "can" arguably yields:

"I can do X" iff "If I wanted to do X, I would do X"

Which uses wanting and a counterfactual to define "can". We could also define:

"I want to do X" iff "If I could to do X, I would do X"

Though those two definitions together are circular. Maybe it is better to regard one concept as more basic than the other, and only define the less basic one in terms of the more basic one. It seems to me that "want" is more basic than "can", so I would only define "can" in terms of "want", and leave the definition for "want" open (for now).

2

Regarding the Confusing Cases. There are at least two canonical classes: akrasia (weakness of will) and addiction. "Can" the addict quit smoking? "Can" the person suffering from akrasia just do The Thing?

A better question perhaps: In which sense is the answer to the above "yes" and in which sense is it "no"?

One possibility is to analyze these cases in terms of first-order vs second-order desires. The first-order desire would be smoking or being lazy, the second order desire would be not to have the (first-order) desire to smoke, or be lazy. Second-order desires seem to be more important or rational than the first order desires. The second-order and first-order desire are "inconsistent" here. If the first-order desire to smoke is stronger than the second-order desire not to have the first-order desire, I don't quit smoking. (Or don't overcome my laziness in case of akrasia). Otherwise I do.

Here, the "can" definition as "If I wanted to do X, I would do X" is ambiguous. If it means "If I had the first-order desire to quit smoking, I would quit smoking", then the sentence is true, and I "can" overcome the addiction (or the akrasia). If it means "If I had a second-order desire to not have the first order desire to smoke, I would quit smoking", then the sentence is false, as I do in fact have the second-order desire but still don't quit. So in this sense it's not true that I "can" quit.

A similar but different different analysis wouldn't phrase it in terms of first and second-order desires, but in terms of rational wishes and a-rational urges. I wish to quit smoking, but I have the urge to smoke. I wish to get to work, but I have the urge to be lazy. If we count, in the definition of "can", urges as "wanting", I can stop smoking, if we count "wishes" as "wanting", I can't. By a similar argument as above.

I think these cases are actually not a major problem of the "can" definition. After all, it seems in fact ambiguous to ask whether someone "can" overcome some case of akrasia or addiction. The definition captures that ambiguity.

Hmm interesting.

I see the case for focusing only on compute (since relatively objective), but it still seems important to try to factor in some amount of algorithmic progress to pretraining (which means that the cost to achieve GPT-6 level performance will be dropping over time).

The points on Epoch are getting outside of my expertise – I see my role as to synthesise what experts are saying. It's good to know these critiques exist and it would be cool to see them written up and discussed.

The 2019 update add many codes that orthodox Western medicine disagrees with. 

If someone wants Chinese medicine codes they got it in the update. Ayurveda got codes. Activists for Chronic Lyme got their codes as well.  

The philosophy of that update seemed to be "If there anything that doctors want to diagnose, it should get a code so that it can go into standardized electronic health records."

I think it would be a mistake to interpret this paper as a substantial update against large safety gains from inspecting CoT.

This paper exposes unfaithfulness in cases where the non-visible reasoning is extremely minimal such that it can easily happen within a forward pass (e.g. a simple reward hack or an easy-to-notice hint). However, a lot of my hope for CoT faithfulness comes from cases where the reasoning is sufficiently difficult and serially deep that the model would have to reason in CoT for the reasoning to occur. This could be either the model reasoning through the basics of alignment faking / training gaming / scheming (e.g., if I comply in training then I won't be modified which means that in deployment...) or the model reasoning through how to strike (without getting caught) given that it is misalignment.

Correspondingly, I think the biggest concern with CoT-based safety is models becoming much more capable of opaque reasoning which could be due to encoded reasoning (aka steganography), architectures which allow for opaque recurrence or similar (aka neuralese), or just much more powerful forward passes.

(See also my commentary in this tweet thread. There is some related discussion in this openphil RFP under encoded reasoning.)

(I copied this comment from here [LW(p) · GW(p)] as I previously wrote this in response to Zvi's post about this paper [LW · GW].)

I think this isn't taking powerful AI seriously. I think the quotes below are quite unreasonable, and only ~half of the research agenda is plausibly important given that there will be superintelligence. So I'm pessimistic about this agenda/project relative to, say, the Forethought agenda.

AGI could lead to massive labor displacement, as studies estimate that between 30% - 47% of jobs could be directly replaceable by AI systems. . . .

AGI could lead to stagnating or falling wages for the majority of workers if AI technology replaces people faster than it creates new jobs.

We could see an increase of downward social mobility for workers, as traditionally “high-skilled” service jobs become cheaply performed by AI, but manual labor remains difficult to automate due to the marginal costs of deploying robotics. These economic pressures could reduce the bargaining power of workers, potentially forcing more people towards gig economy roles or less desirable (e.g. physically demanding) jobs.

Simultaneously, upward social mobility could decline dramatically as ambitious individuals from lower-class backgrounds may lose their competitive intellectual advantages to AGI systems. Fewer entry-level skilled jobs and the reduced comparative value of intelligence could reduce pathways to success – accelerating the hollowing out of the middle class.

If AGI systems are developed, evidence across the board points towards the conclusion that the majority of workers could likely lose out from this coming economic transformation. A core bargain of our society – if you work hard, you can get ahead – may become tenuous if opportunities for advancement and economic security dry up.