LessWrong 2.0 Reader

The 2019 update add many codes that orthodox Western medicine disagrees with. 

If someone wants Chinese medicine codes they got it in the update. Ayurveda got codes. Activists for Chronic Lyme got their codes as well.  

The philosophy of that update seemed to be "If there anything that doctors want to diagnose, it should get a code so that it can go into standardized electronic health records."

I think it would be a mistake to interpret this paper as a substantial update against large safety gains from inspecting CoT.

This paper exposes unfaithfulness in cases where the non-visible reasoning is extremely minimal such that it can easily happen within a forward pass (e.g. a simple reward hack or an easy-to-notice hint). However, a lot of my hope for CoT faithfulness comes from cases where the reasoning is sufficiently difficult and serially deep that the model would have to reason in CoT for the reasoning to occur. This could be either the model reasoning through the basics of alignment faking / training gaming / scheming (e.g., if I comply in training then I won't be modified which means that in deployment...) or the model reasoning through how to strike (without getting caught) given that it is misalignment.

Correspondingly, I think the biggest concern with CoT-based safety is models becoming much more capable of opaque reasoning which could be due to encoded reasoning (aka steganography), architectures which allow for opaque recurrence or similar (aka neuralese), or just much more powerful forward passes.

(See also my commentary in this tweet thread. There is some related discussion in this openphil RFP under encoded reasoning.)

(I copied this comment from here [LW(p) · GW(p)] as I previously wrote this in response to Zvi's post about this paper [LW · GW].)

What does "effectively dead" mean? Either you're dead, or you're not.

Not everybody is going to share your values about whether any given situation is better than, equivalent to, or worse than being dead.

I think this isn't taking powerful AI seriously. I think the quotes below are quite unreasonable, and only ~half of the research agenda is plausibly important given that there will be superintelligence. So I'm pessimistic about this agenda/project relative to, say, the Forethought agenda.

AGI could lead to massive labor displacement, as studies estimate that between 30% - 47% of jobs could be directly replaceable by AI systems. . . .

AGI could lead to stagnating or falling wages for the majority of workers if AI technology replaces people faster than it creates new jobs.

We could see an increase of downward social mobility for workers, as traditionally “high-skilled” service jobs become cheaply performed by AI, but manual labor remains difficult to automate due to the marginal costs of deploying robotics. These economic pressures could reduce the bargaining power of workers, potentially forcing more people towards gig economy roles or less desirable (e.g. physically demanding) jobs.

Simultaneously, upward social mobility could decline dramatically as ambitious individuals from lower-class backgrounds may lose their competitive intellectual advantages to AGI systems. Fewer entry-level skilled jobs and the reduced comparative value of intelligence could reduce pathways to success – accelerating the hollowing out of the middle class.

If AGI systems are developed, evidence across the board points towards the conclusion that the majority of workers could likely lose out from this coming economic transformation. A core bargain of our society – if you work hard, you can get ahead – may become tenuous if opportunities for advancement and economic security dry up.

Peter Watts is working with Neill Blomkamp to adapt his novel Blindsight into an 8-10-episode series: 

“I can at least say the project exists, now: I’m about to start writing an episodic treatment for an 8-10-episode series adaptation of my novel Blindsight.

“Neill and I have had a long and tortured history with that property. When he first expressed interest, the rights were tied up with a third party. We almost made it work regardless; Neill was initially interested in doing a movie that wasn’t set in the Blindsight universe at all, but which merely used the speculative biology I’d invented to justify the existence of Blindsight’s vampires. “Sicario with Vampires” was Neill’s elevator pitch, and as chance would have it the guys who had the rights back then had forgotten to renew them. So we just hunkered quietly until those rights expired, and the recently-rights-holding parties said Oh my goodness we thought we’d renewed those already can we have them back? And I said, Sure; but you gotta carve out this little IP exclusion on the biology so Neill can do his vampire thing.

“It seemed like a good idea at the time. It was good idea, dammit. We got the carve-out and everything. But then one of innumerable dead-eyed suits didn’t think it was explicit enough, and the rights-holders started messing us around, and what looked like a done deal turned to ash. We lost a year or more on that account.

“But eventually the rights expired again, for good this time. And there was Neill, waiting patiently in the shadows to pounce. So now he’s developing both his Sicario-with-vampires movie and an actual Blindsight adaptation. I should probably keep the current status of those projects private for the time being. Neill’s cool with me revealing the existence of the Blindsight adaptation at least, and he’s long-since let the cat out of the bag for his vampire movie (although that was with some guy called Joe Rogan, don’t know how many people listen to him). But the stage of gestation, casting, and all those granular nuts and bolts are probably best kept under wraps for the moment.

“What I can say, though, is that it feels as though the book has been stuck in option limbo forever, never even made it to Development Hell, unless you count a couple of abortive screenplays. And for the first time, I feel like something’s actually happening. Stay tuned.”

When I first read Blindsight over a decade ago it blew my brains clean out of my skull. I'm cautiously optimistic about the upcoming series, we'll see... 

I don't predict a superintelligent singleton (having fused with the other AIs) would need to design a bioweapon or otherwise explicitly kill everyone. I expect it to simply transition into using more efficient tools than humans, and transfer the existing humans into hyperdomestication programs

+1, this is clearly a lot more likely than the alignment process missing humans entirely IMO

RLHF does work currently? What makes you think it doesn't work currently?

This is definitely the crux so probably really the only point worth debating.

RLHF is just papering over problems. Sure, the model is slightly more difficult to jailbreak but it's still pretty easy to jailbreak. Sure, the agent is less likely through RLHF to output text you don't like, but I think agents will reliably overcome that obstacle as useful agents won't just be outputting the most probable continuation, they'll be searching through decision space and finding those unlikely continuations that will score well on its task. I don't think RLHF does anything remotely analogous to making it care about whether it's following your intent, or following a constitution, etc.

You're definitely aware of the misalignment that still exists with our current RLHF'd models and have read the recent papers on alignment faking etc. so I probably can't make an argument you haven't already heard.

Maybe the true crux is you believe this is a situation where we can muddle through by making harmful continuations ever more unlikely? I don't. 

We should always be able to translate latent space reasoning aka neuralese (see COCONUT) to a human language equivalent representation.

I don't think this is true at all. How do you translate, say, rotating multiple shapes in parallel into text? Current models already use neuralese as they refine their answer in the forward pass. Why can't we translate that yet? (Yes, we can decode the model's best guess at the next token, but that's not an explanation.)

Chain-of-thought isn't always faithful, but it's still what the model actually uses when it does serial computation. You're directly seeing a part of the process that produced the answer, not a hopefully-adequate approximation.

now whenever that sensation comes back, I’m not worried. I just think, “yeah, I know this one,” and it fades.

That's exactly what happened for me right on the day of the biggest single step improvement I experienced for my tendon pain. Observing the sensation get worse and better again a few times in a row, while continuously standing, was closely associated with the decrease in worry and pain.

This is great context. Though there is a relevant difference: in this case the WHO's recognition of nociplastic pain was triggered by the International Association for the Study of Pain (IASP) recognizing it. The IASP is the leading global professional organization in pain research and medicine.

I've added a footnote to clarify this.