LessWrong 2.0 Reader

Originally, the parents visited the apartment with James Webb, who was an independent journalist and a bit of a conspiracy theorist. On that visit, James Webb said that the way that the hair was under the doorframe was unusual. The parents might also discuss it in interviews.

There was no mention of it in the police report, though.

Denji is indeed a caricature of himself, both diagetically and metaphorically. I believe this is a deliberate metatextual self-reference to how popular Chainsaw Man has gotten in the real world.

Thanks. Also, in the case of crosscoders, where you have multiple output spaces, do you have any thoughts on the best way to aggregate across these? currently I'm just computing them separately and taking the mean. But I could see imagine it perhaps being better to just concat the spaces and do fvu on that, using l2 norm of the concated vectors.

Apologies in advance if this is a midwit take. Chess engines are “smarter” than humans at chess, but they aren’t automatically better at real-world strategizing as a result. They don’t take over the world. Why couldn’t the same be true for STEMlord LLM-based agents? 

It doesn’t seem like any of the companies are anywhere near AI that can “learn” or generalize in real time like a human or animal. Maybe a superintelligent STEMlord could hack their way around learning, but that still doesn’t seem the same as or as dangerous as fooming, and it also seems much easier to monitor. Does it not seem plausible that the current paradigm drastically accelerates scientific research while remaining tools? The counter is that people will just use the tools to try and figure out learning. But we don’t know how hard learning is, and the tools could also enable people to make real progress on alignment before learning is cracked.

Asa's story started fairly strong, and I enjoyed the first 10 or so chapters. But as Asa was phased out of the story, and it focused more on Denji, I felt it got worse. There were still a few good moments, but it's kinda spoilt the rest of the story, and even Chainsaw Man for me. Denji feels like a caricature of himself.  Hm, writing this, I realize that it isn't that I dislike most of the components of the story. It's really just Denji. 

EDIT: Anyway, thanks for prompting me to reflect on my current opinion of Asa Mitaka's story, or CSM 2 as I think of it.  I don't think I ever intended that to wind up as my cached-opinion.  So it goes.

Yep, that’s the generalisation that would make most sense

Overrefusal issues were way more common 1-2 years ago. models like gemini 1, and claude 1-2 had severe overrefusal issues.

I think what makes Chainsaw Man great is that the characters are dangerous, insane, and relatable. What really sold me on Asa Mitaka's story was Asa's conversation with Yuko about the murder. Asa's story has strengths and weaknesses compared to Denji's. I much prefer that over a retread of the original Chainsaw Man story.

I feel the whole aquarium arc was genius, especially the ending. But to understand it on all the different levels requires knowing that the beginning of the aquarium date, where Asa lectures about fish, is a riff on the aquarium date scene from Rent-A-Girlfriend.

I recently shared this idea with several engineers at EAG and received encouraging feedback:

Split LLM outputs into two distinct parts with separate RL reward structures:

Current format: [Problem solution]<end-of-turn> Proposed format: [Problem solution]<end-of-main-output>[alignment-relevant critique]<end-of-turn>

The first segment functions normally and is what users see. The second segment serves as a dedicated channel for alignment-relevant observations and remains invisible to users. This critique section would typically be empty and gets removed when feeding the conversation back to the model.

Crucially, the critique segment would be trained exclusively on an artificial dataset with rewards tied solely to alignment properties. Since this segment never influences the training of the main output, it has no incentive to be sycophantic or align with the main output's decisions.

For example, if the main output engages in alignment faking, the critique segment could report this behavior to developers. We already have data to train for this specific case, and the approach might generalize to other alignment issues.

The <end-of-main-output> token essentially acts as a 'truth serum' for the model - once reached, previous reward mechanisms become irrelevant, and reward maximization comes only from revealing alignment-relevant issues in the preceding reasoning.

It might.

My understanding (which could be off base) from reading the paper is the method's accuracy in detecting various forms of deception was basically 96-99%. But they acknowledge that the sophisticated deception they're ultimately worried about will be harder to detect.

Still 96-99% seems like a great start. And this was on detecting strategic deception, not just factual falsehoods. And they didn't even utilize the CoT outputs of the models.

(I think the "strategic deception" framing is also probably more general and not as dependent on unnecessary assumptions about how models work, compared to the "mesaoptimizer" framing.)