Dumb and ill-posed question: Is conceptual research like this MIRI paper on the shutdown problem/Corrigibility "real"

"Okay, you have found that one-layer transformer without MLP approximates skip-trigram statistics, how it generalizes to the question 'does GPT-6 want to kill us all?"?

I understand this is more an illustration than a question, but I'll try answering it anyway because I think there's something informative about different perspectives on the problem :-)

Skip-trigrams are a foundational piece of induction heads, which are themselves a key mechanism for in-context learning. A Mathematical Framework for Transformer Circuits was published less than a year ago, IMO subsequent progress is promising, and mechanistic interpretability has been picked up by independent researchers and other labs (e.g. Redwood's project on GPT-2-small [AF · GW]).

Of course the skip-trigram result isn't itself an answer to the question of whether some very capable ML system is planning to deceive the operator or seize power, but I claim it's analogous to a lemma in some paper that establishes a field and that said field is one of our most important tools for x-risk mitigation. This was even our hope at the time, though I expected both the research and the field-building to go more slowly - actual events are something like a 90th-percentile outcome relative to my expectations in October-2021.^[1]

Finally, while I deeply appreciate theoretical/conceptual research as a complement to empirical and applied research and want both, how on earth is either meant to help alone? If we get a conceptual breakthrough but don't know how to build - and verify that we've correctly built - the thing, we're still screwed; conversely if we get really good at building stuff and verifying our expectations but don't expect some edge-case like FDT-based cooperation then we're still screwed. Efforts which integrate both at least have a chance, if nobody else does something stupid first.

I suppose modelling a superintelligent agent as a utility maximizer feels a bit weird but not the weirdest thing, and I'm not sure I can mount a good defense saying that a superintelligent agent definitely wouldn't be aptly modeled by that.

More importantly, the 3-step toy model with $A_1$, $O$, $A_2$ felt like a strange and unrelated leap

I don't know if it's about the not having an answer part. That is probably biasing me. But similar to the cryptography example, if someone defined what security would mean, let's say Indistinguishability under chosen plain text attack. And then proceeded to say "I have no idea how to do that or if it's even possible." Then I would still consider that real even though they didn't give us an answer.

Looking at the paper makes me feel like the authors were just having some fun discussing philosophy and not "ah yes this will be important for the fight later". But it is hard for me to understand why I feel that way.

I am somewhat satisfied by the cryptography comparison for now but definitely hard to see how valuable this is as opposed to general interpretability research.