Have you tried it with a book that doesn't have self-contained chapters?
hastings-greer on Inner Optimization Mechanisms in Neural NetsI've laid out a concrete example of this at https://www.lesswrong.com/posts/FgXjuS4R9sRxbzE5w/medical-image-registration-the-obscure-field-where-deep [LW · GW] , following the "optimization on a scaffold level" route. I found a real example of a misaligned inner objective outside of RL, which is cool
habryka4 on LessWrong's (first) album: I Have Been A Good BingHmm, most of the ordering should be the same. Here is the ordering on Youtube Music:
The Road To Wisdom
Moloch
Thought That Faster (feat. Eliezer Yudkowsky)
The Litany of Tarrrrrski (feat. Eliezer Yudkowsky)
The Litany of Gendlin
Dath Ilan's Song (feat. Eliezer Yudkowsky)
Half An Hour Before Dawn In San Francisco (feat. Scott Alexander)
AGI and the EMH (mit Basil Halperin, J. Zachary Mazlish & Trevor Chow)
First they came for the epistemology (feat. Michael Vassar)
Prime Factorization (feat. Scott Alexander)
We Do Not Wish to Advance (feat. Anthropic)
Nihil Supernum (feat. Godric Gryffindor)
More Dakka (feat. Zvi Mowshowitz)
FHI at Oxford (feat. Nick Bostrom)
Answer to Job (feat. Scott Alexander)
Which is pretty similar to the order here. The folk album is in a slightly different order (which I do think is worse and we sadly can't change), but otherwise things are the same.
dpiepgrass on LessWrong's (first) album: I Have Been A Good BingEven if the stars should die in heaven
Our sins can never be undone
No single death will be forgiven
When fades at last the last lit sun.Then in the cold and silent black
As light and matter end
We’ll have ourselves a last look back.And toast an absent friend.
[verse 2]
I heard that song which left me bitter
For all the sins that had been done
But I had thought the wrong way 'bout it
[cuz] I won't be there to see that sun
I noticed then I could let go
Before my own life ends
It could have been much worse you know
Relaxing with my friends
Hard work I leave with them
Someday they'll get it done
A million years too young
For now we'll have some fun
'Any ways to combine all the previously mentioned ideas (A Causal Explainable Guardrails for Large Language Models, Discussion: Challenges with Unsupervised LLM Knowledge Discovery and A Multimodal Automated Interpretability Agent (MAIA), 'Catching AIs red-handed', including paulfchristiano's comment) with ideas from 'Searching for a model's concepts by their shape – a theoretical framework' to 'catch AIs red-handed'?
Here are a few additional ideas for combining all the mentioned papers and comments to try catching deceptive models red-handed:
Extending my previous thoughts, I think one of the most useful things the "Searching for concepts by their shape" framing can contribute is mathematical precision in defining hypotheses about what characterizes deceptive cognition. If we can frame agency/deception in terms of precise relations between features (even if those features are hard to interpretably extract in practice), it gives us a crisper target to aim at with our testing, rather than just relying on intuitive notions of what "seems deceptive".
However, I continue to be quite pessimistic about the prospects for catching competent deceptive models before it's too late, for the reasons outlined previously. I worry that by the time a model is smart enough that we could meaningfully apply these combined interpretability strategies to it, it would also be smart enough to notice what we're doing and come up with countermeasures.
Unless interpretability tools far exceed my expectations in their ability to crack open the black box of neural nets (which would be great!), I suspect the most practical path is to use simpler, more behavioral detection strategies (like the model consistency testing from your Control paper) to avoid building models that are competent enough to be robustly deceptive in the first place. Catching a superintelligent deceptive model after the fact seems like a much harder challenge than noticing warning signs of misalignment in weaker models and heading them off early.
But I would still encourage pursuing all these ideas, even if they end up not being enough on their own to solve the problem. The interpretability insights gained would be valuable for alignment more broadly. And we may get lucky in extracting some key structural signature of deceptive cognition that even very sophisticated models can't fully mask. It's an arms race we can't afford to unilaterally forfeit.'
donqueror on Can we build a better Public Doublecrux?my most important point here is to approach this as an unsolved "product development" problem
This is a good take. I'd take it one step further and suggest that an even-more material product is the ideal target.
Some of this may be obvious, but for sake of clarity: there exists a process of attempting to identify assumptions which comprise a model which projects to the potential disagreement, for all possible disagreements. In the case of unstated or unnoticed supporting assumptions, this may involve identifying significantly differing probabilities placed on some assertion, and working backwards to identify its supports. Tracking this process explicitly via one or more belief networks, on a whiteboard, a sheet of paper, or even better, purpose-tailored software, allows for reasoners to identify where they may have a crux, and directly focus on building out or otherwise contending with its supports. Reasoners may prefer to produce their own separate belief networks and attempt to merge them.
Mapping out belief networks leads to probability distributions for assertions following from the combinatorics of the supports. When an assertion is under-supported relative to the reasoner's credence, this is made highly visible. The structure makes it apparent when an argument is hand-waved rather than contending with it via the supports it should have or via offering conflicting supports, and helps to fix attention on unfinished work implied by the raising of each assertion. Many rhetorical techniques would rightly fail under these conditions. It also makes for a high utility game of solitaire, whether in advance of an upcoming double crux with another reasoner or not.
A drawback is that some reasoners may prefer not to reveal all of their supports, as in the case of those which may contain infohazardous or exfohazardous content, or ones which may cause those things to be easier to derive or notice. In some cases, reasoners may prefer to engage in this sort of protocol in private, with the option to multilaterally make the results available after-the-fact.
d0themath on D0TheMath's ShortformI strong downvoted this because it's too much like virtue signaling, and imports too much of the culture of Twitter. Not only the hashtags, but also the authoritative & absolute command, and hero-worship wrapped with irony in order to make it harder to call out what it is.
bogdan-ionut-cirstea on Bogdan Ionut Cirstea's ShortformContinued:
'How could the above be combined with 'Catching AIs red-handed' and, in particular, the ideas mentioned in paulfchristiano's comment?
The approaches discussed in "A Multimodal Automated Interpretability Agent" (MAIA) and "Causal Explainable Guardrails for Large Language Models" (LLMGuardaril) could potentially be combined with the ideas in "Catching AIs red-handed" and paulfchristiano's comment to create a more comprehensive strategy for detecting and mitigating risks from deceptively aligned AI systems:
Significant open questions remain about the feasibility of scaling these approaches to be robust against highly capable AI systems. There are also important limitations to consider: for example, MAIA's causal analysis is unlikely to provide a fully conclusive "smoking gun" that an AI system is deceptive, as opposed to strong probabilistic evidence. And sufficiently capable systems may find ways to avoid exhibiting any detectable deceptive behavior under the kind of systematic experimentation enabled by MAIA.
Nonetheless, I believe the combination of automated interpretability agents, causally grounded analysis of AI reasoning, and the overall adversarial "red team" mindset advocated in "Catching AIs red-handed" represents a promising direction for building more robust monitoring and containment of potentially deceptive AI systems. A concerted effort to develop and integrate these approaches could significantly improve our odds of detecting and mitigating catastrophic deceptive alignment failures.'
quetzal_rainbow on Bogdan Ionut Cirstea's ShortformEverything Turing-complete requires infinite memory. When we are saying "x86 set of instructions is Turing-complete" we imply "assuming that processor operates on infinite memory". It's in definition of Turing machine to include infinite tape, after all.
It's hard to pinpoint, but the trick is that it's very nuanced difference between the sense in which transformers are limited in complexity-theoretic sense and "transformers can't do X". Like, there is nothing preventing transformers from playing chess perfectly - they just need to be sufficiently large for this. To answer the question "can transformers do X" you need to ask "how much computing power transformer has" and "can this computing power be shaped by SGD into solution".
bogdan-ionut-cirstea on Bogdan Ionut Cirstea's ShortformConversation with Claude Opus on A Causal Explainable Guardrails for Large Language Models, Discussion: Challenges with Unsupervised LLM Knowledge Discovery [LW · GW] and A Multimodal Automated Interpretability Agent (MAIA). To me it seems surprisingly good at something like coming up with plausible alignment research follow-ups, which e.g. were highlighted here as an important part of the superalignment agenda.
Prompts bolded:
'Summarize 'Causal Explainable Guardrails for Large
Language Models'. In particular, could this be useful to deal with some of the challenges mentioned in 'Discussion: Challenges with Unsupervised LLM Knowledge Discovery'?
The paper "A Causal Explainable Guardrails for Large Language Models" proposes LLMGuardaril, a framework that incorporates causal analysis and adversarial learning to obtain unbiased steering representations in LLMs. Key points:Existing methods for steering LLMs towards desired attributes rely on unbiased representations and steering prompts, but the representations learned from pre-training can introduce semantic biases that negatively impact steering.
LLMGuardaril systematically identifies and blocks the confounding effects of biases, enabling extraction of unbiased steering representations. It also includes an explainable component providing insights into alignment between generated output and desired direction.
Experiments show LLMGuardaril can effectively steer LLMs towards desired attributes while mitigating biases.Regarding the challenges mentioned in "Discussion: Challenges with Unsupervised LLM Knowledge Discovery":One key challenge is distinguishing the model's own knowledge from knowledge of simulated entities. LLMGuardaril aims to disentangle the influence of semantic biases to obtain steering representations capturing desired attributes. This may help identify model's own knowledge if those attributes are specific to the model itself.
Another challenge is the hypothesized existence of many features satisfying consistency properties beyond just the model's knowledge. LLMGuardaril's causal analysis and adversarial learning aims to block confounding semantic biases, potentially reducing the set of candidate knowledge-like features.
Empirically, contrast-consistent search seemed to find prominent arbitrary features rather than knowledge. LLMGuardaril's explainable component could provide insight into whether extracted features actually correspond to model knowledge.
So in summary, while not a complete solution, LLMGuardaril's approach of causally analyzing and debiasing semantic influences may help address some key challenges around identifying model knowledge in an unsupervised way. But significant open questions remain around definitively distinguishing model knowledge from simulated knowledge of other entities. Further research is needed to determine how much traction causal analysis provides on the hardest aspects of the ELK problem.'
'How might MAIA be useful, potentially combined with 'Causal Explainable Guardrails for Large Language Models', to deal with some of the challenges mentioned in 'Discussion: Challenges with Unsupervised LLM Knowledge Discovery'?
MAIA and the techniques from "Causal Explainable Guardrails for Large Language Models" (LLMGuardaril) could potentially be combined to help address some of the challenges mentioned in "Discussion: Challenges with Unsupervised LLM Knowledge Discovery":
However, significant challenges would likely need to be overcome to fruitfully combine these techniques. For example, MAIA is currently focused on interpreting components of vision models, while LLMGuardaril focuses on steering language models, so unifying them would require bridging this modality gap. More fundamentally, even with access to MAIA's automated experimentation and LLMGuardaril's causal analysis, rigorously identifying an LLM's "own" knowledge in a fully unsupervised way, as opposed to the knowledge of a specific simulated entity, remains an open challenge and likely requires further conceptual breakthroughs. Nonetheless, exploring the intersection of scalable interpretability agents like MAIA and causally grounded analysis techniques like LLMGuardaril seems like a promising direction for future work on the ELK problem.'