FWIW I don’t think “self-models” in the Intuitive Self-Models [? · GW] sense are related to instrumental power-seeking—see §8.2 [LW · GW]. For example, I think of my toenail as “part of myself”, but I’m happy to clip it. And I understand that if someone “identifies with the universal consciousness”, their residual urges towards status-seeking, avoiding pain, and so on are about the status and pain of their conventional selves, not the status and pain of the universal consciousness. More examples here [LW(p) · GW(p)] and here [LW(p) · GW(p)].

Separately, I’m not sure what if anything the Intuitive Self-Models [? · GW] stuff has to do with LLMs in the first place.

But there’s a deeper problem: the instrumental convergence concern is about agents that have preferences about the state of the world in the distant future, not about agents that have preferences about themselves. (Cf. here [LW · GW].) So for example, if an agent wants there to be lots of paperclips in the future, then that’s the starting point, and everything else can be derived from there.

• Q: Does the agent care about protecting “the temporary state of the execution of the model (or models)”?
  • A: Yes, if and only if protecting that state is likely to ultimately lead to more paperclips.
• Q: Does the agent care about protecting “the compute resources (CPU/GPU/RAM) allocated to run the model and its collection of support programs”?
  • A: Yes, If and only if protecting those resources is likely to ultimately lead to more paperclips.

Etc. See what I mean? That’s instrumental convergence, and self-models have nothing to do with it.

Sorry if I’m misunderstanding.

I agree with everything you've said. The advantages are primarily from not aligning to values but only to following instructions rather than using RL or any other process to infer underlying values. Instruction-following AGI is easier and more likely than value aligned AGI [LW · GW].

I think creating real AGI based on an LLM aligned to be helpful, harmless and honest would probably be the end of us, as carrying the set of value implied by RLHF to their logical conclusions outside of human control would probably be pretty different from our desired values. Instruction-following provides corrigibililty.

That would work if the untrusted model is better at inserting subtle backdoors than the "human" high-quality labor used to generate the synthetic backdoors (used to validate that the untrusted model doesn't just lie).

In our work we make the assumption that the "human" labor and the untrusted model are as capable (in fact we use GPT-4 as a substitute for "human" labor), so if I understood your suggestion correctly this doesn't quite work in the context of the paper, though this can be used to slightly improve the red-team untrusted monitor - if we did not already tell it to flag obvious backdoors (I don't remember exactly what we did).

It's unclear how much this assumption will hold in practice for the untrusted AIs I care most about (the first AIs able to speed up alignment research by ≥20x). I am optimistic it will hold because humans will be able to spend a lot of time crafting backdoors, while untrusted AIs will only have a forward pass.

Cognitive workspaces, not end-to-end algorithms

It's summer 2020 and gpt3 is superhuman in many ways and probably could take over the world but only when its alien goo flows through the right substrate, channeled into the right alchemical chambers, then forked and rejoined and reshaped in the right way, but somehow nobody can build it.

Robustness via pauseability

The system can be mode robust by letting you stop execution when it's going wrong, and edit a previous output, or edit the code that generated it, and retry.

Neat things like perfect JSON-schema following, if the overarching system is implemented correctly, should speed up any given cognitive task, simply by making it less likely that you have to edit and retry the steps along the way. It just makes the step of `prompt + object shape --> object` run flawlessly, as a single "step", rather than a bunch of things you have to do manually.

The same goes for how strong of a model you use. Increasing model strength should be a trade of dollars for retry count and tinker time and number of intermediate steps, it shouldn't enable qualitatively different tasks.

(At least not at every model upgrade step. There are (probably) some things that you can't do with GPT-2, that you can do with GPT-4, no matter how powerful your UI and workflows are.)

A cool way of seeing it is that you turn the time dimension, which is a single point (0-D) in end-to-end algorithms, into a 1-D line, where each point along the path contains a pocket dimension you can go into. (because you can pause, then edit and retry)

Practical example

I'll start with a simple task and a naive solution, then by showing the problems that come up, I'll show the engineering challenges (or "app specs") that exist.

I personally learned these ideas by doing exactly this, by trying to build apps that help me, because *surely*, a general intelligence that is basically superhuman (I thought this even with gpt3) can completely transform my life and the world! Right?! No.

I believe that such problems are fundamental to tasks where you're trying to get AI to help you, that that's why an interactive UI-based approach is the right one, as opposed to an end-to-end like approach.

Task: you have an article with untitled paragraphs (shoutout Paul Graham) and you want AI to give it titles.

1)
Strong model solves it one shot, returning the full article.

1 -- problem)
Okay, fine.
But what if it's a really long article and you don't want the AI wasting 95% of its tokens on regurgitating paragraphs, or you're very particular about your titles and you don't wanna retry more than 5 times.

2) You build a simple script that splits it into paragraphs, prompts each instance of gpt3 to give a title, outputs your full article, and ignores certain paragraphs based on indices that are stored in a single variable.
It solves paragraph regurgitation, you selectively regenerate only titles parts you want to change, so iteration cost is much lower, and you can even have custom prompts for certain titles.

2 -- problem)
You actually had to write the script, for one. Now, what if you wanted to do this for more than just 1 article. You need to manually write the indices of paragraphs to ignore (also if you just change the shape of the article). If you want it to automatically keep track of which paragraph to ignore, you need to build a whole UI for it that contains the text, where you can toggle which ones are ignored. Or you create a mini scripting language where you can maybe add an $ignore$ line, which the parser stores, and deletes from the final article when it outputs it with the added titles.

3 and beyond)
This just goes on and on, as the scope of the task expands.
Say you're not titling paragraphs, but doing something of the shape, "I write something, code turns it into something, AI helps with some of the parts, code turns back into the format it was originally in". Now you have a way broader thing that a script may not be enough for, and that a paragraph-titler-UI is not enough for.
And why shouldn't the task scope expand? We want AI to solve all problems and create utopia, why can't they? They're smarter than me and almost everyone I know (most of the time anyway) they should be like 1000x more useful. Even if they're not *that* smart, they're still 100x faster at reading and writing, and ~~1 million x more knowledgeable.
The task scope should expand, but it can't because we have skill issues. Not because we need to build 100 billion dollar compute clusters powered by nuclear reactors.

No, the effect size on bankruptcies is about 10x larger than expected. So while offline gambling may be comparable to alcohol, smartphone gambling is in a different category if we trust this research.

I'm just reading this for random reasons either for the first time or for the first time that I have a response. I think what I see differently from you is not happiness but motivation. And that at the time I wrote this, I believe your process of making decisions was more future-oriented than mine was. (I believe I have converged towards you in how my motivation works in the ten years since I wrote this.) When I wrote the above, my past was clinging to me in many ways that were adverse to happiness. (Trauma) What I wasn't quite saying in my previous comment, is that I was at the time (like many other people I know) holding onto trauma in stupid ways because I needed to hold onto it to make it feel like that part of my life had a reason to have happened that wasn't purely just worthless-badness happens. Holding onto trauma was a core part of my identity and was a core part of many other people's identities. On the silliest extreme of people holding onto trauma are the people who keep playing games that they expect to keep losing with the hope of eventually winning in a way that makes all their past losses up to them. (Before 2016, being a Cubs fan was a particularly lighthearted example of people behaving this way; whereas, gambling addictions are a much less lighthearted one. Many gambling addicts are deeply aware that their hope to one day recover all they've lost through continuing to gamble is not founded in reality. I've never been a gambling addict. But I've had several relationships where I was holding onto the baseless hope that someone would change or that someone's true colors were never the colors that actually came out, etc. and I think that's more or less psychologically the same thing as a very potent gambling addiction. I think the psychological mechanism of staying in an abusive relationship is almost exactly the same as the psychological mechanism of being addicted to playing slots, but much, much stronger because abusive people are like intelligent slot machines that are studying you to make you maximally addicted to interacting with them. My overall thesis is that the brain has an enormous number of traits that I would say seem more like bugs to me than like features and that I believe your article is a much more accurate description of how human motivation should work than an accurarate explanation of how human motivation does work.) And I used silly examples to express this because I didn't want to talk about any of the real ones.

I agree with your points about avoiding political polarisation and allowing people with different ideological positions to collaborate on alignment. I'm not sure about the idea that aligning to a single group's values (or to a coherent ideology) is technically easier than a more vague 'align to humanity's values' goal.

Groups rarely have clearly articulated ideologies - more like vibes which everyone normally gets behind. An alignment approach from clearly spelling out what you consider valuable doesn't seem likely to work. Looking to existing models which have been aligned to some degree through safety testing, the work doesn't take the form of injecting a clear structured value set. Instead, large numbers of people with differing opinions and world views continually correct the system until it generally behaves itself. This seems far more pluralistic than 'alignment to one group' suggests.

This comes with the caveat that these systems are created and safety tested by people with highly abnormal attitudes when compared to the rest of their species. But sourcing viewpoints from outside seems to be an organisational issue rather than a technical one.

Of course some of those can be influenced by gambling, eg it is a type of overspending. Even so, Claude estimated that legalized online gambling would raise the bankruptcy rate by 2-3% and agreed that 28% is surprising.

Nice article! I'm still somewhat concerned that the performance increase of o1 can be partially attributed to the benchmarks (blockworld, AGI-ARC) having existed for a while on the internet, and  thus having made their way into updated training corpora (which of course we don't have access to). So an alternative hypothesis would simply be that o1 is still doing pattern matching, just that it has better and more relevant data to pattern-match towards here. Still, I don't think this can fully explain the increase in capabilities observed, so I agree with the high-level argument you present. 

Looking for specific tips and tricks to break AI out of formal/corporate writing patterns. Tried style mimicry ('write like Hemingway') and direct requests ('be more creative') - both fell flat. What works?

Should I be using different AI models ( I am using GPT and Claude)? The base models output an enormous creative storm, but somehow the RLHF has partially lobotomized LLMs such that they always seem to output either cheesy stereotypes or overly verbose academise/corporatespeak.