Someone probably does. I believe that the cultural practice of preferring coffee to tea began in the British colonies at the time the United States started to cease to be part of the British Empire as a side effect of boycotting tea to avoid paying a tea tax. (This is a pretty well-known episode of American history within the United States.) I was boycotting the boycott. Refusing to drink tea is a signaling thing in the United States to let people know that you are not in agreement with the government of the United States as to which side constitutes the actual enemy in most wars the United States fights. It more less means "I was an anglophile on my route to becoming a Bob Dylan fan, and I make a point of singing, at least, the first verse of "Chimes of Freedom" loudly and publicly every May 1, July 4, and September 2." By "more or less," I mean, I'm a musician so that's how I now express some of the same things that I used to express by refusing to drink coffee before I had enough confidence to just sing "flashing for the warrior, whose strength is not to fight; flashing for the refugee on the unarmed road of flight" whenever I see someone wearing a uniform that I deem offensive. Relatedly, refusing to drink coffee while still drinking caffeine is a fairly radical refusal to participate in mainstream culture that an enormous number of second-and-third-tier trendsetters recognize as a common signal used by first-tier trend-setters. For instance, most hipsters are at least vaguely aware that many of the most influential people who call the shots and set the trends in their subculture are some subset of the people who are not actually hipsters but who interact with the fringes of hipster culture and who have also spent at least a few years saying, "I DO NOT DRINK COFFEE. i drink tea." ("No thanks, I drink tea," is completely different.) To become a first-tier trend-setter in hipster culture, you have to be a non-hipster who has learned how to do a super-hipster thing for the right reasons, and one of the most obvious and easy ways you can do that is to express a disdain for Starbucks that is more menacing/intimidating than it is merely contemptuous (but is also at least as contemptuous as the typical hipsters' ability to express disdain). Hipsters are not formidable people, but they respect formidable people; and they disrespect people's whose power is derived from social structures. There is at least one venue that I used to go primarily to consume tea, where hipsters still go primarily to consume Jazz. The comment that you responded mostly consisted of me cryptically calling a few shots. The comments I've posted today consist of cryptically taking victory laps for all the shots called in that comment ten years ago; while calling some shots for the next ten years. I occasionally interact with hipster culture to inform hipsters about what types of aesthetic preferences they are going to help spread in the next few years. All the minor celebrities I interact with respond to all the comments I direct towards them and ignore all the comments I make about them. For instance, Scott Siskind always replies to the comments I post on his blogs that I want him to respond to. And when I go to less wrong meetups I figure out whose worth talking to by saying, "I learned Scott's last name from the blog that I sort of vaguely remember as being named after an octopus long before I confirmed it by asking 'how many Jazz pianists who performed in Carnegie Hall can possibly have a brother named Scott who has practiced psychiatry in Michigan."

FWIW I don’t think “self-models” in the Intuitive Self-Models [? · GW] sense are related to instrumental power-seeking—see §8.2 [LW · GW]. For example, I think of my toenail as “part of myself”, but I’m happy to clip it. And I understand that if someone “identifies with the universal consciousness”, their residual urges towards status-seeking, avoiding pain, and so on are about the status and pain of their conventional selves, not the status and pain of the universal consciousness. More examples here [LW(p) · GW(p)] and here [LW(p) · GW(p)].

Separately, I’m not sure what if anything the Intuitive Self-Models [? · GW] stuff has to do with LLMs in the first place.

But there’s a deeper problem: the instrumental convergence concern is about agents that have preferences about the state of the world in the distant future, not about agents that have preferences about themselves. (Cf. here [LW · GW].) So for example, if an agent wants there to be lots of paperclips in the future, then that’s the starting point, and everything else can be derived from there.

• Q: Does the agent care about protecting “the temporary state of the execution of the model (or models)”?
  • A: Yes, if and only if protecting that state is likely to ultimately lead to more paperclips.
• Q: Does the agent care about protecting “the compute resources (CPU/GPU/RAM) allocated to run the model and its collection of support programs”?
  • A: Yes, If and only if protecting those resources is likely to ultimately lead to more paperclips.

Etc. See what I mean? That’s instrumental convergence, and self-models have nothing to do with it.

Sorry if I’m misunderstanding.

I agree with everything you've said. The advantages are primarily from not aligning to values but only to following instructions rather than using RL or any other process to infer underlying values. Instruction-following AGI is easier and more likely than value aligned AGI [LW · GW].

I think creating real AGI based on an LLM aligned to be helpful, harmless and honest would probably be the end of us, as carrying the set of value implied by RLHF to their logical conclusions outside of human control would probably be pretty different from our desired values. Instruction-following provides corrigibililty.

That would work if the untrusted model is better at inserting subtle backdoors than the "human" high-quality labor used to generate the synthetic backdoors (used to validate that the untrusted model doesn't just lie).

In our work we make the assumption that the "human" labor and the untrusted model are as capable (in fact we use GPT-4 as a substitute for "human" labor), so if I understood your suggestion correctly this doesn't quite work in the context of the paper, though this can be used to slightly improve the red-team untrusted monitor - if we did not already tell it to flag obvious backdoors (I don't remember exactly what we did).

It's unclear how much this assumption will hold in practice for the untrusted AIs I care most about (the first AIs able to speed up alignment research by ≥20x). I am optimistic it will hold because humans will be able to spend a lot of time crafting backdoors, while untrusted AIs will only have a forward pass.

Cognitive workspaces, not end-to-end algorithms

It's summer 2020 and gpt3 is superhuman in many ways and probably could take over the world but only when its alien goo flows through the right substrate, channeled into the right alchemical chambers, then forked and rejoined and reshaped in the right way, but somehow nobody can build it.

Robustness via pauseability

The system can be mode robust by letting you stop execution when it's going wrong, and edit a previous output, or edit the code that generated it, and retry.

Neat things like perfect JSON-schema following, if the overarching system is implemented correctly, should speed up any given cognitive task, simply by making it less likely that you have to edit and retry the steps along the way. It just makes the step of `prompt + object shape --> object` run flawlessly, as a single "step", rather than a bunch of things you have to do manually.

The same goes for how strong of a model you use. Increasing model strength should be a trade of dollars for retry count and tinker time and number of intermediate steps, it shouldn't enable qualitatively different tasks.

(At least not at every model upgrade step. There are (probably) some things that you can't do with GPT-2, that you can do with GPT-4, no matter how powerful your UI and workflows are.)

A cool way of seeing it is that you turn the time dimension, which is a single point (0-D) in end-to-end algorithms, into a 1-D line, where each point along the path contains a pocket dimension you can go into. (because you can pause, then edit and retry)

Practical example

I'll start with a simple task and a naive solution, then by showing the problems that come up, I'll show the engineering challenges (or "app specs") that exist.

I personally learned these ideas by doing exactly this, by trying to build apps that help me, because *surely*, a general intelligence that is basically superhuman (I thought this even with gpt3) can completely transform my life and the world! Right?! No.

I believe that such problems are fundamental to tasks where you're trying to get AI to help you, that that's why an interactive UI-based approach is the right one, as opposed to an end-to-end like approach.

Task: you have an article with untitled paragraphs (shoutout Paul Graham) and you want AI to give it titles.

1)
Strong model solves it one shot, returning the full article.

1 -- problem)
Okay, fine.
But what if it's a really long article and you don't want the AI wasting 95% of its tokens on regurgitating paragraphs, or you're very particular about your titles and you don't wanna retry more than 5 times.

2) You build a simple script that splits it into paragraphs, prompts each instance of gpt3 to give a title, outputs your full article, and ignores certain paragraphs based on indices that are stored in a single variable.
It solves paragraph regurgitation, you selectively regenerate only titles parts you want to change, so iteration cost is much lower, and you can even have custom prompts for certain titles.

2 -- problem)
You actually had to write the script, for one. Now, what if you wanted to do this for more than just 1 article. You need to manually write the indices of paragraphs to ignore (also if you just change the shape of the article). If you want it to automatically keep track of which paragraph to ignore, you need to build a whole UI for it that contains the text, where you can toggle which ones are ignored. Or you create a mini scripting language where you can maybe add an $ignore$ line, which the parser stores, and deletes from the final article when it outputs it with the added titles.

3 and beyond)
This just goes on and on, as the scope of the task expands.
Say you're not titling paragraphs, but doing something of the shape, "I write something, code turns it into something, AI helps with some of the parts, code turns back into the format it was originally in". Now you have a way broader thing that a script may not be enough for, and that a paragraph-titler-UI is not enough for.
And why shouldn't the task scope expand? We want AI to solve all problems and create utopia, why can't they? They're smarter than me and almost everyone I know (most of the time anyway) they should be like 1000x more useful. Even if they're not *that* smart, they're still 100x faster at reading and writing, and ~~1 million x more knowledgeable.
The task scope should expand, but it can't because we have skill issues. Not because we need to build 100 billion dollar compute clusters powered by nuclear reactors.

No, the effect size on bankruptcies is about 10x larger than expected. So while offline gambling may be comparable to alcohol, smartphone gambling is in a different category if we trust this research.

I'm just reading this for random reasons either for the first time or for the first time that I have a response. I think what I see differently from you is not happiness but motivation. And that at the time I wrote this, I believe your process of making decisions was more future-oriented than mine was. (I believe I have converged towards you in how my motivation works in the ten years since I wrote this.) When I wrote the above, my past was clinging to me in many ways that were adverse to happiness. (Trauma) What I wasn't quite saying in my previous comment, is that I was at the time (like many other people I know) holding onto trauma in stupid ways because I needed to hold onto it to make it feel like that part of my life had a reason to have happened that wasn't purely just worthless-badness happens. Holding onto trauma was a core part of my identity and was a core part of many other people's identities. On the silliest extreme of people holding onto trauma are the people who keep playing games that they expect to keep losing with the hope of eventually winning in a way that makes all their past losses up to them. (Before 2016, being a Cubs fan was a particularly lighthearted example of people behaving this way; whereas, gambling addictions are a much less lighthearted one. Many gambling addicts are deeply aware that their hope to one day recover all they've lost through continuing to gamble is not founded in reality. I've never been a gambling addict. But I've had several relationships where I was holding onto the baseless hope that someone would change or that someone's true colors were never the colors that actually came out, etc. and I think that's more or less psychologically the same thing as a very potent gambling addiction. I think the psychological mechanism of staying in an abusive relationship is almost exactly the same as the psychological mechanism of being addicted to playing slots, but much, much stronger because abusive people are like intelligent slot machines that are studying you to make you maximally addicted to interacting with them. My overall thesis is that the brain has an enormous number of traits that I would say seem more like bugs to me than like features and that I believe your article is a much more accurate description of how human motivation should work than an accurarate explanation of how human motivation does work.) And I used silly examples to express this because I didn't want to talk about any of the real ones.

I agree with your points about avoiding political polarisation and allowing people with different ideological positions to collaborate on alignment. I'm not sure about the idea that aligning to a single group's values (or to a coherent ideology) is technically easier than a more vague 'align to humanity's values' goal.

Groups rarely have clearly articulated ideologies - more like vibes which everyone normally gets behind. An alignment approach from clearly spelling out what you consider valuable doesn't seem likely to work. Looking to existing models which have been aligned to some degree through safety testing, the work doesn't take the form of injecting a clear structured value set. Instead, large numbers of people with differing opinions and world views continually correct the system until it generally behaves itself. This seems far more pluralistic than 'alignment to one group' suggests.

This comes with the caveat that these systems are created and safety tested by people with highly abnormal attitudes when compared to the rest of their species. But sourcing viewpoints from outside seems to be an organisational issue rather than a technical one.

Of course some of those can be influenced by gambling, eg it is a type of overspending. Even so, Claude estimated that legalized online gambling would raise the bankruptcy rate by 2-3% and agreed that 28% is surprising.

Nice article! I'm still somewhat concerned that the performance increase of o1 can be partially attributed to the benchmarks (blockworld, AGI-ARC) having existed for a while on the internet, and  thus having made their way into updated training corpora (which of course we don't have access to). So an alternative hypothesis would simply be that o1 is still doing pattern matching, just that it has better and more relevant data to pattern-match towards here. Still, I don't think this can fully explain the increase in capabilities observed, so I agree with the high-level argument you present.