Wei Dai's Shortform

Apparently Gemini 1.5 Pro isn't working great with large contexts:

While this worked well, for even a slightly more complicated problem the model failed. One Twitter user suggested just adding a random ‘iPhone 15’ in the book text and then asking the model if there is anything in the book that seems out of place in the book. And the model failed to locate it.

The same was the case when the model was asked to summarize a 30-minute Mr. Beast video (over 300k tokens). It generated the summary but many people who had watched the video pointed out that the summary was mostly incorrect.

So while on paper this looked like a huge leap forward for Google, it seems that in practice it's not performing as well as they might have hoped.

But is this due to limitations of RLHF training, or something else?

RLHF with humans might also soon get obsoleted by things like online DPO where another chatbot produces preference data for on-policy responses of the tuned model, and there is no separate reward model in the RL sense. If generalization from labeling instructions through preference decisions works in practice, even weak-to-strong setting won't necessarily be important, if tuning of a stronger model gets bootstrapped by a weaker model (where currently SFT from an obviously off-policy instruct dataset seems to suffice), but then the stronger model re-does the tuning of its equally strong successor that starts with the same base model (as in the self-rewarding paper), using some labeling instructions ("constitution"). So all that remains of human oversight that actually contributes to the outcome is labeling instructions written in English, and possibly some feedback on them from spot checking what's going on as a result of choosing particular instructions.

My guess is that we're currently effectively depending on generalization. So "Good" from your decomposition. (Though I think depending on generalization will produce big issues if the model is scheming, so I would prefer avoiding this.)

since it would be too expensive to do RLHF with humans reading diverse 1M+ contexts from scratch

It's plausible to me that after doing a bunch of RLHF on short contexts, RLHF on long contexts is extremely sample efficient (when well tuned) such that only (e.g.) 1,000s of samples sufficies^[1]. If you have a $2,000,000 budget for long context RLHF and need only 1,000 samples, you can spend $2,000 per sample. This gets you perhaps (e.g.) 10 hours of time of an experienced software engineer which might suffice for good long context supervision without necessarily needing any fancy scalable oversight approaches. (That said, probably people will use another LLM by default when trying to determine the reward if their spending this long: recursive reward modeling seems almost certain by default if we're assuming that people spend this much time labeling.)

That said, I doubt that anyone has actually started doing extremely high effort data labeling like this, though plausibly they should...

From a previous comment: [...] This seems to be evidence that RLHF does not tend to generalize well out-of-distribution

It's some evidence, but exploiting a reward model seems somewhat orthogonal to generalization out of distribution: exploitation is heavily selected for.

(Separately, I expect that the quoted comment results in a misleadingly negative perception of the current situation.)

Bad: AI developers haven't taken alignment seriously enough to have invested enough in scalable oversight, and/or those techniques are unworkable or too costly, causing them to be unavailable.

Turns out at least one scalable alignment team has been struggling for resources. From Jan Leike (formerly co-head of Superalignment at OpenAI):

Over the past few months my team has been sailing against the wind. Sometimes we were struggling for compute and it was getting harder and harder to get this crucial research done.

Even worse, apparently the whole Superalignment team has been disbanded.

I have access to Gemini 1.5 Pro. Willing to run experiments if you provide me with an exact experiment to run, plus cover what they charge me (I'm assuming it's paid, I haven't used it yet).

This probably does help capitalize AI companies a little bit, demand for call options will create demand for the underlying. This is probably a relatively small effect (?), but I'm not confident in my ability to estimate this at all.

Do these options have a chance to default / are the sellers stable enough?

It seems hard to buy AI companies at the moment. The only way is to buy tech giants like Microsoft, Google, nVidea which are already valuad very highly - seems like's it's somewhat priced in. It's also not clear to me that most of the value of AI will accrue to them. I'm confused about this though. 

It would seem one would want to buy Nasdaq rather than SPX? On the other hand, maybe most tech companies will be wiped out by AI - it's the world of atoms that would gain relative value. 

I think the main reason why UDT is not discussed in academia is that it is not a sufficiently rigorous proposal, as well as there not being a published paper on it. Hilary Greaves says the following in this 80k episode:

Then as many of your listeners will know, in the space of AI research, people have been throwing around terms like ‘functional decision theory’ and ‘timeless decision theory’ and ‘updateless decision theory’. I think it’s a lot less clear exactly what these putative alternatives are supposed to be. The literature on those kinds of decision theories hasn’t been written up with the level of precision and rigor that characterizes the discussion of causal and evidential decision theory. So it’s a little bit unclear, at least to my likes, whether there’s genuinely a competitor to decision theory on the table there, or just some intriguing ideas that might one day in the future lead to a rigorous alternative.

I also think it is unclear to what extent UDT and updateless are different from existing ideas in academia that are prima facie similar, like McClennen's (1990) resolute choice and Meacham's (2010, §4.2) cohesive decision theory.^[1] Resolute choice in particular has been discussed in a lot of detail, and for a long time (see the citations of McClennen's book). (And, FWIW, my sense is that most philosophers think that resolute choice is irrational and/or doesn't make sense, at least if it is cashed out as a decision rule based on preference.)

It also doesn't help that it is unclear what the difference between FDT and UDT is supposed to be. 

(If UDT is supposed to be an LDT of some sort, then you might want to check out Spohn's (2012)^[2] version of CDT, Fisher's (n.d) disposition-based decision theory, and Poellinger's (2013) discussion of Spohn's theory, for ideas in academia that are similar to the LDT-part of the theory. And then there is also Schwarz' critique of FDT, which would then also apply to UDT, at least partially.)

1. ^{^}

   My own take, using the terminology listed here [LW · GW], is that the causalist version of Meacham's cohesive decision theory is basically "updateless CDT", that the evidentialist version is basically "updateless EDT", and that a Spohn-CDT version of cohesive decision theory is basically "U(C)DT/F(C)DT". I also think that resolute choice is much more permissive than e.g. cohesive decision theory and updatelessness. As a decision rule, it doesn't recommend anything close to "maximizing EU relative to your prior". Instead, it just states that (i) what you prefer ex ante in a dynamic choice problem should be the same as you what you prefer in the normalised version of the problem, and (ii) you should be dynamically consistent (i.e., the most preferred plan should not change throughout the decision problem). 

2. ^{^}

   Note that in the published article, it says that the article was received in 2008.

I was thinking of writing a short post kinda on this topic (EDIT TO ADD: it’s up! See Some (problematic) aesthetics of what constitutes good work in academia [LW · GW]), weaving together:

• Holden on academia not answering important questions [LW · GW]
• This tweet I wrote on the aesthetics of what makes a “good” peer-reviewed psychology paper
• Something about the aesthetics of what makes a “good” peer-reviewed AI/ML paper, probably including the anecdote where DeepMind wrote a whole proper academia-friendly ML paper whose upshot was the same as a couple sentences in an old Abram Demski blog post [LW(p) · GW(p)]
• Something about the aesthetics of what makes a “good” peer-reviewed physics paper, based on my personal experience, probably including my anecdote about solar cell R&D from here [LW · GW]
• Not academia but bhauth on the aesthetics of what makes a “good” VC pitch [LW · GW]
• maybe a couple other things (suggestions anyone?)
• Homework problem for the reader: what are your “aesthetics of success”, and how are they screwing you over?

Aside from the literature on international relations, I don't know much about academic dysfunction (mostly from reading parts of Inadequate Equilibria [? · GW], particularly the visitor dialog [? · GW]) and other Lesswrong people can probably cover it better. I think that planecrash [LW · GW], Yud's second HPMOR-scale work, mentions that everyone in academia just generally avoids citing things published outside of academia, because they risk losing status if they do.

EDIT: I went and found that section, it is here:

It turns out that Earth economists are locked into powerful incentive structures of status and shame, which prevent them from discussing the economic work of anybody who doesn't get their paper into a journal.  The journals are locked into very powerful incentive structures that prevent them from accepting papers unless they're written in a very weird Earth way that Thellim can't manage to imitate, and also, Thellim hasn't gotten tenure at a prestigious university which means they'll probably reject the paper anyways.  Thellim asks if she can just rent temporary tenure and buy somebody else's work to write the paper, and gets approximately the same reaction as if she asked for roasted children recipes.

The system expects knowledge to be contributed to it only by people who have undergone painful trials to prove themselves worthy.  If you haven't proven yourself worthy in that way, the system doesn't want your knowledge even for free, because, if the system acknowledged your contribution, it cannot manage not to give you status, even if you offer to sign a form relinquishing it, and it would be bad and unfair for anyone to get that status without undergoing the pains and trials that others had to pay to get it.

She went and talked about logical decision theory online before she'd realized the full depth of this problem, and now nobody else can benefit from writing it up, because it would be her idea and she would get the status for it and she's not allowed to have that status.  Furthermore, nobody else would put in the huge effort to push forward the idea if she'll capture their pay in status.  It does have to be a huge effort; the system is set up to provide resistance to ideas, and disincentivize people who quietly agreed with those ideas from advocating them, until that resistance is overcome.  This ensures that pushing any major idea takes a huge effort that the idea-owner has to put in themselves, so that nobody will be rewarded with status unless they have dedicated several years to pushing an idea through a required initial ordeal before anyone with existing status is allowed to help, thereby proving themselves admirable enough and dedicated enough to have as much status as would come from contributing a major idea.

To suggest that the system should work in any different way is an obvious plot to steal status that is only deserved by virtuous people who work hard, play by the proper rules, and don't try to cheat by doing anything with less effort than it's supposed to take.

It's glowfic, so of course I don't know how accurate it is as it's intended to plausibly deniable enough to facilitate free writing (while keeping things entertaining enough to register as not-being-work).

I think that UDASSA and UDT might be in academia's blind spots in the same way that the Everett interpretation is [LW · GW]: more correct theories that came after less correct theories with mostly only theoretical evidence to support changing over to the new theories.

Many parts of academia have a strong Not Invented Here tendency. Not just research outside of academia is usually ignored, but even research outside a specific academic citation bubble, even if another bubble investigates a pretty similar issue. For example, economic decision theorists ignore philosophical decisions theorists, which in turn mostly ignore the economic decision theorists. They each have their own writing style and concerns and canonical examples or texts. Which makes it hard for outsiders to read the literature or even contribute to it, so they don't.

A striking example is statistics, where various fields talk about the same mathematical thing with their own idiosyncratic names, unaware or unconcerned whether it already had a different name elsewhere.

Edit: Though LessWrong is also a citation bubble to some degree.

Have you tried talking to professors about these ideas?

Indeed, there is no need for sorrow, for by choosing to remain anonymous, you have done great things. The world owes you a Nobel Prize in Economics and a Turing Award. It is time for the world to seriously recognize your achievements and lead it towards a financial system without bubbles.

Why you hadn't wrote academic articles on these topics? 

The secret is that academic article is just a formatting type and anyone can submit to scientific journals. No need to have a PhD or even work in a scientific institution.

Clearly academia has some blind spots, but how big? Do I just have a knack for finding ideas that academia hates, or are the blind spots actually enormous?

From someone who left a corner of it: the blindspots could be arbitrarily large as far as I know, because there seemed to me to be no real explicit culture of Hamming questions/metalooking for anything neglected. You worked on something vaguely similar/related to your advisor's work, because otherwise you can't get connections to people who know how to attack the problem.

Isn't the basic idea of Constitutional AI just having the AI provide its own training feedback using written instruction? My guess is there was a substantial amount of self-evaluation in the o1 training with complicated written instructions, probably kind of similar to a constituion (though this is just a guess).

These posts might be relevant:

• A recipe for frontier model post-training
• Futures of the data foundry business model

The details of Constitutional AI seem highly contingent, while the general idea is simply automation of data for post-training, so that the remaining external input is the "constitution". In the original paper there are recipes both for instruction tuning data and for preference data. RLAIF is essentially RLHF that runs on synthetic preference data, maybe together with a recipe for generating it. But preference data could also be used to run DPO or something else, in which case RLAIF becomes a misnomer for describing automation of that preference data.

Llama 3 report suggests that instruction tuning data can be largely automated, but human preference data is still better. And data foundry business is still alive, so a lot of human data is at least not widely recognized as useless. But it's unclear if future models won't soon do better than humans at labeling, or possibly already do better at some leading labs. Meta didn't have a GPT-4 level model as a starting point before Llama 3, and then there are the upcoming 5e26 FLOPs models, and o1-like reasoning models.

As a tangent to my question, I wonder how many AI companies are already using RLAIF and not even aware of it. From a recent WSJ story [LW(p) · GW(p)]:

Early last year, Meta Platforms asked the startup to create 27,000 question-and-answer pairs to help train its AI chatbots on Instagram and Facebook.

When Meta researchers received the data, they spotted something odd. Many answers sounded the same, or began with the phrase “as an AI language model…” It turns out the contractors had used ChatGPT to write-up their responses—a complete violation of Scale’s raison d’être.

So they detected the cheating that time, but in RLHF how would they know if contractors used AI to select which of two AI responses is more preferred?

BTW here's a poem(?) I wrote for Twitter, actually before coming across the above story:

The people try to align the board. The board tries to align the CEO. The CEO tries to align the managers. The managers try to align the employees. The employees try to align the contractors. The contractors sneak the work off to the AI. The AI tries to align the AI.

Maybe others are using it in secret but don't want to admit it for some reason? I can't find any mention of Anthropic having filed a patent on the idea, but maybe other companies would feel too much like it looked like they were second-rate imitators if they said they were copying Anthropic's idea?

Just speculating, I don't know. Sure seems like a useful idea to copy.

Philosophy is frequently (probably most of the time) done in order to signal group membership rather than as an attempt to accurately model the world. Just look at political philosophy or philosophy of religion. Most of the observations you note can be explained by philosophers operating at simulacrum level 3 instead of level 1.

I think most academic philosophers take the difficult of philosophy quite seriously. Metaphilosophy is a flourishing subfield of philosophy; you can find recent papers on the topic here https://philpapers.org/browse/metaphilosophy.  There is also a growing group of academic philosophers working on AI safety and alignment; you can find some recent work here https://link.springer.com/collections/cadgidecih. I think that sometimes the tone of specific papers sounds confident; but that is more stylistic convention than a reflection of the underlying credences. Finally, I think that uncertainty / decision theory is a persistent theme in recent philosophical work on AI safety and other issues in philosophy of AI; see for example this paper, which is quite sensitive to issues about chances of welfare https://link.springer.com/article/10.1007/s43681-023-00379-1.

Hm. I think modern academic philosophy is a raging shitshow, but I thought philosophy on LW was quite good. I haven't been a regular LW user until a couple of years ago, and the philosophical takes here, particularly Eliezer's, converge with my own conclusions after a half lifetime of looking at philosophical questions through the lens of science, particularly neuroscience and psychology.

So: what do you see as the limitations in LW/Yudkowskian philosophy? Perhaps I've overlooked them.

I am currently skeptical that we need better philosophy for good AGI outcomes, vs. better practical work on technical AGI alignment (a category that barely exists) and PR work to put the likely personal intent aligned AGI into the hands of people that give half a crap about understanding or implementing ethics. Deciding on the long term future will be a matter of a long contemplation if we get AGI into good hands. We should decide if that logic is right, and if so, plan the victory party after we've won the war.

I did read your metaphilosophy post and remain unconvinced that there's something big the rest of us are missing.

I'm happy to be corrected (I love becoming less wrong, and I'm aware of many of my biases that might prevent it): 

Here's how it currently looks to me: Ethics are ultimately a matter of preference, the rest is game theory and science (including the science of human preferences). Philosophical questions boil down to scientific questions in most cases, so epistemology is metaphilosophy for the most part.

Change my mind! Seriously, I'll listen. It's been years since I've thought about philosophy hard.

I was just reading Daniel Dennett’s memoir for no reason in particular, it had some interesting glimpses into how professional philosophers actually practice philosophy. Like I guess there’s a thing where one person reads their paper (word-for-word!) and then someone else is the designated criticizer? I forget the details. Extremely different from my experience in physics academia though!!

(Obviously, reading that memoir is probably not the most time-efficient way to learn about the day-to-day practice of academic philosophy.)

(Oh, there was another funny anecdote in the memoir where the American professional philosopher association basically had a consensus against some school of philosophy, and everyone was putting it behind them and moving on, but then there was a rebellion where the people who still liked that school of philosophy did a hostile takeover of the association’s leadership!) 

Academic culture/norms - no or negative rewards for being more modest or expressing confusion. (Moral uncertainty being sometimes expressed because one can get rewarded by proposing some novel mechanism for dealing with it.)

A non-ethics example that jumps to my mind is David Chalmers on the Hard Problem of Consciousness here: “So if I’m giving my overall credences, I’m going to give, 10% to illusionism, 30% to panpsychism, 30% to dualism, and maybe the other 30% to, I don’t know what else could be true, but maybe there’s something else out there.” That’s the only example I can think of but I read very very little philosophy.

What are the issues that are "difficult" in philosophy, in your opinion? What makes them difficult?

I remember you and others talking about the need to "solve philosophy", but I was never sure what it meant by that.

To whom does this not apply? Most people who "work on AI alignment" don't even think that thinking is a thing.

I blame science, math, engineering, entrepreneurship. Philosophy is the practice of the esoteric method [LW(p) · GW(p)], meaning it can't be made truly legible for very long stretches of investigation. This results in accumulation of anti-epistemic hazards, which science doesn't particularly need to have tools for dealing with, because it can filter its reasoning through frequent transitions into legibility.

Philosophy can't rely on such filtering through legibility, it has to maintain sanity the hard way. But as philosophy enviously looks at the more successful endeavors of science, it doesn't see respect for such methods of maintaining sanity in its reasoning, instead it sees that merely moving fast and breaking things works very well. And so the enthusiasm for their development wanes, instead philosophy remains content with the object level questions that investigate particular truths, rather than methods for getting better at telling which cognitive algorithms can more robustly arrive at truths [LW · GW] (rationality, metaphilosophy).

I find the idea interesting: To achieve a certain value of "total genius", we either need a large population with a small fraction of geniuses, or a small population with a large fraction of geniuses.

(A third option is a small population with a small fraction of geniuses... and it takes a lot of time. The geniuses read each other's books, rather than talk to each other directly. I think it was like this in the past. Very inefficient, because the information transfer by reading books is one-sided; does not allow collaboration in real time.)

I wonder how the heritability of IQ works, versus the reversion to the mean. Despite Pol Pot's dystopian project, the average IQ in Cambodia seems to be... average. What would happen to a country where let's say half of the children are produced by artificial insemination, and half of the sperm comes from fathers with IQ 130 and above? If the mother is average, the child is likely to be an average between 100 and 130, so 115. On one hand, nothing exceptional; on the other hand, if the baseline is now slightly higher, then the next generation... and here comes the question how exactly the reversion to the mean works, and whether the constant injections of IQ 130 genes in the population could outrun it.

Possibly relevant anecdote: Once I was with a group of people who tried various psychological experiments. That day, the organizers proposed that we play iterated Prisonner's Dilemma. I was like "yay, I know the winning strategy, this will be so easy!"

I lost. Almost everyone always defected against me; there wasn't much I could do to get points comparable to other people who mostly cooperated with each other.

After the game, I asked why. (During the game, we were not allowed to communicate, just to write our moves.) The typical answer was something like: "well, you are obviously very smart, so no matter what I do, you will certainly find a way to win against me, so my best option is to play it safe and always defect, to avoid the worst outcome".

I am not even sure if I should be angry at them. I suppose that in real life, when you have about average intelligence, "don't trust people visibly smarter than you" is probably a good strategy, on average, because there are just too many clever scammers walking around. At the same time I feel hurt, because I am a natural altruist and cooperator, so this feels extremely unfair, and a loss for both sides.

(There were other situations in my life where the same pattern probably also applied, but most of the time, you just don't know why other people do whatever they do. This time I was told their reasoning explicitly.)

What (human or not) phenomena do you think are well explained by this model? I tried to think of any for 5 minutes and the best I came up with was the strong egalitarianism among hunter gatherers. I don't actually know that much about hunter gatherers though. In the modern world something where "high IQ" people are doing worse is sex, but it doesn't seem to fit your model.

Comparing my current message to his, he talks about "selfishness" and explicitly disclaims, "most humans are not evil" (why did he say this?), and focuses on everyday (e.g. consumer) behavior instead of what "power reveals".

The reason I said "most humans are not evil" is because I honestly don't think the concept of evil, as normally applied, is a truthful way to describe most people. Evil typically refers to an extraordinary immoral behavior, in the vicinity of purposefully inflicting harm to others in order to inflict harm intrinsically, rather than out of indifference, or as a byproduct of instrumental strategies to obtain some other goal. I think the majority of harms that most people cause are either (1) byproducts of getting something they want, which is not in itself bad (e.g. wanting to eat meat), or (2) the result of their lack of will to help others (e.g. refusing to donate any income to those in poverty).

By contrast, I focused on consumer behavior because the majority of the world's economic activity is currently engaged in producing consumer products and services. There exist possible worlds in which this is not true. During World War 2, the majority of GDP in Nazi Germany was spent on hiring soldiers, producing weapons of war, and supporting the war effort more generally—which are not consumer goods and services.

Focusing on consumer preferences a natural thing to focus on if you want to capture intuitively "what humans are doing with their wealth", at least in our current world. Before focusing on something else by default—such as moral preferences—I'd want to hear more about why those things are more likely to be influential than ordinary consumer preferences in the future. 

You mention one such argument along these lines:

I guess I wasn't as worried because it seemed like humans are altruistic enough, and their selfish everyday desires limited enough that as they got richer and more powerful, their altruistic values would have more and more influence.

I just think it's not clear it's actually true that humans get more altruistic as they get richer. For example, is it the case that selfish consumer preferences have gotten weaker in the modern world, compared to centuries ago when humans were much poorer on a per capita basis? I have not seen a strong defense of this thesis, and I'd like to see one before I abandon my focus on "everyday (e.g. consumer) behavior".

I like the insight regarding power corrupting or revealing. I think perhaps both might be true and, if so, we should keep both lines of though in mind when thinking about these types of questions.

My general view is that most people are generally good when you're talking about individual interactions. I'm less confident in that when one brings in the in group-out of group aspects. I just am not sure how to integrate all that into a general view or princple about human nature.

A line I heard in some cheesey B-grade horror movies, related to the question of a personal nature and the idea that we all have competing good and bad wolves inside. One of the characters asks which wolve was strongest, the good wolf or the bad wolf. The answer was "Which do you feed the most?"

My model is that the concept of "morality" is a fiction which has 4 generators that are real:

• People have empathy, which means they intrinsically care about other people (and sufficiently person-like entities), but, mostly about those in their social vicinity [LW(p) · GW(p)]. Also, different people have different strength of empathy, a minority might have virtually none.
• Superrational cooperation is something that people understand intuitively to some degree. Obviously, a minority of people understand it on System 2 level as well.
• There is something virtue-ethics-like which I find in my own preferences, along the lines of "some things I would prefer not to do, not because of their consequences, but because I don't want to be the kind of person who would do that". However, I expect different people to differ in this regard.
• The cultural standards of morality, which it might be selfishly beneficial to go along with, including lying to yourself that you're doing it for non-selfish reasons. Which, as you say, becomes irrelevant once you secure enough power. This is a sort of self-deception which people are intuitively skilled at.

I don't think altruism is evolutionarily connected to power as you describe. Caesar didn't come to power by being better at altruism, but by being better at coordinating violence. For a more general example, the Greek and other myths don't give many examples of compassion (though they give many other human values), it seems the modern form of compassion only appeared with Jesus, which is too recent for any evolutionary explanation.

So it's possible that the little we got of altruism and other nice things are merely lucky memes. Not even a necessary adaptation, but more like a cultural peacock's tail, which appeared randomly and might fix itself or not. While our fundamental nature remains that of other living creatures, who eat each other without caring much.

"willing to inflict large amounts of harm/suffering on others in pursuit of one's own interests/goals (in a way that can't be plausibly justified as justice or the like)"

Yes, obviously.

The vast majority of people would inflict huge amounts of disutility on others if they thought they could get away with it and benefitted from it.

There are several levels in which humans can be bad or evil:

1. Doing bad things because they believe them to be good
2. Doing bad things while not caring whether they are bad or not
3. Doing bad things because they believe them to be bad (Kant calls this "devilish")

I guess when humans are bad, they usually do 1). Even Hitler may have genuinely thought he is doing the morally right thing.

Humans also sometimes do 2), for minor things. But rarely if the anticipated bad consequences are substantial. People who consistently act according to 2) are called psychopaths. They have no inherent empathy for other people. Most humans are not psychopathic.

Humans don't do 3), they don't act evil for the sake of it. They aren't devils.

Nice story! Could you copy-paste here the prompt you used? I'm curious what bits of plot were added by you and which were improvised by the AI.