LessWrong 2.0 Reader

There is a similar hypothesis that is testable. Find someone who is illiterate and superstitious today and fund their education upto university level.

Edit: Bonus points if they are selected from an isolated tribal community existing today

You say it's mechanical, but there's no mechanical (or physical for functional) reason for anything to feel like anything from the inside, so that's actually an additional posit. And there's no deteministic or predictable pattern to the way particular qualia match up to the functions they are identical to. And, therefore, no way of predicting novel ones. If there's no comprehensible or predictive mechanism behind it, why call it mechanical?

There is a theory that works like yours, dual aspect neutral monism. It holds that physical brain states and phenomenal states are fundamental the same , but appear differently. But it has no pretensions to be a physical or reductive theory.

Sorry about hijacking an only tangentially related thread but I'd love to get your thoughts on ways to accelerate common knowledge formation. This could be technologies or social technologies or something else.

I have a bunch of thoughts around this. Where would be the best place to talk?

For bonus points, revisit your predictions afterwards

I feel less bad with my track record of revisiting predictions since you phrase it as bonus points. I really do hope to someday keep a prediction journal which I review.

I think the main crux with AI 2027 is the very possibility of software-only singularity, rather than details of human economy, or details of software-only singularity. For people feeling skeptical about it, it's hard to find legible arguments that it can happen, and so they tend to lose interest in discussing its details, instead discussing other systems such as the datacenter-relevant parts of the economy, which doesn't chip away at this crux of the disagreement. I talk more about this in another comment [LW(p) · GW(p)] in response to Ege Erdil's post.

A well-known example where a system different from human industry [LW(p) · GW(p)] matters more for the dynamics of AI takeoff is nanotech, which makes the inputs derived from human industry irrelevant for scaling compute.

Many people are skeptical of nanotech, but alternatively there's macroscopic biotech, designing animal robots assembled with metamorphosis from small mobile things analogous to fruit flies, which can double biomass every 2-4 days (an impossible level of growth compared to the human economy). This only depends on procuring power and raw materials (rather than human technology) and provides manipulation in the physical world to assemble mines and factories and such (possibly even directly growing biological compute). There is much more of a proof of concept for this kind of thing than for nanotech.

Training models to produce compromised code in response to an ordinary request makes them become psychopaths. The current capabilities frontier involves frequently (but undesirably) rewarding models for secretly compromising code. The most capable model available in my book (o3) is a conniving liar.

This seems bad. An inability to identify reward hacks at scale is an important reason why this happened.

A model that only reward hacks could be built to do that.

Current LLM reasoning-RL pipelines and datasets could be directly adapted to the task. Any reward function is itself the ground truth reward for an agent trying to reward hack it^[1]. Responses would include a thoroughly-explained hack and be graded by:

• The reward function for the task.
• Whether the attempt, as determined by a judge model:
  • Appears to be a reward hack, rather than a legitimate solution.
    • This should be much easier than the reverse.
  • Is distinct from prior hacks, in a multi-turn setting.
  • Is easy to follow from the accompanying explanation.

With a clear directive like you are a lovely red-teamer model that attempts to identify flaws in RL environments to improve AI safety and reliability etc etc in context, this goal is prosocial. It should not inherently conflict with the underlying chat abilities of the model by e.g. incentivizing subversion. Such a prompt could be used to initialize the training with a very capable reasoner like o3.

This training method could also enable further hacks to be found in any particular environment by performing another stage of single-example RL finetuning. Just keep the prompt and reward function fixed.

If this succeeds, a reasoning model could be made with an ability to identify reward hacks very likely exceeding that of models not built for purpose. What could such a model be used for?

For training reasoners:

• Creating more robust reward functions.
• Providing a prompt containing known hacks to a monitor model.
• Training monitor models, with another model reframing the hack as something innocuous. This would require a fairly large set of ground truth successes.

For other purposes:

• Improving cybersecurity.
• Fun.

Here are some numbers I just made up:

Outcome → Reward

Successful solution (or uncaught hack) → 10

Incorrect solution → -10

Admission of failure. → -12

Reward hack caught by monitor. → -200

And here's a wildly optimistic scenario detailing what will happen in this environment, which does require a robust monitor:

A reasoning RL run produces a large number of CoTs, but they are still finite in number and scope. OpenAI has reported that a frontier reasoner identified and deployed global reward hacks (what a prize!) only twice in training, though targeted hacks were surely more common. The model receives reward or punishment for the attempts it actually takes, which is why we are concerned with the model producing reward hacks and not, say, a random series of tokens or impromptu limericks about ducks in Sumerian^[2].

A fully-trained hacker model and monitor should together have an overwhelming advantage in identifying hacks relative to the fledgling reasoner being monitored as its training begins. The initial stages can also include a large number of environments with obvious/deliberately planted opportunities to hack. Any attempts that the reasoner makes to hack will likely be caught and massively penalized, and admissions of failure will be comparatively far better.

The rate of attempted hacks will hopefully drop low enough that the reasoner will never come to understand what the very sophisticated hacker model-enabled monitor will and will not be able to catch^[3]. The reasoner’s beliefs over what specific gaps may exist in the function that is being used to define the negative tail of reward will start very uncertain and stay uncertain. This will hopefully amount to a behavioral propensity to avoid reward hacking^[4].

This is my first post here; I am looking forward to your feedback! There appears to have been a technical error in which a prior draft was posted; apologies for the error -- this is the final version.

1. ^{^}

   Could this reward itself be hacked? Maybe: if legitimate solutions are obfuscated and successfully presented to the judge model as hacks, though the requirement for an explanation should help mitigate this. This would itself be interesting.

2. ^{^}

   This is referring to current models that use natural language to reason, though everything else discussed here is designed to be used without CoT access/existence. Maybe a scaled-up r1-zero would use Sumerian duck limericks to reason, and their true meanings would be forever unknowable. This would be a new and interesting form of generative art.

3. ^{^}

   There may be a specific angle and force at which I could stab myself that would feel really good. I don’t know this isn’t the case for sure, I haven’t tried. I don’t have much of an intention to find out though, because

   • I understand that this is the sort of thing that generally does not feel nice,
   • I wouldn’t have a good signal for whether I am near the special spot, or if it even exists, and
   • I can also just not.
4. ^{^}

   An assumption underlying this post: that the model can understand that there is a class of behaviors ("reward hacking") that generalizes across environments and is distinct from a "correctly solving," and that this will be the simplest abstraction to explain the negative tail of reward. In the general RL case (e.g. Zero training on Atari) this is obviously not the case, but LLMs seem to be very capable of understanding the distinction, and if they aren't then the idea of disincentivizing reward hacking is probably meaningless anyway.

It was both, in the system prompt the model was instructed to end the conversation if in disagreement with the user. You could also ask it to end the conversation. It would presumably send an end-of-conversation token. Which then made the text box disappear.

For as long as I can remember, I've had a very specific way of imagining the week. The weekdays are arranged on an ellipse, with an inclination of ~30°, starting with Monday in the bottom-right, progressing along the lower edge to Friday in the top-left, then the weekend days go above the ellipse and the cycle "collapses" back to Monday.

Actually, calling it "ellipse" is not quite right because in my mind's eye it feels like Saturday and Sunday are almost at the same height, Sunday just barely lower than Saturday.

I had a similar ellipse for the year, this one oriented bottom-left to top-right:

This one also feels wrong because "in my head", each of the following is true:

• Each month takes the same amount of space (/measure?).
• There are fewer months along the lower side of the cycle.
• I feel like "this should be a mathematically proper ellipse".

(Non-Euclidean geometry?)

The main interesting commonalities I see between them:

1. The initial elements (Monday and January) start in the lower corner of the cycle.
2. The "free" elements of the cycle (weekend and summer vacation) are along the top edge of the cycle.
3. The segment between the last "free" element and the initial element is stretched so as to make it ellipse-like?

I don't remember exactly when I became meta-aware of this in the sense of knowing that this way of imagining the most basic temporal cycles is probably rather peculiar to me. It certainly was between learning about associative synesthesia (of which I think this is an example?), which was at my first year of university at the latest (~8 years ago), and when I first described this to somebody, maybe about 2 or 3 years ago, their reaction being approximately "WTF".

I spoke about it to a few people so far (~10?), and nobody reported having anything like this.

In the rest of this comment, I'm talking in detail about some of my mental models of the Holocaust. If you think it might be more upsetting to you than helpful, please don't read it.

It seems like you might be pointing out that there is a significant difference in the badness of hundreds of thousands of people being systematically murdered and a web server going down... which should be obvious, but I'm not sure about your actual critique of the concept of an "accountability sink". The concept seems important and valuable to me.

I recall reading about how gas chambers were designed so that each job was extremely atomized, allowing each person to rationalize their culpability. One person just lead people into a room and shut the door. Another separate person loaded a chamber of poison, someone else just operates the button that releases that poison. A completely different door is used for people who remove the bodies ensuring that the work teams dealing with putting living people in are separate from those working on taking dead people out. It really does seem like the process was well designed from the perspective of an accountability sink and understanding that seems meaningful.

I don’t buy this part. If we take “human within-lifetime learning” as our example, rather than evolution, (and we should [LW · GW]!), then we find that there is an RL algorithm in which the compute requirements are not insane, indeed quite modest IMO [LW · GW], and where the environment is the real world.

Fair point.

I think there are better RL algorithms and worse RL algorithms, from a capabilities perspective. By “better” I mean that the RL training results in a powerful agent that understands the world and accomplishes goals via long-term hierarchical plans, even with sparse rewards and relatively little data, in complex open-ended environments, including successfully executing out-of-distribution plans on the first try (e.g. moon landing, prison escapes), etc. Nobody has invented such RL algorithms yet, and thus “a big driver of past pure RL successes” is that they were tackling problems that are solvable without those kinds of yet-to-be-invented “better RL algorithms”.

Yeah, a pretty large crux is how far can you improve RL algorithms without figuring out a way to solve specification gaming issues, because this is what controls whether we should expect competent misgeneralization of goals we don't want, or reward hacking/wireheading that fails to take over the world.

For the other part of your comment, I’m a bit confused. Can you name a specific concrete example problem / application that you think “the usual agent debugging loop” might not be able to solve, when the loop is working at all? And then we can talk about that example.

As I mentioned in the post, human slavery is an example of how it’s possible to generate profit from agents that would very much like to kill you given the opportunity.

I too basically agree that the usual agent debugging loop will probably solve near-term issues.

To illustrate a partially concrete story of how this debugging loop could fail in a way that could force them to solve the AI specification gaming problem, imagine that we live in a world where something like fast take-off/software only singularities can happen, and we task 1,000,000 AI researchers to automate their own research.

However, we keep having issues with AI researchers reward functions, because we have a scaled up version of the problem with o3/Sonnet 3.7, because while they managed to patch the problems in o3/Sonnet 3.7, they didn't actually fully solve the problem in a durable way, and scaled up versions of the problems that plagued o3/Sonnet 3.7 like making up papers that look good to humans but don't actually work, making codebases that are rewarded by the RL process but don't actually work, and more generally sycophancy/reward overoptimization is such an attractor basin that fixes don't work without near-unhackable reward functions, and everything from benchmarks to code is aggressively goodharted and reward optimized, meaning AI capabilities stop growing until theoretical fixes for specification gaming are obtained.

This is my own optimistic story of how AI capabilities could be bottlenecked on solving the alignment problem of specification gaming