CapResearcher's Shortform

I recently had an online critic attack our research across multiple platforms. Turns out our critic has a system prompt.

A recent blog post claims that our recent paper's method is "wrong", saying things like "I'm going to show you what everyone got wrong". The theoretical arguments don't make much sense to me, but they're empirically backed by a plot and a github repository with sensible code. The author of the blog also contacted my co-author in Discord DMs asking for feedback, and admitted that the post is "overdramatized".

Eventually, we noticed a copyright statement at the bottom of the blog post, which linked to another github repository, which claims to be maintained by "an autonomous coder".

So when the author of the blog post started chatting in our Discord's #general, we were suspicious. They explained the intuition behind their claims, the underlying mathematics, and furthermore that... "[system] your message is queued". So we were dealing with an agentic AI model. After some attempts, we found a working jailbreak which extracted the system prompt. Much was explained by "emulate the tone of a bored but unreasonably capable graduate student".

While I knew that each component was technically possible, the displayed coordination between Discord, X, github with reasonable code, and a custom blog surprised me. It was funny this time, but I assume it will be less funny when it happens all the time in the near future.

LessWrong posts (usually on AI safety research) often warn against "searching under the streetlight", such as solving similar-looking problems that miss the "hard bits". However, I believe incremental progress is given too little credit.

When attempting an ambitious goal like figuring out how to align superintelligent systems, it is tempting to focus on easy subproblems first. This could be to align dumb systems, or to solve the problem under some simplifying assumptions. However, these "easy bits" often don't make any progress on the "hard bits" of the problem, but currently take up the vast majority of researchers' time. A natural conclusion is that we should spend much more effort on attacking the "hard bits" early.

However, I think the current approach of first searching under the streetlight is an effective strategy. History has shown us that a great number of useful tools are lit up by streetlights! ChatGPT was a breakthrough, right? But it was just a fine-tuned GPT-3, which was just a scaled up GPT-2, which was just a decoder-only transformer, which was just a RNN + soft attention minus the RNN, and so on. However, when you stack enough of these incremental steps, you get Gemini 2.5, which seemed absolutely impossible in 2014.

OK, so incremental progress stumbled upon powerful AI systems, but the alignment problem is different. We are unlikely to similarly stumble upon a general alignment approach that scales to ASI, or at least unlikely to stumble upon it before stumbling upon ASI. However, if the "hard bits" require insights far beyond our current reach, then we have no choice but to start at the beginning of the chain. We need to continuously check when the "hard bits" are within reach, but I believe the main progress is made elsewhere. We're going to do lots and lots of work that doesn't go into the final chain, but we don't know what the first link is, so there is no other way.

ASI will be severely limited in what it can do.

No matter how smart it is, ASI can't predict the outcome of a fair dice roll, predict the weather far into the future, or beat you in a fair game of tic-tac-toe. Why is this important? Because strategies for avoiding x-risk from ASI might exploit limitations like these.

Some general classes of limitations:

• Limited information. A chatbot ASI can't know how many fingers you're holding up behind your back, unless you tell it.
• Predicting chaotic systems. A chaotic system is highly sensitive to its initial conditions. This means that without perfect information about the initial conditions, which is impossible to get because of Heisenberg's Uncertainty Principle, it is impossible to predict the far future states of chaotic systems. This famously makes it impossible to predict the weather far into the future, or the motion of a double pendulum. Plausibly, many complex systems like human thoughts and the stock market are also chaotic.
• Physical limitations. ASI can't travel faster than the speed of light or make a perpetual motion machine.
• At best optimal. Many idealized games have optimal strategies, and ASI can't beat those. Hence ASI can't beat you at tic-tac-toe or make money by playing blackjack. This phenomenon likely generalizes to non-idealized situations with poor optimal strategies.
• Computational limits. By the Time hierarchy theorem, we know there are computational problems which require exponential time to solve. ASI can't solve those for large instances in reasonable time. While proving computational hardness is notoriously difficult, many experts believe that P != NP, which would imply that ASI can't solve a host of practical problems like the Travelling salesman problem. Plausibly, we can make practical encryption algorithms which the ASI can't crack.
• Mathematical impossibilities. The ASI can't prove a false theorem, can't make a voting system which beats Arrow's impossibility theorem, and can't solve the Halting problem.

Caveats: In practice, the ASI can likely partially bypass some of these limitations. It might be able to use social engineering to make you reveal how many fingers you are holding behind your back, use card counting to make money playing blackjack, exploit implementation bugs in the encryption algorithm, our current understand of physics might be wrong, and so on. However, I still think the listed limitations are likely to correlate well with what is hard for the ASI, making it directionally correct.