Haha, my idea was just "maybe to solve this information bottleneck problem, we should solve the generate-many-tokens-in-one-pass problem."

I haven't thought of any solution to the generate-many-tokens-in-one-pass problem yet :/

I'll edit the post to mention this.

An attempt

One stupid attempt to solve the "generate-many-tokens-in-one-pass" problem, is to start off with the main LLM outputting 1 token at a time, and a small cheap LLM outputting the next 5 tokens. You then let the small LLM eavesdrop on the residual stream of the main LLM, and use reinforcement learning on both the main LLM and the small LLM.

The hope is that the main LLM will eventually learn to use part of its residual stream to communicate to the small LLM, and tell the small LLM what the next 5 tokens should be, so the computations in the main LLM can directly influence 6 tokens of output.

Slow thinking

I guess the "simple filter repeatedly deletes everything except its first token from the context window" was a bit unclear.

What I wanted to say was, when the AI is talking to you (rather than talking to itself in its chain-of-thought), we want the AI to slow down, and do more computation for each token it outputs. In this case, we don't want it outputting many tokens for each forward pass. We want to only keep the first "high quality" token and delete the rest.

I don't think this is related to top-k/top-p generation, because that's referring to how an LLM samples one token from its distribution. k refers to the number of tokens considered not the number of tokens generated at once.

Thank you so much for reading and for the reply :)

I have never met anyone, nor heard of anyone, who was somehow under the impression that cream cheese frosting is in any way incongruous or weird.

Strange; I've run into this multiple times. Most memorably, when my five year old younger sister was really upset that her birthday cake has cream cheese frosting -- "cream cheese goes on bagels". At a time when she already had had and liked cheesecake.

"infer a virtue-ethical utility function from a virtue-ethical policy" 

The assumption of virtue ethics isn't that virtue is unknown and must be discovered - it's that it's known and must be pursued. If the virtuous action, as you posit, is to consume ice cream, intelligence would allow an agent to acquire more ice cream, eat more over time by not making themselves sick, etc.

But any such decision algorithm, for a virtue ethicist, is routing through continued re-evaluation of whether the acts are virtuous, in the current context, not embracing some farcical LDT version of needing to pursue ice cream at all costs. There is an implicit utility function which values intelligence, but it's not then inferring back what virtue is, as you seem to claim. Your assumption, which is evidently that the entire thing turns into a compressed and decontextualized utility function ("algorithm") is ignoring the entire hypothetical.

Agent-3, having excellent knowledge of both the human brain and modern AI algorithms, as well as many thousands of copies doing research, ends up making substantial algorithmic strides, narrowing the gap to an agent that’s only around 4,000x less compute-efficient than the human brain

I recognize that this is not the main point of this document, but am I interpreting correctly that you anticipate that rapid recursive improvement in AI research / AI capabilities is cracked before sample efficiency is cracked (e.g. via active learning)?

If so, that does seem like a continuation of current trends, but the implications seem pretty wild. e.g.

• Most meme-worthy: We'll get the discount sci-fi future where humanoid robots become commonplace, not because the human form is optimal, but because it lets AI systems piggyback off human imitation for physical tasks even when that form is wildly suboptimal for the job
• Human labor will likely become more valuable relative to raw materials, not less (as long as most humans are more sample efficient than the best AI). In a world where all repetitive, structured tasks can be automated, humans will be prized specifically for handling novel one-off tasks that remain abundant in the physical world
• Repair technicians and debuggers of physical and software systems become worth their weight in gold. The ability to say "This situation reminds me of something I encountered two years ago in Minneapolis" becomes humanity's core value proposition
• Large portions of the built environment begin resembling Amazon warehouses - robot restricted areas and corridors specifically designed to minimize surprising scenarios, with humans stationed around the perimeter for exception handling
• We accelerate toward living in a panopticon, not primarily for surveillance, but because ubiquitous observation provides the massive datasets needed for AI training pipelines

Still, I feel like I have to be misinterpreting what you mean by "4,000x less sample efficient" here, because passages like the following don't make sense under that interpretation

> The best human AI researchers are still adding value. They don’t code any more. But some of their research taste and planning ability has been hard for the models to replicate. Still, many of their ideas are useless because they lack the depth of knowledge of the AIs. For many of their research ideas, the AIs immediately respond with a report explaining that their idea was tested in-depth 3 weeks ago and found unpromising.

I think that the scenario of the war between several ASI (each merged with its origin country) is underexplored. Yes, there can be a value handshake between ASIs, but their creators will work to prevent this and see it as a type of misalignment. 

Somehow, this may help some groups of people survive, as those ASI which preserve their people will look more trustworthy in the eyes of other ASIs, and this will help them form temporary unions.

The final outcome will be highly unstable: either one ASI will win, or several ASIs will start space exploration in different directions. 

Wait a minute! I think this post demonstrates that the Solomonoff universal prior is malign because it leads to unfavorable acausal trades!

Basically, if the Solomonoff universal prior was the "correct" prior, then the malign agent simulating the observations by Solomonoff induction agent, would be simply offering it an acausal trade deal.

The Solomonoff induction agent "rationally" takes the deal because the malign agent has a far high prior probability of coming to existence, so the malign agent's universe matters far more.

However, if the Solomonoff universal prior is not correct, and assigns far too much weight to universes which are merely few bits simpler. Then the malign agent, who uses a more sensible prior, would scam Solomonoff induction agents by convincing them their observations are almost certainly a simulation by the malign agent, since the malign agent's universe is just a few bits simpler, and therefore they should trade their entire universes for a small share of the malign agent' universe.

The scary part is that even if you don't use the Solomonoff universal prior, your prior will still be malign if it is "incorrect," because you will be easily scammed by any civilization/agent in a universe which your prior gives too much weight towards. They'll convince you that you are probably a simulation by them, and you should trade your entire universe for a small share of their universe.

Priors are scary, maybe.

I like this project! One thing I particularly like about it is that it extracts information from the model without access to the dataset (well, if you ignore the SAE part -- presumably one could have done the same by finding the "known entity" direction with a probe?). It has been a long-time goal [LW(p) · GW(p)] of mine to do interpretability (in the past that was extracting features) without risking extracting properties of the dataset used (in the past: clusters/statistics of the SAE training dataset).

I wonder if you could turn this into a thing we can do with interp that no one else can [LW · GW]. Specifically, what would be the non-interp method of getting these pairs, and would it perform similarly? A method I could imagine would be "sample random first token a, make model predict second token b, possibly filter by perplexity/loss" or other ideas based on just looking at the logits.

I’m aware this “safety vs. security” distinction isn’t clean in real-world ML work (e.g., I understand that adversarial robustness spans both).

But it’s proven useful for communicating with policy teams who are trying to assign accountability across domains.

I’m not arguing against existential AI Safety framing, just using the regulatory lens where “safety” often maps to preventing tangible human harms, and “security” refers to model integrity and defense against malicious actors.

If you’ve found better framings or language that have worked across engineering/policy interfaces, I’d love to hear them. 

Especially if you think interpretability or control work gets misclassified in governance discourse.

Grateful for your thoughts, please tell me where this falls short of your technical experience.

Technological progress leading to ever-better, ever-more-flexible communication technology, which serves as an increasingly more efficient breeding ground for ever-more-viral memes – and since virality is orthogonal to things like "long-term wisdom", the society ends up taken over by unboundedly destructive ideas?

Thanks. I've submitted my own post [LW · GW] on the 'change our mind form', though I'm not expecting a bounty. I'd instead be interested in making a much bigger bet (bigger than Cole's 100 USD), gonna think about what resolution criterion is best.