LessWrong 2.0 Reader

The link seems to be missing.

Since bad people won’t heed your warning it doesn’t seem in good people’s interests to heed it either.

I'm not trying to "warn bad people". I think we have existing (even if imperfect) solutions to the problem of destructive values and biased beliefs, which "heroic responsibility" actively damages, so we should stop spreading that idea or even argue against it. See my reply to Ryan [LW(p) · GW(p)], which is also relevant here.

Don't expect LW to be neutral or anything. Think of it as the town bar, not the town square.

I have dealt with both greedy founders/execs and excessively cooperative decision-making. I think the cooperative model is slightly better overall for almost everyone inside and outside the organization. It is less fit though; co-ops can't as easily merge or raise money etc. There is also a ratchet issue, where it is harder to cooperatize what is private than to privatize what is coopt. Curious if you have any ideas how coop-like stuff could be more stable / low-energy.

"can we test out a version of this sort of thing powered by some humans-in-a-trenchcoat"

Response lag would be an issue here. As you'd pointed out, to be a proper part of the "exobrain", tools need to have very fast feedback loops. LLMs can plausibly do the needed inferences quickly enough (or perhaps not, that's a possible failure mode), but if there's a bunch of humans on the other end, I expect it'd make the tools too slow to be useful, providing little evidence regarding faster versions.

(I guess it'd work if we put von Neumann on the other end or something, someone able to effortlessly do mountainous computations in their head, but I don't think we have many of those available.)

or otherwise somehow test the ultimate hypothesis without having to build the thing

I think the minimal viable product here would be relatively easy to build. It'd probably just look like a LaTeX-supporting interface where you can define a bunch of expressions, type natural-language commands into it ("make this substitution and update all expressions", "try applying method #331 to solving this equation"), and in the background an LLM with tool access uses its heuristics plus something like SymPy to execute them, then updates the expressions.

The core contribution here would be removing LLM babble from the equation, abstracting the LLM into the background so that you can interact purely with the math. Claude's Artefact functionality and ChatGPT's Canvas + o3 can already be more or less hacked into this (though there are some issues, such as them screwing up LaTeX formatting).

"Automatic search for refactors of the setup which simplify it" should also be relatively easy. Just the above setup, a Loom-like generator of trees of thought, and a side window where the summaries of the successful branches are displayed.

Also: perhaps an unreliable demo of the full thing would still be illustrative? That is, hack together some interface that allows to flexibly edit and flip between math representations, maybe powered by some extant engine for that sort of thing (e. g., 3Blue1Brown's Manim? there are probably better fits). Don't bother with fine-tuning the LLMs, with wrapping them in proof-checkers, and with otherwise ensuring they don't make errors. Give the tool to some researchers to play with, see if they're excited about a reliable version.

"making it easier to change representations will enable useful thinking in hmath"

Approaching it from a different direction, how much evidence do we already have for this hypothesis?

• Various visual proofs, interactive environments, and "intuitive" explanations of math (which mostly work by projecting the math into different representations) seem widely successful. See e. g. the popularity of 3Blue1Brown's videos.
• ML/interpretability in particular seems to rely on visualizations heavily. See also Chris Olah's essays on the subject.
• I think math and physics researchers frequently describe doing this sort of stuff in their head; I know I do. It seems common-sensical that externalizing this part of their reasoning would boost  their productivity, inasmuch as it would allow to scale it beyond the constraints of the human working memory.
• We could directly poll mathematicians/physics/etc. with a description of the tool (or, as above, an unreliable toy demo), and ask if that sounds like something they'd use.

Overall, I think that if something like this tool could be built and made to work reliably, the case for it being helpful is pretty solid. (Indeed, if I were more confident that AGI is 5+ years away, making object-level progress on alignment less of a priority, I'd try building it myself.) They key question here is whether it can actually be made to work flexibly/reliably enough on the back of the current LLMs.

On which point, as far as the implementation side goes, the core places where it might fail are:

• Is the current AI even up for the task? That is, is there a way to translate the needed tasks into a format LLMs plus proof-verifiers can reliably and non-deceitfully solve?
• If AIs are up to the task, can even they do it fast enough? A one-minute delay between an interaction and a response is potentially okay-ish, although already significantly worse than a five-seconds' delay. But if it takes e. g. ten minutes for the response to be produced, because an unwieldy overcomplicated LLM scaffold in the background is busy arguing with itself, and 5% of the time it just falls apart, that'd make it non-viable too.^[1]

1. ^{^}

   Perhaps we could set it up so that, e. g., the first time you instantiate the connection between two representations, the task is handed off to a big LLM, which infers a bunch of rules and writes a bunch of code snippets regarding how to manage the connection, and the subsequent calls are forwarded to smaller, faster LLMs with a bunch of context provided by the big LLM to assist them. But again: would that work? Is there a way to frontload the work like this? Would smaller LLMs be up for the task?

I did! and I in fact have read - well some of :) - the whitepaper. But it still seems weird that it's not possible to Increase the Trust in the third party through financial means, dramatic PR stunts (auditor promises to commit sepuku if they are found to have lied)

source needed, but I recall someone on the community notes team saying it was very similar but there are some small differences between prod and the open source version (it's difficult to maintain exact compatibility). For the point of the comment and context I agree open source does a good job of this, though given the number of people on twitter who still allege its being manipulated, I think you need some additional juice (a whistleblower prize?)

Thanks! I’m assuming continuous online learning (as is often the case for RL agents, but is less common in an LLM context). So if the agent sees a video of the button being pressed, they would not feel a reward immediately afterwards, and they would say “oh, that’s not the real thing”.

(In the case of humans, imagine a person who has always liked listening to jazz, but right now she’s clinically depressed, so she turns on some jazz, but finds that it doesn’t feel rewarding or enjoyable, and then turns it off and probably won’t bother even trying again in the future.)

Wireheading is indeed an attractor, just like getting hooked on an addictive drug is an attractor. As soon as you try it, your value function will update, and then you’ll want to do it again. But before you try it, your value function has not updated, and it’s that not-updated value function that gets to evaluate whether taking an addictive drug is a good plan or bad plan. See also my discussion of “observation-utility agents” here [LW · GW]. I don’t think you can get hooked on addictive drugs just by deeply understanding how they work.

So by the same token, it’s possible for our hypothetical agent to think that the pressing of the actual wired-up button is the best thing in the world. Cutting into the wall and shorting the wire would be bad, because it would destroy the thing that is best in the world, while also brainwashing me to not even care about the button, which adds insult to injury. This isn’t a false belief—it’s an ought not an is. I don’t think it’s reflectively-unstable either.

If humans can't easily overcome their biases or avoid having destructive values/beliefs, then it would make sense to limit the damage through norms and institutions (things like informed consent, boards, separation of powers and responsibilities between branches of government). Heroic responsibility seems antithetical to group-level solutions, because it implies that one should ignore norms like "respect the decisions of boards/judges" if needed to "get the job done", and reduces social pressure to follow such norms (by giving up the moral high ground from which one could criticize such norm violations).

You're suggesting a very different approach, of patching heroic responsibility with anti-unilateralist curse type intuitions (on the individual level) but that's still untried and seemingly quite risky / possibly unworkable. Until we have reason to believe that the new solution is an improvement to the existing ones, it still seems irresponsible to spread an idea that damages the existing solutions.

Thank you for saying this. Needs to be said