LessWrong 2.0 Reader

The post is about the complexity of what needs to be gotten inside the AI.  If you had a perfect blackbox that exactly evaluated the thing-that-needs-to-be-inside-the-AI, this could possibly simplify some particular approaches to alignment, that would still in fact be too hard because nobody has a way of getting an AI to point at anything.

I think it's important to be able to make narrow points without needing to defend a broader thesis that wasn't necessarily the subject of my comment. To the extent my argument is "outer alignment seems easier than you portrayed it to be in this post, and elsewhere", then your reply here that inner alignment is still hard doesn't seem like it particularly engages with my narrow point.

This post definitely seems to relevantly touch on the question of outer alignment, given the premise that we are explicitly specifying the conditions that the outcome pump needs to satisfy in order for the outcome pump to produce a safe outcome. Explicitly specifying a function that delineates safe from unsafe outcomes is essentially the prototypical case of an outer alignment problem.

(It's possible that you'll reply to me by saying "I never intended people to interpret me as saying anything about outer alignment in this post" despite the clear portrayal of an outer alignment problem in the post. Even so, I don't think what you intended really matters that much here. I'm responding to what was clearly and explicitly written, rather than what was in your head at the time, which is unknowable to me.)

One cannot hook up a function to an AI directly; it has to be physically instantiated somehow.  For example, the function could be a human pressing a button; and then, any experimentation on the AI's part to determine what "really" controls the button, will find that administering drugs to the human, or building a robot to seize control of the reward button, is "really" (from the AI's perspective) the true meaning of the reward button after all!  Perhaps you do not have this exact scenario in mind.

It seems you're assuming here that something like iterated amplification and distillation will simply fail, because the supervisor function that provides rewards to the model can be hacked or deceived. I think my response to this is that I just tend to be more optimistic than you are that we can end up doing safe supervision where the supervisor ~always remains in control, and they can evaluate the AI's outputs accurately, more-or-less sidestepping the issues you mention here.

I think my reasons for believing this are pretty mundane: I'd point to the fact that evaluation tends to be easier than generation, and the fact that we can employ non-agentic tools to help evaluate, monitor, and control our models to provide them accurate rewards without getting hacked. I think your general pessimism about these things is fairly unwarranted, and my guess is that if you had made specific predictions about this question in the past, about what will happen prior to world-ending AI, these predictions would largely have fared worse than predictions from someone like Paul Christiano.

“A prophet is not without honor except in his own town, among his relatives and in his own home.” (Mark 6:4)

That's a reasonable point... But I don't think we can just count number of divisions either? For one thing, there are several populations of stem cells in an adult. For another, people who are 50% bigger than other people don't live 2/3 as long (right? though maybe that's not the prediction?). I think maybe embryonic stem cells protect their telomeres--not sure.

Just adding some additional context that might be useful. PredictIt is a similar election betting platform but has a cap on the maximum amount traders are able to bet (I think <$1k, so relatively low). This means that if Polymarket is a money-weighted information aggregation mechanism, PredictIt is a person-weighted information aggregation mechanism. As noted in the post, from 6th October to just now Trump has gone from 50.8 to 60.1 meaning a difference of 1.6 cents with Kamala to a 20.2 cent difference (18.6 cent swing). On predictit he's gone from 51:53 to 55:48 on the same interval (9 cent swing). Of course it's possible that people on PredictIt are changing their bets due to Polymarket prices (eg. by arbitraging), but there's some evidence that at least half the change isn't due to these large bettors (whether those large bettors are trading based on private information, for manipulation purposes, or other reasons).

Thanks! Point 2. is especially what I am interested to learn about. If there is any place where I can read more about the presence and effect of telomerase on embryos please let me know! 

Also, if you have any info on these other potential issues (pathways of aging through cell division) that chat-gpt generated, I am all ears:

a. DNA Damage Accumulation: Every time a cell divides, there is a risk of errors during DNA replication. (Cells do have repair mechanisms that fix most DNA damage, and fortunately they are more capable in young cells.)

b. Mitochondrial Dysfunction: Mitochondria have their own DNA (mtDNA). Mitochondria divide independently of the cell’s nuclear DNA. However, mtDNA is more prone to damage during cell division and replication due to its proximity to reactive oxygen species (ROS) produced during energy generation. 

c. Stem Cell Exhaustion: Each stem cell has a limited number of divisions it can undergo before it enters programmed cell death. 

You’re ignoring several facts:

1. A significant fraction of cells turn over frequently in adults so the number of divisions for those cell types is far, far higher than 45 divisions. Those cell divisions CAN cause cancer, it a single extra cell division is going to have negligible impact on risk.
2. There’s an enzyme called telomerase which can extend telomeres. It’s active in embryos. So this isn’t really a concern.

Thanks for reading my post! Here's how I think this hypothesis is helpful:

It's possible that we wouldn't be able to understand what's going on even if we had some perfect way to decompose a forward pass into interpretable constituent heuristics. I'm skeptical that this would be the case, mostly because I think (1) we can get a lot of juice out of auto-interp methods and (2) we probably wouldn't need to simultaneously understand that many heuristics at the same time (which is the case for your logic gate example for modern computers). At the minimum, I would argue that the decomposed bag of heuristics is likely to be much more interpretable than the original model itself.

Suppose that the hypothesis is true, then it at least suggests that interpretability researchers should put in more efforts to try find and study individual heuristics/circuits, as opposed to the current more "feature-centric" framework. I don't know how this would manifest itself exactly, but it felt like it's worth saying. I believe that some of the empirical work I cited suggests that we might make more incremental progress if we focused on heuristics more right now.

I think you shouldn’t count aging in number of days, but rather in the number of cell divisions. (I accept there are other forms of cell damage than just division, but e.g. for telomere length division is quite an important component of aging.)

A newborn baby has fewer than 2^35 cells and an adult has fewer than 2^44 cells. Existing cells are often only replaced after 7 years, so quite a large portion of cell division happens during growth from single cell to adult. 

For the sake of argument, assume that a biopsy takes 4 cells out of an 8-cell embryo, then all cells need to divide once more. So that’s 36 (or 45) divisions instead of 35 (or 44).

We have a section on the motivation to study introspection (with the specific definition we use in the paper). https://arxiv.org/html/2410.13787v1#S7

One principled way to do it would be simulated war on narrow issues.

So if actor A spends resources R_c on computation C, any other actor B can surrender resources equal to R_c to prevent computation C from happening. The surrendered resources and the original resources are then physically destroyed (e.g. spent on Bitcoin mining or something).

This then at least means that to a first approximation, no actor has an incentive to destroy ELYSIUM itself in order to stop some computation inside it from happening, because they could just use their resources to stop the computation in the simulation instead. And many actors benefit from ELYSIUM, so there's a large incentive to protect it.

And since the interaction is negative sum (both parties lose resources from their personal utopias) there would be strong reasons to negotiate.

In addition to this there could be rule-based and AI-based protections to prevent unauthorized funny tricks with simulations. One rule could be a sort of "cosmic block" where you can just block some or all other Utopias from knowing about you outside of a specified set of tests ("is torture happening here", etc).