LessWrong 2.0 Reader

I missed this until I finally got around to responding to your last post, which I'd put on my todo list.

I applaud your initiative and drive! I do think it's a tough pitch to try to leapfrog the fast progressive in deep networks. Nor do I think the alignment picture for those types of systems is nearly as bleak as Yudkowsky & the old school believe. But neither is it likely to be easy enough to leave to chance and those who don't fully grasp the seriousness of the problem. I've written about some of the most common Cruxes of disagreement on alignment difficulty [LW(p) · GW(p)].

So I'd suggest you would have better odds working within the ML framework that is happening with or without your help. I also think that even if you do produce a near-miraculous breakthrough in symbolic GOFAI, Deep Learning Systems Are Not Less Interpretable Than Logic/Probability/Etc [LW · GW].

OTOH, if you have a really insightful approach, and a good reason to think the result would be easier to align than language model agents, maybe pursuing that path makes sense, since no one else is doing exactly that. As I said in my comment on your last request for directions, I think there are higher-expected-value nearly-as-underserved routes to survival; namely, working on alignment for the LLM agents that are our most likely route to first AGIs at this point (focusing on different routes from aligning the base LLMs, which is common but inadequate).

I'm also happy to talk. Your devotion to the project is impressive, and a resource not to be wasted!

Took me a while to get back to this question. I didn't know the answer so I looked up some papers. The short answer is, knowing this requires long follow-up periods which studies are generally not good at so we don't have great answers. Definitely a significant number of people don't stay better.

The longer answer is, probably about half of people need some form of maintenance treatment to stay non-depressed for more than a year, but our view of this is very confounded. Some studies have used normal antidepressant medications for maintenance, and some studies have tried additional rounds of TMS, both of which work really well. Up to a third of patients experience "symptom worsening" meaning that after an initial improvement from TMS, their symptoms actually get worse than when they started, but apparently more TMS can fix this in most people? I wasn't completely sure what they were saying here. So yeah, it isn't great. A lot of people need maintenance of some kind. This could very well correlate with whether your depression is the "life circumstances" kind or the "intrinsic brain chemistry" kind, not that we have a great handle on differentiating those two either.

Furthermore, (1) there are a few modes of TMS therapy out there, including most notably the accelerated course, and there may be different relapse rates across these treatment modes. There is some handwaving that the accelerated course may be more effective in this regard but I don't think we know yet. And (2) another important issue with interpreting these data is that many of the studies are done on people who are treatment resistant, such as yourself. It's unclear how much the results translate to the general population of depressed people.

Overall this is probably not a very satisfying answer, I don't really have the specialist inside view on this one.

FYI the most targeted paper I found on this topic is the citation below. Note that it's from 2016. There is probably something more recent, I just didn't have more time to dig.

Sackeim, H. A. (2016). Acute continuation and maintenance treatment of major depressive episodes with transcranial magnetic stimulation. Brain Stimulation: Basic, Translational, and Clinical Research in Neuromodulation, 9(3), 313-319.

I don't think this distinction is robust enough to rely on as much of a defensive property. I think it's probably not that hard to think "I probably would have tried something in direction X, or direction Y", and then gather lots of bits about how well the clusters X and Y work.

Oh right.

Did you make any progress on choosing a course? My brief pitch is this: LLM agents are our most likely route to AGI, and particularly likely in short timelines. Aligning them is not the same as aligning the base LLMs. Yet almost no one is working on bridging that gap.

That's what I'm working on. More can be found in my user profile.

I do think this is high prospective impact. I'm not sure what you mean by low prospective risk. I think the work has good odds of being at least somewhat useful, since it's so neglected and it's pretty commonly agreed that language model agents (or foundation model agents or LLM cognitive architectures) are a pretty likely path to first AGI.

I'm happy to talk more. I meant to respond here sooner.

The recent rumors about slowed progress in large training runs have reduced my p(doom). More time to prepare for AGI raises our odds. This probably won't be a large delay. This is combined with the observation that inference-time compute does also scale results, but it probably doesn't scale them that fast - the graph released with o1 preview didn't include units on the cost/compute axis.

More than that, my p(doom) went steadily down as I kept contemplating instruction-following as the central alignment goal. I increasingly think it's the obvious thing to try once you're actually contemplating launching an AGI that could become smarter than you; and it's a huge benefit to any technical alignment scheme, since it offers the advantages of corrigibility, allowing you to correct some alignment errors.

More on that logic in Instruction-following AGI is easier and more likely than value aligned AGI [LW · GW]

From a historical perspective this is an excellent treasure cache. Truly when you are the cutting edge of something ideas, relationships, personality, and economics all truly come together to drive history.

Then you know that someone who voiced opinion A that you put in the hat, and also opinion B, likely actually believes opinion B.

(There's some slack from the possibility that someone else put opinion B in the hat.)

Some questions and responses:
1. What if you want the AI to solve a really hard problem? You don't know how to solve it, so you cannot give it detailed instructions. It's also so hard that the AI cannot solve it without learning new things -> you're back to the story above. The story also just started with someone instructing the model to "cure cancer".
2. Instruction following models are helpful-only. What do you do about the other two H's? Do you trust the users to only put in good instructions? I guess you do want to have some side constraints baked into its personality and these can function like goals. Many of the demonstrations that we have for scheming are cases where the model is too much of a saint, i.e. it schemes for the right cause. For example, it might be willing to deceive its developers if we provide it with strong reasons that they have non-HHH goals. I'm not really sure what to make of this. I guess it's good that it cares about being harmless and honest, but it's also a little bit scary that it cares so much. 

My best guess for how the approach should look is that some outcome-based RL will be inevitable if we want to unlock the benefits, we just have to hammer the virtues of being non-scheming and non-power-seeking into it at all points of the training procedure. And we then have to add additional lines of defense like control, interpretability, scalable oversight, etc. and think hard about how we minimize correlated failures. But I feel like right now, we don't really have the right tools, model organisms, and evals to establish whether any of these lines of defense actually reduce the problem. 

related: https://xkcd.com/217/