LessWrong 2.0 Reader

Ah, got it. Thanks a ton!

Great post, thank you!

SAD: When I did a very brief lit search, the research showed much larger effects of vitamin D supplementation than light exposure therapy. Of course, they weren't using enough dakka on the light, so both should be used. But two of my close friends with severe SAD were dramatically improved when I got them to supplement D regularly. It's handy that you don't need to take it regularly, just in large doses occasionally (probably don't do more than 50k IU at a time for safety). Sorry I didn't keep the references where I can find them!

Again, doing both is probably a good idea, but most people seem to be vit. D deficient, as you'd expect from a light-exposure-synthesized vitamin, with all of this modern unnatural clothes-wearing and indoors-dwelling.

Back to light: as the standard male night owl (particularly on a WFH flexible schedule): Am I understanding you correctly that if I wanted to go to bed earlier (not sure I do but I probably should), I'd wake up earlier and blast my eyeballs with light right away, then avoid bright light 3-4 hours before bed? Anything else?

I'm talking about finding world-models in which real objects (such as "strawberries" or "chairs") can be identified.

My point is that chairs and humans can be considered in a similar way.

The most straightforward way of finding a world-model is just predicting your sensory input. But then you're not guaranteed to get a model in which something corresponding to "real objects" can be easily identified.

There's the world as a whole that generates observations, and particular objects on their own. A model that cares about individual objects needs to consider them separately from the world. The same object in a different world/situation should still make sense, so there are many possibilities for the way an object can be when placed in some context and allowed to develop. This can be useful for modularity, but also for formulating properties of particular objects, in a way that doesn't get distorted by the influence of the rest of the world. Human preferences is one such property.

I think you're assuming a sharp line between sincere ethics motivations and self-interest. In my view, that doesn't usually exist. People are prone to believe things that suit their self-interest. That motivated reasoning is the biggest problem with public discourse. People aren't lying, they're just confused. I think Musk definitely and even probably Altman believe they're doing the best thing for humanity - they're just confused and not taking the effort to get un-confused.

I'm really sorry all of that happened to you. Capitalism is a harsh system, and humans are harsh beings when we're competing. And confused beings, even when we're trying not to be harsh. I didn't have time to go through your whole story, but I fully believe you were wronged.

I think most villains are the heroes of their own stories. Some of us are more genuinely altruistic than others - but we're all confused in our own favor to one degree or another.

So reducing confusion while playing to everyone's desire to be a hero is one route to survival.

I'm so glad you found that response helpful!

I primarily mean reward in the sense of reinforcement - a functional definition from animal psychology and neuroscience: reinforcement is whatever makes the previous behavior more likely in the future.

But I also mean a positive feeling (qualia if you like, although I find that term too contentious to use much). I think we have a positive feeling when we're getting a reward (reinforcement), but I'm not sure that all positive feelings work as enforcement. Maybe.

As to how deep can that recursive learning mechanism go: very deep. When people spend time arguing about logic and abstract values online, they've gone deep. There's no limit- until the world intervenes to tell you your chain of predicted-reward inferences has gone off-track. For instance, if that person has lost their job, and they're cold and hungry, they might track down the (correct) logic that they ascribed too much value to proving people wrong on the internet, and reduce their estimate of its value.

Hey, I remember your medical miracle post. I enjoyed it!

"Objectively" for me would translate to "biomarker" i.e., a bio-physical signal that predicts a clinical outcome. Note that for depression and many psychological issues this means that we find the biomarkers by asking people how they feel...but maybe this is ok because we do huge studies with good controls, and the biomarkers may take on a life of their own after they are identified.

I'm assuming you mean biomarkers for psychological / mental health outcomes specifically. This is spiritually pretty close to what my lab studies - ways to predict how TMS will affect individuals, and adjust it to make it work better in each person. Our philosophy - which I had to think about for a bit to even articulate, it's so baked into our thinking - is that the effects of an intervention will manifest most reliably in reactions to very simple cognitive tasks like vigilance, working memory, and so on. Most serious health issues impact your reaction times, accuracy, bias, etc. in subtle but statistically reliable ways. Measuring these with random sampling from a phone app and doing good statistics on the data is probably your best bet for objectively assessing interventions. Maybe that is what Quantified Mind does, I'm not sure?

The short answer is that if this were easy, it would already be popular, because we clearly need it. A lot of academic labs and industry people are trying to do this all the time. There is growing success, but it's slow growing and fraught with non-replicable work.

i think i agree that this does justified harm, but maybe for some subgroups or communities the justified harm is worth the benefits of such an event? our local rationality community has developed to a point where i think people are comfortable talking about "controversial" statements with their real faces on because the vibes are one where any attempt at cancellation instead of dialogue will be met with eyerolls and social exclusion but like, you know, it took a pretty long time and sustained effort for us to get here. (and maybe im wrong and there are people in the group with opinions they are still afraid to voice!)

im modelling this as something kind of like authentic relating - you're hacking the group's intimacy module and ratcheting up the feeling of closeness with a shortcut. it's not going to be as good as the genuine thing, but maybe it's a lot better than what one would have general access to. it's not everyone's thing, people with enough access to the genuine goods are likely to be like "wtf this is weird", sometimes it can go catastrophically wrong if the facilitator drops the ball... but despite all of that, for some people it's a good thing to do occasionally bc otherwise they will never get enough of that social nutrient naturally

Question 1:

I stay in the loop when my AGI is solving hard problems. Absolutely it will need persistent goals, new reasoning, and continuous learning to make progress. That changing mind opens up The alignment stability problem [LW · GW] as you note in your comment on the other thread. My job is making sure it's not going off the rails WRT my intent as it works. 

People will do this by default. Letting it run for any length of time without asking questions about what it's up to would be both very expensive, and beyond the bounds of patience and curiosity for almost any humans. I instructed it to cure cancer, but I'm going to keep asking it how it's planning to do that and what progress its making. My important job is asking it about its alignment continually as it learns and plans. I'm frequently asking if it's had ideas about scheming to get its (subgoal) accomlished (while of course re-iterating the standing instructions to tell me the whole truth relevant to my requests). Its alignmnt is my job, until it's so much smarter than me, and clearly understands my intent, to trust it to keep itself aligned.

Question 2:

Yes, instruction-following should be helpful-only. Giving a bunch of constraints on the instructions it will follow adds risk that it won't obey your instructions to shut down or amend its goals or its understanding of previous instructions. That's the principal advantage of corrigibility. Max Harms details this logic [LW · GW] in much more compelling detail.

Yes, this definitely opens up the prospect of misuse, and that is terrifying. But this is not only the safer early route, it's the one AGI project leaders will choose - because they're people who like power.

An org that's created instruction-following AGI would have it follow instructions only from one or a few top "principals". They would instruct it to follow a limited set of instructions from any users they license its instances to. Some of those users would try to jailbreak it to follow dangerous instructions.

And having even a few humans from different groupos (e.g., rival governments) fully in charge of real AGIs would be terrifying. Much more on this in [If we solve alignment, do we die anyway?](https://www.lesswrong.com/posts/kLpFvEBisPagBLTtM/if-we-solve-alignment-do-we-die-anyway-1).

(Provisional current answer after that discussion: it becomes a tricky political negotiation. Who knows. But maybe. 

I favor this approach far above just giving a list of goals and side constraints and hoping they're not too badly interpreted. The reasoning in your post pretty much describes what I'm worried about - although there are also some less intuitive but logically sound misinterpretations of lots of goals that might come into play as the AGI becomes thoroughly superhuman and does more of its own thinking.

To your final observation:

Yes, some outcome-based RL is probably inevitable. o1 was probably trained that way, and others will follow. Let us hope it is not too much to overwhelm the non-instrumental training and the explicit goal of following instructions. And let us figure out how much is likely to be too much before we learn the hard way.

The bet is indeed on. See you back here in 2029 :)

I missed this until I finally got around to responding to your last post, which I'd put on my todo list.

I applaud your initiative and drive! I do think it's a tough pitch to try to leapfrog the fast progressive in deep networks. Nor do I think the alignment picture for those types of systems is nearly as bleak as Yudkowsky & the old school believe. But neither is it likely to be easy enough to leave to chance and those who don't fully grasp the seriousness of the problem. I've written about some of the most common Cruxes of disagreement on alignment difficulty [LW(p) · GW(p)].

So I'd suggest you would have better odds working within the ML framework that is happening with or without your help. I also think that even if you do produce a near-miraculous breakthrough in symbolic GOFAI, Deep Learning Systems Are Not Less Interpretable Than Logic/Probability/Etc [LW · GW].

OTOH, if you have a really insightful approach, and a good reason to think the result would be easier to align than language model agents, maybe pursuing that path makes sense, since no one else is doing exactly that. As I said in my comment on your last request for directions, I think there are higher-expected-value nearly-as-underserved routes to survival; namely, working on alignment for the LLM agents that are our most likely route to first AGIs at this point (focusing on different routes from aligning the base LLMs, which is common but inadequate).

I'm also happy to talk. Your devotion to the project is impressive, and a resource not to be wasted!