That does sound like you might've made some progress in the Way, at least; well done on having acquired that much more dignity.

Thanks Alex for writing this. I think the social failure modes you described in the Mistakes section are all too common, and I've often found myself held back by these. 

I agree that impact measures are not super useful for alignment (apart from deconfusion) and I've also moved on from working on this topic. Improving our understanding of power-seeking seems pretty useful though, so I'm curious why you wish you had stopped working on it sooner. 

I wanted to be more like Eliezer Yudkowsky and Buck Shlegeris and Paul Christiano. They know lots of facts and laws about lots of areas (e.g. general relativity and thermodynamics and information theory). I focused on building up dependencies (like analysis [? · GW] and geometry [? · GW] and topology [? · GW]) not only because I wanted to know the answers, but because I felt I owed a debt, that I was in the red until I could at least meet other thinkers at their level of knowledge. 

I feel kind of bad about some actions of mine related to this. (This has been on my list to write about for a while, but this post seemed like a reasonable prompt to actually write it.)

I personally love studying science, and I get a lot of mileage out of making science analogies when I'm doing research (even though the research is basically just CS and applied math), and I find it incredibly satisfying to find a connection between my current research and some science topic I studied once. (I think there are also some other reasons why studying science feels healthy and good to me.)

But my guess is that studying applied math and CS would have been better for me per hour than studying science, and the reason I spent that time learning science was largely because I think it's exciting and cool rather than because I endorse it as a direct path to knowing things that are useful for doing alignment research (or understanding other important topics).

If some young person asked me what to study, I'd say something like: "You should probably err on the side of learning things that seem relevant to your long term goals, but it's way more efficient to study things you feel really excited about, and so if you (like me) find science more fun than studying applied math, then you should study it with some of your time."

But I definitely don't want to pressure indifferent people into studying random science because they think that people will judge them for not knowing it--people who happen to be passionate about math and CS directly should mostly just study that rather than intentionally diversifying.

I know more science than Paul, and he knows way more CS than me, and basically every time I tell him about a science analogy, he says something like "I know this fact via learning applied math/CS directly, but don't know that science fact".

Nate Soares is a smart guy who's learned a lot of physics, and he told me once that he didn't think it had been that helpful for his research.

(Maybe another reason why I like studying science is that I have a different source of analogies than my computer scientist colleagues, and this diversity of perspectives is sometimes helpful.)

Congratulations on finishing your PhD!

I definitely feel your points about social dynamics having a negative influence on the quality of alignment thinking. Long time LW alignment researchers have status in our social circle. The social gradient pushes towards imitating and impressing them, as opposed to directly pursuing the lines of thought that seem most fruitful to you. 

The instinct is to think in the frames of higher status researchers and to ensure your work is defensible under someone else's frame of alignment. This will never be as efficient / natural as thinking in your own frames, and will invariably push the community towards more conformity with the views of higher-status researchers. Note that the social conformation pressure affects both internal thoughts, and externally expressed opinions, and so it is doubly crippling because it reduces your ability to both think original thoughts and to communicate those original thoughts in your own frame. 

I also feel like I've gotten sharper recently. I feel like I've made connections that I'm not sure the Quintin of a year ago would have spotted. E.g., "values as modular factorizations of a utility function" or "variance in human alignment implies the generators of human alignment can be optimized for more alignment". I've also had moments of introspective awareness into my own cognitive algorithms that seem clearer than had been typical for me. 

I can't describe how I've gotten sharper in as much detail as you can. I think one of the bigger improvements in my own thinking was when I finally grokked that high status alignment researchers can actually be wrong about alignment, and that they can be wrong in huge, important, and obvious-seeming-to-me ways. If you think a higher status person is making a simple mistake, social conformity bias will push you in two incredibly unhelpful directions:

1. Assume the higher status person is right, so as to avoid the possible status hit or potential ementy that might come from contradicting a higher status person.
2. Complicate the mistake that the higher status person seems to be making, so as to avoid "insulting" the higher status person by claiming they made a simple mistake.

These are both, of course, very bad if your actual goal is to accurately identify mistakes in another person's thinking.

Ironically, I think that another factor in improving my reasoning was that I've moved away from trying to force my thoughts to have a "Bayesian" style (or what I imagined as a "Bayesian" style). Previously, I'd been worried about how to correctly update my credence in discrete propositions in the light of new evidence. E.g., "will human level alignment scale to superintelligence? Yes or no?" Now, I instead think in terms of updating an ensemble of latent abstractions over the generators of my observations. 

I.e., there's some distribution over possible learning processes and their downstream alignment properties. I'm trying to model this distribution with abstractions, and I use my observations about humans and their alignment properties as empirical evidence to update my abstractions over learning processes and their alignment properties. The most important thing to think about isn't the (poorly specified) first order question of how well human alignment scales, but the deeper questions about the generators of my current observations.

(I realize that this is still a fundamentally Bayesian way of thinking about inference. But speaking for myself, the admittedly limited concept I had of what "proper Bayesian reasoning" ought to look like was something of a roadblock to acquiring what I now consider to be an improved inferential process.)

I actually have a new puppy - and it certainly has taken a little while for him to figure out that kibble is food so this part stood out to me :)

For example, dogs don’t learn to salivate whenever they see food. This reflex is ‘hard-wired’ into the dog

This doesn't require that dog have a hardwired food classifier. The behaviour is the dog salivating when the dog sees something it recognizes as food, not that the dog is able to recognize all foods. It just needs one hardwired FOOD neuron, that can be attached to a classifier that is later trained. (Idk the technical terms sorry!)

It might still be bullshit - but you'd have to do some experiment that determined whether dogs knew immediately to salivate in the presence of things they know to be food, or if dogs have to learn to salivate in the presence of things they know to be food, which Zitovich's experiment doesn't address because he gave the dog stuff it didn't realize was food.

Unless I'm just missing context, and Pavlov really did think the dog could recognize all types of food that exist from birth...

PS Also congratulations on finishing your PhD. I started one but didn't finish, so much respect from me.

Congrats on the accomplishments. Leaving aside the rest, I like the prompt: why don't people wirehead? Realistically, they're cautious due to having but one brain and a low visibility into what they'd become. A digital-copyable agent would, if curious about what slightly different versions of themselves would do, not hesitate to simulate one in a controlled environment.

Generally I would tweak my brain if it would reliably give me the kind of actions I'd now approve of, while providing at worst the same sort of subjective state as I'd have if managing the same results without the intervention. I wouldn't care if the center of my actions was different as long as the things I value today were bettered.

Anyway, it's a nice template for generating ideas for: when would an agent want to allow its values to shift?

I'm glad you broke free of trying to equal others' bragged-about abilities. Not everyone needs to be great at everything. People who invest in learning something generally talk up the benefits of what they paid for. I'm thinking of Heinlein's famous "specialization is for insects" where I presume much of the laundry lists of things every person should know how to do are exactly the arbtirary things he knows how to do.

I really like this post and found it very interesting, particularly because I'm generally interested in the relationship between the rationality side of the AI Alignment community and academia, and I wanted to register some thoughts. Sorry for the long comment on an old post and I hope this doesn't come across as pernickety. If anything I sort of feel like TurnTrout is being hard on himself. 

I think the tl;dr for my comment is sort of that to me the social dynamics "mistakes" don't really seem like mistakes - or at least not ones that were actually made by the author. 

Broadly speaking, these "mistakes" seem to me like mostly normal ways of learning and doing a PhD that happen for mostly good reasons and my reaction to the fact that these "mistakes" were "figured out" towards the end of the PhD is that this is a predictable part of the transition from being primarily a student to primarily an independent researcher (the fast-tracking of which would be more difficult than a lot of rationalists would like to believe). 

I also worry that emphasizing these things as "mistakes" might actually lead people to infer that they should 'do the opposite' from the start, which to me would sound like weird/bad advice: e.g Don't try to catch up with people who are more knowledgeable than you; don't try to seem smart and defensible; don't defer, you can do just as good by thinking everything through for yourself. 

I broadly agree that

rationality is not about the bag of facts you know.
 

but AI alignment/safety/x-risk isn't synonymous with rationality (Or is it? I realise TurnTrout does not directly claim that it is, which is why I'm maybe more cautioning against a misreading than disagreeing with him head on, but maybe he or others think there is a much closer relationship between rationality and alignment work than I do?). 

Is there not, by this point, something at least a little bit like "a bag of facts" that one should know in AI Alignment? People have been thinking about AI alignment for at least a little while now.  And so like, what have they achieved? Do we or do we not actually have some knowledge about the alignment problem? It seems to me that it would be weird if we didn't have any knowledge - like if there was basically nothing that we should count as established and useful enough to be codified and recorded as part of the foundations of the subject. It's worth wondering whether this has perhaps changed significantly in the last 5-10 years though, i.e. during TurnTrout's PhD. That is, perhaps - during that time - the subject has grown a lot and at least some things have been sufficiently 'deconfused' to have become more established concepts etc.  But generally, if there are now indeed such things, then these are probably things that people entering the field should learn about.  And it would seem likely that a lot of the more established 'big names'/productive people actually know a lot of these things and that "catching up with them" is a pretty good instrumental/proxy way to get relevant knowledge that will help you do alignment work. (I almost want to say: I know it's not fashionable in rationality to think this, but wanting to impress the teacher really does work pretty well in practice when starting out!)

Focussing on seeming smart and defensible probably can ultimately lead to a bad mistake. But when framed more as "It's important to come across as credible" or "It's not enough to be smart or even right; you actually do need to think about how others view you and interact with you", it's not at all clear that it's a bad thing; and certainly it more clearly touches on a regular topic of discussion in EA/rationality about how much to focus on how one is seen or how 'we' are viewed by outsiders. Fwiw I don't see any real "mistake" being actually described in this part of the post. In my opinion, when starting out, probably it is kinda important to build up your credibility more carefully. Then when Quintin came to TurnTrout, he writes that it took "a few days" to realize that Quintin's ideas could be important and worth pursuing.  Maybe the expectation in hindsight would be that he should have had the 'few days' old reaction immediately?? But my gut reaction is that that would be way too critical of oneself and actually my thought is more like 'woah he realised that after thinking about it for only a few days; that's great'. Can the whole episode not be read as a straightforward win: "Early on, it is important to build your own credibility by being careful about your arguments and being able to back up claims that you make in formal, public ways. Then as you gain respect for the right reasons, you can choose when and where to 'spend' your credibility... here's a great example of that..."

And then re: deference, certainly it was true for me that when I was starting out in my PhD, if I got confused reading a paper or listening to talk, I was likely to be the one who was wrong.  Later on or after my PhD, then, yeah, when I got confused by someone else's presentation, I was less likely to be wrong and it was more likely I was spotting an error in someone else's thinking. To me this seems like a completely normal product of the education and sort of the correct thing to be happening. i.e. Maybe the correct thing to do is to defer more when you have less experience and to gradually defer less as you gain knowledge and experience? I'm thinking that under the simple model that when one is confused about something, either you're misunderstanding or the other person is wrong, one starts out in the regime where your confusion is much more often better explained by the fact you have misunderstood and you end up in the regime where you actually just have way more experience thinking about these things and so are now more reliably spotting other people's errors. The rational response to the feeling of confusion changes because once fully accounted for the fact you just know way more stuff and are a way more experienced thinker about alignment. (One also naturally gains a huge boost to confidence as it becomes clear you will get your PhD and have good postdoc prospects etc... so it becomes easier to question 'authority' for that reason too, but it's not a fake confidence boost; this is mostly a good/useful effect because you really do now have experience of doing research yourself, so you actually are more likely to be better at spotting these things).

I'm midway through your post and I want to say that I've also been deep in Mad Investor Chaos. In addition to loving it, I am very inspired by it. 
Inspired to look for ways to improve my thinking. Looking for things I'm flinching away from - and narrow in on those things for inspection.
I keep asking myself - what is it that I already know about the world that I'm pretending not to see? What are things that I can expect to believe later that I should just update and believe now?

I imagine your writing style here reminds me of the manic way of the characters figuring Law out in Mad Investor Chaos, and I like it.
I really appreciate you for writing how despite verbally agreeing that modesty is unproductive, you nevertheless never judged high-status people as dumb. That's totally the kind of noticing/Law I imagine we need more of.
And I also imagine this is the sort of mindset Eliezer is looking for - the mindset where you figure those things out unprompted, without an Eliezer there to correct you.  And I also judge that in your case, even though you could say you are being prompted - by your desire for validation - well, that's good enough. 

Anyway, back to reading.

"Attainable utility preservation is pointless for AGI alignment"

Have you written about this? I recently read that sequence for the first time and thought it was very exciting.

I have new inferences about why I didn't realize AI alignment thinking seemed doomed:

After I became more optimistic about alignment due to having a sharper understanding of the overall problem and of how human values formed to begin with, I also became more pessimistic about other approaches, like IDA/ELK/RRM/AUP/[anything else with a three-letter acronym]. But my new understanding didn't seem to present any specific objections. So why did I suddenly feel worse about these older ideas?

I suspect that part of the explanation is: I hadn't wanted to admit how confused I was about alignment, and I (implicitly) clutched to "but it could work"-style hopefulness. But now that I had a different reason to hope, resting upon a more solid and mechanistic understanding, now it was apparently emotionally safe for me to admit I didn't have much hope at all for the older approaches. 

Yikes. 

If that's what happened, I was seriously deluding myself. I will do better next time.

I think I was, in fact, deluding myself. 

But also I think that, in better understanding the alignment problem, I implicitly realized the inappropriateness of much outer/inner alignment reasoning [LW(p) · GW(p)]. Sentences which used to seem short and descriptive (e.g. "get the AI to care about a robust outer objective") became long and confused in my new ontology. The outer/inner frame, unfortunately, underlies quite a lot of AI alignment thinking. 

I am sad and concerned that I haven't been able to deeply communicate this insight to many existing researchers, but I'm going to keep trying. 

When I get stuck on a problem (e.g. what is the type signature of human values?), I do not stay stuck. I notice I am stuck, I run down a list of tactics, I explicitly note what works, I upweight that for next time. 

What tactics in particular?

Curated. I've been inspired by your work on alignment over the years, your fairly unique efforts, and I think this is a great retrospective on ways you could've cut closer to the enemy. Good luck on succeeding, I look forward to all of your future writing and work.

(Note: this post is available as a podcast, and I added a note to the top of the post letting folks who get the curated emails know.)

That was an inspiring and enjoyable read!

Can you say why you think AUP is "pointless" for Alignment? It seems to me attaining cautious behavior out of a reward learner might turn out to be helpful. Overall my intuition is it could turn out to be an essential piece of the puzzle.

I can think of one or two reasons myself, but I barely grasp the finer points of AUP as it is, so speculation on my part here might be counterproductive.

I wanted to be more like Eliezer Yudkowsky and Buck Shlegeris and Paul Christiano. They know lots of facts and laws about lots of areas (e.g. general relativity and thermodynamics and information theory). I focused on building up dependencies (like analysis [? · GW] and geometry [? · GW] and topology [? · GW]) not only because I wanted to know the answers, but because I felt I owed a debt, that I was in the red until I could at least meet other thinkers at their level of knowledge. 

But rationality is not about the bag of facts you know, nor is it about the concepts you have internalized. Rationality is about how your mind holds itself, it is how you weigh evidence, it is how you decide where to look next when puzzling out a new area. 

If I had been more honest with myself, I could have nipped the "catching up with other thinkers" mistake in 2018. I could have removed the bad mental habits using certain introspective techniques [? · GW]; or at least been aware of the badness.

It makes sense to me that the generator here wasn't ideal, but I'm not currently convinced your actual actions were actually wrong. The first quoted paragraph brings to mind the virtue of scholarship ("If you swallow enough sciences the gaps between them will diminish and your knowledge will become a unified whole". Or, that John Wentworth quote I can't find now about AI alignment work requiring you to have some depth of knowledge in some domains). The reasons you cite in your second paragraph don't seem very connected to whether the actions in the first paragraph mattered. 

I viewed my life through narratives

This is listed as a mistake. But I don't know that the alternative is to "not view my life through narratives".

One alternative model is that humans run on narratives, and as such you need to be good at building good narratives for yourself that cut to what you care about and capture key truths, as opposed to narratives that are solely based on what people will reward you for saying about yourself or something else primarily socially mediated rather than mediated by your goals and how reality works.

Insofar as that model is accurate, I somewhat suspect I will read another post 5 years from now (similar to Hazar's post "How to Ignore Your Emotions (while also thinking you're awesome at emotions) [LW · GW]") where you'll say "I found out the narratives I told myself about myself were hurting me, so I decided to become someone who didn't believe narratives about himself, and it turns out this just worked to hide narratives from my introspective processes, and I hurt myself by acting according to pretty dumb narratives that I couldn't introspect on. Now I instead am conscious of the narratives I live by, and work to change them [LW · GW] as evidence comes in. I agree there are surely faults with them, but I think that's the constraint of my computational architecture that I have to work through, not around."

I'm not sure of this story or prediction. Maybe I'm wrong about the human mind having narratives built-in, though it feels quite a tempting story to me. Maybe you will pull the full transhumanist here and break free from the architecture, but I have a lot of prior probability mass on people making many kinds of "ignore the native architecture" mistake. And of course, maybe you'll discover this mistake in 3 months and not 5 years! Making mistakes faster is another way to get over them.

Very inspiring, of the rationality escape velocity points I do: (a) usually, (b) not yet, (c) always, (d) mostly yes.

May we all become more rational!

I'm having trouble understanding your first point on wanting to 'catch up' to other thinkers. Was your primary message advocating against feeling as if you are 'in dept' until you improve your rationality skills? If so, I can understand that.

But if that is the case, I don't understand the relevance of the lack of a "rationality tech-tree" - sure, there may not be clearly defined pathways to learn rationality. Even so, I think its fair to say that I perceive some people on this blog to currently be better thinkers than I, and that I would like to catch up to their thinking abilities so that I can effectively contribute to many discussions. Would you advocate against that mindset as well?

Thank you for writing this post, I especially appreciate the Mistakes section, since I've seen many rationalists (including me) making similar mistakes at one time or another.

Looking forward to a shard theory sequence.

Our world is so inadequate that seminal psychology experiments are described in mangled, misleading ways. Inadequacy abounds, and status only weakly tracks adequacy. Even if the high-status person belongs to your in-group. Even if all your smart friends are nodding along.

It says he started with the belief. Not, that he was right, or ended with it. Keeping the idea contained to the source, so it's clear it's not being stated could be improved, yes.

On a related note, I recently had the thought "Wow, I feel like the quality of TurnTrout's writing/thinking has noticably improved over the course of 2022. Nice." So there's at least one point of independent outside perception noticing effects related to the internal changes you discuss here.

I think I ended up achieving rationality escape velocity. 
...

The general rhythm is: I feel agentic and capable and self-improving, and these traits are strengthening over time, as is the rate of strengthening.

I found this very inspiring, and it's made me reflect more deeply on how to be purpose-driven, proactive, and intentional in my own life. 


> This definitely didn't have to happen, but I made it happen (with the help of some friends and resources).

I'd be curious how you made it happen! 

Rationality is about how your mind holds itself, it is how you weigh evidence, it is how you decide where to look next when puzzling out a new area. 

I really liked this line. A couple of years I was with friends and we were playing Spades together along with the dad of one of the friends. We were all computer science majors and the dad was farmer. In all of our games he was either first or second. We also played mafia and he was clearly very good at reading people and appearing innocent. 

I've played some games since then with my friends dad and he's always been very skilled. I've always wondered why he was so good at these games we played since it's not something he did often, only the 1 or 2 times a year we'd visit. However, I've reflected on some of the conversations I've had with him and I've realized that he's a very detailed thinker and his job requires him to be constantly making decisions under uncertainty. For example, he has in his head about an hour long talk about what is the ideal corn row width which takes into account the trade-off between output volume and fungal spread and many other things that I can't remember. He'll also casually mention that one of his fields flooded with a nonchalance and a quip that he'll make do. He always sharing how much rain has fallen in the last x days and how much is predicted to fall in the next x days and I've realized that the weather not only affects his crop but also dictates what kind of work he's gonna do and when he'll do it. He's a detailed planner but his plans are always flexible. In games like Spades or Poker where you need make decisions under uncertainty his training as a farmer is far more valuable than my training as a programmer. At a programming job or in competitive programming I'm always looking for an exact answer and either I know it or I don't. There's no real uncertainty outside of saying things like "I'm 70% sure this will work" which is something I generally try to avoid. The biggest advantage my friend's dad has gained as a farmer in these uncertainty laden games is his intuitive sense of probabilities and the planning that should follow which is IMO best gained from accumulating real-life experience with a dedicated effort to improving this intuition and planning. 

This post is not only useful, but beautiful.

This, more than anything else on this website, reflects for me the lived experiences which demonstrate we can become more rational and effective at helping the world.

Many points of resonance with my experience since discovering this community. Many same blind-spots that I unfortunately haven't been able to shortcut, and have had to re-discover by myself. Although this does make me wish I had read some of your old posts earlier.

I'm a natural at this kind of stuff. Yet, growing up during my twenties, I became more like that.

It seems to me you imply that rationalism was a key component. Sometimes I wonder about that for myself. Currently I put more probability on there being many many trajectories of this kind and the specificities not mattering much. Other people have completely different experiences, converge on the same conclusions, and then think their experiences were key. Maybe it's good old growing up.