LessWrong 2.0 Reader

I curated this post because

1. this is a rare productivity system post that made me consider actually implementing it. Right now I can’t because my energy levels are too variable, but if that weren’t true I would definitely be trying it.
2. lots of details, on lots of levels. Things like “I fail 5% of the time” and then translating that too “therefore i price things such that if I could pay 5% of the failure fee to just have it done, I would do so.”
3. Practical advice like “yes verification sometimes takes a stupid amount of time, the habit is nonetheless worth it” or “arrange things to verify the day after”

I was depressed once for ten years and didn’t realize that it was fixable.  I thought it was normal to have no fun and be disagreeable and grumpy and out of sorts all the time.  Now that I’ve fixed it, I’m much better off, and everyone around me is better off.  I enjoy enjoyable activities, I’m pleasant to deal with, and I’m only out of sorts when I’m tired or hungry, as is normal.

If you think you might be depressed, you might be right, so try fixing it.  The cost seems minor compared to the possible benefit (at least it was in my case.). I don’t think there’s a high possibility of severe downside consequences, but I’m not a psychiatrist, so what do I know.

I had been depressed for a few weeks at a time in my teens and twenties and I thought I knew how to fix it: withdraw from stressful situations, plenty of sleep, long walks in the rain.  (In one case I talked to a therapist, which didn’t feel like it helped.)  But then it crept up on me slowly in my forties and in retrospect I spent ten years being depressed.

So fixing it started like this.  I have a good friend at work, of many years standing.  I’ll call him Barkley, because that‘s not his name.  I was riding in the car with my wife, complaining about some situation at work.  My wife said “well, why don’t you ask Barkley to help?”  And I said “Ahh, Barkley doesn’t care.”  And my wife said “What are you saying?  Of course he cares about you.”  And I realized in that moment that I was detached from reality, that Barkley was a good friend who had done many good things for me, and yet my brain was saying he didn’t care.  And thus my brain was lying to me to make me miserable.  So I think for a bit and say “I think I may be depressed.”  And my wife thinks (she told me later) “No duh, you’re depressed. It’s been obvious for years to people who know you.”  But she says “What would you like to do about it?” And I say, “I don’t know, suffer I guess, do you have a better idea?”  And she says “How about if I find you a therapist?”  And my brain told me this was doomed to fail, but I didn’t trust my brain any more, so I said “Okay”.

So I go to the therapist, and conversing with him has many desirable mind-improving effects, and he sends me to a psychiatrist, who takes one look at me and starts me on SSRIs.  And years pass, and I see a different therapist (not as good) and I see a different psychiatrist (better).  
 

And now I’ve been fine for years.  Looking back, here are the things I think worked:

—Talking for an hour a week to a guy who was trying to fix my thinking was initially very helpful.  After about a year, the density of improvements dropped off, and, in retrospect, all subsequent several years of therapy don’t seem that useful.  But of course that’s only clear in retrospect.  Eventually I stopped, except for three-monthly check-ins with my psychiatrist.  And I recently stopped that.

—Wellbutrin, AKA Bupropion.  Other SSRIs had their pluses and minuses and I needed a few years of feeling around for which drug and what dosage was best.  I ended up on low doses of Bupropion and escitalopram.  The Escitalopram doesn‘t feel like it does anything, but I trust my psychiatrist that it does.   Your mileage will vary.

—The ability to detect signs of depression early is very useful.  I can monitor my own mind, spot a depression flare early, and take steps to fix it before it gets bad.  It took a few actual flares, and professional help, to learn this trick.

—The realization that I have a systematic distortion in mental evaluation of plans, making actions seem less promising that they are.  When I’m deciding whether to do stuff, I can apply a conscious correction to this, to arrive at a properly calibrated judgement.

—The realization that, in general, my thinking can have systematic distortions, and that I shouldn’t believe everything I think.  This is basic less-wrong style rationalism, but it took years to work through all the actual consequences on actual me.

—Exercise helps.  I take lots of long walks when I start feeling depressed.  Rain is optional. 

I found this article useful. Any plans to update this for 2024?

Personally, I most enjoyed the first one in the the series, and enjoyed listening to Joe's reading of it even more than when I just read it. My top three are 1, 6, 7.

Idk the Nobel prize committee thought it wasn't significant enough to give out a separate prize 🤷

Agreed code as coordination mechanism

Code nowadays can do lots of things, from buying items to controlling machines. This presents code as a possible coordination mechanism, if you can get multiple people to agree on what code should be run in particular scenarios and situations, that can take actions on behalf of those people that might need to be coordinated.

This would require moving away from the “one person committing code and another person reviewing” code model. 

This could start with many people reviewing the code, people could write their own test sets against the code or AI agents could be deputised to review the code (when that becomes feasible). Only when an agreed upon number of people thinking the code should it be merged into the main system.

Code would be automatically deployed, using gitops and the people administering the servers would be audited to make sure they didn’t interfere with running of the system without people noticing.

Code could replace regulation in fast moving scenarios, like AI. There might have to be legal contracts that you can’t deploy the agreed upon code or use the code by itself outside of the coordination mechanism.

Thinking about AI training runs scaling to the $100b/1T range. It seems really hard to do this as an independent AGI company (not owned by tech giants, governments, etc.). It seems difficult to raise that much money, especially if you're not bringing in substantial revenue or it's not predicted that you'll be making a bunch of money in the near future. 

What happens to OpenAI if GPT-5 or the ~5b training run isn't much better than GPT-4? Who would be willing to invest the money to continue? It seems like OpenAI either dissolves or gets acquired. Were Anthropic founders pricing in that they're likely not going to be independent by the time they hit AGI — does this still justify the existence of a separate safety-oriented org?  

This is not a new idea, but I feel like I'm just now taking some of it seriously. Here's Dario talking about it recently, 

I basically do agree with you. I think it’s the intellectually honest thing to say that building the big, large scale models, the core foundation model engineering, it is getting more and more expensive. And anyone who wants to build one is going to need to find some way to finance it. And you’ve named most of the ways, right? You can be a large company. You can have some kind of partnership of various kinds with a large company. Or governments would be the other source.

Now, maybe the corporate partnerships can be structured so that AGI companies are still largely independent but, idk man, the more money invested the harder that seems to make happen. Insofar as I'm allocating probability mass between 'acquired by big tech company', 'partnership with big tech company', 'government partnership', and 'government control', acquired by big tech seems most likely, but predicting the future is hard. 

On the topic of related work, Mallen et al performed a similar experiment in Eliciting Latent Knowledge from Quirky Language Models, and found similar results.

(As in this work, they did linear probing to distinguish what-model-knows from what-model-says, where models were trained to be deceptive conditional on a trigger word, and the probes weren't trained on any examples of deception behaviors; they found that probing "works," that middle layers are most informative, that deceptive activations "look different" in a way that can be mechanically detected w/o supervision about deception [reminiscent of the PCA observations here], etc.)

Got it, that makes sense. Thanks!

Now replace the word the transformer should define with a real, normal word and repeat the earlier experiment. You will see that it decides to predict [generic object] in a later layer

So trying to imagine a concrete example of this, I imagine a prompt like: "A typical definition of 'goy' would be: a person who is not a" and you would expect the natural completion to be " Jew" regardless of whether attention to " person" is suppressed (unlike in the empty-string case). Does that correctly capture what you're thinking of? ('goy' is a bit awkward here since it's an unusual & slangy word but I couldn't immediately think of a better example)

Someone said they dislike bullet point lists. Here is the same piece formatted as paragraphs. Do you prefer it? (in which case I will edit and change it)

Carlsmith tackles two linked questions:

• How should we behave towards future beings (future humans, AIs etc)?
• What should our priors be about how AIs will behave towards us?
   

Let's consider the first question - what should our poise be towards the future? Perhaps we are worried about the future being less valuable than it could be or that humans will be killed or outcompeted by AI. 

The blog posts contain a range of answers to this question, broadly categorised as follows:

1.  We could trust in some base goodness - the universe, god, AIs being good [? · GW]
2. We could accept that all future beings will be alien to us and stop worrying (see here [? · GW] and here [? · GW])
3. We rely on moral systems or concepts of goodness/niceness [? · GW]
4. We could seize power over the future (see here [? · GW] and here [? · GW])
5. We could adopt a different poise which centred around notions like growth/harmony/ “attunement” (here [? · GW] and here [? · GW])

Let's look at each. Generally each point links to a blog or pair of blog posts.

Trusting in basic goodness, eg god or the universe. I might think God holds the future in His hands, or more broadly that things tend to be okay. Carlsmith considers a distrust of this as a feature of Yudkowskianism, [? · GW] which he labels "deep atheism". Not merely not trusting in God, but not trusting that things will be 'okay' unless we make them so. For similar reasons, Yudkowskians don't assume AIs will be good.  AIs will be good [? · GW]. For them this isn't a good answer.

Next, I might decide this isn't fixable. Hanson argues [? · GW] future people of any stripe might be as deeply alien to us as we are to, say the Ancient Greeks. He doesn't expect a future we consider good to be possible or, likely, desirable. Carlsmith notes that Yudkowskians don't hold this view and muses why. Are they avoiding how alien future people will be? [? · GW] Do they have a clear notion of good that's robust over time? ( Yudkowsky doensn't seem to think so). [? · GW]Or are they avoiding thinking about something uncomfortable? [? · GW]

Many answers seem to rely on moral systems, but these present their own problems. Moral systems vary wildly at edge cases [? · GW] meaning that at the scale of the future, many beliefs systems would advocate for seizing control against others.  We fear paperclipping partially because it is involuntary and aesthetically dull. But the arguments also extend to law abiding and even relatively joyful beings [? · GW] taking increasing control of the future via legal and positive sum means. 

However, without the above, most justifications for seizing control of the future look like those of the AIs [? · GW]. I too would be trying to gain the most resources for my aims at the cost of others, regardless of their ethical stances. In this sense, the AIs aren't bad because they foom, they are bad because they are.. not us. However this looks worrying like a justification that Stalin or the paperclippers could use. See here [? · GW] and here [? · GW].

Finally, Carlsmith posits a hidden fifth option, for which we current lack good concepts. He points to a notion of  trust/ growth/ balance/ attunement [? · GW]. He talks about the colour, 'green' from Magic the Gathering, which is about growing in harmony with complex systems, sometimes trusting, sometimes acting. He notes rationalists and EAs are quite historically inimical to [? · GW] this (favouring 'blue' and 'black'). He repeatedly tries to point [? · GW] at this missing way of being. 

There are therefore a number of ways of dealing with the future, with a number of flaws.

But there is a second parallel discussion, about how might AIs treat us [? · GW]. Perhaps because our imagination of our future selves informs how we imagine AIs. 

For instance if we assume that we cannot trust things (not 1,3) then it's very easy to see AIs as a tool or competitor. Either it is more powerful than us or we are more powerful than it. 

However, if there is a meaningful position on (5). There may be other ways to relate to AIs and future people. Here we might not control them and they might not control us.  We might relates as gentle aliens (eg octopus [? · GW]), as dead-but-not-supreme nature (like a bear [? · GW]). Or something even more other than that. Something we cannot imagine but should attempt to. 

Note that this doesn't mean we shouldn't fear AIs, they might still be capable of ruining the future, but this poise feels different. 

In conclusion, this is my shortest summary of this set of blogs (though there is much much more in there).  We should consider other ways to be towards those we could control but who might control us [? · GW]. We should consider other possible relationships towards AI [? · GW]. In AI discourse there is a lack of clarity in notions of attunement, respect, harmony in relation to the sub-optimal choices of other conscious beings. It is possible that this affects our priors about what AI might be like, possibly pushing towards a worse equilibrium.