LessWrong 2.0 Reader

Curated. This post hit me exactly at a moment where I was trying to become a more "numerate person" (i.e. gain some skills where it was easier to convert problems into 'things with numbers' and then solve the problems with simple math).

I liked the worked examples of the post, and I appreciated the emphasis on why it's important to keep track of units (which I had roughly known, but seeing how it played out in the examples drove the point home more).

To avoid the interview-style conversation, I don't have to wait for someone to ask me a question. I can just open up and share relatable things (via conversational signposts) about my life. And if they're not biting, like at all, it's possible we may not have much in common. At that point, it's okay to make a polite excuse to exit the conversation.

The other consideration is that some people feel awkward asking personal questions. To those more introverted people, I praise any questions they ask me. Here's an example:

Them: "How's work going?"
Me: "Thanks for asking! It's going well, though, I'm a bit frustrated at the moment with my boss."
Them: "Frustrated in what way?"
Me: "Hmm, good question...[thinking] I guess it's because he doesn't listen to my input."

^I'm demonstrating that I value their input and encouraging them to ask more questions. (See: Don't Shoot the Dog [LW · GW]).

(Also, I removed the McKinsey example. I don't think it was adding much to the concept of conversational signposts).

I used the word obligation because it felt too hard to find a better one, but I don't like it, even for saving children in shallow ponds. In my mind, obligations are for things you signed up for. In our imperfect world I also feel okay using it for things you got signed up for and benefit from (e.g. I never agreed to be born in the US as a citizen, but I sure do benefit from it, so taxes are an obligation). In my world obligations are always to a specific entity, not general demands. 

I think that for some people, rescuing drowning children is an obligation to society, similar to taxes. Something feels wrong about that to me, although I'd think very badly of someone who could have trivially saved a child and chose not to. 

A key point for me is that people are allowed to be shitty. This right doesn't make them not-shitty or free them from the consequences of being shitty, but it is an affordance available to everyone. Not being shitty requires a high average on erogatory actions, plus some number of supererogatory ones. 

How many supererogatory actions? The easiest way to define this is relative to capacity, but that seems toxic to me, like people to don't have a right to their own gains. It also seems likely to drive lots of people crazy with guilt. I don't know what the right answer is.

TBH I've been really surprised at my reaction to "~obligation to maximal growth". I would have predicted it would feel constraining and toxic, but it feels freeing and empowering, like I've been given a more chances to help people at no cost to me. I feel more powerful. I also feel more permission to give up on what is currently too hard, since sacrificing myself for one short term goal hurts my long term obligation. 

Maybe the key is that this is a better way to think achieve goals I already had. It's not a good frame for deciding what one's goals should be. 

Use a smart but not self-improving AI agent to antagonize the world with the goal of making advanced societies believe that AGI is a bad idea and precipitating effective government actions. You could call this the Ozymandias approach.

ChaosGPT already exists. It's incompetent to the point of being comical at the moment, but maybe more powerful analogues will appear and wreak havoc. Considering the current prevalence of malware, it might be more surprising if something like this didn't happen.

We've already seen developments that could have been considered AI "warning shots" in the past. So far, they haven't been enough to stop capabilities advancement. Why would the next one be any different? We're already living in a world with literal wars killing people right now, and crazy terrorists with various ideologies. It's surprising what people get used to. How bad would a warning shot have to be to shock the world into action given that background noise? Or would we be desensitized by then by the smaller warning shots leading up to it? Boiling the frog, so to speak. I honestly don't know. And by the time a warning shot gets that bad, can we act in time to survive the next one?

Intentionally causing earlier warning shots would be evil, illegal, destructive, and undignified. Even "purely" economic damage at sufficient scale is going to literally kill people. Our best chance is civilization stepping up and coordinating. That means regulations and treaties, and only then the threat of violence to enforce the laws and impose the global consensus on any remaining rogue nations. That looks like the police and the army, not terrorists and hackers.

Be careful that you don't say "the incentives are bad :(" as an easy out. "The incentives!" might be an infohazard, promoting a sophisticated sounding explanation for immoral behavior:

If you find yourself unable to do your job without regularly engaging in practices that clearly devalue the very science you claim to care about, and this doesn’t bother you deeply, then maybe the problem is not actually The Incentives—or at least, not The Incentives alone. Maybe the problem is You.

~ No, it’s not The Incentives—it’s you

The lesson extends beyond science to e.g. Twitter conversations where you're incentivized to sound snappy and confident and not change your mind publicly. 

I don't understand this formulation. If Beauty always says that the probability of Heads is 1/7, does she win? Whatever "win" means...

The primary motive for funding NASA was definitely related to competing with the USSR, but I doubt that it was heavily focused on military applications. It was more along the lines of demonstrating the general superiority of the US system, in order to get neutral countries to side with us because we were on track to win the cold war.

We have already identified some key resources involved in AI development that could be restricted. The economic bottlenecks are mainly around high energy requirements and chip manufacturing.

Energy is probably too connected to the rest of the economy to be a good regulatory lever, but the U.S. power grid can't currently handle the scale of the data centers the AI labs want for model training. That might buy us a little time. Big tech is already talking about buying small modular nuclear reactors to power the next generation of data centers. Those probably won't be ready until the early 2030s. Unfortunately, that also creates pressures to move training to China or the Middle East where energy is cheaper, but where governments are less concerned about human rights.

A recent hurricane flooding high-purity quartz mines made headlines because chip producers require it for the crucibles used in making silicon wafers. Lower purity means accidental doping of the silicon crystal, which means lower chip yields per wafer, at best. Those mines aren't the only source, but they seem to be the best one. There might also be ways to utilize lower-purity materials, but that might take time to develop and would require a lot more energy, which is already a bottleneck.

The very cutting-edge chips required for AI training runs require some delicate and expensive extreme-ultraviolet lithography machines to manufacture. They literally have to plasmify tin droplets with a pulsed laser to reach those frequencies. ASML Holdings is currently the only company that sells these systems, and machines that advanced have their own supply chains. They have very few customers, and (last I checked) only TSMC was really using them successfully at scale. There are a lot of potential policy levers in this space, at least for now.

Who said biological immortality (do you mean a complete cure for ageing?) requires nanobots?

We know individual cell lines can go on indefinitely, the challenge is to have an intelligent multicellular organism that can too.

What exactly do you mean by "different tools need to be used"? Can you give me an example?

I mean that Beauty should maintain full model of experiment, and use decision theory as well as probability theory (if latter is even useful, which it admittedly seems to be). If she didn't keep track of full setup but only "a fair coin was flipped, so the odds are 1:1", she would predictably lose when betting on the coin outcome.

Also, I've minted another "paradox" version. I can predict you'll take issue with one of formulations in it, but what do you think about it?

A fair coin is flipped, hidden from you.

On Heads, you're waken up on Monday, asked "what credence do you have that coin landed Heads?"; on Tuesday, you're let go.

If coin landed Tails, you're waken up on Monday and still asked "what credence do you have that coin landed Heads?"; then, with no memory erasure, you're waken up on Tuesday, and experimenter says to you: "Name the credence that coin landed Heads, but you must name the exact same number as yesterday". Afterwards, you're let go.

If you don't follow experiment protocol, you lose/lose out on some reward.