LessWrong 2.0 Reader

Be careful that you don't say "the incentives are bad :(" as an easy out. "The incentives!" might be an infohazard, promoting a sophisticated sounding explanation for immoral behavior:

If you find yourself unable to do your job without regularly engaging in practices that clearly devalue the very science you claim to care about, and this doesn’t bother you deeply, then maybe the problem is not actually The Incentives—or at least, not The Incentives alone. Maybe the problem is You.

~ No, it’s not The Incentives—it’s you

The lesson extends beyond science to e.g. Twitter conversations where you're incentivized to sound snappy and confident and not change your mind publicly. 

I don't understand this formulation. If Beauty always says that the probability of Heads is 1/7, does she win? Whatever "win" means...

The primary motive for funding NASA was definitely related to competing with the USSR, but I doubt that it was heavily focused on military applications. It was more along the lines of demonstrating the general superiority of the US system, in order to get neutral countries to side with us because we were on track to win the cold war.

We have already identified some key resources involved in AI development that could be restricted. The economic bottlenecks are mainly around high energy requirements and chip manufacturing.

Energy is probably too connected to the rest of the economy to be a good regulatory lever, but the U.S. power grid can't currently handle the scale of the data centers the AI labs want for model training. That might buy us a little time. Big tech is already talking about buying small modular nuclear reactors to power the next generation of data centers. Those probably won't be ready until the early 2030s. Unfortunately, that also creates pressures to move training to China or the Middle East where energy is cheaper, but where governments are less concerned about human rights.

A recent hurricane flooding high-purity quartz mines made headlines because chip producers require it for the crucibles used in making silicon wafers. Lower purity means accidental doping of the silicon crystal, which means lower chip yields per wafer, at best. Those mines aren't the only source, but they seem to be the best one. There might also be ways to utilize lower-purity materials, but that might take time to develop and would require a lot more energy, which is already a bottleneck.

The very cutting-edge chips required for AI training runs require some delicate and expensive extreme-ultraviolet lithography machines to manufacture. They literally have to plasmify tin droplets with a pulsed laser to reach those frequencies. ASML Holdings is currently the only company that sells these systems, and machines that advanced have their own supply chains. They have very few customers, and (last I checked) only TSMC was really using them successfully at scale. There are a lot of potential policy levers in this space, at least for now.

Who said biological immortality (do you mean a complete cure for ageing?) requires nanobots?

We know individual cell lines can go on indefinitely, the challenge is to have an intelligent multicellular organism that can too.

What exactly do you mean by "different tools need to be used"? Can you give me an example?

I mean that Beauty should maintain full model of experiment, and use decision theory as well as probability theory (if latter is even useful, which it admittedly seems to be). If she didn't keep track of full setup but only "a fair coin was flipped, so the odds are 1:1", she would predictably lose when betting on the coin outcome.

Also, I've minted another "paradox" version. I can predict you'll take issue with one of formulations in it, but what do you think about it?

A fair coin is flipped, hidden from you.

On Heads, you're waken up on Monday, asked "what credence do you have that coin landed Heads?"; on Tuesday, you're let go.

If coin landed Tails, you're waken up on Monday and still asked "what credence do you have that coin landed Heads?"; then, with no memory erasure, you're waken up on Tuesday, and experimenter says to you: "Name the credence that coin landed Heads, but you must name the exact same number as yesterday". Afterwards, you're let go.

If you don't follow experiment protocol, you lose/lose out on some reward.

Thanks, edited! Please let me know if it still doesn't work.

Somewhat related: I just published the LLM Deceptiveness and Gullibility Benchmark. This benchmark evaluates both how well models can generate convincing disinformation and their resilience against deceptive arguments. The analysis covers 19,000 questions and arguments derived from provided articles.

I get a "Something's not right. This page doesn't exist" error when I click on the dinner donation link

I agree that the conflation between maintaining human control and alignment and safety is a problem, and to be clear I'm not saying that the outcome of human-controlled AI taking over because someone ordered to do that is an objectively safe outcome.

I agree at present the AI safety field is poorly equipped to avoid catastrophic outcomes that don't involve extinction from uncontrolled AIs.