LessWrong 2.0 Reader

One notebook per paper figure

• When you get to paper writing, it is very helpful to have one notebook per paper figure. This allows other collaborators to easily reproduce the figure and make changes as required. It also reduces the risk of horrible git merge conflicts if everyone is iterating in parallel.

After our BoN paper I actually kind of disagree with this take. I think I definitely leaned more towards liking to use the notebooks that had all the plots (cleanly). At the very least, I would put this much lower on the list

e.g. during the rebuttal period
specific that this is a paper rebuttal period 

This seems like a superficial reading of the blog post. The part you focus on that the MS President announced was not really that much news - $80b is a nice specific citable number, yes, but I don't know of widespread belief that MS's datacenter capex was going to be much below that, and this is consistent with past reporting like Stargate being $100b+.

The important thing about the post, and why the MS President wrote it and posted it when he did, is the messaging it contains to the incoming Trump administration and to the natsec establishment, lobbying for specific policies around AI regulation and chip exports and how they should be wielded for foreign policy. (See also the Oracle blog post, the new chip export regulation proposal, and Masayoshi Son.) Probably the most important part of the post was the paragraph about G42, and that's the one you and pretty much all commentators appear to have not even read.

As TurnTrout explained in Reward is not the optimization target, this is technically incorrect. The technically correct description is that reward chisels cognition in ways that led to reward during training. Again, talking as if reward is the optimization target is common and practical, in humans and other intelligences, but because it is technically incorrect it can lead us astray.

Wrong, and yet another example of why that was such a harmful essay. TurnTrout's claim apply only to a narrow (and largely obsolete) class of RL agents - which does not cover humans or LLMs (you know, the actual RL agents we are dealing with today) - and he concedes that, but readers like you nevertheless come away with a grossly inflated belief. In reality, for humans and LLMs, reward is the optimization target, and this is why things like Claude's reward-hacking exist. Because that is what they optimize: the reward.

Quicky thoughts, not fully fledged, sorry.

Maybe it depends on the precise way you see the human take-over, but some benefits of Stalin over Clippy include:

Humans have to sleep, have biological functions, and have need to be validated and loved etc which is useful for everyone else.

Humans also have limited life span and their progeny has decent random chances of wanting things to go well for everyone.

Humans are mortal and posses one body which can be harmed if need be making them more likely to cooperate with other humans.

I guess the opposite point of view is that aligning AIs to AI companies' money interests is harmful to the rest of us, so it might actually be better if AI companies didn't have much time to do it, and the AIs got to keep some leftover morality from human texts. And WBE would enable the powerful to do some pretty horrible things to the powerless, so without some kind of benevolent oversight a world with WBE might be scary. But I'm not sure about any of this, maybe your points are right and mine are wrong.

A crux I have on the point about disincentivising developers from developing parts of their own land - how common is this? In my own country, the answer is - not at all, almost all development comes from the government building infrastructure, schools, etc. and developers buy land near where they know the government will build a metro line or whatever to leech off the benefits. Is the situation in the US that developers often buy big plots of cheap land and develop them with roads, hospitals, schools, to benefit from the rise in value of all the other land?

Is it the case that the tech would exist without him? I think that's pretty unclear, especially for SpaceX, where despite other startups in the space, nobody else managed to radically reduce the cost per launch in a way that transformed the industry.

Even for Tesla, which seems more pedestrian (heh) now, there were a number of years where they had the only viable car in the market. It was only once they proved it was feasible that everyone else piled in.

I basically think your sixth to last (or so) bulllet point is key - an AI that takes over is likely to be using a lot more RL on real world problems, i.e. drawn from a different distribution than present-day AI. This will be worse for us than conditioning on a present-day AI taking over.

I think this view is quite US-centric as in fact most countries in the world do not include mineral rights with the land ownership (and yet, minerals are explored everywhere, not just US, meaning imo that profit motive is alive and well when you need to buy licences on top of the land, it's just priced in differently). From Claude:

In a relatively small number of countries, private landowners own mineral rights (including oil) under their property. The United States is the most notable example, where private mineral rights are common through the concept of "mineral estate." Even in the US though, there are some limitations and government regulations on extraction.

The vast majority of countries follow the "state ownership" model, where subsurface minerals including oil are owned by the government regardless of who owns the surface land. This includes:

Most of Europe (including UK, France, Germany)

Russia

China

Most Middle Eastern countries

Most African nations

Most Latin American countries

Canada (where the provinces generally own mineral rights)

Mexico (where oil specifically is constitutionally defined as state property)

Australia (where states own mineral rights)

Even in countries that technically allow private mineral ownership, state-owned companies often have exclusive rights to develop oil resources (like Saudi Aramco in Saudi Arabia or PEMEX in Mexico).

The US system of widespread private mineral rights is quite unique globally. There are a few other countries that have limited forms of private mineral rights, but none with the same extensive private ownership system as the US.