LessWrong 2.0 Reader

Quicky thoughts, not fully fledged, sorry.

Maybe it depends on the precise way you see the human take-over, but some benefits of Stalin over Clippy include:

Humans have to sleep, have biological functions, and have need to be validated and loved etc which is useful for everyone else.

Humans also have limited life span and their progeny has decent random chances of wanting things to go well for everyone.

Humans are mortal and posses one body which can be harmed if need be making them more likely to cooperate with other humans.

I guess the opposite point of view is that aligning AIs to AI companies' money interests is harmful to the rest of us, so it might actually be better if AI companies didn't have much time to do it, and the AIs got to keep some leftover morality from human texts. And WBE would enable the powerful to do some pretty horrible things to the powerless, so without some kind of benevolent oversight a world with WBE might be scary. But I'm not sure about any of this, maybe your points are right and mine are wrong.

A crux I have on the point about disincentivising developers from developing parts of their own land - how common is this? In my own country, the answer is - not at all, almost all development comes from the government building infrastructure, schools, etc. and developers buy land near where they know the government will build a metro line or whatever to leech off the benefits. Is the situation in the US that developers often buy big plots of cheap land and develop them with roads, hospitals, schools, to benefit from the rise in value of all the other land?

Is it the case that the tech would exist without him? I think that's pretty unclear, especially for SpaceX, where despite other startups in the space, nobody else managed to radically reduce the cost per launch in a way that transformed the industry.

Even for Tesla, which seems more pedestrian (heh) now, there were a number of years where they had the only viable car in the market. It was only once they proved it was feasible that everyone else piled in.

I basically think your sixth to last (or so) bulllet point is key - an AI that takes over is likely to be using a lot more RL on real world problems, i.e. drawn from a different distribution than present-day AI. This will be worse for us than conditioning on a present-day AI taking over.

I think this view is quite US-centric as in fact most countries in the world do not include mineral rights with the land ownership (and yet, minerals are explored everywhere, not just US, meaning imo that profit motive is alive and well when you need to buy licences on top of the land, it's just priced in differently). From Claude:

In a relatively small number of countries, private landowners own mineral rights (including oil) under their property. The United States is the most notable example, where private mineral rights are common through the concept of "mineral estate." Even in the US though, there are some limitations and government regulations on extraction.

The vast majority of countries follow the "state ownership" model, where subsurface minerals including oil are owned by the government regardless of who owns the surface land. This includes:

Most of Europe (including UK, France, Germany)

Russia

China

Most Middle Eastern countries

Most African nations

Most Latin American countries

Canada (where the provinces generally own mineral rights)

Mexico (where oil specifically is constitutionally defined as state property)

Australia (where states own mineral rights)

Even in countries that technically allow private mineral ownership, state-owned companies often have exclusive rights to develop oil resources (like Saudi Aramco in Saudi Arabia or PEMEX in Mexico).

The US system of widespread private mineral rights is quite unique globally. There are a few other countries that have limited forms of private mineral rights, but none with the same extensive private ownership system as the US.

Huh? Environmentalism means let things work as they naturally worked, not change them for the sake of "reversibility" or some other human idea.

Kinda sucks how it is easy to have infinity rules or zero rules but really hard to have a reasonable amount of rules. It reminds me of how I check my email — either every 5 seconds or every 5 weeks.

The difference between a person with an IQ of 90 and one with an IQ of 180 is not in gaps of knowledge or having access to information that's right or wrong but in reasoning ability.

There are enormous differences in gaps of knowledge and further, self-selection into niches and lifestyles and skills like looking up information in Google Scholar, between an IQ 90 and a 180 person. Look at vocab norms or simple ordinary trivia questions such as 'does the earth go around the sun or vice-versa?'. You can't do much reasoning about what you don't know about.

(This is one of the biggest reasons that 'retrieval heavy' LLMs have underperformed so much. There is no 'small logical core' you can easily cheaply learn free of actual real-world knowledge. At best, like the Phi series, you can steal reasoning from a much larger model like GPT-4 which learned it the hard way and has predigested it for you.)

but it still says "it's easy for others to get their own superintelligences with different values", with 'superintelligence' referring to the 'superhuman' AI of 2035?

my response is the same, the story ends before what i meant by superintelligence has occurred.

(it's okay if this discussion was secretly a definition difference till now!)