LessWrong 2.0 Reader

Thanks @TomasD [LW · GW], that's interesting! I agree - most words in my random list seem like random "objects/things/organisms" so there might be some conditioning going on there. Going over your code to see if there's something else that's different.

It's set in about 1995...

God exists because the most reasonable take is the Solomonoff Prior. 

A funny consequence of that is that Intelligent Design will have a fairly large weight in the Solomonoff prior. Indeed the simulation argument can be seen as a version of Intelligent Design. 

The Abrahamic God hypothesis is still substantially downweighted because it seems to involve many contigent bits - i.e noisy random bits that can't be compressed. The Solomonoff prior therefore has to downweight them. 

NotebookLM is able to generate a good podcast from this post. There are some bugs though.

One feature of every lesswrong Petrov day ritual is the understanding that the people on the other side of the button have basically similar goals and reasoning processes, especially when aggregated into a group. I wonder if the mods at /r/sneerclub would be interested in a Petrov day collaboration in the future.

But it's also relevant that we're not asking the superintelligence to grant a random wish, we're asking it for the right to keep something we already have. This seems more easily granted than the random wish, since it doesn't imply he has to give random amounts of money to everyone.

My preferred analogy would be:

You founded a company that was making $77/year. Bernard launched a hostile takeover, took over the company, then expanded it to make $170 billion/year. You ask him to keep paying you the $77/year as a pension, so that you don't starve to death.

This seems like a very sympathetic request, such that I expect the real, human Bernard would grant it. I agree this doesn't necessarily generalize to superintelligences, but that's Zack's point - Eliezer should choose a different example.

I doubt it matters much. 

If Xi Jinping would care enough then they will build A(G)I. 

China has 1,4 billion people.The Chinese economy is 18 trillion dollars per annum. What is a 1,10,100, even a 1000 billion dollar training run on that kind of industrial might? The stickler is political will not economic possibility. 

The only real question is whether Western controls on chip import/exports is working. It seems that China is about 10 years behind on Western chip manufactures - but catching up quickly. 

In general I think it's often helpful to ignore mood afilliatiated stories about this or that country doing well or not well. Focus on the actual numbers*. In many cases the narrative is about an effect that is <1% of the economy. That's not going to matter. 

*well caveat that a Chinese GDP statistics are maybe probably somewhat fake. e.g. satelite light photos etc. 

The things you mentioned were probably all net positive, they just had some negative consequences as well. If you want to drive the far-ish future in a particular direction you’ve just got to accept that you’ll never know for sure that you’re doing a good job.

"Winning The democratic party cares about laws and democratic norms. Trump cares about winning."

I doubt that you passed the IIT. I presume MAGA republicans would say that they're the ones that care about Freedom and Democracy while the democrats are the two-faced cronies of the Deep State.

Back in the day when this discourse first took off I remember meeting a lot of people that told me something in the vein of 'Yudkowskys position that AI will just FOOM like that is silly, not credible, sci-fi. You should really be listening to this Paul guy. His continuous slow takeoff perspective is much more sensible'. 

I remember pointing out to them that "Paul slow" according to Christiano was 'like the industrial revolution but 100x faster'... 

Depending on who you ask, the industrial revolution lasted ~150 years - so that's 1,5 years. That's pretty much what we are seeing. This was just status quo bias and respectability bias. 

Actually, ofc the discourse was double silly. FOOM can still happen even in a 'slow takeoff' scenario. It's a 'slow' straight line until some threshold is hit until it absolutely explodes. [although imho people equivocate between different versions of FOOM in confusing ways [LW(p) · GW(p)]]

Whether or not a certain takeoff will happen only partially has to do with predictable features like scaling compute. Part of it may be predicted by straight-line numerology but a lot of it comes down to harder-to-predict or even impossible-to-predict features about the future like algorithmic & architectural changes, government intervention etc. That was the reasonable position in 2020 anyway. Today we have enough bits to have a pretty good guess when where and how superintelligence will happen.