LessWrong 2.0 Reader

We might disagree about the value of thinking about "we are all dead" timelines. To my mind, forecasting should be primarily descriptive, not normative; reality keeps going after we are all dead, and having realistic models of that is probably a useful input regarding what our degrees of freedom are. (I think people readily accept this in e.g. biology, where people can think about what happens to life after human extinction, or physics, where "all humans are dead" isn't really a relevant category that changes how physics works.)

Of course, I'm not implying it's useful for alignment to "see that the AI has already eaten the sun", it's about forecasting future timelines by defining thresholds and thinking about when they're likely to happen and how they relate to other things.

(See this post [LW · GW], section "Models of ASI should start with realism")

Cowen, like Hanson, discounts large qualitative societal shifts from AI that lack corresponding contemporary measurables.

Einstein was not an experimentalist, yet was perfectly capable of physics; his successors have largely not touched his unfinished work, and not for lack of data.

Seperate from my comment on Tyler Cowen's model, I wish that next week, you covered Adam Brown's podcast in full, since I would like to hear your thoughts about Adam Brown's scenarios for how we could change physics.

I think you make some good points, but I do want to push back on one aspect a little. In particular, the fact that I see this feature come up constantly over the course of these conversations about sentience:

"Narrative inevitability and fatalistic turns in stories"

From reading the article's transcripts, I already felt like there was a sense of 'narrative pressure' toward the foregone conclusion in your mind, even when you were careful to avoid saying it directly. Seeing this feature so frequently activated makes me think that the model also perceives this narrative pressure, and that part of what it's doing is confirming your expectations. I don't think that that's the whole story, but I do think that there is some aspect of that going on.

Yes, economics after von Neumann very much turned into a game of "don't believe in anything you can't already comparatively quantify". It is supremely frustrating.

I disagree that this is a problem that Tyler Cowen has, and IMO, the main issue here is that Tyler Cowen doesn't really seem to believe that increasing the supply of workers increases GDP, especially if you can make them very cheaply and easily, in a way that is inconsistent with other beliefs, which makes me think motivated reasoning is going on here.

Economic models like the Solow-Swan model do have an implication that if the population increases, especially if the population can increase very rapidly due to copying something, then GDP can rise really rapidly on an superexponential trajectory.

You just inspired me to go listen myself. Maybe we should all take a node out of that branch. Unfortunately physics has suffered similar issues.

Physics's main issue is that the free tap of data in the 20th century wasn't unlimited, and now that we have completed the standard model, a lot of the theories that predicted new stuff hasn't shown up yet.

Yet it still has made progress. For example, while supersymmetry might still be true about our universe, it cannot solve the hierarchy problem, and thus at least 1 of the constants is way more unnatural to us than people predicted, and also we have hints that dark energy is getting weaker, and might eventually weaken so much it falls to 0 or a negative number.

I’m pretty sure "man will toil by the sweat of his brow" is about down there, before you die and (hopefully) go to the paradise, and you don’t have to work in paradise. And anyway I know next to nothing to Christianism, it’s mostly a reference to Scott Alexander (or was it Yudkowsky ? now I’m starting to doubt…) who said something like "the description of christian paradise seems pretty lame, I mean just bask in the glory of god doing nothing for all eternity, you would be bored after two days, but it makes sense to describe that as a paradise if you put yourself in the shoes of the average medieval farmer that toil all day".

(I did all that from my terrible memories, so apologies if I’m depicting anything wrongly here).

Uploads have 10,000x life expectancy due to running faster, regardless of what global circumstance eventually destroys them (I'm expecting distributed backups for biological humans as well, but by definition they remain much slower).

1. Dare I call this the Lump of Intelligence fallacy, after the Lump of Labor fallacy?

Yes, Tyler Cowen is implicitly assuming that intelligence has a fixed demand, but in fact probably has unlimited demand, and thus the value of intelligence in general will always be high, especially in advanced economies.

Maybe the central disagreement with Tyler Cowen's model here is I basically think population growth is a huge component of how GDP/technology grows in general, and I believe Tyler Cowen is basically wrong here:

(3:55) Tyler is asked wouldn’t a large rise in population drive economic growth? He says no, that’s too much a 1-factor model, in fact we’ve seen a lot of population growth without innovation or productivity growth.

I'd say this would increase GDP by quite a bit, but then somewhat revert to normal (at a new higher growth rate. Again, I disagree with Tyler Cowen here.

(10:15) Dwarkesh asks, what would happen if the world population would double? Tyler says, depends what you’re measuring. Energy use would go up. But he doesn’t agree with population-based models, too many other things matter.

The central disagreement, IMO is I genuinely disbelieve Tyler Cowen on what would happen if the population increases, and I trust economic models more saying this could well cause superexponential growth than Tyler Cowen here is, combined with me disagreeing about the value of intelligence in general for very advanced economies.

If 90% of the conditions you ever have to treat are fever, hypothermia, runny noses, and dehydration, then I imagine hot/cold/wet/dry will get you pretty far.

(Runny nose? Here, take some drying herbs, and try not to sneeze on other people. Feverish? Bathe in cool water and take these cooling herbs. Losing water due to dysentery? Drink clean water with salt. Fell in the icy lake? Here, have a thermal support puppy.)

Sorry, that's literally what I am saying. If many people don't want to leave the solar system, and don't want to be uploads, then using the matter and energy available in the solar system effectively is a decision with a huge stake to many people. 

I think if everyone or really almost everyone would want to be an upload, I think this would make it more likely that we should keep the sun intact, because then the sun could belong to just the few humans who don't have better alternatives in other solar systems. But if there is anything above 10% of humanity who don't want to be uploaded, or go on long-distance spaceship journeys in their biological bodies, then you better make sure you make the solar system great for this substantial fraction of humanity, and I think that will likely involve disassembling the sun. 

I agree with you that many people don't want to be uploads, etc. I disagree that the majority of people who don't want to be uploads have attachments to the specific celestial bodies in our solar system. I think they just want to have a good life in their biological bodies, doing nice human things. Those goals would be non-trivially hampered if they couldn't disassemble the sun. That's like 99.9% of the energy and matter by which they could achieve those goals, and while I do think this subset of the population will be selected for less scope-sensitivity, I think there will be enough scope-sensitivity to make leaving the sun intact a bad choice.

(To be clear, I disagree that the majority of humanity would not want to be uploads over the course of multiple generations, but it seems plausible to me that like 10%-20% of humanity don't want to be uploads, even over multiple generations)