Posts

Forecasting Transformative AI, Part 1: What Kind of AI? 2021-09-24T00:46:49.279Z
This Can't Go On 2021-09-18T23:50:33.307Z
Digital People FAQ 2021-09-13T15:24:11.350Z
Digital People Would Be An Even Bigger Deal 2021-09-13T15:23:41.195Z
The Duplicator: Instant Cloning Would Make the World Economy Explode 2021-09-08T00:06:27.721Z
The Most Important Century: Sequence Introduction 2021-09-03T20:19:27.917Z
All Possible Views About Humanity's Future Are Wild 2021-09-03T20:19:06.453Z
Thoughts on the Singularity Institute (SI) 2012-05-11T04:31:30.364Z
Maximizing Cost-effectiveness via Critical Inquiry 2011-11-10T19:25:14.904Z
Why We Can't Take Expected Value Estimates Literally (Even When They're Unbiased) 2011-08-18T23:34:12.099Z

Comments

Comment by HoldenKarnofsky on Forecasting Transformative AI, Part 1: What Kind of AI? · 2021-09-24T00:00:38.648Z · LW · GW

1. Of course, the answer could be "A kajillion years from now" or "Never."

Comment by HoldenKarnofsky on Forecasting Transformative AI, Part 1: What Kind of AI? · 2021-09-24T00:00:11.463Z · LW · GW

2. See this section of "Forecasting TAI with Biological Anchors" (Cotra (2020)) for a more full definition of "transformative AI."

Comment by HoldenKarnofsky on Forecasting Transformative AI, Part 1: What Kind of AI? · 2021-09-23T23:59:51.803Z · LW · GW

3. I'm sorry. But I do think the rest of the series will be slightly more fun to read this way.

Comment by HoldenKarnofsky on Forecasting Transformative AI, Part 1: What Kind of AI? · 2021-09-23T23:59:34.720Z · LW · GW

4. The examples here are of course simplified. For example, both Deep Blue and AlphaGo incorporate substantial amounts of "tree search," a traditionally-programmed algorithm that has its own "trial and error" process.

Comment by HoldenKarnofsky on Forecasting Transformative AI, Part 1: What Kind of AI? · 2021-09-23T23:59:14.191Z · LW · GW

5. And they can include simulating long chains of future game states.

Comment by HoldenKarnofsky on Forecasting Transformative AI, Part 1: What Kind of AI? · 2021-09-23T23:58:56.266Z · LW · GW

6. Some AIs could be used to determine whether papers are original contributions based on how they are later cited; others could be used to determine whether papers are original contributions based only on the contents of the paper and on previous literature. The former could be used to train the latter, by providing a "That's correct" or "That's wrong" signal for judgments of originality. Similar methods could be used for training AIs to assess the correctness of papers.

Comment by HoldenKarnofsky on Forecasting Transformative AI, Part 1: What Kind of AI? · 2021-09-23T23:58:00.890Z · LW · GW

7. E.g., https://openai.com/blog/improving-language-model-behavior/

Comment by HoldenKarnofsky on Forecasting Transformative AI, Part 1: What Kind of AI? · 2021-09-23T23:57:34.627Z · LW · GW

8. Due to improvements in hardware and software.

Comment by HoldenKarnofsky on Forecasting Transformative AI, Part 1: What Kind of AI? · 2021-09-23T23:57:10.696Z · LW · GW

9. It's even worse than spaghetti code.

Comment by HoldenKarnofsky on Forecasting Transformative AI, Part 1: What Kind of AI? · 2021-09-23T00:41:56.177Z · LW · GW

10. More books: Human Compatible, Life 3.0, and The Alignment Problem

Comment by HoldenKarnofsky on Forecasting Transformative AI, Part 1: What Kind of AI? · 2021-09-23T00:41:33.081Z · LW · GW

Footnotes Container

Comment by HoldenKarnofsky on This Can't Go On · 2021-09-18T23:21:45.815Z · LW · GW

1. If you have no idea what that means, try my short economic growth explainer.

Comment by HoldenKarnofsky on This Can't Go On · 2021-09-18T23:20:51.266Z · LW · GW

2. Global real growth has generally ranged from slightly negative to ~7% per year.

Comment by HoldenKarnofsky on This Can't Go On · 2021-09-18T23:20:27.138Z · LW · GW

3. I'm skipping over 2020 here since it was unusually different from past years, due to the global pandemic and other things.

Comment by HoldenKarnofsky on This Can't Go On · 2021-09-18T23:20:16.237Z · LW · GW

4. For the historical data, see Modeling the Human Trajectory. The projections are rough and meant to be visually suggestive rather than using the best modeling approaches.

Comment by HoldenKarnofsky on This Can't Go On · 2021-09-18T23:19:44.485Z · LW · GW

5. This refers to real GDP growth (adjusted for inflation). 2% is lower than the current world growth figure, and using the world growth figure would make my point stronger. But I think that 2% is a decent guess for "frontier growth" - growth occurring in the already-most-developed economies - as opposed to total world growth, which includes “catchup growth” (previously poor countries growing rapidly, such as China today).

To check my 2% guess, I downloaded this US data and looked at the annualized growth rate between 2000-2020, 2010-2020, and 2015-2020 (all using July since July was the latest 2020 point). These were 2.5%, 2.2% and 2.05% respectively.

Comment by HoldenKarnofsky on This Can't Go On · 2021-09-18T23:19:28.703Z · LW · GW

6. 2% growth over 35 years is (1 + 2%)^35 = 2x growth.

Comment by HoldenKarnofsky on This Can't Go On · 2021-09-18T23:18:48.992Z · LW · GW

7. Wikipedia's highest listed estimate for the Milky Way's mass is 4.5*10^12 solar masses, each of which is about 2*10^30 kg, each of which is estimated as the equivalent of about 1.67*10^-27 hydrogen atoms. (4.5*10^12 * 2*10^30)/(1.67*10^-27) =~ 5.4*10^69.

Comment by HoldenKarnofsky on This Can't Go On · 2021-09-18T23:17:03.690Z · LW · GW

8. Wikipedia: "In March 2019, astronomers reported that the mass of the Milky Way galaxy is 1.5 trillion solar masses within a radius of about 129,000 light-years." I'm assuming we can't travel more than 129,000 light-years in the next 8200 years, because this would require far-faster-than-light travel.

Comment by HoldenKarnofsky on This Can't Go On · 2021-09-18T23:15:44.026Z · LW · GW

9. This calculation isn't presented straightforwardly in the post. The key lines are "No matter what the technology, a sustained 2.3% energy growth rate would require us to produce as much energy as the entire sun within 1400 years" and "The Milky Way galaxy hosts about 100 billion stars. Lots of energy just spewing into space, there for the taking. Recall that each factor of ten takes us 100 years down the road. One-hundred billion is eleven factors of ten, so 1100 additional years." 1400 + 1100 = 2500, the figure I cite. This relies on the assumption that the average star in our galaxy offers about as much energy as the sun; I don't know whether that's the case.

Comment by HoldenKarnofsky on This Can't Go On · 2021-09-18T23:15:26.053Z · LW · GW

10. There is an open debate on whether Modeling the Human Trajectory is fitting the right sort of shape to past historical data. I discuss how the debate could change my conclusions here.

Comment by HoldenKarnofsky on This Can't Go On · 2021-09-17T23:23:59.601Z · LW · GW

11. 250 doublings would be a growth factor of about 1.8*10^75, over 10,000 times the number of atoms in our galaxy.

Comment by HoldenKarnofsky on This Can't Go On · 2021-09-17T23:23:44.171Z · LW · GW

12. 20 years would be 240 months, so if each one saw a doubling in the world economy, that would be a growth factor of about 1.8*10^72, over 100 times the number of atoms in our galaxy.

Comment by HoldenKarnofsky on This Can't Go On · 2021-09-17T23:23:27.603Z · LW · GW

13. That’s because of the above observation that today’s growth rate can’t last for more than another 8200 years (82 centuries) or so. So the only way we could have more than 82 more centuries with growth equal to today’s is if we also have a lot of centuries with negative growth, ala the zig-zag dotted line in the "This Can't Go On" chart.

Comment by HoldenKarnofsky on This Can't Go On · 2021-09-17T23:23:13.544Z · LW · GW

14. This dataset assigns significance to historical figures based on how much they are covered in reference works. It has over 10x as many "Science" entries after 1500 as before; the data set starts in 800 BC. I don't endorse the book that this data set is from, as I think it draws many unwarranted conclusions from the data; here I am simply supporting my claim that most reference works will disproportionately cover years after 1500. 

Comment by HoldenKarnofsky on This Can't Go On · 2021-09-17T23:22:45.903Z · LW · GW

15. To be fair, reference works like this may be biased toward the recent past. But I think the big-picture impression they give on this point is accurate nonetheless. Really supporting this claim would be beyond the scope of this post, but the evidence I would point to is (a) the works I'm referencing - I think if you read or skim them yourselves you'll probably come out with a similar impression; (b) the fact that economic growth shows a similar pattern (although the explosion starts more recently; I think it makes intuitive sense that economic growth would follow scientific progress with a lag).

Comment by HoldenKarnofsky on This Can't Go On · 2021-09-17T23:22:21.451Z · LW · GW

16. The papers cited in The Duplicator on this point specifically model an explosion in innovation as part of the dynamic driving explosive economic growth.

Comment by HoldenKarnofsky on This Can't Go On · 2021-09-17T23:22:00.918Z · LW · GW

Footnotes Container

Comment by HoldenKarnofsky on Digital People Would Be An Even Bigger Deal · 2021-09-13T02:18:17.426Z · LW · GW

1.The best example I can think of, but surely not the only one.

Comment by HoldenKarnofsky on Digital People Would Be An Even Bigger Deal · 2021-09-13T02:17:49.029Z · LW · GW

2. The movie The Matrix gives a decent intuition for the idea with its fully-immersive virtual reality, but unlike the heroes of The Matrix, a digital person need not be connected to any physical person - they could exist as pure software.

The agents ("bad guys") are more like digital people than the heroes are. In fact, one extensively copies himself.

Comment by HoldenKarnofsky on Digital People Would Be An Even Bigger Deal · 2021-09-13T02:17:17.639Z · LW · GW

3. See Age of Em Chapter 6, starting with "Regarding the computation ..."

Comment by HoldenKarnofsky on Digital People Would Be An Even Bigger Deal · 2021-09-13T02:16:38.736Z · LW · GW

4. For example, when multiple teams of digital people need to coordinate on a project, they might speed up (or slow down) particular steps and teams in order to make sure that each piece of the project is completed just on time. This would allow more complex, "fragile" plans to work out. (This point is from Age of Em Chapter 17, "Preparation" section.)

Comment by HoldenKarnofsky on Digital People Would Be An Even Bigger Deal · 2021-09-13T02:16:19.432Z · LW · GW

5. See Age of Em Chapter 11, "Retirement" section.

Comment by HoldenKarnofsky on Digital People Would Be An Even Bigger Deal · 2021-09-13T02:15:58.152Z · LW · GW

6. Without human bodies - and depending on what kinds of robots were available - digital people might not be good substitutes for humans when it comes to jobs that rely heavily on human physical abilities, or jobs that require in-person interaction with biological humans.

However, digital people would likely be able to do everything needed to cause an explosive economic growth, even if they couldn't do everything. In particular, it seems they could do everything needed to increase the supply of computers, and thereby increase the population of digital people.

Creating more computing power requires (a) raw materials - mostly metal; (b) research and development - to design the computers; (c) manufacturing - to carry out the design and turn raw materials into computers; (d) energy. Digital people could potentially make all of these things a great deal cheaper and more plentiful:

  • Raw materials. It seems that mining could, in principle, be done entirely with robots. Digital people could design and instruct these robots to extract raw materials as efficiently as possible.
  • Research and development. My sense is that this is a major input into the cost of computing today: the work needed to design ever-better microprocessors and other computer parts. Digital people could do this entirely virtually.
  • Manufacturing. My sense is that this is the other major input into the cost of computing today. Like mining, it could in principle be done entirely with robots.
  • Energy. Solar panels are also subject to (a) better research and development; (b) robot-driven manufacturing. Good enough design and manufacturing of solar panels could lead to radically cheaper and more plentiful energy.
  • Space exploration. Raw materials, energy, and "real estate" are all super-abundant outside of Earth. If digital people could design and manufacture spaceships, along with robots that could build solar panels and computer factories, they could take advantage of massive resources compared to what we have on earth.
Comment by HoldenKarnofsky on Digital People Would Be An Even Bigger Deal · 2021-09-13T02:15:25.403Z · LW · GW

7. It is debatable whether the world is getting somewhat better at these things, somewhat worse, or neither. But it seems pretty clear that the progress isn't as impressive as in computing.

Comment by HoldenKarnofsky on Digital People Would Be An Even Bigger Deal · 2021-09-13T02:15:00.009Z · LW · GW

8. Why would the copy cooperate in the experiment? Perhaps because they simply were on board with the goal (I certainly would cooperate with a copy of myself trying to learn about meditation!). Perhaps because they were paid (in the form of a nice retirement after the experiment). Perhaps because they saw themselves and their copies (and/or original) as the same person (or at least cared a lot about these very similar people). A couple of factors that would facilitate this kind of experimentation: (a) digital people could examine their own state of mind to get a sense of the odds of cooperation (since the copy would have the same state of mind); (b) if only a small number of digital people experimented, large numbers of people could still learn from the results.

Comment by HoldenKarnofsky on Digital People Would Be An Even Bigger Deal · 2021-09-13T02:13:26.254Z · LW · GW

9. I'd also expect them to be able to try more radical things. For example, in today's world, it's unlikely that you could run a randomized experiment on what happens if people currently living in New York just decide to move to Chicago. It would be too hard to find people willing to be randomly assigned to stay in New York or move to Chicago. But in a world of digital people, experimenters could pay New Yorkers to make copies of themselves who move to Chicago. And after the experiment, each Chicago copy that wished it had stayed in New York could choose to replace itself with another copy of the New York version. (The latter brings up questions about philosophy of personal identity, but for social science purposes, all that matters is that some people would be happy to participate in experiments due to this option, and everyone could learn from the experiments.)

Comment by HoldenKarnofsky on Digital People Would Be An Even Bigger Deal · 2021-09-13T02:12:57.424Z · LW · GW

10. See footnote from the first bullet point on why people's copies might cooperate with them. 

Comment by HoldenKarnofsky on Digital People Would Be An Even Bigger Deal · 2021-09-13T02:12:39.191Z · LW · GW

11. And air for cooling.

Comment by HoldenKarnofsky on Digital People Would Be An Even Bigger Deal · 2021-09-13T02:12:27.515Z · LW · GW

12. See the estimates in Astronomical Waste for a rough sense of how big the numbers can get here (although these estimates are extremely speculative).

Comment by HoldenKarnofsky on Digital People Would Be An Even Bigger Deal · 2021-09-13T02:10:30.255Z · LW · GW

Footnotes Container

This comment is a container for our temporary "footnotes-as-comments" implementation that gives us hover-over-footnotes.

Comment by HoldenKarnofsky on Digital People FAQ · 2021-09-12T22:41:25.196Z · LW · GW

1. The agents ("bad guys") are more like digital people. In fact, one extensively copies himself.

Comment by HoldenKarnofsky on Digital People FAQ · 2021-09-12T22:39:21.075Z · LW · GW

2. These are all taken from this video, except for the last one.

Comment by HoldenKarnofsky on Digital People FAQ · 2021-09-12T22:38:47.588Z · LW · GW

3. Football video games have already expanded to simulate offseason tradings, signings and setting ticket prices.

Comment by HoldenKarnofsky on Digital People FAQ · 2021-09-12T22:38:35.028Z · LW · GW

4. It's also possible there could be conscious "digital people" who did not resemble today's humans, but I won't go into that here - I'll just focus on the concrete example of "digital people" that are virtual versions of humans. 

Comment by HoldenKarnofsky on Digital People FAQ · 2021-09-12T22:38:00.163Z · LW · GW

5. According to the PhilPapers Surveys, 56.5% of philosophers endorse physicalism, vs. 27.1% who endorse non-physicalism and 16.4% "other." I expect the vast majority of philosophers who endorse physicalism to agree that a sufficiently detailed simulation of a human would be conscious. (My understanding is that biological naturalism is a fringe/unpopular position, and that physicalism + rejecting biological naturalism would imply believing that sufficiently detailed simulations of humans would be conscious.) I also expect that some philosophers who don't endorse physicalism would still believe that such simulations would be conscious (David Chalmers is an example - see The Conscious Mind). These expectations are just based on my impressions of the field. 

Comment by HoldenKarnofsky on Digital People FAQ · 2021-09-12T22:37:36.935Z · LW · GW

6. From an email from a physicist friend: "I think a lot of people have the intuition that real neural activity, produced by real chemical reactions from real neurotransmitters, and real electrical activity that you can feel with your hand, somehow has some property that mere computer code can't have. But one of the overwhelming messages of modern physics has been that everything that exists -- particles, fields, atoms, etc, is best thought of in terms of information, and may simply *be* information. The universe may perhaps be best described as a mathematical abstraction. Chemical reactions don't come from some essential property of atoms but instead from subtle interactions between their valence electron shells. Electrons and protons aren't well-defined particles, but abstract clouds of probability mass. Even the concept of "particles" is misleading; what seems to actually exist is quantum fields which are the solutions of abstract mathematical equations, and some of whose states are labeled by humans as "1 particle" or "2 particles". To be a bit metaphorical, we are like tiny ripples on vast abstract mathematical waves, ripples whose patterns and dynamics happen to execute the information processing corresponding to what we call sentience. If you ask me our existence and the substrate we live on is already much weirder and more ephemeral than anything we might upload humans onto."

Comment by HoldenKarnofsky on Digital People FAQ · 2021-09-12T22:36:08.715Z · LW · GW

7. I actually expect it would start off very expensive, but become cheaper very quickly due to a productivity explosion, discussed below.

Comment by HoldenKarnofsky on Digital People FAQ · 2021-09-12T22:35:46.118Z · LW · GW

8. For an illustration of this, see this report: How much computational power does it take to match the human brain? (Particularly the Uncertainty in neuroscience section.) Even estimating how many meaningful operations the human brain performs is, today, very difficult and fraught - let alone characterizing what those operations are.

Comment by HoldenKarnofsky on Digital People FAQ · 2021-09-12T22:35:07.416Z · LW · GW

9. This statement is based on my understanding of conventional wisdom plus the fact that recorded video and audio often seems quite realistic, implying that the camera/microphone didn't fail to record much important information about its source.