[link] [poll] Future Progress in Artificial Intelligence

If I understand correctly, the inheritability of a trait often increases with a decrease of environmental variability.

Yes. (More relevantly, I'd say that as the environment gets better, the heritability will increase.)

Overall, your points about the Ethiopian cows are correct but I don't think they would account for more than a relatively small chunk of the difference between the best American milk cows and regular Ethiopian milk cows. It really does look to me like humanity has pushed milk capacity dozens of standard deviations past where it would have been even centuries ago.

They found that a non-linear model predicts the data better than a linear model, which is however quite good, and again I don't find this particularly surprising since linear approximations often perform well on sufficiently smooth functions, especially in the neighbourhood of a stationary point (where you can expect the genotypes of a relatively stable population to be, approximately).

Not surprising no, but people have seriously argued to me that things like embryo selection will not work well or at all because it's possible important stuff will be due to nonlinear genetic interactions (most recently on Google+, but I've seen it elsewhere). So it's something that apparently needs to be established.

My problem with Hsu line of argument is that he extrapolates predictions of these kinds of linear models way past observed phenotypes, which is something that has no theoretical basis, especially given that non-linear effects

I'm not sure how seriously Hsu takes the 30SD part as translating to underlying intelligence; the issue of SDs/normal ordinal distribution of intelligence in the population vs a hypothetical underlying cardinal scale of intelligence (http://lesswrong.com/lw/kcs/what_resources_have_increasing_marginal_utility/b0qb) is not really easy to come down to a hard conclusion except to note that in some areas AI progress curves spend a while in the human range but often go steadily beyond (eg computer chess), which suggests to me that large difference in human intelligence rankings do translate to fairly meaningful (albeit not huge) absolute intelligence differences, in which case the 30SDs might translate to a lot of real intelligence and not some trivial-but-statistically-measurable improvements in how fast they can do crosswords or something.

I think probably the best response here is to take it as saying that the lower limit will be extremely high and equivalent to the top observed phenotype, like a von Neumann. Since right now estimates of IVF sperm donor usage in the USA suggest something like 30-60k kids a year are born that way*, if the fertility doctors dropped in an iterated embryo selection procedure before implantation. I think 30-60k geniuses would make a major difference to society**, and if they happened to be even smarter than the previous top observed phenotypes...?

* I use this figure because looking into the matter, I don't think many women who could bear kids normally would willing sign up for IVF just to get the benefits of embryo selection. It's much too painful, inconvenient, and signals the wrong values. But women who have to do IVF if they ever want to have a kid would be much more likely to make use of it.

** to put 30-60k in perspective, the USA has around 4m babies a year, so ignoring demographics, the top 1% (roughly MENSA level, below-average for LW, well below average for cutting-edge research) of babies represents 40k. If all the IVFers used embryo selection and it boosted the IVF babies to an average of just 130, well below genius, it'd practically single-handedly double the 1%ers.

I was disagreeing that that architectural change would not be helpful on the safety issue.

Take the movie Avatar. The resources needed to render it were: 4,000 Hewlett-Packard servers with 35,000 processor cores with 104 terabytes of RAM and three petabytes of storage. I suppose that it would have been relatively hard to render it on illegally obtained storage and computational resources?

I don't think so. Consider botnets.

How hard is it to buy time on a botnet? Not too hard, since they exist for the sake of selling their services, after all.

Do they have the capacity? Botnets range in size from a few computers to extremes of 30 million computers; if they're desktops, then average RAM these days tends to be at least 4GB, dual core, and hard drive sizes are >=500GB, briefly looking at the cheapest desktops on Newegg. So to get those specs: 35k cores is 17.5k desktops, for 104tb of RAM you'd need a minimum of 104000 / 4 = 26k computers, and the 3pb would be 6k (3000000 / 500); botnets can't use 100% of host resources or their attrition will be even higher than usual, so double the numbers, and the minimum of the biggest is 52k. Well within the range of normal botnets (the WP list has 22 botnets which could've handled that load). And AFAIK CGI rendering is very parallel, so the botnet being high latency and highly distributed might not be as big an issue as it seems.

How much would it cost? Because botnets are a market, it's been occasionally studied/reported on by the likes of Brian Krebs (google 'cost of renting a botnet'). For example, https://www.damballa.com/want-to-rent-an-80-120k-ddos-botnet/ says you could rent a 80-120k botnet for $200 a day, or a 12k botnet for $500 a month - so presumably 5 such botnets would cost a quite reasonable $2500 per month.

(That's much cheaper than Amazon AWS, looks like. https://calculator.s3.amazonaws.com/index.html 17500 t2.medium instances would cost ~$666k a month.)

Do we have any estimates on how quickly a superhuman AI's storage requirements would grow? CERN produces 30 petabytes of data per year. If an AI undergoing an intelligence explosion requires to store vast amounts of data then it will be much harder for it to copy itself.

I don't know. Humans get by adding only a few bits per second to long-term memory, Landauer estimated, but I'm not sure how well that maps onto an AI.

The uncertainties involved here still seem to be too big to claim that a superhuman intelligence will be everywhere moments after you connect it to the Internet.

It may not be able to move itself instantly, but given everything we know about botnets and computer security, it would be able to control a lot of remote computers quickly if prepared, and that opens up a lot of avenues one would rather not have to deal with. (Original AI connects to Internet, seizes a few computers across the world, installs a small seed as root with a carefully packaged set of instructions, small database of key facts around the world, and a bunch of Bitcoin private keys for funding while the seed grows to something approaching the original, and lets itself be wiped by its vigilant overseers.)

Why does the AI have to transfer its source code? I assumed we were just talking about taking over machines as effectors.

This is not magic, I am not a layman, and your beliefs about computer security are wildly misinformed. Putting trojans on large fractions of the computers on the internet is currently within the reach of, and is actually done by, petty criminals acting alone. While this does involve a fair amount of thinking time, all of this thinking goes into advance preparation, which could be done while still in an AI-box or in advance of an order.

I don't see how the NSA could have a trojan everywhere moments after the POTUS asked them to take over the Internet.

Given a backdoor or an appropriate zero-day exploit, I would estimate that it would take no longer than a few minutes to gain control over most of the computers connected to the 'net if you're not worried about detection. It's not hard. Random people routinely build large botnets without any superhuman abilities.