LessWrong 2.0 Reader

(that means a classical computer can run software that acts the same way).

No. Computability shows that you can have a classical computer that has the same input/output behavior, not that you can have a classical computer that acts the same way. Input/Output behavior is generally not considered to be enough to guarantee same consciousness, so this doesn't give you what you need. Without arguing about the internal workings of the brain, a simulation of a brain is just a different physical process doing different computational steps that arrives at the same result. A GLUT (giant look-up table) is also a different physical process doing different computational steps that arrives at the same result, and Eliezer himself argued that GLUT isn't conscious.

The "let's swap neurons in the brain with artificial neurons" is actually a much better argument than "let's build a simulation of the human brain on a different physical system", and I don't think it's a coincidence that Eliezer used the second argument in his post.

A recently commonly heard viewpoint on the development of AI states that AI will be economically impactful but will not upend the dominancy of humans. Instead AI and humans will flourish together, trading and cooperating with one another. This view is particularly popular with a certain kind of libertarian economist: Tyler Cowen, Matthew Barnett, Robin Hanson.

They share the curious conviction that the probablity of AI-caused extinction p(Doom) is neglible. They base this with analogizing AI with previous technological transition of humanity, like the industrial revolution or the development of new communication mediums. A core assumption/argument is that AI will not disempower humanity because they will respect the existing legal system, apparently because they can gain from trades with humans.

I think this summarizes my view quite poorly on a number of points. For example, I think that:

1. AI is likely to be much more impactful than the development of new communication mediums. My default prediction is that AI will fundamentally transform and increase the economic growth rate, rather than merely continuing the trend of the last few centuries.

2. Biological humans are very unlikely to remain dominant in the future, pretty much no matter how this is measured. Instead, I predict that artificial minds and humans who upgrade their cognition will likely capture the majority of future wealth, political influence, and social power, with non-upgraded biological humans becoming an increasingly small force in the world over time.

3. The legal system will likely evolve to cope with the challenges of incorporating and integrating non-human minds. This will likely involve a series of revolutionary reforms, and will look very different from the idea of "AIs will fit nearly into human social roles and obey human-controlled institutions indefinitely".

A more accurate description of my view is that humans will become economically obsolete after AGI, but this obsolescence will happen peacefully, without a massive genocide of biological humans. In the scenario I find most likely, humans will have time to prepare and adapt to the changing world, allowing us to secure a comfortable retirement, and/or join the AIs via mind uploading.

A word has to be real already to get into a dictionary.

It's not cheaper in reality. Net metering is effectively a major subsidy that goes away pretty much everywhere that solar generation starts to make up a significant fraction of the supply.

Electricity companies don't want to pay all that capital expense, so it makes sense for them to shift it onto consumers up until home solar generation starts approaching daytime demand. After that point, they can discontinue the net metering and push for "smart meters" that track usage by time of day and charge or pay variable amounts applicable for that particular time, and/or have separate "feed in" credits that are radically smaller per kWh than consumption charges (in practice often up to 85% less).

With smart meters and cheaper home battery systems the incentives starts to shift toward wealthier solar enthusiasts buying batteries and selling excess power to the grid at peak times (or consuming it themselves), lowering peak demand at no additional capital or maintenance cost to the grid operators.

In principle the endgame could involve no wholesale generators at all, just grid operators charging fees to net consumers and paying some nominal amount to net suppliers, but I expect it to not converge to anything as simple as that. Economies of scale will still favour larger-scale operations and local geographic and economic conditions will maintain a mixture of types and scales of generation, storage, distribution, and consumption. Regulation, contracts, and other conditions will also continue to vary greatly from place to place.

fwiw, I think it'd be helpful if this post had the transcript posted as part of the main post body.

I think I actually agree with Lincoln here and think he was saying a different thing than your comment here seems to be oriented around.

I don't think Lincoln's comment had much to do with assuming there was a shadow EA cabal that was aligned with your values. He said "your words are having an impact."

Words having impacts just does actually take time. I updated from stuff Ben Hoffman said, but it did take 3-4 years or something for the update to fully happen (for me in particular), and when I did ~finish updating the amount I was going to update, it wasn't exactly the way Ben Hoffman wanted. In the first 3 years, it's not like I can show Ben Hoffman "I am ready for your approval", or even that I've concretely updated any particular way, because it was a slow messy process and it wasn't like I knew for sure how close to his camp I was going to land.

But, it wouldn't have been true to say "his critiques dropped like a stone through water". (Habryka has said they also affected him, and this seems generally to have actually reveberated a lot). 

I don't know whether or not your critiques have landed, but I think it is too soon to judge.

Specific examples might include criticisms of RSPs, Kelsey’s coverage of the OpenAI NDA stuff, alleged instances of labs or lab CEOs misleading the public/policymakers, and perspectives from folks like Tegmark and Leahy (who generally see a lot of lab governance as safety-washing and probably have less trust in lab CEOs than the median AIS person).

Isn't much of that criticism also forms of lab governance? I've always understood the field of "lab governance" as something like "analysing and suggesting improvements for practices, policies, and organisational structures in AI labs". By that definition, many critiques of RSPs would count as lab governance, as could the coverage of OpenAI's NDAs. But arguments of the sort "labs aren't responsive to outside analyses/suggestions, dooming such analyses/suggestions" would indeed be criticisms of lab governance as a field or activity.

(ETA: Actually, I suppose there's no reason why a piece of X research cannot critique X (the field it's a part of). So my whole comment may be superfluous. But eh, maybe it's worth pointing out that the stuff you propose adding can also be seen as a natural part of the field.)

I think feedback loops are good, but how is that incompatible with taking AI seriously? At this point, even if you want to work on things with tighter feedback loops, AI seems like the central game in town (probably by developing technology that leverages it, while thinking carefully about the indirect effects of that, or at the very least, by being in touch with how it will affect whatever other problem you are trying to solve, since it will probably affect all of them).

so the idea is that you can describe the brain by treating each neuron as a little black box about which you just know its input/output behavior, and then describe the interactions between those little black boxes. Then, assuming you can implement the input/output behavior of your black boxes with a different substrate (i.e., an artificial neuron)

This is guaranteed, because the universe (and any of its subsets) is computable (that means a classical computer can run software that acts the same way).

And there are orders of magnitude more detail going on in my body (and even just in my brain) than I perceive, let alone that I communicate.

There are no sentient details going on that you wouldn't perceive.

It doesn't matter if you communicate something, the important part is that you are capable of communicating it, which means that in changes your input/output pattern (if it didn't, you wouldn't be capable of communicating it even in principle).

Circular arguments that "something is discussed, therefore that thing exists"

This isn't the argument in the OP (even though, when reading quickly, I can see how someone could get that impression).