[LINK] Nick Szabo: Beware Pascal's Scams

Or is at least a symptom of bad PR.

People currently understand the physical world sufficiently to see that supernatural claims are bogus, and so there is certainty about impossibility of developments predicated on supernatural. People know robust and general laws of physics that imply impossibility of perpetual motion, and so we can conclude in advance with great certainty that any perpetual motion engineering project is going to fail. Some long-standing problems in mathematics were attacked unsuccessfully for a long time, and so we know that making further progress on them is hard. In all these cases, there are specific pieces of positive knowledge that enable the inference of impossibility or futility of certain endeavors.

In contrast, a lot of questions concerning Friendly AI remain confusing and unexplored. It might turn out to be impossibly difficult to make progress on them, or else a simple matter of figuring out how to apply standard tools of mainstream mathematics. We don't know, but neither do we have positive knowledge that implies impossibility or extreme difficulty of progress on these questions. In particular, the enormity of consequences does not imply extreme improbability of influencing those consequences. It looks plausible that the problem can be solved.

This kind of seems like political slander to me. Maybe I'm miscalibrated? But it seems like you're thinking of "reasonable estimates" as things produced by groups or factions, treating SI as a single "estimate" in this sense, and lumping them with a vaguely negative but non-specified reference class of "prophetic groups".

The packaged claims function to reduce SI's organizational credibility, and yet it references no external evidence and makes no testable claims. For your "prophetic groups" reference class, does it include 1930's nuclear activists, 1950's environmentalists, or 1970's nanotechnology activists? Those examples come from the socio-political reference class I generally think of SI as belonging to, and I think of them in a mostly positive way.

Personally, I prefer to think of "estimates" as specific predictions produced by specific processes at specific times, and they seem like they should be classified as "reasonable" or not on the basis of their mechanisms and grounding in observables in the past and the future.

The politics and social dynamics surrounding an issue can give you hints about what's worth thinking about, but ultimately you have to deal with the object level issues, and the object level issues will screen off the politics and social dynamics once you process them. The most reasonable tool for extracting a "coherent opinion" from someone on the subject of AGI that is available to the public that I'm aware of is the uncertain future.

(Endgame: Singularity is a more interesting tool in some respects. It's interesting for building intuitions about certain kinds of reality/observable correlations because it has you play as a weak but essentially benevolent AGI rather than as humanity, but (1) it is ridiculously over-specific as a prediction tool, and (2) seems to give the AGI certain unrealistic advantages and disadvantages for the sake of making it more fun as a game. I've had a vague thought to fork it, try to change it to be more realistic, write a bot for playing it, and use that as an engine for Monte-carlo simulator of singularity scenarios. Alas: a day job prevents me from having the time, and if that constraint were removed I bet I could find many higher value things to work on, reality being what it is, and people being motivated to action the way they are.)

Do you know of anything more epistemically helpful than the uncertain future? If so, can you tell me about it? If not, could you work through it and say how it affected your model of the world?

I think your estimate of their chances of success is low. But even given that estimate, I don't think it's Pascalian. To me, it's Pascalian when you say "my model says the chances of this are zero, but I have to give it non-zero odds because there may be an unknown failing in my model". I think Heaven and Hell are actually impossible, I'm just not 100% confident of that. By contrast, it would be a bit odd if your model of the world said "there is this risk to us all, but the odds of a group of people causing a change that averts that risk are actually zero".

I'm not seeing how you get from "doesn't state an explicit probability or timescale publically" to "argues that SI should be supported on Pascalian grounds".

It's not clear to me how approaching your response with an assumption of bad faith will convince him or his readers of the correctness of your position. Let us know how it works out for you.

I would have preferred that he use my (even more passive-aggressive) approach, which is to say, "I'm not going to name any names[1]", and then have a footnote saying "[1] A 'name' is an identifier used to reference a proper noun. An example of a name might be 'Singularity Institute'."

Get it? You're not "naming names", you're just giving an example of name in the exact neighborhood of the accusation! Tee hee!

(Of course, it's even better if you actually make the accusation directly, but that's obviously not an option here.)

From zero up to $1,000,000,000.

I do, in that there are nonfatal actions that i would not take in exchange for that much money. Of course, at numbers over several hundred thousand, money loses unit utility very fast. One billion dollars has significantly less than one thousand times the value to me of one million dollars, because the things I can buy with a billion dollars are less than one thousand times as valuable to me as the things I can buy with a million.

It can be summarized as follows: for basic reasons of economics and computer science, specialized algorithms are generally far superior to general ones.

It would be better to present, as your main reason, "the kinds of general algorithms that humans are likely to develop and implement, even absent impediments caused by AI-existential risk activism, will almost certainly be far inferior to specialized ones". That there exist general-purpose algorithms which subsume the competitive abilities of all existing human-engineered special-purpose algorithms, given sufficient advantages of scale in number of problem domains, is trivial by the existence proof constituted by the human economy.

Put another way: There is some currently-unsubstitutable aspect of the economy which is contained strictly within human cognition and communication. Consider the case where the intellectual difficulties involved in understanding the essence of this unsubstitutable function were overcome, and it were implemented in silico, with an initial level of self-engineering insight already equal to that which was used to create it, and with starting capital and education sufficient to overcome transient learning-curve effects on its initial success. There would then be some fraction of the economy directed by the newly engineered process. Would this fraction of the economy inevitably be at a net competitive advantage, or disadvantage, relative to the fraction of the economy which was directed by humans?

If that fraction of the economy would have an advantage, then this would be an example of a general algorithm ultimately superior to all contemporarily-available specialized algorithms. In that case, what you claim to be the core of your argument would be defeated; the strength of your argument would instead have to come from a focus on the reasons why it were improbable that anyone had a relevant chance of ever achieving this kind of software substitute for human strategy and insight (that is, before everyone else was adequately prepared for it to prevent catastrophe), and that even to the point that supposing otherwise deserves to be tarred with a label of "scam". And if the software-directed economy would have a disadvantage even at steady state, then this would be a peculiar fact about software and computing machinery relative to neural states and brains, and it could not be assumed without argument. Digital software and computing machinery both have properties that have made them, in most respects, much more tractable to large returns to scale from purposeful re-engineering for higher performance than neural states and brains, and this is likely to continue to be true into the future.

It can be summarized as follows: for basic reasons of economics and computer science, specialized algorithms are generally far superior to general ones.

I don't understand your reasoning here. If you have a general AI, it can always choose to apply or invent a specialized algorithm when the situation calls for that, but if all you have is a collection of specialized algorithms, then you have to try to choose/invent the right algorithm yourself, and will likely do a worse (possibly much worse) job than the general AI if it is smarter than you are. So why do we not have to worry about "extreme consequences from general AI"?