Discussion: Yudkowsky's actual accomplishments besides divulgation

Have you tried getting into analytical philosophy\?

I don't know how to say this without sounding rude, but the Future of Humanity Institute and Nick Bostrom give a formal/professional appearance that the SIAI and Eliezer Yudkowsky lack.

There's some truth to that - but I can't say I am particularly sold on the FHI either. Yudkowsky seems less deluded about brain emulation than they are. Both organisations are basically doom-mongering. Doom-mongers are not known for their sanity or even-headedness:

History is peppered with false prognostications of imminent doom. Blustering doomsayers are harmful: Not only do they cause unnecessary fear and disturbance, but worse: they deplete our responsiveness and make even sensible efforts to understand or reduce existential risk look silly by association.

It seems difficult to study this subject and remain objective. Those organisations that have tried so far have mostly exaggerated the prospects for the end of the world. They form from those who think the end of the world is more likely than most, associate with others with the same mindset, and their funding often depends on how convincing and dramatic picture of DOOM they can paint. The results tend to lead to something of a credibility gap.

I am under the impression that SIAI is well aware that they could use more appearance of seriousness.

Oh, I'm not saying he doesn't deserve praise, Guy's works changed my life forever. I 'm just saying I got points for praising him without properly justifying it on more than one occasion, which I feel guilty for. I also don't think he should be bashed for the sake of bashing or other Why Our Kind Can't Cooperate gratuitous dissenting.

Yes, populus and vulgus are basically synonims, with vulgus having the worst connotations ("folk" VS "the mob" basically), but semantic sliding and usage have made "popular" and its derivates get a base connotation. People don't as easily link "divulgation" and "vulgar".

It'd be nice to have a word that basically means "spreading elevated knowledge to the untrained" without making it sound like we're abasing it. Every time I hear the term "Popular Science" I think of Dr. Sheldon Cooper deriding and ridiculing any of his colleagues who are trying to do just that.

Pleased to meet you, Dr Gablehouser. How fortunate for you that the University has chosen to hire you, despite the fact that you’ve done no original research in 25 years, and instead have written a series of popular books that reduce the great concepts of science to a series of anecdotes, each one dumbed down to accommodate the duration of an average bowel movement.

That sort of elitism just makes me sick*, and I've seen it in Real Life, even among scientists and from scientists towards engineers ("The Oompa Loompas of Science", another Sheldonism)..

If only for self-serving reasons, it is very counterproductive. The more people know about Science, the more likely they are to understand the importance of any given work... and fund it. Also, the more likely they are to show respect to science-folk and freaking listen to them. That means investing time and effort to make this stuff reach the masses, and it's perfectly understandable that a researcher spend their entire career on that: understanding scientific concepts proprely and then managing to grab untrained people's interest and eloquently explain advanced concepts to them so that they grasp even a pale reflection of them is not trivial.

Oh. It looks pretty nice actually. Still, inflection-latin might be more fun to learn, but I guess if you just want people to learn Latin vocabulary and use it for simple thing so they aren't baffled by the huge things, it might be a good idea to popularize it.

I think jimrandomh is slightly too harsh about Flare, the idea of using a pattern-matching object database as the foundation of a language rather than a bolted-on addition is at least an interesting concept. However, it seems like Eliezer focused excessively on bizarre details like supporting HTML in code comments, and having some kind of reference counting garbage collection which would be unlike anything to come before (even though the way he described it sounded pretty much exactly like the kind of reference counting GC that had been in use for decades), and generally making grandiose, highly detailed plans that were mostly impractical and/or far too ambitious for a small team to hope to implement in anything less than a few lifetimes. And then the whole thing was suddenly abandoned unfinished.

I've looked at the Flare docs and been similarly unimpressed. Most of that is hindsight bias - knowing that the project remained (that I'm aware of) at the vaporware stage without delivering an actual language.

Some of the proposed language features are indeed attractive; the existing language that most closely resembles it is Javascript, which shares with LambdaMOO (mentioned in the Flare docs) the interesting feature of prototype inheritance ("parenting").

Part of the negative impression comes from the docs being a catalog of proposed features, without a clear explanation of how each of those features participates in a coherent whole; it comes across as a "kitchen sink" approach to language design. Using XML as an underlying representation scheme being the most grating instance. The docs are long on how great Flare will be but short on programs written in Flare itself illustrating how and why the things you can do with Flare would be compelling to a programmer with a particular kind of problem to solve.

To give you an idea of my qualifications (or lack thereof) for evaluating such an effort: I'm an autodidact; I've never designed a new language, but I have fair implementation experience. I've written a LambdaMOO compiler targeting the Java VM as part of a commercial project (shipped), and attempted writing a Java VM in Java (never shipped, impratical without also writing a JIT, but quite instructive). That was back in 1998. These projects required learning quite a bit about language design and implementation.

It's harder to comment on Eliezer's other accomplishments - I'm rather impressed by the whole conceptual framework of FAI and CEV but it's the kind of thing to be judged by the detailed drudge work required to make it all work afterward, rather than by the grand vision itself. I'm impressed (you have to be) with the AI box experiments.

Also, Drescher points out a particular error that DBDT makes: in Newcomb's problem, if Omega chooses the contents of the box before the agent is born, the agent will two-box.

The actual objection was:

I don't think DBDT gives the right answer if the predictor's snapshot of the local universe-state was taken before the agent was born (or before humans evolved, or whatever), because the "critical point", as Fisher defines it, occurs too late.

Surely, as I pointed out at the time, the author already covered that in the paper. See this bit:

For now, let us take it for granted that, in short-duration scenarios like Newcomb’s problem and the psychologically-similar prisoners’ dilemma, the critical point comes prior to the first events mentioned in standard descriptions of these scenarios. (See Figure 1.)

...and this bit:

The critical point in Newcomb’s problem comes prior to the visit to the predictor.

Yudkowsky's objection is based on the same mistake. He says:

there's still a possibility that DBDT will end up two-boxing if Omega takes a snapshot of the (classical) universe a billion years ago before DBDT places the "critical point"

...but this directly contradicts what it says in the paper about where that point is located:

the critical point must come before the contents of the opaque box are determined.

...and...

the critical point comes prior to the first events mentioned in standard descriptions of these scenarios.

I meant more: does it make any decisions differently.

It certainly seems like a rather similar perspective. It was published back in 2002.

Anyway, the point is to find out if a transhuman AI would mind-control the operator into letting it out. Eliezer is smart, but is no transhuman (yet). If he got out, then any strong AI will.

Minor emendation: replace "would"/"will" above with "could (and for most non-Friendly goal systems, would)".

EY's point would be even stronger if transcripts were released and people still let him out regularly.

Why "fooled"? Why assume the AI would have duplicitous intentions? I can imagine an unfriendly AI à la "Literal Genie" and "Zeroth Law Rebellion", but an actually malevolent "Turned Against Their Masters" AI seems like a product of the Mind Projection Fallacy.

Downvoting Roman strikes again!