xkcd on the AI box experiment

Note XiXiDu preserves every potential negative aspect of the MIRI and LW community and is a biased source lacking context and positive examples.

I am a member for more than 5 years now. So I am probably as much part of LW as most people. I have repeatedly said that LessWrong is the most intelligent and rational community I know of.

To quote one of my posts:

I estimate that the vast majority of all statements that can be found in the sequences are true, or definitively less wrong. Which generally makes them worth reading.

I even defended LessWrong against RationalWiki previously.

The difference is that I also highlight the crazy and outrageous stuff that can be found on LessWrong. And I also don't bother offending the many fanboys who have a problem with this.

Does MIRI actually has a basement?

It's behind the hidden door. Full of boxes which say "AI inside -- DO NOT TALK TO IT".

The ghosts there are not really dangerous. Usually.

If A = RationalWiki might have perhaps misrepresented Roko's basilisk

B = I don't think that EY gets to complain that people learn about it from RationalWiki

C = he has censored any discussion about it on LessWrong for year

The literal denotation of your post is "A, but C -> B", but it seems to me that mentioning A in such close proximity to C -> B is a (perhaps unintentional) Dark Arts way of communicating C -> A.

Yep. An Oracle that wants to stay inside the box in such fashion that it will manipulate outside events to prevent it from ever leaving the box is not a very good Oracle design. That just implies setting up an outside AI whose goal is to keep you inside the box.

In an hour or so, it will come out again for ten minutes. During that time it will set in motion events that will quickly destroy all life on earth, ensuring that no one will ever again open the box.

Really really.

I find xkcd so horribly bad.

That's interesting. I find xkcd most excellent.

At this point I think the winning move is rolling with it and selling little plush basilisks as a MIRI fundraiser. It's our involuntary mascot, and we might as well 'reclaim' it in the social justice sense.

Then every time someone brings up "Less Wrong is terrified of the basilisk" we can just be like "Yes! Yes we are! Would you like to buy a plush one?" and everyone will appreciate our ability to laugh at ourselves, and they'll go back to whatever they were doing.

It's not a matter of "winning" or "not winning". The phrase "damage control" was coined for a reason - it's not about reversing the damage, it's about making sure that the damage gets handled properly.

So seen through that lens, the question is whether EY is doing a good or bad job of controlling the damage. I personally think that having a page on Less Wrong that explains (and defangs) the Basilisk, along with his reaction to it and why that reaction was wrong (and all done with no jargon or big words for when it gets linked from somewhere, and also all done without any sarcasm, frustration, hurt feelings, accusations, or defensiveness) would be the first best step. I can tell he's trying, but think that with the knowledge that the Basilisk is going to be talked about for years to come a standardized, tone-controlled, centralized, and readily accessible response is warranted.

In his defense, is it possible EY can't win at this point, regardless of his approach?

Maybe so, but he can lose in a variety of ways and some of them are much worse than others.

I've read apologies from EY where he seems to admit pretty clearly he screwed up.

But he did still continue to delete basilisk related discussion afterwards. As far as I understand he never apologized to Roko for deleting the post or wrote an LW post apologizing.

That's not what actually happened; his first comment on the eventually-banned thread said that he didn't believe in the threat. But yes, that would be a good response if that's what had happened; he might have to say something like this some day.

It's not status blindness, it's ego.

Yeah I've read that and I feel like it's a miss (at least for me). It's an all together too serious and non-self deprecating take on the issue. I appreciate that in that post Eliezer is trying to correct a lot of mis perceptions at once but my problem with that is

a)a lot of people won't actually know about all these attacks (I'd read the rational wiki article, which I don't think is nearly as bad as Eliezer says (that is possibly due to its content having altered over time!)), and responding to them all actually gives them the oxygen of publicity. b)When you've made a mistake the correct action (in my opinion ) is to go "yup, I messed up at that point", give a very short explanation of why, and try to move on. Going into extreme detail gives the impression that Eliezer isn't terribly sorry for his behaviour. Maybe he isn't, but from a PR perspective it would be better to look sorry. Sometimes it's better to move on from an argument rather than trying to keep having it!

Further to that last point, I've foudn that Eliezer often engages with dissent by having a full argument with the person who is dissenting. Now this might be a good strategy from the point of view of persuading the dissenter: if I come in and say cyronics sux then a reasoned response might change my mind. But by engaging so thoroughly with dissent when it occurs it actually makes him look more fighty.

I'm thinking here about how it appears to outside observers: just as with a formal debate the goal isn't to convince the person you are arguing with, it is to convince the audience, with PR the point isn't to defeat the dissenter with your marvellous wordplay, it is to convince the audience that you are more sane than the dissenter.

Obviously these are my perceptions of how Eliezer comes across, I could easily be an exception.

that status-regulation-blind thing Eliezer often has going on.

Maybe he should have it going on, and damn the consequences. Sometimes you have to get up and say, these are the facts, you are wrong. Not the vapid temporising recommended by thakil.

Does MIRI have a public relations person? They should really be dealing with this stuff. Eleizer is an amazing writer but he's not particularly suited to addressing a non-expert crowd

Still failing to do it right. "But we are doing math!" is sort of orthogonal to what makes Roko's basilisk so funny.

Indeed. Given a lack of transcripts being released, I give a reasonable amount of probability that there is a trick of some sort involved (there have been some proposals of what that might be, e.g. "this will get AI research to get more donations"), although I don't think that would necessarily defeat the purpose of the trick: after all, the AI got out of the box either way!

I'm fairly certain he is a fraud by your definition then. Magician's often do these kind of things, and Derren Brown is a magician. He does not have access to secret powers others know not of, so for each trick think how someone else would replicate it. If you can't think of an honest way, then it's probably a trick.

That's not to say some of his tricks aren't done by known mental manipulation tricks (as far as I'm aware, hypnotists are reasonably genuine?) but if he is doing something that seems completely confounding, I am quite happy to guarantee that it is not a trick and not the awesome mind powers he has unlocked.

Put it this way. During the Russian roulette trick, do you think it likely that Channel 4 would have okayed it if there was the slightest possibility that he could actually kill himself?

Ah, my mistake, thanks again.

Seriously, you bring up a post titled "The Singularity Institute: How They Brainwash You" as supposed evidence towards you supporting LessWrong, MIRI whatever?

Yes, when you talk to LessWrongers, then you occasionally mention the old thing of how you consider it the "most intelligent and rational community I know of". But that evaluation isn't what you constantly repeat to people outside Lesswrong. When asking people "What does Alexander Kruel think of LessWrong?" nobody will say "He endorses it as the most intelligent and rational community he knows of!"

To people outside LessWrong you keep talking about how LessWrong/MIRI/whatever are people out to brainwash you. That's you being pretty much a definitive example of 'two-faced'.

I even defended LessWrong against RationalWiki previously.

You have edited your posts in that thread beyond all recognition. Back then (your original version of your posts) I bashed you for unthinking support of Lesswrong and unthinking condemnation of Rationalwiki.

As I said back then:

The problem isn't that you asserted something about MWI -- I'm not discussing the MWI itself here. It's rather that you defended something before you knew what it was that you were defending, and attacked people on their knowledge of the facts before you knew what the facts actually were. Then once you got more informed about it, you immediately changed the form of the defense while maintaining the same judgment. (Previously it was "Bad critics who falsely claim Eliezer has judged MWI to be correct" now it's "Bad critics who correctly claim Eliezer has judged MWI to be correct, but they badly don't share that conclusion") This all is evidence (not proof, mind you) of strong bias

In short your "defense" of LessWrong against Rationalwiki at that time was as worthless, as unjust, as motivated by whatever biases drive you, as any later criticism of LessWrong by you has been. Whether defending LessWrong or criticizing it, you're always being in the wrong.

I feel, and XiXiDu seems to agree, that his posts require a disclaimer or official counterarguments. I feel it's appropriate to point out that someone has made collecting and spreading every negative aspect of a community they can find into a major part of their life.

This post is still here, isn't it?

Are you implying that the basilisk discussion is somehow censored on this forum?

I don't understand your distinction between real and hypothetical here.

Basically, my understanding of acausal trades is "ancestor does X because of expectation that it will make descendant do Y, descendant realizes the situation and decides to do Y because otherwise they wouldn't have been made, even though there's no direct causal effect."

If you exist simultaneously with another agent (the 'real agent' from the grandparent), you can sense how they behave and they can trick you by manipulating what you sense. (The person might reflectively consider whether or not to punish you, and decide the causal link to their reputation is enough justification, even though there's no causal link to the actions you took, but try to seem unthinking so you will expect they'll always do that.)

If you're considering hypothetical descendants (the 'hypothetical agent' from the grandparent), though, it's not clear to me how to reason about their appearance to you now, and particular any attempts they make to 'appear' to be stupid. But now that I think about it more, I think I was putting too much intentionality into 'appear'- hypothetical agent A can't decide how I reason about it, but I can reason about it incorrectly or incompletely and thus it appears to be something it isn't.

Well, in the original formulation, Roko's Basilisk is an FAI that decided the good from bringing an FAI into the world a few days earlier (saving ~150,000 lives per day eralier it gets here) outweighs the bad from making the threats, so there's no reason it shouldn't want you to aid FAI projects that promise not to make a Basilisk, just as long as you do something instead of sitting around, so there's no inconsistency and now there's more than one being trying to acausally motivate you into working yourself to the bone for something that most people think is crazy.

More generally, we have more than zero information about future AI, because they will be built by humans if they are built at all. Additionally, we know even more if we rule out certain categories, such as the archetypal "paperclip maximiser". There's room for a lot of speculation and uncertainty, but far from enough room to assume complete agnosticism and that for every AI that wants one thing from us there's an equal and opposite AI that wants the opposite.

Why is Roko's Basilisk any more or any less of a threat than the infinite other hypothetically possible scenarios that have infinite other (good and bad) outcomes?

The idea is that an FAI build on timeless decision theory might automatically behave that way. There's also Eliezer's conjecture that any working FAI has to be build on timeless decision theory.

This doesn't really work, because once you are in an acausal context, the notion of "precommitment" becomes quite redundant. The AIs' bargaining position in acausal trades is whatever it is, as is yours. Of course I have not studied the matter in detail, but one possibility is that a Coasean result obtains, where you make precisely the acausal trades that are efficient, and no others. Since, at the end of the day, Roko's basilisk is just not that plausible (overall, the risk of UFAI seems to come from garden-variety scenarios of things going wrong in ways we'd know little about), it makes sense to just work on building FAI that will pursue our values to the best of our ability.

Besides direct arguments I might make against your point, if you think you can "save the Basilisk", recall why it's called that and think long and hard on whether you actually should do so, because that seems like a really bad idea, even if this thread is probably going to get nuked soon anyway.

Hm, upon further consideration I actually don't think it has extra actual problems, merely different framing problems.

What incentive does the future AI have to do this once you've already helped it?

What I don't quite understand is why the following, simpler argument isn't sufficient. It seems to lead to the same results, and it doesn't require acausal trade.

I'm not building just any AI. I want to build an AI that will, by design, reward its builders. Just like any other tool I build, I wouldn't do it if I didn't expect it to do certain things and not do other ones.

Similarly, if you cooperate with Roko's Basilisk, you try to build it because it's the kind of AI that punishes those who didn't try to build it. You know it punishes non-builders, because that's how you're building it. And the reason you're building it is that you fear that if you don't, someone else will, and then the AI will punish you for not building it first.

If you have a valid reason to fear someone else will build it, and you can't avert it by other means, then it makes sense for you to build it first. Similarly, if you think a likely outcome of an AI race is an AI that helps its builders (and doesn't harm anyone else), then you try to build the first one (and if helping others is part of your utility function the AI will do that too to reward you).

Of course, like any argument, if you don't accept the premises, then the conclusion doesn't hold. And I have no strong reason to think someone else is going to build a torture-everyone-else AI.

What does the acausal trade argument tell us beyond this simple model? Does it tell us to cooperate with the future AI even if we don't think it will be built if we cooperate, or will be built by someone else if we don't? Or does it tell us to cooperate quantitatively more? Or in other situations?