Tallinn-Evans $125,000 Singularity Challenge

Okay, let's think this through with a particular case.

Assume only your two factors: John has a reliable source of income and overdraft protection on an account. Since you assert that those two factors are sufficient, we can suppose John doesn't have any line of credit, doesn't own anything valuable that could be converted to cash, doesn't know anybody that could give him a loan or a job, etc

John donates all his savings, and loses his job the next day. He has overdraft protection on his empty bank account, which will save him from some fees when he starts bouncing checks, but the overdraft protection will expire pretty quickly once checks start bouncing.

Things will spiral out of control quickly unless John is able to get another source of income sufficient to cover his recurring expenses or there is some other compensating factor than the two you mentioned (which shows they are not sufficient). Or do you think he's doing okay a month later when the overdraft protection is no longer in effect, he has tons of bills due, needs to pay his rent, still hasn't found a job, has run out of food, etc.? And if he hasn't found work within a few months more -- which is quite possible -- he'll be evicted from his home and his credit will be ruined from not having paid any of his bills for several months.

ETA: the point isn't that all of that will happen or is even likely to happen, but that a bank account represents some amount of time that the person can stay afloat while they're looking for work. It greatly increases the likelihood that they will find a new source of income before they hit the catastrophe point of being evicted and having their credit ruined.

No, I definitely meant size, not need (or effectiveness or quality of goals or anything else). A larger charity can mount more effective campaigns than a smaller one. This is from the Iron Law of Institutions perspective, in which charities are blobs for sucking in money from a more or less undifferentiated pool of donations. An oversimplification, but not too much of one, I fear - there's a reason charity is a sector in employment terms.

How does everyone splitting donations go against the interests of the neediest charities, if we don't have an efficient market in philanthropy and the lumped donations would have gone to the most popular (hypothetically = largest) charities rather than the neediest?

Or did you interpret "splitting donations" as referring to something other than everyone doing so?

If everyone splitting donations were against the interests of the neediest charities, wouldn't that imply that we did have an efficient market in philanthropy — that the lumped donations would have gone to the neediest charities, rather than the most popular (hypothetically = largest)?

In this case, a decision to donate 100% of your money to SIAI would mean that all those people implementing a decision process sufficiently similar to your own would donate 100% of their money to SIAI. A decision to donate 50% of your money to SIAI and 50% to Charity Option B would imply a similar split for all those people as well.

This argument assumes that the people using a similar decision process are faced with the same evidence. In particular, if they made their decision significantly later then they would know about your donation (not directly, but if SIAI now had significantly more funds they could know about it).

If all decision makers were perfectly rational and omniscient, but didn't have to make their decisions at the same time, then you wouldn't expect to see the 50/50 splitting. You would expect everyone to donate to the charity for which the current marginal usefulness is greatest. In the situation you envision, the marginal usefulness would decrease over time, until eventually donors would notice that it was no longer the best option, and then start diverting their funding. Perhaps once this sort of equilibrium is reached splitting your money is advisable, but we are extremely unlikely to be anywhere near such an equilibrium (with respect to my personal values) unless there is an explicit mechanism pushing us towards it. This would probably require postulating a lot of brilliant rational donors with identical values.

Sweet, I can still be the one to push it over! [1]

[1] so long as you disregard the fungibility of money and therefore my contribution's indistinguishability from that of all the others.

Thanks. How long until a donation is reflected in the picture? Is it possible the 125k goal is already met?

Eliezer has really got to do something about his fictional villains escaping into real life. First Clippy, now you too?

Posts in which people announce that they have changed their mind are usually upvoted

As a total newbie to this site, I applaud this sentiment, but have just gone through an experience where this has not, in fact, happened.

After immediately retracting my erroneous statement (and explaining exactly where and why I'd gone wrong), I continued to be hammered over arguments that I had not actually made. My retracted statements (which I've left in place, along with the edits explaining why they're wrong) stay just as down-voted as before...

My guess is that some of the older members of this site may realise that this is how it's supposed to work... but it's certainly not got through to us newbies yet ;)

Perhaps it should be added to the etiquette section in the newbie pages (eg the karma-section in the FAQ) ?

Translation: I haven't managed to convince you therefore you must be punished for your insolent behaviour of not being convinced by my arguments. I cannot walk away from this and leave you being wrong, you must profess to agree with me and if you are not rational enough to understand and accept logical arguments then you will be forced to profess agreement.

Who did you say hasn't learned how to lose?

I'm beginning to think that LW needs some better mechanism for dealing with the phenomenon of commenters who are polite, repetitive, immune to all correction, and consistently wrong about everything. I know people don't like it when I say this sort of thing, but seriously, people like that can lower the perceived quality of a whole website.

Warn, then ban the people involved.

If you decide that refusing to be convinced by evidence while also unable to convincingly counter it, and at the same time continuing to argue is bad form for the LW that you want to create, then stand by that decision and act on it.

Therefore, I propose adding a "plonk" button on each comment.

From the username, I was expecting that the suggestion was going to be to say avada kedavra.

and a form letter reminder that self-consistency is not a virtue [..] making it clear that this community's respect is contingent on [..]

Is changing professed beliefs to something else without understanding / agreeing with the new position, but just doing it to gain community respect, a virtue?

Tim, and the rest of the class of commenters to which you refer, simply haven't learned how to lose.

Or still isn't convinced that he is wrong by the time you have passed your tolerance of explaining so you give up and decide he must be broken. Your proposed 'solution' is a hack so you can give up on convincing him but still have him act convinced for the benefit of appearances - maybe you are simply expecting far far too short inferential distances?

Escalating punishment so someone "learns better" can work, but it requires real punishments, not symbolic ones. It's not clear to me that "plonking" would accomplish that.

And, of course, it has all the same problems that punishment-based behavior modification always has.

...self-consistency is not a virtue and a short guide to losing gracefully.

As long as you are no fooming FAI...

online communities, being largely by and for geeks, dislike overt exclusionary tactics because it brings up painful associations. I think well established communities often have more to gain from elitism than they stand to lose.

These two statements are contradictory. Did you swap "gain" and "lose" in the second statement?

One solution is to try and convince people to downvote more aggressively.

Or at least to downvote established users more aggressively, given that new people have said they felt intimidated.

Alternatively just allow people to have an "ignored users" file. You can "click to ignore this user" on anybody that you find to be continuously less worthwhile on average.

Or, even better, you can apply a "handicap" to certain people. eg that you will only view comments by a certain person if the comment has been upvoted to at least 4 (or whatever).

Another solution is to change the karma system to remove the psychological obstacles that may keep people from downvoting. It feels a little mean to directly cause a comment to be filtered, even when it would probably improve the mean quality of discourse. It may be a little easier to express your opinion that a comment is not constructive, and have a less direct mechanism responsible for converting a consensus into moderation / karma penalty.

I do this by keeping kibitzing turned off most of the time, and always showing all comments regardless of karma. This won't work for everyone, but it works for me: I think my upvotes and downvotes are less biased this way.

Something I've thought about (in the context of other venues) is a rating system where vote-totals V are stored but not displayed. Instead what gets displayed is a ranking R, where R=f(V)... a comment utility function, of sorts. That way a single vote in isolation does not cause a state transition.

The same function can apply to user "scores." Which may reduce the inclination to stress about one's karma, if one is inclined to that.

To pick a simple example, suppose (R=V^.37). So 1 upvote gets R1, but a second upvote stays R1. The 3rd upvote gets R2; the 12th gets R3; the 30th gets R4. Downvoting an R2 comment might bring it to R1, or it might not. Eliezer is R68, Yvain is R45, Alicorn is R37, cousin_it and wei_dai and annasolomon are all R30, I'm R13, and so forth.

(The specific function is just intended for illustrative purposes.)

The function needn't be symmetrical around zero... e.g., if R=(-1*ABS(V)^.37) when V<0, rankings go negative faster than they go positive.

====

Along a different axis, it might be interesting to allow voters to be nymous... for example, a preferences setting along those lines, or a mechanism for storing notes along with votes if one chooses to, visible on a different channel. The idea being that you can provide more detailed feedback without cluttering the comment-channel with meta-discussions about other comments.

Another solution is to change the karma system to remove the psychological obstacles that may keep people from downvoting.

The way Hacker News does this is by not displaying comment scores that are below the filter threshold.

I personally haven't downvoted Tim, although I now feel like I ought to have, simply because it felt like bad etiquette to downvote someone else while in a sustained argument with them, even if you feel like they're engaging in bad reasoning. I should probably be more liberal with my downvotes in future than I have been.

If a person is making positive contributions to the board with some regularity though, I think it's worth having them around even if they are also frequently making negative contributions. At least the karma system gives people a mechanism to filter posts so that they can ignore the less worthwhile ones if they so choose.

Thanks. I'm not sure votes have that much to do with being right, though. My perception is more that people vote up things they like seeing - and vote down things they don't. It sometimes seems more like applause.

Look at his actual comments.

I'm not sure I understand what you mean. Would you mind explaining?

This posting above, which begins with an argument that is absolutely silly, managed to receive 11 votes.

Envy is unbecoming; I recommend against displaying it. You'd be better off starting with your 3rd sentence and cutting the word "silly."

I would ask this question: what comparison did donors make to decide the Singularity Institute is a better recipient than the one mentioned in Yvain's preceding entry, where each $500 saves a human life.

They have worked out this math, and it's available in most of their promotional stuff that I've seen. Their argument is essentially "instead of operating on the level of individuals, we will either save all of humanity, present and future, or not." And so if another $500 gives SIAI an additional 1 out of 7 billion chance of succeeding, then it's a better bet than giving $500 to get one guaranteed life (and that only looks at present lives).

The question as to whether SIAI is the best way to nudge the entire future of humanity is a separate question from whether or not SIAI is a better bet than preventing malaria deaths. I don't know if SIAI folks have made quantitative comparisons to other x-risk reduction plans, but I strongly suspect that if they have, a key feature of the comparison is that if we stop the Earth from getting hit by an asteroid, we just prevent bad stuff. If we get Friendly AI, we get unimaginably good stuff (and if we prevent Unfriendly AI without getting Friendly AI, we also prevent bad stuff).

Numerous posters wouldn't pick this particular charity, even if it happened to be among the best, unless they were motivated by signaling aspirations rather than the rational choice of the best recipient.

You know this is a blog started by and run by Eliezer Yudkowsky - right? Many of the posters are fans. Looking at the rest of this thread, signaling seems to be involved in large quantities - but consider also the fact that there is a sampling bias.

Do you have any argument for why the SIAI is unlikely to be the best other than the sheer size of the option space?

This is a community where a lot of the members have put substantial thought into locating the optimum in that option space, and have well developed reasons for their conclusion. Further, there are not a lot of real charities clustered around that optimum. Simply claiming a low prior probability of picking the right charity is not a strong argument here. If you have additional arguments, I suggest you explain them further.

(I'll also add that I personally arrived at the conclusion that an SIAI-like charity would be the optimal recipient for charitable donations before learning that it existed, or encountering Overcoming Bias, Less Wrong, or any of Eliezer's writings, and in fact can completely discount the possibility that my rationality in reaching my conclusion was corrupted by an aura effect around anyone I considered to be smarter or more moral than myself.)

It is obvious that a number of smart people have decided that SIAI is currently the most important cause to devote their time and money to. This in itself constitutes an extremely strong form of evidence. This is, or at least was, basically Eliezer's blog; if the thing that unites its readers is respect for his intelligence and judgment, then you should be completely unsurprised to see that many support SIAI. It is not clear how this is a form of irrationality, unless you are claiming that the facts are so clearly against the SIAI that we should be interpreting them as evidence against the intelligence of supporters of the SIAI.

Someone who is trying to have an effect on the course of an intelligence explosion is more likely to than someone who isn't. I think many readers (myself included) believe very strongly that an intelligence explosion is almost certainly going to happen eventually and that how it occurs will have a dominant influence on the future of humanity. I don't know if the SIAI will have a positive, negative, or negligible influence, but based on my current knowledge all of these possibilities are still reasonably likely (where even 1% is way more than likely enough to warrant attention).

Of course, the Singularity argument in no way relies on nanotech.

Without advanced real-world nanotechnology it will be considerable more difficult for an AI to FOOM and therefore pose an existential risk. It will have to make use of existing infrastructure, e.g. buy stocks of chip manufactures and get them to create more or better CPU's. It will have to rely on puny humans for a lot of tasks. It won't be able to create new computational substrate without the whole economy of the world supporting it. It won't be able to create an army of robot drones overnight without it either.

Doing so it would have to make use of considerable amounts of social engineering without its creators noticing it. But more importantly it will have to make use of its existing intelligence to do all of that. The AGI would have to acquire new resources slowly, as it couldn't just self-improve to come up with faster and more efficient solutions. In other words, self-improvement would demand resources, therefore the AGI could not profit from its ability to self-improve regarding the necessary acquisition of resources to be able to self-improve in the first place.

So the absence of advanced nanotechnology constitutes an immense blow to any risk estimates including already available nanotech. Further, if one assumes that nanotech is a prerequisite for AI going FOOM then another question arises. It should be easier to create advanced replicators to destroy the world than creating AGI that then creates advanced replicators that then fails hold and then destroys the world. Therefore one might ask what is the bigger risk here.

"The Singularity argument"? What's that, then?

One of my main problems regarding risks from AI is that I do not see anything right now that would hint at the possibility of FOOM.

I see foom as a completely separate argument from FAI or AGI or extinction risks. Certainly it would make things more chaotic and difficult to handle, increasing risk and uncertainty, but it's completely unnecessary for chaos, risk, and destruction to occur - humans are quite capable of that on their own.

Once an AGI is "out there" and starts getting copied (assuming no foom), I want to make sure they're all pointed in the right direction, regardless of capabilities, just as I want that for nuclear and other weapons. I think there's a possibility we'll be arguing over the politics of enemy states getting an AGI. That doesn't seem to be a promising future. FAI is arms control, and a whole lot more.

We have had quite a bit of self-improvement so far - according to my own:

http://www.alife.co.uk/essays/the_intelligence_explosion_is_happening_now/

Progress is accelerating in a sense - due to synergy between developments making progress easier.

...when progress is at its fastest, things might well get pretty interesting. Much in the way of "existential" risk seems pretty unlikely to me - but with speculative far-future events, it is hard to be certain.

What does look as though it will go up against the wall are unmodified human beings - and other multicellular DNA-based lifeforms. There is no way these can compete against engineering and intelligent design - and they look like an unsuitable foundation for building directly on top of.

Some will paint that as an apocalypse - though to me it looks like a sensible, obvious and practically inevitable move. The most reasonable hope for the continued existence of biological humans is in future equivalents of museums and historical simulations - IMO.

the fervor in this thread seemed to me completely unjustified. [...] My main problem is how far I should go to neglect other problems in favor of some high-impact low-probability event.

I agree with SIAI's goals. I don't see it as "fervor". I see it as: I can do something to make this world a better place (according to my own understanding, in a better way than any other possible), therefore I will do so.

I compartmentalize. Humans are self-contradictory in many ways. I can send my entire bank account to some charity in the hopes of increasing the odds of friendly AI, and I can buy a hundred dollar bottle of bourbon for my own personal enjoyment. Sometimes on the same day. I'm not ultra-rational or pure utilitarian. I'm a regular person with various drives and desires. I save frogs from my stairwell rather than driving straight to work and earning more money. I do what I can.

One should seriously consider if there are other groups worthy of promotion or if there should be other groups doing the same as the SIAI or being dealing with one of its sub-goals.

I have seriously considered it. I have looked for such groups, here and elsewhere, and no one has ever presented a contender. That's why I made my question as simple and straightforward as possible: name something more important. No one's named anything so far, and I still read for many hours each week on this and other such topics, so hopefully if one arises, I'll know and be able to evaluate it.

I donate based on relative merit. As I said at the end of my original supporting post: so far, no one else seems to come close to SIAI. I'm comfortable with giving away a large portion of my income because I don't have much use for it myself. I post it here because it encourages others to give of themselves. I think it's the right thing to do.

I know it's hard to see why. I wish they had better marketing materials. I was really hoping the last challenge, with projects like a landing page, a FAQ, etc., would make a difference. So far, I don't see much in the way of results, which is upsetting.

I still think it's the right place to put my money.

Well, bypassing the issue of FOOMingness, I am pretty sure that machine intelligence represents an upcoming issue that could go better or worse than average - and which humanity should try and steer in a positive direction - x-risk or no.

My concerns about the SIAI are mostly about their competence. It seems rather easy for me to imagine another organisation in the SIAI's niche doing a much better job. Are 63 chapters of a Harry Potter fanfic really helping, for instance?

Also, if they think using fear of THE END OF THE WORLD is a good way to stimulate donations, I would be very interested to see information about the effect on society of such marketing. Will it produce a culture of fear? What about The risks of caution?

My general impression is that spreading the DOOM virus around is rarely very constructive. It may well be actively harmful. In financial markets, prophesying market crashes may actually help make them happen, since the whole system works like a big rumour mill - and if a crash is coming, it makes sense to cash in and buy gold - and, if everyone does that, then the crash happens. A case of the self-fulfilling prophesy. The prophet may look smug - but if only they had kept their mouth shut!

Have the DOOM merchants looked into this kind of thing? Where are their reassurances that prophesying DOOM - and separating passing punters from their cash in the process - is a harmless pass-time, with no side effects?

I am pretty sure that the "FOOM" term is an attempt to say something about the timescale of the growth of machine intelligence. So, I am sceptical about definitions which involve the concept of trickery. Surely rapid growth need not necessarily involve trickery. My FOOM sources don't seem to mention trickery. Do you have any references relating to the point?

Just to clarify: are you asserting that this comment, and the associated post about the size of mindspace, represent the "convince even a skeptic" collection of evidence you were alluding to in its grandparent (which XiXiDu quotes)?

Or was there a conversational disconnect somewhere along the line?

This doesn't address the most controversial aspect, which is that AI would go foom. If extreme fooming doesn't occur this isn't nearly as big an issue. That is an issue where many people have discussed it and not all have come away convinced. Robin Hanson had a long debate with Eliezer over this and Robin was not convinced. Personally, I consider fooming to be unlikely but plausible. But how likely one thinks it is matters a lot.

Your post right here seems like a good example. You could say something along the lines of "This is a dismissal, not an argument; merely naming a bias isn't enough to convince me. If you provide some specific examples, I'd be happy to listen and respond as best as I can." You can even tack on an "But until then, I'm downvoting this because it seems like it's coming from hostility rather than a desire to find the truth together."

Heck, you could even copy that and have it saved somewhere as a form response to comments like that.

It wasn't an argument, it was a dismissal.

I noticed the tendency on LW to portray comments as attacks. They may seem that way to trained rationalists and otherwise highly educated folks. But not every negative comment is actually intended to be just a rhetorical device or simple dismissal. It won't help if you just downvote people or call them logical rude. Some people are honestly interested but fail to express themselves adequately. Usually newcomers won't know about the abnormally high standards on LW. You have to tell them about it. You also have to take into account those who are linked to this post, or come across it by other means, who don't know anything about LW. How does this thread appear to them, what are they likely to conclude, especially if no critical comment is being answered kindly but simply downvoted or snidely rejected?

This is the internet, though; skins are assumed to be tough. There is some benefit to saying "It looks like you wanted to say 'X'. Please try to be less nasty next time. Here's why I don't agree with X" instead of just "wow, you're nasty."