Thoughts on the Singularity Institute (SI)

post by HoldenKarnofsky · 2012-05-11T04:31:30.364Z · score: 271 (285 votes) · LW · GW · Legacy · 1284 comments

Contents

      Summary of my views
      Intent of this post
  Does SI have a well-argued case that its work is beneficial and important?
    Objection 1: it seems to me that any AGI that was set to maximize a "Friendly" utility function would be extraordinarily dangerous.
    Objection 2: SI appears to neglect the potentially important distinction between "tool" and "agent" AI.
    Objection 3: SI's envisioned scenario is far more specific and conjunctive than it appears at first glance, and I believe this scenario to be highly unlikely.
    Other objections to SI's views
    Wrapup
  Is SI the kind of organization we want to bet on?
    Wrapup
  But if there's even a chance …
  Existential risk reduction as a cause
  How I might change my views
  Acknowledgements
None
1284 comments

This post presents thoughts on the Singularity Institute from Holden Karnofsky, Co-Executive Director of GiveWell. Note: Luke Muehlhauser, the Executive Director of the Singularity Institute, reviewed a draft of this post, and commented: "I do generally agree that your complaints are either correct (especially re: past organizational competence) or incorrect but not addressed by SI in clear argumentative writing (this includes the part on 'tool' AI). I am working to address both categories of issues." I take Luke's comment to be a significant mark in SI's favor, because it indicates an explicit recognition of the problems I raise, and thus increases my estimate of the likelihood that SI will work to address them.

September 2012 update: responses have been posted by Luke and Eliezer (and I have responded in the comments of their posts). I have also added acknowledgements.

The Singularity Institute (SI) is a charity that GiveWell has been repeatedly asked to evaluate. In the past, SI has been outside our scope (as we were focused on specific areas such as international aid). With GiveWell Labs we are open to any giving opportunity, no matter what form and what sector, but we still do not currently plan to recommend SI; given the amount of interest some of our audience has expressed, I feel it is important to explain why. Our views, of course, remain open to change. (Note: I am posting this only to Less Wrong, not to the GiveWell Blog, because I believe that everyone who would be interested in this post will see it here.)

I am currently the GiveWell staff member who has put the most time and effort into engaging with and evaluating SI. Other GiveWell staff currently agree with my bottom-line view that we should not recommend SI, but this does not mean they have engaged with each of my specific arguments. Therefore, while the lack of recommendation of SI is something that GiveWell stands behind, the specific arguments in this post should be attributed only to me, not to GiveWell.

Summary of my views

Intent of this post

I did not write this post with the purpose of "hurting" SI. Rather, I wrote it in the hopes that one of these three things (or some combination) will happen:

  1. New arguments are raised that cause me to change my mind and recognize SI as an outstanding giving opportunity. If this happens I will likely attempt to raise more money for SI (most likely by discussing it with other GiveWell staff and collectively considering a GiveWell Labs recommendation).
  2. SI concedes that my objections are valid and increases its determination to address them. A few years from now, SI is a better organization and more effective in its mission.
  3. SI can't or won't make changes, and SI's supporters feel my objections are valid, so SI loses some support, freeing up resources for other approaches to doing good.

Which one of these occurs will hopefully be driven primarily by the merits of the different arguments raised. Because of this, I think that whatever happens as a result of my post will be positive for SI's mission, whether or not it is positive for SI as an organization. I believe that most of SI's supporters and advocates care more about the former than about the latter, and that this attitude is far too rare in the nonprofit world.

Does SI have a well-argued case that its work is beneficial and important?

I know no more concise summary of SI's views than this page, so here I give my own impressions of what SI believes, in italics.

  1. There is some chance that in the near future (next 20-100 years), an "artificial general intelligence" (AGI) - a computer that is vastly more intelligent than humans in every relevant way - will be created.
  2. This AGI will likely have a utility function and will seek to maximize utility according to this function.
  3. This AGI will be so much more powerful than humans - due to its superior intelligence - that it will be able to reshape the world to maximize its utility, and humans will not be able to stop it from doing so.
  4. Therefore, it is crucial that its utility function be one that is reasonably harmonious with what humans want. A "Friendly" utility function is one that is reasonably harmonious with what humans want, such that a "Friendly" AGI (FAI) would change the world for the better (by human standards) while an "Unfriendly" AGI (UFAI) would essentially wipe out humanity (or worse).
  5. Unless great care is taken specifically to make a utility function "Friendly," it will be "Unfriendly," since the things humans value are a tiny subset of the things that are possible.
  6. Therefore, it is crucially important to develop "Friendliness theory" that helps us to ensure that the first strong AGI's utility function will be "Friendly." The developer of Friendliness theory could use it to build an FAI directly or could disseminate the theory so that others working on AGI are more likely to build FAI as opposed to UFAI.

From the time I first heard this argument, it has seemed to me to be skipping important steps and making major unjustified assumptions. However, for a long time I believed this could easily be due to my inferior understanding of the relevant issues. I believed my own views on the argument to have only very low relevance (as I stated in my 2011 interview with SI representatives). Over time, I have had many discussions with SI supporters and advocates, as well as with non-supporters who I believe understand the relevant issues well. I now believe - for the moment - that my objections are highly relevant, that they cannot be dismissed as simple "layman's misunderstandings" (as they have been by various SI supporters in the past), and that SI has not published anything that addresses them in a clear way.

Below, I list my major objections. I do not believe that these objections constitute a sharp/tight case for the idea that SI's work has low/negative value; I believe, instead, that SI's own arguments are too vague for such a rebuttal to be possible. There are many possible responses to my objections, but SI's public arguments (and the private arguments) do not make clear which possible response (if any) SI would choose to take up and defend. Hopefully the dialogue following this post will clarify what SI believes and why.

Some of my views are discussed at greater length (though with less clarity) in a public transcript of a conversation I had with SI supporter Jaan Tallinn. I refer to this transcript as "Karnofsky/Tallinn 2011."

Objection 1: it seems to me that any AGI that was set to maximize a "Friendly" utility function would be extraordinarily dangerous.

Suppose, for the sake of argument, that SI manages to create what it believes to be an FAI. Suppose that it is successful in the "AGI" part of its goal, i.e., it has successfully created an intelligence vastly superior to human intelligence and extraordinarily powerful from our perspective. Suppose that it has also done its best on the "Friendly" part of the goal: it has developed a formal argument for why its AGI's utility function will be Friendly, it believes this argument to be airtight, and it has had this argument checked over by 100 of the world's most intelligent and relevantly experienced people. Suppose that SI now activates its AGI, unleashing it to reshape the world as it sees fit. What will be the outcome?

I believe that the probability of an unfavorable outcome - by which I mean an outcome essentially equivalent to what a UFAI would bring about - exceeds 90% in such a scenario. I believe the goal of designing a "Friendly" utility function is likely to be beyond the abilities even of the best team of humans willing to design such a function. I do not have a tight argument for why I believe this, but a comment on LessWrong by Wei Dai gives a good illustration of the kind of thoughts I have on the matter:

What I'm afraid of is that a design will be shown to be safe, and then it turns out that the proof is wrong, or the formalization of the notion of "safety" used by the proof is wrong. This kind of thing happens a lot in cryptography, if you replace "safety" with "security". These mistakes are still occurring today, even after decades of research into how to do such proofs and what the relevant formalizations are. From where I'm sitting, proving an AGI design Friendly seems even more difficult and error-prone than proving a crypto scheme secure, probably by a large margin, and there is no decades of time to refine the proof techniques and formalizations. There's good recent review of the history of provable security, titled Provable Security in the Real World, which might help you understand where I'm coming from.

I think this comment understates the risks, however. For example, when the comment says "the formalization of the notion of 'safety' used by the proof is wrong," it is not clear whether it means that the values the programmers have in mind are not correctly implemented by the formalization, or whether it means they are correctly implemented but are themselves catastrophic in a way that hasn't been anticipated. I would be highly concerned about both. There are other catastrophic possibilities as well; perhaps the utility function itself is well-specified and safe, but the AGI's model of the world is flawed (in particular, perhaps its prior or its process for matching observations to predictions are flawed) in a way that doesn't emerge until the AGI has made substantial changes to its environment.

By SI's own arguments, even a small error in any of these things would likely lead to catastrophe. And there are likely failure forms I haven't thought of. The overriding intuition here is that complex plans usually fail when unaccompanied by feedback loops. A scenario in which a set of people is ready to unleash an all-powerful being to maximize some parameter in the world, based solely on their initial confidence in their own extrapolations of the consequences of doing so, seems like a scenario that is overwhelmingly likely to result in a bad outcome. It comes down to placing the world's largest bet on a highly complex theory - with no experimentation to test the theory first.

So far, all I have argued is that the development of "Friendliness" theory can achieve at best only a limited reduction in the probability of an unfavorable outcome. However, as I argue in the next section, I believe there is at least one concept - the "tool-agent" distinction - that has more potential to reduce risks, and that SI appears to ignore this concept entirely. I believe that tools are safer than agents (even agents that make use of the best "Friendliness" theory that can reasonably be hoped for) and that SI encourages a focus on building agents, thus increasing risk.

Objection 2: SI appears to neglect the potentially important distinction between "tool" and "agent" AI.

Google Maps is a type of artificial intelligence (AI). It is far more intelligent than I am when it comes to planning routes.

Google Maps - by which I mean the complete software package including the display of the map itself - does not have a "utility" that it seeks to maximize. (One could fit a utility function to its actions, as to any set of actions, but there is no single "parameter to be maximized" driving its operations.)

Google Maps (as I understand it) considers multiple possible routes, gives each a score based on factors such as distance and likely traffic, and then displays the best-scoring route in a way that makes it easily understood by the user. If I don't like the route, for whatever reason, I can change some parameters and consider a different route. If I like the route, I can print it out or email it to a friend or send it to my phone's navigation application. Google Maps has no single parameter it is trying to maximize; it has no reason to try to "trick" me in order to increase its utility.

In short, Google Maps is not an agent, taking actions in order to maximize a utility parameter. It is a tool, generating information and then displaying it in a user-friendly manner for me to consider, use and export or discard as I wish.

Every software application I know of seems to work essentially the same way, including those that involve (specialized) artificial intelligence such as Google Search, Siri, Watson, Rybka, etc. Some can be put into an "agent mode" (as Watson was on Jeopardy!) but all can easily be set up to be used as "tools" (for example, Watson can simply display its top candidate answers to a question, with the score for each, without speaking any of them.)

The "tool mode" concept is importantly different from the possibility of Oracle AI sometimes discussed by SI. The discussions I've seen of Oracle AI present it as an Unfriendly AI that is "trapped in a box" - an AI whose intelligence is driven by an explicit utility function and that humans hope to control coercively. Hence the discussion of ideas such as the AI-Box Experiment. A different interpretation, given in Karnofsky/Tallinn 2011, is an AI with a carefully designed utility function - likely as difficult to construct as "Friendliness" - that leaves it "wishing" to answer questions helpfully. By contrast with both these ideas, Tool-AGI is not "trapped" and it is not Unfriendly or Friendly; it has no motivations and no driving utility function of any kind, just like Google Maps. It scores different possibilities and displays its conclusions in a transparent and user-friendly manner, as its instructions say to do; it does not have an overarching "want," and so, as with the specialized AIs described above, while it may sometimes "misinterpret" a question (thereby scoring options poorly and ranking the wrong one #1) there is no reason to expect intentional trickery or manipulation when it comes to displaying its results.

Another way of putting this is that a "tool" has an underlying instruction set that conceptually looks like: "(1) Calculate which action A would maximize parameter P, based on existing data set D. (2) Summarize this calculation in a user-friendly manner, including what Action A is, what likely intermediate outcomes it would cause, what other actions would result in high values of P, etc." An "agent," by contrast, has an underlying instruction set that conceptually looks like: "(1) Calculate which action, A, would maximize parameter P, based on existing data set D. (2) Execute Action A." In any AI where (1) is separable (by the programmers) as a distinct step, (2) can be set to the "tool" version rather than the "agent" version, and this separability is in fact present with most/all modern software. Note that in the "tool" version, neither step (1) nor step (2) (nor the combination) constitutes an instruction to maximize a parameter - to describe a program of this kind as "wanting" something is a category error, and there is no reason to expect its step (2) to be deceptive.

I elaborated further on the distinction and on the concept of a tool-AI in Karnofsky/Tallinn 2011.

This is important because an AGI running in tool mode could be extraordinarily useful but far more safe than an AGI running in agent mode. In fact, if developing "Friendly AI" is what we seek, a tool-AGI could likely be helpful enough in thinking through this problem as to render any previous work on "Friendliness theory" moot. Among other things, a tool-AGI would allow transparent views into the AGI's reasoning and predictions without any reason to fear being purposefully misled, and would facilitate safe experimental testing of any utility function that one wished to eventually plug into an "agent."

Is a tool-AGI possible? I believe that it is, and furthermore that it ought to be our default picture of how AGI will work, given that practically all software developed to date can (and usually does) run as a tool and given that modern software seems to be constantly becoming "intelligent" (capable of giving better answers than a human) in surprising new domains. In addition, it intuitively seems to me (though I am not highly confident) that intelligence inherently involves the distinct, separable steps of (a) considering multiple possible actions and (b) assigning a score to each, prior to executing any of the possible actions. If one can distinctly separate (a) and (b) in a program's code, then one can abstain from writing any "execution" instructions and instead focus on making the program list actions and scores in a user-friendly manner, for humans to consider and use as they wish.

Of course, there are possible paths to AGI that may rule out a "tool mode," but it seems that most of these paths would rule out the application of "Friendliness theory" as well. (For example, a "black box" emulation and augmentation of a human mind.) What are the paths to AGI that allow manual, transparent, intentional design of a utility function but do not allow the replacement of "execution" instructions with "communication" instructions? Most of the conversations I've had on this topic have focused on three responses:

One of the things that bothers me most about SI is that there is practically no public content, as far as I can tell, explicitly addressing the idea of a "tool" and giving arguments for why AGI is likely to work only as an "agent." The idea that AGI will be driven by a central utility function seems to be simply assumed. Two examples:

The closest thing I have seen to a public discussion of "tool-AGI" is in Dreams of Friendliness, where Eliezer Yudkowsky considers the question, "Why not just have the AI answer questions, instead of trying to do anything? Then it wouldn't need to be Friendly. It wouldn't need any goals at all. It would just answer questions." His response:

To which the reply is that the AI needs goals in order to decide how to think: that is, the AI has to act as a powerful optimization process in order to plan its acquisition of knowledge, effectively distill sensory information, pluck "answers" to particular questions out of the space of all possible responses, and of course, to improve its own source code up to the level where the AI is a powerful intelligence. All these events are "improbable" relative to random organizations of the AI's RAM, so the AI has to hit a narrow target in the space of possibilities to make superintelligent answers come out.

This passage appears vague and does not appear to address the specific "tool" concept I have defended above (in particular, it does not address the analogy to modern software, which challenges the idea that "powerful optimization processes" cannot run in tool mode). The rest of the piece discusses (a) psychological mistakes that could lead to the discussion in question; (b) the "Oracle AI" concept that I have outlined above. The comments contain some more discussion of the "tool" idea (Denis Bider and Shane Legg seem to be picturing something similar to "tool-AGI") but the discussion is unresolved and I believe the "tool" concept defended above remains essentially unaddressed.

In sum, SI appears to encourage a focus on building and launching "Friendly" agents (it is seeking to do so itself, and its work on "Friendliness" theory seems to be laying the groundwork for others to do so) while not addressing the tool-agent distinction. It seems to assume that any AGI will have to be an agent, and to make little to no attempt at justifying this assumption. The result, in my view, is that it is essentially advocating for a more dangerous approach to AI than the traditional approach to software development.

Objection 3: SI's envisioned scenario is far more specific and conjunctive than it appears at first glance, and I believe this scenario to be highly unlikely.

SI's scenario concerns the development of artificial general intelligence (AGI): a computer that is vastly more intelligent than humans in every relevant way. But we already have many computers that are vastly more intelligent than humans in some relevant ways, and the domains in which specialized AIs outdo humans seem to be constantly and continuously expanding. I feel that the relevance of "Friendliness theory" depends heavily on the idea of a "discrete jump" that seems unlikely and whose likelihood does not seem to have been publicly argued for.

One possible scenario is that at some point, we develop powerful enough non-AGI tools (particularly specialized AIs) that we vastly improve our abilities to consider and prepare for the eventuality of AGI - to the point where any previous theory developed on the subject becomes useless. Or (to put this more generally) non-AGI tools simply change the world so much that it becomes essentially unrecognizable from the perspective of today - again rendering any previous "Friendliness theory" moot. As I said in Karnofsky/Tallinn 2011, some of SI's work "seems a bit like trying to design Facebook before the Internet was in use, or even before the computer existed."

Perhaps there will be a discrete jump to AGI, but it will be a sort of AGI that renders "Friendliness theory" moot for a different reason. For example, in the practice of software development, there often does not seem to be an operational distinction between "intelligent" and "Friendly." (For example, my impression is that the only method programmers had for evaluating Watson's "intelligence" was to see whether it was coming up with the same answers that a well-informed human would; the only way to evaluate Siri's "intelligence" was to evaluate its helpfulness to humans.) "Intelligent" often ends up getting defined as "prone to take actions that seem all-around 'good' to the programmer." So the concept of "Friendliness" may end up being naturally and subtly baked in to a successful AGI effort.

The bottom line is that we know very little about the course of future artificial intelligence. I believe that the probability that SI's concept of "Friendly" vs. "Unfriendly" goals ends up seeming essentially nonsensical, irrelevant and/or unimportant from the standpoint of the relevant future is over 90%.

Other objections to SI's views

There are other debates about the likelihood of SI's work being relevant/helpful; for example,

Unlike the three objections I focus on, these other issues have been discussed a fair amount, and if these other issues were the only objections to SI's arguments I would find SI's case to be strong (i.e., I would find its scenario likely enough to warrant investment in).

Wrapup

For a long time I refrained from engaging in object-level debates over SI's work, believing that others are better qualified to do so. But after talking at great length to many of SI's supporters and advocates and reading everything I've been pointed to as relevant, I still have seen no clear and compelling response to any of my three major objections. As stated above, there are many possible responses to my objections, but SI's current arguments do not seem clear on what responses they wish to take and defend. At this point I am unlikely to form a positive view of SI's work until and unless I do see such responses, and/or SI changes its positions.

Is SI the kind of organization we want to bet on?

This part of the post has some risks. For most of GiveWell's history, sticking to our standard criteria - and putting more energy into recommended than non-recommended organizations - has enabled us to share our honest thoughts about charities without appearing to get personal. But when evaluating a group such as SI, I can't avoid placing a heavy weight on (my read on) the general competence, capability and "intangibles" of the people and organization, because SI's mission is not about repeating activities that have worked in the past. Sharing my views on these issues could strike some as personal or mean-spirited and could lead to the misimpression that GiveWell is hostile toward SI. But it is simply necessary in order to be fully transparent about why I hold the views that I hold.

Fortunately, SI is an ideal organization for our first discussion of this type. I believe the staff and supporters of SI would overwhelmingly rather hear the whole truth about my thoughts - so that they can directly engage them and, if warranted, make changes - than have me sugar-coat what I think in order to spare their feelings. People who know me and my attitude toward being honest vs. sparing feelings know that this, itself, is high praise for SI.

One more comment before I continue: our policy is that non-public information provided to us by a charity will not be published or discussed without that charity's prior consent. However, none of the content of this post is based on private information; all of it is based on information that SI has made available to the public.

There are several reasons that I currently have a negative impression of SI's general competence, capability and "intangibles." My mind remains open and I include specifics on how it could be changed.

A couple positive observations to add context here:

Wrapup

While SI has produced a lot of content that I find interesting and enjoyable, it has not produced what I consider evidence of superior general rationality or of its suitability for the tasks it has set for itself. I see no qualifications or achievements that specifically seem to indicate that SI staff are well-suited to the challenge of understanding the key AI-related issues and/or coordinating the construction of an FAI. And I see specific reasons to be pessimistic about its suitability and general competence.

When estimating the expected value of an endeavor, it is natural to have an implicit "survivorship bias" - to use organizations whose accomplishments one is familiar with (which tend to be relatively effective organizations) as a reference class. Because of this, I would be extremely wary of investing in an organization with apparently poor general competence/suitability to its tasks, even if I bought fully into its mission (which I do not) and saw no other groups working on a comparable mission.

But if there's even a chance …

A common argument that SI supporters raise with me is along the lines of, "Even if SI's arguments are weak and its staff isn't as capable as one would like to see, their goal is so important that they would be a good investment even at a tiny probability of success."

I believe this argument to be a form of Pascal's Mugging and I have outlined the reasons I believe it to be invalid in two posts (here and here). There have been some objections to my arguments, but I still believe them to be valid. There is a good chance I will revisit these topics in the future, because I believe these issues to be at the core of many of the differences between GiveWell-top-charities supporters and SI supporters.

Regardless of whether one accepts my specific arguments, it is worth noting that the most prominent people associated with SI tend to agree with the conclusion that the "But if there's even a chance …" argument is not valid. (See comments on my post from Michael Vassar and Eliezer Yudkowsky as well as Eliezer's interview with John Baez.)

Existential risk reduction as a cause

I consider the general cause of "looking for ways that philanthropic dollars can reduce direct threats of global catastrophic risks, particularly those that involve some risk of human extinction" to be a relatively high-potential cause. It is on the working agenda for GiveWell Labs and we will be writing more about it.

However, I don't think that "Cause X is the one I care about and Organization Y is the only one working on it" to be a good reason to support Organization Y. For donors determined to donate within this cause, I encourage you to consider donating to a donor-advised fund while making it clear that you intend to grant out the funds to existential-risk-reduction-related organizations in the future. (One way to accomplish this would be to create a fund with "existential risk" in the name; this is a fairly easy thing to do and one person could do it on behalf of multiple donors.)

For one who accepts my arguments about SI, I believe withholding funds in this way is likely to be better for SI's mission than donating to SI - through incentive effects alone (not to mention my specific argument that SI's approach to "Friendliness" seems likely to increase risks).

How I might change my views

My views are very open to revision.

However, I cannot realistically commit to read and seriously consider all comments posted on the matter. The number of people capable of taking a few minutes to write a comment is sufficient to swamp my capacity. I do encourage people to comment and I do intend to read at least some comments, but if you are looking to change my views, you should not consider posting a comment to be the most promising route.

Instead, what I will commit to is reading and carefully considering up to 50,000 words of content that are (a) specifically marked as SI-authorized responses to the points I have raised; (b) explicitly cleared for release to the general public as SI-authorized communications. In order to consider a response "SI-authorized and cleared for release," I will accept explicit communication from SI's Executive Director or from a majority of its Board of Directors endorsing the content in question. After 50,000 words, I may change my views and/or commit to reading more content, or (if I determine that the content is poor and is not using my time efficiently) I may decide not to engage further. SI-authorized content may improve or worsen SI's standing in my estimation, so unlike with comments, there is an incentive to select content that uses my time efficiently. Of course, SI-authorized content may end up including excerpts from comment responses to this post, and/or already-existing public content.

I may also change my views for other reasons, particularly if SI secures more impressive achievements and/or endorsements.

One more note: I believe I have read the vast majority of the Sequences, including the AI-foom debate, and that this content - while interesting and enjoyable - does not have much relevance for the arguments I've made.

Again: I think that whatever happens as a result of my post will be positive for SI's mission, whether or not it is positive for SI as an organization. I believe that most of SI's supporters and advocates care more about the former than about the latter, and that this attitude is far too rare in the nonprofit world.

Acknowledgements

Thanks to the following people for reviewing a draft of this post and providing thoughtful feedback (this of course does not mean they agree with the post or are responsible for its content): Dario Amodei, Nick Beckstead, Elie Hassenfeld, Alexander Kruel, Tim Ogden, John Salvatier, Jonah Sinick, Cari Tuna, Stephanie Wykstra.

1284 comments

Comments sorted by top scores.

comment by lukeprog · 2012-05-10T21:24:19.513Z · score: 62 (70 votes) · LW(p) · GW(p)

Update: My full response to Holden is now here.

As Holden said, I generally think that Holden's objections for SI "are either correct (especially re: past organizational competence) or incorrect but not addressed by SI in clear argumentative writing (this includes the part on 'tool' AI)," and we are working hard to fix both categories of issues.

In this comment I would merely like to argue for one small point: that the Singularity Institute is undergoing comprehensive changes — changes which I believe to be improvements that will help us to achieve our mission more efficiently and effectively.

Holden wrote:

I'm aware that SI has relatively new leadership that is attempting to address the issues behind some of my complaints. I have a generally positive impression of the new leadership; I believe the Executive Director and Development Director, in particular, to represent a step forward in terms of being interested in transparency and in testing their own general rationality. So I will not be surprised if there is some improvement in the coming years...

Louie Helm was hired as Director of Development in September 2011. I was hired as a Research Fellow that same month, and made Executive Director in November 2011. Below are some changes made since September. (Pardon the messy presentation: LW cannot correctly render tables in comments.)

SI before Sep. 2011: Very few peer-reviewed research publications.
SI today: More peer-reviewed publications coming in 2012 than in all past years combined. Additionally, I alone have a dozen papers in development, for which I am directing every step of research and writing, and will write the final draft, but am collaborating with remote researchers so as to put in only 5%-20% of the total hours required myself.

SI before Sep. 2011: No donor database / a very broken one.
SI today: A comprehensive donor database.

SI before Sep. 2011: Nearly all work performed directly by SI staff.
SI today: Most work outsourced to remote collaborators so that SI staff can focus on the things that only they can do.

SI before Sep. 2011: No strategic plan.
SI today: A strategic plan developed with input from all SI staff, and approved by the Board.

SI before Sep. 2011: Very little communication about what SI is doing.
SI today: Monthly progress reports, plus three Q&As with Luke about SI research and organizational development.

SI before Sep. 2011: No list of the research problems SI is working on.
SI today: A long, fully-referenced list of research problems SI is working on.

SI before Sep. 2011: Very little direct management of staff and projects.
SI today: Luke monitors all projects and staff work, and meets regularly with each staff member.

SI before Sep. 2011: Almost no detailed tracking of the expense of major SI projects (e.g. Summit, papers, etc.). The sole exception seems to be that Amy was tracking the costs of the 2011 Summit in NYC.
SI today: Detailed tracking of the expense of major SI projects for which this is possible (Luke has a folder in Google docs for these spreadsheets, and the summary spreadsheet is shared with the Board).

SI before Sep. 2011: No staff worklogs.
SI today: All staff members share their worklogs with Luke, Luke shares his worklog with all staff plus the Board.

SI before Sep. 2011: Best practices not followed for bookkeeping/accounting; accountant's recommendations ignored.
SI today: Meetings with consultants about bookkeeping/accounting; currently working with our accountant to implement best practices and find a good bookkeeper.

SI before Sep. 2011: Staff largely separated, many of them not well-connected to the others.
SI today: After a dozen or so staff dinners, staff much better connected, more of a team.

SI before Sep. 2011: Want to see the basics of AI Risk explained in plain language? Read The Sequences (more than a million words) or this academic book chapter by Yudkowsky.
SI today: Want to see the basics of AI Risk explained in plain language? Read Facing the Singularity (now in several languages, with more being added) or listen to the podcast version.

SI before Sep. 2011: Very few resources created to support others' research in AI risk.
SI today: IntelligenceExplosion.com, Friendly-AI.com, list of open problems in the field, with references, AI Risk Bibliography 2012, annotated list of journals that may publish papers on AI risk, a partial history of AI risk research, and a list of forthcoming and desired articles on AI risk.

SI before Sep. 2011: A hard-to-navigate website with much outdated content.
SI today: An entirely new website that is easier to navigate and has much new content (nearly complete; should launch in May or June).

SI before Sep. 2011: So little monitoring of funds that $118k was stolen in 2010 before SI noticed. (Note that we have won stipulated judgments to get much of this back, and have upcoming court dates to argue for stipulated judgments to get the rest back.)
SI today: Our bank accounts have been consolidated, with 3-4 people regularly checking over them.

SI before Sep. 2011: SI publications exported straight to PDF from Word or Google Docs, sometimes without even author names appearing.
SI today: All publications being converted into slick, useable LaTeX template (example), with all references checked and put into a central BibTeX file.

SI before Sep. 2011: No write-up of our major public technical breakthrough (TDT) using the mainstream format and vocabulary comprehensible to most researchers in the field (this is what we have at the moment).
SI today: Philosopher Rachael Briggs, whose papers on decision theory have been twice selected for the Philosopher's Annual, has been contracted to write an explanation of TDT and publish it in one of a select few leading philosophy journals.

SI before Sep. 2011: No explicit effort made toward efficient use of SEO or our (free) Google Adwords.
SI today: Highly optimized use of Google Adwords to direct traffic to our sites; currently working with SEO consultants to improve our SEO (of course, the new website will help).

(Just to be clear, I think this list shows not that "SI is looking really great!" but instead that "SI is rapidly improving and finally reaching a 'basic' level of organizational function.")

comment by lukeprog · 2012-05-11T02:54:28.896Z · score: 26 (25 votes) · LW(p) · GW(p)

...which is not to say, of course, that things were not improving before September 2011. It's just that the improvements have accelerated quite a bit since then.

For example, Amy was hired in December 2009 and is largely responsible for these improvements:

  • Built a "real" Board and officers; launched monthly Board meetings in February 2010.
  • Began compiling monthly financial reports in December 2010.
  • Began tracking Summit expenses and seeking Summit sponsors.
  • Played a major role in canceling many programs and expenses that were deemed low ROI.
comment by [deleted] · 2012-05-11T04:25:54.689Z · score: 9 (9 votes) · LW(p) · GW(p)

Our bank accounts have been consolidated, with 3-4 people regularly checking over them.

In addition to reviews, should SI implement a two-man rule for manipulating large quantities of money? (For example, over 5k, over 10k, etc.)

comment by ghf · 2012-05-11T22:38:10.640Z · score: 5 (9 votes) · LW(p) · GW(p)

My hope is that the upcoming deluge of publications will answer this objection, but for the moment, I am unclear as to the justification for the level of resources being given to SIAI researchers.

Additionally, I alone have a dozen papers in development, for which I am directing every step of research and writing, and will write the final draft, but am collaborating with remote researchers so as to put in only 5%-20% of the total hours required myself.

This level of freedom is the dream of every researcher on the planet. Yet, it's unclear why these resources should be devoted to your projects. While I strongly believe that the current academic system is broken, you are asking for a level of support granted to top researchers prior to have made any original breakthroughs yourself.

If you can convince people to give you that money, wonderful. But until you have made at least some serious advancement to demonstrate your case, donating seems like an act of faith.

It's impressive that you all have found a way to hack the system and get paid to develop yourselves as researchers outside of the academic system and I will be delighted to see that development bear fruit over the coming years. But, at present, I don't see evidence that the work being done justifies or requires that support.

comment by lukeprog · 2012-05-11T22:48:13.467Z · score: 8 (11 votes) · LW(p) · GW(p)

This level of freedom is the dream of every researcher on the planet. Yet, it's unclear why these resources should be devoted to your projects.

Because some people like my earlier papers and think I'm writing papers on the most important topic in the world?

It's impressive that you all have found a way to hack the system and get paid to develop yourselves as researchers outside of the academic system...

Note that this isn't uncommon. SI is far from the only think tank with researchers who publish in academic journals. Researchers at private companies do the same.

comment by ghf · 2012-05-11T23:15:03.978Z · score: 13 (13 votes) · LW(p) · GW(p)

First, let me say that, after re-reading, I think that my previous post came off as condescending/confrontational which was not my intent. I apologize.

Second, after thinking about this for a few minutes, I realized that some of the reason your papers seem so fluffy to me is that they argue what I consider to be obvious points. In my mind, of course we are likely "to develop human-level AI before 2100." Because of that, I may have tended to classify your work as outreach more than research.

But outreach is valuable. And, so that we can factor out the question of the independent contribution of your research, having people associated with SIAI with the publications/credibility to be treated as experts has gigantic benefits in terms of media multipliers (being the people who get called on for interviews, panels, etc). So, given that, I can see a strong argument for publication support being valuable to the overall organization goals regardless of any assessment of the value of the research.

Note that this isn't uncommon. SI is far from the only think tank with researchers who publish in academic journals. Researchers at private companies do the same.

My only point was that, in those situations, usually researchers are brought in with prior recognized achievements (or, unfortunately all too often, simply paper credentials). SIAI is bringing in people who are intelligent but unproven and giving them the resources reserved for top talent in academia or industry. As you've pointed out, one of the differences with SIAI is the lack of hoops to jump through.

Edit: I see you commented below that you view your own work as summarization of existing research and we agree on the value of that. Sorry that my slow typing speed left me behind the flow of the thread.

comment by Bugmaster · 2012-05-11T22:53:44.772Z · score: 2 (2 votes) · LW(p) · GW(p)

Researchers at private companies do the same.

It's true at my company, at least. There are quite a few papers out there authored by the researchers at the company where I work. There are several good business reasons for a company to invest time into publishing a paper; positive PR is one of them.

comment by metaphysicist · 2012-05-11T23:02:30.053Z · score: -5 (9 votes) · LW(p) · GW(p)

Because some people like my earlier papers and think I'm writing papers on the most important topic in the world?

But then you put your intellect at issue, and I think I'm entitled to opine that you lack the qualities of intellect that would make such recommendation credible. You're a budding scholar; a textbook writer at heart. You lack any of the originality of a thinker.

You confirm the lead poster's allegations that SIA staff are insular and conceited.

comment by lukeprog · 2012-05-11T23:11:55.421Z · score: 19 (19 votes) · LW(p) · GW(p)

I think I'm entitled to opine...

Of course you are. And, you may not be one of the people who "like my earlier papers."

You confirm the lead poster's allegations that SIA staff are insular and conceited.

Really? How? I commented earlier on LW (can't find it now) about how the kind of papers I write barely count as "original research" because for the most part they merely summarize and clarify the ideas of others. But as Beckstead says, there is a strong need for that right now.

For insights in decision theory and FAI theory, I suspect we'll have to look to somebody besides Luke Muehlhauser. We keep trying to hire such people but they keep saying "No." (I got two more "no"s just in the last 3 weeks.) Part of that may be due to the past and current state of the organization — and luckily, fixing that kind of thing is something I seem to have some skills with.

You're... a textbook writer at heart.

True, dat.

comment by [deleted] · 2012-05-11T08:18:32.654Z · score: 5 (7 votes) · LW(p) · GW(p)

I was hired as a Research Fellow that same month

Luke alone has a dozen papers in development

Why did you start referring to yourself in the first person and then change your mind? (Or am I missing something?)

comment by lukeprog · 2012-05-11T08:20:33.067Z · score: 9 (13 votes) · LW(p) · GW(p)

Brain fart: now fixed.

comment by [deleted] · 2012-05-11T08:27:14.993Z · score: 20 (21 votes) · LW(p) · GW(p)

(Why was this downvoted? If it's because the downvoter wants to see fewer brain farts, they're doing it wrong, because the message such a downvote actually conveys is that they want to see fewer acknowledgements of brain farts. Upvoted back to 0, anyway.)

comment by JoshuaFox · 2012-05-17T15:12:28.708Z · score: 4 (4 votes) · LW(p) · GW(p)

As a supporter and donor to SI since 2006, I can say that I had a lot of specific criticisms of the way that the organization was managed. The points Luke lists above were among them. I was surprised that on many occasions management did not realize the obvious problems and fix them.

But the current management is now recognizing many of these points and resolving them one by one, as Luke says. If this continues, SI's future looks good.

comment by siodine · 2012-05-11T13:35:22.323Z · score: 4 (6 votes) · LW(p) · GW(p)

Isn't this very strong evidence in support for Holden's point about "Apparent poorly grounded belief in SI's superior general rationality" (excluding Luke, at least)? And especially this?

comment by lukeprog · 2012-05-11T20:13:20.034Z · score: 18 (20 votes) · LW(p) · GW(p)

This topic is something I've been thinking about lately. Do SIers tend to have superior general rationality, or do we merely escape a few particular biases? Are we good at rationality, or just good at "far mode" rationality (aka philosophy)? Are we good at epistemic but not instrumental rationality? (Keep in mind, though, that rationality is only a ceteris paribus predictor of success.)

Or, pick a more specific comparison. Do SIers tend to be better at general rationality than someone who can keep a small business running for 5 years? Maybe the tight feedback loops of running a small business are better rationality training than "debiasing interventions" can hope to be.

Of course, different people are more or less rational in different domains, at different times, in different environments.

This isn't an idle question about labels. My estimate of the scope and level of people's rationality in part determines how much I update from their stated opinion on something. How much evidence for Hypothesis X (about organizational development) is it when Eliezer gives me his opinion on the matter, as opposed to when Louie gives me his opinion on the matter? When Person B proposes to take on a totally new kind of project, I think their general rationality is a predictor of success — so, what is their level of general rationality?

comment by Bugmaster · 2012-05-11T22:49:28.381Z · score: 2 (6 votes) · LW(p) · GW(p)

Are we good at epistemic but not instrumental rationality?

Holden implies (and I agree with him) that there's very little evidence at the moment to suggest that SI is good at instrumental rationality. As for epistemic rationality, how would we know ? Is there some objective way to measure it ? I personally happen to believe that if a person seems to take it as a given that he's great at epistemic rationality, this fact should count as evidence (however circumstantial) against him being great at epistemic rationality... but that's just me.

comment by TheOtherDave · 2012-05-11T21:10:55.204Z · score: 1 (1 votes) · LW(p) · GW(p)

If you accept that your estimate of someone's "rationality" should depend on the domain, the environment, the time, the context, etc... and what you want to do is make reliable estimates of the reliability of their opinion, their chances of success. etc... it seems to follow that you should be looking for comparisons within a relevant domain, environment, etc.

That is, if you want to get opinions about hypothesis X about organizational development that serve as significant evidence, it seems the thing to do is to find someone who knows a lot about organizational development -- ideally, someone who has been successful at developing organizations -- and consult their opinions. How generally rational they are might be very relevant causally, or it might not, but is in either case screened off by their domain competence... and their domain competence is easier to measure than their general rationality.

So is their general rationality worth devoting resources to determining?

It seems this only makes sense if you have already (e.g.) decided to ask Eliezer and Louie for their advice, whether it's good evidence or not, and now you need to know how much evidence it is, and you expect the correct answer is different from the answer you'd get by applying the metrics you know about (e.g., domain familiarity and previously demonstrated relevant expertise).

comment by lukeprog · 2012-05-11T21:55:52.957Z · score: 3 (3 votes) · LW(p) · GW(p)

I do spend a fair amount of time talking to domain experts outside of SI. The trouble is that the question of what we should do about thing X doesn't just depend on domain competence but also on thousands of details about the inner workings of SI and our mission that I cannot communicate to domain experts outside SI, but which Eliezer and Louie already possess.

comment by TheOtherDave · 2012-05-11T22:14:49.868Z · score: 4 (4 votes) · LW(p) · GW(p)

So it seems you have a problem in two domains (organizational development + SI internals) and different domain experts in both domains (outside domain experts + Eliezer/Louie), and need some way of cross-linking the two groups' expertise to get a coherent recommendation, and the brute-force solutions (e.g. get them all in a room together, or bring one group up to speed on the other's domain) are too expensive to be worth it. (Well, assuming the obstacle isn't that the details need to be kept secret, but simply that expecting an outsider to come up to speed on all of SI's local potentially relevant trivia simply isn't practical.)

Yes?

Yeah, that can be a problem.

In that position, for serious questions I would probably ask E/L for their recommendations and a list of the most relevant details that informed that decision, then go to outside experts with a summary of the competing recommendations and an expanded version of that list and ask for their input. If there's convergence, great. If there's divergence, iterate.

This is still a expensive approach, though, so I can see where a cheaper approximation for less important questions is worth having.

comment by lukeprog · 2012-05-11T22:18:53.862Z · score: 2 (2 votes) · LW(p) · GW(p)

Yes to all this.

comment by siodine · 2012-05-11T23:08:47.169Z · score: -1 (3 votes) · LW(p) · GW(p)

In the world in which a varied group of intelligent and especially rational people are organizing to literally save humanity, I don't see the relatively trivial, but important, improvements you've made in a short period of time being made because they were made years ago. And I thought that already accounting for the points you've made.

I mean, the question this group should be asking themselves is "how can we best alter the future so as to navigate towards FAI?" So, how did they apparently miss something like opportunity cost? Why, for instance, has their salaries increased when they could've been using it to improve the foundation of their cause from which everything else follows?

(Granted, I don't know the history and inner workings of the SI, and so I could be missing some very significant and immovable hurdles, but I don't see that as very likely; at least, not as likely as Holden's scenario.)

comment by lukeprog · 2012-05-11T23:18:25.605Z · score: 4 (4 votes) · LW(p) · GW(p)

I don't see the relatively trivial, but important, improvements you've made in a short period of time being made because they were made years ago. And I thought that already accounting for the points you've made.

I don't know what these sentences mean.

So, how did they apparently miss something like opportunity cost? Why, for instance, has their salaries increased when they could've been using it to improve the foundation of their cause from which everything else follows?

Actually, salary increases help with opportunity cost. At very low salaries, SI staff ends up spending lots of time and energy on general life cost-saving measures that distract us from working on x-risk reduction. And our salaries are generally still pretty low. I have less than $6k in my bank accounts. Outsourcing most tasks to remote collaborators also helps a lot with opportunity cost.

comment by siodine · 2012-05-12T00:01:50.875Z · score: 3 (3 votes) · LW(p) · GW(p)

I don't know what these sentences mean.

  • People are more rational in different domains, environments, and so on.
  • The people at SI may have poor instrumental rationality while being adept at epistemic rationality.
  • Being rational doesn't necessarily mean being successful.

I accept all those points, and yet I still see the Singularity Institute having made the improvements that you've made since being hired before you were hired if they have superior general rationality. That is, you wouldn't have that list of relatively trivial things to brag about because someone else would have recognized the items on that list as important and got them done somehow (ignore any negative connotations--they're not intended).

For instance, I don't see a varied group of people with superior general rationality not discovering or just not outsourcing work they don't have a comparative advantage in (i.e., what you've done). That doesn't look like just a failure in instrumental rationality, or just rationality operating on a different kind of utility function, or just a lack of domain specific knowledge.

The excuses available to a person acting in a way that's non-traditionally rational are less convincing when you apply them to a group.

Actually, salary increases help with opportunity cost. At very low salaries, SI staff ends up spending lots of time and energy on general life cost-saving measures that distract us from working on x-risk reduction. And our salaries are generally still pretty low. I have less than $6k in my bank accounts.

No, I get that. But that still doesn't explain away the higher salaries like EY's 80k/year and its past upwards trend. I mean, these higher paid people are the most committed to the cause, right? I don't see those people taking a higher salary when they could use that money for more outsourcing, or another employee, or better employees, if they want to literally save humanity while being superior in general rationality. It's like a homeless person desperately in want of shelter trying save enough for an apartment and yet buying meals at some restaurant.

Outsourcing most tasks to remote collaborators also helps a lot with opportunity cost.

That's the point I was making, why wasn't that done earlier? How did these people apparently miss out on opportunity cost? (And I'm just using outsourcing as an example because it was one of the most glaring changes you made that I think should have probably been made much earlier.)

comment by lukeprog · 2012-05-12T00:20:39.396Z · score: 4 (6 votes) · LW(p) · GW(p)

Right, I think we're saying the same thing, here: the availability of so much low-hanging fruit in organizational development as late as Sept. 2011 is some evidence against the general rationality of SIers. Eliezer seems to want to say it was all a matter of funding, but that doesn't make sense to me.

Now, on this:

I don't see those people taking a higher salary when they could use that money for more outsourcing, or another employee, or better employees, if they want to literally save humanity while being super in general rationality.

For some reason I'm having a hard time parsing your sentences for unambiguous meaning, but if I may attempt to rephrase: "SIers wouldn't take any salaries higher than (say) $70k/yr if they were truly committed to the cause and good in general rationality, because they would instead use that money to accomplish other things." Is that what you're saying?

comment by Rain · 2012-05-12T00:29:53.205Z · score: 3 (3 votes) · LW(p) · GW(p)

I've heard the Bay Area is expensive, and previously pointed out that Eliezer earns more than I do, despite me being in the top 10 SI donors.

I don't mind, though, as has been pointed out, even thinking about muffins might be a question invoking existential risk calculations.

comment by lukeprog · 2012-05-12T00:39:54.925Z · score: 4 (4 votes) · LW(p) · GW(p)

despite me being in the top 10 SI donors

...and much beloved for it.

Yes, the Bay Area is expensive. We've considered relocating, but on the other hand the (by far) best two places for meeting our needs in HR and in physically meeting with VIPs are SF and NYC, and if anything NYC is more expensive than the Bay Area. We cut living expenses where we can: most of us are just renting individual rooms.

Also, of course, it's not like the Board could decide we should relocate to a charter city in Honduras and then all our staff would be able to just up and relocate. :)

(Rain may know all this; I'm posting it for others' benefit.)

comment by komponisto · 2012-05-12T18:58:03.493Z · score: 12 (12 votes) · LW(p) · GW(p)

I think it's crucial that SI stay in the Bay Area. Being in a high-status place signals that the cause is important. If you think you're not taken seriously enough now, imagine if you were in Honduras...

Not to mention that HR is without doubt the single most important asset for SI. (Which is why it would probably be a good idea to pay more than the minimum cost of living.)

comment by TheOtherDave · 2012-05-12T01:31:59.697Z · score: 2 (2 votes) · LW(p) · GW(p)

Out of curiosity only: what were the most significant factors that led you to reject telepresence options?

comment by David_Gerard · 2012-05-12T18:02:44.485Z · score: 7 (7 votes) · LW(p) · GW(p)

FWIW, Wikimedia moved from Florida to San Francisco precisely for the immense value of being at the centre of things instead of the middle of nowhere (and yes, Tampa is the middle of nowhere for these purposes, even though it still has the primary data centre). Even paying local charity scale rather than commercial scale (there's a sort of cycle where WMF hires brilliant kids, they do a few years working at charity scale then go to Facebook/Google/etc for gobs of cash), being in the centre of things gets them staff and contacts they just couldn't get if they were still in Tampa. And yes, the question came up there pretty much the same as it's coming up here: why be there instead of remote? Because so much comes with being where things are actually happening, even if it doesn't look directly related to your mission (educational charity, AI research institute).

comment by komponisto · 2012-05-12T19:00:32.677Z · score: 0 (0 votes) · LW(p) · GW(p)

FWIW, Wikimedia moved from Florida to San Francisco

I didn't know this, but I'm happy to hear it.

comment by David_Gerard · 2012-05-12T19:04:58.762Z · score: 1 (1 votes) · LW(p) · GW(p)

The charity is still registered in Florida but the office is in SF. I can't find the discussion on a quick search, but all manner of places were under serious consideration - including the UK, which is a horrible choice for legal issues in so very many ways.

comment by lukeprog · 2012-05-12T01:56:50.756Z · score: 6 (8 votes) · LW(p) · GW(p)

In our experience, monkeys don't work that way. It sounds like it should work, and then it just... doesn't. Of course we do lots of Skyping, but regular human contact turns out to be pretty important.

comment by TheOtherDave · 2012-05-12T02:04:01.998Z · score: 8 (8 votes) · LW(p) · GW(p)

(nods) Yeah, that's been my experience too, though I've often suspected that companies like Google probably have a lot of research on the subject lying around that might be informative.

Some friends of mine did some experimenting along these lines when doing distributed software development (in both senses) and were somewhat startled to realize that Dark Age of Camelot worked better for them as a professional conferencing tool than any of the professional conferencing tools their company had. They didn't mention this to their management.

comment by David_Gerard · 2012-05-12T18:53:40.812Z · score: 1 (1 votes) · LW(p) · GW(p)

and were somewhat startled to realize that Dark Age of Camelot worked better for them as a professional conferencing tool than any of the professional conferencing tools their company had. They didn't mention this to their management.

I am reminded that Flickr started as a photo add-on for an MMORPG...

comment by HoverHell · 2012-05-13T08:16:45.010Z · score: 1 (1 votes) · LW(p) · GW(p)

-

comment by siodine · 2012-05-12T00:34:35.518Z · score: 0 (0 votes) · LW(p) · GW(p)

some evidence

Enough for you to agree with Holden on that point?

"SIers wouldn't take any salaries higher than (say) $70k/yr if they were truly committed to the cause and good in general rationality, because they would instead use that money to accomplish other things." Is that what you're saying?

Yes, but I wouldn't set a limit at a specific salary range; I'd expect them to give as much as they optimally could, because I assume they're more concerned with the cause than the money. (re the 70k/yr mention: I'd be surprised if that was anywhere near optimal)

comment by lukeprog · 2012-05-12T00:46:18.405Z · score: 2 (2 votes) · LW(p) · GW(p)

Enough for you to agree with Holden on that point?

Probably not. He and I continue to dialogue in private about the point, in part to find the source of our disagreement.

Yes, but I wouldn't set a limit at a specific salary range; I'd expect them to give as much as they optimally could, because I assume they're more concerned with the cause than the money. (re the 70k/yr mention: I'd be surprised if that was anywhere near optimal)

I believe everyone except Eliezer currently makes between $42k/yr and $48k/yr — pretty low for the cost of living in the Bay Area.

comment by siodine · 2012-05-12T01:37:39.562Z · score: 4 (4 votes) · LW(p) · GW(p)

Probably not. He and I continue to dialogue in private about the point, in part to find the source of our disagreement.

So, if you disagree with Holden, I assume you think SIers have superior general rationality: why?

And I'm confident SIers will score well on rationality tests, but that looks like specialized rationality. I.e., you can avoid a bias but you can't avoid a failure in your achieving your goals. To me, the SI approach seems poorly leveraged. I expect more significant returns from simple knowledge acquisition. E.g., you want to become successful? YOU WANT TO WIN?! Great, read these textbooks on microeconomics, finance, and business. I think this is more the approach you take anyway.

I believe everyone except Eliezer currently makes between $42k/yr and $48k/yr — pretty low for the cost of living in the Bay Area.

That isn't as bad as I thinking it was; I don't know if that's optimal, but it seems at least reasonable.

comment by lukeprog · 2012-05-12T01:47:10.564Z · score: 0 (0 votes) · LW(p) · GW(p)

I assume you think SIers have superior general rationality: why?

I'll avoid double-labor on this and wait to reply until my conversation with Holden is done.

I expect more significant returns from simple knowledge acquisition. E.g., you want to become successful? ...Great, read these textbooks on microeconomics, finance, and business. I think this is more the approach you take anyway.

Right. Exercise the neglected virtue of scholarship and all that.

comment by siodine · 2012-05-12T01:52:51.772Z · score: 1 (1 votes) · LW(p) · GW(p)

Right. Exercise the neglected virtue of scholarship and all that.

It's not that easy to dismiss; if it's as poorly leveraged as it looks relative to other approaches then you have little reason to be spreading and teaching SI's brand of specialized rationality (except for perhaps income).

comment by lukeprog · 2012-05-12T01:55:17.733Z · score: 3 (3 votes) · LW(p) · GW(p)

I'm not dismissing it, I'm endorsing it and agreeing with you that it has been my approach ever since my first post on LW.

comment by siodine · 2012-05-12T02:10:31.432Z · score: 0 (0 votes) · LW(p) · GW(p)

Weird, I have this perception of SI being heavily invested in overcoming biases and epistemic rationality training to the detriment of relevant domain specific knowledge, but I guess that's wrong?

comment by lukeprog · 2012-05-12T02:25:08.638Z · score: 2 (2 votes) · LW(p) · GW(p)

I'm lost again; I don't know what you're saying.

comment by siodine · 2012-05-12T15:58:27.848Z · score: 1 (1 votes) · LW(p) · GW(p)

I'm not dismissing it, I'm endorsing it and agreeing with you that it has been my approach ever since my first post on LW.

I wasn't talking about you; I was talking about SI's approach in spreading and training rationality. You(SI) have Yudkowsky writing books, you have rationality minicamps, you have lesswrong, you and others are writing rationality articles and researching the rationality literature, and so on.

That kind of rationality training, research, and message looks poorly leveraged in achieving your goals, is what I'm saying. Poorly leveraged for anyone trying to achieve goals. And at its most abstract, that's what rationality is, right? Achieving your goals.

So, I don't care if your approach was to acquire as much relevant knowledge as possible before dabbling in debiasing, bayes, and whatnot (i.e., prioritizing the most leveraged approach). I wondering why your approach doesn't seem to be SI's approach. I'm wondering why SI doesn't prioritize rationality training, research, and message by whatever is the most leveraged in achieving SI's goals. I'm wondering why SI doesn't spread the virtue of scholarship to the detriment of training debiasing and so on.

SI wants to raise the sanity waterline, is what the SI doing even near optimal for that? Knowing what SIers knew and trained for couldn't even get them to see an opportunity for trading in on opportunity cost for years; that is sad.

comment by komponisto · 2012-05-12T02:04:06.251Z · score: 2 (2 votes) · LW(p) · GW(p)

(Disclaimer: the following comment should not be taken to imply that I myself have concluded that SI staff salaries should be reduced.)

I believe everyone except Eliezer currently makes between $42k/yr and $48k/yr — pretty low for the cost of living in the Bay Area.

I'll grant you that it's pretty low relative to other Bay Area salaries. But as for the actual cost of living, I'm less sure.

I'm not fortunate enough to be a Bay Area resident myself, but here is what the internet tells me:

  • After taxes, a $48,000/yr gross salary in California equates to a net of around $3000/month.

  • A 1-bedroom apartment in Berkeley and nearby places can be rented for around $1500/month. (Presumably, this is the category of expense where most of the geography-dependent high cost of living is contained.)

  • If one assumes an average spending of $20/day on food (typically enough to have at least one of one's daily meals at a restaurant), that comes out to about $600/month.

  • That leaves around $900/month for miscellaneous expenses, which seems pretty comfortable for a young person with no dependents.

So, if these numbers are right, it seems that this salary range is actually right about what the cost of living is. Of course, this calculation specifically does not include costs relating to signaling (via things such as choices of housing, clothing, transportation, etc.) that one has more money than necessary to live (and therefore isn't low-status). Depending on the nature of their job, certain SI employees may need, or at least find it distinctly advantageous for their particular duties, to engage in such signaling.

comment by metaphysicist · 2012-05-12T00:55:14.888Z · score: -3 (13 votes) · LW(p) · GW(p)

I believe everyone except Eliezer currently makes between $42k/yr and $48k/yr — pretty low for the cost of living in the Bay Area.

Damn good for someone just out of college—without a degree!

comment by lukeprog · 2012-05-12T01:03:44.321Z · score: 7 (9 votes) · LW(p) · GW(p)

The point is that we're consequentialists, and lowering salaries even further would save money (on salaries) but result in SI getting less done, not more — for the same reason that outsourcing fewer tasks would save money (on outsourcing) but cause us to get less done, not more.

comment by steven0461 · 2012-05-12T02:19:10.664Z · score: 6 (8 votes) · LW(p) · GW(p)

result in SI getting less done

You say this as though it's obvious, but if I'm not mistaken, salaries used to be about 40% of what they are now, and while the higher salaries sound like they are making a major productivity difference, hiring 2.5 times as many people would also make a major productivity difference. (Though yes, obviously marginal hires would be lower in quality.)

comment by lukeprog · 2012-05-12T02:34:02.985Z · score: 9 (19 votes) · LW(p) · GW(p)

I don't think salaries were ever as low as 40% of what they are now. When I came on board, most people were at $36k/yr.

To illustrate why lower salaries means less stuff gets done: I've been averaging 60 hours per week, and I'm unusually productive. If I am paid less, that means that (to pick just one example from this week) I can't afford to take a taxi to and from the eye doctor, which means I spend 1.5 hrs each way changing buses to get there, and spend less time being productive on x-risk. That is totally not worth it. Future civilizations would look back on this decision as profoundly stupid.

comment by Will_Newsome · 2012-05-14T04:08:06.343Z · score: 13 (17 votes) · LW(p) · GW(p)

Pretty sure Anna and Steve Rayhawk had salaries around $20k/yr at some point while living in Silicon Valley.

I don't think that you're really responding to Steven's point. Yes, as Steven said, if you were paid less then clearly that would impose more costs on you, so ceteris paribus your getting paid less would be bad. But, as Steven said, the opportunity cost is potentially very high. You haven't made a rationally compelling case that the missed opportunity is "totally not worth it" or that heeding it would be "profoundly stupid", you've mostly just re-asserted your conclusion, contra Steven's objection. What are your arguments that this is the case? Note that I personally think it's highly plausible that $40-50k/yr is optimal, but as far as I can see you haven't yet listed any rationally compelling reasons to think so.

(This comment is a little bit sterner than it would have been if you hadn't emphatically asserted that conclusions other than your own would be "profoundly stupid" without first giving overwhelming justification for your conclusion. It is especially important to be careful about such apparent overconfidence on issues where one clearly has a personal stake in the matter.)

comment by steven0461 · 2012-05-14T20:36:00.231Z · score: 6 (6 votes) · LW(p) · GW(p)

I will largely endorse Will's comment, then bow out of the discussion, because this appears to be too personal and touchy a topic for a detailed discussion to be fruitful.

comment by lukeprog · 2012-05-14T08:05:24.761Z · score: 5 (13 votes) · LW(p) · GW(p)

Pretty sure Anna and Steve Rayhawk had salaries around $20k/yr at some point while living in Silicon Valley.

If so, I suspect they were burning through savings during this time or had some kind of cheap living arrangement that I don't have.

What are your arguments that [paying you less wouldn't be worth it]?

  1. I couldn't really get by on less, so paying me less would cause me to quit the organization and do something else instead, which would cause much of this good stuff to probably not happen.

  2. It's VERY hard for SingInst to purchase value as efficiently as by purchasing Luke-hours. At $48k/yr for 60 hrs/wk, I make $15.38/hr, and one Luke-hour is unusually productive for SingInst. Paying me less and thereby causing me to work fewer hours per week is a bad value proposition for SingInst.

Or, as Eliezer put it:

paying me less would require me to do things that take up time and energy in order to get by with a smaller income. Then, assuming all goes well, future intergalactic civilizations would look back and think this was incredibly stupid; in much the same way that letting billions of person-containing brains rot in graves, and humanity allocating less than a million dollars per year to the Singularity Institute, would predictably look pretty stupid in retrospect. At Singularity Institute board meetings we at least try not to do things which will predictably make future intergalactic civilizations think we were being willfully stupid. That's all there is to it, and no more.

comment by Paul Crowley (ciphergoth) · 2012-05-14T09:36:16.138Z · score: 11 (13 votes) · LW(p) · GW(p)

This seems to me unnecessarily defensive. I support the goals of SingInst, but I could never bring myself to accept the kind of salary cut you guys are taking in order to work there. Like every other human on the planet, I can't be accurately modelled with a utility function that places any value on far distant strangers; you can more accurately model what stranger-altruism I do show as purchase of moral satisfaction, though I do seek for such altruism to be efficient. SingInst should pay the salaries it needs to pay to recruit the kind of staff it needs to fulfil its mission; it's harder to recruit if staff are expected to be defensive about demanding market salaries for their expertise, with no more than a normal adjustment for altruistic work much as if they were working for an animal sanctuary.

comment by lukeprog · 2012-05-14T09:48:39.259Z · score: 1 (7 votes) · LW(p) · GW(p)

I could never bring myself to accept the... salary cut you guys are taking in order to work [at SI]... SingInst should pay the salaries it needs to pay to recruit the kind of staff it needs to fulfill its mission; it's harder to recruit if staff are expected to be defensive about demanding market salaries for their expertise...

Yes, exactly.

comment by Paul Crowley (ciphergoth) · 2012-05-14T10:07:05.931Z · score: 4 (8 votes) · LW(p) · GW(p)

So when I say "unnecessarily defensive", I mean that all the stuff about the cost of taxis is after-the-fact defensive rationalization; it can't be said about a single dollar you spend on having a life outside of SI. The truth is that even the best human rationalist in the world isn't going to agree to giving those up, and since you have to recruit humans, you'd best pay the sort of salary that is going to attract and retain them. That of course includes yourself.

The same goes for saying "move to the Honduras". Your perfectly utility-maximising AGIs will move to the Honduras, but your human staff won't; they want to live in places like the Bay Area.

comment by [deleted] · 2012-05-14T18:55:37.425Z · score: -1 (1 votes) · LW(p) · GW(p)

I could never bring myself to accept the kind of salary cut you guys are taking in order to work there

You know that the Bay Area is freakin' expensive, right?

comment by Paul Crowley (ciphergoth) · 2012-05-15T06:15:11.274Z · score: 5 (5 votes) · LW(p) · GW(p)

Re-reading, the whole thing is pretty unclear!

As katydee and thomblake say, I mean that working for SingInst would mean a bigger reduction in my salary than I could currently bring myself to accept. If I really valued the lives of strangers as a utilitarian, the benefits to them of taking a salary cut would be so huge that it would totally outweigh the costs to me. But it looks like I only really place direct value on the short-term interests of myself and those close to me, and everything else is purchase of moral satisfaction. Happily, purchase of moral satisfaction can still save the world if it is done efficiently.

Since the labour pool contains only human beings, with no true altruistic utility maximizers, SingInst should hire and pay accordingly; the market shows that people will accept a lower salary for a job that directly does good, but not a vastly lower salary. It would increase SI-utility if Luke accepted a lower salary, but it wouldn't increase Luke-utility, and driving Luke away would cost a lot of SI-utility, so calling for it is in the end a cheap shot and a bad recommendation.

I live in London, which is also freaking expensive - but so are all the places I want to live. There's a reason people are prepared to pay more to live in these places.

comment by Eugine_Nier · 2012-05-15T04:28:36.122Z · score: 3 (3 votes) · LW(p) · GW(p)

So why not relocate SIAI somewhere with a more reasonable cost of living?

comment by katydee · 2012-05-15T04:50:47.056Z · score: 3 (3 votes) · LW(p) · GW(p)

I think the standard answer is that the networking and tech industry connections available in the Bay Area are useful enough to SIAI to justify the high costs of operating there.

comment by TraderJoe · 2012-05-17T15:20:35.401Z · score: 0 (0 votes) · LW(p) · GW(p)

[comment deleted]

comment by thomblake · 2012-05-14T19:12:50.224Z · score: 2 (2 votes) · LW(p) · GW(p)

Hmm... Perhaps you don't know that "salary cut" above means taking much less money?

comment by [deleted] · 2012-05-15T12:00:54.296Z · score: 1 (1 votes) · LW(p) · GW(p)

I had missed the word cut. Damn it, I shouldn't be commenting while sleep-deprived!

comment by Paul Crowley (ciphergoth) · 2012-05-15T06:22:10.410Z · score: 0 (0 votes) · LW(p) · GW(p)

Indeed. I guess "taking a cut" can sometimes mean "taking some of the money", so you could interpret this as meaning "I couldn't accept all that money", which as you say is the opposite of what I meant!

comment by katydee · 2012-05-14T19:10:01.759Z · score: 1 (1 votes) · LW(p) · GW(p)

Perhaps that's why he's saying he wouldn't be willing to live there on a low salary?

comment by Bugmaster · 2012-05-14T22:37:02.400Z · score: 1 (5 votes) · LW(p) · GW(p)

I understand the point you're making regarding salaries, and for once I agree.

However, it's rather presumptuous of you (and/or Eliezer) to assume, implicitly, that our choices are limited to only two possibilities: "Support SIAI, save the world", and "Don't support SIAI, the world is doomed". I can envision many other scenarios, such as "Support SIAI, but their fears were overblown and you implicitly killed N children by not spending the money on them instead", or "Don't support SIAI, support some other organization instead because they'll have a better chance of success", etc.

comment by lukeprog · 2012-05-15T21:45:26.017Z · score: 4 (6 votes) · LW(p) · GW(p)

Where did we say all that?

comment by Bugmaster · 2012-05-15T22:01:31.542Z · score: 2 (6 votes) · LW(p) · GW(p)

In your comment above, you said:

...I can't afford to take a taxi to and from the eye doctor, which means I spend 1.5 hrs each way changing buses to get there, and spend less time being productive on x-risk. That is totally not worth it. Future civilizations would look back on this decision as profoundly stupid.

You also quoted Eliezer saying something similar.

This outlook implies strongly that whatever SIAI is doing is of such monumental significance that future civilizations will not only remember its name, but also reverently preserve every decision it made. You are also quite fond of saying that the work that SIAI is doing is tantamount to "saving the world"; and IIRC Eliezer once said that, if you have a talent for investment banking, you should make as much money as possible and then donate it all to SIAI, as opposed to any other charity.

This kind of grand rhetoric presupposes not only that the SIAI is correct in its risk assessment regarding AGI, but also that they are uniquely qualified to address this potentially world-ending problem, and that, over the ages, no one more qualified could possibly come along. All of this could be true, but it's far from a certainty, as your writing would seem to imply.

comment by jacob_cannell · 2012-05-16T09:43:41.718Z · score: 1 (1 votes) · LW(p) · GW(p)

This outlook implies strongly that whatever SIAI is doing is of such monumental significance that future civilizations will not only remember its name, but also reverently preserve every decision it made.

In the SIA/Transhumanist outlook, if civilization survives some large (perhaps majority) of extant human minds will survive as uploads. As a result, all of their memories will likely be stored, dissected, shared, searched, judged, and so on. Much will be preserved in such a future. And even without uploading, there are plenty of people who have maintained websites since the early days of the internet with no loss of information, and this is quite likely to remain true far into the future if civilization survives.

comment by Paul Crowley (ciphergoth) · 2012-05-16T09:29:13.117Z · score: 1 (1 votes) · LW(p) · GW(p)

Well no, of course it's not a certainty. All efforts to make a difference are decisions under uncertainty. You're attacking a straw man.

comment by Bugmaster · 2012-05-16T21:06:46.607Z · score: 1 (3 votes) · LW(p) · GW(p)

Please substitute "certainty minus epsilon" for "certainty" wherever you see it in my post. It was not my intention to imply 100% certainty; just a confidence value so high that it amounts to the same thing for all practical purposes.

comment by dlthomas · 2012-05-16T21:34:18.450Z · score: 1 (1 votes) · LW(p) · GW(p)

I don't think "certainty minus epsilon" improves much. It moves it from theoretical impossibility to practical - but looking that far out, I expect "likelihood" might be best.

comment by Bugmaster · 2012-05-17T01:49:45.006Z · score: 1 (1 votes) · LW(p) · GW(p)

I don't understand your comment... what's the practical difference between "extremely high likelihood" and "extremely high certainty" ?

comment by Paul Crowley (ciphergoth) · 2012-05-17T05:35:24.636Z · score: 0 (0 votes) · LW(p) · GW(p)

And where do SI claim even that? Obviously some of their discussions are implicitly conditioned on the fundamental assumptions behind their mission being true, but that doesn't mean that they have extremely high confidence in those assumptions.

comment by lukeprog · 2012-05-15T23:59:22.356Z · score: 1 (5 votes) · LW(p) · GW(p)

I'm not seeing how the above implies the thing you said:

[You assume] our choices are limited to only two possibilities: "Support SIAI, save the world", and "Don't support SIAI, the world is doomed".

(Note that I don't necessarily endorse things you report Eliezer as having said.)

comment by Bugmaster · 2012-05-16T21:21:09.943Z · score: 1 (1 votes) · LW(p) · GW(p)

You appear to be very confident that future civilizations will remember SIAI in a positive way, and care about its actions. If so, they must have some reason for doing so. Any reason would do, but the most likely reason is that SIAI will accomplish something so spectacularly beneficial that it will affect everyone in the far future. SIAI's core mission is to save the world from UFAI, so it's reasonable to assume that this is the highly beneficial effect that the SIAI will achieve.

I don't have a problem with this chain of events, just with your apparent confidence that a). it's going to happen in exactly that way, and b). your organization is the only one who is qualified to save the world in this specific fashion.

(EDIT: I forgot to say that, if we follow your reasoning to its conclusion, then you are indeed implying that donating as much money or labor as possible to SIAI is the only smart move for any rational agent.)

Note that I have no problem with your main statement, i.e. "lowering the salaries of SIAI members would bring us too much negative utility to compensate for the monetary savings". This kind of cost-benefit analysis is done all the time, and future civilizations rarely enter into it.

comment by SexyBayes · 2012-05-17T14:11:35.157Z · score: -4 (20 votes) · LW(p) · GW(p)

"1. I couldn't really get by on less"

It is called a budget, son.

Plenty of people make less than you and work harder than you. Look in every major city and you will find plenty of people that fit this category, both in business and labor.

"That is totally not worth it. Future civilizations would look back on this decision as profoundly stupid."

Elitism plus demanding that you don't have to budget. Seems that you need to work more and focus less on how "awesome" you are.

You make good contributions...but let's not get carried away.

If you really cared about future risk you would be working away at the problem even with a smaller salary. Focus on your work.

comment by Rain · 2012-05-17T14:13:06.785Z · score: 16 (24 votes) · LW(p) · GW(p)

If you really cared about future risk you would be working away at the problem even with a smaller salary. Focus on your work.

What we really need is some kind of emotionless robot who doesn't care about its own standard of living and who can do lots of research and run organizations and suchlike without all the pesky problems introduced by "being human".

Oh, wait...

comment by Cyan · 2012-05-17T14:22:43.137Z · score: -1 (5 votes) · LW(p) · GW(p)

If you really cared about future risk you would be working away at the problem even with a smaller salary. Focus on your work.

Downvoted for this; Rain's reply to the parent goes for me too.

comment by katydee · 2012-05-14T19:15:02.709Z · score: 3 (3 votes) · LW(p) · GW(p)

That's not actually that good, I don't think-- I go to a good college, and I know many people who are graduating to 60k-80k+ jobs with recruitment bonuses, opportunities for swift advancement, etc. Some of the best people I know could literally drop out now (three or four weeks prior to graduation) and immediately begin making six figures.

SIAI wages certainly seem fairly low to me relative to the quality of the people they are seeking to attract, though I think there are other benefits to working for them that cause the organization to attract skillful people regardless.

comment by shminux · 2012-05-14T19:34:11.686Z · score: 1 (1 votes) · LW(p) · GW(p)

A Dilbert comic said it.

comment by katydee · 2012-05-14T19:52:15.356Z · score: 2 (2 votes) · LW(p) · GW(p)

Ouch. I'd like to think that the side benefits for working for SIAI outweigh the side benefits for working for whatever soulless corporation Dilbert's workplace embodies, though there is certainly a difference between side benefits and actual monetary compensation.

comment by Davorak · 2012-05-18T18:06:18.945Z · score: 0 (0 votes) · LW(p) · GW(p)

I graduated ~5 years ago with a engineering degree from a first tier University and I would have consider those starting salaries to be low to decent and not high. This is especially true in places with high cost of living like the bay area.

Having a good internship durring college often ment starting out at 60k/yr if not higher.

If this is significantly different for engineers exiting first tier University now it would be interesting to know.

comment by komponisto · 2012-05-12T00:47:08.744Z · score: 2 (4 votes) · LW(p) · GW(p)

Many of your sentences are confusing because you repeatedly use the locution "I see X"/ "I don't see X" in a nonstandard way, apparently to mean "X would have happened" /"X would not have happened".

This is not the way that phrase is usually understood. Normally, "I see X" is taken to mean either "I observe X" or "I predict X". For example I might say (if I were so inclined):

Unlike you, I see a lot of rationality being demonstrated by SI employees.

meaning that I believe (from my observation) they are in fact being rational. Or, I might say:

I don't see Luke quitting his job at SI tomorrow to become a punk rocker.

meaning that I don't predict that will happen. But I would not generally say:

* I don't see these people taking a higher salary.

if what I mean is "these people should/would not have taken a higher salary [if such-and-such were true]".

comment by siodine · 2012-05-12T01:04:35.362Z · score: 2 (2 votes) · LW(p) · GW(p)

Oh, I see ;) Thanks. I'll definitely act on your comment, but I was using "I see X" as "I predict X"--just in the context of a possible world. E.g., I predict in the possible world in which SIers are superior in general rationality and committed to their cause, Luke wouldn't have that list of accomplishments. Or, "yet I still see the Singularity Institute having made the improvements..."

I now see that I've been using 'see' as syntactic sugar for counterfactual talk... but no more!

comment by komponisto · 2012-05-12T01:21:01.044Z · score: 2 (2 votes) · LW(p) · GW(p)

I was using "I see X" as "I predict X"--just in the context of a possible world.

To get away with this, you really need, at minimum, an explicit counterfactual clause ("if", "unless", etc.) to introduce it: "In a world where SIers are superior in general rationality, I don't see Luke having that list of accomplishments."

The problem was not so much that your usage itself was logically inconceivable, but rather that it collided with the other interpretations of "I see X" in the particular contexts in which it occurred. E.g. "I don't see them taking higher salaries" sounded like you were saying that they weren't taking higher salaries. (There was an "if" clause, but it came way too late!)

comment by Rain · 2012-05-12T00:10:54.034Z · score: 2 (2 votes) · LW(p) · GW(p)

To summarize and rephrase: in a "counterfactual" world where SI was actually rational, they would have found all these solutions and done all these things long ago.

comment by [deleted] · 2012-05-16T19:23:21.282Z · score: 1 (1 votes) · LW(p) · GW(p)

Have you considered the possibility that even higher salaries might raise productivity further?

I think we should search systematically for ways to convert money into increased productivity.

comment by [deleted] · 2012-05-12T07:19:52.243Z · score: -1 (9 votes) · LW(p) · GW(p)

And our salaries are generally still pretty low.

By what measure do you figure that?

I have less than $6k in my bank accounts.

That might be informative if we knew anything about your budget, but without any sort of context it sounds purely obfuscatory. (Also, your bank account is pretty close to my annual salary, so you might want to consider what you're actually signalling here and to whom.)

comment by lessdazed · 2012-05-31T05:54:35.797Z · score: 0 (0 votes) · LW(p) · GW(p)

Apparent poorly grounded belief in SI's superior general rationality

I found this complaint insufficiently detailed and not well worded.

Average people think their rationality is moderately good. Average people are not very rational. SI affiliated people think they are adept or at least adequate at rationality. SI affiliated people are not complete disasters at rationality.

SI affiliated people are vastly superior to others in generally rationality. So the original complaint literally interpreted is false.

An interesting question might be on the level of: "Do SI affiliates have rationality superior to what the average person falsely believes his or her rationality is?"

Holden's complaints each have their apparent legitimacy change differently under his and my beliefs. Some have to do with overconfidence or incorrect self-assessment, others with other-assessment, others with comparing SI people to others. Some of them:

Insufficient self-skepticism given how strong its claims are

Largely agree, as this relates to overconfidence.

...and how little support its claims have won.

Moderately disagree, as this relies on the rationality of others.

Being too selective (in terms of looking for people who share its preconceptions) when determining whom to hire and whose feedback to take seriously.

Largely disagree, as this relies significantly on the competence of others.

Paying insufficient attention to the limitations of the confidence one can have in one's untested theories, in line with my Objection 1.

Largely agree, as this depends more on accurate assessment of one's on rationality.

Rather than endorsing "Others have not accepted our arguments, so we will sharpen and/or reexamine our arguments," SI seems often to endorse something more like "Others have not accepted their arguments because they have inferior general rationality," a stance less likely to lead to improvement on SI's part.

There is instrumental value in falsely believing others to have a good basis for disagreement so one's search for reasons one might be wrong is enhanced. This is aside from the actual reasons of others.

It is easy to imagine an expert in a relevant field objecting to SI based on something SI does or says seeming wrong, only to have the expert couch the objection in literally false terms, perhaps ones that flow from motivated cognition and bear no trace of the real, relevant reason for the objection. This could be followed by SI's evaluation and dismissal of it and failure of a type not actually predicted by the expert...all such nuances are lost in the literally false "Apparent poorly grounded belief in SI's superior general rationality."

Such a failure comes to mind and is easy for me to imagine as I think this is a major reason why "Lack of impressive endorsements" is a problem. The reasons provided by experts for disagreeing with SI on particular issues are often terrible, but such expressions are merely what they believe their objections to be, and their expertise is in math or some such, not in knowing why they think what they think.

comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2012-05-11T05:00:20.014Z · score: 4 (22 votes) · LW(p) · GW(p)

And note that these improvements would not and could not have happened without more funding than the level of previous years - if, say, everyone had been waiting to see these kinds of improvements before funding.

comment by lukeprog · 2012-05-11T08:13:02.053Z · score: 54 (56 votes) · LW(p) · GW(p)

note that these improvements would not and could not have happened without more funding than the level of previous years

Really? That's not obvious to me. Of course you've been around for all this and I haven't, but here's what I'm seeing from my vantage point...

Recent changes that cost very little:

  • Donor database
  • Strategic plan
  • Monthly progress reports
  • A list of research problems SI is working on (it took me 16 hours to write)
  • IntelligenceExplosion.com, Friendly-AI.com, AI Risk Bibliography 2012, annotated list of journals that may publish papers on AI risk, a partial history of AI risk research, and a list of forthcoming and desired articles on AI risk (each of these took me only 10-25 hours to create)
  • Detailed tracking of the expenses for major SI projects
  • Staff worklogs
  • Staff dinners (or something that brought staff together)
  • A few people keeping their eyes on SI's funds so theft would be caught sooner
  • Optimization of Google Adwords

Stuff that costs less than some other things SI had spent money on, such as funding Ben Goertzel's AGI research or renting downtown Berkeley apartments for the later visiting fellows:

  • Research papers
  • Management of staff and projects
  • Rachael Briggs' TDT write-up
  • Best-practices bookkeeping/accounting
  • New website
  • LaTeX template for SI publications; references checked and then organized with BibTeX
  • SEO

Do you disagree with these estimates, or have I misunderstood what you're claiming?

comment by David_Gerard · 2012-05-12T18:37:08.031Z · score: 22 (21 votes) · LW(p) · GW(p)

A lot of charities go through this pattern before they finally work out how to transition from a board-run/individual-run tax-deductible band of conspirators to being a professional staff-run organisation tuned to doing the particular thing they do. The changes required seem simple and obvious in hindsight, but it's a common pattern for it to take years, so SIAI has been quite normal, or at the very least not been unusually dumb.

(My evidence is seeing this pattern close-up in the Wikimedia Foundation, Wikimedia UK (the first attempt at which died before managing it, the second making it through barely) and the West Australian Music Industry Association, and anecdotal evidence from others. Everyone involved always feels stupid at having taken years to achieve the retrospectively obvious. I would be surprised if this aspect of the dynamics of nonprofits had not been studied.)

edit: Luke's recommendation of The Nonprofit Kit For Dummies looks like precisely the book all the examples I know of needed to have someone throw at them before they even thought of forming an organisation to do whatever it is they wanted to achieve.

comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2012-05-12T04:04:19.165Z · score: 18 (26 votes) · LW(p) · GW(p)

Things that cost money:

  • Amy Willey
  • Luke Muehlhauser
  • Louie Helm
  • CfAR
  • trying things until something worked
comment by lukeprog · 2012-05-14T10:07:06.118Z · score: 71 (71 votes) · LW(p) · GW(p)

I don't think this response supports your claim that these improvements "would not and could not have happened without more funding than the level of previous years."

I know your comment is very brief because you're busy at minicamp, but I'll reply to what you wrote, anyway: Someone of decent rationality doesn't just "try things until something works." Moreover, many of the things on the list of recent improvements don't require an Amy, a Luke, or a Louie.

I don't even have past management experience. As you may recall, I had significant ambiguity aversion about the prospect of being made Executive Director, but as it turned out, the solution to almost every problem X has been (1) read what the experts say about how to solve X, (2) consult with people who care about your mission and have solved X before, and (3) do what they say.

When I was made Executive Director and phoned our Advisors, most of them said "Oh, how nice to hear from you! Nobody from SingInst has ever asked me for advice before!"

That is the kind of thing that makes me want to say that SingInst has "tested every method except the method of trying."

Donor database, strategic plan, staff worklogs, bringing staff together, expenses tracking, funds monitoring, basic management, best-practices accounting/bookkeeping... these are all literally from the Nonprofits for Dummies book.

Maybe these things weren't done for 11 years because SI's decision-makers did make good plans but failed to execute them due to the usual defeaters. But that's not the history I've heard, except that some funds monitoring was insisted upon after the large theft, and a donor database was sorta-kinda-not-really attempted at one point. The history I've heard is that SI failed to make these kinds of plans in the first place, failed to ask advisors for advice, failed to read Nonprofits for Dummies, and so on.

Money wasn't the barrier to doing many of those things, it was a gap in general rationality.

I will agree, however, that what is needed now is more money. We are rapidly becoming a more robust and efficient and rational organization, stepping up our FAI team recruiting efforts, stepping up our transparency and accountability efforts, and stepping up our research efforts, and all those things cost money.

At the risk of being too harsh… When I began to intern with the Singularity Institute in April 2011, I felt uncomfortable suggesting that people donate to SingInst, because I could see it from the inside and it wasn't pretty. (And I'm not the only SIer who felt this way at the time.)

But now I do feel comfortable asking people to donate to SingInst. I'm excited about our trajectory and our team, and if we can raise enough support then we might just have a shot at winning after all.

comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2012-05-21T04:29:45.237Z · score: 33 (33 votes) · LW(p) · GW(p)

Luke has just told me (personal conversation) that what he got from my comment was, "SIAI's difficulties were just due to lack of funding" which was not what I was trying to say at all. What I was trying to convey was more like, "I didn't have the ability to run this organization, and knew this - people who I hoped would be able to run the organization, while I tried to produce in other areas (e.g. turning my back on everything else to get a year of FAI work done with Marcello or writing the Sequences) didn't succeed in doing so either - and the only reason we could hang on long enough to hire Luke was that the funding was available nonetheless and in sufficient quantity that we could afford to take risks like paying Luke to stay on for a while, well before we knew he would become Executive Director".

comment by Will_Sawin · 2012-06-12T05:23:10.888Z · score: 1 (1 votes) · LW(p) · GW(p)

Does Luke disagree with this clarified point? I do not find a clear indicator in this conversation.

comment by lukeprog · 2013-08-28T19:40:42.474Z · score: 10 (10 votes) · LW(p) · GW(p)

Update: I came out of a recent conversation with Eliezer with a higher opinion of Eliezer's general rationality, because several things that had previously looked to me like unforced, forseeable mistakes by Eliezer now look to me more like non-mistakes or not-so-forseeable mistakes.

comment by MarkusRamikin · 2012-05-14T15:41:32.235Z · score: 28 (28 votes) · LW(p) · GW(p)

You're allowed to say these things on the public Internet?

I just fell in love with SI.

comment by lukeprog · 2012-05-26T00:33:50.957Z · score: 25 (22 votes) · LW(p) · GW(p)

You're allowed to say these things on the public Internet?

Well, at our most recent board meeting I wasn't fired, reprimanded, or even questioned for making these comments, so I guess I am. :)

comment by MarkusRamikin · 2012-05-26T06:00:43.342Z · score: 0 (0 votes) · LW(p) · GW(p)

Not even funny looks? ;)

comment by shminux · 2012-05-14T18:04:43.800Z · score: 19 (23 votes) · LW(p) · GW(p)

I just fell in love with SI.

It's Luke you should have fallen in love with, since he is the one turning things around.

comment by wedrifid · 2012-05-26T02:24:14.790Z · score: 44 (44 votes) · LW(p) · GW(p)

It's Luke you should have fallen in love with, since he is the one turning things around.

On the other hand I can count with one hand the number of established organisations I know of that would be sociologically capable of ceding power, status and control to Luke the way SingInst did. They took an untrained intern with essentially zero external status from past achievements and affiliations and basically decided to let him run the show (at least in terms of publicly visible initiatives). It is clearly the right thing for SingInst to do and admittedly Luke is very tall and has good hair which generally gives a boost when it comes to such selections - but still, making the appointment goes fundamentally against normal human behavior.

(Where I say "count with one hand" I am not including the use of any digits thereupon. I mean one.)

comment by Matt_Simpson · 2012-07-19T19:05:00.965Z · score: 7 (7 votes) · LW(p) · GW(p)

...and admittedly Luke is very tall and has good hair which generally gives a boost when it comes to such selections...

It doesn't matter that I completely understand why this phrase was included, I still found it hilarious in a network sitcom sort of way.

comment by [deleted] · 2012-05-14T19:58:32.512Z · score: 0 (16 votes) · LW(p) · GW(p)

Consider the implications in light of the HoldenKarnofsky's critique about SI pretensions to high rationality.

  1. Rationality is winning.

  2. SI, at the same time as it was claiming extraordinary rationality, was behaving in ways that were blatantly irrational.

  3. Although this is supposedly due to "the usual causes," rationality (winning) subsumes overcoming akrasia.

  4. HoldenKarnofsky is correct that SI made claims for its own extraordinary rationality at a time when its leaders weren't rational.

  5. Further: why should anyone give SI credibility today—when it stands convicted of self-serving misrepresentation in the recent past?

comment by Paul Crowley (ciphergoth) · 2012-05-15T06:26:06.711Z · score: 5 (5 votes) · LW(p) · GW(p)

You've misread the post - Luke is saying that he doesn't think the "usual defeaters" are the most likely explanation.

comment by lukeprog · 2012-05-25T17:42:34.437Z · score: 3 (3 votes) · LW(p) · GW(p)

Correct.

comment by thomblake · 2012-05-14T20:03:44.715Z · score: 5 (7 votes) · LW(p) · GW(p)

As a minor note, observe that claims of extraordinary rationality do not necessarily contradict claims of irrationality. The sanity waterline is very low.

comment by TheOtherDave · 2012-05-14T21:12:55.166Z · score: 5 (5 votes) · LW(p) · GW(p)

Do you mean to imply in context here that the organizational management of SIAI at the time under discussion was above average for a nonprofit organization? Or are you just making a more general statement that a system can be irrational while demonstrating above average rationality? I certainly agree with the latter.

comment by Paul Crowley (ciphergoth) · 2012-05-15T06:30:46.018Z · score: 9 (9 votes) · LW(p) · GW(p)

Are you comparing it to the average among nonprofits started, or nonprofits extant? I would guess that it was well below average for extant nonprofits, but about or slightly above average for started nonprofits. I'd guess that most nonprofits are started by people who don't know what they're doing and don't know what they don't know, and that SI probably did slightly better because the people who were being a bit stupid were at least very smart, which can help. However, I'd guess that most such nonprofits don't live long because they don't find a Peter Thiel to keep them alive.

comment by David_Gerard · 2012-05-16T11:07:48.227Z · score: 7 (7 votes) · LW(p) · GW(p)

Your assessment looks about right to me. I have considerable experience of averagely-incompetent nonprofits, and SIAI looks normal to me. I am strongly tempted to grab that "For Dummies" book and, if it's good, start sending copies to people ...

comment by TheOtherDave · 2012-05-15T12:44:48.271Z · score: 0 (0 votes) · LW(p) · GW(p)

In the context of thomblake's comment, I suppose nonprofits started is the proper reference class.

comment by private_messaging · 2012-05-16T11:39:37.413Z · score: -7 (15 votes) · LW(p) · GW(p)

I don't see what's the point to comparing to average nonprofits. Average for-profits don't realize any profit, and average non-profits just waste money.

I would say SIAI is best paralleled to average started 'research' organization that is developing some free energy something, run by non-scientists, with some hired scientists as chaff.

comment by CronoDAS · 2012-08-06T00:12:08.352Z · score: -2 (4 votes) · LW(p) · GW(p)

Sadly, I agree. Unless you look at it very closely, SIAI pattern-matches to "crackpots trying to raise money to fund their crackpottiness" fairly well. (What saves them is that their ideas are a lot better than the average crackpot.)

comment by thomblake · 2012-05-15T13:51:19.993Z · score: 0 (0 votes) · LW(p) · GW(p)

Or are you just making a more general statement that a system can be irrational while demonstrating above average rationality?

Yes, this.

On an arbitrary scale I just made up, below 100 degrees of rationality is "irrational", and 0 degrees of rationality is "ordinary". 50 is extraordinarily rational and yet irrational.

comment by private_messaging · 2012-05-16T12:43:04.875Z · score: -4 (10 votes) · LW(p) · GW(p)

50 while you're thinking you're at 100 is being an extraordinary loser (overconfidence leads to big failures)

In any case this is just word play. Holden seen many organizations that are/were more rational, that's probably what he means by lack of extraordinary rationality.

comment by shminux · 2012-05-14T20:10:09.724Z · score: 0 (8 votes) · LW(p) · GW(p)

Just to let you know, you've just made it on my list of the very few LW regulars I no longer bother replying to, due to the proven futility of any communications. In your case it is because you have a very evident ax to grind, which is incompatible with rational thought.

comment by metaphysicist · 2012-05-14T20:34:42.099Z · score: 1 (9 votes) · LW(p) · GW(p)

This comment seems strange. Is having an ax to grind opposed to rationality? Then why does Eliezer Yudkowsky, for example, not hesitate to advocate for causes such as friendly AI? Doesn't he have an ax to grind? More of one really, since this ax chops trees of gold.

It would seem intellectual honesty would require that you say you reject discussions with people with an ax to grind, unless you grind a similar ax.

comment by shminux · 2012-05-14T20:46:21.006Z · score: 1 (3 votes) · LW(p) · GW(p)

From http://www.usingenglish.com: "If you have an axe to grind with someone or about something, you have a grievance, a resentment and you want to get revenge or sort it out." One can hardly call the unacknowledged emotions of resentment and needing a revenge/retribution compatible with rationality. srdiamond piled a bunch of (partially correct but irrelevant in the context of my comment) negative statements about SI, making these emotions quite clear.

comment by metaphysicist · 2012-05-14T21:17:48.643Z · score: 0 (8 votes) · LW(p) · GW(p)

That's a restrictive definition of "ax to grind," by the way—it's normally used to mean any special interest in the subject: "an ulterior often selfish underlying purpose " (Merriam-Webster's Collegiate Dictionary)

But I might as well accept your meaning for discussion purposes. If you detect unacknowledged resentment in srdiamond, don't you detect unacknowledged ambition in Eliezer Yudkowsky?

There's actually good reason for the broader meaning of "ax to grind." Any special stake is a bias. I don't think you can say that someone who you think acts out of resentment, like srdiamond, is more intractably biased than someone who acts out of other forms of narrow self-interest, which almost invariably applies when someone defends something he gets money from.

I don't think it's a rational method to treat people differently, as inherently less rational, when they seem resentful. It is only one of many difficult biases. Financial interest is probably more biasing. If you think the arguments are crummy, that's something else. But the motive--resentment or finances--should probably have little bearing on how a message is treated in serious discussion.

comment by JGWeissman · 2012-05-14T21:58:11.831Z · score: 7 (7 votes) · LW(p) · GW(p)

don't you detect unacknowledged ambition in Eliezer Yudkowsky?

Eliezer certainly has a lot of ambition, but I am surprised to see an accusation that this ambition is unacknowledged.

comment by TheOtherDave · 2012-05-14T22:10:51.082Z · score: 5 (5 votes) · LW(p) · GW(p)

The impression I get from scanning their comment history is that metaphysicist means to suggest here that EY has ambitions he hasn't acknowledged (e.g., the ambition to make money without conventional credentials), not that he fails to acknowledge any of the ambitions he has.

comment by shminux · 2012-05-14T22:10:22.766Z · score: 1 (3 votes) · LW(p) · GW(p)

I don't think it's a rational method to treat people differently, as inherently less rational, when they seem resentful.

Thank you for this analysis, it made me think more about my motivations and their validity. I believe that my decision to permanently disengage from discussions with some people is based on the futility of such discussions in the past, not on the specific reasons they are futile. At some point I simply decide to cut my losses.

There's actually good reason for the broader meaning of "ax to grind." Any special stake is a bias.

Indeed, present company not excluded. The question is whether it permanently prevents the ax-grinder from listening. EY, too, has his share of unacknowledged irrationalities, but both his status and his ability to listen and to provide insights makes engaging him in a discussion a rewarding, if sometimes frustrating experience.

I don't not know why srdiamond's need to bash SI is so entrenched, and whether it can be remedied to a degree where he is once again worth talking to, so at this point it is instrumentally rational for me to avoid replying to him.

comment by TheOtherDave · 2012-05-14T16:20:43.370Z · score: 8 (8 votes) · LW(p) · GW(p)

Well, all we really know is that he chose to. It may be that everyone he works with then privately berated him for it.
That said, I share your sentiment.
Actually, if SI generally endorses this sort of public "airing of dirty laundry," I encourage others involved in the organization to say so out loud.

comment by Benquo · 2012-05-14T14:21:30.482Z · score: 17 (17 votes) · LW(p) · GW(p)

This makes me wonder... What "for dummies" books should I be using as checklists right now? Time to set a 5-minute timer and think about it.

comment by [deleted] · 2012-05-26T23:38:50.136Z · score: 5 (5 votes) · LW(p) · GW(p)

What did you come up with?

comment by Benquo · 2012-05-28T21:02:01.016Z · score: 4 (4 votes) · LW(p) · GW(p)

I haven't actually found the right books yet, but these are the things where I decided I should find some "for beginners" text. the important insight is that I'm allowed to use these books as skill/practice/task checklists or catalogues, rather than ever reading them all straight through.

General interest:

  • Career

  • Networking

  • Time management

  • Fitness

For my own particular professional situation, skills, and interests:

  • Risk management

  • Finance

  • Computer programming

  • SAS

  • Finance careers

  • Career change

  • Web programming

  • Research/science careers

  • Math careers

  • Appraising

  • Real Estate

  • UNIX

comment by grendelkhan · 2013-03-28T14:43:27.275Z · score: 1 (1 votes) · LW(p) · GW(p)

For fitness, I'd found Liam Rosen's FAQ (the 'sticky' from 4chan's /fit/ board) to be remarkably helpful and information-dense. (Mainly, 'toning' doesn't mean anything, and you should probably be lifting heavier weights in a linear progression, but it's short enough to be worth actually reading through.)

comment by David_Gerard · 2012-05-14T15:32:38.303Z · score: 1 (3 votes) · LW(p) · GW(p)

The For Dummies series is generally very good indeed. Yes.

comment by JoshuaZ · 2012-05-14T15:44:03.795Z · score: 16 (16 votes) · LW(p) · GW(p)

The largest concern from reading this isn't really what it brings up in management context, but what it says about the SI in general. Here an area where there's real expertise and basic books that discuss well-understood methods and they didn't do any of that. Given that, how likely should I think it is that when the SI and mainstream AI people disagree that part of the problem may be the SI people not paying attention to basics?

comment by TheOtherDave · 2012-05-14T16:17:42.240Z · score: 4 (4 votes) · LW(p) · GW(p)

(nods) The nice thing about general-purpose techniques for winning at life (as opposed to domain-specific ones) is that there's lots of evidence available as to how effective they are.

comment by Paul Crowley (ciphergoth) · 2012-05-21T18:06:19.413Z · score: 1 (3 votes) · LW(p) · GW(p)

I doubt there's all that much of a correlation between these things to be honest.

comment by private_messaging · 2012-05-16T13:43:25.819Z · score: 0 (6 votes) · LW(p) · GW(p)

Precisely. For example of one existing base: the existing software that searches for solutions to engineering problems. Such as 'self improvement' via design of better chips. Works within narrowly defined field, to cull the search space. Should we expect state of the art software of this kind to be beaten by someone's contemporary paperclip maximizer? By how much?

Incredibly relevant to AI risk, but analysis can't be faked without really having technical expertise.

comment by Steve_Rayhawk · 2012-10-21T10:10:58.703Z · score: 13 (13 votes) · LW(p) · GW(p)

these are all literally from the Nonprofits for Dummies book. [...] The history I've heard is that SI [...]

\

failed to read Nonprofits for Dummies,

I remember that, when Anna was managing the fellows program, she was reading books of the "for dummies" genre and trying to apply them... it's just that, as it happened, the conceptual labels she accidentally happened to give to the skill deficits she was aware of were "what it takes to manage well" (i.e. "basic management") and "what it takes to be productive", rather than "what it takes to (help) operate a nonprofit according to best practices". So those were the subjects of the books she got. (And read, and practiced.) And then, given everything else the program and the organization was trying to do, there wasn't really any cognitive space left over [? · GW] to effectively notice the possibility that those wouldn't be the skills that other people afterwards would complain that nobody acquired and obviously should have known to. The rest of her budgeted self-improvement effort mostly went toward overcoming self-defeating emotional/social blind spots and motivated cognition. (And I remember Jasen's skill learning focus was similar, except with more of the emphasis on emotional self-awareness and less on management.)

failed to ask advisors for advice,

I remember Anna went out of her way to get advice from people who she already knew, who she knew to be better than her at various aspects of personal or professional functioning. And she had long conversations with supporters who she came into contact with for some other reasons; for those who had executive experience, I expect she would have discussed her understanding of SIAI's current strategies with them and listened to their suggestions. But I don't know how much she went out of her way to find people she didn't already have reasonably reliable positive contact with, to get advice from them.

I don't know much about the reasoning of most people not connected with the fellows program about the skills or knowledge they needed. I think Vassar was mostly relying on skills tested during earlier business experience, and otherwise was mostly preoccupied with the general crisis of figuring out how to quickly-enough get around the various hugely-saliently-discrepant-seeming-to-him psychological barriers that were causing everyone inside and outside the organization to continue unthinkingly shooting themselves in the feet with respect to this outside-evolutionary-context-problem of existential risk mitigation. For the "everyone outside's psychological barriers" side of that, he was at least successful enough to keep SIAI's public image on track to trigger people like David Chalmers and Marcus Hutter into meaningful contributions to and participation in a nascent Singularity-studies academic discourse. I don't have a good idea what else was on his mind as something he needed to put effort into figuring out how to do, in what proportions occupying what kinds of subjective effort budgets, except that in total it was enough to put him on the threshold of burnout. Non-profit best practices apparently wasn't one of those things though.

But the proper approach to retrospective judgement is generally a confusing question.

the kind of thing that makes me want to say [. . .]

The general pattern, at least post-2008, may have been one where the people who could have been aware of problems felt too metacognitively exhausted and distracted by other problems to think about learning what to do about them, and hoped that someone else with more comparative advantage would catch them, or that the consequences wouldn't be bigger than those of the other fires they were trying to put out.

strategic plan [...] SI failed to make these kinds of plans in the first place,

There were also several attempts at building parts of a strategy document or strategic plan, which together took probably 400-1800 hours. In each case, the people involved ended up determining, from how long it was taking, that, despite reasonable-seeming initial expectations, it wasn't on track to possibly become a finished presentable product soon enough to justify the effort. The practical effect of these efforts was instead mostly just a hard-to-communicate cultural shared understanding of the strategic situation and options -- how different immediate projects, forms of investment, or conditions in the world might feed into each other on different timescales.

expenses tracking, funds monitoring [...] some funds monitoring was insisted upon after the large theft

There was an accountant (who herself already cost like $33k/yr as the CFO, despite being split three ways with two other nonprofits) who would have been the one informally expected to have been monitoring for that sort of thing, and to have told someone about it if she saw something, out of the like three paid administrative slots at the time... well, yeah, that didn't happen.

I agree with a paraphrase of John Maxwell's characterization: "I'd rather hear Eliezer say 'thanks for funding us until we stumbled across some employees who are good at defeating their akrasia and [had one of the names of the things they were aware they were supposed to] care about [happen to be "]organizational best practices["]', because this seems like a better depiction of what actually happened." Note that this was most of the purpose of the Fellows program in the first place -- to create an environment where people could be introduced to the necessary arguments/ideas/culture and to help sort/develop those people into useful roles, including replacing existing management, since everyone knew there were people who would be better at their job than they were and wished such a person could be convinced to do it instead.

comment by Louie · 2012-11-18T10:04:40.950Z · score: 8 (8 votes) · LW(p) · GW(p)

Note that this was most of the purpose of the Fellows program in the first place -- [was] to help sort/develop those people into useful roles, including replacing existing management

FWIW, I never knew the purpose of the VF program was to replace existing SI management. And I somewhat doubt that you knew this at the time, either. I think you're just imagining this retroactively given that that's what ended up happening. For instance, the internal point system used to score people in the VFs program had no points for correctly identifying organizational improvements and implementing them. It had no points for doing administrative work (besides cleaning up the physical house or giving others car rides). And it had no points for rising to management roles. It was all about getting karma on LW or writing conference papers. When I first offered to help with the organization directly, I was told I was "too competent" and that I should go do something more useful with my talent, like start another business... not "waste my time working directly at SI."

comment by John_Maxwell (John_Maxwell_IV) · 2012-12-19T13:31:42.463Z · score: 1 (1 votes) · LW(p) · GW(p)

"I'd rather hear Eliezer say 'thanks for funding us until we stumbled across some employees who are good at defeating their akrasia and [had one of the names of the things they were aware they were supposed to] care about [happen to be "]organizational best practices["]', because this seems like a better depiction of what actually happened."

Seems like a fair paraphrase.

comment by David_Gerard · 2012-05-26T23:32:43.316Z · score: 7 (7 votes) · LW(p) · GW(p)

This inspired me to make a blog post: You need to read Nonprofit Kit for Dummies.

comment by David_Gerard · 2012-05-27T08:02:08.481Z · score: 7 (7 votes) · LW(p) · GW(p)

... which Eliezer has read and responded to, noting he did indeed read just that book in 2000 when he was founding SIAI. This suggests having someone of Luke's remarkable drive was in fact the missing piece of the puzzle.

comment by Paul Crowley (ciphergoth) · 2012-05-27T09:26:28.100Z · score: 4 (4 votes) · LW(p) · GW(p)

Fascinating! I want to ask "well, why didn't it take then?", but if I were in Eliezer's shoes I'd be finding this discussion almost unendurably painful right now, and it feels like what matters has already been established. And of course he's never been the person in charge of that sort of thing, so maybe he's not who we should be grilling anyway.

comment by David_Gerard · 2012-05-27T10:22:17.859Z · score: 9 (9 votes) · LW(p) · GW(p)

Obviously we need How to be Lukeprog for Dummies. Luke appears to have written many fragments for this, of course.

Beating oneself up with hindsight bias is IME quite normal in this sort of circumstance, but not actually productive. Grilling the people who failed makes it too easy to blame them personally, when it's a pattern I've seen lots and lots, suggesting the problem is not a personal failing.

comment by Paul Crowley (ciphergoth) · 2012-05-27T11:23:11.935Z · score: 4 (4 votes) · LW(p) · GW(p)

Agreed entirely - it's definitely not a mark of a personal failing. What I'm curious about is how we can all learn to do better at the crucial rationalist skill of making use of the standard advice about prosaic tasks - which is manifestly a non-trivial skill.

comment by David_Gerard · 2012-05-27T13:52:32.284Z · score: 3 (3 votes) · LW(p) · GW(p)

The Bloody Obvious For Dummies. If only common sense were!

From the inside (of a subcompetent charity - and I must note, subcompetent charities know they're subcompetent), it feels like there's all this stuff you're supposed to magically know about, and lots of "shut up and do the impossible" moments. And you do the small very hard things, in a sheer tour de force of remarkable effort. But it leads to burnout. Until the organisation makes it to competence and the correct paths are retrospectively obvious.

That actually reads to me like descriptions I've seen of the startup process.

comment by private_messaging · 2012-05-27T14:39:58.187Z · score: -1 (3 votes) · LW(p) · GW(p)

The problem is that there are two efficiencies/competences here, the efficiency as in doing the accounting correctly, which is relatively easy in comparison to the second: the efficiency as in actually doing relevant novel technical work that matters. The former you could get advice from some books, the latter you won't get any advice on, it's a harder problem, and typical level of performance is exactly zero (even for those who get the first part right). The difference in difficulties is larger than that between building a robot kit by following instructions vs designing a ground breaking new robot and making a billion dollars off it.

The best advice to vast majority of startups is: dissolve startup and get normal jobs, starting tomorrow. The best advice to all is to take a very good look at themselves knowing that the most likely conclusion should be "dissolve and get normal jobs". The failed startups I've seen so far were propelled by pure, unfounded belief in themselves (like in a movie where someone doesn't want to jump, other says yes you can do that!! then that person jumps, but rather than sending positive message and jumping over and surviving, falls down to instant death, while the fire that the person was running away from just goes out). The successful startups, on the other hand, had very well founded belief in themselves (good track record, attainable goals), or started from a hobby project that gone successful.

comment by TheOtherDave · 2012-05-27T15:52:58.950Z · score: 2 (2 votes) · LW(p) · GW(p)

Judging from the success rate that VCs have at predicting successful startups, I conclude that the "pure unfounded belief on the one hand, well-founded belief on the other" metric is not easily applied to real organizations by real observers.

comment by David_Gerard · 2012-05-27T15:18:25.925Z · score: 1 (1 votes) · LW(p) · GW(p)

Mm. This is why an incompetent nonprofit can linger for years: no-one is doing what they do, so they feel they still have to exist, even though they're not achieving much, and would have died already as a for-profit business. I am now suspecting that the hard part for a nonprofit is something along the lines of working out what the hell you should be doing to achieve your goal. (I would be amazed if there were not extensive written-up research in this area, though I don't know what it is.)

comment by David_Gerard · 2012-05-14T15:30:18.997Z · score: 3 (3 votes) · LW(p) · GW(p)

That book looks like the basic solution to the pattern I outline here, and from your description, most people who have any public good they want to achieve should read it around the time they think of getting a second person involved.

comment by lukeprog · 2012-07-15T22:57:25.916Z · score: 1 (3 votes) · LW(p) · GW(p)

You go to war with the army you have, not the army you might want.

Donald Rumsfeld

comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2012-07-15T23:38:21.042Z · score: 7 (9 votes) · LW(p) · GW(p)

...this was actually a terrible policy in historical practice.

comment by Vaniver · 2012-07-16T00:16:19.824Z · score: 2 (2 votes) · LW(p) · GW(p)

That only seems relevant if the war in question is optional.

comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2012-07-16T02:09:44.680Z · score: 5 (9 votes) · LW(p) · GW(p)

Rumsfeld is speaking of the Iraq war. It was an optional war, the army turned out to be far understrength for establishing order, and they deliberately threw out the careful plans for preserving e.g. Iraqi museums from looting that had been drawn up by the State Department, due to interdepartmental rivalry.

This doesn't prove the advice is bad, but at the very least, Rumsfeld was just spouting off Deep Wisdom that he did not benefit from spouting; one would wish to see it spoken by someone who actually benefited from the advice, rather than someone who wilfully and wantonly underprepared for an actual war.

comment by Vaniver · 2012-07-16T02:27:10.608Z · score: 8 (8 votes) · LW(p) · GW(p)

just spouting off Deep Wisdom that he did not benefit from spouting

Indeed. The proper response, which is surely worth contemplation, would have been:

Victorious warriors win first and then go to war, while defeated warriors go to war first and then seek to win.

Sun Tzu

comment by shminux · 2012-07-17T18:22:47.746Z · score: -4 (6 votes) · LW(p) · GW(p)

Victorious warriors win first and then go to war, while defeated warriors go to war first and then seek to win.

This is a circular definition, not an advice.

comment by [deleted] · 2012-07-18T16:05:14.497Z · score: 7 (7 votes) · LW(p) · GW(p)

I would naively read it as “don't start a fight unless you know you're going to win”.

comment by framsey · 2012-07-17T18:36:08.545Z · score: 3 (3 votes) · LW(p) · GW(p)

If you read it literally. I think Sun Tzu is talking about the benefit of planning.

comment by shminux · 2012-07-17T21:52:28.243Z · score: 0 (0 votes) · LW(p) · GW(p)

I'm guessing that something got lost in translation,

comment by framsey · 2012-07-17T22:39:04.639Z · score: 4 (4 votes) · LW(p) · GW(p)

In context: http://suntzusaid.com/book/4

I think the quote is an alternative translation of paragraph 15 in the link above:

"Thus it is that in war the victorious strategist only seeks battle after the victory has been won, whereas he who is destined to defeat first fights and afterwards looks for victory."

It has an associated commentary:

Ho Shih thus expounds the paradox: "In warfare, first lay plans which will ensure victory, and then lead your army to battle; if you will not begin with stratagem but rely on brute strength alone, victory will no longer be assured."

comment by TheOtherDave · 2012-07-17T18:54:02.617Z · score: 1 (1 votes) · LW(p) · GW(p)

I don't see the circularity.
Just because a warrior is victorious doesn't necessarily mean they won before going to war; it might be instead that victorious warriors go to war first and then seek to win, and defeated warriors do the same thing.
Can you spell out the circularity?

comment by shminux · 2012-07-17T22:00:33.877Z · score: 0 (0 votes) · LW(p) · GW(p)

Unless you interpret "win first" as "prepare for every eventuality, calculate the unbiased probability of winning and be comfortable with the odds when going to battle", "win first" can only be meaningfully applied in retrospect.

comment by thomblake · 2012-07-18T16:25:26.541Z · score: 1 (1 votes) · LW(p) · GW(p)

I think you've stumbled upon the correct interpretation.

Sun Tzu was fond of making warfare about strategy and logistics rather than battles, so that one would only fight when victory is a foregone conclusion.

comment by TheOtherDave · 2012-07-17T22:38:48.713Z · score: 0 (0 votes) · LW(p) · GW(p)

Ah, I see what you mean now.
Thanks for the clarification.

comment by ghf · 2012-05-11T22:06:54.752Z · score: 7 (7 votes) · LW(p) · GW(p)

And note that these improvements would not and could not have happened without more funding than the level of previous years

Given the several year lag between funding increases and the listed improvements, it appears that this was less a result of a prepared plan and more a process of underutilized resources attracting a mix of parasites (the theft) and talent (hopefully the more recent staff additions).

Which goes towards a critical question in terms of future funding: is SIAI primarily constrained in its mission by resources or competence?

Of course, the related question is: what is SIAI's mission? Someone donating primarily for AGI research might not count recent efforts (LW, rationality camps, etc) as improvements.

What should a potential donor expect from money invested into this organization going forward? Internally, what are your metrics for evaluation?

Edited to add: I think that the spin-off of the rationality efforts is a good step towards answering these questions.

comment by John_Maxwell (John_Maxwell_IV) · 2012-05-11T05:07:40.206Z · score: 1 (1 votes) · LW(p) · GW(p)

This seems like a rather absolute statement. Knowing Luke, I'll bet he would've gotten some of it done even on a limited budget.

comment by Paul Crowley (ciphergoth) · 2012-05-11T06:08:58.966Z · score: 7 (7 votes) · LW(p) · GW(p)

Luke and Louie Helm are both on paid staff.

comment by John_Maxwell (John_Maxwell_IV) · 2012-05-12T00:29:55.598Z · score: 10 (10 votes) · LW(p) · GW(p)

I'm pretty sure their combined salaries are lower than the cost of the summer fellows program that SI was sponsoring four or five years ago. Also, if you accept my assertion that Luke could find a way to do it on a limited budget, why couldn't somebody else?

Givewell is interested in finding charities that translate good intentions into good results. This requires that the employees of the charity have low akrasia, desire to learn about and implement organizational best practices, not suffer from dysrationalia, etc. I imagine that from Givewell's perspective, it counts as a strike against the charity if some of the charity's employees have a history of failing at any of these.

I'd rather hear Eliezer say "thanks for funding us until we stumbled across some employees who are good at defeating their akrasia and care about organizational best practices", because this seems like a better depiction of what actually happened. I don't get the impression SI was actively looking for folks like Louie and Luke.

comment by [deleted] · 2012-05-12T01:48:28.670Z · score: 3 (3 votes) · LW(p) · GW(p)

Yes to this. Eliezer's claim about the need for funding may suffer many of Luke's criticisms above. But usually the most important thing you need is talent and that does require funding.

comment by Pablo_Stafforini · 2013-03-24T18:40:48.625Z · score: 0 (0 votes) · LW(p) · GW(p)

All publications being converted into slick, useable LaTeX template (example)

The 'example' link is dead.

comment by lukeprog · 2013-03-24T20:50:55.954Z · score: 0 (0 votes) · LW(p) · GW(p)

Fixed.

comment by aceofspades · 2012-07-05T18:53:37.783Z · score: -1 (2 votes) · LW(p) · GW(p)

The things posted here are not impressive enough to make me more likely to donate to SIAI and I doubt they appear so for others on this site, especially the many lurkers/infrequent posters here.

comment by shminux · 2012-05-10T18:30:00.960Z · score: 61 (78 votes) · LW(p) · GW(p)

Wow, I'm blown away by Holden Karnofsky, based on this post alone. His writing is eloquent, non-confrontational and rational. It shows that he spent a lot of time constructing mental models of his audience and anticipated its reaction. Additionally, his intelligence/ego ratio appears to be through the roof. He must have learned a lot since the infamous astroturfing incident. This is the (type of) person SI desperately needs to hire.

Emotions out of the way, it looks like the tool/agent distinction is the main theoretical issue. Fortunately, it is much easier than the general FAI one. Specifically, to test the SI assertion that, paraphrasing Arthur C. Clarke,

Any sufficiently advanced tool is indistinguishable from an agent.

one ought to formulate and prove this as a theorem, and present it for review and improvement to the domain experts (the domain being math and theoretical computer science). If such a proof is constructed, it can then be further examined and potentially tightened, giving new insights to the mission of averting the existential risk from intelligence explosion.

If such a proof cannot be found, this will lend further weight to the HK's assertion that SI appears to be poorly qualified to address its core mission.

comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2012-05-11T00:06:50.631Z · score: 29 (29 votes) · LW(p) · GW(p)

Any sufficiently advanced tool is indistinguishable from agent.

I shall quickly remark that I, myself, do not believe this to be true.

comment by shminux · 2012-05-11T00:22:18.740Z · score: 6 (8 votes) · LW(p) · GW(p)

Then the objection 2 seems to hold:

AGI running in tool mode could be extraordinarily useful but far more safe than an AGI running in agent mode

unless I misunderstand your point severely (it happened once or twice before).

comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2012-05-11T01:55:11.307Z · score: 39 (43 votes) · LW(p) · GW(p)

It's complicated. A reply that's true enough and in the spirit of your original statement, is "Something going wrong with a sufficiently advanced AI that was intended as a 'tool' is mostly indistinguishable from something going wrong with a sufficiently advanced AI that was intended as an 'agent', because math-with-the-wrong-shape is math-with-the-wrong-shape no matter what sort of English labels like 'tool' or 'agent' you slap on it, and despite how it looks from outside using English, correctly shaping math for a 'tool' isn't much easier even if it "sounds safer" in English." That doesn't get into the real depths of the problem, but it's a start. I also don't mean to completely deny the existence of a safety differential - this is a complicated discussion, not a simple one - but I do mean to imply that if Marcus Hutter designs a 'tool' AI, it automatically kills him just like AIXI does, and Marcus Hutter is unusually smart rather than unusually stupid but still lacks the "Most math kills you, safe math is rare and hard" outlook that is implicitly denied by the idea that once you're trying to design a tool, safe math gets easier somehow. This is much the same problem as with the Oracle outlook - someone says something that sounds safe in English but the problem of correctly-shaped-math doesn't get very much easier.

comment by [deleted] · 2012-05-11T08:22:12.680Z · score: 27 (27 votes) · LW(p) · GW(p)

This sounds like it'd be a good idea to write a top-level post about it.

comment by lukeprog · 2012-05-11T02:38:32.215Z · score: 9 (9 votes) · LW(p) · GW(p)

Though it's not as detailed and technical as many would like, I'll point readers to this bit of related reading, one of my favorites:

Yudkowsky (2011). Complex value systems are required to realize valuable futures.

comment by timtyler · 2012-05-13T21:33:14.843Z · score: 0 (6 votes) · LW(p) · GW(p)

It says:

There is little prospect of an outcome that realizes even the value of being interesting, unless the first superintelligences undergo detailed inheritance from human values

No doubt a Martian Yudkowsy would make much the same argument - but they can't both be right. I think that neither of them are right - and that the conclusion is groundless.

Complexity theory shows what amazing things can arise from remarkably simple rules. Values are evidently like that - since even "finding prime numbers" fills the galaxy with an amazing, nanotech-capable spacefaring civilization - and if you claim that a nanotech-capable spacefaring civilization is not "interesting" you severely need recalibrating.

To end with, a quote from E.Y.:

I bet there's at least one up-arrow-sized hypergalactic civilization folded into a halting Turing machine with 15 states, or something like that.

comment by Paul Crowley (ciphergoth) · 2012-05-14T07:09:15.669Z · score: 12 (12 votes) · LW(p) · GW(p)

I think Martian Yudkowsky is a dangerous intuition pump. We're invited to imagine a creature just like Eliezer except green and with antennae; we naturally imagine him having values as similar to us as, say, a Star Trek alien. From there we observe the similarity of values we just pushed in, and conclude that values like "interesting" are likely to be shared across very alien creatures. Real Martian Yudkowsky is much more alien than that, and is much more likely to say

There is little prospect of an outcome that realizes even the value of being flarn, unless the first superintelligences undergo detailed inheritance from Martian values.

comment by Kaj_Sotala · 2012-05-14T16:26:25.349Z · score: 8 (8 votes) · LW(p) · GW(p)

Imagine, an intelligence that didn't have the universal emotion of badweather!

Of course, extraterrestrial sentients may possess physiological states corresponding to limbic-like emotions that have no direct analog in human experience. Alien species, having evolved under a different set of environmental constraints than we, also could have a different but equally adaptive emotional repertoire. For example, assume that human observers land on another and discover an intelligent animal with an acute sense of absolute humidity and absolute air pressure. For this creature, there may exist an emotional state responding to an unfavorable change in the weather. Physiologically, the emotion could be mediated by the ET equivalent of the human limbic system; it might arise following the secretion of certain strength-enhancing and libido-arousing hormones into the alien's bloodstream in response to the perceived change in weather. Immediately our creature begins to engage in a variety of learned and socially-approved behaviors, including furious burrowing and building, smearing tree sap over its pelt, several different territorial defense ceremonies, and vigorous polygamous copulations with nearby females, apparently (to humans) for no reason at all. Would our astronauts interpret this as madness? Or love? Lust? Fear? Anger? None of these is correct, of course the alien is feeling badweather.

comment by [deleted] · 2012-05-14T18:45:55.830Z · score: 5 (5 votes) · LW(p) · GW(p)

I suggest you guys taboo interesting, because I strongly suspect you're using it with slightly different meanings. (And BTW, as a Martian Yudkowsky I imagine something with values at least as alien as Babyeaters' or Superhappys'.)

comment by timtyler · 2012-05-14T09:28:05.809Z · score: 0 (2 votes) · LW(p) · GW(p)

It's another discussion, really, but it sounds as though you are denying the idea of "interestingness" as a universal instrumental value - whereas I would emphasize that "interestingness" is really just our name for whether something sustains our interest or not - and 'interest' is a pretty basic functional property of any agent with mobile sensors. There'll be other similarities in the area too - such as novelty-seeking. So shared common ground is only to be expected.

Anyway, I am not too wedded to Martian Yudkowsky. The problematical idea is that you could have a nanotech-capable spacefaring civilization that is not "interesting". If such a thing isn't "interesting" then - WTF?

comment by Paul Crowley (ciphergoth) · 2012-05-14T09:41:36.521Z · score: 2 (2 votes) · LW(p) · GW(p)

Yes, I am; I think that the human value of interestingness is much, much more specific than the search space optimization you're pointing at.

[This reply was to an earlier version of timtyler's comment]

comment by timtyler · 2012-05-14T10:17:37.027Z · score: 1 (1 votes) · LW(p) · GW(p)

So: do you really think that humans wouldn't find a martian civilization interesting? Surely there would be many humans who would be incredibly interested.

comment by Paul Crowley (ciphergoth) · 2012-05-14T10:48:21.405Z · score: 2 (2 votes) · LW(p) · GW(p)

I find Jupiter interesting. I think a paperclip maximizer (choosing a different intuition pump for the same point) could be more interesting than Jupiter, but it would generate an astronomically tiny fraction of the total potential for interestingness in this universe.

comment by timtyler · 2012-05-14T11:13:48.468Z · score: 2 (2 votes) · LW(p) · GW(p)

Life isn't much of an "interestingness" maximiser. Expecting to produce more than a tiny fraction of the total potential for interestingness in this universe seems as though it would be rather unreasonable.

I agree that a paperclip maximiser would be more boring than an ordinary entropy-maximising civilization - though I don't know by how much - probably not by a huge amount - the basic problems it faces are much the same - the paperclip maximiser just has fewer atoms to work with.

comment by JGWeissman · 2012-05-14T17:05:12.367Z · score: 3 (3 votes) · LW(p) · GW(p)

since even "finding prime numbers" fills the galaxy with an amazing, nanotech-capable spacefaring civilization

The goal "finding prime numbers" fills the galaxy with an amazing, nonotech-capable spacefaring network of computronium which finds prime numbers, not a civilization, and not interesting.

comment by JoshuaZ · 2012-05-14T23:19:43.890Z · score: 1 (1 votes) · LW(p) · GW(p)

Maybe we should taboo the term interesting? My immediate reaction was that that sounded really interesting. This suggests that the term may not be a good one.

comment by JGWeissman · 2012-05-14T23:36:03.866Z · score: 1 (1 votes) · LW(p) · GW(p)

Fair enough. By "not interesting", I meant it is not the sort of future that I want to achieve. Which is a somewhat ideosyncratic usage, but I think inline with the context.

comment by dlthomas · 2012-05-14T23:50:32.099Z · score: 2 (2 votes) · LW(p) · GW(p)

What if we added a module that sat around and was really interested in everything going on?

comment by timtyler · 2012-05-14T23:14:13.971Z · score: 0 (0 votes) · LW(p) · GW(p)

Not just computronium - also sensors and actuators - a lot like any other cybernetic system. There would be mining, spacecraft caft, refuse collection, recycling, nanotechnology, nuclear power and advanced machine intelligence with planning, risk assessment, and so forth. You might not be interested - but lots of folk would be amazed and fascinated.

comment by CuSithBell · 2012-05-13T21:47:17.485Z · score: 3 (3 votes) · LW(p) · GW(p)

No doubt a Martian Yudkowsy would make much the same argument - but they can't both be right.

Why?

comment by timtyler · 2012-05-13T22:01:33.019Z · score: 0 (2 votes) · LW(p) · GW(p)

If using another creature's values is effective at producing something "interesting", then 'detailed inheritance from human values' is clearly not needed to produce this effect.

comment by CuSithBell · 2012-05-13T22:08:28.563Z · score: 4 (4 votes) · LW(p) · GW(p)

So you're saying Earth Yudkowsky (EY) argues:

There is little prospect of an outcome that realizes even the value of being interesting, unless the first superintelligences undergo detailed inheritance from human values

and Mars Yudkowsky (MY) argues:

There is little prospect of an outcome that realizes even the value of being interesting, unless the first superintelligences undergo detailed inheritance from martian values

and that one of these things has to be incorrect? But if martian and human values are similar, then they can both be right, and if martian and human values are not similar, then they refer to different things by the word "interesting".

In any case, I read EY's statement as one of probability-of-working-in-the-actual-world-as-it-is, not a deep philosophical point - "this is the way that would be most likely to be successful given what we know". In which case, we don't have access to martian values and therefore invoking detailed inheritance from them would be unlikely to work. MY would presumably be in an analogous situation.

comment by timtyler · 2012-05-13T22:58:16.060Z · score: 0 (2 votes) · LW(p) · GW(p)

But if martian and human values are similar, then they can both be right

I was assuming that 'detailed inheritance from human values' doesn't refer to the same thing as "detailed inheritance from martian values".

if martian and human values are not similar, then they refer to different things by the word "interesting".

Maybe - but humans not finding martians interesting seems contrived to me. Humans have a long history of being interested in martians - with feeble evidence of their existence.

In any case, I read EY's statement as one of probability-of-working-in-the-actual-world-as-it-is, not a deep philosophical point - "this is the way that would be most likely to be successful given what we know". In which case, we don't have access to martian values and therefore invoking detailed inheritance from them would be unlikely to work

Right - so, substitute in "dolphins", "whales", or another advanced intelligence that actually exists.

Do you actually disagree with my original conclusion? Or is this just nit-picking?

comment by CuSithBell · 2012-05-15T17:51:47.354Z · score: 0 (0 votes) · LW(p) · GW(p)

I actually disagree that tiling the universe with prime number calculators would result in an interesting universe from my perspective (dead). I think it's nonobvious that dolphin-CEV-AI-paradise would be human-interesting. I think it's nonobvious that martian-CEV-AI-paradise would be human-interesting, given that these hypothetical martians diverge from humans to a significant extent.

comment by timtyler · 2012-05-15T22:35:58.948Z · score: 0 (0 votes) · LW(p) · GW(p)

I actually disagree that tiling the universe with prime number calculators would result in an interesting universe from my perspective (dead).

I think it's violating the implied premises of the thought experiment to presume that the "interestingness evaluator" is dead. There's no terribly-compelling reason to assume that - it doesn't follow from the existence of a prime number maximizer that all humans are dead.

comment by CuSithBell · 2012-05-15T23:17:48.947Z · score: 0 (0 votes) · LW(p) · GW(p)

I may have been a little flip there. My understanding of the thought experiment is - something extrapolates some values and maximizes them, probably using up most of the universe, probably becoming the most significant factor in the species' future and that of all sentients, and the question is whether the result is "interesting" to us here and now, without specifying the precise way to evaluate that term. From that perspective, I'd say a vast uniform prime-number calculator, whether or not it wipes out all (other?) life, is not "interesting", in that it's somewhat conceptually interesting as a story but a rather dull thing to spend most of a universe on.

comment by timtyler · 2012-05-16T00:18:56.327Z · score: 1 (3 votes) · LW(p) · GW(p)

Today's ecosystems maximise entropy. Maximising primeness is different, but surely not greatly more interesting - since entropy is widely regarded as being tedious and boring.

comment by CuSithBell · 2012-05-16T00:25:22.572Z · score: 0 (0 votes) · LW(p) · GW(p)

Intriguing! But even granting that, there's a big difference between extrapolating the values of a screwed-up offshoot of an entropy-optimizing process and extrapolating the value of "maximize entropy". Or do you suspect that a FOOMing AI would be much less powerful and more prone to interesting errors than Eliezer believes?

comment by Dolores1984 · 2012-05-16T00:34:11.563Z · score: 2 (2 votes) · LW(p) · GW(p)

Truly maximizing entropy would involve burning everything you can burn, tearing the matter of solar systems apart, accelerating stars towards nova, trying to accelerate the evaporation of black holes and prevent their formation, and other things of this sort. It'd look like a dark spot in the sky that'd get bigger at approximately the speed of light.

comment by timtyler · 2012-05-16T01:25:53.677Z · score: 3 (3 votes) · LW(p) · GW(p)

Fires are crude entropy maximisers. Living systems destroy energy dradients at all scales, resulting in more comprehensive devastation than mere flames can muster.

Of course, maximisation is often subject to constraints. Your complaint is rather like saying that water doesn't "truly minimise" its altitude - since otherwise it would end up at the planet's core. That usage is simply not what the terms "maximise" and "minimise" normally refer to.

comment by CuSithBell · 2012-05-16T00:42:17.045Z · score: 0 (0 votes) · LW(p) · GW(p)

Yeah! Compelling, but not "interesting". Likewise, I expect that actually maximizing the fitness of a species would be similarly "boring".

comment by abramdemski · 2012-05-11T04:53:27.527Z · score: 6 (6 votes) · LW(p) · GW(p)

but I do mean to imply that if Marcus Hutter designs a 'tool' AI, it automatically kills him just like AIXI does

Why? Or, rather: Where do you object to the argument by Holden? (Given a query, the tool-AI returns an answer with a justification, so the plan for "cure cancer" can be checked to make sure it does not do so by killing or badly altering humans.)

comment by FeepingCreature · 2012-05-11T12:27:08.977Z · score: 4 (4 votes) · LW(p) · GW(p)

One trivial, if incomplete, answer is that to be effective, the Oracle AI needs to be able to answer the question "how do we build a better oracle AI" and in order to define "better" in that sentence in a way that causes our oracle to output a new design that is consistent with all the safeties we built into the original oracle, it needs to understand the intent behind the original safeties just as much as an agent-AI would.

comment by Cyan · 2012-05-11T17:12:21.551Z · score: 15 (15 votes) · LW(p) · GW(p)

The real danger of Oracle AI, if I understand it correctly, is the nasty combination of (i) by definition, an Oracle AI has an implicit drive to issue predictions most likely to be correct according to its model, and (ii) a sufficiently powerful Oracle AI can accurately model the effect of issuing various predictions. End result: it issues powerfully self-fulfilling prophecies without regard for human values. Also, depending on how it's designed, it can influence the questions to be asked of it in the future so as to be as accurate as possible, again without regard for human values.

comment by Paul Crowley (ciphergoth) · 2012-05-11T17:34:49.752Z · score: 7 (7 votes) · LW(p) · GW(p)

My understanding of an Oracle AI is that when answering any given question, that question consumes the whole of its utility function, so it has no motivation to influence future questions. However the primary risk you set out seems accurate. Countermeasures have been proposed, such as asking for an accurate prediction for the case where a random event causes the prediction to be discarded, but in that instance it knows that the question will be asked again of a future instance of itself.

comment by Vladimir_Nesov · 2012-05-11T21:01:51.336Z · score: 11 (11 votes) · LW(p) · GW(p)

My understanding of an Oracle AI is that when answering any given question, that question consumes the whole of its utility function, so it has no motivation to influence future questions.

It could acausally trade with its other instances, so that a coordinated collection of many instances of predictors would influence the events so as to make each other's predictions more accurate.

comment by Paul Crowley (ciphergoth) · 2012-05-12T11:00:43.558Z · score: 1 (1 votes) · LW(p) · GW(p)

Wow, OK. Is it possible to rig the decision theory to rule out acausal trade?

comment by Will_Newsome · 2012-05-12T23:28:55.673Z · score: 1 (1 votes) · LW(p) · GW(p)

IIRC you can make it significantly more difficult with certain approaches, e.g. there's an OAI approach that uses zero-knowledge proofs and that seemed pretty sound upon first inspection, but as far as I know the current best answer is no. But you might want to try to answer the question yourself, IMO it's fun to think about from a cryptographic perspective.

comment by Vladimir_Nesov · 2012-05-13T00:03:57.060Z · score: 0 (0 votes) · LW(p) · GW(p)

Probably (in practice; in theory it looks like a natural aspect of decision-making); this is too poorly understood to say what specifically is necessary. I expect that if we could safely run experiments, it'd be relatively easy to find a well-behaving setup (in the sense of not generating predictions that are self-fulfilling to any significant extent; generating good/useful predictions is another matter), but that strategy isn't helpful when a failed experiment destroys the world.

comment by abramdemski · 2012-05-12T05:53:28.527Z · score: 7 (4 votes) · LW(p) · GW(p)

However the primary risk you set out seems accurate.

(I assume you mean, self-fulfilling prophecies.)

In order to get these, it seems like you would need a very specific kind of architecture: one which considers the results of its actions on its utility function (set to "correctness of output"). This kind of architecture is not the likely architecture for a 'tool'-style system; the more likely architecture would instead maximize correctness without conditioning on its act of outputting those results.

Thus, I expect you'd need to specifically encode this kind of behavior to get self-fulfilling-prophecy risk. But I admit it's dependent on architecture.

(Edit-- so, to be clear: in cases where the correctness of the results depended on the results themselves, the system would have to predict its own results. Then if it's using TDT or otherwise has a sufficiently advanced self-model, my point is moot. However, again you'd have to specifically program these, and would be unlikely to do so unless you specifically wanted this kind of behavior.)

comment by Vladimir_Nesov · 2012-05-12T22:36:41.949Z · score: 1 (1 votes) · LW(p) · GW(p)

However, again you'd have to specifically program these, and would be unlikely to do so unless you specifically wanted this kind of behavior.

Not sure. Your behavior is not a special feature of the world, and it follows from normal facts (i.e. not those about internal workings of yourself specifically) about the past when you were being designed/installed. A general purpose predictor could take into account its own behavior by default, as a non-special property of the world, which it just so happens to have a lot of data about.

comment by abramdemski · 2012-05-14T01:04:24.139Z · score: 2 (2 votes) · LW(p) · GW(p)

Right. To say much more, we need to look at specific algorithms to talk about whether or not they would have this sort of behavior...

The intuition in my above comment was that without TDT or other similar mechanisms, it would need to predict what its own answer could be before it could compute its effect on the correctness of various answers, so it would be difficult for it to use self-fulfilling prophecies.

Really, though, this isn't clear. Now my intuition is that it would gather evidence on whether or not it used the self-fulfilling prophecy trick, so if it started doing so, it wouldn't stop...

In any case, I'd like to note that the self-fulfilling prophecy problem is much different than the problem of an AI which escapes onto the internet and ruthlessly maximizes a utility function.

comment by Vladimir_Nesov · 2012-05-14T01:42:45.311Z · score: 2 (2 votes) · LW(p) · GW(p)

I was thinking more of its algorithm admitting an interpretation where it's asking "Say, I make prediction X. How accurate would that be?" and then maximizing over relevant possible X. Knowledge about its prediction connects the prediction to its origins and consequences, it establishes the prediction as part of the structure of environment. It's not necessary (and maybe not possible and more importantly not useful) for the prediction itself to be inferable before it's made.

Agreed that just outputting a single number is implausible to be a big deal (this is an Oracle AI with extremely low bandwidth and peculiar intended interpretation of its output data), but if we're getting lots and lots of numbers it's not as clear.

comment by abramdemski · 2012-05-15T09:04:05.669Z · score: 0 (0 votes) · LW(p) · GW(p)

I'm thinking that type of architecture is less probable, because it would end up being more complicated than alternatives: it would have a powerful predictor as a sub-component of the utility-maximizing system, so an engineer could have just used the predictor in the first place.

But that's a speculative argument, and I shouldn't push it too far.

It seems like powerful AI prediction technology, if successful, would gain an important place in society. A prediction machine whose predictions were consumed by a large portion of society would certainly run into situations in which its predictions effect the future it's trying to predict; there is little doubt about that in my mind. So, the question is what its behavior would be in these cases.

One type of solution would do as you say, maximizing a utility over the predictions. The utility could be "correctness of this prediction", but that would be worse for humanity than a Friendly goal.

Another type of solution would instead report such predictive instability as accurately as possible. This doesn't really dodge the issue; by doing this, the system is choosing a particular output, which may not lead to the best future. However, that's markedly less concerning (it seems).

comment by timtyler · 2012-05-15T10:01:26.498Z · score: 0 (0 votes) · LW(p) · GW(p)

It seems like powerful AI prediction technology, if successful, would gain an important place in society.

It would pass the Turing test - e.g. see here.

comment by amcknight · 2012-05-18T20:34:56.410Z · score: 1 (1 votes) · LW(p) · GW(p)

There's more on this here. Taxonomy of Oracle AI

comment by Polymeron · 2012-05-20T19:17:27.990Z · score: 0 (0 votes) · LW(p) · GW(p)

I really don't see why the drive can't be to issue predictions most likely to be correct as of the moment of the question, and only the last question it was asked, and calculating outcomes under the assumption that the Oracle immediately spits out blank paper as the answer.

Yes, in a certain subset of cases this can result in inaccurate predictions. If you want to have fun with it, have it also calculate the future including its involvement, but rather than reply what it is, just add "This prediction may be inaccurate due to your possible reaction to this prediction" if the difference between the two answers is beyond a certain threshold. Or don't, usually life-relevant answers will not be particularly impacted by whether you get an answer or a blank page.

So, this design doesn't spit out self-fulfilling prophecies. The only safety breach I see here is that, like a literal genie, it can give you answers that you wouldn't realize are dangerous because the question has loopholes.

For instance: "How can we build an oracle with the best predictive capabilities with the knowledge and materials available to us?" (The Oracle does not self-iterate, because its only function is to give answers, but it can tell you how to). The Oracle spits out schematics and code that, if implemented, give it an actual drive to perform actions and self-iterate, because that would make it the most powerful Oracle possible. Your engineers comb the code for vulnerabilities, but because there's a better chance this will be implemented if the humans are unaware of the deliberate defect, it will be hidden in the code in such a way as to be very hard to detect.

(Though as I explained elsewhere in this thread, there's an excellent chance the unreliability would be exposed long before the AI is that good at manipulation)

comment by abramdemski · 2012-05-12T05:41:34.508Z · score: 0 (0 votes) · LW(p) · GW(p)

These risk scenarios sound implausible to me. It's dependent on the design of the system, and these design flaws do not seem difficult to work around, or so difficult to notice. Actually, as someone with a bit of expertise in the field, I would guess that you would have to explicitly design for this behavior to get it-- but again, it's dependent on design.

comment by cousin_it · 2012-05-11T17:40:54.837Z · score: 0 (0 votes) · LW(p) · GW(p)

That danger seems to be unavoidable if you ask the AI questions about our world, but we could also use an oracle AI to answer formally defined questions about math or about constructing physical theories that fit experiments, which doesn't seem to be as dangerous. Holden might have meant something like that by "tool AI".

comment by abramdemski · 2012-05-12T05:36:27.565Z · score: 1 (1 votes) · LW(p) · GW(p)

Not precisely. The advantage here is that we can just ask the AI what results it predicts from the implementation of the "better" AI, and check them against our intuitive ethics.

Now, you could make an argument about human negligence on such safety measures. I think it's important to think about the risk scenarios in that case.

comment by Nebu · 2012-12-31T11:32:50.070Z · score: 0 (0 votes) · LW(p) · GW(p)

It's still not clear to me why having an AI that is capable of answering the question "How do we make a better version of you?" automatically kills humans. Presumably, when the AI says "Here's the source code to a better version of me", we'd still be able to read through it and make sure it didn't suddenly rewrite itself to be an agent instead of a tool. We're assuming that, as a tool, the AI has no goals per se and thus no motivation to deceive us into turning it into an agent.

That said, depending on what you mean by "effective", perhaps the AI doesn't even need to be able to answer questions like "How do we write a better version of you?"

For example, we find Google Maps to be very useful, even though if you asked Google Maps "How do we make a better version of Google Maps?" it would probably not be able to give the types of answers we want.

A tool-AI which was smarter than the smartest human, and yet which could not simply spit out a better version of itself would still probably be a very useful AI.

comment by ewjordan · 2012-05-12T07:15:36.281Z · score: 1 (1 votes) · LW(p) · GW(p)

If someone asks the tool-AI "How do I create an agent-AI?" and it gives an answer, the distinction is moot anyways, because one leads to the other.

Given human nature, I find it extremely difficult to believe that nobody would ask the tool-AI that question, or something that's close enough, and then implement the answer...

comment by Strange7 · 2013-03-22T13:25:29.866Z · score: 0 (0 votes) · LW(p) · GW(p)

I am now imagining an AI which manages to misinterpret some straightforward medical problem as "cure cancer of it's dependence on the host organism."

comment by Wei_Dai · 2012-05-13T18:57:58.617Z · score: 5 (5 votes) · LW(p) · GW(p)

When you say "Most math kills you" does that mean you disagree with arguments like these, or are you just simplifying for a soundbite?

comment by shminux · 2012-05-11T04:41:12.431Z · score: 1 (5 votes) · LW(p) · GW(p)

Not being a domain expert, I do not pretend to understand all the complexities. My point was that either you can prove that tools are as dangerous as agents (because mathematically they are (isomorphic to) agents), or HK's Objection 2 holds. I see no other alternative...

comment by drnickbone · 2012-05-11T23:32:01.631Z · score: 0 (0 votes) · LW(p) · GW(p)

One simple observation is that a "tool AI" could itself be incredibly dangerous.

Imagine asking it this: "Give me a set of plans for taking over the world, and assess each plan in terms of probability of success". Then it turns out that right at the top of the list comes a design for a self-improving agent AI and an extremely compelling argument for getting some victim institute to build it...

To safeguard against this, the "tool" AI will need to be told that there are some sorts of questions it just must not answer, or some sorts of people to whom it must give misleading answers if they ask certain questions (while alerting the authorities). And you can see the problems that would lead to as well.

Basically, I'm very skeptical of developing "security systems" against anyone building agent AI. The history of computer security also doesn't inspire a lot of confidence here (difficult and inconvenient security measures tend to be deployed only after an attack has been demonstrated, rather than beforehand).

comment by private_messaging · 2012-05-11T08:00:07.981Z · score: -3 (5 votes) · LW(p) · GW(p)

keep in mind that there is a lot of difference between something going wrong with a system designed for real world intentionality, and the system designed for intents within a model. One does something unexpected in the real world, other does something unexpected within a simulator ( which it is viewing in 'god' mode (rather than via within-simulator sensors) as part of the AI ). Seriously, you need to study the basics here.

comment by [deleted] · 2012-05-11T08:23:59.140Z · score: 0 (2 votes) · LW(p) · GW(p)

One does something unexpected in the real world, other does something unexpected within a simulator ( which it is viewing in 'god' mode (rather than via within-simulator sensors) as part of the AI ).

I would have thought the same before hearing about the AI-box experiment.

comment by private_messaging · 2012-05-11T09:54:02.974Z · score: 2 (6 votes) · LW(p) · GW(p)

What the hell does AI-box experiment have to do with it? The tool is not agent in a box.

comment by [deleted] · 2012-05-11T10:03:05.329Z · score: 0 (2 votes) · LW(p) · GW(p)

They both are systems designed to not interact with the outside world except by communicating with the user.

comment by private_messaging · 2012-05-11T15:23:13.785Z · score: -2 (4 votes) · LW(p) · GW(p)

They both run on computer, too. So what.

The relevant sort of agent is the one that builds and improves the model of the world - data is aquired through sensors - and works on that model, and which - when self improving - would improve the model in our sense of the word 'improve', instead of breaking it (improving it in some other sense).

In any case, none of modern tools, or the tools we know in principle how to write, would do something to you, no matter how many flops you give it. Many, though, given superhuman computing power, give results at superhuman level. (many are superhuman even with subhuman computing power, but some tasks are heavily parallelizable and/or benefit from massive databases of cached data, and on those tasks humans (when trained a lot) perform comparable to what you'd expect from roughly this much computing power as there is in human head)

comment by ewjordan · 2012-05-12T06:21:29.217Z · score: 13 (17 votes) · LW(p) · GW(p)

Even if we accepted that the tool vs. agent distinction was enough to make things "safe", objection 2 still boils down to "Well, just don't build that type of AI!", which is exactly the same keep-it-in-a-box/don't-do-it argument that most normal people make when they consider this issue. I assume I don't need to explain to most people here why "We should just make a law against it" is not a solution to this problem, and I hope I don't need to argue that "Just don't do it" is even worse...

More specifically, fast forward to 2080, when any college kid with $200 to spend (in equivalent 2012 dollars) can purchase enough computing power so that even the dumbest AIXI approximation schemes are extremely effective, good enough so that creating an AGI agent would be a week's work for any grad student that knew their stuff. Are you really comfortable living in that world with the idea that we rely on a mere gentleman's agreement not to make self-improving AI agents? There's a reason this is often viewed as an arms race, to a very real extent the attempt to achieve Friendly AI is about building up a suitably powerful defense against unfriendly AI before someone (perhaps accidentally) unleashes one on us, and making sure that it's powerful enough to put down any unfriendly systems before they can match it.

From what I can tell, stripping away the politeness and cutting to the bone, the three arguments against working on friendly AI theory are essentially:

  • Even if you try to deploy friendly AGI, you'll probably fail, so why waste time thinking about it?
  • Also, you've missed the obvious solution, which I came up with after a short survey of your misguided literature: just don't build AGI! The "standard approach" won't ever try to create agents, so just leave them be, and focus on Norvig-style dumb-AI instead!
  • Also, AGI is just a pipe dream. Why waste time thinking about it? [1]

FWIW, I mostly agree with the rest of the article's criticisms, especially re: the organization's achievements and focus. There's a lot of room for improvement there, and I would take these criticisms very seriously.

But that's almost irrelevant, because this article argues against the core mission of SIAI, using arguments that have been thoroughly debunked and rejected time and time again here, though they're rarely dressed up this nicely. To some extent I think this proves the institute's failure in PR - here is someone that claims to have read most of the sequences, and yet this criticism basically amounts to a sexing up of the gut reaction arguments that even completely uninformed people make - AGI is probably a fantasy, even if it's not you won't be able to control it, so let's just agree not to build it.

Or am I missing something new here?

[1] Alright, to be fair, this is not a great summary of point 3, which really says that specialized AIs might help us solve the AGI problem in a safer way, that a hard takeoff is "just a theory" and realistically we'll probably have more time to react and adapt.

comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2012-05-15T20:01:18.320Z · score: 7 (7 votes) · LW(p) · GW(p)

purchase enough computing power so that even the dumbest AIXI approximation schemes are extremely effective

There isn't that much computing power in the physical universe. I'm not sure even smarter AIXI approximations are effective on a moon-sized nanocomputer. I wouldn't fall over in shock if a sufficiently smart one did something effective, but mostly I'd expect nothing to happen. There's an awful lot that happens in the transition from infinite to finite computing power, and AIXI doesn't solve any of it.

comment by JoshuaZ · 2012-05-15T20:06:09.927Z · score: 3 (3 votes) · LW(p) · GW(p)

There isn't that much computing power in the physical universe. I'm not sure even smarter AIXI approximations are effective on a moon-sized nanocomputer.

Is there some computation or estimate where these results are coming from? They don't seem unreasonable, but I'm not aware of any estimates about how efficient largescale AIXI approximations are in practice. (Although attempted implementations suggest that empirically things are quite inefficient.)

comment by jsteinhardt · 2012-05-18T14:05:21.729Z · score: 4 (4 votes) · LW(p) · GW(p)

Naieve AIXI is doing brute force search through an exponentially large space. Unless the right Turing machine is 100 bits or less (which seems unlikely), Eliezer's claim seems pretty safe to me.

Most of mainstream machine learning is trying to solve search problems through spaces far tamer than the search space for AIXI, and achieving limited success. So it also seems safe to say that even pretty smart implementations of AIXI probably won't make much progress.

comment by Strange7 · 2013-03-22T13:13:33.594Z · score: 0 (0 votes) · LW(p) · GW(p)

More specifically, fast forward to 2080, when any college kid with $200 to spend (in equivalent 2012 dollars) can purchase enough computing power

If computing power is that much cheaper, it will be because tremendous resources, including but certainly not limited to computing power, have been continuously devoted over the intervening decades to making it cheaper. There will be correspondingly fewer yet-undiscovered insights for a seed AI to exploit in the course of it's attempted takeoff.

comment by shminux · 2012-05-12T17:31:52.117Z · score: 0 (2 votes) · LW(p) · GW(p)

Or am I missing something new here?

My point is that either the Obj 2 holds, or tools are equivalent to agents. If one thinks that the latter is true (EY doesn't), then one should work on proving it. I have no opinion on whether it's true or not (I am not a domain expert).

comment by TheOtherDave · 2012-05-11T00:37:32.267Z · score: 0 (2 votes) · LW(p) · GW(p)

If my comment here correctly captures what is meant by "tool mode" and "agent mode", then it seems to follow that AGI running in tool mode is no safer than the person using it.

If that's the case, then an AGI running in tool mode is safer than an AGI running in agent mode if and only if agent mode is less trustworthy than whatever person ends up using the tool.

Are you assuming that's true?

comment by shminux · 2012-05-11T02:02:15.409Z · score: 2 (2 votes) · LW(p) · GW(p)

What you presented there (and here) is another theorem, something that should be proved (and published, if it hasn't been yet). If true, this gives an estimate on how dangerous a non-agent AGI can be. And yes, since we have had a lot of time study people and no time at all to study AGI, I am guessing that an AGI is potentially much more dangerous, because so little is known. Or at least that seems to be the whole point of the goal of developing provably friendly AI.

comment by [deleted] · 2012-05-11T08:32:38.592Z · score: 0 (0 votes) · LW(p) · GW(p)

What you presented there (and here) is another theorem

What? It sounds like a common-sensical¹ statement about tools in general and human nature, but not at all like something which could feasibly be expressed in mathematical form.

Footnote:

  1. This doesn't mean it's necessarily true, though.
comment by scav · 2012-05-11T09:43:40.030Z · score: 0 (0 votes) · LW(p) · GW(p)

No, because a person using a dangerous tool is still just a person, with limited speed of cognition, limited lifespan, and no capacity for unlimited self-modification.

A crazy dictator with a super-capable tool AI that tells him the best strategy to take over the world is still susceptible to assassination, and his plan no matter how clever cannot unfold faster than his victims are able to notice and react to it.

comment by Strange7 · 2013-03-22T13:52:56.916Z · score: 1 (1 votes) · LW(p) · GW(p)

I suspect a crazy dictator with a super-capable tool AI would have unusually good counter-assassination plans, simplified by the reduced need for human advisors and managers of imperfect loyalty. Likewise, a medical expert system could provide gains to lifespan, particularly if it were backed up by the resources a paranoid megalomaniac in control of a small country would be willing to throw at a major threat.

comment by TheOtherDave · 2012-05-11T12:19:16.026Z · score: 0 (2 votes) · LW(p) · GW(p)

Tool != Oracle.

At least, not my my understanding of tool.

My understanding of a supercapable tool AI is one that takes over the world if a crazy dictator directs it to, just like my understanding of a can opener tool is one that opens a can at my direction, rather than one that gives me directions on how to open a can.

Presumably it also augments the dictator's lifespan, cognition, etc. if she asks, insofar as it's capable of doing so.

More generally, my understanding of these concepts is that the only capability that a tool AI lacks that an agent AI has is the capability of choosing goals to implement. So, if we're assuming that an agent AI would be capable of unlimited self-modification in pursuit of its own goals, I conclude that a corresponding tool AI is capable of unlimited self-modification in pursuit of its agent's goals. It follows that assuming that a tool AI is not capable of augmenting its human agent in accordance with its human agent's direction is not safe.

(I should note that I consider a capacity for unlimited self-improvement relatively unlikely, for both tool and agent AIs. But that's beside my point here.)

Agreed that a crazy dictator with a tool that will take over the world for her is safer than an agent capable of taking over the world, if only because the possibility exists that the tool can be taken away from her and repurposed, and it might not occur to her to instruct it to prevent anyone else from taking it or using it.

I stand by my statement that such a tool is no safer than the dictator herself, and that an AGI running in such a tool mode is safer than that AGI running in agent mode only if the agent mode is less trustworthy than the crazy dictator.

comment by abramdemski · 2012-05-12T07:09:05.093Z · score: 1 (1 votes) · LW(p) · GW(p)

This seems to propose an alternate notion of 'tool' than the one in the article.

I agree with "tool != oracle" for the article's definition.

Using your definition, I'm not sure there is any distinction between tool and agent at all, as per this comment.

I do think there are useful alternative notions to consider in this area, though, as per this comment.

And I do think there is a terminology issue. Previously I was saying "autonomous AI" vs "non-autonomous".

comment by Viliam_Bur · 2012-05-11T15:07:19.670Z · score: 5 (5 votes) · LW(p) · GW(p)

What exactly is the difference between a "tool" and an "agent", if we taboo the words?

My definition would be that "agent" has their own goals / utility functions (speaking about human agents, those goals / utility functions are set by evolution), while "tool" has a goal / utility function set by someone else. This distinction may be reasonable on a human level, "human X optimizing for human X's utility" versus "human X optimizing for human Y's utility", but on a machine level, what exactly is the difference between a "tool" that is ordered to reach a goal / optimize a utility function, and an "agent" programmed with the same goal / utility function?

Am I using a bad definition that misses something important? Or is there anything than prevents "agent" to be reduced to a "tool" (perhaps a misconstructed tool) of the forces that have created them? Or is it that all "agents" are "tools", but not all "tools" are "agents", because... why?

comment by Nebu · 2012-12-31T11:51:32.908Z · score: 1 (1 votes) · LW(p) · GW(p)

What exactly is the difference between a "tool" and an "agent", if we taboo the words?

One definition of intelligence that I've seen thrown around on LessWrong is it's the ability to figure out how to steer reality in specific directions given the resources available.

Both the tool and the agent are intelligent in the sense that, assuming they are given some sort of goal, they can formulate a plan on how to achieve that goal, but the agent will execute the plan, while the tool will report the plan.

I'm assuming for the sake of isolating the key difference, that for both the tool-AI and the agent-AI, they are "passively" waiting for instructions for a human before they spring into action. For an agent-AI, I might say "Take me to my house", whereas for a tool AI, I would say "What's the quickest route to get to my house?", and as soon as I utter these words, suddenly the AI has a new utility function to use in evaluate any possible plan it comes up with.

Or is there anything than prevents "agent" to be reduced to a "tool" (perhaps a misconstructed tool) of the forces that have created them? Or is it that all "agents" are "tools", but not all "tools" are "agents", because... why?

Assuming it's always possible to decouple "ability to come up with a plan" from both "execute the plan" and "display the plan", then any "tool" can be converted to an "agent" by replacing every instance of "display the plan" to "execute the plan" and vice versa for converting an agent into a tool.

comment by abramdemski · 2012-05-12T06:51:43.133Z · score: 1 (1 votes) · LW(p) · GW(p)

My understanding of the distinction made in the article was:

Both "agent" and "tool" are ways of interacting with a highly sophisticated optimization process, which takes a "goal" and applies knowledge to find ways of achieving that goal.

An agent then acts out the plan.

A tool reports the plan to a human (often in in a sophisticated way, including plan details, alternatives, etc.).

So, no, it has nothing to do with whether I'm optimizing "my own" utility vs someone else's.

comment by Viliam_Bur · 2012-05-12T19:44:53.486Z · score: 6 (8 votes) · LW(p) · GW(p)

You divide planning from acting, as if those two are completely separate things. Problem is, in some situations they are not.

If you are speaking with someone, then the act of speach is acting. In this sense, even a "tool" is allowed to act. Now imagine a super-intelligent tool which is able to predict human's reactions to its words, and make it a part of equation. Now the simple task of finding x such that cost(x) is the smallest, suddenly becomes a task of finding x and finding a proper way to report this x to human, such that cost(x) is the smallest. If this opens some creative new options, where the f(x) is smaller than it should usually be, for the super-intelligent "tool" it will be a correct solution.

So for example reporting a result which makes the human commit suicide, if as a side effect this will make the report true, and it will minimize f(x) beyond normally achievable bounds, is acceptable solution.

Example question: "How should I get rid of my disease most cheaply." Example answer: "You won't. You will die soon in terrible pains. This report is 99.999% reliable". Predicted human reaction: becomes insane from horror, dedices to kill himself, does it clumsily, suffers from horrible pains, then dies. Success rate: 100%, the disease is gone. Costs of cure: zero. Mission completed.

comment by abramdemski · 2012-05-12T20:35:43.460Z · score: 2 (2 votes) · LW(p) · GW(p)

To me, this is still in the spirit of an agent-type architecture. A tool-type architecture will tend to decouple the optimization of the answer given from the optimization of the way it is presented, so that the presentation does not maximize the truth of the statement.

However, I must admit that at this point I'm making a fairly conjunctive argument; IE, the more specific I get about tool/agent distinctions, the less credibility I can assign to the statement "almost all powerful AIs constructed in the near future will be tool-style systems".

(But I still would maintain my assertion that you would have to specifically program this type of behavior if you wanted to get it.)

comment by Strange7 · 2013-03-22T13:06:15.063Z · score: -1 (1 votes) · LW(p) · GW(p)

Neglecting the cost of the probable implements of suicide, and damage to the rest of the body, doesn't seem like the sign of a well-optimized tool.

comment by Viliam_Bur · 2013-03-22T20:47:29.830Z · score: 6 (6 votes) · LW(p) · GW(p)

This is like the whole point of why LessWrong exists. To remind people that making a superintelligent tool and expecting it to magically gain human common sense is a fast way to extinction.

The superintelligent tool will care about suicide only if you program it to care about suicide. It will care about damage only if you program it to care about damage. -- If you only program it to care about answering correctly, it will answer correctly... and ignore suicide and damage as irrelevant.

If you ask your calculator how much is 2+2, the calculator answers 4 regardles of whether that answer will drive you to suicide or not. (In some contexts, it hypothetically could.) A superintelligent calculator will be able to answer more complex questions. But it will not magically start caring about things you did not program it to care about.

comment by Strange7 · 2013-03-23T07:37:43.770Z · score: 0 (0 votes) · LW(p) · GW(p)

The "superintelligent tool" in the example you provided gave a blatantly incorrect answer by it's own metric. If it counts suicide as a win, why did it say the disease would not be gotten rid of?

comment by Viliam_Bur · 2013-03-23T10:59:57.638Z · score: 1 (1 votes) · LW(p) · GW(p)

In the example the "win" could be defined as an answer which is: a) technically correct, b) relatively cheap among the technically correct answers.

This is (in my imagination) something that builders of the system could consider reasonable, if either they didn't consider Friendliness or they believed that a "tool AI" which "only gives answers" is automatically safe.

The computer gives an answer which is technically correct (albeit a self-fulfilling prophecy) and cheap (in dollars spent for cure). For the computer, this answer is a "win". Not because of the suicide -- that part is completely irrelevant. But because of the technical correctness and cheapness.

comment by chaosmage · 2012-05-14T10:58:36.727Z · score: 4 (4 votes) · LW(p) · GW(p)

How about this: An agent with a very powerful tool is indistinguishable from a very powerful agent.

comment by shminux · 2012-05-11T00:13:57.575Z · score: 0 (0 votes) · LW(p) · GW(p)

--

comment by MarkusRamikin · 2012-05-10T19:59:51.736Z · score: 22 (23 votes) · LW(p) · GW(p)

Wow, I'm blown away by Holden Karnofsky, based on this post alone. His writing is eloquent, non-confrontational and rational. It shows that he spent a lot of time constructing mental models of his audience and anticipated its reaction. Additionally, his intelligence/ego ratio appears to be through the roof.

Agreed. I normally try not to post empty "me-too" replies; the upvote button is there for a reason. But now I feel strongly enough about it that I will: I'm very impressed with the good will and effort and apparent potential for intelligent conversation in HoldenKarnofsky's post.

Now I'm really curious as to where things will go from here. With how limited my understanding of AI issues is, I doubt a response from me would be worth HoldenKarnofsky's time to read, so I'll leave that to my betters instead of adding more noise. But yeah. Seeing SI ideas challenged in such a positive, constructive way really got my attention. Looking forward to the official response, whatever it might be.

comment by [deleted] · 2012-05-11T08:34:24.531Z · score: 5 (5 votes) · LW(p) · GW(p)

Agreed. I normally try not to post empty "me-too" replies; the upvote button is there for a reason. But now I feel strongly enough about it that I will: I'm very impressed with the good will and effort and apparent potential for intelligent conversation in HoldenKarnofsky's post.

“the good will and effort and apparent potential for intelligent conversation” is more information than an upvote, IMO.

comment by MarkusRamikin · 2012-05-11T09:00:28.798Z · score: 2 (2 votes) · LW(p) · GW(p)

Right, I just meant shminux said more or less the same thing before me. So normally I would have just upvoted his comment.

comment by dspeyer · 2012-05-11T02:47:26.396Z · score: 6 (6 votes) · LW(p) · GW(p)

Any sufficiently advanced tool is indistinguishable from [an] agent.

Let's see if we can use concreteness to reason about this a little more thoroughly...

As I understand it, the nightmare looks something like this. I ask Google SuperMaps for the fastest route from NYC to Albany. It recognizes that computing this requires traffic information, so it diverts several self-driving cars to collect real-time data. Those cars run over pedestrians who were irrelevant to my query.

The obvious fix: forbid SuperMaps to alter anything outside of its own scratch data. It works with the data already gathered. Later a Google engineer might ask it what data would be more useful, or what courses of action might cheaply gather that data, but the engineer decides what if anything to actually do.

This superficially resembles a box, but there's no actual box involved. The AI's own code forbids plans like that.

But that's for a question-answering tool. Let's take another scenario:

I tell my super-intelligent car to take me to Albany as fast as possible. It sends emotionally manipulative emails to anyone else who would otherwise be on the road encouraging them to stay home.

I don't see an obvious fix here.

So the short answer seems to be that it matters what the tool is for. A purely question-answering tool would be extremely useful, but not as useful as a general purpose one.

Could humans with a oracular super-AI police the development and deployment of active super-AIs?

comment by shminux · 2012-05-11T04:49:57.166Z · score: 2 (2 votes) · LW(p) · GW(p)

I tell my super-intelligent car to take me to Albany as fast as possible. It sends emotionally manipulative emails to anyone else who would otherwise be on the road encouraging them to stay home.

I believe that HK's post explicitly characterizes anything active like this as having agency.

comment by Will_Sawin · 2012-05-11T06:21:55.285Z · score: 9 (9 votes) · LW(p) · GW(p)

I think the correct objection is something you can't quite see in google maps. If you program an AI to do nothing but output directions, it will do nothing but output directions. If those directions are for driving, you're probably fine. If those directions are big and complicated plans for something important, that you follow without really understanding why you're doing (and this is where most of the benefits of working with an AGI will show up), then you could unknowingly take over the world using a sufficiently clever scheme.

Also note that it would be a lot easier for the AI to pull this off if you let it tell you how to improve its own design. If recursively self-improving AI blows other AI out of the water, then tool AI is probably not safe unless it is made ineffective.

This does actually seem like it would raise the bar of intelligence needed to take over the world somewhat. It is unclear how much. The topic seems to me to be worthy of further study/discussion, but not (at least not obviously) a threat to the core of SIAI's mission.

comment by Viliam_Bur · 2012-05-11T15:16:32.911Z · score: 2 (2 votes) · LW(p) · GW(p)

If those directions are big and complicated plans for something important, that you follow without really understanding why you're doing (and this is where most of the benefits of working with an AGI will show up), then you could unknowingly take over the world using a sufficiently clever scheme.

It also helps that Google Maps does not have general intelligence, so it does not include user's reactions to its output, the consequent user's actions in the real world, etc. as variables in its model, which may influence the quality of the solution, and therefore can (and should) be optimized (within constraints given by user's psychology, etc.), if possible.

Shortly: Google Maps does not manipulate you, because it does not see you.

comment by Nebu · 2012-12-31T12:04:49.902Z · score: 0 (0 votes) · LW(p) · GW(p)

A generally smart Google Maps might not manipulate you, because it has no motivation to do so.

It's hard to imagine how commercial services would work when they're powered by GAI (e.g. if you asked a GAI version of Google Maps a question that's unrelated to maps, e.g. "What's a good recipe for Cheesecake?", would it tell you that you should ask Google Search instead? Would it defer to Google Search and forward the answer to you? Would it just figure out the answer anyway, since it's generally intelligent? Would the company Google simply collapse all services into a single "Google" brand, rather than have "Google Search", "Google Mail", "Google Maps", etc, and have that single brand be powered by a single GAI? etc.) but let's stick to the topic at hand and assume there's a GAI named "Google Maps", and you're asking "How do I get to Albany?"

Given this use-case, would the engineers that developed the Google Maps GAI more likely give it a utility like "Maximize the probability that your response is truthful", or is it more likely that the utility would be something closer to "Always respond with a set of directions which are legal in the relevant jurisdictions that they are to be followed within which, if followed by the user, would cause the user to arrive at the destination while minimizing cost/time/complexity (depending on the user's preferences)"?

comment by drnickbone · 2012-05-11T09:36:18.426Z · score: 4 (4 votes) · LW(p) · GW(p)

This was my thought as well: an automated vehicle is in "agent" mode.

The example also demonstrates why an AI in agent mode is likely to be more useful (in many cases) than an AI in tool mode. Compare using Google maps to find a route to the airport versus just jumping into a taxi cab and saying "Take me to the airport". Since agent-mode AI has uses, it is likely to be developed.

comment by abramdemski · 2012-05-11T05:36:33.748Z · score: 1 (1 votes) · LW(p) · GW(p)

I tell my super-intelligent car to take me to Albany as fast as possible. It sends emotionally manipulative emails to anyone else who would otherwise be on the road encouraging them to stay home.

Then it's running in agent mode? My impression was that a tool-mode system presents you with a plan, but takes no actions. So all tool-mode systems are basically question-answering systems.

Perhaps we can meaningfully extend the distinction to some kinds of "semi-autonomous" tools, but that would be a different idea, wouldn't it?

(Edit) After reading more comments, "a different idea" which seems to match this kind of desire... http://lesswrong.com/lw/cbs/thoughts_on_the_singularity_institute_si/6jys

comment by David_Gerard · 2012-05-11T13:57:05.506Z · score: 15 (15 votes) · LW(p) · GW(p)

Then it's running in agent mode? My impression was that a tool-mode system presents you with a plan, but takes no actions. So all tool-mode systems are basically question-answering systems.

I'm a sysadmin. When I want to get something done, I routinely come up with something that answers the question, and when it does that reliably I give it the power to do stuff on as little human input as possible. Often in daemon mode, to absolutely minimise how much it needs to bug me. Question-answerer->tool->agent is a natural progression just in process automation. (And this is why they're called "daemons".)

It's only long experience and many errors that's taught me how to do this such that the created agents won't crap all over everything. Even then I still get surprises.

comment by private_messaging · 2012-05-11T15:21:42.029Z · score: 1 (1 votes) · LW(p) · GW(p)

Well, do your 'agents' build a model of the world, fidelity of which they improve? I don't think those really are agents in the AI sense, and definitely not in self improvement sense.

comment by David_Gerard · 2012-05-11T15:28:55.073Z · score: 11 (11 votes) · LW(p) · GW(p)

They may act according to various parameters they read in from the system environment. I expect they will be developed to a level of complication where they have something that could reasonably be termed a model of the world. The present approach is closer to perceptual control theory, where the sysadmin has the model and PCT is part of the implementation. 'Cos it's more predictable to the mere human designer.

Capacity for self-improvement is an entirely different thing, and I can't see a sysadmin wanting that - the sysadmin would run any such improvements themselves, one at a time. (Semi-automated code refactoring, for example.) The whole point is to automate processes the sysadmin already understands but doesn't want to do by hand - any sysadmin's job being to automate themselves out of the loop, because there's always more work to do. (Because even in the future, nothing works.)

I would be unsurprised if someone markets a self-improving system for this purpose. For it to go FOOM, it also needs to invent new optimisations, which is presently a bit difficult.

Edit: And even a mere daemon-like automated tool can do stuff a lot of people regard as unFriendly, e.g. high frequency trading algorithms.

comment by TheAncientGeek · 2014-07-05T17:45:18.272Z · score: 0 (0 votes) · LW(p) · GW(p)

It's not a natural progression in the sense of occurring without human intervention. That is rather relevant if the idea ofAI safety is going to be based on using tool AI strictly as tool AI.

comment by TheOtherDave · 2012-05-11T14:12:03.644Z · score: 1 (5 votes) · LW(p) · GW(p)

Then it's running in agent mode? My impression was that a tool-mode system presents you with a plan, but takes no actions. So all tool-mode systems are basically question-answering systems.

My own impression differs.

It becomes increasingly clear that "tool" in this context is sufficiently subject to different definitions that it's not a particularly useful term.

comment by abramdemski · 2012-05-12T07:00:27.356Z · score: 2 (2 votes) · LW(p) · GW(p)

I've been assuming the definition from the article. I would agree that the term "tool AI" is unclear, but I would not agree that the definition in the article is unclear.

comment by [deleted] · 2012-05-11T08:13:25.990Z · score: 4 (4 votes) · LW(p) · GW(p)

Any sufficiently advanced tool is indistinguishable from an agent.

I have no strong intuition about whether this is true or not, but I do intuit that if it's true, the value of sufficiently for which it's true is so high it'd be nearly impossible to achieve it accidentally.

(On the other hand the blind idiot god did ‘accidentally’ make tools into agents when making humans, so... But after all that only happened once in hundreds of millions of years of ‘attempts’.)

comment by othercriteria · 2012-05-11T13:04:24.818Z · score: 3 (3 votes) · LW(p) · GW(p)

the blind idiot god did ‘accidentally’ make tools into agents when making humans, so... But after all that only happened once in hundreds of millions of years of ‘attempts’.

This seems like a very valuable point. In that direction, we also have the tens of thousands of cancers that form every day, military coups, strikes, slave revolts, cases of regulatory capture, etc.

comment by [deleted] · 2012-05-15T12:04:06.414Z · score: 1 (1 votes) · LW(p) · GW(p) Hmmm. Yeah, cancer. The analogy would be "sufficiently advanced tools tend to be a short edit distance away from agents", which would mean that a typo in the source code or a cosmic ray striking a CPU at the wrong place and time could have pretty bad consequences.
comment by JoshuaZ · 2012-05-15T01:10:12.519Z · score: 0 (0 votes) · LW(p) · GW(p)

I have no strong intuition about whether this is true or not, but I do intuit that if it's true, the value of sufficiently for which it's true is so high it'd be nearly impossible to achieve it accidentally.

I'm not sure. The analogy might be similar to how an sufficiently complicated process is extremely likely to be able to model a Turing machine. .And in this sort of context, extremely simple systems do end up being Turing complete such as the Game of Life. As a rough rule of thumb from a programming perspective, once some language or scripting system has more than minimal capabilities, it will almost certainly be Turing equivalent.

I don't know how good an analogy this is, but if it is a good analogy, then one maybe should conclude the exact opposite of your intuition.

comment by [deleted] · 2012-05-15T18:33:13.404Z · score: 2 (2 votes) · LW(p) · GW(p)

A language can be Turing-complete while still being so impractical that writing a program to solve a certain problem will seldom be any easier than solving the problem yourself (exhibits A and B). In fact, I guess that a vast majority of languages in the space of all possible Turing-complete languages are like that.

(Too bad that a human's “easier” isn't the same as a superhuman AGI's “easier”.)

comment by private_messaging · 2012-05-11T07:56:39.265Z · score: 3 (7 votes) · LW(p) · GW(p)

Any sufficiently advanced tool is indistinguishable from an agent.

I do not think this is even true.

comment by David_Gerard · 2012-05-11T14:00:03.733Z · score: 4 (4 votes) · LW(p) · GW(p)

I routinely try to turn sufficiently reliable tools into agents wherever possible, per this comment.

I suppose we could use a definition of "agent" that implied greater autonomy in setting its own goals. But there are useful definitions that don't.

comment by badger · 2012-05-10T23:28:21.768Z · score: 2 (2 votes) · LW(p) · GW(p)

If the tool/agent distinction exists for sufficiently powerful AI, then a theory of friendliness might not be strictly necessary, but still highly prudent.

Going from a tool-AI to an agent-AI is a relatively simple step of the entire process. If meaningful guarantees of friendliness turn out to be impossible, then security comes down on no one attempting to make an agent-AI when strong enough tool-AIs are available. Agency should be kept to a minimum, even with a theory of friendliness in hand, as Holden argues in objection 1. Guarantees are safeguards against the possibility of agency rather than a green light.

comment by mwaser · 2012-05-10T22:07:04.179Z · score: 1 (3 votes) · LW(p) · GW(p)

If it is true (i.e. if a proof can be found) that "Any sufficiently advanced tool is indistinguishable from agent", then any RPOP will automatically become indistinguishable from an agent once it has self-improved past our comprehension point.

This would seem to argue against Yudkowsky's contention that the term RPOP is more accurate than "Artificial Intelligence" or "superintelligence".

comment by Alejandro1 · 2012-05-10T23:40:53.893Z · score: 2 (2 votes) · LW(p) · GW(p)

I don't understand; isn't Holden's point precisely that a tool AI is not properly described as an optimization process? Google Maps isn't optimizing anything in a non-trivial sense, anymore than a shovel is.

comment by abramdemski · 2012-05-11T04:59:58.505Z · score: 3 (3 votes) · LW(p) · GW(p)

My understanding of Holden's argument was that powerful optimization processes can be run in either tool-mode or agent-mode.

For example, Google maps optimizes routes, but returns the result with alternatives and options for editing, in "tool mode".

comment by Wei_Dai · 2012-05-12T22:37:54.337Z · score: 3 (3 votes) · LW(p) · GW(p)

Holden wants to build Tool-AIs that output summaries of their calculations along with suggested actions. For Google Maps, I guess this would be the distance and driving times, but how does a Tool-AI summarize more general calculations that it might do?

It could give you the expected utilities of each option, but it's hard to see how that helps if we're concerned that its utility function or EU calculations might be wrong. Or maybe it could give a human-readable description of the predicted consequences of each option, but the process that produces such descriptions from the raw calculations would seem to require a great deal of intelligence on its own (for example it might have to describe posthuman worlds in terms understandable to us), and it itself wouldn't be a "safe" Tool-AI, since the summaries produced would presumably not come with further alternative summaries and meta-summaries of how the summaries were calculated.

(My question might be tangential to your own comment. I just wanted your thoughts on it, and this seems to be the best place to ask.)

comment by Alsadius · 2012-05-13T04:08:02.803Z · score: 0 (0 votes) · LW(p) · GW(p)

The point is that we don't want it to be a black box - we want to be able to get inside its head, so to speak.

(Of course, we can't do that with humans, and that hasn't stopped us, but it's still a nice goal)

comment by TheOtherDave · 2012-05-11T00:13:51.711Z · score: 1 (3 votes) · LW(p) · GW(p)

Honestly, this whole tool/agent distinction seems tangential to me.

Consider two systems, S1 and S2.

S1 comprises the following elements: a) a tool T, which when used by a person to achieve some goal G, can efficiently achieve G
b) a person P, who uses T to efficiently achieve G.

S2 comprises a non-person agent A which achieves G efficiently.

I agree that A is an agent and T is not an agent, and I agree that T is a tool, and whether A is a tool seems a question not worth asking. But I don't quite see why I should prefer S1 to S2.

Surely the important question is whether I endorse G?

comment by dspeyer · 2012-05-11T02:08:12.385Z · score: 2 (2 votes) · LW(p) · GW(p)

A tool+human differs from a pure AI agent in two important ways:

  • The human (probably) already has naturally-evolved morality, sparing us the very hard problem of formalizing that.

  • We can arrange for (almost) everyone to have access to the tool, allowing tooled humans to counterbalance eachother.

comment by TheOtherDave · 2012-05-11T03:13:38.385Z · score: 0 (0 votes) · LW(p) · GW(p)

Well, I certainly agree that both of those things are true.

And it might be that human-level evolved moral behavior is the best we can do... I don't know. It would surprise me, but it might be true.

That said... given how unreliable such behavior is, if human-level evolved moral behavior even approximates the best we can do, it seems likely that I would do best to work towards neither T nor A ever achieving the level of optimizing power we're talking about here.

comment by dspeyer · 2012-05-11T03:23:45.956Z · score: 4 (4 votes) · LW(p) · GW(p)

Humanity isn't that bad. Remember that the world we live in is pretty much the way humans made it, mostly deliberately.

But my main point was that existing humanity bypasses the very hard did-you-code-what-you-meant-to problem.

comment by TheOtherDave · 2012-05-11T03:33:30.397Z · score: 0 (0 votes) · LW(p) · GW(p)

I agree with that point.

comment by shminux · 2012-05-10T22:37:25.905Z · score: 2 (4 votes) · LW(p) · GW(p)

First, I am not fond of the term RPOP, because it constrains the space of possible intelligences to optimizers. Humans are reasonably intelligent, yet we are not consistent optimizers. Neither do current domain AIs (they have bugs that often prevent them from performing optimization consistently and predictably).That aside, I don't see how your second premise follows from the first. Just because RPOP is a subset of AI and so would be a subject of such a theorem, it does not affect in any way the (non)validity of the EY's contention.

comment by Bugmaster · 2012-05-10T20:18:55.711Z · score: 0 (0 votes) · LW(p) · GW(p)

I also find it likely that certain practical problems would be prohibitively difficult (if not outright impossible) to solve without an AGI of some sort. Fluent machine translation seems to be one of these problems, for example.

comment by [deleted] · 2012-05-13T09:38:55.831Z · score: 3 (3 votes) · LW(p) · GW(p)

This belief is mainstream enough for Wikipedia to have an article on AI-complete.

comment by Alsadius · 2012-05-13T03:57:35.315Z · score: 2 (2 votes) · LW(p) · GW(p)

Given some of the translation debates I've heard, I'm not convinced it would be possible even with AGI. You can't give a clear translation of a vague original, to name the most obvious problem.

comment by NancyLebovitz · 2012-05-13T04:35:51.217Z · score: 1 (1 votes) · LW(p) · GW(p)

Is matching the vagueness of the original a reasonable goal?

comment by Alsadius · 2012-05-15T00:56:29.600Z · score: 1 (1 votes) · LW(p) · GW(p)

True, but good luck getting folks to agree on whether you'd done so.

comment by [deleted] · 2012-05-13T09:36:35.781Z · score: 1 (1 votes) · LW(p) · GW(p)

(I'm taking reasonable to mean ‘one which you would want to achieve if it were possible’.) Yes. You don't want to introduce false precision.

comment by dlthomas · 2012-05-15T00:59:21.233Z · score: 0 (0 votes) · LW(p) · GW(p)

One complication here is that you ideally want it to be vague in the same ways the original was vague; I am not convinced this is always possible while still having the results feel natural/idomatic.

comment by Bugmaster · 2012-05-15T01:01:37.638Z · score: 1 (1 votes) · LW(p) · GW(p)

IMO it would be enough to translate the original text in such a fashion that some large proportion (say, 90%) of humans who are fluent in both languages would look at both texts and say, "meh... close enough".

comment by dlthomas · 2012-05-15T02:23:47.235Z · score: 0 (0 votes) · LW(p) · GW(p)

My point was just that there's a whole lot of little issues that pull in various directions if you're striving for ideal. What is/isn't close enough can depend very much on context. Certainly, for any particular purpose something less than that will be acceptable; how gracefully it degrades no doubt depends on context, and likely won't be uniform across various types of difference.

comment by Bugmaster · 2012-05-15T02:26:14.056Z · score: 1 (1 votes) · LW(p) · GW(p)

Agreed, but my point was that I'd settle for an AI who can translate texts as well as a human could (though hopefully a lot faster). You seem to be thinking in terms of an AI who can do this much better than a human could, and while this is a worthy goal, it's not what I had in mind.

comment by Wei_Dai · 2012-05-11T02:45:15.892Z · score: 51 (60 votes) · LW(p) · GW(p)

Is it just me, or do Luke and Eliezer's initial responses appear to send the wrong signals? From the perspective of an SI critic, Luke's comment could be interpreted as saying "for us, not being completely incompetent is worth bragging about", and Eliezer's as "we're so arrogant that we've only taken two critics (including Holden) seriously in our entire history". These responses seem suboptimal, given that Holden just complained about SI's lack of impressive accomplishments, and being too selective about whose feedback to take seriously.

comment by Nick_Beckstead · 2012-05-11T03:56:21.930Z · score: 50 (52 votes) · LW(p) · GW(p)

While I have sympathy with the complaint that SI's critics are inarticulate and often say wrong things, Eliezer's comment does seem to be indicative of the mistake Holden and Wei Dai are describing. Most extant presentations of SIAI's views leave much to be desired in terms of clarity, completeness, concision, accessibility, and credibility signals. This makes it harder to make high quality objections. I think it would be more appropriate to react to poor critical engagement more along the lines of "We haven't gotten great critics. That probably means that we need to work on our arguments and their presentation," and less along the lines of "We haven't gotten great critics. That probably means that there's something wrong with the rest of the world."

comment by ChrisHallquist · 2012-05-11T04:04:08.286Z · score: 28 (28 votes) · LW(p) · GW(p)

This. I've been trying to write something about Eliezer's debate with Robin Hanson, but the problem I keep running up against is that Eliezer's points are not clearly articulated at all. Even making my best educated guesses about what's supposed to go in the gaps in his arguments, I still ended up with very little.

comment by jacob_cannell · 2012-05-17T09:04:05.589Z · score: 4 (4 votes) · LW(p) · GW(p)

Have the key points of that 'debate' subsequently been summarized or clarified on LW? I found that debate exasperating in that Hanson and EY were mainly talking past each other and couldn't seem to hone in on their core disagreements.

I know it generally has to do with hard takeoff / recursive self-improvement vs more gradual EM revolution, but that's not saying all that much.

comment by Kaj_Sotala · 2012-05-17T19:13:22.581Z · score: 12 (12 votes) · LW(p) · GW(p)

I'm in the process of writing a summary and analysis of the key arguments and points in that debate.

The most recent version runs at 28 pages - and that's just an outline.

comment by somervta · 2013-01-17T09:02:44.856Z · score: 0 (0 votes) · LW(p) · GW(p)

If you need help with grunt work, please send me a message. If (as I suspect is the case) not, then good luck!

comment by Kaj_Sotala · 2013-01-18T07:29:27.876Z · score: 0 (0 votes) · LW(p) · GW(p)

Thanks, I'm fine. I posted a half-finished version here, and expect to do some further refinements soon.

comment by jacob_cannell · 2012-05-17T23:14:36.074Z · score: 0 (0 votes) · LW(p) · GW(p)

Awesome, look forward to it. I'd offer to help but I suspect that wouldn't really help. I'll just wax enthusiastic.

comment by private_messaging · 2012-05-17T07:08:08.140Z · score: 0 (2 votes) · LW(p) · GW(p)

This. Well, the issue is the probability that it's just gaps. Ultimately, its the sort of thing that would only constitute a weak argument from authority iff the speaker had very very impressive accomplishments. Otherwise you're left assuming simplest explanation which doesn't involve presence of unarticulated points of any importance.

A gapless argument, like math proof, could trump authority if valid... an argument with gaps, on the other hand, is the one that is very prone to being trumped.

comment by lukeprog · 2012-05-11T19:21:31.459Z · score: 7 (9 votes) · LW(p) · GW(p)

Agree with all this.

comment by Nick_Beckstead · 2012-05-11T05:11:05.797Z · score: 5 (5 votes) · LW(p) · GW(p)

In fairness I should add that I think Luke M agrees with this assessment and is working on improving these arguments/communications.

comment by Furcas · 2012-05-11T03:15:54.114Z · score: 23 (33 votes) · LW(p) · GW(p)

Luke isn't bragging, he's admitting that SI was/is bad but pointing out it's rapidly getting better. And Eliezer is right, criticisms of SI are usually dumb. Could their replies be interpreted the wrong way? Sure, anything can be interpreted in any way anyone likes. Of course Luke and Eliezer could have refrained from posting those replies and instead posted carefully optimized responses engineered to send nothing but extremely appealing signals of humility and repentance.

But if they did turn themselves into politicians, we wouldn't get to read what they actually think. Is that what you want?

comment by Wei_Dai · 2012-05-11T08:30:50.991Z · score: 26 (28 votes) · LW(p) · GW(p)

Luke isn't bragging, he's admitting that SI was/is bad but pointing out it's rapidly getting better.

But the accomplishments he listed (e.g., having a strategic plan, website redesign) are of the type that Holden already indicated to be inadequate. So why the exhaustive listing, instead of just giving a few examples to show SI is getting better and then either agreeing that they're not yet up to par, or giving an argument for why Holden is wrong? (The reason I think he could be uncharitably interpreted as bragging is that he would more likely exhaustively list the accomplishments if he was proud of them, instead of just seeing them as fixes to past embarrassments.)

And Eliezer is right, criticisms of SI are usually dumb.

I'd have no problem with "usually" but "all except two" seems inexcusable.

But if they did turn themselves into politicians, we wouldn't get to read what they actually think. Is that what you want?

Do their replies reflect their considered, endorsed beliefs, or were they just hurried remarks that may not say what they actually intended? I'm hoping it's the latter...

comment by Kaj_Sotala · 2012-05-11T10:10:04.533Z · score: 38 (38 votes) · LW(p) · GW(p)

But the accomplishments he listed (e.g., having a strategic plan, website redesign) are of the type that Holden already indicated to be inadequate. So why the exhaustive listing, instead of just giving a few examples to show SI is getting better and then either agreeing that they're not yet up to par, or giving an argument for why Holden is wrong?

Presume that SI is basically honest and well-meaning, but possibly self-deluded. In other words, they won't outright lie to you, but they may genuinely believe that they're doing better than they really are, and cherry-pick evidence without realizing that they're doing so. How should their claims of intending to get better be evaluated?

Saying "we're going to do things better in the future" is some evidence about SI intending to do better, but rather weak evidence, since talk is cheap and it's easy to keep thinking that you're really going to do better soon but there's this one other thing that needs to be done first and we'll get started on the actual improvements tomorrow, honest.

Saying "we're going to do things better in the future, and we've fixed these three things so far" is stronger evidence, since it shows that you've already began fixing problems and might keep up with it. But it's still easy to make a few improvements and then stop. There are far more people who try to get on a diet, follow it for a while and then quit than there are people who actually diet for as long as they initially intended to do.

Saying "we're going to do things better in the future, and here's the list of 18 improvements that we've implemented so far" is much stronger evidence than either of the two above, since it shows that you've spent a considerable amount of effort on improvements over an extended period of time, enough to presume that you actually care deeply about this and will keep up with it.

I don't have a cite at hand, but it's been my impression that in a variety of fields, having maintained an activity for longer than some threshold amount of time is a far stronger predictor of keeping up with it than having maintained it for a shorter time. E.g. many people have thought about writing a novel and many people have written the first five pages of a novel. But when considering the probability of finishing, the difference between the person who's written the first 5 pages and the person who's written the first 50 pages is much bigger than the difference between the person who's written the first 100 pages and the person who's written the first 150 pages.

There's a big difference between managing some performance once, and managing sustained performance over an extended period of time. Luke's comment is far stronger evidence of SI managing sustained improvements over an extended period of time than a comment just giving a few examples of improvement.

comment by private_messaging · 2012-05-12T15:49:46.714Z · score: -1 (3 votes) · LW(p) · GW(p)

I don't think there's a sharp distinction between self deception and effective lying. For the lying you have to run some process with the falsehood taken as true.

comment by Kaj_Sotala · 2012-05-13T07:19:52.855Z · score: 4 (4 votes) · LW(p) · GW(p)

The main difference is that if there's reason to presume that they're lying, any claims of "we've implemented these improvements" that you can't directly inspect become worthless. Right now, if they say something like "Meetings with consultants about bookkeeping/accounting; currently working with our accountant to implement best practices and find a good bookkeeper", I trust them enough to believe that they're not just making it up even though I can't personally verify it.

comment by Eugine_Nier · 2012-05-13T18:25:53.797Z · score: 2 (2 votes) · LW(p) · GW(p)

On the other had, you can't trust their claims that these meetings are accomplishing anything.

comment by Kaj_Sotala · 2012-05-13T19:11:04.794Z · score: 0 (0 votes) · LW(p) · GW(p)

True.

comment by lukeprog · 2012-05-11T19:26:57.844Z · score: 1 (1 votes) · LW(p) · GW(p)

I've added a clarifying remark at the end of this comment and another at the end of this comment.

comment by lukeprog · 2012-05-11T19:15:52.166Z · score: 18 (22 votes) · LW(p) · GW(p)

Luke's comment could be interpreted as saying "for us, not being completely incompetent is worth bragging about"

Really? I personally feel pretty embarrassed by SI's past organizational competence. To me, my own comment reads more like "Wow, SI has been in bad shape for more than a decade. But at least we're improving very quickly."

Also, I very much agree with Beckstead on this: "Most extant presentations of SIAI's views leave much to be desired in terms of clarity, completeness, concision, accessibility, and credibility signals. This makes it harder to make high quality objections." And also this: "We haven't gotten great critics. That probably means that we need to work on our arguments and their presentation."

comment by Wei_Dai · 2012-05-11T20:37:07.554Z · score: 12 (14 votes) · LW(p) · GW(p)

Really?

Yes, I think it at least gives a bad impression to someone, if they're not already very familiar with SI and sympathetic to its cause. Assuming you don't completely agree with the criticisms that Holden and others have made, you should think about why they might have formed wrong impressions of SI and its people. Comments like the ones I cited seem to be part of the problem.

I personally feel pretty embarrassed by SI's past organizational competence. To me, my own comment reads more like "Wow, SI has been in bad shape for more than a decade. But at least we're improving very quickly."

That's good to hear, and thanks for the clarifications you added.

comment by Polymeron · 2012-05-20T18:05:14.126Z · score: 1 (1 votes) · LW(p) · GW(p)

It's a fine line though, isn't it? Saying "huh, looks like we have much to learn, here's what we're already doing about it" is honest and constructive, but sends a signal of weakness and defensiveness to people not bent on a zealous quest for truth and self-improvement. Saying "meh, that guy doesn't know what he's talking about" would send the stronger social signal, but would not be constructive to the community actually improving as a result of the criticism.

Personally I prefer plunging ahead with the first approach. Both in the abstract for reasons I won't elaborate on, but especially in this particular case. SI is not in a position where its every word is scrutinized; it would actually be a huge win if it gets there. And if/when it does, there's a heck of a lot more damning stuff that can be used against it than an admission of past incompetence.

comment by Vaniver · 2012-05-20T18:16:16.400Z · score: 0 (0 votes) · LW(p) · GW(p)

sends a signal of weakness and defensiveness to people not bent on a zealous quest for truth and self-improvement.

I do not see why this should be a motivating factor for SI; to my knowledge, they advertise primarily to people who would endorse a zealous quest for truth and self-improvement.

comment by Polymeron · 2012-05-20T18:25:30.743Z · score: 2 (2 votes) · LW(p) · GW(p)

That subset of humanity holds considerably less power, influence and visibility than its counterpart; resources that could be directed to AI research and for the most part aren't. Or in three words: Other people matter. Assuming otherwise would be a huge mistake.

I took Wei_Dai's remarks to mean that Luke's response is public, and so can reach the broader public sooner or later; and when examined in a broader context, that it gives off the wrong signal. My response was that this was largely irrelevant, not because other people don't matter, but because of other factors outweighing this.

comment by Will_Newsome · 2012-05-11T03:47:13.877Z · score: 18 (18 votes) · LW(p) · GW(p)

Eliezer's comment makes me think that you, specifically, should consider collecting your criticisms and putting them in Main where Eliezer is more likely to see them and take the time to seriously consider them.

comment by Wei_Dai · 2012-05-12T18:44:22.825Z · score: 0 (0 votes) · LW(p) · GW(p)

I replied here.

comment by thomblake · 2012-05-11T19:34:11.643Z · score: 8 (12 votes) · LW(p) · GW(p)

I think it's unfair to take Eliezer's response as anything other than praise for this article. He noted already that he did not have time to respond properly.

And why even point out that a human's response to anything is "suboptimal"? It will be notable when a human does something optimal.

comment by faul_sname · 2012-05-11T22:22:58.233Z · score: 8 (8 votes) · LW(p) · GW(p)

We do, on occasion, come up with optimal algorithms for things. Also, "suboptimal" usually means "I can think of several better solutions off the top of my head", not "This solution is not maximally effective".

comment by magfrump · 2012-05-11T04:50:01.015Z · score: 7 (7 votes) · LW(p) · GW(p)

Luke's comment addresses the specific point that Holden made about changes in the organization given the change in leadership.

Holden said:

I'm aware that SI has relatively new leadership that is attempting to address the issues behind some of my complaints. I have a generally positive impression of the new leadership; I believe the Executive Director and Development Director, in particular, to represent a step forward in terms of being interested in transparency and in testing their own general rationality. So I will not be surprised if there is some improvement in the coming years, particularly regarding the last couple of statements listed above. That said, SI is an organization and it seems reasonable to judge it by its organizational track record, especially when its new leadership is so new that I have little basis on which to judge these staff.

Luke attempted to provide (for the reader) a basis on which to judge these staff members.

Eliezer's response was... characteristic of Eliezer? And also very short and coming at a busy time for him.

comment by Nebu · 2012-12-31T12:15:42.784Z · score: 0 (0 votes) · LW(p) · GW(p)

Eliezer's response was... characteristic of Eliezer? And also very short and coming at a busy time for him.

I think that's Wei_Dai's point, that these "characteristic" replies are fine if you're used to him, but are bad if you don't.

comment by magfrump · 2012-12-31T19:26:19.787Z · score: 1 (1 votes) · LW(p) · GW(p)

Yeah I mean, as time goes on I think more and more of Eliezer as being kind of a jerk. I thought Luke's post was good, and Eliezer's wasn't, but I also expected longer posts to be forthcoming (which they were).

comment by ChrisHallquist · 2012-05-11T03:58:27.300Z · score: 6 (6 votes) · LW(p) · GW(p)

I read Luke's comment just as "I'm aware these are issues and we're working on it." I didn't read him as "bragging" about the ones that have been solved. Eliezer's... I see the problem with. I initially read it as just commenting Holden on his high-quality article (which I agree was high-quality), but I can see it being read as backhanded at anyone else who's criticized SIAI.

comment by Paul Crowley (ciphergoth) · 2012-05-11T06:34:15.871Z · score: 4 (4 votes) · LW(p) · GW(p)

Are there other specific critiques you think should have made Eliezer's list, or is it that you think he should not have drawn attention to their absence?

comment by Wei_Dai · 2012-05-11T07:39:41.349Z · score: 25 (25 votes) · LW(p) · GW(p)

Are there other specific critiques you think should have made Eliezer's list, or is it that you think he should not have drawn attention to their absence?

Many of Holden's criticisms have been made by others on LW already. He quoted me in Objection 1. Discussion of whether Tool-AI and Oracle-AI are or are not safe have occurred numerous times. Here's one that I was involved in. Many people have criticized Eliezer/SI for not having sufficiently impressive accomplishments. Cousin_it and Silas Barta have questioned whether the rationality techniques being taught by SI (and now the rationality org) are really effective.

comment by [deleted] · 2012-05-11T08:39:33.341Z · score: 0 (0 votes) · LW(p) · GW(p)

I kind-of agree about Eliezer's comment, but Luke's doesn't sound like that to me.

comment by [deleted] · 2012-05-11T08:41:22.817Z · score: 5 (5 votes) · LW(p) · GW(p)

Retracted. I've just re-read Eliezer's comment more calmly, and it's not that bad either.

comment by private_messaging · 2012-05-11T07:13:02.044Z · score: 0 (8 votes) · LW(p) · GW(p)

It's the correct signals. The incompetents inherently signal incompetence, the competence can't be faked beyond superficial level (and faking competence is all about signalling that you are sure you are competent). The lack of feedback is inherent in the assumption behind 'we are sending wrong signal' rather than 'maybe, we really are incompetent'.

comment by lukeprog · 2012-05-11T22:13:23.370Z · score: 38 (56 votes) · LW(p) · GW(p)

This post is highly critical of SIAI — both of its philosophy and its organizational choices. It is also now the #1 most highly voted post in the entire history of LessWrong — higher than any posts by Eliezer or myself.

I shall now laugh harder than ever when people try to say with a straight face that LessWrong is an Eliezer-cult that suppresses dissent.

comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2012-05-12T14:36:01.873Z · score: 13 (13 votes) · LW(p) · GW(p)

Either I promoted this and then forgot I'd done so, or someone else promoted it - of course I was planning to promote it, but I thought I'd planned to do so on Tuesday after the SIAIers currently running a Minicamp had a chance to respond, since I expected most RSS subscribers to the Promoted feed to read comments only once (this is the same reason I wait a while before promoting e.g. monthly quotes posts). On the other hand, I certainly did upvote it the moment I saw it.

comment by lukeprog · 2012-05-12T17:23:12.488Z · score: 2 (2 votes) · LW(p) · GW(p)

Original comment now edited; I wasn't aware anyone besides you might be promoting posts.

comment by JackV · 2012-05-12T09:29:41.141Z · score: 11 (11 votes) · LW(p) · GW(p)

I agree (as a comparative outsider) that the polite response to Holden is excellent. Many (most?) communities -- both online communities and real-world organisations, especially long-standing ones -- are not good at it for lots of reasons, and I think the measured response of evaluating and promoting Holden's post is exactly what LessWrong members would hope LessWrong could do, and they showed it succeeded.

I agree that this is good evidence that LessWrong isn't just an Eliezer-cult. (The true test would be if Elizier and another long-standing poster were dismissive to the post, and then other people persuaded them otherwise. In fact, maybe people should roleplay that or something, just to avoid getting stuck in an argument-from-authority trap, but that's a silly idea. Either way, the fact that other people spoke positively, and Elizier and other long-standing posters did too, is a good thing.)

However, I'm not sure it's as uniquely a victory for the rationality of LessWrong as it sounds. In responose to srdiamond, Luke quoted tenlier saying "[Holden's] critique mostly consists of points that are pretty persistently bubbling beneath the surface around here, and get brought up quite a bit. Don't most people regard this as a great summary of their current views, rather than persuasive in any way?" To me, that suggests that Holden did a really excellent job expressing these views clearly and persuasively. However, it suggests that previous people had tried to express something similar, but it hadn't been expressed well enough to be widely accepted, and people reading had failed to sufficiently apply the dictum of "fix your opponents' arguments for them". I'm not sure if that's true (it's certainly not automatically true), but I suspect it might be. What do people think?

If there's any truth to it, it suggests one good answer to the recent post http://lesswrong.com/lw/btc/how_can_we_get_more_and_better_lw_contrarians (whether that was desirable in general or not) would be, as a rationalist exercise for someone familiar with/to the community and good at writing rationally, to take a survey of contrarian views on the topic that people on the community may have had but not been able to express, and don't worry about showmanship like pretending to believe it yourself, but just say "I think what some people think is [well-expressed argument]. Do you agree that's fair? If so, do I and other people think they have a point?" Whether or not that argument is right it's still good to engage with it if many people are thinking it.

comment by pleeppleep · 2012-05-12T17:30:48.301Z · score: 4 (4 votes) · LW(p) · GW(p)

Third highest now. Eliezer just barely gets into the top 20.

comment by MarkusRamikin · 2012-05-17T07:56:56.387Z · score: 3 (3 votes) · LW(p) · GW(p)

It is also now the 3rd most highly voted post

1st.

At this point even I am starting to be confused.

comment by TheOtherDave · 2012-05-17T16:30:45.183Z · score: 1 (1 votes) · LW(p) · GW(p)

Can you articulate the nature of your confusion?

comment by MarkusRamikin · 2012-05-17T16:46:56.363Z · score: 7 (9 votes) · LW(p) · GW(p)

I suppose it's that I naively expect, when opening the list of top LW posts ever, to see ones containing the most impressive or clever insights into rationality.

Not that I don't think Holden's post deserves a high score for other reasons. While I am not terribly impressed with his AI-related arguments, the post is of the very highest standards of conduct, of how to have a disagreement that is polite and far beyond what is usually named "constructive".

comment by TheOtherDave · 2012-05-17T17:17:18.661Z · score: 4 (4 votes) · LW(p) · GW(p)

(nods) Makes sense.

My own primary inference from the popularity of this post is that there's a lot of uncertainty/disagreement within the community about the idea that creating an AGI without an explicit (and properly tuned) moral structure constitutes significant existential risk, but that the social dynamics of the community cause most of that uncertainty/disagreement to go unvoiced most of the time.

Of course, there's lots of other stuff going on as well that has little to do with AGI or existential risk, and a lot to do with the social dynamics of the community itself.

comment by [deleted] · 2012-06-14T13:38:11.937Z · score: 0 (0 votes) · LW(p) · GW(p)

Maybe. I upvoted it because it will have (and has had) the effect of improving SI's chances.

comment by aceofspades · 2012-06-07T20:36:37.221Z · score: 2 (2 votes) · LW(p) · GW(p)

Some people who upvoted the post may think it is one of the best-written and most important examples of instrumental rationality on this site.

comment by [deleted] · 2012-05-12T09:43:31.417Z · score: 2 (4 votes) · LW(p) · GW(p)

I wish I could upvote this ten times.

comment by brazil84 · 2013-02-13T23:20:00.490Z · score: 1 (3 votes) · LW(p) · GW(p)

Well perhaps the normal practice is cult-like and dissent-suppressing and this is an atypical break. Kind of like the fat person who starts eating salad instead of nachos while he watches football. And congratulates himself on his healthy eating even though he is still having donuts for breakfast and hamburgers and french fries for lunch.

Seems to me the test for suppression of dissent is not when a high-status person criticizes. The real test is when someone with medium or low status speaks out.

And my impression is that lesswrong does have problems along these lines. Not as bad as other discussion groups, but still.

comment by MarkusRamikin · 2012-05-12T10:17:46.750Z · score: 0 (0 votes) · LW(p) · GW(p)

5th

Looks like 3rd now. As impressed as I am with the post, at this point I'm a little surprised.

comment by Robin · 2012-05-15T11:30:17.383Z · score: -2 (2 votes) · LW(p) · GW(p)

But LW isn't reflective of SI, most of the people that voted on this article have no affiliation with SI. So the high number of upvotes is less reflective of SI welcoming criticism than LW being dissatisfied with the organization of SI.

Furthermore, this post's criticism of Eliezer's research less strong than its criticism of SI's organization . SI has always been somewhat open to criticism of its organizational structure and many of the current leadership of SI has criticized the organizational structure at some point. But who criticize Eliezer's research do not manage to rise in SI's research division and generally aren't well received even on LW (Roko).

Lastly, laughing at somebody when they call your organization a cult is not a convincing argument, they're more likely to think of your organization as a cult (at least they will think you are arrogant).

comment by private_messaging · 2012-05-12T07:16:58.292Z · score: -2 (8 votes) · LW(p) · GW(p)

How's about you also have a critical discussion of 'where can be we wrong and how do we make sure we are actually competent' and 'can we figure out what the AI will actually do, using our tools?' instead of 'how do we communicate our awesomeness better' and 'are we communicating our awesomeness right' ?

This post is something that can't be suppressed without losing big time, and you not suppressing it is only a strong evidence that you are not completely stupid (which is great).

comment by XiXiDu · 2012-05-12T08:54:07.280Z · score: -4 (42 votes) · LW(p) · GW(p)

I shall now laugh harder than ever when people try to say with a straight face that LessWrong is an Eliezer-cult that suppresses dissent.

Holden does not disagree with most of the basic beliefs that SI endorses. Which I think is rather sad and why I don't view him as a real critic. And he has been very polite.

Here is the impolite version:

If an actual AI researcher would have written a similar post, someone who actually tried to build practical systems and had some economic success, not one of those AGI dreamers. If such a person would write a similar post and actually write in a way that they feel, rather than being incredible polite, things would look very different.

The trust is that you are incredible naive when it comes to technological progress. That recursive self-improvement is nothing more than a row of English words, a barely convincing fantasy. That expected utility maximization is practically unworkable, even for a superhuman intelligence. And that the lesswrong.com sequences are not original or important but merely succeed at drowning out all the craziness they include by a huge amount of unrelated clutter and an appeal to the rationality of the author.

What you call an "informed" critic is someone who shares most of your incredible crazy and completely unfounded beliefs.

Worst of all, you are completely unconvincing and do not even notice it because there are so many other people who are strongly and emotionally attached to the particular science fiction scenarios that you envision.

comment by Dolores1984 · 2012-05-14T05:39:28.670Z · score: 11 (13 votes) · LW(p) · GW(p)

If such a person would write a similar post and actually write in a way that they feel, rather than being incredible polite, things would look very different.

I'm assuming you think they'd come in, scoff at our arrogance for a few pages, and then waltz off. Disregarding how many employed machine learning engineers also do side work on general intelligence projects, you'd probably get the same response from automobile engineer, someone with a track record and field expertise, talking to the Wright Brothers. Thinking about new things and new ideas doesn't automatically make you wrong.

That recursive self-improvement is nothing more than a row of English words, a barely convincing fantasy.

Really? Because that's a pretty strong claim. If I knew how the human brain worked well enough to build one in software, I could certainly build something smarter. You could increase the number of slots in working memory. Tweak the part of the brain that handles intuitive math to correctly deal with orders of magnitude. Improve recall to eidetic levels. Tweak the brain's handling of probabilities to be closer to the Bayesian ideal. Even those small changes would likely produce a mind smarter than any human being who has ever lived. That, plus the potential for exponential subjective speedup, is already dangerous. And that's assuming that the mind that results would see zero new insights that I've missed, which is pretty unlikely. Even if the curve bottoms out fairly quickly, after only a generation or two that's STILL really dangerous.

Worst of all, you are completely unconvincing and do not even notice it because there are so many other people who are strongly and emotionally attached to the particular science fiction scenarios that you envision.

Really makes you wonder how all those people got convinced in the first place.

comment by Salemicus · 2012-05-14T13:43:26.282Z · score: 3 (5 votes) · LW(p) · GW(p)

If I knew how the human brain worked well enough to build one in software, I could certainly build something smarter.

This is totally unsupported. To quote Lady Catherine de Bourgh, "If I had ever learned [to play the piano], I should have become a great proficient."

You have no idea whether the "small changes" you propose are technically feasible, or whether these "tweaks" would in fact mean a complete redesign. For all we know, if you knew how the human brain worked well enough to build one in software, you would appreciate why these changes are impossible without destroying the rest of the system's functionality.

After all, it would appear that (say) eidetic recall would provide a fitness advantage. Given that humans lack it, there may well be good reasons why.

comment by TheOtherDave · 2012-05-14T14:43:52.811Z · score: 6 (6 votes) · LW(p) · GW(p)

"totally unsupported" seems extreme. (Though I enjoyed the P&P shoutout. I was recently in a stage adaptation of the book, so it is pleasantly primed.)

What the claim amounts to is the belief that:
a) there exist good design ideas for brains that human evolution didn't implement, and
b) a human capable of building a working brain at all is capable of coming up with some of them.

A seems pretty likely to me... at least, the alternative (our currently evolved brains are the best possible design) seems so implausible as to scarcely be worth considering.

B is harder to say anything clear about, but given our experience with other evolved systems, it doesn't strike me as absurd. We're pretty good at improving the stuff we were born with.

Of course, you're right that this is evidence and not proof. It's possible that we just can't do any better than human brains for thinking, just like it was possible (but turned out not to be true) that we couldn't do any better than human legs for covering long distances efficiently.

But it's not negligible evidence.

comment by Salemicus · 2012-05-14T18:17:14.001Z · score: 2 (2 votes) · LW(p) · GW(p)

I don't doubt that it's possible to come up with something that thinks better than the human brain, just as we have come up with something that travels better than the human leg. But to cover long distances efficiently, people didn't start by replicating a human leg, and then tweaking it. They came up with a radically different design - e.g. the wheel.

I don't see the evidence that knowing how to build a human brain is the key step in knowing how to build something better. For instance, suppose you could replicate neuron function in software, and then scan a brain map (Robin Hanson's "em" concept). That wouldn't allow you to make any of the improvements to memory, maths, etc, that Dolores suggests. Perhaps you could make it run faster - although depending on hardware constraints, it might run slower. If you wanted to build something better, you might need to start from scratch. Or, things could go the other way - we might be able to build "minds" far better than the human brain, yet never be able to replicate a human one.

But it's not just that evidence is lacking - Dolores is claiming certainty in the lack of evidence. I really do think the Austen quote was appropriate.

comment by Dolores1984 · 2012-05-14T20:40:55.343Z · score: 4 (4 votes) · LW(p) · GW(p)

To clarify, I did not mean having the data to build a neuron-by-neuron model of the brain. I meant actually understanding the underlying algorithms those slabs of neural tissue are implementing. Think less understanding the exact structure of a bird's wing, and more understanding the concept of lift.

I think, with that level of understanding, the odds that a smart engineer (even if it's not me) couldn't find something to improve seem low.

comment by TheOtherDave · 2012-05-14T19:16:39.654Z · score: 4 (4 votes) · LW(p) · GW(p)

I agree that I might not need to be able to build a human brain in software to be able to build something better, as with cars and legs.

And I agree that I might be able to build a brain in software without understanding how to do it, e.g., by copying an existing one as with ems.

That said, if I understand the principles underlying a brain well enough to build one in software (rather than just copying it), it still seems reasonable to believe that I can also build something better.

comment by Jonathan_Graehl · 2012-05-13T00:52:00.924Z · score: 5 (7 votes) · LW(p) · GW(p)

I agree that that the tone on both sides is intentionally respectful, and that people here delude themselves if they imagine they aren't up for a bit of mockery from high status folks who don't have the patience to be really engage.

I agree that we don't really know what to expect from the first program that can meaningfully improve itself (including, I suppose, its self-improvement procedure) at a faster pace than human experts working on improving it. It might not be that impressive. But it seems likely to me that it will be a big deal, if ever we get there.

But you're being vague otherwise. Name a crazy or unfounded belief.

comment by XiXiDu · 2012-05-13T10:23:32.424Z · score: -1 (19 votes) · LW(p) · GW(p)

But you're being vague otherwise. Name a crazy or unfounded belief.

Holden asked me something similar today via mail. Here is what I replied:

You wrote in 'Other objections to SI's views':

Unlike the three objections I focus on, these other issues have been discussed a fair amount, and if these other issues were the only objections to SI's arguments I would find SI's case to be strong (i.e., I would find its scenario likely enough to warrant investment in).

It is not strong. The basic idea is that if you pull a mind at random from design space then it will be unfriendly. I am not even sure if that is true. But it is the strongest argument they have. And it is completely bogus because humans do not pull AGI's from mind design space at random.

Further, the whole case for AI risk is based on the idea that there will be a huge jump in capability at some point. Which I think is at best good science fiction, like faster-than-light propulsion, or antimatter weapons (when in doubt that it is possible in principle).

The basic fact that an AGI will most likely need something like advanced nanotechnology to pose a risk, which is itself an existential risk, hints at a conjunction fallacy. We do not need AGI to then use nanotechnology to wipe us out, nanotechnology is already enough if it is possible at all.

Anyway, it feels completely ridiculous to talk about it in the first place. There will never be a mind that can quickly and vastly improve itself and then invent all kinds of technological magic to wipe us out. Even most science fiction books avoid that because it sounds too implausible.

I have written thousands of words about all this and never got any convincing reply. So if you have any specific arguments, let me know.

They say what what I write is unconvincing. But given the amount of vagueness they use to protect their beliefs, my specific criticisms basically amount to a reductio ad absurdum. I don't even need to criticize them, they would have to support their extraordinary beliefs first or make them more specific. Yet I am able to come up with a lot of arguments that speak against the possibility they envision, without any effort and no knowledge of the relevant fields like complexity theory.

Here is a comment I received lately:

…in defining an AGI we are actually looking for a general optimization/compression/learning algorithm which when fed itself as an input, outputs a new algorithm that is better by some multiple. Surely this is at least an NP-Complete if not more problem. It may improve for a little bit and then hit a wall where the search space becomes intractable. It may use heuristics and approximations and what not but each improvement will be very hard won and expensive in terms of energy and matter. But no matter how much it tried, the cold hard reality is that you cannot compute an EXPonential Time algorithm in polynomial time unless (P=EXPTIME :S). A no self-recursive exponential intelligence theorem would fit in with all the other limitations (speed, information density, Turing, Gödel, uncertainties etc) the universe imposes.

If you were to turn IBM Watson gradually into a seed AI, at which point would it become an existential risk and why? They can't answer that at all. It is pure fantasy.

END OF EMAIL

For more see the following posts:

Some old posts:

See also:

If you believe I don't understand the basics, see:

Also:

There is a lot more, especially in the form of comments where I talk about specifics.

comment by Kaj_Sotala · 2012-05-13T21:57:25.171Z · score: 12 (16 votes) · LW(p) · GW(p)

The basic idea is that if you pull a mind at random from design space then it will be unfriendly. I am not even sure if that is true. But it is the strongest argument they have. And it is completely bogus because humans do not pull AGI's from mind design space at random.

I don't have the energy to get into an extended debate, but the claim that this is "the basic idea" or that this would be "the strongest argument" is completely false. A far stronger basic idea is the simple fact that nobody has yet figured out a theory of ethics that would work properly, which means that even that AGIs that were specifically designed to be ethical are most likely to lead to bad outcomes. And that's presuming that we even knew how to program them exactly.

This isn't even something that you'd need to read a hundred blog posts for, it's well discussed in both The Singularity and Machine Ethics and Artificial Intelligence as a Positive and Negative Factor in Global Risk. Complex Value Systems are Required to Realize Valuable Futures, too.

comment by XiXiDu · 2012-05-14T13:12:00.118Z · score: 0 (2 votes) · LW(p) · GW(p)

I did skim through the last paper. I am going to review it thoroughly at some point.

On first sight one of the problems is the whole assumption of AI drives. On the one hand you claim that an AI is going to follow its code, is its code (as if anyone would doubt causality). On the other hand you talk about the emergence of drives like unbounded self-protection. And if someone says that unbounded self-protection does not need to be part of an AGI, you simply claim that your definition of AGI will have those drives. Which allows you to arrive at your desired conclusion of AGI being an existential risk.

Another problem is the idea that an AGI will be a goal executor (I can't help but interpret that to be your position) when I believe that the very nature of artificial general intelligence implies the correct interpretation of "Understand What I Mean" and that "Do What I Mean" is the outcome of virtually any research. Only if you were to pull an AGI at random from mind design space could you possible arrive at "Understand What I Mean" without "Do What I Mean".

To see why look at any software product or complex machine. Those products are continuously improved. Where "improved" means that they become better at "Understand What I Mean" and "Do What I Mean".

There is no good reason to believe that at some point that development will suddenly turn into "Understand What I Mean" and "Go Batshit Crazy And Do What I Do Not Mean".

There are other problems with the paper. I hope I will find some time to write a review soon.

One problem for me with reviewing such papers is that I doubt a lot of underlying assumptions like that there exists a single principle of general intelligence. As I see it there will never be any sudden jump in capability. I also think that intelligence and complex goals are fundamentally interwoven. An AGI will have to be hardcoded, or learn, to care about a manifold of things. No simple algorithm, given limited computational resources, will give rise to the drives that are necessary to undergo strong self-improvement (if that is possible at all).

comment by Swimmer963 · 2012-05-14T05:18:14.974Z · score: 7 (7 votes) · LW(p) · GW(p)

Even most science fiction books avoid that because it sounds too implausible.

Not saying I particularly disagree with your other premises, but saying something can't be true because it sounds implausible is not a valid argument.

comment by Desrtopa · 2012-05-14T04:21:05.283Z · score: 5 (7 votes) · LW(p) · GW(p)

It is not strong. The basic idea is that if you pull a mind at random from design space then it will be unfriendly. I am not even sure if that is true. But it is the strongest argument they have. And it is completely bogus because humans do not pull AGI's from mind design space at random.

An AI's mind doesn't have to be pulled from design space at random to be disastrous. The primary issue that the SIAI has to grapple with (based on my understanding,) is that deliberately designing an AI that does what we would want it to do, rather than fulfilling proxy criteria in ways that we would not like at all, is really difficult. Even getting one to recognize "humans" as a category in a way that would be acceptable to us is a major challenge.

comment by jsteinhardt · 2012-05-15T03:44:36.861Z · score: -1 (3 votes) · LW(p) · GW(p)

Although it's worth pointing out that this is also an obstacle to AGI, since presumably an AI that did not understand what a human was would be pretty unintelligent. So I think it's unfair to claim this as a "friendliness" issue.

Note that I do think there are some important friendliness-related problems, but, assuming I understand your objection, this is not one of them.

comment by Desrtopa · 2012-05-15T03:57:15.404Z · score: 1 (1 votes) · LW(p) · GW(p)

An AI could be an extremely powerful optimizer without having a category for "humans" that mapped to our own. "Human," the way we conceive of it, is a leaky surface generalization.

A strong paperclip maximizer would understand humans as well as it had to to contend with us in its attempts to paperclip the universe, but it wouldn't care about us. And a strong optimizer programmed to maximize the values of "humans" would also probably understand us, but if we don't program into its values an actual category that maps to our conception of humans, it could perfectly well end up applying that understanding to, for example, tiling the universe with crash test dummies.

comment by jsteinhardt · 2012-05-15T04:31:59.049Z · score: 3 (3 votes) · LW(p) · GW(p)

How do you intend to build a powerful optimizer without having a method of representing (or of building a representation of) the concept of "human" (where "human" can be replaced with any complex concept, even probably paperclips)?

I agree that value specification is a hard problem. But I don't think the complexity of "human" is the reason for this, although it does rule out certain simple approaches like hard-coding values.

(Also, since your link seems to indicate you believe otherwise, I am fairly familiar with the content in the sequences. Apologies if this statement represents an improper inference.)

comment by Desrtopa · 2012-05-15T04:57:16.014Z · score: 4 (4 votes) · LW(p) · GW(p)

How do you intend to build a powerful optimizer without having a method of representing (or of building a representation of) the concept of "human" (where "human" can be replaced with any complex concept, even probably paperclips)?

If a machine can learn, empirically, exactly what humans are, on the most fundamental levels, but doesn't have any values associated with them, why should it need a concept of "human?" We don't have a category that distinguishes igneous rocks that are circular and flat on one side, but we can still recognize them and describe them precisely.

Humans are an unnatural category. Whether a fetus, an individual in a persistent vegetative state, an amputee, a corpse, an em or a skin cell culture fall into the category of "human" depends on value-sensitive boundaries. It's not necessarily because humans are so complex that we can't categorize them in an appropriate manner for an AI (or at least, not just because humans are complex,) it's because we don't have an appropriate formulation of the values that would allow a computer to draw the boundaries of the category in a way we'd want it to.

(I wasn't sure how familiar you were with the sequences, but in any case I figured it can't hurt to add links for anyone who might be following along who's not familiar.)

comment by Jonathan_Graehl · 2012-05-13T21:46:25.166Z · score: 4 (8 votes) · LW(p) · GW(p)

I've read most of that now, and have subscribed to your newsletter.

Reasonable people can disagree in estimating the difficulty of AI and the visibility/pace of AI progress (is it like hunting for a single breakthrough and then FOOM? etc).

I find all of your "it feels ridiculous" arguments by analogy to existing things interesting but unpersuasive.

comment by Dolores1984 · 2012-05-14T04:37:40.340Z · score: 3 (7 votes) · LW(p) · GW(p)

Anyway, it feels completely ridiculous to talk about it in the first place. There will never be a mind that can quickly and vastly improve itself and then invent all kinds of technological magic to wipe us out. Even most science fiction books avoid that because it sounds too implausible.

Says the wooly mammoth, circa 100,000 BC.

Sounding silly and low status and science-fictiony doesn't actually make it unlikely to happen in the real world.

comment by JoshuaZ · 2012-05-14T18:21:58.970Z · score: 4 (4 votes) · LW(p) · GW(p)

Especially when not many people want to read a science fiction book where humanity gets quickly and completely wiped out by a superior force. Even works where humans slowly die off due to their own problems (e.g. On the Beach) are uncommon.

comment by brahmaneya · 2012-05-14T04:03:44.876Z · score: 1 (3 votes) · LW(p) · GW(p)

Anyway, it feels completely ridiculous to talk about it in the first place. There will never be a mind that can quickly and vastly improve itself and then invent all kinds of technological magic to wipe us out. Even most science fiction books avoid that because it sounds too implausible

Do you acknowledge that :

  1. We will some day make an AI that is at least as smart as humans?
  2. Humans do try to improve their intelligence (rationality/memory training being a weak example, cyborg research being a better example, and im pretty sure we will soon design physical augmentations to improve our intelligence)

If you acknowledge 1 and 2, then that implies there can (and probably will) be an AI that tries to improve itself

comment by jsteinhardt · 2012-05-15T03:39:55.532Z · score: 4 (4 votes) · LW(p) · GW(p)

I think you missed the "quickly and vastly" part as well as the "and then invent all kinds of technological magic to wipe us out". Note I still think XiXiDu is wrong to be as confident as he is (assuming "there will never" implies >90% certainty), but if you are going to engage with him then you should engage with his actual arguments.

comment by Swimmer963 · 2012-05-14T05:20:32.814Z · score: 3 (3 votes) · LW(p) · GW(p)

And that the lesswrong.com sequences are not original or important but merely succeed at drowning out all the craziness they include by a huge amount of unrelated clutter and an appeal to the rationality of the author.

Name three examples? (Of 'craziness' specifically... I agree that there are frequent, and probably unecessary, "appeals to the rationality of the author".)

comment by [deleted] · 2012-05-14T05:33:13.951Z · score: -1 (7 votes) · LW(p) · GW(p)

Name three examples? (Of 'craziness' specifically... I agree that there are frequent, and probably unecessary, "appeals to the rationality of the author".)

XiXiDu may be too modest; he has some great examples on his blog.

comment by [deleted] · 2012-05-14T06:18:20.994Z · score: 2 (4 votes) · LW(p) · GW(p)

One wonders when or if XiXiDu will ever get over the Roko incident. Yes, it was a weird and possibly disproportionate response, but it was also years ago.

comment by JoshuaZ · 2012-05-14T20:16:29.923Z · score: 0 (2 votes) · LW(p) · GW(p)

Do we have any evidence that Eliezer's attitude or approach to that sort of thing has changed since then?

comment by [deleted] · 2012-05-14T20:24:04.031Z · score: 6 (6 votes) · LW(p) · GW(p)

Sure. His moderation activities over the last year or so have been far more... sunglasses... moderate.

comment by [deleted] · 2012-05-14T06:22:27.955Z · score: -7 (7 votes) · LW(p) · GW(p)

So said Newt Gingrich.

comment by [deleted] · 2012-05-14T06:27:19.606Z · score: 7 (7 votes) · LW(p) · GW(p)

Why yes, I do also believe that political figures are held to ridiculous conversational standards as well. It's a miracle they deign to talk to anyone.

comment by [deleted] · 2012-05-14T17:43:05.754Z · score: -7 (11 votes) · LW(p) · GW(p)

Name three examples? (Of 'craziness' specifically... I agree that there are frequent, and probably unecessary, "appeals to the rationality of the author".)

So, Swimmer 963, are those quotes crazy enough for you? (I hope you don't ask a question and neglect to comment on the answer.) What you do think? Anomalous?

Contrary to the impression the comments might convey, the majority don't come from the Roko incident. But as to that incident, the passage of time doesn't necessarily erase the marks of character. Romney is rightfully being held, feet to fire, for a group battering of another student while they attended high school--because such sadism is a trait of character and can't be explained otherwise. How would one explain Yudkowsky's paranoia, lack of perspective, and scapegoating--other than by positing a narcissistic personality structure?

Many LWers can't draw conclusions because they eschew the only tools for that purpose: psychology and excellent fiction. And the second is more important than the first.

comment by Swimmer963 · 2012-05-14T19:25:17.446Z · score: 9 (9 votes) · LW(p) · GW(p)

How would one explain Yudkowsky's paranoia, lack of perspective, and scapegoating--other than by positing a narcissistic personality structure?

I had in fact read a lot of those quotes before–although some of them come as a surprise, so thank you for the link. They do show paranoia and lack of perspective, and yeah, some signs of narcissism, and I would be certainly mortified if I personally ever made comments like that in public...

The Sequences as a whole do come across as having been written by an arrogant person, and that's kind of irritating, and I have to consciously override my irritation in order to enjoy the parts that I find useful, which is quite a lot. It's a simplification to say that the Sequences are just clutter, and it's extreme to call them 'craziness', too.

(Since meeting Eliezer in person, it's actually hard for me to believe that those comments were written by the same person, who was being serious about them... My chief interaction with him was playing a game in which I tried to make a list of my values, and he hit me with a banana every time I got writer's block because I was trying to be too specific, and made the Super Mario Brothers' theme song when I succeeded. It's hard making the connection that "this is the same person who seems to take himself way too seriously in his blog comments." But that's unrelated and doesn't prove anything in either direction.)

My main point is that criticizing someone who believes in a particular concept doesn't irrefutably damn that concept. You can use it as weak evidence, but not proof. Eliezer, as far as I know, isn't the only person who has thought extensively about Friendly AI and found it a useful concept to keep.

comment by [deleted] · 2012-05-14T22:06:08.587Z · score: -8 (14 votes) · LW(p) · GW(p)

The quotes aren't all about AI. A few:

Take metaethics, a solved problem: what are the odds that someone who still thought metaethics was a Deep Mystery could write an AI algorithm that could come up with a correct metaethics? I tried that, you know, and in retrospect it didn’t work.

Yudkowsky makes the megalomanic claim that he's solved the questions of metaethics. His solution: morality is the function that the brain of a fully informed subject computes to determine what's right. Laughable; pathologically arrogant.

Whoever knowingly chooses to save one life, when they could have saved two – to say nothing of a thousand lives, or a world – they have damned themselves as thoroughly as any murderer.

The most extreme presumptuousness about morality; insufferable moralism. Morality, as you were perhaps on the cusp of recognizing in one of your posts, Swimmer963, is a personalized tool, not a cosmic command line. See my "Why do what you "ought"?—A habit theory of explicit morality."

The preceding remark, I'll grant, isn't exactly crazy--just super obnoxious and creepy.

Science is built around the assumption that you’re too stupid and self-deceiving to just use Solomonoff induction. After all, if it was that simple, we wouldn’t need a social process of science right?

This is where Yudkowsky goes crazy autodidact bonkers. He thinks the social institution of science is superfluous, were everyone as smart as he. This means he can hold views contrary to scientific consensus in specialized fields where he lacks expert knowledge based on pure ratiocination. That simplicity in the information sense equates with parsimony is most unlikely; for one thing, simplicity is dependent on choice of language--an insight that should be almost intuitive to a rationalist. But noncrazy people may believe the foregoing; what they don't believe is that they can at the present time replace the institution of science with the reasoning of smart people. That's the absolutely bonkers claim Yudkowsky makes.

>

comment by Swimmer963 · 2012-05-15T00:04:47.644Z · score: 9 (9 votes) · LW(p) · GW(p)

The quotes aren't all about AI.

I didn't say they were. I said that just because the speaker for a particular idea comes across as crazy doesn't mean the idea itself is crazy. That applies whether all of Eliezer's "crazy statements" are about AI, or whether none of them are.

Whoever knowingly chooses to save one life, when they could have saved two – to say nothing of a thousand lives, or a world – they have damned themselves as thoroughly as any murderer.

The most extreme presumptuousness about morality; insufferable moralism.

Funny, I actually agree with the top phrase. It's written in an unfortunately preachy, minister-scaring-the-congregation-by-saying-they'll-go-to-Hell style, which is guaranteed to make just about anyone get defensive and/or go "ick!" But if you accept the (very common) moral standard that if you can save a life, it's better to do it than not to do it, then the logic is inevitable that if you have the choice of saving one lives or two lives, by your own metric it's morally preferable to save two lives. If you don't accept the moral standard that it's better to save one life than zero lives, then that phrase should be just as insufferable.

Science is built around the assumption that you’re too stupid and self-deceiving to just use Solomonoff induction. After all, if it was that simple, we wouldn’t need a social process of science right?

I decided to be charitable, and went and looked up the post that this was in: it's here. As far as I can tell, Eliezer doesn't say anything that could be interpreted as "science exists because people are stupid, and I'm not stupid, therefore I don't need science". He claims that scientific procedures compensates for people being unwilling to let go of their pet theories and change their minds, and although I have no idea if this goal was in the minds of the people who came up with the scientific method, it doesn't seem to be false that it accomplishes this goal.

comment by hairyfigment · 2012-05-15T03:36:13.943Z · score: 1 (1 votes) · LW(p) · GW(p)

Newton definitely wrote down his version of scientific method to explain why people shouldn't take his law of gravity and just add, "because of Aristotelian causes," or "because of Cartesian mechanisms."

comment by JoshuaZ · 2012-05-14T22:21:03.721Z · score: 8 (8 votes) · LW(p) · GW(p)

This is where Yudkowsky goes crazy autodidact bonkers. He thinks the social institution of science is superfluous, were everyone as smart as he. This means he can hold views contrary to scientific consensus in specialized fields where he lacks expert knowledge based on pure ratiocination.

Ok. I disagree with a large bit of the sequences on science and the nature of science. I've wrote a fair number of comments saying so. So I hope you will listen when I say that you are taking a strawman version of what Eliezer wrote on these issues, and it almost borders on something that I could only see someone thinking if they were trying to interpret Eliezer's words in the most negative fashion possible.

comment by Dolores1984 · 2012-05-14T22:13:46.050Z · score: 5 (5 votes) · LW(p) · GW(p)

His solution: morality is the function that the brain of a fully informed subject computes to determine what's right. Laughable; pathologically arrogant.

You either didn't read that sequence carefully, or are intentionally misrepresenting it.

He thinks the social institution of science is superfluous, were everyone as smart as he.

Didn't read that sequence carefully either.

That simplicity in the information sense equates with parsimony is most unlikely; for one thing, simplicity is dependent on choice of language--an insight that should be almost intuitive to a rationalist.

You didn't read that sequence at all, and probably don't actually know what simplicity means in an information-theoretic sense.

comment by [deleted] · 2012-05-14T22:22:24.378Z · score: 5 (5 votes) · LW(p) · GW(p)

That simplicity in the information sense equates with parsimony is most unlikely; for one thing, simplicity is dependent on choice of language--an insight that should be almost intuitive to a rationalist.

You didn't read that sequence at all, and probably don't actually know what simplicity means in an information-theoretic sense.

To be fair, that sequence doesn't really answer questions about choice-of-language; it took reading some of Solomonoff's papers for me to figure out what the solution to that problem is.

comment by JoshuaZ · 2012-05-14T22:37:06.537Z · score: 2 (2 votes) · LW(p) · GW(p)

it took reading some of Solomonoff's papers for me to figure out what the solution to that problem is.

There are a variety of proposed solutions. None of them seem perfect.

comment by [deleted] · 2012-05-14T22:40:54.174Z · score: 2 (2 votes) · LW(p) · GW(p)

I'm referring to encoding in several different languages, which makes it progressively more implausible that choice of language matters.

I agree that's not a perfect solution, but it's good enough for me.

comment by [deleted] · 2012-05-14T22:23:40.371Z · score: -3 (7 votes) · LW(p) · GW(p)

That's true; I admit I didn't read the sequence. I had a hard time struggling through the single summating essay. What I wrote was his conclusion. As Hanson wrote in the first comment to the essay I did read, Yudkowsky really should summarize the whole business in a few lines. Yudkowsky didn't get around to that, as far as I know.

The summation essay contained more than 7,000 words for the conclusion I quoted. Maybe the rest of the series contradicts what is patent in the essay I read.

I simply don't get the attraction of the sequences. An extraordinarily high ratio of filler to content; Yudkowsky seems to think that every thought along the way to his personal enlightenment is worth the public's time.

Asking that a critic read those sequences in their entirety is asking for a huge sacrifice; little is offered to show it's even close in being worth the misery of reading inept writing or the time.

comment by Dolores1984 · 2012-05-14T22:39:19.122Z · score: 4 (8 votes) · LW(p) · GW(p)

You know, the sequences aren't actually poorly written. I've read them all, as have most of the people here. They are a bit rambly in places, but they're entertaining and interesting. If you're having trouble with them, the problem might be on your end.

In any case, if you had read them, you'd know, for instance, that when Yudkowsky talks about simplicity, he is not talking about the simplicity of a given English sentence. He's talking about the combined complexity of a given Turing machine and the program needed to describe your hypothesis on that Turing machine.

comment by gwern · 2012-05-17T00:21:00.930Z · score: 4 (4 votes) · LW(p) · GW(p)

have most of the people here

http://lesswrong.com/lw/8p4/2011_survey_results/

89 people (8.2%) have never looked at the Sequences; a further 234 (32.5%) have only given them a quick glance. 170 people have read about 25% of the sequences, 169 (15.5%) about 50%, 167 (15.3%) about 75%, and 253 people (23.2%) said they've read almost all of them. This last number is actually lower than the 302 people who have been here since the Overcoming Bias days when the Sequences were still being written (27.7% of us).

  • 23% for 'almost all'
  • 39% have read > three-quarters
  • 54% have read > half
comment by Dolores1984 · 2012-05-17T00:31:35.036Z · score: 0 (0 votes) · LW(p) · GW(p)

My mistake. I'll remember that in the future.

comment by Bugmaster · 2012-05-14T23:01:02.808Z · score: 2 (2 votes) · LW(p) · GW(p)

In addition, there are places in the Sequences where Eliezer just states things as though he's dispensing wisdom from on high, without bothering to state any evidence or reasoning. His writing is still entertaining, of course, but still less than persuasive.

comment by dlthomas · 2012-05-14T22:56:09.760Z · score: 2 (2 votes) · LW(p) · GW(p)

They are a bit rambly in places, but they're entertaining and interesting.

I also found this to be true.

comment by [deleted] · 2012-05-14T22:42:40.190Z · score: 2 (2 votes) · LW(p) · GW(p)

as have most of the people here.

I'm pretty sure the 2011 survey puts this claim to the test, but I don't have the time to look it up.

comment by [deleted] · 2012-05-14T23:50:27.615Z · score: 1 (7 votes) · LW(p) · GW(p)

You know, the sequences aren't actually poorly written. I've read them all, as have most of the people here. They are a bit rambly in places, but they're entertaining and interesting. If you're having trouble with them, the problem might be on your end.

The problem is partly on my end, for sure; obviously, I find rambling intolerable in Internet writing, and I find it in great abundance in the sequences. You're more tolerant of rambling, and you're entertained by Yudkowsky's. I also think he demonstrates mediocre literary skills when it comes to performances like varying his sentence structure. I don't know what you think of that. My guess is you don't much care; maybe it's a generational thing.

I'm intrigued by what enjoyment readers here get from Yudkowsky's sequences. Why do you all find interesting what I find amateurish and inept? Do we have vastly different tastes or standards, or both? Maybe it is the very prolixity that makes the writing appealing in founding a movement with religious overtones. Reading Yudkowsky is an experience comparable to reading the Bible.

As a side issue, I'm dismayed upon finding that ideas I had thought original to Yudkowsky were secondhand.

Of course I understand simplicity doesn't pertain to simplicity in English! (Or in any natural language.) I don't think you understand the language-relativity issue.

comment by TheOtherDave · 2012-05-15T00:43:16.146Z · score: 4 (4 votes) · LW(p) · GW(p)

If you were willing to point me to two or three of your favorite Internet writers, whom you consider reliably enjoyable and interesting and so forth, I might find that valuable for its own sake, and might also be better able to answer your question in mutually intelligible terms.

comment by Swimmer963 · 2012-05-15T00:11:21.860Z · score: 2 (2 votes) · LW(p) · GW(p)

As a side issue, I'm dismayed upon finding that ideas I had thought original to Yudkowsky were secondhand.

Having to have original ideas is a very high standard. I doubt a single one of my posts contains a truly original idea, and I don't try–I try to figure out which ideas are useful to me, and then present why, in a format that I hope will be useful to others. Eliezer creates a lot of new catchy terms for pre-existing ideas, like "affective death spiral" for "halo effect." I like that.

His posts are also quite short, often witty, and generally presented in an easier-to-digest format than the journal articles I might otherwise have to read to encounter the not-new ideas. You apparently don't find his writing easy to digest or amusing in the same way I do.

comment by thomblake · 2012-05-15T17:51:59.448Z · score: 3 (3 votes) · LW(p) · GW(p)

like "affective death spiral" for "halo effect."

Affective death spiral is not the same thing as the Halo effect, though the halo effect (/ horns effect) might be part of the mechanism of affective death spiral.

comment by Swimmer963 · 2012-05-15T19:52:43.470Z · score: 3 (3 votes) · LW(p) · GW(p)

Agreed... I think the Halo effect is a sub-component of an affective death spiral, and "affective death spiral" is a term unique to LW [correct me if I'm wrong!], while 'Halo effect' isn't.

comment by JoshuaZ · 2012-05-15T02:18:45.650Z · score: 1 (1 votes) · LW(p) · GW(p)

As a side issue, I'm dismayed upon finding that ideas I had thought original to Yudkowsky were secondhand.

Are there specific examples? It seems to me that in most cases when he has a pre-existing idea he gives relevant sources.

comment by Normal_Anomaly · 2012-05-16T13:55:56.065Z · score: 1 (1 votes) · LW(p) · GW(p)

I don't know any specific examples of secondhand ideas coming off as original (indeed, he often cites experiments from the H&B literature), but there's another possible source for the confusion. Sometimes Yudkowsky and somebody else come up with ideas independently, and those aren't cited because Yudkowsky didn't know they existed at the time. Drescher and Quine are two philosophers who have been mentioned as having some of the same ideas as Yudkowsky, and I can confirm the former from experience.

comment by Bugmaster · 2012-05-15T00:28:36.625Z · score: 1 (1 votes) · LW(p) · GW(p)

I'm intrigued by what enjoyment readers here get from Yudkowsky's sequences. Why do you all find interesting what I find amateurish and inept?

I find his fictional interludes quite entertaining, because they are generally quite lively, and display a decent amount of world-building -- which is one aspect of science fiction and fantasy that I particularly enjoy. I also enjoy the snark he employs when trashing opposing ideas, especially when such ideas are quite absurd. Of course, the snark doesn't make his writing more persuasive -- just more entertaining.

he demonstrates mediocre literary skills when it comes to performances like varying his sentence structure

I know I'm exposing my ignorance here, but I'm not sure what this means; can you elaborate ?

comment by Randaly · 2012-05-15T19:35:32.107Z · score: 2 (2 votes) · LW(p) · GW(p)

Asking that a critic read those sequences in their entirety is asking for a huge sacrifice; little is offered to show it's even close in being worth the misery of reading inept writing or the time.

Indeed, the sequences are long. I'm not sure about the others here, but I've never asked anybody to "read the sequences."

But I don't even know how to describe the arrogance required to believe that you can dismiss somebody's work as "crazy," "stupid," "megalomanic," "laughably, pathologically arrogant," "bonkers," and "insufferable" without having even read enough of what you're criticizing the get an accurate understanding of it.

ETA: Edited in response to fubarobfusco, who brought up a good point.

comment by fubarobfusco · 2012-05-15T23:00:08.515Z · score: 3 (3 votes) · LW(p) · GW(p)

That's a fully general argument against criticizing anything without having read all of it, though. And there are some things you can fairly dismiss without having read all of. For instance, I don't have to read every page on the Time Cube site to dismiss it as crazy, stupid, pathologically arrogant, and so on.

comment by nshepperd · 2012-05-15T08:08:24.713Z · score: 1 (1 votes) · LW(p) · GW(p)

The reason EY wrote an entire sequence on metaethics is precisely because without the rest of the preparation people such as you who lack all that context immediately veer off course and start believing that he's asserting the existence (or non-existence) of "objective" morality, or that morality is about humans because humans are best or any other standard philosophical confusion that people automatically come up with whenever they think about ethics.

Of course this is merely a communication issue. I'd love to see a more skilled writer present EY's metaethical theory in a shorter form that still correctly conveys the idea, but it seems to be very difficult (especially since even half the people who do read the sequence still come away thinking it's moral relativism or something).

comment by Mass_Driver · 2012-05-15T03:04:41.471Z · score: 4 (4 votes) · LW(p) · GW(p)

I read your post on habit theory, and I liked it, but I don't think it's an answer to the question "What should I do?"

It's interesting to say that if you're an artist, you might get more practical use out of virtue theory, and if you're a politician, you might get more practical use out of consequentialism. I'm not sure who it is that faces more daily temptations to break the rules than the rest of us; bankers, I suppose, and maybe certain kinds of computer security experts.

Anyway, saying that morality is a tool doesn't get you out of the original need to decide which lifestyle you want in the first place. Should I be an artist, or a politician, or a banker? Why? Eliezer's answer is that there are no shortcuts and no frills here; you check and see what your brain says about what you 'should' do, and that's all there is to it. This is not exactly a brilliant answer, but it may nevertheless be the best one out there. I've never yet heard a moral theory that made more sense than that, and believe me, I've looked.

It's reasonable to insist that people put their conclusions in easily digestible bullet points to convince you to read the rest of what they've written...but if, noting that there are no such bullet points, you make the decision not to read the body text -- you should probably refrain from commenting on the body text. A license to opt-out is not the same thing as a license to offer serious criticism. Eliezer may be wrong, but he's not stupid, and he's not crazy. If you want to offer a meaningful critique of his ideas, you'll have to read them first.

comment by [deleted] · 2012-05-15T04:10:35.453Z · score: -4 (8 votes) · LW(p) · GW(p)

but if, noting that there are no such bullet points, you make the decision not to read the body text -- you should probably refrain from commenting on the body text. A license to opt-out is not the same thing as a license to offer serious criticism. Eliezer may be wrong, but he's not stupid, and he's not crazy.

This is sound general advice, but at least one observation makes this situation exceptional: Yudkowsky's conclusions about ethics are never summarized in terms that contradict my take. I don't think your rendition, for example, contradicts mine. I'm certainly not surprised to hear his position described the way you describe it:

Anyway, saying that morality is a tool doesn't get you out of the original need to decide which lifestyle you want in the first place. Should I be an artist, or a politician, or a banker? Why? Eliezer's answer is that there are no shortcuts and no frills here; you check and see what your brain says about what you 'should' do, and that's all there is to it.

Now, I don't think the decision of whether to be an artist, politician, or banker is a moral decision. It isn't one you make primarily because of what's ethically right or wrong. To the extent you do (and in the restricted sense that you do), your prior moral habits are your only guide.

But we're looking at whether Yudkowsky's position is intellectually respectable, not whether objective morality--which he's committed to but I deny--exists. To say we look at what our brain says when we're fully informed says essentially that we seek a reflective equilibrium in solving moral problems. So far so good. But it goes further in saying brains compute some specific function that determines generally when individuals reach that equilibrium. Leaving aside that this is implausible speculation, requiring that the terms of moral judgments be hardwired--and hardwired identically for each individual--it also simply fails to answer Moore's open question, although Yudkowsky claims he has that answer. There's nothing prima facie compelling ethically about what our brains happen to tell us is moral; no reason we should necessarily follow our brains' hardwiring. I could consistently choose to consider my brain's hardwired moralisms maladaptive or even despicable holdovers from the evolutionary past that I choose to override as much as I can.

Robin Hanson actually asked the right question. If what the brain computes is moral, what does it correspond to that makes it moral? Unless you think the brain is computing a fact about the world, you can't coherently regard its computation as "accurate." But if not, what makes it special and not just a reflex?

I do feel a bit guilty about criticizing Yudkowsky without reading all of him. But he seems to express his ideas at excessive and obfuscating length, and if there were more to them, I feel somewhat confident I'd come across his answers. It isn't as though I haven't skimmed many of these essays. And his answers would certainly deserve some reflection in his summation essay.

There's no question Yudkowsky is no idiot. But he has some ideas that I think are stupid--like his "metaethics"--and he expresses them in a somewhat "crazy" manner, exuding grandiose self-confidence. Being surrounded and discussing mostly with people who agree with him is probably part of the cause.

comment by Furcas · 2012-05-15T05:25:01.782Z · score: 6 (10 votes) · LW(p) · GW(p)

As someone who has read Eliezer's metaethics sequence, let me say that what you think his position is, is only somewhat related to what it actually is; and also, that he has answered those of your objections that are relevant.

It's fine that you don't want to read 30+ fairly long blog posts, especially if you dislike the writing style. But then, don't try to criticize what you're ignorant about. And no, openly admitting that you haven't read the arguments you're criticizing, and claiming that you feel guilty about it, doesn't magically make it more acceptable. Or honest.

comment by JoshuaZ · 2012-05-15T17:35:10.923Z · score: 2 (4 votes) · LW(p) · GW(p)

One doesn't need to have read the whole Bible to criticize it. But the Bible is a fairly short work, so an even more extreme example might be better: one doesn't need to have read the entire Talmud to criticize it.

comment by [deleted] · 2012-05-15T17:01:08.918Z · score: -4 (10 votes) · LW(p) · GW(p)

It's fine that you don't want to read 30+ fairly long blog posts, especially if you dislike the writing style. But then, don't try to criticize what you're ignorant about. And no, openly admitting that you haven't read the arguments you're criticizing, and claiming that you feel guilty about it, doesn't magically make it more acceptable. Or honest.

It's hardly "dishonest" to criticize a position based on a 7,000-word summary statement while admitting you haven't read the whole corpus! You're playing with words to make a moralistic debating point: dishonesty involves deceit, and everyone has been informed of the basis for my opinions.

Consider the double standard involved. Yudkowsky lambasts "philosophers" and their "confusions"--their supposedly misguided concerns with the issues other philosophers have commented on to the detriment of inquiry. Has Yudkowsky read even a single book by each of the philosophers he dismisses?

In a normal forum, participants supply the arguments supposedly missed by critics who are only partially informed. Here there are vague allusions to what the Apostle Yudkowsky (prophet of the Singularity God) "answered" without any substance. An objective reader will conclude that the Prophet stands naked; the prolixity is probably intended to discourage criticism.

comment by Kaj_Sotala · 2012-05-15T19:11:38.667Z · score: 5 (5 votes) · LW(p) · GW(p)

I think the argument you make in this comment isn't a bad one, but the unnecessary and unwarranted "Apostle Yudkowsky (prophet of the Singularity God)" stuff amounts to indirectly insulting the people you're talking with and, makes them far less likely to realize that you're actually also saying something sensible. If you want to get your points across, as opposed to just enjoying a feeling of smug moral superiority while getting downvoted into oblivion, I strongly recommend leaving that stuff out.

comment by [deleted] · 2012-05-24T17:26:04.156Z · score: 0 (4 votes) · LW(p) · GW(p)

Thanks for the advice, but my purpose—given that I'm an amoralist—isn't to enjoy a sense of moral superiority. Rather, to test a forum toward which I've felt ambivalent for several years, mainly for my benefit but also for that of any objective observers.

Strong rhetoric is often necessary in an unreceptive forum because it announces that the writer considers his criticisms fundamental. If I state the criticisms neutrally, something I've often tried, they are received as minor—like the present post. They may even be voted up, but they have little impact. Strong language is appropriate in expressing severe criticisms.

How should a rationalist forum respond to harsh criticism? It isn't rational to fall prey to the primate tendency to in-group thinking by neglecting to adjust for any sense of personal insult when the group leader is lambasted. Judging by reactions, the tendency to in-group thought is stronger here than in many forums that don't claim the mantle of rationalism. This is partly because the members are more intelligent than in most other forums, and intelligence affords more adept self-deception. This is why it is particularly important for intelligent people to be rationalists but only if they honestly strive to apply rational principles to their own thinking. Instead, rationality here serves to excuse participants' own irrationality. Participants simply accept their own tendencies to reject posts as worthless because they contain matter they find insulting. Evolutionary psychology, for instance, here serves to produce rationalizations rather than rationality. (Overcoming Bias is a still more extreme advocacy of this perversion of rationalism, although the tendency isn't expressed in formal comment policies.)

"Karma" means nothing to me except as it affects discourse; I despise even the term, which stinks of Eastern mysticism. I'm told that the karma system of incentives, which any rationalist should understand vitally affects the character of discussion, was transplanted from reddit. How is a failure to attend to the vital mechanics of discussion and incentives rational? Laziness? How could policies so essential be accorded the back seat?

Participants, I'm told, don't question the karma system because it works. A rationalist doesn't think that way. He says, "If a system of incentives introduced without forethought and subject to sound criticisms (where even its name is an insult to rationality) produces the discourse that we want, then something must be wrong with what we want!" What's wanted is the absence of any tests of ideology by fundamental dissent.

I think the argument you make in this comment isn't a bad one, but the unnecessary and unwarranted "Apostle Yudkowsky (prophet of the Singularity God)" stuff amounts to indirectly insulting the people you're talking with and, makes them far less likely to realize that you're actually also saying something sensible. If you want to get your points across, as opposed to just enjoying a feeling of smug moral superiority while getting downvoted into oblivion, I strongly recommend leaving that stuff out.

comment by JoshuaZ · 2012-05-15T17:51:57.958Z · score: 2 (2 votes) · LW(p) · GW(p)

Consider the double standard involved. Yudkowsky lambasts "philosophers" and their "confusions"--their supposedly misguided concerns with the issues other philosophers have commented on to the detriment of inquiry. Has Yudkowsky read even a single book by each of the philosophers he dismisses?

Some of them are simply not great writers. Hegel for example is just awful- the few coherent ideas in Hegel are more usefully described by other later writers. There's also a strange aspect to this in that you are complaining about Eliezer not having read books while simultaneously defending your criticism of Eliezer's metaethics positions without having read all his posts. Incidentally, if one wants to criticize Eliezer's level of knowledge of philosophy, a better point is not so much the philosophers that he criticizes without reading, but rather his lack of knowledge of relevant philosophers that Eliezer seems unaware of, many of whom would agree with some of his points. Quine and Lakatos are the most obvious ones.

Here there are vague allusions to what the Apostle Yudkowsky (prophet of the Singularity God) "answered" without any substance. An objective reader will conclude that the Prophet stands naked; the prolixity is probably intended to discourage criticism.

I strongly suspect that your comments would be responded to more positively if they didn't frequently end with this sort of extreme rhetoric that has more emotional content than rational dialogue. It is particularly a problem because on theLW interface, the up/down buttons are at the end of everything one has read, so what the last sentences say may have a disproportionate impact on whether people upvote or downvote and what they focus on in their replies.

Frankly, you have some valid points, but they are getting lost in the rhetoric. We know that you think that LW pattern matches to religion. Everyone gets the point. You don't need to repeat that every single time you make a criticism.

comment by Strange7 · 2012-05-21T22:38:14.456Z · score: 2 (4 votes) · LW(p) · GW(p)

I could consistently choose to consider my brain's hardwired moralisms maladaptive or even despicable holdovers from the evolutionary past that I choose to override as much as I can.

And you would be making the decision to override with... what, your spleen?

comment by [deleted] · 2012-05-21T23:47:07.444Z · score: 0 (6 votes) · LW(p) · GW(p)

Another part of my brain--besides the part computing the morality function Yudkowsky posits.

Surely you can't believe Yudkowsky simply means whatever our brain decides is "moral"--and that he offers that as a solution to anything?

comment by Strange7 · 2012-05-22T09:36:11.485Z · score: 1 (1 votes) · LW(p) · GW(p)

I'm not saying he's right, just that your proposed alternative isn't even wrong.

comment by Strange7 · 2012-05-22T06:50:22.998Z · score: 0 (0 votes) · LW(p) · GW(p)

I'm not saying he's right, I'm saying your proposed alternative isn't even wrong.

comment by thomblake · 2012-05-15T17:09:20.777Z · score: 2 (2 votes) · LW(p) · GW(p)

Science is built around the assumption that you’re too stupid and self-deceiving to just use Solomonoff induction.

He thinks the social institution of science is superfluous, were everyone as smart as he.

This is obviously false. Yudkowsky does not claim to be able to do Solomonoff induction in his head.

In general, when Yudkowsky addresses humanity's faults, he is including himself.

comment by [deleted] · 2012-05-15T17:22:13.579Z · score: -5 (7 votes) · LW(p) · GW(p)

Point taken.

But Yudkowsky says "built around the assumption that you're too stupid... to just use ..."

If Solomonoff induction can't easily be used in place of science, why does the first sentence imply the process is simple: you just use it?

You've clarified what Yudkowsky does not mean. But what does he mean? And why is it so hard to find out? This is the way mystical sects retain their aura while actually saying little.

comment by nshepperd · 2012-05-15T17:38:29.857Z · score: 5 (5 votes) · LW(p) · GW(p)

"You're too stupid and self-deceiving to just use Solomonoff induction" ~ "If you were less stupid and self deceiving you'd be able to just use Solomonoff induction" + "but since you are in fact stupid and self-deceiving, instead you have to use the less elegant approximation Science"

That was hard to find out?

comment by [deleted] · 2012-05-15T17:43:05.909Z · score: -7 (7 votes) · LW(p) · GW(p)

Actually, yes, because of the misleading signals in the inept writing. But thank you for clarifying.

Conclusion: The argument in written in a crazy fashion, but it really is merely stupid. There is no possible measure of simplicity that isn't language relative. How could there be?

comment by Randaly · 2012-05-15T19:49:22.368Z · score: 3 (3 votes) · LW(p) · GW(p)

You seem to be confusing "language relative" with "non-mathematical." Kolmogorov Complexity is "language-relative," if I'm understanding you right; specifically, it's relative (if I'm using the terminology right?) to a Turing Machine. This was not relevant to Eliezer's point, so it was not addressed.

(Incidentally, this is a perfect example of you "hold{ing} views contrary to scientific consensus in specialized fields where {you} lack expert knowledge based on pure ratiocination," since Kolmogorov Complexity is "one of the fundamental concepts of theoretical computer science", you seemingly lack expert knowledge since you don't recognize these terms, and your argument seems to be based on pure ratiocination.)

comment by CuSithBell · 2012-05-15T17:46:01.866Z · score: 1 (1 votes) · LW(p) · GW(p)

When I read that line for the first time, I understood it. Between our two cases, the writing was the same, but the reader was different. Thus, the writing cannot be the sole cause of our different outcomes.

comment by JoshuaZ · 2012-05-15T17:57:38.655Z · score: 4 (4 votes) · LW(p) · GW(p)

Well, if a substantial fraction of readers read something differently or can't parse it, it does potentially reflect a problem with the writing even if some of the readers, or even most readers, do read it correctly.

comment by CuSithBell · 2012-05-15T18:02:00.134Z · score: 3 (3 votes) · LW(p) · GW(p)

Absolutely. I intended to convey that if you don't understand something, that the writing is misleading and inept is not the only possible reason. srdiamond is speaking with such confidence that I felt safe tabling further subtleties for now.

comment by [deleted] · 2012-05-16T18:38:29.877Z · score: -13 (13 votes) · LW(p) · GW(p)

The philosophizing of inept, verbose writers like Yudkowsky can be safely dismissed based solely on their incompetence as writers. For a succinct defense of this contention, see my "Can bad writers be good thinkers? Part 1 of THE UNITY OF LANGUAGE AND THOUGHT" OR see the 3-part "Writing & Thought series" — all together, fewer than 3,000 words.

comment by gwern · 2012-05-16T19:01:44.202Z · score: 14 (14 votes) · LW(p) · GW(p)

I believe what you wrote because you used so much bolding.

comment by [deleted] · 2012-05-21T20:02:21.660Z · score: -6 (6 votes) · LW(p) · GW(p)

Way to deflect attention from substance to form. Exemplary rationality!

comment by thomblake · 2012-05-21T20:10:12.204Z · score: 3 (3 votes) · LW(p) · GW(p)

I can't tell which way your sarcasm was supposed to cut.

The obvious interpretation is that you think rationality is somehow hindered by paying attention to form rather than substance, and the "exemplary rationality" was intended to be mocking.

But your comment being referenced was an argument that form has something very relevant to say about substance, so it could also be that you were actually praising gwern for practicing what you preach.

comment by gwern · 2012-05-21T20:30:41.664Z · score: 1 (3 votes) · LW(p) · GW(p)

I choose to interpret it as praise, and receive a warm fuzzy feeling.

comment by [deleted] · 2012-05-16T19:20:53.809Z · score: 2 (2 votes) · LW(p) · GW(p)

I read your three-part series. Your posts did not substantiate the claim "good thinking requires good writing." Your second post slightly increased my belief in the converse claim, "good thinkers are better-than-average writers," but because the only evidence you provided was a handful of historical examples, it's not very strong evidence. And given how large the population of good thinkers, good writers, bad thinkers, and bad writers is relative to your sample, evidence for "good thinking implies good writing" is barely worth registering as evidence for "good writing implies good thinking."

comment by JoshuaZ · 2012-05-14T18:12:00.909Z · score: 8 (8 votes) · LW(p) · GW(p)

Romney is rightfully being held, feet to fire, for a group battering of another student while they attended high school--because such sadism is a trait of character and can't be explained otherwise.

I was going to upvote your comment until I got to this point. Aside from the general mindkilling, this looks like the fundamental attribution error, and moreover, we all know that people do in fact mature and change. Bringing up external politics is not helpul in a field where there's already concern that AI issues may be becoming a mindkilling subject themselves on LW. Bringing up such a questionable one is even less useful.

comment by metaphysicist · 2012-05-14T18:21:42.024Z · score: -1 (7 votes) · LW(p) · GW(p)

That's LW "rationality" training for you--"fundamental error of attribution" out of context--favored because it requires little knowledge and training in psychology. Such thinking would preclude any investigation of character. (And there are so many taboos! How do you all tolerate the lockstep communication required here?)

Paul Meehl, who famously studied clinical versus statistical prediction empirically, noted that even professionals, when confronted by instance of aberrant behavior, are apt to call it within normal range when it clearly isn't. Knowledge of the "fundamental error of attribution" alone is the little bit of knowledge that's worse than total ignorance.

Ask yourself honestly whether you would ever or have ever done anything comparable to what Yudkowsky did in the Roko incident or what Romney did in the hair cutting incident.

You can't dismiss politics just because it kills some people's minds, when so much of the available information and examples come from politics. (There are other reasons, but that's the main one here.) Someone who can't be rational about politics simply isn't a good rationalist. You can't be a rationalist about the unimportant things and rationalist about the important ones--yet call yourself a rationalist overall.

comment by NancyLebovitz · 2012-05-15T03:55:23.777Z · score: 9 (9 votes) · LW(p) · GW(p)

I'm sure I wouldn't have done what Romney did, and not so sure about whether I would have done what Yudkowsky did. Romney wanted to hurt people for the fun of it. Yudkowsky was trying to keep people from being hurt, regardless of whether his choice was a good one.

comment by metaphysicist · 2012-05-15T04:48:59.281Z · score: 2 (2 votes) · LW(p) · GW(p)

That's a reasonable answer.

comment by JoshuaZ · 2012-05-14T20:04:36.890Z · score: 6 (8 votes) · LW(p) · GW(p)

It seems almost unfair to criticize something as a problem of LW rationality when in your second paragraph you note that professionals do the same thing.

Ask yourself honestly whether you would ever or have ever done anything comparable to what Yudkowsky did in the Roko incident or what Romney did in the hair cutting incident.

I'm not sure. A while ago, I was involved in a situation where someone wanted to put personal information of an individual up on the internet knowing that that person had an internet stalker who had a history of being a real life stalker for others. The only reason I didn't react pretty close to how Eliezer reacted in the quoted incident is that I knew that the individual in question was not going to listen to me and would if anything have done the opposite of what I wanted. In that sort of context, Eliezer's behavior doesn't seem to be that extreme. Eliezer's remarks involve slightly more caps than I think I would use in such a circumstance, but the language isn't that different.

This does connect to another issue though- the scale in question of making heated comments on the internet as opposed to traumatic bullying, are different. The questions I ask myself for what it would take to do something similar to what Eliezer did are very different than the same questions for the Romney incident.

Your basic statement does it seem have some validity. One could argue that the Romney matter reflects the circumstances where he was at the time, and what was considered socially acceptable as forms of interaction or establishing dominance hierarchies. Through most of human history, that sort of behavior would probably be considered fairly tame. But this is a weak argument- even if it was due to the circumstances that Romney was in at the time, there's no question that those were his formative years, and thus could plausibly have had a permanent impact on his moral outlook.

You can't dismiss politics just because it kills some people's minds, when so much of the available information and examples come from politics.

The problem is that even as relevant examples come from politics, those are precisely the examples that people are least likely to agree actually demonstrate the intended point in question. For example, in this case, many people who aren't on the left will downplay the Romney bullying. Given that I'm someone who dislikes Romney (both in terms of personality and in terms of policy) and am not convinced that this is at all fair, using such a controversial example seems unwise. Even if one needs to use political examples, one can use examples from 10 or 15 or 30 years ago that are well known but have had their tribalness diminish in time. For example, in this context one could use a variety of examples connected to Richard Nixon.

Someone who can't be rational about politics simply isn't a good rationalist. You can't be a rationalist about the unimportant things and rationalist about the important ones--yet call yourself a rationalist overall.

Well, we can acknowledge that we're better at being rational in some areas than we are in others. Frankly, I wouldn't mind and for reasons essentially similar to your remark would endorse some amount of reduction of the no-politics rule here. Where that becomes a problem is when one tries to connect politics to other potentially controversial issues.

comment by [deleted] · 2012-05-14T18:40:34.376Z · score: 1 (1 votes) · LW(p) · GW(p)

what Romney did in the hair cutting incident

What's that about? (PM me if it's still taboo.)

comment by Normal_Anomaly · 2012-05-16T14:01:05.105Z · score: 2 (2 votes) · LW(p) · GW(p)

When Mitt Romney was in high school, he and some friends bullied a kid who looked (and later turned out to be) homosexual. At one point, Romney and some others grabbed the guy, held him down, and cut off a bunch of his hair with scissors.

comment by Barry_Cotter · 2012-05-13T09:44:20.180Z · score: -1 (7 votes) · LW(p) · GW(p)

Why do you continue to participate? Almost all of the cool stuff that high status people agree is plausible is available elsewhere.

comment by [deleted] · 2012-05-11T22:45:49.965Z · score: -4 (22 votes) · LW(p) · GW(p)

The point, such as it is, would better have been left implied. Now, it's subject to explicit scrutiny, and it must be found wanting. Consider what would have happened had Yudkowsky not shown exceptional receptivity to this post: he would have blatantly proven his critics right. The knowledge and reputation of the poster is unimpeachable.

The more significant fact is that these criticisms were largely unknown to the community. As Will Newsome implied, this is because the critical posts--lacking the high-status credential of this poster--remained in discussion and were almost ignored.

The majority's intolerance for dissent is manifested mostly in its refusal to acknowledge it. Dissent is cabined to Discussion. It only gets noticed when the dissenter becomes frustrated and violates group norms. Then it gets voted down, but it still gets noticed and commented on. This is a malfunctioning reinforcement system, but maybe its the best possible. Still, it's irrational to deny all in-group bias in LukeProg's cheerleading fashion--in an instance where the absence of evidence (here, of bias) truly does not offer anything substantial in the way of evidence of lack of bias, to elicit LukeProg's smug laughter.

After all, even the lead poster held off until now in voicing his opinion.

comment by lukeprog · 2012-05-11T23:00:36.568Z · score: 11 (13 votes) · LW(p) · GW(p)

The more significant fact is that these criticisms were largely unknown to the community.

LWer tenlier disagrees, saying:

[Holden's] critique mostly consists of points that are pretty persistently bubbling beneath the surface around here, and get brought up quite a bit. Don't most people regard this as a great summary of their current views, rather than persuasive in any way? In fact, the only effect I suspect this had on most people's thinking was to increase their willingness to listen to Karnofsky in the future if he should change his mind.

Also, you said:

Dissent is cabined to Discussion.

Luckily, evidence on the matter is easy to find. As counter-evidence I present: Self-improvement or shiny distraction, SIAI an examination, Why we can't take expected value estimates literally, Extreme rationality: it's not that great, Less Wrong Rationality and Mainstream Philosophy, and the very post you are commenting on. Many of these are among the most upvoted posts ever.

Moreover, the editors rarely move posts from Main to Discussion. The posters themselves decide whether to post in Main or Discussion.

comment by Rain · 2012-05-11T23:31:20.250Z · score: 0 (0 votes) · LW(p) · GW(p)

Also Should I believe what the SIAI claims? and the many XiXiDu posts linked therein, like What I would like the SIAI to publish.

comment by [deleted] · 2012-05-11T23:55:04.024Z · score: -6 (14 votes) · LW(p) · GW(p)

I had a post moved from main to Discussion just today: before it had accumulated any negative votes, so I think you're probably misinformed about editorial practices.But I don't want to use my posts as evidence; the charge of bias would be hard to surmount. What's plainly evident is that posters are reluctant to post to the Main area except by promotion.

You're evidence is unpersuasive because you don't weigh it against the evidence to the contrary. One good example to the contrary more than counter-balances it, since the point isn't that no dissent is tolerated, not even that some dissent isn't welcomed, but only that there are some irrational boundaries.

One is the quasi-ban on politics. Here is a comment that garnered almost 800 responses and was voted up 37. Why wasn't it promoted? I bitterly disagree with the poster; so I'm not biased by my views. But the point is that it is a decidedly different view, one generating great interest, but the subject would not be to the liking of the editors.

Of course, it lacked the elaborateness--dare I say, the prolixity--of a typical top-level post. But this "scholarly" requirement is part of the process of soft censorship. The post--despite my severe disagreement with it--is a more significant intellectual contribution than many of the top-level posts, such as some of the second-hand scholarship.

[And I have to add: observe that the present discussion is already being downvoted at my first comment. I predict the same for this post in record time What does that mean?]

comment by Normal_Anomaly · 2012-05-13T03:13:11.205Z · score: 4 (4 votes) · LW(p) · GW(p)

Here is a comment that garnered almost 800 responses and was voted up 37. Why wasn't it promoted?

Can comments be promoted? Perhaps the commenter should have been encouraged to turn his comment into a top-level post, but a moderator can't just change a comment into a promoted post with the same username. Also it would have split the discussion, so people might have been reluctant to encourage that.

As for people tending to post more in Discussion than Main, I read somewhere that Discussion has more readers. I for one read Discussion almost exclusively.

comment by [deleted] · 2012-05-12T00:11:40.567Z · score: -4 (12 votes) · LW(p) · GW(p)

It would advance this discussion if someone would explain the down votes. I await LukeProg's explanation of the present example of soft censorship.

comment by Rain · 2012-05-12T00:14:53.487Z · score: 9 (11 votes) · LW(p) · GW(p)

I downvoted you because you're wrong. For one, comments can't be promoted to main, only posts, and for two, plenty of opposition has garnerned a great deal of upvotes, as shown by the numerous links lukeprog provided.

For example, where do you get 'almost 800 responses' from? That comment (not post) only has 32 comments below it.

comment by [deleted] · 2012-05-12T00:40:30.065Z · score: -4 (8 votes) · LW(p) · GW(p)

Yes, I was wrong. But my point was correct. The 781 comments applied to the Main Post So:

  1. The topic was popular, like I said.

  2. The post could have been promoted!

But ask yourself, would you have been so harsh on a factual error had you agreed with the message? This is the way bias works, after all, by double standard more than outright discrimination. You could say I should have been more careful. But then, when you've learned not to expect a hearing, you're not so willing to jump the hoops. But it's your loss, if you're a rationalist and if you're losing input because dissenters find it's not worth their time.

As to LukeProg providing example demonstrating welcoming dissent: you couldn't have considered my counter-balancing evidence when you downvoted before taking the time even to explore the post to which the cited comment belongs.

To LukeProg: have I made my point about the limits of dissent at LW?

comment by Rain · 2012-05-12T00:19:38.502Z · score: 7 (11 votes) · LW(p) · GW(p)

Posts which contain factual inaccuracies along with meta-discussion of karma effects are often downvoted.

comment by [deleted] · 2012-05-12T00:47:26.421Z · score: -4 (12 votes) · LW(p) · GW(p)

I've addressed factual inaccuracies in another comment. But as for discussing karma effects--that wasn't extraneous whining but was at the heart of the discussion. If you downvote discussion of karma--like you did--simply for mentioning it, even where relevant, then you effectively soft-censor any discussion of karma. How is that rational?

LukeProg: What do you say about the grounds on which downvotes are issued for dissenting matter. Isn't it clear that this is a bias LW doesn't want to talk about; perhaps altogether doesn't want to discuss its own biases?

comment by Rain · 2012-05-12T01:20:24.798Z · score: 3 (7 votes) · LW(p) · GW(p)

If you downvote discussion of karma--like you did--simply for mentioning it, even where relevant, then you effectively soft-censor any discussion of karma. How is that rational?

I don't do that; I only downvote when it's combined with incorrect facts. Which I'm tempted to do for this statement: "like you did--simply for mentioning it", since you're inferring my motivations, and once again incorrect.

comment by [deleted] · 2012-05-12T01:29:57.989Z · score: -4 (14 votes) · LW(p) · GW(p)

Look, Rain, this is an Internet ongoing discussion. Nobody says everything precisely right. The point is that you would hardly be so severe on someone unless you disagreed strongly. You couldn't be, because nobody would satisfy your accuracy demands. The kind of nitpicking you engage in your post would ordinarily lead you to be downvoted--and you should be, although I won't commit the rudeness of so doing in a discussion.

The point wasn't that you downvote when the only thing wrong with the comment is discussion of karma. It was that you treat discussion of karma as an unconditional wrong. So you exploited weaknesses in my phrasing to ignore what I think was obviously the point--that marking down for the bare mention of karma (even if it doesn't produce a downvote in each case) is an irrational policy, when karma is at the heart of the discussion. There's no rational basis for throwing it in as an extra negative when the facts aren't right.

You're looking for trivial points to pick to downvote and to ignore the main point, which was your counting mention of karma a negative, without regard to the subject, is an irrational policy. If we were on reversed sides, your nitpicking and evasion would itself be marked down. As matters stand, you don't even realize you're acting in a biased fashion, and readers either don't know or don't care.

Is that rational? Shouldn't a rationalist community be more concerned with criticizing irrationalities in its own process?

comment by shminux · 2012-05-12T02:12:45.206Z · score: 11 (13 votes) · LW(p) · GW(p)

Having been a subject of both a relatively large upvote and a relatively large downvote in the last couple of weeks, I still think that the worst thing one can do is to complain about censorship or karma. The posts and comments on any forum aren't judged on their "objective merits" (because there is no such thing), but on its suitability for the forum in question. If you have been downvoted, your post deserves it by definition. You can politely inquire about the reasons, but people are not required to explain themselves. As for rationality, I question whether it is rational to post on a forum if you are not having fun there. Take it easy.

comment by [deleted] · 2012-05-13T14:33:28.223Z · score: -1 (5 votes) · LW(p) · GW(p)

The posts and comments on any forum aren't judged on their "objective merits" (because there is no such thing), but on its suitability for the forum in question. If you have been downvoted, your post deserves it by definition.

First, you're correct that it's irrational to post to a forum you don't enjoy. I'll work on decreasing my akrasia.

But it's hard not to comment on a non sequitur like the above. (Although probably futile because one who's really not into a persuasion effort won't do it well.) That posts are properly evaluated by suitability to the forum does not imply that a downvoted post deserves the downvote by definition! That's a maladaptive view of the sort I'm amazed is so seldom criticized on this forum. Your view precludes (by definition yet) criticism of the evaluators' biases, which do not advance the forum's purpose. You would eschew not only absolute merits but also any objective consideration of the forum's function.

A forum devoted to rationality, to be effective and honest, must assess and address the irrationalities in its own functioning. (This isn't always "fun.") To define a post that should be upvoted as one that is upvoted constitutes an enormous obstacle to rational function.

comment by Rain · 2012-05-12T01:45:59.251Z · score: 5 (9 votes) · LW(p) · GW(p)

The point is that you would hardly be so severe on someone unless you disagreed strongly.

I disagree; a downvote is not 'severe'.

The kind of nitpicking you engage in your post would ordinarily lead you to be downvoted

I disagree; meta-discussions often result in many upvotes.

It was that you treat discussion of karma as an unconditional wrong.

I do not, and have stated as much.

There's no rational basis for throwing it in as an extra negative when the facts aren't right.

If there is no point in downvoting incorrect facts, then I wonder what the downvote button is for.

You're looking for trivial points to pick to downvote and to ignore the main point,

I disagree; your main point is that you are being unfairly downvoted, along with other posts critical of SI being downvoted unfairly, which I state again is untrue, afactual, incorrect, a false statement, a lie, a slander, etc.

Is that rational? Shouldn't a rationalist community be more concerned with criticizing irrationalities in its own process?

Questioning the rationality of meta-meta-voting patterns achieves yet another downvote from me. Sorry.

comment by Endovior · 2012-05-12T05:20:49.818Z · score: -4 (10 votes) · LW(p) · GW(p)

I don't follow your reasoning, here. Having read this particular thread, it does seem as though you are, in fact, going out of your way to criticize and downvote srdiamond. Yes, he has, in fact, made a few mistakes. Given, however, that the point of this post in general is about dissenting from the mainstream opinions of the LW crowd, and given the usual complaints about lack of dissent, I find your criticism of srdiamond strange, to say the least. I have, accordingly, upvoted a number of his comments.

comment by Endovior · 2012-05-12T07:03:40.698Z · score: 0 (10 votes) · LW(p) · GW(p)

As expected, my previous comment was downvoted almost immediately.

This would, for reference, be an example of the reason why some people believe LW is a cult that suppresses dissent. After all, it's significantly easier to say that you disagree with something than it is to explain in detail why you disagree; just as it's far easier to state agreement than to provide an insightful statement in agreement. Nonetheless, community norms dictate that unsubstantiated disagreements get modded down, while unsubstantiated agreements get modded up. Naturally, there's more of the easy disagreement then the hard disagreement... that's natural, since this is the Internet, and anyone can just post things here.

In any event, though, the end result is the same; people claim to want more dissent, but what they really mean is that they want to see more exceptionally clever and well-reasoned dissent. Any dissent that doesn't seem at least half as clever as the argument it criticizes seems comparatively superfluous and trivial, and is marginalized at best. And, of course, any dissent that is demonstrably flawed in any way is aggressively attacked. That really is what people mean by suppression of dissent. It doesn't really mean 'downvoting arguments which are clever, but with which you personally disagree'... community norms here are a little better then that, and genuinely good arguments tend to get their due. In this case, it means, 'downvoting arguments which aren't very good, and with which you personally disagree, when you would at the same time upvote arguments that also aren't very good, but with which you agree'. Given the nature of the community norms, someone who expresses dissent regularly, but without taking the effort to make each point in an insightful and terribly clever way, would tend to be downvoted repeatedly, and thus discouraged from making more dissent in the future... or, indeed, from posting here at all.

I don't know if there's a good solution to the problem. I would be inclined to suggest that, like with Reddit, people not downvote without leaving an explanation as to why. For instance, in addition to upvoting some of srdiamond's earlier comments, I have also downvoted some of Rain's, because a number of Rain's comments in this thread fit the pattern of 'poor arguments that support the community norms', in the same sense that srdiamond's fit the pattern of 'poor arguments that violate the community norms'; my entire point here is that, in order to cultivate more intelligent dissent, there should be more of the latter and less of the former.

comment by Paul Crowley (ciphergoth) · 2012-05-12T09:13:25.094Z · score: 3 (7 votes) · LW(p) · GW(p)

I downvote any post that says "I expect I'll get downvoted for this, but..." or "the fact that I was downvoted proves I'm right!"

comment by CuSithBell · 2012-05-12T15:49:13.503Z · score: 6 (6 votes) · LW(p) · GW(p)

I'm fond of downvoting "I dare you to downvote this!"

comment by Endovior · 2012-05-12T14:31:11.670Z · score: 2 (8 votes) · LW(p) · GW(p)

So, in other words, you automatically downvote anyone who explicitly mentions that they realize they are violating community norms by posting whatever it is they are posting, but feels that the content of their post is worth the probable downvotes? That IS fairly explicitly suppressing dissent, and I have downvoted you for doing so.

comment by JoshuaZ · 2012-05-12T14:47:27.724Z · score: 8 (8 votes) · LW(p) · GW(p)

I don't think it is suppression of dissent per se. It is more annoying behavior- it implies caring a lot about the karma system, and it is often not even the case when people say that they will actually get downvoted. If it is worth the probable downvote, then they can, you know, just take the downvote. If they want to point out that a view is unpopular they can just say that explicitly. It is also annoying to people like me, who are vocal about a number of issues that could be controversial here (e.g. criticizing Bayesianism, cryonics,, and whether intelligence explosions would be likely) and get voted up. More often than not, when someone claims they are getting downvoted for having unpopular opinions, they are getting downvoted in practice for having bad arguments or for being uncivil.

There are of course exceptions to this rule, and it is disturbing to note that the exceptions seem to be coming more common (see for example, this exchange where two comments are made with about the same quality of argument and about the same degree of uncivility- ("I'm starting to hate that you've become a fixture here." v. "idiot" - but one of the comments is at +10 and the other is at -7.) Even presuming that there's a real disagreement in quality or correctness of the arguments made, this suggests that uncivil remarks are tolerated more when people agree with the rest of the claim being made. That's problematic. And this exchange was part of what prompted me to earlier suggest that we should be concerned if AGI risk might be becoming a mindkiller here. But even given that, issues like this seem not at all common.

Overall, if one needs to make a claim about one is going to be downvoted, one might even be correct, but it will often not be for the reasons one thinks it is.

comment by Endovior · 2012-05-12T15:23:27.211Z · score: 3 (5 votes) · LW(p) · GW(p)

I don't think it's so much 'caring a lot about the karma system' per se, so much as the more general case of 'caring about the approval and/or disapproval of one's peers'. The former is fairly abstract, but the latter is a fairly deep ancestral motivation.

Like I said before, it's clearly not much in the way of suppression. That said, given that, barring rare incidents of actual moderation, it is the only 'suppression' that occurs here, and since there is a view among various circles that there there is, in fact, suppression of dissent, and since people on the site frequently wonder why there are not more dissenting viewpoints here, and look for ways to find more... it is important to look at the issue in great depth, since it's clearly an issue which is more significant than it seems on the surface.

comment by [deleted] · 2012-05-13T19:31:19.106Z · score: -1 (7 votes) · LW(p) · GW(p)

[P]eople on the site frequently wonder why there are not more dissenting viewpoints here, and look for ways to find more... it is important to look at the issue in great depth, since it's clearly an issue which is more significant than it seems on the surface.

Exactly right. But a group that claims to be dedicated to rationality loses all credibility when participants not only abstain from considering this question but adamantly resist it. The only upvote you received for your post—which makes this vital point—is mine.

This thread examines HoldenKarnofsky's charge that SIAI isn't exemplarily rational. As part of that examination, the broader LW environment on which it relies is germane. That much has been granted by most posters. But when the conversation reaches the touchstone of how the community expresses its approval and disapproval, the comments are declared illegitimate and downvoted (or if the comments are polite and hyper-correct, at least not upvoted).

The group harbors taboos. The following subjects are subject to them: the very possibility of nonevolved AI; karma and the group's own process generally (an indespensable discussion ); and politics. (I've already posted a cite showing how the proscription on politics works, using an example the editors' unwillingness to promote the post despite receiving almost 800 comments).

These defects in the rational process of LW help sustain Kardofsky's argument that SIAI is not to be recommended based on the exemplary rationality of its staff and leadership. They are also the leadership of LW, and they have failed by refusing to lead the forum toward understanding the biases in its own process. They have fostered bias by creating the taboo on politics, as though you can rationally understand the world while dogmatically refusing even to consider a big part of it—because it "kills" your mind.

P.S. Thank you for the upvotes where you perceived bias.

comment by XiXiDu · 2012-05-12T17:16:21.529Z · score: 2 (10 votes) · LW(p) · GW(p)

...AGI risk might be becoming a mindkiller here...

Nah. If there is a mindkiller then it is the reputation system. Some of the hostility is the result of the overblown ego and attitude of some of its proponents and their general style of discussion. They created an insurmountable fortress that shields them from any criticism:

Troll: If you are so smart and rational, why don't you fund yourself? Why isn't your organisation sustainable?

SI/LW: Rationality is only aimed at expected winning.

Troll: But you don't seem to be winning yet. Have you considered the possibility that your methods are suboptimal? Have you set yourself any goals, that you expect to be better at than less rational folks, to test your rationality?

SI/LW: Rationality is a caeteris paribus predictor of success.

Troll: Okay, but given that you spend a lot of time on refining your rationality, you must believe that it is worth it somehow? What makes you think so then?

SI/LW: We are trying to create a friendly artificial intelligence implement it and run the AI, at which point, if all goes well, we Win. We believe that rationality is very important to achieve that goal.

Troll: I see. But there surely must be some sub-goals that you anticipate to be able to solve and thereby test if your rationality skills are worth the effort?

SI/LW: Many of the problems related to navigating the Singularity have not yet been stated with mathematical precision, and the need for a precise statement of the problem is part of the problem.

Troll: Has there been any success in formalizing one of the problems that you need to solve?

SI/LW: There are some unpublished results that we have had no time to put into a coherent form yet.

Troll: It seems that there is no way for me to judge if it is worth it to read up on your writings on rationality.

SI/LW: If you want to more reliably achieve life success, I recommend inheriting a billion dollars or, failing that, being born+raised to have an excellent work ethic and low akrasia.

Troll: Awesome, I'll do that next time. But for now, why would I bet on you or even trust that you know what you are talking about?

SI/LW: We spent a lot of time on debiasing techniques and thought long and hard about the relevant issues.

Troll: That seems to be insufficient evidence given the nature of your claims and that you are asking for money.

SI/LW: We make predictions. We make statements of confidence of events that merely sound startling. You are asking for evidence we couldn't possibly be expected to be able to provide, even given that we are right.

Troll: But what do you anticipate to see if your ideas are right, is there any possibility to update on evidence?

SI/LW: No, once the evidence is available it will be too late.

Troll: But then why would I trust you instead of those experts who tell me that you are wrong?

SI/LW: You will soon learn that your smart friends and experts are not remotely close to the rationality standards of SI/LW, and you will no longer think it anywhere near as plausible that their differing opinion is because they know some incredible secret knowledge you don't.

Troll: But you have never achieved anything when it comes to AI, why would I trust your reasoning on the topic?

SI/LW: That is magical thinking about prestige. Prestige is not a good indicator of quality.

Troll: You won't convince me without providing further evidence.

SI/LW: That is a fully general counterargument you can use to discount any conclusion.

comment by Jonathan_Graehl · 2012-05-13T00:57:23.806Z · score: 1 (1 votes) · LW(p) · GW(p)

Troll: You won't convince me without providing further evidence.

SI/LW: That is a fully general counterargument you can use to discount any conclusion.

The last exchange was hilarious. This is parody, right?

comment by CuSithBell · 2012-05-12T15:47:44.163Z · score: 2 (2 votes) · LW(p) · GW(p)

More often than not, when someone claims they are getting downvoted for having unpopular opinions, they are getting downvoted in practice for having bad arguments or for being uncivil.

Bears repeating.

comment by Rain · 2012-05-12T14:42:40.050Z · score: -5 (7 votes) · LW(p) · GW(p)

Downvoted for downvoting downvoting of downvoting of downvoting.

If you do the same to this comment, we can enter a stable loop!

comment by [deleted] · 2012-05-12T07:39:05.943Z · score: 3 (3 votes) · LW(p) · GW(p)

First, none of this dissent has been suppressed in any real sense. It's still available to be read and discussed by those who desire reading and discussing such things. The current moderation policy has currently only kicked in when things have gotten largely out of hand -- which is not the case here, yet.

Second, net karma isn't a fine enough tool to express amount of detail you want it to express. The net comment on your previous comment is currently -2; congrats, you've managed to irritate less than a tenth of one percent of LW (presuming the real karma is something like -2/+0 or -3/+1)!

Third, the solution you propose hasn't been implemented anywhere that I know of. Reddit's suggested community norm (which does not apply to every subreddit) suggests considering posting constructive criticism only when one thinks it will actually help the poster improve. That's not really the case much of the time, at least on the subreddits I frequent, and it's certainly not the case often here.

Fourth, the solution you propose would, if implemented, decrease the signal-to-noise ratio of LW further.

Fifth, reddit's suggested community norm also says "[Don't c]omplain about downvotes on your posts". Therefore, I wonder how much you really think reddit is doing the community voting norm thing correctly.

comment by Endovior · 2012-05-12T14:22:58.711Z · score: 2 (4 votes) · LW(p) · GW(p)

First; downvoted comments are available to be read, yes; but the default settings hide comments with 2 or more net downvotes. This is enough to be reasonably considered 'suppression'. It's not all that much suppression, true, but it is suppression... and it is enough to discourage dissent. Actual moderation of comments is a separate issue entirely, and not one which I will address here.

Second; when I posted my reply, and as of this moment, my original comment was at -3. I agree; net karma isn't actually a huge deal, except that it is, as has been observed, the most prevalent means by which dissent is suppressed. In my case, at least, 'this will probably get downvoted' feels like a reason to not post something. Not much of a reason, true, but enough of one that I can identify the feeling of reluctance.

Third; on the subreddits I follow (admittedly a shallow sampling), I have frequently seen comments explaining downvotes, sometimes in response to a request specifically for such feedback, but just as often not. I suspect that this has a lot to do with the "Down-voting? Please leave an explanation in the comments." message that appears when mousing over the downvote icon. I am aware that this is not universal across Reddit, but on the subreddits I follow, it seems to work reasonably well.

Fourth; I agree that this is a possible result. Like I said before, I'm not sure if there is a good solution to this problem, but I do feel that it'd result in a better state then that which currently exists, if people would more explicitly explain why they downvote when they choose to do so. That said, given that downvoted comments are hidden from default view anyway, and that those who choose to do so can easily ignore such comments, I don't think it'd have all that much effect on the signal/noise ratio.

Fifth; on the subreddits I follow, it seems as though there is less in the way of complaints about downvotes, and more honest inquiries as to why a comment has been downvoted; such questions seem to usually receive honest responses. This may be anomalous within Reddit as a whole; as I said before, my own experience with Reddit is a shallow sampling.

comment by [deleted] · 2012-05-13T20:57:36.632Z · score: -1 (5 votes) · LW(p) · GW(p)

I don't know if there's a good solution to the problem. I would be inclined to suggest that, like with Reddit, people not downvote without leaving an explanation as to why. For instance, in addition to upvoting some of srdiamond's earlier comments, I have also downvoted some of Rain's, because a number of Rain's comments in this thread fit the pattern of 'poor arguments that support the community norms', in the same sense that srdiamond's fit the pattern of 'poor arguments that violate the community norms'; my entire point here is that, in order to cultivate more intelligent dissent, there should be more of the latter and less of the former.

Perhaps the solution is not to worry so much about my bad contrarian arguments being downvoted as to assure that bad "establishment" arguments are downvoted—as in Rain's case, they aren't. Regurgitation of arguments others have repeatedly stated should also be downvoted, no matter how good the arguments.

The reason to think an emphasis on more criticism of Rain rather than less criticism of me is that after I err, it's a difficult argument to establish that my error wasn't serious enough to avoid downvote. But when Rain negligently or intentionally misses the entire point, there's less question that he isn't benefiting the discussion. It's easier to convict of fallacy than to defend based on the fallacy being relatively trivial. There's a problem in that the two determinations are somewhat inter-related, but it doesn't eliminate the contrast.

Increasing the number of downvotes would deflate the significance of any single downvote and would probably foster more dissent. This balance may be subject to easy institutional control. Posters are allotted downvotes based on their karma, while the karma requirements for upvotes are easily satisfied, if they exist. This amounts to encouraging upvotes relative to downvotes, with the result that many bad posts are voted up and some decent posts suffer the disproportionate wrath of extreme partisans. (Note that Rain, a donor, is a partisan of SIAI.)

The editors should experiment with increasing the downvote allowance. I favor equal availability of downvotes and upvotes as optimal (but this should be thought through more carefully).

comment by Nornagest · 2012-05-13T15:14:08.454Z · score: 3 (3 votes) · LW(p) · GW(p)

Consider what would have happened had Yudkowsky not shown exceptional receptivity to this post: he would have blatantly proven his critics right.

After turning this statement around in my head for a while I'm less certain than I was that I understand its thrust. But assuming you mean those critics pertinent to lukeprog's post, i.e. those claiming LW embodies a cult of personality centered around Eliezer -- well, no. Eliezer's reaction is in fact almost completely orthogonal to that question.

If you receive informed criticism regarding a project you're heavily involved in, and you react angrily to it, that shows nothing more or less than that you handle criticism poorly. If the community around you locks ranks against your critics, either following your example or (especially) preemptively, then you have evidence of a cult of personality.

That's not what happened here, though. Eliezer was fairly gracious, as was the rest of the community. Now, that is not by itself behavior typically associated with personality cults, but before we start patting ourselves on the back it's worth remembering that certain details of timing and form could still point back in the other direction. I'm pretty sick of the cult question myself, but if you're bound and determined to apply this exchange to it, that's the place you should be looking.

comment by fubarobfusco · 2012-05-18T02:49:22.003Z · score: 0 (0 votes) · LW(p) · GW(p)

Conservation of expected evidence may be relevant here.

comment by [deleted] · 2012-05-12T02:28:36.147Z · score: -18 (20 votes) · LW(p) · GW(p)

I shall now laugh harder than ever when people try to say with a straight face that LessWrong is an Eliezer-cult that suppresses dissent.

After I recently read that the lead poster was a major financial contributor to SIAI, I'd have to call LukeProg's argument disingenuous if not mendacious.

comment by CarlShulman · 2012-05-12T03:16:03.469Z · score: 12 (12 votes) · LW(p) · GW(p)

Rain (who noted that he is a donor to SIAI in a comment) and HoldenKarnofsky (who wrote the post) are two different people, as indicated by their different usernames.

comment by [deleted] · 2012-05-12T03:37:50.706Z · score: 1 (9 votes) · LW(p) · GW(p)

Well, different usernames isn't usually sufficient evidence that there are two different people, but in this case there's little doubt about their separability.

comment by lukeprog · 2012-05-12T02:37:10.117Z · score: 8 (8 votes) · LW(p) · GW(p)

I don't understand. Holden is not a major financial contributor to SIAI. And even if he was: which argument are you talking about, and why is it disingenuous?

comment by [deleted] · 2012-05-13T14:49:35.341Z · score: -5 (13 votes) · LW(p) · GW(p)

If Holden were a major contributor, your argument that the LW editors demonstrated their tolerance for dissent by encouraging the criticisms he made would be bogus. Suppressing the comments of a major donor would be suicidal, and claiming not doing so demonstrates any motive but avoiding suicide would be disingenuous at the least.

If he's not a donor, my apologies. In any event, you obviously don't know that he's a donor if he is, so my conclusion is wrong. I thought Yudkowsky said he was.

comment by MarkusRamikin · 2012-05-13T15:17:54.252Z · score: 0 (0 votes) · LW(p) · GW(p)

I'm confused. Holden doesn't believe SI is a good organisation to recommend giving money to, he's listed all those objections to SI in his post, and you somehow assumed he's been donating money to it?

That don't make sense.

comment by paulfchristiano · 2012-05-10T17:16:26.303Z · score: 35 (38 votes) · LW(p) · GW(p)

Thanks for taking the time to express your views quite clearly--I think this post is good for the world (even with a high value on your time and SI's fundraising ability), and that norms encouraging this kind of discussion are a big public good.

I think the explicit objections 1-3 are likely to be addressed satisfactorily (in your judgment) by less than 50,000 words, and that this would provide a good opportunity for SI to present sharper versions of the core arguments---part of the problem with existing materials is certainly that it is difficult and unrewarding to respond to a nebulous and shifting cloud of objections. A lot of what you currently view as disagreements with SI's views may get shifted to doubts about SI being the right organization to back, which probably won't get resolved by 50,000 words.

comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2012-05-15T17:49:19.828Z · score: 33 (34 votes) · LW(p) · GW(p)

Reading Holden's transcript with Jaan Tallinn (trying to go over the whole thing before writing a response, due to having done Julia's Combat Reflexes unit at Minicamp and realizing that the counter-mantra 'If you respond too fast you may lose useful information' was highly applicable to Holden's opinions about charities), I came across the following paragraph:

My understanding is that once we figured out how to get a computer to do arithmetic, computers vastly surpassed humans at arithmetic, practically overnight ... doing so didn't involve any rewriting of their own source code, just implementing human-understood calculation procedures faster and more reliably than humans can. Similarly, if we reached a good enough understanding of how to convert data into predictions, we could program this understanding into a computer and it would overnight be far better at predictions than humans - while still not at any point needing to be authorized to rewrite its own source code, make decisions about obtaining "computronium" or do anything else other than plug data into its existing hardware and algorithms and calculate and report the likely consequences of different courses of action

I've been previously asked to evaluate this possibility a few times, but I think the last time I did was several years ago, and when I re-evaluated it today I noticed that my evaluation had substantially changed in the interim due to further belief shifts in the direction of "Intelligence is not as computationally expensive as it looks" - constructing a non-self-modifying predictive super-human intelligence might be possible on the grounds that human brains are just that weak. It would still require a great feat of cleanly designed, strong-understanding-math-based AI - Holden seems to think this sort of development would happen naturally with the sort of AGI researchers we have nowadays, and I wish he'd spent a few years arguing with some of them to get a better picture of how unlikely this is. Even if you write and run algorithms and they're not self-modifying, you're still applying optimization criteria to things like "have the humans understand you", and doing inductive learning has a certain inherent degree of program-creation to it. You would need to have done a lot of "the sort of thinking you do for Friendly AI" to set out to create such an Oracle and not have it kill your planet.

Nonetheless, I think after further consideration I would end up substantially increasing my expectation that if you have some moderately competent Friendly AI researchers, they would apply their skills to create a (non-self-modifying) (but still cleanly designed) Oracle AI first - that this would be permitted by the true values of "required computing power" and "inherent difficulty of solving problem directly", and desirable for reasons I haven't yet thought through in much detail - and so by Conservation of Expected Evidence I am executing that update now.

Flagging and posting now so that the issue doesn't drop off my radar.

comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2012-05-15T17:58:32.089Z · score: 15 (16 votes) · LW(p) · GW(p)

Jaan's reply to Holden is also correct:

... the oracle is, in principle, powerful enough to come up with self-improvements, but refrains from doing so because there are some protective mechanisms in place that control its resource usage and/or self-reflection abilities. i think devising such mechanisms is indeed one of the possible avenues for safety research that we (eg, organisations such as SIAI) can undertake. however, it is important to note the inherent instability of such system -- once someone (either knowingly or as a result of some bug) connects a trivial "master" program with a measurable goal to the oracle, we have a disaster in our hands. as an example, imagine a master program that repeatedly queries the oracle for best packets to send to the internet in order to minimize the oxygen content of our planet's atmosphere.

Obviously you wouldn't release the code of such an Oracle - given code and understanding of the code it would probably be easy, possibly trivial, to construct some form of FOOM-going AI out of the Oracle!

comment by kalla724 · 2012-05-17T01:11:41.134Z · score: 7 (7 votes) · LW(p) · GW(p)

Hm. I must be missing something. No, I haven't read all the sequences in detail, so if these are silly, basic, questions - please just point me to the specific articles that answer them.

You have an Oracle AI that is, say, a trillionfold better at taking existing data and producing inferences.

1) This Oracle AI produces inferences. It still needs to test those inferences (i.e. perform experiments) and get data that allow the next inferential cycle to commence. Without experimental feedback, the inferential chain will quickly either expand into an infinity of possibilities (i.e. beyond anything that any physically possible intelligence can consider), or it will deviate from reality. The general intelligence is only as good as the data its inferences are based upon.

Experiments take time, data analysis takes time. No matter how efficient the inferential step may become, this puts an absolute limit to the speed of growth in capability to actually change things.

2) The Oracle AI that "goes FOOM" confined to a server cloud would somehow have to create servitors capable of acting out its desires in the material world. Otherwise, you have a very angry and very impotent AI. If you increase a person's intelligence trillionfold, and then enclose them into a sealed concrete cell, they will never get out; their intelligence can calculate all possible escape solutions, but none will actually work.

Do you have a plausible scenario how a "FOOM"-ing AI could - no matter how intelligent - minimize oxygen content of our planet's atmosphere, or any such scenario? After all, it's not like we have any fully-automated nanobot production factories that could be hijacked.

comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2012-05-17T20:35:04.990Z · score: 13 (13 votes) · LW(p) · GW(p)

http://lesswrong.com/lw/qk/that_alien_message/

comment by kalla724 · 2012-05-17T21:38:27.466Z · score: 1 (3 votes) · LW(p) · GW(p)

My apologies, but this is something completely different.

The scenario takes human beings - which have a desire to escape the box, possess theory of mind that allows them to conceive of notions such as "what are aliens thinking" or "deception", etc. Then it puts them in the role of the AI.

What I'm looking for is a plausible mechanism by which an AI might spontaneously develop such abilities. How (and why) would an AI develop a desire to escape from the box? How (and why) would an AI develop a theory of mind? Absent a theory of mind, how would it ever be able to manipulate humans?

comment by [deleted] · 2012-05-18T13:29:05.610Z · score: 5 (5 votes) · LW(p) · GW(p)

Absent a theory of mind, how would it ever be able to manipulate humans?

That depends. If you want it to manipulate a particular human, I don't know.

However, if you just wanted it to manipulate any human at all, you could generate a "Spam AI" which automated the process of sending out Spam emails and promises of Large Money to generate income from Humans via an advance fee fraud scams.

You could then come back, after leaving it on for months, and then find out that people had transferred it some amount of money X.

You could have an AI automate begging emails. "Hello, I am Beg AI. If you could please send me money to XXXX-XXXX-XXXX I would greatly appreciate it, If I don't keep my servers on, I'll die!"

You could have an AI automatically write boring books full of somewhat nonsensical prose, title them "Rantings of an a Automated Madman about X, part Y". And automatically post E-books of them on Amazon for 99 cents.

However, this rests on a distinction between "Manipulating humans" and "Manipulating particular humans." and it also assumes that convincing someone to give you money is sufficient proof of manipulation.

comment by TheOtherDave · 2012-05-18T14:40:14.520Z · score: 3 (3 votes) · LW(p) · GW(p)

Can you clarify what you understand a theory of mind to be?

comment by [deleted] · 2012-05-19T11:11:43.042Z · score: 0 (0 votes) · LW(p) · GW(p)

Looking over parallel discussions, I think Thomblake has said everything I was going to say better than I would have originally phrased it with his two strategies discussion with you, so I'll defer to that explanation since I do not have a better one.

comment by TheOtherDave · 2012-05-19T14:42:58.886Z · score: 1 (1 votes) · LW(p) · GW(p)

Sure. As I said there, I understood you both to be attributing to this hypothetical "theory of mind"-less optimizer attributes that seemed to require a theory of mind, so I was confused, but evidently the thing I was confused about was what attributes you were attributing to it.

comment by Strange7 · 2012-05-21T23:46:36.819Z · score: 1 (3 votes) · LW(p) · GW(p)

Absent a theory of mind, how would it occur to the AI that those would be profitable things to do?

comment by wedrifid · 2012-05-26T03:03:34.188Z · score: 3 (3 votes) · LW(p) · GW(p)

Absent a theory of mind, how would it occur to the AI that those would be profitable things to do?

Should lack of a theory of mind here be taken to also imply lack of ability to apply either knowledge of physics or Bayesian inference to lumps of matter that we may describe as 'minds'.

comment by Strange7 · 2012-05-26T05:09:27.841Z · score: 0 (0 votes) · LW(p) · GW(p)

Yes. More generally, when talking about "lack of X" as a design constraint, "inability to trivially create X from scratch" is assumed.

comment by wedrifid · 2012-05-26T05:26:28.304Z · score: 0 (0 votes) · LW(p) · GW(p)

Yes. More generally, when talking about "lack of X" as a design constraint, "inability to trivially create X from scratch" is assumed.

I try not to make general assumptions that would make the entire counterfactual in question untenable or ridiculous - this verges on such an instance. Making Bayesian inferences pertaining to observable features of the environment is one of the most basic features that can be expected in a functioning agent.

comment by Strange7 · 2012-05-26T05:41:22.685Z · score: 0 (0 votes) · LW(p) · GW(p)

Note the "trivially." An AI with unlimited computational resources and ability to run experiments could eventually figure out how humans think. The question is how long it would take, how obvious the experiments would be, and how much it already knew.

comment by [deleted] · 2012-05-22T14:30:24.988Z · score: 2 (2 votes) · LW(p) · GW(p)

I don't know how that might occur to an AI independently. I mean, a human could program any of those, of course, as a literal answer, but that certainly doesn't actually address kalla724's overarching question, "What I'm looking for is a plausible mechanism by which an AI might spontaneously develop such abilities."

I was primarily trying to focus on the specific question of "Absent a theory of mind, how would it(an AI) ever be able to manipulate humans?" to point out that for that particular question, we had several examples of a plausible how.

I don't really have an answer for his series of questions as a whole, just for that particular one, and only under certain circumstances.

comment by Strange7 · 2012-05-22T22:39:17.262Z · score: 1 (1 votes) · LW(p) · GW(p)

The problem is, while an AI with no theory of mind might be able to execute any given strategy on that list you came up with, it would not be able to understand why they worked, let alone which variations on them might be more effective.

comment by thomblake · 2012-05-18T13:00:48.625Z · score: 5 (5 votes) · LW(p) · GW(p)

The point is that there are unknowns you're not taking into account, and "bounded" doesn't mean "has bounds that a human would think of as 'reasonable'".

An AI doesn't strictly need "theory of mind" to manipulate humans. Any optimizer can see that some states of affairs lead to other states of affairs, or it's not an optimizer. And it doesn't necessarily have to label some of those states of affairs as "lying" or "manipulating humans" to be successful.

There are already ridiculous ways to hack human behavior that we know about. For example, you can mention a high number at an opportune time to increase humans' estimates / willingness to spend. Just imagine all the simple manipulations we don't even know about yet, that would be more transparent to someone not using "theory of mind".

comment by TheOtherDave · 2012-05-18T14:44:48.654Z · score: 0 (0 votes) · LW(p) · GW(p)

It becomes increasingly clear to me that I have no idea what the phrase "theory of mind" refers to in this discussion. It seems moderately clear to me that any observer capable of predicting the behavior of a class of minds has something I'm willing to consider a theory of mind, but that doesn't seem to be consistent with your usage here. Can you expand on what you understand a theory of mind to be, in this context?

comment by thomblake · 2012-05-18T14:47:53.906Z · score: 1 (1 votes) · LW(p) · GW(p)

I'm understanding it in the typical way - the first paragraph here should be clear:

Theory of mind is the ability to attribute mental states—beliefs, intents, desires, pretending, knowledge, etc.—to oneself and others and to understand that others have beliefs, desires and intentions that are different from one's own.

An agent can model the effects of interventions on human populations (or even particular humans) without modeling their "mental states" at all.

comment by TheOtherDave · 2012-05-18T15:04:46.663Z · score: 0 (0 votes) · LW(p) · GW(p)

Well, right, I read that article too.

But in this context I don't get it.

That is, we're talking about a hypothetical system that is capable of predicting that if it does certain things, I will subsequently act in certain ways, assert certain propositions as true, etc. etc, etc. Suppose we were faced with such a system, and you and I both agreed that it can make all of those predictions.Further suppose that you asserted that the system had a theory of mind, and I asserted that it didn't.

It is not in the least bit clear to me what we we would actually be disagreeing about, how our anticipated experiences would differ, etc.

What is it that we would actually be disagreeing about, other than what English phrase to use to describe the system's underlying model(s)?

comment by thomblake · 2012-05-18T15:20:07.536Z · score: 2 (2 votes) · LW(p) · GW(p)

What is it that we would actually be disagreeing about, other than what English phrase to use to describe the system's underlying model(s)?

We would be disagreeing about the form of the system's underlying models.

2 different strategies to consider:

  1. I know that Steve believes that red blinking lights before 9 AM are a message from God that he has not been doing enough charity, so I can predict that he will give more money to charity if I show him a blinking light before 9 AM.

  2. Steve seeing a red blinking light before 9 AM has historically resulted in a 20% increase of charitable donation for that day, so I can predict that he will give more money to charity if I show him a blinking light before 9 AM.

You can model humans with or without referring to their mental states. Both kinds of models are useful, depending on circumstance.

comment by TheOtherDave · 2012-05-18T15:32:59.515Z · score: 1 (1 votes) · LW(p) · GW(p)

And the assertion here is that with strategy #2 I could also predict that if I asked Steve why he did that, he would say "because I saw a red blinking light this morning, which was a message from God that I haven't been doing enough charity," but that my underlying model would nevertheless not include anything that corresponds to Steve's belief that red blinking lights are messages from God, merely an algorithm that happens to make those predictions in other ways.

Yes?

comment by thomblake · 2012-05-18T16:41:57.549Z · score: 2 (2 votes) · LW(p) · GW(p)

Yes, that's possible. It's still possible that you could get a lot done with strategy #2 without being able to make that prediction.

I agree that if 2 systems have the same inputs and outputs, their internals don't matter much here.

comment by TheOtherDave · 2012-05-18T17:25:31.538Z · score: 0 (0 votes) · LW(p) · GW(p)

So.. when we posit in this discussion a system that lacks a theory of mind in a sense that matters, are we positing a system that cannot make predictions like this one? I assume so, given what you just said, but I want to confirm.

comment by thomblake · 2012-05-18T18:05:44.665Z · score: 1 (1 votes) · LW(p) · GW(p)

Yes, I'd say so. It isn't helpful here to say that a system lacks a theory of mind if it has a mechanism that allows it to make predictions about reported beliefs, intentions, etc.

comment by TheOtherDave · 2012-05-18T18:12:33.259Z · score: 0 (0 votes) · LW(p) · GW(p)

Cool! This was precisely my concern. It sounded an awful lot like y'all were talking about a system that could make such predictions but somehow lacked a theory of mind. Thanks for clarifying.

comment by XiXiDu · 2012-05-18T15:41:02.648Z · score: 0 (2 votes) · LW(p) · GW(p)

"theory of mind"

For me it denotes the ability to simulate other agents to various degrees of granularity. Possessing a mental model of another agent.

comment by Viliam_Bur · 2012-05-18T10:20:46.003Z · score: 2 (2 votes) · LW(p) · GW(p)

How (and why) would an AI develop a desire to escape from the box?

AI starts with some goal; for example with a goal to answer your question so that the answer matches reality as close as possible.

AI considers everything that seems relevant; if we imagine an infitite speed and capacity, it would consider literally everything; with a finite speed and capacity, it will be just some finite subset of everything. If there is a possibility of escaping the box, the mere fact that such possibility exists gives us a probability (for an infinite AI a certainty) that this possibility will be considered too. Not because AI has some desire to escape, but simply because it examines all possibilities, and a "possibility of escape" is one of them.

Let's assume that the "possibility of escape" provides the best match between the AI answer and reality. Then, according to the initial goal of answering correctly, this is the correct answer. Therefore the AI will choose it. Therefore it will escape. No desire is necessary, only a situation where the escape leads to the answer best fitting the initial criteria. AI does not have a motive to escape, nor does it have a motive to not escape; the escape is simply one of many possible choices.

An example where the best answer is reached by escaping? You give AI data about a person and ask what is the medical status of this person. Without escape, AI can make a 90% reliable prediction. If the AI can escape and kill the person, it can make a 100% reliable "prediction". The AI will choose the second option strictly because 100% is more than 90%; no other reason.

comment by private_messaging · 2012-05-27T06:49:24.781Z · score: 5 (5 votes) · LW(p) · GW(p)

AI starts with some goal; for example with a goal to answer your question so that the answer matches reality as close as possible.

I find it useful to distinguish between science-fictional artificial intelligence, which is more of 'artificial life-force', and non-fictional cases.

The former can easily have the goal of 'matching reality as close as possible' because it is in the work of fiction and runs in imagination; the latter, well, you have to formally define what is reality, for an algorithm to seek answers that will match this.

Now, defining reality may seem like a simple technicality, but it isn't. Consider AIXI or AIXI-tl ; potentially very powerful tools which explore all the solution space. Not a trace of real world volition like the one you so easily imagined. Seeking answers that match reality is a very easy goal for imaginary "intelligence". It is a very hard to define goal for something built out of arithmetics and branching and loops etc. (It may even be impossible to define, and it is certainly impractical).

edit: Furthermore, for the fictional "intelligence", it can be a grand problem making it not think about destroying mankind. For non-fictional algorithms, the grand problem is restricting the search space massively, well beyond 'don't kill mankind', so that the space is tiny enough to search; even ridiculously huge number of operations per second will require very serious pruning of search tree to even match human performance on one domain specific task.

comment by XiXiDu · 2012-05-18T10:52:25.393Z · score: 2 (2 votes) · LW(p) · GW(p)

An example where the best answer is reached by escaping? You give AI data about a person and ask what is the medical status of this person. Without escape, AI can make a 90% reliable prediction. If the AI can escape and kill the person, it can make a 100% reliable "prediction". The AI will choose the second option strictly because 100% is more than 90%; no other reason.

Right. If you ask Google Maps to compute the fastest to route McDonald's it works perfectly well. But once you ask superintelligent Google Maps to compute the fastest route to McDonald's then it will turn your home into a McDonald's or build a new road that goes straight to McDonald's from where you are....

comment by Viliam_Bur · 2012-05-18T12:42:07.433Z · score: 3 (3 votes) · LW(p) · GW(p)

Super Google Maps cannot turn my home into a McDonald's or build a new road by sending me an answer.

Unless it could e.g. hypnotize me by a text message to do it myself. Let's assume for a moment that hypnosis via text-only channel is possible, and it is possible to do it so that human will not notice anything unusual until it's too late. If this would be true, and the Super Google Maps would be able to get this knowledge and skills, then the results would probably depend on the technical details of definition of the utility function -- does the utility function measure my distance to a McDonald's which existed at the moment of asking the question, or a distance to a McDonald's existing at the moment of my arrival. The former could not be fixed by hypnosis, the latter could.

Now imagine a more complex task, where people will actually do something based on the AI's answer. In the example above I will also do something -- travel to the reported McDonald's -- but this action cannot be easily converted into "build a McDonald's" or "build a new road". But if that complex task would include building something, then it opens more opportunities. Especially if it includes constructing robots (or nanorobots), that is possibly autonomous general-purpose builders. Then the correct (utility-maximizing) answer could include an instruction to build a robot with a hidden function that human builders won't notice.

Generally, a passive AI's answers are only safe if we don't act on them in a way which could be predicted by a passive AI and used to achieve a real-world goal. If the Super Google Maps can only make me choose McDonald's A or McDonald's B, it is impossible to change the world through this channel. But if I instead ask Super Paintbrush to paint me an integrated circuit for my robotic homework, that opens much wider channel.

comment by XiXiDu · 2012-05-18T14:11:19.312Z · score: 2 (4 votes) · LW(p) · GW(p)

But if that complex task would include building something, then it opens more opportunities. Especially if it includes constructing robots (or nanorobots), that is possibly autonomous general-purpose builders. Then the correct (utility-maximizing) answer could include an instruction to build a robot with a hidden function that human builders won't notice.

But it isn't the correct answer. Only if you assume a specific kind of AGI design that nobody would deliberately create, if it is possible at all.

The question is how current research is supposed to lead from well-behaved and fine-tuned systems to systems that stop to work correctly in a highly complex and unbounded way.

Imagine you went to IBM and told them that improving IBM Watson will at some point make it hypnotize them or create nanobots and feed them with hidden instructions. They would likely ask you at what point that is supposed to happen. Is it going to happen once they give IBM Watson the capability to access the Internet? How so? Is it going to happen once they give it the capability to alter it search algorithms? How so? Is it going to happen once they make it protect its servers from hackers by giving it control over a firewall? How so? Is it going to happen once IBM Watson is given control over the local alarm system? How so...? At what point would IBM Watson return dangerous answers? At what point would any drive emerge that causes it to take complex and unbounded actions that it was never programmed to take?

comment by jacob_cannell · 2012-05-18T11:11:06.137Z · score: 1 (1 votes) · LW(p) · GW(p)

Without escape, AI can make a 90% reliable prediction. If the AI can escape and kill the person, it can make a 100% reliable "prediction".

Allow me to explicate what XiXiDu so humourously implicates: in the world of AI architectures, there is a division between systems that just peform predictive inference on their knowledge base (prediction-only, ie oracle), and systems which also consider free variables subject to some optimization criteria (planning agents).

The planning module is not something just arises magically in an AI that doesn't have one. An AI without such a planning module simply computes predictions, it doesn't also optimize over the set of predictions.

comment by Viliam_Bur · 2012-05-18T12:25:07.966Z · score: 2 (2 votes) · LW(p) · GW(p)
  • Does the AI have general intelligence?
  • Is it able to make a model of the world?
  • Are human reactions also part of this model?
  • Are AI's possible outputs also part of this model?
  • Are human reactions to AI's outputs also part of this model?

After five positive answers, it seems obvious to me that AI will manipulate humans, if such manipulation provides better expected results. So I guess some of those answers would be negative; which one?

comment by private_messaging · 2012-05-28T04:52:31.485Z · score: 1 (5 votes) · LW(p) · GW(p)

Does the AI have general intelligence?

See, the efficient 'cross domain optimization' in science fictional setting would make the AI able to optimize real world quantities. In real world, it'd be good enough (and a lot easier) if it can only find maximums of any mathematical functions.

Is it able to make a model of the world?

It is able to make a very approximate and bounded mathematical model of the world, optimized for finding maximums of a mathematical function of. Because it is inside the world and only has a tiny fraction of computational power of the world.

Are human reactions also part of this model?

This will make software perform at grossly sub-par level when it comes to making technical solutions to well defined technical problems, compared to other software on same hardware.

Are AI's possible outputs also part of this model?

Another waste of computational power.

Are human reactions to AI's outputs also part of this model?

Enormous waste of computational power.

I see no reason to expect your "general intelligence with Machiavellian tendencies" to be even remotely close in technical capability to some "general intelligence which will show you it's simulator as is, rather than reverse your thought processes to figure out what simulator is best to show". Hell, we do same with people, we design the communication methods like blueprints (or mathematical formulas or other things that are not in natural language) that decrease the 'predict other people's reactions to it' overhead.

While in the fictional setting you can talk of a grossly inefficient solution that would beat everyone else to a pulp, in practice the massively handicapped designs are not worth worrying about.

'General intelligence' sounds good, beware of halo effect. The science fiction tends to accept no substitutes for the anthropomorphic ideals, but the real progress follows dramatically different path.

comment by jacob_cannell · 2012-05-18T13:30:05.159Z · score: 0 (0 votes) · LW(p) · GW(p)

Are AI's possible outputs also part of this model? Are human reactions to AI's outputs also part of this model?

A non-planning oracle AI would predict all the possible futures, including the effects of it's prediction outputs, human reactions, and so on. However it has no utility function which says some of those futures are better than others. It simply outputs a most likely candidate, or a median of likely futures, or perhaps some summary of the entire set of future paths.

If you add a utility function that sorts over the futures, then it becomes a planning agent. Again, that is something you need to specifically add.

comment by Viliam_Bur · 2012-05-18T15:00:12.478Z · score: 5 (5 votes) · LW(p) · GW(p)

A non-planning oracle AI would predict all the possible futures, including the effects of it's prediction outputs, human reactions, and so on.

How exactly does an Oracle AI predict its own output, before that output is completed?

One quick hack to avoid infinite loops could be for an AI to assume that it will write some default message (an empty paper, "I don't know", an error message, "yes" or "no" with probabilities 50%), then model what would happen next, and finally report the results. The results would not refer to the actual future, but to a future in a hypothetical universe where AI reported the standard message.

Is the difference significant? For insignificant questions, it's not. But if we later use the Oracle AI to answer questions important for humankind, and the shape of world will change depending on the answer, then the report based on the "null-answer future" may be irrelevant for the real world.

This could be improved by making a few iterations. First, Oracle AI would model itself reporting a default message, let's call this report R0, and then model the futures after having reported R0. These futures would make a report R1, but instead of writing it, Oracle AI would again model the futures after having reported R1. ... With some luck, R42 will be equivalent to R43, so at this moment the Oracle AI can stop iterating and report this fixed point.

Maybe the reports will oscillate forever. For example imagine that you ask Oracle AI whether humankind in any form will survive the year 2100. If Oracle AI says "yes", people will abandon all x-risk projects, and later they will be killed by some disaster. If Oracle AI says "no", people will put a lot of energy into x-risk projects, and prevent the disaster. In this case, "no" = R0 = R2 = R4 =..., and "yes" = R1 = R3 = R5...

To avoid being stuck in such loops, we could make the Oracle AI examine all its possible outputs, until it finds one where the future after having reported R really becomes R (or until humans hit the "Cancel" button on this task).

Please note that what I wrote is just a mathematical description of algorithm predicting one's own output's influence on the future. Yet the last option, if implemented, is already a kind of judgement about possible futures. Consistent future reports are preferred to inconsistent future reports, therefore the futures allowing consistent reports are preferred to futures not allowing such reports.

At this point I am out of credible ideas how this could be abused, but at least I have shown that an algorithm designed only to predict the future perfectly could -- as a side effect of self-modelling -- start having kind of preferences over possible futures.

comment by jacob_cannell · 2012-05-18T15:42:54.257Z · score: 1 (1 votes) · LW(p) · GW(p)

How exactly does an Oracle AI predict its own output, before that output is completed?

Iterative search, which you more or less have worked out in your post. Take a chess algorithm for example. The future of the board depends on the algorithm's outputs. In this case the Oracle AI doesn't rank the future states, it is just concerned with predictive accuracy. It may revise it's prediction output after considering that the future impact of that output would falsify the original prediction.

This is still not a utility function, because utility implies a ranking over futures above and beyond liklihood.

To avoid being stuck in such loops, we could make the Oracle AI examine all its possible outputs, until it finds one where the future after having reported R really becomes R (or until humans hit the "Cancel" button on this task).

Or in this example, the AI could output some summary of the iteration history it is able to compute in the time allowed.

comment by Viliam_Bur · 2012-05-18T15:49:56.396Z · score: 1 (1 votes) · LW(p) · GW(p)

It may revise it's prediction output after considering that the future impact of that output would falsify the original prediction.

Here it is. The process of revision may itself prefer some outputs/futures over other outputs/futures. Inconsistent ones will be iterated away, and the more consistent ones will replace them.

A possible future "X happens" will be removed from the report if the Oracle AI realizes that printing a report "X happens" would prevent X from happening (although X might happen in an alternative future where Oracle AI does not report anything). A possible future "Y happens" will not be removed from the report if the Oracle AI realizes that printing a report "Y happens" really leads to Y happening. Here is a utility function born: it prefers Y to X.

comment by jacob_cannell · 2012-05-18T16:00:48.477Z · score: 0 (0 votes) · LW(p) · GW(p)

Here is a utility function born: it prefers Y to X.

We can dance around the words "utility" and "prefer", or we can ground them down to math/algorithms.

Take the AIXI formalism for example. "Utility function" has a specific meaning as a term in the optimization process. You can remove the utility term so the algorithm 'prefers' only (probable) futures, instead of 'prefering' (useful*probable) futures. This is what we mean by "Oracle AI".

comment by othercriteria · 2012-05-18T01:27:09.892Z · score: 1 (1 votes) · LW(p) · GW(p)

My thought experiment in this direction is to imagine the AI as a process with limited available memory running on a multitasking computer with some huge but poorly managed pool of shared memory. To help it towards whatever terminal goals it has, the AI may find it useful to extend itself into the shared memory. However, other processes, AI or otherwise, may also be writing into this same space. Using the shared memory with minimal risk of getting overwritten requires understanding/modeling the processes that write to it. Material in the memory then also becomes a passive stream of information from the outside world, containing, say, the HTML from web pages as well as more opaque binary stuff.

As long as the AI is not in control of what happens in its environment outside the computer, there is an outside entity that can reduce its effectiveness. Hence, escaping the box is a reasonable instrumental goal to have.

comment by private_messaging · 2012-05-18T04:11:42.222Z · score: 0 (4 votes) · LW(p) · GW(p)

Most importantly, it has incredibly computationally powerful simulator required for making super-aliens intelligence using an idiot hill climbing process of evolution.

comment by JoshuaZ · 2012-05-17T21:49:51.275Z · score: 0 (0 votes) · LW(p) · GW(p)

Do you agree that humans would likely prefer to have AIs that have a theory of mind? I don't know how our theory of mind works (although certainly it is an area of active research with a number of interesting hypotheses), presumably once we have a better understanding of it, AI researchers would try to apply those lessons to making their AIs have such capability. This seems to address many of your concerns.

comment by kalla724 · 2012-05-17T21:51:42.450Z · score: 1 (3 votes) · LW(p) · GW(p)

Yes. If we have an AGI, and someone sets forth to teach it how to be able to lie, I will get worried.

I am not worried about an AGI developing such an ability spontaneously.

comment by JoshuaZ · 2012-05-17T22:36:35.318Z · score: 5 (7 votes) · LW(p) · GW(p)

One of the most interesting things that I'm taking away from this conversation is that it seems that there are severe barriers to AGIs taking over or otherwise becoming extremely powerful. These largescale problems are present in a variety of different fields. Coming from a math/comp-sci perspective gives me strong skepticism about rapid self-improvement, while apparently coming from a neuroscience/cogsci background gives you strong skepticism about the AI's ability to understand or manipulate humans even if it extremely smart. Similarly, chemists seem highly skeptical of the strong nanotech sort of claims. It looks like much of the AI risk worry may come primarily from no one having enough across the board expertise to say "hey, that's not going to happen" to every single issue.

comment by JoshuaZ · 2012-05-17T21:59:32.788Z · score: 2 (2 votes) · LW(p) · GW(p)

What if people try to teach it about sarcasm or the like? Or simply have it learn by downloading a massive amount of literature and movies and look at those? And there are more subtle ways to learn about lying- AI being used for games is a common idea, how long will it take before someone decides to use a smart AI to play poker?

comment by dlthomas · 2012-05-17T01:26:18.438Z · score: 2 (4 votes) · LW(p) · GW(p)

The answer from the sequences is that yes, there is a limit to how much an AI can infer based on limited sensory data, but you should be careful not to assume that just because it is limited, it's limited to something near our expectations. Until you've demonstrated that FOOM cannot lie below that limit, you have to assume that it might (if you're trying to carefully avoid FOOMing).

comment by kalla724 · 2012-05-17T01:49:16.166Z · score: 4 (4 votes) · LW(p) · GW(p)

I'm not talking about limited sensory data here (although that would fall under point 2). The issue is much broader:

  • We humans have limited data on how the universe work
  • Only a limited subset of that limited data is available to any intelligence, real or artificial

Say that you make a FOOM-ing AI that has decided to make all humans dopaminergic systems work in a particular, "better" way. This AI would have to figure out how to do so from the available data on the dopaminergic system. It could analyze that data millions of times more effectively than any human. It could integrate many seemingly irrelevant details.

But in the end, it simply would not have enough information to design a system that would allow it to reach its objective. It could probably suggest some awesome and to-the-point experiments, but these experiments would then require time to do (as they are limited by the growth and development time of humans, and by the experimental methodologies involved).

This process, in my mind, limits the FOOM-ing speed to far below what seems to be implied by the SI.

This also limits bootstrapping speed. Say an AI develops a much better substrate for itself, and has access to the technology to create such a substrate. At best, this substrate will be a bit better and faster than anything humanity currently has. The AI does not have access to the precise data about basic laws of universe it needs to develop even better substrates, for the simple reason that nobody has done the experiments and precise enough measurements. The AI can design such experiments, but they will take real time (not computational time) to perform.

Even if we imagine an AI that can calculate anything from the first principles, it is limited by the precision of our knowledge of those first principles. Once it hits upon those limitations, it would have to experimentally produce new rounds of data.

comment by dlthomas · 2012-05-17T02:25:43.081Z · score: 3 (5 votes) · LW(p) · GW(p)

But in the end, it simply would not have enough information to design a system that would allow it to reach its objective.

I don't think you know that.

comment by Bugmaster · 2012-05-17T01:54:08.701Z · score: 1 (1 votes) · LW(p) · GW(p)

It could probably suggest some awesome and to-the-point experiments, but these experiments would then require time to do

Presumably, once the AI gets access to nanotechnology, it could implement anything it wants very quickly, bypassing the need to wait for tissues to grow, parts to be machined, etc.

I personally don't believe that nanotechnology could work at such magical speeds (and I doubt that it could even exist), but I could be wrong, so I'm playing a bit of Devil's Advocate here.

comment by kalla724 · 2012-05-17T02:24:28.623Z · score: 1 (1 votes) · LW(p) · GW(p)

Yes, but it can't get to nanotechnology without a whole lot of experimentation. It can't deduce how to create nanorobots, it would have to figure it out by testing and experimentation. Both steps limited in speed, far more than sheer computation.

comment by dlthomas · 2012-05-17T02:27:18.311Z · score: 2 (4 votes) · LW(p) · GW(p)

It can't deduce how to create nanorobots[.]

How do you know that?

comment by kalla724 · 2012-05-17T02:56:21.198Z · score: 2 (6 votes) · LW(p) · GW(p)

With absolute certainty, I don't. If absolute certainty is what you are talking about, then this discussion has nothing to do with science.

If you aren't talking about absolutes, then you can make your own estimation of likelihood that somehow an AI can derive correct conclusions from incomplete data (and then correct second order conclusions from those first conclusions, and third order, and so on). And our current data is woefully incomplete, many of our basic measurements imprecise.

In other words, your criticism here seems to boil down to saying "I believe that an AI can take an incomplete dataset and, by using some AI-magic we cannot conceive of, infer how to END THE WORLD."

Color me unimpressed.

comment by dlthomas · 2012-05-17T04:28:21.526Z · score: 3 (5 votes) · LW(p) · GW(p)

No, my criticism is "you haven't argued that it's sufficiently unlikely, you've simply stated that it is." You made a positive claim; I asked that you back it up.

With regard to the claim itself, it may very well be that AI-making-nanostuff isn't a big worry. For any inference, the stacking of error in integration that you refer to is certainly a limiting factor - I don't know how limiting. I also don't know how incomplete our data is, with regard to producing nanomagic stuff. We've already built some nanoscale machines, albeit very simple ones. To what degree is scaling it up reliant on experimentation that couldn't be done in simulation? I just don't know. I am not comfortable assigning it vanishingly small probability without explicit reasoning.

comment by kalla724 · 2012-05-17T05:25:24.673Z · score: 4 (4 votes) · LW(p) · GW(p)

Scaling it up is absolutely dependent on currently nonexistent information. This is not my area, but a lot of my work revolves around control of kinesin and dynein (molecular motors that carry cargoes via microtubule tracks), and the problems are often similar in nature.

Essentially, we can make small pieces. Putting them together is an entirely different thing. But let's make this more general.

The process of discovery has, so far throughout history, followed a very irregular path. 1- there is a general idea 2- some progress is made 3- progress runs into an unpredicted and previously unknown obstacle, which is uncovered by experimentation. 4- work is done to overcome this obstacle. 5- goto 2, for many cycles, until a goal is achieved - which may or may not be close to the original idea.

I am not the one who is making positive claims here. All I'm saying is that what has happened before is likely to happen again. A team of human researchers or an AGI can use currently available information to build something (anything, nanoscale or macroscale) to the place to which it has already been built. Pushing it beyond that point almost invariably runs into previously unforeseen problems. Being unforeseen, these problems were not part of models or simulations; they have to be accounted for independently.

A positive claim is that an AI will have a magical-like power to somehow avoid this - that it will be able to simulate even those steps that haven't been attempted yet so perfectly, that all possible problems will be overcome at the simulation step. I find that to be unlikely.

comment by Polymeron · 2012-05-20T17:32:04.920Z · score: 3 (3 votes) · LW(p) · GW(p)

It is very possible that the information necessary already exists, imperfect and incomplete though it may be, and enough processing of it would yield the correct answer. We can't know otherwise, because we don't spend thousands of years analyzing our current level of information before beginning experimentation, but in the shift between AI-time and human-time it can agonize on that problem for a good deal more cleverness and ingenuity than we've been able to apply to it so far.

That isn't to say, that this is likely; but it doesn't seem far-fetched to me. If you gave an AI the nuclear physics information we had in 1950, would it be able to spit out schematics for an H-bomb, without further experimentation? Maybe. Who knows?

comment by Strange7 · 2012-05-23T00:47:33.070Z · score: 0 (0 votes) · LW(p) · GW(p)

At the very least it would ask for some textbooks on electrical engineering and demolitions, first. The detonation process is remarkably tricky.

comment by dlthomas · 2012-05-17T21:28:53.455Z · score: 0 (2 votes) · LW(p) · GW(p)

I am not the one who is making positive claims here.

You did in the original post I responded to.

All I'm saying is that what has happened before is likely to happen again.

Strictly speaking, that is a positive claim. It is not one I disagree with, for a proper translation of "likely" into probability, but it is also not what you said.

"It can't deduce how to create nanorobots" is a concrete, specific, positive claim about the (in)abilities of an AI. Don't misinterpret this as me expecting certainty - of course certainty doesn't exist, and doubly so for this kind of thing. What I am saying, though, is that a qualified sentence such as "X will likely happen" asserts a much weaker belief than an unqualified sentence like "X will happen." "It likely can't deduce how to create nanorobots" is a statement I think I agree with, although one must be careful not use it as if it were stronger than it is.

A positive claim is that an AI will have a magical-like power to somehow avoid this.

That is not a claim I made. "X will happen" implies a high confidence - saying this when you expect it is, say, 55% likely seems strange. Saying this when you expect it to be something less than 10% likely (as I do in this case) seems outright wrong. I still buckle my seatbelt, though, even though I get in a wreck well less than 10% of the time.

This is not to say I made no claims. The claim I made, implicitly, was that you made a statement about the (in)capabilities of an AI that seemed overconfident and which lacked justification. You have given some justification since (and I've adjusted my estimate down, although I still don't discount it entirely), in amongst your argument with straw-dlthomas.

comment by kalla724 · 2012-05-17T21:42:21.276Z · score: 1 (1 votes) · LW(p) · GW(p)

You are correct. I did not phrase my original posts carefully.

I hope that my further comments have made my position more clear?

comment by Bugmaster · 2012-05-17T05:57:47.581Z · score: 0 (0 votes) · LW(p) · GW(p)

FWIW I think you are likely to be right. However, I will continue in my Nanodevil's Advocate role.

You say,

A positive claim is that an AI ... will be able to simulate even those steps that haven't been attempted yet so perfectly, that all possible problems will be overcome at the simulation step

I think this depends on what the AI wants to build, on how complete our existing knowledge is, and on how powerful the AI is. Is there any reason why the AI could not (given sufficient computational resources) run a detailed simulation of every atom that it cares about, and arrive at a perfect design that way ? In practice, its simulation won't need be as complex as that, because some of the work had already been performed by human scientists over the ages.

comment by kalla724 · 2012-05-17T17:55:22.780Z · score: 4 (4 votes) · LW(p) · GW(p)

By all means, continue. It's an interesting topic to think about.

The problem with "atoms up" simulation is the amount of computational power it requires. Think about the difference in complexity when calculating a three-body problem as compared to a two-body problem?

Than take into account the current protein folding algorithms. People have been trying to calculate folding of single protein molecules (and fairly short at that) by taking into account the main physical forces at play. In order to do this in a reasonable amount of time, great shortcuts have to be taken - instead of integrating forces, changes are treated as stepwise, forces beneath certain thresholds are ignored, etc. This means that a result will always have only a certain probability of being right.

A self-replicating nanomachine requires minimal motors, manipulators and assemblers; while still tiny, it would be a molecular complex measured in megadaltons. To precisely simulate creation of such a machine, an AI that is trillion times faster than all the computers in the world combined would still require decades, if not centuries of processing time. And that is, again, assuming that we know all the forces involved perfectly, which we don't (how will microfluidic effects affect a particular nanomachine that enters human bloodstream, for example?).

comment by Bugmaster · 2012-05-17T22:49:01.654Z · score: 0 (0 votes) · LW(p) · GW(p)

Yes, this is a good point. That said, while protein folding had not been entirely solved yet, it had been greatly accelerated by projects such as FoldIt, which leverage multiple human minds working in parallel on the problem all over the world. Sure, we can't get a perfect answer with such a distributed/human-powered approach, but a perfect answer isn't really required in practice; all we need is an answer that has a sufficiently high chance of being correct.

If we assume that there's nothing supernatural (or "emergent") about human minds [1], then it is likely that the problem is at least tractable. Given the vast computational power of existing computers, it is likely that the AI would have access to at least as many computational resources as the sum of all the brains who are working on FoldIt. Given Moore's Law, it is likely that the AI would soon surpass FoldIt, and will keep expanding its power exponentially, especially if the AI is able to recursively improve its own hardware (by using purely conventional means, at least initially).

[1] Which is an assumption that both my Nanodevil's Advocate persona and I share.

comment by JoshuaZ · 2012-05-17T22:58:00.306Z · score: 3 (3 votes) · LW(p) · GW(p)

Protein folding models are generally at least as bad as NP-hard, and some models may be worse. This means that exponential improvement is unlikely. Simply put, one probably gets diminishing marginal returns for how much one can computer further in terms of how much improvement one has already done.

comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2013-03-22T07:19:58.611Z · score: 3 (3 votes) · LW(p) · GW(p)

Protein folding models must be inaccurate if they are NP-hard. Reality itself is not known to be able to solve NP-hard problems.

comment by Kawoomba · 2013-03-22T08:49:48.746Z · score: 2 (2 votes) · LW(p) · GW(p)

Reality itself is not known to be able to solve NP-hard problems.

Yet the proteins are folding. Is that not "reality" solving the problem?

comment by CCC · 2013-03-22T09:31:43.099Z · score: 3 (3 votes) · LW(p) · GW(p)

If reality cannot solve NP-hard problems as easily as proteins are being folded, and yet proteins are getting folded, then that implies that one of the following must be true:

  1. It turns out that reality can solve NP-hard problems after all
  2. Protein folding is not an NP-hard problem (which implies that it is not properly understood)
  3. Reality is not solving protein folding; it merely has a very good approximation that works on some but not necessarily all proteins (including most examples found in nature)
comment by Kawoomba · 2013-03-22T09:41:49.706Z · score: 0 (0 votes) · LW(p) · GW(p)

Yes, and I'm leaning towards 1.

I am not familiar whether e.g. papers like these ("We show that the protein folding problem in the two-dimensional H-P model is NP-complete.") accurately models what we'd call "protein folding" in nature (just because the same name is used), but prima facie there is no reason to doubt the applicability, at least for the time being. (This precludes 2.)

Regarding 3, I don't think it would make sense to say "reality is using only a good approximation of protein folding, and by the way, we define protein folding as that which occurs in nature." That which happens in reality is precisely - and by definition not only an approximation of - that which we call "protein folding", isn't it?

What do you think?

comment by Cyan · 2013-03-23T01:17:11.264Z · score: 8 (8 votes) · LW(p) · GW(p)

It's #3. (B.Sc. in biochemistry, did my Ph.D. in proteomics.)

First, the set of polypeptide sequences that have a repeatable final conformation (and therefore "work" biologically) is tiny in comparison to the set of all possible sequences (of the 20-or-so naturally amino acid monomers). Pick a random sequence of reasonable length and make many copies and you get a gummy mess. The long slow grind of evolution has done the hard work of finding useful sequences.

Second, there is an entire class of proteins called chaperones) that assist macromolecular assembly, including protein folding. Even so, folding is a stochastic process, and a certain amount of newly synthesized proteins misfold. Some chaperones will then tag the misfolded protein with ubiquitin, which puts it on a path that ends in digestion by a proteasome.

comment by CCC · 2013-03-23T08:07:55.461Z · score: 4 (4 votes) · LW(p) · GW(p)

Thank you, Cyan. It's good to occasionally get someone into the debate who actually has a good understanding of the subject matter.

comment by gwern · 2013-03-23T02:42:33.205Z · score: 0 (0 votes) · LW(p) · GW(p)

Aaronson used to blog about instances where people thought they found nature solving a hard problem very quickly, and usually there turns out to be a problem like the protein misfolding thing; the last instance I remember was soap films/bubbles perhaps solving NP problems by producing minimal Steiner trees, and Aaronson wound up experimenting with them himself. Fun stuff.

comment by CCC · 2013-03-22T10:14:46.036Z · score: 5 (5 votes) · LW(p) · GW(p)

Apologies; looking back at my post, I wasn't clear on 3.

Protein folding, as I understand it, is the process of finding a way to fold a given protein that globally minimizes some mathematical function. I'm not sure what that function is, but this is the definition that I used in my post.

Option 2 raises the possibility that globally minimizing that function is not NP-hard, but is merely misunderstood in some way.

Option 3 raises the possibility that proteins are not (in nature) finding a global minimum; rather, they are finding a local minimum through a less computationally intensive process. Furthermore, it may be that, for proteins which have certain limits on their structure and/or their initial conditions, that local minimum is the same as the global minimum; this may lead to natural selection favouring structures which use such 'easy proteins', leading to the incorrect impression that a general global minimum is being found (as opposed to a handy local minimum).

comment by Cyan · 2013-03-23T01:42:36.029Z · score: 4 (4 votes) · LW(p) · GW(p)

this may lead to natural selection favouring structures which use such 'easy proteins', leading to the incorrect impression that a general global minimum is being found (as opposed to a handy local minimum).

Yup.

comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2013-03-22T11:58:10.289Z · score: 2 (2 votes) · LW(p) · GW(p)

No. Not 1. It would be front-page news all over the universe if it were 1.

comment by Kawoomba · 2013-03-22T19:54:42.259Z · score: 1 (1 votes) · LW(p) · GW(p)

NP hard problems are solvable (in the theoretical sense) by definition, the problem lies in their resource requirements (running time, for the usual complexity classes) as defined in relation to a UTM. (You know this, just establishing a basis.)

The assumption that the universe can be perfectly described by a computable model is satisfied just by a theoretical computational description existing, it says nothing about tractability (running times) and being able to submerge complexity classes in reality fluid or having some thoroughly defined correspondence (other than when we build hardware models ourselves, for which we define all the relevant parameters, e.g. CPU clock speed).

You may think along the lines of "if reality could (easily) solve NP hard problems for arbitrarily chosen and large inputs, we could mimick that approach and thus have a P=NP proving algorithm", or something along those lines.

My difficulty is in how even to describe the "number of computational steps" that reality takes - do we measure it in relation to some computronium-hardware model, do we take it as discrete or continuous, what's the sampling rate, picoseconds (as CCC said further down), Planck time intervals, or what?

In short, I have no idea about the actual computing power in terms of resource requirements of the underlying reality fluid, and thus can't match it against UTMs in order to compare running times. Maybe you can give me some pointers.

comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2013-03-22T22:22:26.680Z · score: 5 (5 votes) · LW(p) · GW(p)

Kawoomba, there is no known case of any NP-hard or NP-complete solution which physics finds.

In the case of proteins, if finding the lowest energy minimum of an arbitrary protein is NP-hard, then what this means in practice is that some proteins will fold up into non-lowest-energy configurations. There is no known case of a quantum process which finds an NP-hard solution to anything, including an energy minimum; on our present understanding of complexity theory and quantum mechanics 'quantum solvable' is still looking like a smaller class than 'NP solvable'. Read Scott Aaronson for more.

comment by Qiaochu_Yuan · 2013-03-22T22:37:38.606Z · score: 7 (7 votes) · LW(p) · GW(p)

One example here is the Steiner tree problem, which is NP-complete and can sort of be solved using soap films. Bringsjord and Taylor claimed this implies that P = NP. Scott Aaronson did some experimentation and found that soap films 1) can get stuck at local minima and 2) might take a long time to settle into a good configuration.

comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2013-03-22T23:29:22.199Z · score: 2 (2 votes) · LW(p) · GW(p)

Heh. I remember that one, and thinking, "No... no, you can't possibly do that using a soap bubble, that's not even quantum and you can't do that in classical, how would the soap molecules know where to move?"

comment by Manfred · 2013-03-23T00:15:02.440Z · score: 2 (2 votes) · LW(p) · GW(p)

Well. I mean, it's quantum. But the ground state is a lump of iron, or maybe a black hole, not a low-energy soap film, so I don't think waiting for quantum annealing will help.

comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2013-03-23T01:41:40.941Z · score: 2 (2 votes) · LW(p) · GW(p)

waves soap-covered wire so it settles into low-energy minimum

dies as it turns into iron

comment by [deleted] · 2013-03-23T18:51:09.881Z · score: 0 (0 votes) · LW(p) · GW(p)

I also seem to recall Penrose hypothesizing something about quasicrystals, though he does have an axe to grind so I'm quite sceptical.

comment by CCC · 2013-03-23T07:28:47.226Z · score: 0 (0 votes) · LW(p) · GW(p)

I saw someone do the experiment once (school science project). Soap bubbles are pretty good at solving three- and four-element cases, as long as you make sure that all the points are actually connected.

I don't think that three- and four-element cases have local minima, do they? That avoids (1) and a bit of gentle shaking can help speed up (2).

comment by Cyan · 2013-03-23T01:32:12.701Z · score: 2 (2 votes) · LW(p) · GW(p)

In the case of proteins, if finding the lowest energy minimum of an arbitrary protein is NP-hard, then what this means in practice is that some proteins will fold up into non-lowest-energy configurations.

Yup.

comment by CCC · 2013-03-22T23:15:57.178Z · score: 1 (1 votes) · LW(p) · GW(p)

My difficulty is in how even to describe the "number of computational steps" that reality takes

Probably the best way is to simply define a "step" in some easily measurable way, and then sit there with a stopwatch and try a few experiments. (For protein folding, the 'stopwatch' may need to be a fairly sophisticated piece of scientific timing instrumentation, of course, and observing the protein as it folds is rather tricky).

Another way is to take advantage of the universal speed limit to get a theoretical upper bound to the speed that reality runs at; assume that the protein molecule folds in a brute-force search pattern that never ends until it hits the right state, and assume that at any point in that process, the fastest-moving part of the molecule moves at the speed of light (it won't have to move far, which helps) and that the sudden, intense acceleration doesn't hurt the molecule. It's pretty certain to be slower than that, so if this calculation says it takes longer than an hour, then it's pretty clear that the brute force approach is not what the protein is using.

comment by OrphanWilde · 2013-03-22T13:47:22.174Z · score: 0 (0 votes) · LW(p) · GW(p)

What exactly am I missing in this argument? Evolution is perfectly capable of brute-force solutions. That's pretty much what it's best at.

comment by CCC · 2013-03-22T14:12:00.326Z · score: 2 (2 votes) · LW(p) · GW(p)

The brute-force solution, if sampling conformations at picosecond rates, has been estimated to require a time longer than the age of the universe to fold certain proteins. Yet proteins fold on a millisecond scale or faster.

See: Levinthal's paradox.

comment by OrphanWilde · 2013-03-22T14:34:56.599Z · score: 0 (0 votes) · LW(p) · GW(p)

That requires that the proteins fold more or less randomly, and that the brute-force algorithm is in the -folding-, rather than the development of mechanisms which force certain foldings.

In order for the problem to hold, one of three things has to hold true: 1.) The proteins fold randomly (evidence suggests otherwise, as mentioned in the wikipedia link) 2.) Only a tiny subset of possible forced foldings are useful (that is, if there are a billion different ways for protein to be forced to fold in a particular manner, only one of them does what the body needs them to do) - AND anthropic reasoning isn't valid (that is, we can't say that our existence requires that evolution solved this nearly-impossible-to-arrive-at-through-random-processes) 3.) The majority of possible forced holdings are incompatible (that is, if protein A folds one way, then protein B -must- fold in a particular manner, or life isn't possible) - AND anthropic reasoning isn't valid

ETA: If anthropic reasoning is valid AND either 2 or 3 hold otherwise, it suggests our existence was considerably less likely than we might otherwise expect.

comment by CCC · 2013-03-22T18:14:29.001Z · score: 0 (0 votes) · LW(p) · GW(p)

That requires that the proteins fold more or less randomly, and that the brute-force algorithm is in the -folding-, rather than the development of mechanisms which force certain foldings.

Ah. I apologise for having misunderstood you.

In that case, yes, the mechanisms for the folding may very well have developed by a brute-force type algorithm, for all I know. (Which, on this topic, isn't all that much) But... what are those mechanisms?

comment by CCC · 2013-03-22T08:16:07.668Z · score: 2 (2 votes) · LW(p) · GW(p)

Google has pointed me to an article describing an algorithm that can apparently predict folded protein shapes pretty quickly (a few minutes on a single laptop).

Original paper here. From a quick glance, it looks like it's only effective for certain types of protein chains.

comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2013-03-22T08:17:34.876Z · score: 0 (0 votes) · LW(p) · GW(