Thoughts on the Singularity Institute (SI)

post by HoldenKarnofsky · 2012-05-11T04:31:30.364Z · score: 265 (285 votes) · LW · GW · Legacy · 1284 comments

Contents

      Summary of my views
      Intent of this post
  Does SI have a well-argued case that its work is beneficial and important?
    Objection 1: it seems to me that any AGI that was set to maximize a "Friendly" utility function would be extraordinarily dangerous.
    Objection 2: SI appears to neglect the potentially important distinction between "tool" and "agent" AI.
    Objection 3: SI's envisioned scenario is far more specific and conjunctive than it appears at first glance, and I believe this scenario to be highly unlikely.
    Other objections to SI's views
    Wrapup
  Is SI the kind of organization we want to bet on?
    Wrapup
  But if there's even a chance …
  Existential risk reduction as a cause
  How I might change my views
  Acknowledgements
None
1284 comments

This post presents thoughts on the Singularity Institute from Holden Karnofsky, Co-Executive Director of GiveWell. Note: Luke Muehlhauser, the Executive Director of the Singularity Institute, reviewed a draft of this post, and commented: "I do generally agree that your complaints are either correct (especially re: past organizational competence) or incorrect but not addressed by SI in clear argumentative writing (this includes the part on 'tool' AI). I am working to address both categories of issues." I take Luke's comment to be a significant mark in SI's favor, because it indicates an explicit recognition of the problems I raise, and thus increases my estimate of the likelihood that SI will work to address them.

September 2012 update: responses have been posted by Luke and Eliezer (and I have responded in the comments of their posts). I have also added acknowledgements.

The Singularity Institute (SI) is a charity that GiveWell has been repeatedly asked to evaluate. In the past, SI has been outside our scope (as we were focused on specific areas such as international aid). With GiveWell Labs we are open to any giving opportunity, no matter what form and what sector, but we still do not currently plan to recommend SI; given the amount of interest some of our audience has expressed, I feel it is important to explain why. Our views, of course, remain open to change. (Note: I am posting this only to Less Wrong, not to the GiveWell Blog, because I believe that everyone who would be interested in this post will see it here.)

I am currently the GiveWell staff member who has put the most time and effort into engaging with and evaluating SI. Other GiveWell staff currently agree with my bottom-line view that we should not recommend SI, but this does not mean they have engaged with each of my specific arguments. Therefore, while the lack of recommendation of SI is something that GiveWell stands behind, the specific arguments in this post should be attributed only to me, not to GiveWell.

Summary of my views

Intent of this post

I did not write this post with the purpose of "hurting" SI. Rather, I wrote it in the hopes that one of these three things (or some combination) will happen:

  1. New arguments are raised that cause me to change my mind and recognize SI as an outstanding giving opportunity. If this happens I will likely attempt to raise more money for SI (most likely by discussing it with other GiveWell staff and collectively considering a GiveWell Labs recommendation).
  2. SI concedes that my objections are valid and increases its determination to address them. A few years from now, SI is a better organization and more effective in its mission.
  3. SI can't or won't make changes, and SI's supporters feel my objections are valid, so SI loses some support, freeing up resources for other approaches to doing good.

Which one of these occurs will hopefully be driven primarily by the merits of the different arguments raised. Because of this, I think that whatever happens as a result of my post will be positive for SI's mission, whether or not it is positive for SI as an organization. I believe that most of SI's supporters and advocates care more about the former than about the latter, and that this attitude is far too rare in the nonprofit world.

Does SI have a well-argued case that its work is beneficial and important?

I know no more concise summary of SI's views than this page, so here I give my own impressions of what SI believes, in italics.

  1. There is some chance that in the near future (next 20-100 years), an "artificial general intelligence" (AGI) - a computer that is vastly more intelligent than humans in every relevant way - will be created.
  2. This AGI will likely have a utility function and will seek to maximize utility according to this function.
  3. This AGI will be so much more powerful than humans - due to its superior intelligence - that it will be able to reshape the world to maximize its utility, and humans will not be able to stop it from doing so.
  4. Therefore, it is crucial that its utility function be one that is reasonably harmonious with what humans want. A "Friendly" utility function is one that is reasonably harmonious with what humans want, such that a "Friendly" AGI (FAI) would change the world for the better (by human standards) while an "Unfriendly" AGI (UFAI) would essentially wipe out humanity (or worse).
  5. Unless great care is taken specifically to make a utility function "Friendly," it will be "Unfriendly," since the things humans value are a tiny subset of the things that are possible.
  6. Therefore, it is crucially important to develop "Friendliness theory" that helps us to ensure that the first strong AGI's utility function will be "Friendly." The developer of Friendliness theory could use it to build an FAI directly or could disseminate the theory so that others working on AGI are more likely to build FAI as opposed to UFAI.

From the time I first heard this argument, it has seemed to me to be skipping important steps and making major unjustified assumptions. However, for a long time I believed this could easily be due to my inferior understanding of the relevant issues. I believed my own views on the argument to have only very low relevance (as I stated in my 2011 interview with SI representatives). Over time, I have had many discussions with SI supporters and advocates, as well as with non-supporters who I believe understand the relevant issues well. I now believe - for the moment - that my objections are highly relevant, that they cannot be dismissed as simple "layman's misunderstandings" (as they have been by various SI supporters in the past), and that SI has not published anything that addresses them in a clear way.

Below, I list my major objections. I do not believe that these objections constitute a sharp/tight case for the idea that SI's work has low/negative value; I believe, instead, that SI's own arguments are too vague for such a rebuttal to be possible. There are many possible responses to my objections, but SI's public arguments (and the private arguments) do not make clear which possible response (if any) SI would choose to take up and defend. Hopefully the dialogue following this post will clarify what SI believes and why.

Some of my views are discussed at greater length (though with less clarity) in a public transcript of a conversation I had with SI supporter Jaan Tallinn. I refer to this transcript as "Karnofsky/Tallinn 2011."

Objection 1: it seems to me that any AGI that was set to maximize a "Friendly" utility function would be extraordinarily dangerous.

Suppose, for the sake of argument, that SI manages to create what it believes to be an FAI. Suppose that it is successful in the "AGI" part of its goal, i.e., it has successfully created an intelligence vastly superior to human intelligence and extraordinarily powerful from our perspective. Suppose that it has also done its best on the "Friendly" part of the goal: it has developed a formal argument for why its AGI's utility function will be Friendly, it believes this argument to be airtight, and it has had this argument checked over by 100 of the world's most intelligent and relevantly experienced people. Suppose that SI now activates its AGI, unleashing it to reshape the world as it sees fit. What will be the outcome?

I believe that the probability of an unfavorable outcome - by which I mean an outcome essentially equivalent to what a UFAI would bring about - exceeds 90% in such a scenario. I believe the goal of designing a "Friendly" utility function is likely to be beyond the abilities even of the best team of humans willing to design such a function. I do not have a tight argument for why I believe this, but a comment on LessWrong by Wei Dai gives a good illustration of the kind of thoughts I have on the matter:

What I'm afraid of is that a design will be shown to be safe, and then it turns out that the proof is wrong, or the formalization of the notion of "safety" used by the proof is wrong. This kind of thing happens a lot in cryptography, if you replace "safety" with "security". These mistakes are still occurring today, even after decades of research into how to do such proofs and what the relevant formalizations are. From where I'm sitting, proving an AGI design Friendly seems even more difficult and error-prone than proving a crypto scheme secure, probably by a large margin, and there is no decades of time to refine the proof techniques and formalizations. There's good recent review of the history of provable security, titled Provable Security in the Real World, which might help you understand where I'm coming from.

I think this comment understates the risks, however. For example, when the comment says "the formalization of the notion of 'safety' used by the proof is wrong," it is not clear whether it means that the values the programmers have in mind are not correctly implemented by the formalization, or whether it means they are correctly implemented but are themselves catastrophic in a way that hasn't been anticipated. I would be highly concerned about both. There are other catastrophic possibilities as well; perhaps the utility function itself is well-specified and safe, but the AGI's model of the world is flawed (in particular, perhaps its prior or its process for matching observations to predictions are flawed) in a way that doesn't emerge until the AGI has made substantial changes to its environment.

By SI's own arguments, even a small error in any of these things would likely lead to catastrophe. And there are likely failure forms I haven't thought of. The overriding intuition here is that complex plans usually fail when unaccompanied by feedback loops. A scenario in which a set of people is ready to unleash an all-powerful being to maximize some parameter in the world, based solely on their initial confidence in their own extrapolations of the consequences of doing so, seems like a scenario that is overwhelmingly likely to result in a bad outcome. It comes down to placing the world's largest bet on a highly complex theory - with no experimentation to test the theory first.

So far, all I have argued is that the development of "Friendliness" theory can achieve at best only a limited reduction in the probability of an unfavorable outcome. However, as I argue in the next section, I believe there is at least one concept - the "tool-agent" distinction - that has more potential to reduce risks, and that SI appears to ignore this concept entirely. I believe that tools are safer than agents (even agents that make use of the best "Friendliness" theory that can reasonably be hoped for) and that SI encourages a focus on building agents, thus increasing risk.

Objection 2: SI appears to neglect the potentially important distinction between "tool" and "agent" AI.

Google Maps is a type of artificial intelligence (AI). It is far more intelligent than I am when it comes to planning routes.

Google Maps - by which I mean the complete software package including the display of the map itself - does not have a "utility" that it seeks to maximize. (One could fit a utility function to its actions, as to any set of actions, but there is no single "parameter to be maximized" driving its operations.)

Google Maps (as I understand it) considers multiple possible routes, gives each a score based on factors such as distance and likely traffic, and then displays the best-scoring route in a way that makes it easily understood by the user. If I don't like the route, for whatever reason, I can change some parameters and consider a different route. If I like the route, I can print it out or email it to a friend or send it to my phone's navigation application. Google Maps has no single parameter it is trying to maximize; it has no reason to try to "trick" me in order to increase its utility.

In short, Google Maps is not an agent, taking actions in order to maximize a utility parameter. It is a tool, generating information and then displaying it in a user-friendly manner for me to consider, use and export or discard as I wish.

Every software application I know of seems to work essentially the same way, including those that involve (specialized) artificial intelligence such as Google Search, Siri, Watson, Rybka, etc. Some can be put into an "agent mode" (as Watson was on Jeopardy!) but all can easily be set up to be used as "tools" (for example, Watson can simply display its top candidate answers to a question, with the score for each, without speaking any of them.)

The "tool mode" concept is importantly different from the possibility of Oracle AI sometimes discussed by SI. The discussions I've seen of Oracle AI present it as an Unfriendly AI that is "trapped in a box" - an AI whose intelligence is driven by an explicit utility function and that humans hope to control coercively. Hence the discussion of ideas such as the AI-Box Experiment. A different interpretation, given in Karnofsky/Tallinn 2011, is an AI with a carefully designed utility function - likely as difficult to construct as "Friendliness" - that leaves it "wishing" to answer questions helpfully. By contrast with both these ideas, Tool-AGI is not "trapped" and it is not Unfriendly or Friendly; it has no motivations and no driving utility function of any kind, just like Google Maps. It scores different possibilities and displays its conclusions in a transparent and user-friendly manner, as its instructions say to do; it does not have an overarching "want," and so, as with the specialized AIs described above, while it may sometimes "misinterpret" a question (thereby scoring options poorly and ranking the wrong one #1) there is no reason to expect intentional trickery or manipulation when it comes to displaying its results.

Another way of putting this is that a "tool" has an underlying instruction set that conceptually looks like: "(1) Calculate which action A would maximize parameter P, based on existing data set D. (2) Summarize this calculation in a user-friendly manner, including what Action A is, what likely intermediate outcomes it would cause, what other actions would result in high values of P, etc." An "agent," by contrast, has an underlying instruction set that conceptually looks like: "(1) Calculate which action, A, would maximize parameter P, based on existing data set D. (2) Execute Action A." In any AI where (1) is separable (by the programmers) as a distinct step, (2) can be set to the "tool" version rather than the "agent" version, and this separability is in fact present with most/all modern software. Note that in the "tool" version, neither step (1) nor step (2) (nor the combination) constitutes an instruction to maximize a parameter - to describe a program of this kind as "wanting" something is a category error, and there is no reason to expect its step (2) to be deceptive.

I elaborated further on the distinction and on the concept of a tool-AI in Karnofsky/Tallinn 2011.

This is important because an AGI running in tool mode could be extraordinarily useful but far more safe than an AGI running in agent mode. In fact, if developing "Friendly AI" is what we seek, a tool-AGI could likely be helpful enough in thinking through this problem as to render any previous work on "Friendliness theory" moot. Among other things, a tool-AGI would allow transparent views into the AGI's reasoning and predictions without any reason to fear being purposefully misled, and would facilitate safe experimental testing of any utility function that one wished to eventually plug into an "agent."

Is a tool-AGI possible? I believe that it is, and furthermore that it ought to be our default picture of how AGI will work, given that practically all software developed to date can (and usually does) run as a tool and given that modern software seems to be constantly becoming "intelligent" (capable of giving better answers than a human) in surprising new domains. In addition, it intuitively seems to me (though I am not highly confident) that intelligence inherently involves the distinct, separable steps of (a) considering multiple possible actions and (b) assigning a score to each, prior to executing any of the possible actions. If one can distinctly separate (a) and (b) in a program's code, then one can abstain from writing any "execution" instructions and instead focus on making the program list actions and scores in a user-friendly manner, for humans to consider and use as they wish.

Of course, there are possible paths to AGI that may rule out a "tool mode," but it seems that most of these paths would rule out the application of "Friendliness theory" as well. (For example, a "black box" emulation and augmentation of a human mind.) What are the paths to AGI that allow manual, transparent, intentional design of a utility function but do not allow the replacement of "execution" instructions with "communication" instructions? Most of the conversations I've had on this topic have focused on three responses:

One of the things that bothers me most about SI is that there is practically no public content, as far as I can tell, explicitly addressing the idea of a "tool" and giving arguments for why AGI is likely to work only as an "agent." The idea that AGI will be driven by a central utility function seems to be simply assumed. Two examples:

The closest thing I have seen to a public discussion of "tool-AGI" is in Dreams of Friendliness, where Eliezer Yudkowsky considers the question, "Why not just have the AI answer questions, instead of trying to do anything? Then it wouldn't need to be Friendly. It wouldn't need any goals at all. It would just answer questions." His response:

To which the reply is that the AI needs goals in order to decide how to think: that is, the AI has to act as a powerful optimization process in order to plan its acquisition of knowledge, effectively distill sensory information, pluck "answers" to particular questions out of the space of all possible responses, and of course, to improve its own source code up to the level where the AI is a powerful intelligence. All these events are "improbable" relative to random organizations of the AI's RAM, so the AI has to hit a narrow target in the space of possibilities to make superintelligent answers come out.

This passage appears vague and does not appear to address the specific "tool" concept I have defended above (in particular, it does not address the analogy to modern software, which challenges the idea that "powerful optimization processes" cannot run in tool mode). The rest of the piece discusses (a) psychological mistakes that could lead to the discussion in question; (b) the "Oracle AI" concept that I have outlined above. The comments contain some more discussion of the "tool" idea (Denis Bider and Shane Legg seem to be picturing something similar to "tool-AGI") but the discussion is unresolved and I believe the "tool" concept defended above remains essentially unaddressed.

In sum, SI appears to encourage a focus on building and launching "Friendly" agents (it is seeking to do so itself, and its work on "Friendliness" theory seems to be laying the groundwork for others to do so) while not addressing the tool-agent distinction. It seems to assume that any AGI will have to be an agent, and to make little to no attempt at justifying this assumption. The result, in my view, is that it is essentially advocating for a more dangerous approach to AI than the traditional approach to software development.

Objection 3: SI's envisioned scenario is far more specific and conjunctive than it appears at first glance, and I believe this scenario to be highly unlikely.

SI's scenario concerns the development of artificial general intelligence (AGI): a computer that is vastly more intelligent than humans in every relevant way. But we already have many computers that are vastly more intelligent than humans in some relevant ways, and the domains in which specialized AIs outdo humans seem to be constantly and continuously expanding. I feel that the relevance of "Friendliness theory" depends heavily on the idea of a "discrete jump" that seems unlikely and whose likelihood does not seem to have been publicly argued for.

One possible scenario is that at some point, we develop powerful enough non-AGI tools (particularly specialized AIs) that we vastly improve our abilities to consider and prepare for the eventuality of AGI - to the point where any previous theory developed on the subject becomes useless. Or (to put this more generally) non-AGI tools simply change the world so much that it becomes essentially unrecognizable from the perspective of today - again rendering any previous "Friendliness theory" moot. As I said in Karnofsky/Tallinn 2011, some of SI's work "seems a bit like trying to design Facebook before the Internet was in use, or even before the computer existed."

Perhaps there will be a discrete jump to AGI, but it will be a sort of AGI that renders "Friendliness theory" moot for a different reason. For example, in the practice of software development, there often does not seem to be an operational distinction between "intelligent" and "Friendly." (For example, my impression is that the only method programmers had for evaluating Watson's "intelligence" was to see whether it was coming up with the same answers that a well-informed human would; the only way to evaluate Siri's "intelligence" was to evaluate its helpfulness to humans.) "Intelligent" often ends up getting defined as "prone to take actions that seem all-around 'good' to the programmer." So the concept of "Friendliness" may end up being naturally and subtly baked in to a successful AGI effort.

The bottom line is that we know very little about the course of future artificial intelligence. I believe that the probability that SI's concept of "Friendly" vs. "Unfriendly" goals ends up seeming essentially nonsensical, irrelevant and/or unimportant from the standpoint of the relevant future is over 90%.

Other objections to SI's views

There are other debates about the likelihood of SI's work being relevant/helpful; for example,

Unlike the three objections I focus on, these other issues have been discussed a fair amount, and if these other issues were the only objections to SI's arguments I would find SI's case to be strong (i.e., I would find its scenario likely enough to warrant investment in).

Wrapup

For a long time I refrained from engaging in object-level debates over SI's work, believing that others are better qualified to do so. But after talking at great length to many of SI's supporters and advocates and reading everything I've been pointed to as relevant, I still have seen no clear and compelling response to any of my three major objections. As stated above, there are many possible responses to my objections, but SI's current arguments do not seem clear on what responses they wish to take and defend. At this point I am unlikely to form a positive view of SI's work until and unless I do see such responses, and/or SI changes its positions.

Is SI the kind of organization we want to bet on?

This part of the post has some risks. For most of GiveWell's history, sticking to our standard criteria - and putting more energy into recommended than non-recommended organizations - has enabled us to share our honest thoughts about charities without appearing to get personal. But when evaluating a group such as SI, I can't avoid placing a heavy weight on (my read on) the general competence, capability and "intangibles" of the people and organization, because SI's mission is not about repeating activities that have worked in the past. Sharing my views on these issues could strike some as personal or mean-spirited and could lead to the misimpression that GiveWell is hostile toward SI. But it is simply necessary in order to be fully transparent about why I hold the views that I hold.

Fortunately, SI is an ideal organization for our first discussion of this type. I believe the staff and supporters of SI would overwhelmingly rather hear the whole truth about my thoughts - so that they can directly engage them and, if warranted, make changes - than have me sugar-coat what I think in order to spare their feelings. People who know me and my attitude toward being honest vs. sparing feelings know that this, itself, is high praise for SI.

One more comment before I continue: our policy is that non-public information provided to us by a charity will not be published or discussed without that charity's prior consent. However, none of the content of this post is based on private information; all of it is based on information that SI has made available to the public.

There are several reasons that I currently have a negative impression of SI's general competence, capability and "intangibles." My mind remains open and I include specifics on how it could be changed.

A couple positive observations to add context here:

Wrapup

While SI has produced a lot of content that I find interesting and enjoyable, it has not produced what I consider evidence of superior general rationality or of its suitability for the tasks it has set for itself. I see no qualifications or achievements that specifically seem to indicate that SI staff are well-suited to the challenge of understanding the key AI-related issues and/or coordinating the construction of an FAI. And I see specific reasons to be pessimistic about its suitability and general competence.

When estimating the expected value of an endeavor, it is natural to have an implicit "survivorship bias" - to use organizations whose accomplishments one is familiar with (which tend to be relatively effective organizations) as a reference class. Because of this, I would be extremely wary of investing in an organization with apparently poor general competence/suitability to its tasks, even if I bought fully into its mission (which I do not) and saw no other groups working on a comparable mission.

But if there's even a chance …

A common argument that SI supporters raise with me is along the lines of, "Even if SI's arguments are weak and its staff isn't as capable as one would like to see, their goal is so important that they would be a good investment even at a tiny probability of success."

I believe this argument to be a form of Pascal's Mugging and I have outlined the reasons I believe it to be invalid in two posts (here and here). There have been some objections to my arguments, but I still believe them to be valid. There is a good chance I will revisit these topics in the future, because I believe these issues to be at the core of many of the differences between GiveWell-top-charities supporters and SI supporters.

Regardless of whether one accepts my specific arguments, it is worth noting that the most prominent people associated with SI tend to agree with the conclusion that the "But if there's even a chance …" argument is not valid. (See comments on my post from Michael Vassar and Eliezer Yudkowsky as well as Eliezer's interview with John Baez.)

Existential risk reduction as a cause

I consider the general cause of "looking for ways that philanthropic dollars can reduce direct threats of global catastrophic risks, particularly those that involve some risk of human extinction" to be a relatively high-potential cause. It is on the working agenda for GiveWell Labs and we will be writing more about it.

However, I don't think that "Cause X is the one I care about and Organization Y is the only one working on it" to be a good reason to support Organization Y. For donors determined to donate within this cause, I encourage you to consider donating to a donor-advised fund while making it clear that you intend to grant out the funds to existential-risk-reduction-related organizations in the future. (One way to accomplish this would be to create a fund with "existential risk" in the name; this is a fairly easy thing to do and one person could do it on behalf of multiple donors.)

For one who accepts my arguments about SI, I believe withholding funds in this way is likely to be better for SI's mission than donating to SI - through incentive effects alone (not to mention my specific argument that SI's approach to "Friendliness" seems likely to increase risks).

How I might change my views

My views are very open to revision.

However, I cannot realistically commit to read and seriously consider all comments posted on the matter. The number of people capable of taking a few minutes to write a comment is sufficient to swamp my capacity. I do encourage people to comment and I do intend to read at least some comments, but if you are looking to change my views, you should not consider posting a comment to be the most promising route.

Instead, what I will commit to is reading and carefully considering up to 50,000 words of content that are (a) specifically marked as SI-authorized responses to the points I have raised; (b) explicitly cleared for release to the general public as SI-authorized communications. In order to consider a response "SI-authorized and cleared for release," I will accept explicit communication from SI's Executive Director or from a majority of its Board of Directors endorsing the content in question. After 50,000 words, I may change my views and/or commit to reading more content, or (if I determine that the content is poor and is not using my time efficiently) I may decide not to engage further. SI-authorized content may improve or worsen SI's standing in my estimation, so unlike with comments, there is an incentive to select content that uses my time efficiently. Of course, SI-authorized content may end up including excerpts from comment responses to this post, and/or already-existing public content.

I may also change my views for other reasons, particularly if SI secures more impressive achievements and/or endorsements.

One more note: I believe I have read the vast majority of the Sequences, including the AI-foom debate, and that this content - while interesting and enjoyable - does not have much relevance for the arguments I've made.

Again: I think that whatever happens as a result of my post will be positive for SI's mission, whether or not it is positive for SI as an organization. I believe that most of SI's supporters and advocates care more about the former than about the latter, and that this attitude is far too rare in the nonprofit world.

Acknowledgements

Thanks to the following people for reviewing a draft of this post and providing thoughtful feedback (this of course does not mean they agree with the post or are responsible for its content): Dario Amodei, Nick Beckstead, Elie Hassenfeld, Alexander Kruel, Tim Ogden, John Salvatier, Jonah Sinick, Cari Tuna, Stephanie Wykstra.

1284 comments

Comments sorted by top scores.

comment by lukeprog · 2012-05-10T21:24:19.513Z · score: 62 (70 votes) · LW · GW

Update: My full response to Holden is now here.

As Holden said, I generally think that Holden's objections for SI "are either correct (especially re: past organizational competence) or incorrect but not addressed by SI in clear argumentative writing (this includes the part on 'tool' AI)," and we are working hard to fix both categories of issues.

In this comment I would merely like to argue for one small point: that the Singularity Institute is undergoing comprehensive changes — changes which I believe to be improvements that will help us to achieve our mission more efficiently and effectively.

Holden wrote:

I'm aware that SI has relatively new leadership that is attempting to address the issues behind some of my complaints. I have a generally positive impression of the new leadership; I believe the Executive Director and Development Director, in particular, to represent a step forward in terms of being interested in transparency and in testing their own general rationality. So I will not be surprised if there is some improvement in the coming years...

Louie Helm was hired as Director of Development in September 2011. I was hired as a Research Fellow that same month, and made Executive Director in November 2011. Below are some changes made since September. (Pardon the messy presentation: LW cannot correctly render tables in comments.)

SI before Sep. 2011: Very few peer-reviewed research publications.
SI today: More peer-reviewed publications coming in 2012 than in all past years combined. Additionally, I alone have a dozen papers in development, for which I am directing every step of research and writing, and will write the final draft, but am collaborating with remote researchers so as to put in only 5%-20% of the total hours required myself.

SI before Sep. 2011: No donor database / a very broken one.
SI today: A comprehensive donor database.

SI before Sep. 2011: Nearly all work performed directly by SI staff.
SI today: Most work outsourced to remote collaborators so that SI staff can focus on the things that only they can do.

SI before Sep. 2011: No strategic plan.
SI today: A strategic plan developed with input from all SI staff, and approved by the Board.

SI before Sep. 2011: Very little communication about what SI is doing.
SI today: Monthly progress reports, plus three Q&As with Luke about SI research and organizational development.

SI before Sep. 2011: No list of the research problems SI is working on.
SI today: A long, fully-referenced list of research problems SI is working on.

SI before Sep. 2011: Very little direct management of staff and projects.
SI today: Luke monitors all projects and staff work, and meets regularly with each staff member.

SI before Sep. 2011: Almost no detailed tracking of the expense of major SI projects (e.g. Summit, papers, etc.). The sole exception seems to be that Amy was tracking the costs of the 2011 Summit in NYC.
SI today: Detailed tracking of the expense of major SI projects for which this is possible (Luke has a folder in Google docs for these spreadsheets, and the summary spreadsheet is shared with the Board).

SI before Sep. 2011: No staff worklogs.
SI today: All staff members share their worklogs with Luke, Luke shares his worklog with all staff plus the Board.

SI before Sep. 2011: Best practices not followed for bookkeeping/accounting; accountant's recommendations ignored.
SI today: Meetings with consultants about bookkeeping/accounting; currently working with our accountant to implement best practices and find a good bookkeeper.

SI before Sep. 2011: Staff largely separated, many of them not well-connected to the others.
SI today: After a dozen or so staff dinners, staff much better connected, more of a team.

SI before Sep. 2011: Want to see the basics of AI Risk explained in plain language? Read The Sequences (more than a million words) or this academic book chapter by Yudkowsky.
SI today: Want to see the basics of AI Risk explained in plain language? Read Facing the Singularity (now in several languages, with more being added) or listen to the podcast version.

SI before Sep. 2011: Very few resources created to support others' research in AI risk.
SI today: IntelligenceExplosion.com, Friendly-AI.com, list of open problems in the field, with references, AI Risk Bibliography 2012, annotated list of journals that may publish papers on AI risk, a partial history of AI risk research, and a list of forthcoming and desired articles on AI risk.

SI before Sep. 2011: A hard-to-navigate website with much outdated content.
SI today: An entirely new website that is easier to navigate and has much new content (nearly complete; should launch in May or June).

SI before Sep. 2011: So little monitoring of funds that $118k was stolen in 2010 before SI noticed. (Note that we have won stipulated judgments to get much of this back, and have upcoming court dates to argue for stipulated judgments to get the rest back.)
SI today: Our bank accounts have been consolidated, with 3-4 people regularly checking over them.

SI before Sep. 2011: SI publications exported straight to PDF from Word or Google Docs, sometimes without even author names appearing.
SI today: All publications being converted into slick, useable LaTeX template (example), with all references checked and put into a central BibTeX file.

SI before Sep. 2011: No write-up of our major public technical breakthrough (TDT) using the mainstream format and vocabulary comprehensible to most researchers in the field (this is what we have at the moment).
SI today: Philosopher Rachael Briggs, whose papers on decision theory have been twice selected for the Philosopher's Annual, has been contracted to write an explanation of TDT and publish it in one of a select few leading philosophy journals.

SI before Sep. 2011: No explicit effort made toward efficient use of SEO or our (free) Google Adwords.
SI today: Highly optimized use of Google Adwords to direct traffic to our sites; currently working with SEO consultants to improve our SEO (of course, the new website will help).

(Just to be clear, I think this list shows not that "SI is looking really great!" but instead that "SI is rapidly improving and finally reaching a 'basic' level of organizational function.")

comment by lukeprog · 2012-05-11T02:54:28.896Z · score: 23 (23 votes) · LW · GW

...which is not to say, of course, that things were not improving before September 2011. It's just that the improvements have accelerated quite a bit since then.

For example, Amy was hired in December 2009 and is largely responsible for these improvements:

  • Built a "real" Board and officers; launched monthly Board meetings in February 2010.
  • Began compiling monthly financial reports in December 2010.
  • Began tracking Summit expenses and seeking Summit sponsors.
  • Played a major role in canceling many programs and expenses that were deemed low ROI.
comment by [deleted] · 2012-05-11T04:25:54.689Z · score: 9 (9 votes) · LW · GW

Our bank accounts have been consolidated, with 3-4 people regularly checking over them.

In addition to reviews, should SI implement a two-man rule for manipulating large quantities of money? (For example, over 5k, over 10k, etc.)

comment by ghf · 2012-05-11T22:38:10.640Z · score: 5 (9 votes) · LW · GW

My hope is that the upcoming deluge of publications will answer this objection, but for the moment, I am unclear as to the justification for the level of resources being given to SIAI researchers.

Additionally, I alone have a dozen papers in development, for which I am directing every step of research and writing, and will write the final draft, but am collaborating with remote researchers so as to put in only 5%-20% of the total hours required myself.

This level of freedom is the dream of every researcher on the planet. Yet, it's unclear why these resources should be devoted to your projects. While I strongly believe that the current academic system is broken, you are asking for a level of support granted to top researchers prior to have made any original breakthroughs yourself.

If you can convince people to give you that money, wonderful. But until you have made at least some serious advancement to demonstrate your case, donating seems like an act of faith.

It's impressive that you all have found a way to hack the system and get paid to develop yourselves as researchers outside of the academic system and I will be delighted to see that development bear fruit over the coming years. But, at present, I don't see evidence that the work being done justifies or requires that support.

comment by lukeprog · 2012-05-11T22:48:13.467Z · score: 8 (11 votes) · LW · GW

This level of freedom is the dream of every researcher on the planet. Yet, it's unclear why these resources should be devoted to your projects.

Because some people like my earlier papers and think I'm writing papers on the most important topic in the world?

It's impressive that you all have found a way to hack the system and get paid to develop yourselves as researchers outside of the academic system...

Note that this isn't uncommon. SI is far from the only think tank with researchers who publish in academic journals. Researchers at private companies do the same.

comment by ghf · 2012-05-11T23:15:03.978Z · score: 13 (13 votes) · LW · GW

First, let me say that, after re-reading, I think that my previous post came off as condescending/confrontational which was not my intent. I apologize.

Second, after thinking about this for a few minutes, I realized that some of the reason your papers seem so fluffy to me is that they argue what I consider to be obvious points. In my mind, of course we are likely "to develop human-level AI before 2100." Because of that, I may have tended to classify your work as outreach more than research.

But outreach is valuable. And, so that we can factor out the question of the independent contribution of your research, having people associated with SIAI with the publications/credibility to be treated as experts has gigantic benefits in terms of media multipliers (being the people who get called on for interviews, panels, etc). So, given that, I can see a strong argument for publication support being valuable to the overall organization goals regardless of any assessment of the value of the research.

Note that this isn't uncommon. SI is far from the only think tank with researchers who publish in academic journals. Researchers at private companies do the same.

My only point was that, in those situations, usually researchers are brought in with prior recognized achievements (or, unfortunately all too often, simply paper credentials). SIAI is bringing in people who are intelligent but unproven and giving them the resources reserved for top talent in academia or industry. As you've pointed out, one of the differences with SIAI is the lack of hoops to jump through.

Edit: I see you commented below that you view your own work as summarization of existing research and we agree on the value of that. Sorry that my slow typing speed left me behind the flow of the thread.

comment by Bugmaster · 2012-05-11T22:53:44.772Z · score: 2 (2 votes) · LW · GW

Researchers at private companies do the same.

It's true at my company, at least. There are quite a few papers out there authored by the researchers at the company where I work. There are several good business reasons for a company to invest time into publishing a paper; positive PR is one of them.

comment by metaphysicist · 2012-05-11T23:02:30.053Z · score: -5 (9 votes) · LW · GW

Because some people like my earlier papers and think I'm writing papers on the most important topic in the world?

But then you put your intellect at issue, and I think I'm entitled to opine that you lack the qualities of intellect that would make such recommendation credible. You're a budding scholar; a textbook writer at heart. You lack any of the originality of a thinker.

You confirm the lead poster's allegations that SIA staff are insular and conceited.

comment by lukeprog · 2012-05-11T23:11:55.421Z · score: 19 (19 votes) · LW · GW

I think I'm entitled to opine...

Of course you are. And, you may not be one of the people who "like my earlier papers."

You confirm the lead poster's allegations that SIA staff are insular and conceited.

Really? How? I commented earlier on LW (can't find it now) about how the kind of papers I write barely count as "original research" because for the most part they merely summarize and clarify the ideas of others. But as Beckstead says, there is a strong need for that right now.

For insights in decision theory and FAI theory, I suspect we'll have to look to somebody besides Luke Muehlhauser. We keep trying to hire such people but they keep saying "No." (I got two more "no"s just in the last 3 weeks.) Part of that may be due to the past and current state of the organization — and luckily, fixing that kind of thing is something I seem to have some skills with.

You're... a textbook writer at heart.

True, dat.

comment by [deleted] · 2012-05-11T08:18:32.654Z · score: 5 (7 votes) · LW · GW

I was hired as a Research Fellow that same month

Luke alone has a dozen papers in development

Why did you start referring to yourself in the first person and then change your mind? (Or am I missing something?)

comment by lukeprog · 2012-05-11T08:20:33.067Z · score: 9 (13 votes) · LW · GW

Brain fart: now fixed.

comment by [deleted] · 2012-05-11T08:27:14.993Z · score: 18 (20 votes) · LW · GW

(Why was this downvoted? If it's because the downvoter wants to see fewer brain farts, they're doing it wrong, because the message such a downvote actually conveys is that they want to see fewer acknowledgements of brain farts. Upvoted back to 0, anyway.)

comment by JoshuaFox · 2012-05-17T15:12:28.708Z · score: 4 (4 votes) · LW · GW

As a supporter and donor to SI since 2006, I can say that I had a lot of specific criticisms of the way that the organization was managed. The points Luke lists above were among them. I was surprised that on many occasions management did not realize the obvious problems and fix them.

But the current management is now recognizing many of these points and resolving them one by one, as Luke says. If this continues, SI's future looks good.

comment by siodine · 2012-05-11T13:35:22.323Z · score: 4 (6 votes) · LW · GW

Isn't this very strong evidence in support for Holden's point about "Apparent poorly grounded belief in SI's superior general rationality" (excluding Luke, at least)? And especially this?

comment by lukeprog · 2012-05-11T20:13:20.034Z · score: 18 (20 votes) · LW · GW

This topic is something I've been thinking about lately. Do SIers tend to have superior general rationality, or do we merely escape a few particular biases? Are we good at rationality, or just good at "far mode" rationality (aka philosophy)? Are we good at epistemic but not instrumental rationality? (Keep in mind, though, that rationality is only a ceteris paribus predictor of success.)

Or, pick a more specific comparison. Do SIers tend to be better at general rationality than someone who can keep a small business running for 5 years? Maybe the tight feedback loops of running a small business are better rationality training than "debiasing interventions" can hope to be.

Of course, different people are more or less rational in different domains, at different times, in different environments.

This isn't an idle question about labels. My estimate of the scope and level of people's rationality in part determines how much I update from their stated opinion on something. How much evidence for Hypothesis X (about organizational development) is it when Eliezer gives me his opinion on the matter, as opposed to when Louie gives me his opinion on the matter? When Person B proposes to take on a totally new kind of project, I think their general rationality is a predictor of success — so, what is their level of general rationality?

comment by Bugmaster · 2012-05-11T22:49:28.381Z · score: 2 (6 votes) · LW · GW

Are we good at epistemic but not instrumental rationality?

Holden implies (and I agree with him) that there's very little evidence at the moment to suggest that SI is good at instrumental rationality. As for epistemic rationality, how would we know ? Is there some objective way to measure it ? I personally happen to believe that if a person seems to take it as a given that he's great at epistemic rationality, this fact should count as evidence (however circumstantial) against him being great at epistemic rationality... but that's just me.

comment by TheOtherDave · 2012-05-11T21:10:55.204Z · score: 1 (1 votes) · LW · GW

If you accept that your estimate of someone's "rationality" should depend on the domain, the environment, the time, the context, etc... and what you want to do is make reliable estimates of the reliability of their opinion, their chances of success. etc... it seems to follow that you should be looking for comparisons within a relevant domain, environment, etc.

That is, if you want to get opinions about hypothesis X about organizational development that serve as significant evidence, it seems the thing to do is to find someone who knows a lot about organizational development -- ideally, someone who has been successful at developing organizations -- and consult their opinions. How generally rational they are might be very relevant causally, or it might not, but is in either case screened off by their domain competence... and their domain competence is easier to measure than their general rationality.

So is their general rationality worth devoting resources to determining?

It seems this only makes sense if you have already (e.g.) decided to ask Eliezer and Louie for their advice, whether it's good evidence or not, and now you need to know how much evidence it is, and you expect the correct answer is different from the answer you'd get by applying the metrics you know about (e.g., domain familiarity and previously demonstrated relevant expertise).

comment by lukeprog · 2012-05-11T21:55:52.957Z · score: 3 (3 votes) · LW · GW

I do spend a fair amount of time talking to domain experts outside of SI. The trouble is that the question of what we should do about thing X doesn't just depend on domain competence but also on thousands of details about the inner workings of SI and our mission that I cannot communicate to domain experts outside SI, but which Eliezer and Louie already possess.

comment by TheOtherDave · 2012-05-11T22:14:49.868Z · score: 4 (4 votes) · LW · GW

So it seems you have a problem in two domains (organizational development + SI internals) and different domain experts in both domains (outside domain experts + Eliezer/Louie), and need some way of cross-linking the two groups' expertise to get a coherent recommendation, and the brute-force solutions (e.g. get them all in a room together, or bring one group up to speed on the other's domain) are too expensive to be worth it. (Well, assuming the obstacle isn't that the details need to be kept secret, but simply that expecting an outsider to come up to speed on all of SI's local potentially relevant trivia simply isn't practical.)

Yes?

Yeah, that can be a problem.

In that position, for serious questions I would probably ask E/L for their recommendations and a list of the most relevant details that informed that decision, then go to outside experts with a summary of the competing recommendations and an expanded version of that list and ask for their input. If there's convergence, great. If there's divergence, iterate.

This is still a expensive approach, though, so I can see where a cheaper approximation for less important questions is worth having.

comment by lukeprog · 2012-05-11T22:18:53.862Z · score: 2 (2 votes) · LW · GW

Yes to all this.

comment by siodine · 2012-05-11T23:08:47.169Z · score: -1 (3 votes) · LW · GW

In the world in which a varied group of intelligent and especially rational people are organizing to literally save humanity, I don't see the relatively trivial, but important, improvements you've made in a short period of time being made because they were made years ago. And I thought that already accounting for the points you've made.

I mean, the question this group should be asking themselves is "how can we best alter the future so as to navigate towards FAI?" So, how did they apparently miss something like opportunity cost? Why, for instance, has their salaries increased when they could've been using it to improve the foundation of their cause from which everything else follows?

(Granted, I don't know the history and inner workings of the SI, and so I could be missing some very significant and immovable hurdles, but I don't see that as very likely; at least, not as likely as Holden's scenario.)

comment by lukeprog · 2012-05-11T23:18:25.605Z · score: 4 (4 votes) · LW · GW

I don't see the relatively trivial, but important, improvements you've made in a short period of time being made because they were made years ago. And I thought that already accounting for the points you've made.

I don't know what these sentences mean.

So, how did they apparently miss something like opportunity cost? Why, for instance, has their salaries increased when they could've been using it to improve the foundation of their cause from which everything else follows?

Actually, salary increases help with opportunity cost. At very low salaries, SI staff ends up spending lots of time and energy on general life cost-saving measures that distract us from working on x-risk reduction. And our salaries are generally still pretty low. I have less than $6k in my bank accounts. Outsourcing most tasks to remote collaborators also helps a lot with opportunity cost.

comment by siodine · 2012-05-12T00:01:50.875Z · score: 3 (3 votes) · LW · GW

I don't know what these sentences mean.

  • People are more rational in different domains, environments, and so on.
  • The people at SI may have poor instrumental rationality while being adept at epistemic rationality.
  • Being rational doesn't necessarily mean being successful.

I accept all those points, and yet I still see the Singularity Institute having made the improvements that you've made since being hired before you were hired if they have superior general rationality. That is, you wouldn't have that list of relatively trivial things to brag about because someone else would have recognized the items on that list as important and got them done somehow (ignore any negative connotations--they're not intended).

For instance, I don't see a varied group of people with superior general rationality not discovering or just not outsourcing work they don't have a comparative advantage in (i.e., what you've done). That doesn't look like just a failure in instrumental rationality, or just rationality operating on a different kind of utility function, or just a lack of domain specific knowledge.

The excuses available to a person acting in a way that's non-traditionally rational are less convincing when you apply them to a group.

Actually, salary increases help with opportunity cost. At very low salaries, SI staff ends up spending lots of time and energy on general life cost-saving measures that distract us from working on x-risk reduction. And our salaries are generally still pretty low. I have less than $6k in my bank accounts.

No, I get that. But that still doesn't explain away the higher salaries like EY's 80k/year and its past upwards trend. I mean, these higher paid people are the most committed to the cause, right? I don't see those people taking a higher salary when they could use that money for more outsourcing, or another employee, or better employees, if they want to literally save humanity while being superior in general rationality. It's like a homeless person desperately in want of shelter trying save enough for an apartment and yet buying meals at some restaurant.

Outsourcing most tasks to remote collaborators also helps a lot with opportunity cost.

That's the point I was making, why wasn't that done earlier? How did these people apparently miss out on opportunity cost? (And I'm just using outsourcing as an example because it was one of the most glaring changes you made that I think should have probably been made much earlier.)

comment by lukeprog · 2012-05-12T00:20:39.396Z · score: 4 (6 votes) · LW · GW

Right, I think we're saying the same thing, here: the availability of so much low-hanging fruit in organizational development as late as Sept. 2011 is some evidence against the general rationality of SIers. Eliezer seems to want to say it was all a matter of funding, but that doesn't make sense to me.

Now, on this:

I don't see those people taking a higher salary when they could use that money for more outsourcing, or another employee, or better employees, if they want to literally save humanity while being super in general rationality.

For some reason I'm having a hard time parsing your sentences for unambiguous meaning, but if I may attempt to rephrase: "SIers wouldn't take any salaries higher than (say) $70k/yr if they were truly committed to the cause and good in general rationality, because they would instead use that money to accomplish other things." Is that what you're saying?

comment by Rain · 2012-05-12T00:29:53.205Z · score: 3 (3 votes) · LW · GW

I've heard the Bay Area is expensive, and previously pointed out that Eliezer earns more than I do, despite me being in the top 10 SI donors.

I don't mind, though, as has been pointed out, even thinking about muffins might be a question invoking existential risk calculations.

comment by lukeprog · 2012-05-12T00:39:54.925Z · score: 4 (4 votes) · LW · GW

despite me being in the top 10 SI donors

...and much beloved for it.

Yes, the Bay Area is expensive. We've considered relocating, but on the other hand the (by far) best two places for meeting our needs in HR and in physically meeting with VIPs are SF and NYC, and if anything NYC is more expensive than the Bay Area. We cut living expenses where we can: most of us are just renting individual rooms.

Also, of course, it's not like the Board could decide we should relocate to a charter city in Honduras and then all our staff would be able to just up and relocate. :)

(Rain may know all this; I'm posting it for others' benefit.)

comment by komponisto · 2012-05-12T18:58:03.493Z · score: 12 (12 votes) · LW · GW

I think it's crucial that SI stay in the Bay Area. Being in a high-status place signals that the cause is important. If you think you're not taken seriously enough now, imagine if you were in Honduras...

Not to mention that HR is without doubt the single most important asset for SI. (Which is why it would probably be a good idea to pay more than the minimum cost of living.)

comment by TheOtherDave · 2012-05-12T01:31:59.697Z · score: 2 (2 votes) · LW · GW

Out of curiosity only: what were the most significant factors that led you to reject telepresence options?

comment by David_Gerard · 2012-05-12T18:02:44.485Z · score: 7 (7 votes) · LW · GW

FWIW, Wikimedia moved from Florida to San Francisco precisely for the immense value of being at the centre of things instead of the middle of nowhere (and yes, Tampa is the middle of nowhere for these purposes, even though it still has the primary data centre). Even paying local charity scale rather than commercial scale (there's a sort of cycle where WMF hires brilliant kids, they do a few years working at charity scale then go to Facebook/Google/etc for gobs of cash), being in the centre of things gets them staff and contacts they just couldn't get if they were still in Tampa. And yes, the question came up there pretty much the same as it's coming up here: why be there instead of remote? Because so much comes with being where things are actually happening, even if it doesn't look directly related to your mission (educational charity, AI research institute).

comment by komponisto · 2012-05-12T19:00:32.677Z · score: 0 (0 votes) · LW · GW

FWIW, Wikimedia moved from Florida to San Francisco

I didn't know this, but I'm happy to hear it.

comment by David_Gerard · 2012-05-12T19:04:58.762Z · score: 1 (1 votes) · LW · GW

The charity is still registered in Florida but the office is in SF. I can't find the discussion on a quick search, but all manner of places were under serious consideration - including the UK, which is a horrible choice for legal issues in so very many ways.

comment by lukeprog · 2012-05-12T01:56:50.756Z · score: 6 (8 votes) · LW · GW

In our experience, monkeys don't work that way. It sounds like it should work, and then it just... doesn't. Of course we do lots of Skyping, but regular human contact turns out to be pretty important.

comment by TheOtherDave · 2012-05-12T02:04:01.998Z · score: 8 (8 votes) · LW · GW

(nods) Yeah, that's been my experience too, though I've often suspected that companies like Google probably have a lot of research on the subject lying around that might be informative.

Some friends of mine did some experimenting along these lines when doing distributed software development (in both senses) and were somewhat startled to realize that Dark Age of Camelot worked better for them as a professional conferencing tool than any of the professional conferencing tools their company had. They didn't mention this to their management.

comment by David_Gerard · 2012-05-12T18:53:40.812Z · score: 1 (1 votes) · LW · GW

and were somewhat startled to realize that Dark Age of Camelot worked better for them as a professional conferencing tool than any of the professional conferencing tools their company had. They didn't mention this to their management.

I am reminded that Flickr started as a photo add-on for an MMORPG...

comment by HoverHell · 2012-05-13T08:16:45.010Z · score: 1 (1 votes) · LW · GW

-

comment by siodine · 2012-05-12T00:34:35.518Z · score: 0 (0 votes) · LW · GW

some evidence

Enough for you to agree with Holden on that point?

"SIers wouldn't take any salaries higher than (say) $70k/yr if they were truly committed to the cause and good in general rationality, because they would instead use that money to accomplish other things." Is that what you're saying?

Yes, but I wouldn't set a limit at a specific salary range; I'd expect them to give as much as they optimally could, because I assume they're more concerned with the cause than the money. (re the 70k/yr mention: I'd be surprised if that was anywhere near optimal)

comment by lukeprog · 2012-05-12T00:46:18.405Z · score: 2 (2 votes) · LW · GW

Enough for you to agree with Holden on that point?

Probably not. He and I continue to dialogue in private about the point, in part to find the source of our disagreement.

Yes, but I wouldn't set a limit at a specific salary range; I'd expect them to give as much as they optimally could, because I assume they're more concerned with the cause than the money. (re the 70k/yr mention: I'd be surprised if that was anywhere near optimal)

I believe everyone except Eliezer currently makes between $42k/yr and $48k/yr — pretty low for the cost of living in the Bay Area.

comment by siodine · 2012-05-12T01:37:39.562Z · score: 4 (4 votes) · LW · GW

Probably not. He and I continue to dialogue in private about the point, in part to find the source of our disagreement.

So, if you disagree with Holden, I assume you think SIers have superior general rationality: why?

And I'm confident SIers will score well on rationality tests, but that looks like specialized rationality. I.e., you can avoid a bias but you can't avoid a failure in your achieving your goals. To me, the SI approach seems poorly leveraged. I expect more significant returns from simple knowledge acquisition. E.g., you want to become successful? YOU WANT TO WIN?! Great, read these textbooks on microeconomics, finance, and business. I think this is more the approach you take anyway.

I believe everyone except Eliezer currently makes between $42k/yr and $48k/yr — pretty low for the cost of living in the Bay Area.

That isn't as bad as I thinking it was; I don't know if that's optimal, but it seems at least reasonable.

comment by lukeprog · 2012-05-12T01:47:10.564Z · score: 0 (0 votes) · LW · GW

I assume you think SIers have superior general rationality: why?

I'll avoid double-labor on this and wait to reply until my conversation with Holden is done.

I expect more significant returns from simple knowledge acquisition. E.g., you want to become successful? ...Great, read these textbooks on microeconomics, finance, and business. I think this is more the approach you take anyway.

Right. Exercise the neglected virtue of scholarship and all that.

comment by siodine · 2012-05-12T01:52:51.772Z · score: 1 (1 votes) · LW · GW

Right. Exercise the neglected virtue of scholarship and all that.

It's not that easy to dismiss; if it's as poorly leveraged as it looks relative to other approaches then you have little reason to be spreading and teaching SI's brand of specialized rationality (except for perhaps income).

comment by lukeprog · 2012-05-12T01:55:17.733Z · score: 3 (3 votes) · LW · GW

I'm not dismissing it, I'm endorsing it and agreeing with you that it has been my approach ever since my first post on LW.

comment by siodine · 2012-05-12T02:10:31.432Z · score: 0 (0 votes) · LW · GW

Weird, I have this perception of SI being heavily invested in overcoming biases and epistemic rationality training to the detriment of relevant domain specific knowledge, but I guess that's wrong?

comment by lukeprog · 2012-05-12T02:25:08.638Z · score: 2 (2 votes) · LW · GW

I'm lost again; I don't know what you're saying.

comment by siodine · 2012-05-12T15:58:27.848Z · score: 1 (1 votes) · LW · GW

I'm not dismissing it, I'm endorsing it and agreeing with you that it has been my approach ever since my first post on LW.

I wasn't talking about you; I was talking about SI's approach in spreading and training rationality. You(SI) have Yudkowsky writing books, you have rationality minicamps, you have lesswrong, you and others are writing rationality articles and researching the rationality literature, and so on.

That kind of rationality training, research, and message looks poorly leveraged in achieving your goals, is what I'm saying. Poorly leveraged for anyone trying to achieve goals. And at its most abstract, that's what rationality is, right? Achieving your goals.

So, I don't care if your approach was to acquire as much relevant knowledge as possible before dabbling in debiasing, bayes, and whatnot (i.e., prioritizing the most leveraged approach). I wondering why your approach doesn't seem to be SI's approach. I'm wondering why SI doesn't prioritize rationality training, research, and message by whatever is the most leveraged in achieving SI's goals. I'm wondering why SI doesn't spread the virtue of scholarship to the detriment of training debiasing and so on.

SI wants to raise the sanity waterline, is what the SI doing even near optimal for that? Knowing what SIers knew and trained for couldn't even get them to see an opportunity for trading in on opportunity cost for years; that is sad.

comment by komponisto · 2012-05-12T02:04:06.251Z · score: 2 (2 votes) · LW · GW

(Disclaimer: the following comment should not be taken to imply that I myself have concluded that SI staff salaries should be reduced.)

I believe everyone except Eliezer currently makes between $42k/yr and $48k/yr — pretty low for the cost of living in the Bay Area.

I'll grant you that it's pretty low relative to other Bay Area salaries. But as for the actual cost of living, I'm less sure.

I'm not fortunate enough to be a Bay Area resident myself, but here is what the internet tells me:

  • After taxes, a $48,000/yr gross salary in California equates to a net of around $3000/month.

  • A 1-bedroom apartment in Berkeley and nearby places can be rented for around $1500/month. (Presumably, this is the category of expense where most of the geography-dependent high cost of living is contained.)

  • If one assumes an average spending of $20/day on food (typically enough to have at least one of one's daily meals at a restaurant), that comes out to about $600/month.

  • That leaves around $900/month for miscellaneous expenses, which seems pretty comfortable for a young person with no dependents.

So, if these numbers are right, it seems that this salary range is actually right about what the cost of living is. Of course, this calculation specifically does not include costs relating to signaling (via things such as choices of housing, clothing, transportation, etc.) that one has more money than necessary to live (and therefore isn't low-status). Depending on the nature of their job, certain SI employees may need, or at least find it distinctly advantageous for their particular duties, to engage in such signaling.

comment by metaphysicist · 2012-05-12T00:55:14.888Z · score: -3 (13 votes) · LW · GW

I believe everyone except Eliezer currently makes between $42k/yr and $48k/yr — pretty low for the cost of living in the Bay Area.

Damn good for someone just out of college—without a degree!

comment by lukeprog · 2012-05-12T01:03:44.321Z · score: 7 (9 votes) · LW · GW

The point is that we're consequentialists, and lowering salaries even further would save money (on salaries) but result in SI getting less done, not more — for the same reason that outsourcing fewer tasks would save money (on outsourcing) but cause us to get less done, not more.

comment by steven0461 · 2012-05-12T02:19:10.664Z · score: 6 (8 votes) · LW · GW

result in SI getting less done

You say this as though it's obvious, but if I'm not mistaken, salaries used to be about 40% of what they are now, and while the higher salaries sound like they are making a major productivity difference, hiring 2.5 times as many people would also make a major productivity difference. (Though yes, obviously marginal hires would be lower in quality.)

comment by lukeprog · 2012-05-12T02:34:02.985Z · score: 9 (19 votes) · LW · GW

I don't think salaries were ever as low as 40% of what they are now. When I came on board, most people were at $36k/yr.

To illustrate why lower salaries means less stuff gets done: I've been averaging 60 hours per week, and I'm unusually productive. If I am paid less, that means that (to pick just one example from this week) I can't afford to take a taxi to and from the eye doctor, which means I spend 1.5 hrs each way changing buses to get there, and spend less time being productive on x-risk. That is totally not worth it. Future civilizations would look back on this decision as profoundly stupid.

comment by Will_Newsome · 2012-05-14T04:08:06.343Z · score: 13 (17 votes) · LW · GW

Pretty sure Anna and Steve Rayhawk had salaries around $20k/yr at some point while living in Silicon Valley.

I don't think that you're really responding to Steven's point. Yes, as Steven said, if you were paid less then clearly that would impose more costs on you, so ceteris paribus your getting paid less would be bad. But, as Steven said, the opportunity cost is potentially very high. You haven't made a rationally compelling case that the missed opportunity is "totally not worth it" or that heeding it would be "profoundly stupid", you've mostly just re-asserted your conclusion, contra Steven's objection. What are your arguments that this is the case? Note that I personally think it's highly plausible that $40-50k/yr is optimal, but as far as I can see you haven't yet listed any rationally compelling reasons to think so.

(This comment is a little bit sterner than it would have been if you hadn't emphatically asserted that conclusions other than your own would be "profoundly stupid" without first giving overwhelming justification for your conclusion. It is especially important to be careful about such apparent overconfidence on issues where one clearly has a personal stake in the matter.)

comment by steven0461 · 2012-05-14T20:36:00.231Z · score: 6 (6 votes) · LW · GW

I will largely endorse Will's comment, then bow out of the discussion, because this appears to be too personal and touchy a topic for a detailed discussion to be fruitful.

comment by lukeprog · 2012-05-14T08:05:24.761Z · score: 5 (13 votes) · LW · GW

Pretty sure Anna and Steve Rayhawk had salaries around $20k/yr at some point while living in Silicon Valley.

If so, I suspect they were burning through savings during this time or had some kind of cheap living arrangement that I don't have.

What are your arguments that [paying you less wouldn't be worth it]?

  1. I couldn't really get by on less, so paying me less would cause me to quit the organization and do something else instead, which would cause much of this good stuff to probably not happen.

  2. It's VERY hard for SingInst to purchase value as efficiently as by purchasing Luke-hours. At $48k/yr for 60 hrs/wk, I make $15.38/hr, and one Luke-hour is unusually productive for SingInst. Paying me less and thereby causing me to work fewer hours per week is a bad value proposition for SingInst.

Or, as Eliezer put it:

paying me less would require me to do things that take up time and energy in order to get by with a smaller income. Then, assuming all goes well, future intergalactic civilizations would look back and think this was incredibly stupid; in much the same way that letting billions of person-containing brains rot in graves, and humanity allocating less than a million dollars per year to the Singularity Institute, would predictably look pretty stupid in retrospect. At Singularity Institute board meetings we at least try not to do things which will predictably make future intergalactic civilizations think we were being willfully stupid. That's all there is to it, and no more.

comment by Paul Crowley (ciphergoth) · 2012-05-14T09:36:16.138Z · score: 11 (13 votes) · LW · GW

This seems to me unnecessarily defensive. I support the goals of SingInst, but I could never bring myself to accept the kind of salary cut you guys are taking in order to work there. Like every other human on the planet, I can't be accurately modelled with a utility function that places any value on far distant strangers; you can more accurately model what stranger-altruism I do show as purchase of moral satisfaction, though I do seek for such altruism to be efficient. SingInst should pay the salaries it needs to pay to recruit the kind of staff it needs to fulfil its mission; it's harder to recruit if staff are expected to be defensive about demanding market salaries for their expertise, with no more than a normal adjustment for altruistic work much as if they were working for an animal sanctuary.

comment by lukeprog · 2012-05-14T09:48:39.259Z · score: 1 (7 votes) · LW · GW

I could never bring myself to accept the... salary cut you guys are taking in order to work [at SI]... SingInst should pay the salaries it needs to pay to recruit the kind of staff it needs to fulfill its mission; it's harder to recruit if staff are expected to be defensive about demanding market salaries for their expertise...

Yes, exactly.

comment by Paul Crowley (ciphergoth) · 2012-05-14T10:07:05.931Z · score: 4 (8 votes) · LW · GW

So when I say "unnecessarily defensive", I mean that all the stuff about the cost of taxis is after-the-fact defensive rationalization; it can't be said about a single dollar you spend on having a life outside of SI. The truth is that even the best human rationalist in the world isn't going to agree to giving those up, and since you have to recruit humans, you'd best pay the sort of salary that is going to attract and retain them. That of course includes yourself.

The same goes for saying "move to the Honduras". Your perfectly utility-maximising AGIs will move to the Honduras, but your human staff won't; they want to live in places like the Bay Area.

comment by [deleted] · 2012-05-14T18:55:37.425Z · score: -1 (1 votes) · LW · GW

I could never bring myself to accept the kind of salary cut you guys are taking in order to work there

You know that the Bay Area is freakin' expensive, right?

comment by Paul Crowley (ciphergoth) · 2012-05-15T06:15:11.274Z · score: 5 (5 votes) · LW · GW

Re-reading, the whole thing is pretty unclear!

As katydee and thomblake say, I mean that working for SingInst would mean a bigger reduction in my salary than I could currently bring myself to accept. If I really valued the lives of strangers as a utilitarian, the benefits to them of taking a salary cut would be so huge that it would totally outweigh the costs to me. But it looks like I only really place direct value on the short-term interests of myself and those close to me, and everything else is purchase of moral satisfaction. Happily, purchase of moral satisfaction can still save the world if it is done efficiently.

Since the labour pool contains only human beings, with no true altruistic utility maximizers, SingInst should hire and pay accordingly; the market shows that people will accept a lower salary for a job that directly does good, but not a vastly lower salary. It would increase SI-utility if Luke accepted a lower salary, but it wouldn't increase Luke-utility, and driving Luke away would cost a lot of SI-utility, so calling for it is in the end a cheap shot and a bad recommendation.

I live in London, which is also freaking expensive - but so are all the places I want to live. There's a reason people are prepared to pay more to live in these places.

comment by Eugine_Nier · 2012-05-15T04:28:36.122Z · score: 3 (3 votes) · LW · GW

So why not relocate SIAI somewhere with a more reasonable cost of living?

comment by katydee · 2012-05-15T04:50:47.056Z · score: 3 (3 votes) · LW · GW

I think the standard answer is that the networking and tech industry connections available in the Bay Area are useful enough to SIAI to justify the high costs of operating there.

comment by TraderJoe · 2012-05-17T15:20:35.401Z · score: 0 (0 votes) · LW · GW

[comment deleted]

comment by thomblake · 2012-05-14T19:12:50.224Z · score: 2 (2 votes) · LW · GW

Hmm... Perhaps you don't know that "salary cut" above means taking much less money?

comment by [deleted] · 2012-05-15T12:00:54.296Z · score: 1 (1 votes) · LW · GW

I had missed the word cut. Damn it, I shouldn't be commenting while sleep-deprived!

comment by Paul Crowley (ciphergoth) · 2012-05-15T06:22:10.410Z · score: 0 (0 votes) · LW · GW

Indeed. I guess "taking a cut" can sometimes mean "taking some of the money", so you could interpret this as meaning "I couldn't accept all that money", which as you say is the opposite of what I meant!

comment by katydee · 2012-05-14T19:10:01.759Z · score: 1 (1 votes) · LW · GW

Perhaps that's why he's saying he wouldn't be willing to live there on a low salary?

comment by Bugmaster · 2012-05-14T22:37:02.400Z · score: 1 (5 votes) · LW · GW

I understand the point you're making regarding salaries, and for once I agree.

However, it's rather presumptuous of you (and/or Eliezer) to assume, implicitly, that our choices are limited to only two possibilities: "Support SIAI, save the world", and "Don't support SIAI, the world is doomed". I can envision many other scenarios, such as "Support SIAI, but their fears were overblown and you implicitly killed N children by not spending the money on them instead", or "Don't support SIAI, support some other organization instead because they'll have a better chance of success", etc.

comment by lukeprog · 2012-05-15T21:45:26.017Z · score: 4 (6 votes) · LW · GW

Where did we say all that?

comment by Bugmaster · 2012-05-15T22:01:31.542Z · score: 2 (6 votes) · LW · GW

In your comment above, you said:

...I can't afford to take a taxi to and from the eye doctor, which means I spend 1.5 hrs each way changing buses to get there, and spend less time being productive on x-risk. That is totally not worth it. Future civilizations would look back on this decision as profoundly stupid.

You also quoted Eliezer saying something similar.

This outlook implies strongly that whatever SIAI is doing is of such monumental significance that future civilizations will not only remember its name, but also reverently preserve every decision it made. You are also quite fond of saying that the work that SIAI is doing is tantamount to "saving the world"; and IIRC Eliezer once said that, if you have a talent for investment banking, you should make as much money as possible and then donate it all to SIAI, as opposed to any other charity.

This kind of grand rhetoric presupposes not only that the SIAI is correct in its risk assessment regarding AGI, but also that they are uniquely qualified to address this potentially world-ending problem, and that, over the ages, no one more qualified could possibly come along. All of this could be true, but it's far from a certainty, as your writing would seem to imply.

comment by jacob_cannell · 2012-05-16T09:43:41.718Z · score: 1 (1 votes) · LW · GW

This outlook implies strongly that whatever SIAI is doing is of such monumental significance that future civilizations will not only remember its name, but also reverently preserve every decision it made.

In the SIA/Transhumanist outlook, if civilization survives some large (perhaps majority) of extant human minds will survive as uploads. As a result, all of their memories will likely be stored, dissected, shared, searched, judged, and so on. Much will be preserved in such a future. And even without uploading, there are plenty of people who have maintained websites since the early days of the internet with no loss of information, and this is quite likely to remain true far into the future if civilization survives.

comment by Paul Crowley (ciphergoth) · 2012-05-16T09:29:13.117Z · score: 1 (1 votes) · LW · GW

Well no, of course it's not a certainty. All efforts to make a difference are decisions under uncertainty. You're attacking a straw man.

comment by Bugmaster · 2012-05-16T21:06:46.607Z · score: 1 (3 votes) · LW · GW

Please substitute "certainty minus epsilon" for "certainty" wherever you see it in my post. It was not my intention to imply 100% certainty; just a confidence value so high that it amounts to the same thing for all practical purposes.

comment by dlthomas · 2012-05-16T21:34:18.450Z · score: 1 (1 votes) · LW · GW

I don't think "certainty minus epsilon" improves much. It moves it from theoretical impossibility to practical - but looking that far out, I expect "likelihood" might be best.

comment by Bugmaster · 2012-05-17T01:49:45.006Z · score: 1 (1 votes) · LW · GW

I don't understand your comment... what's the practical difference between "extremely high likelihood" and "extremely high certainty" ?

comment by Paul Crowley (ciphergoth) · 2012-05-17T05:35:24.636Z · score: 0 (0 votes) · LW · GW

And where do SI claim even that? Obviously some of their discussions are implicitly conditioned on the fundamental assumptions behind their mission being true, but that doesn't mean that they have extremely high confidence in those assumptions.

comment by lukeprog · 2012-05-15T23:59:22.356Z · score: 1 (5 votes) · LW · GW

I'm not seeing how the above implies the thing you said:

[You assume] our choices are limited to only two possibilities: "Support SIAI, save the world", and "Don't support SIAI, the world is doomed".

(Note that I don't necessarily endorse things you report Eliezer as having said.)

comment by Bugmaster · 2012-05-16T21:21:09.943Z · score: 1 (1 votes) · LW · GW

You appear to be very confident that future civilizations will remember SIAI in a positive way, and care about its actions. If so, they must have some reason for doing so. Any reason would do, but the most likely reason is that SIAI will accomplish something so spectacularly beneficial that it will affect everyone in the far future. SIAI's core mission is to save the world from UFAI, so it's reasonable to assume that this is the highly beneficial effect that the SIAI will achieve.

I don't have a problem with this chain of events, just with your apparent confidence that a). it's going to happen in exactly that way, and b). your organization is the only one who is qualified to save the world in this specific fashion.

(EDIT: I forgot to say that, if we follow your reasoning to its conclusion, then you are indeed implying that donating as much money or labor as possible to SIAI is the only smart move for any rational agent.)

Note that I have no problem with your main statement, i.e. "lowering the salaries of SIAI members would bring us too much negative utility to compensate for the monetary savings". This kind of cost-benefit analysis is done all the time, and future civilizations rarely enter into it.

comment by SexyBayes · 2012-05-17T14:11:35.157Z · score: -4 (20 votes) · LW · GW

"1. I couldn't really get by on less"

It is called a budget, son.

Plenty of people make less than you and work harder than you. Look in every major city and you will find plenty of people that fit this category, both in business and labor.

"That is totally not worth it. Future civilizations would look back on this decision as profoundly stupid."

Elitism plus demanding that you don't have to budget. Seems that you need to work more and focus less on how "awesome" you are.

You make good contributions...but let's not get carried away.

If you really cared about future risk you would be working away at the problem even with a smaller salary. Focus on your work.

comment by Rain · 2012-05-17T14:13:06.785Z · score: 16 (24 votes) · LW · GW

If you really cared about future risk you would be working away at the problem even with a smaller salary. Focus on your work.

What we really need is some kind of emotionless robot who doesn't care about its own standard of living and who can do lots of research and run organizations and suchlike without all the pesky problems introduced by "being human".

Oh, wait...

comment by Cyan · 2012-05-17T14:22:43.137Z · score: -1 (5 votes) · LW · GW

If you really cared about future risk you would be working away at the problem even with a smaller salary. Focus on your work.

Downvoted for this; Rain's reply to the parent goes for me too.

comment by katydee · 2012-05-14T19:15:02.709Z · score: 3 (3 votes) · LW · GW

That's not actually that good, I don't think-- I go to a good college, and I know many people who are graduating to 60k-80k+ jobs with recruitment bonuses, opportunities for swift advancement, etc. Some of the best people I know could literally drop out now (three or four weeks prior to graduation) and immediately begin making six figures.

SIAI wages certainly seem fairly low to me relative to the quality of the people they are seeking to attract, though I think there are other benefits to working for them that cause the organization to attract skillful people regardless.

comment by shminux · 2012-05-14T19:34:11.686Z · score: 1 (1 votes) · LW · GW

A Dilbert comic said it.

comment by katydee · 2012-05-14T19:52:15.356Z · score: 2 (2 votes) · LW · GW

Ouch. I'd like to think that the side benefits for working for SIAI outweigh the side benefits for working for whatever soulless corporation Dilbert's workplace embodies, though there is certainly a difference between side benefits and actual monetary compensation.

comment by Davorak · 2012-05-18T18:06:18.945Z · score: 0 (0 votes) · LW · GW

I graduated ~5 years ago with a engineering degree from a first tier University and I would have consider those starting salaries to be low to decent and not high. This is especially true in places with high cost of living like the bay area.

Having a good internship durring college often ment starting out at 60k/yr if not higher.

If this is significantly different for engineers exiting first tier University now it would be interesting to know.

comment by komponisto · 2012-05-12T00:47:08.744Z · score: 2 (4 votes) · LW · GW

Many of your sentences are confusing because you repeatedly use the locution "I see X"/ "I don't see X" in a nonstandard way, apparently to mean "X would have happened" /"X would not have happened".

This is not the way that phrase is usually understood. Normally, "I see X" is taken to mean either "I observe X" or "I predict X". For example I might say (if I were so inclined):

Unlike you, I see a lot of rationality being demonstrated by SI employees.

meaning that I believe (from my observation) they are in fact being rational. Or, I might say:

I don't see Luke quitting his job at SI tomorrow to become a punk rocker.

meaning that I don't predict that will happen. But I would not generally say:

* I don't see these people taking a higher salary.

if what I mean is "these people should/would not have taken a higher salary [if such-and-such were true]".

comment by siodine · 2012-05-12T01:04:35.362Z · score: 2 (2 votes) · LW · GW

Oh, I see ;) Thanks. I'll definitely act on your comment, but I was using "I see X" as "I predict X"--just in the context of a possible world. E.g., I predict in the possible world in which SIers are superior in general rationality and committed to their cause, Luke wouldn't have that list of accomplishments. Or, "yet I still see the Singularity Institute having made the improvements..."

I now see that I've been using 'see' as syntactic sugar for counterfactual talk... but no more!

comment by komponisto · 2012-05-12T01:21:01.044Z · score: 2 (2 votes) · LW · GW

I was using "I see X" as "I predict X"--just in the context of a possible world.

To get away with this, you really need, at minimum, an explicit counterfactual clause ("if", "unless", etc.) to introduce it: "In a world where SIers are superior in general rationality, I don't see Luke having that list of accomplishments."

The problem was not so much that your usage itself was logically inconceivable, but rather that it collided with the other interpretations of "I see X" in the particular contexts in which it occurred. E.g. "I don't see them taking higher salaries" sounded like you were saying that they weren't taking higher salaries. (There was an "if" clause, but it came way too late!)

comment by Rain · 2012-05-12T00:10:54.034Z · score: 2 (2 votes) · LW · GW

To summarize and rephrase: in a "counterfactual" world where SI was actually rational, they would have found all these solutions and done all these things long ago.

comment by [deleted] · 2012-05-16T19:23:21.282Z · score: 1 (1 votes) · LW · GW

Have you considered the possibility that even higher salaries might raise productivity further?

I think we should search systematically for ways to convert money into increased productivity.

comment by [deleted] · 2012-05-12T07:19:52.243Z · score: -1 (9 votes) · LW · GW

And our salaries are generally still pretty low.

By what measure do you figure that?

I have less than $6k in my bank accounts.

That might be informative if we knew anything about your budget, but without any sort of context it sounds purely obfuscatory. (Also, your bank account is pretty close to my annual salary, so you might want to consider what you're actually signalling here and to whom.)

comment by lessdazed · 2012-05-31T05:54:35.797Z · score: 0 (0 votes) · LW · GW

Apparent poorly grounded belief in SI's superior general rationality

I found this complaint insufficiently detailed and not well worded.

Average people think their rationality is moderately good. Average people are not very rational. SI affiliated people think they are adept or at least adequate at rationality. SI affiliated people are not complete disasters at rationality.

SI affiliated people are vastly superior to others in generally rationality. So the original complaint literally interpreted is false.

An interesting question might be on the level of: "Do SI affiliates have rationality superior to what the average person falsely believes his or her rationality is?"

Holden's complaints each have their apparent legitimacy change differently under his and my beliefs. Some have to do with overconfidence or incorrect self-assessment, others with other-assessment, others with comparing SI people to others. Some of them:

Insufficient self-skepticism given how strong its claims are

Largely agree, as this relates to overconfidence.

...and how little support its claims have won.

Moderately disagree, as this relies on the rationality of others.

Being too selective (in terms of looking for people who share its preconceptions) when determining whom to hire and whose feedback to take seriously.

Largely disagree, as this relies significantly on the competence of others.

Paying insufficient attention to the limitations of the confidence one can have in one's untested theories, in line with my Objection 1.

Largely agree, as this depends more on accurate assessment of one's on rationality.

Rather than endorsing "Others have not accepted our arguments, so we will sharpen and/or reexamine our arguments," SI seems often to endorse something more like "Others have not accepted their arguments because they have inferior general rationality," a stance less likely to lead to improvement on SI's part.

There is instrumental value in falsely believing others to have a good basis for disagreement so one's search for reasons one might be wrong is enhanced. This is aside from the actual reasons of others.

It is easy to imagine an expert in a relevant field objecting to SI based on something SI does or says seeming wrong, only to have the expert couch the objection in literally false terms, perhaps ones that flow from motivated cognition and bear no trace of the real, relevant reason for the objection. This could be followed by SI's evaluation and dismissal of it and failure of a type not actually predicted by the expert...all such nuances are lost in the literally false "Apparent poorly grounded belief in SI's superior general rationality."

Such a failure comes to mind and is easy for me to imagine as I think this is a major reason why "Lack of impressive endorsements" is a problem. The reasons provided by experts for disagreeing with SI on particular issues are often terrible, but such expressions are merely what they believe their objections to be, and their expertise is in math or some such, not in knowing why they think what they think.

comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2012-05-11T05:00:20.014Z · score: 4 (22 votes) · LW · GW

And note that these improvements would not and could not have happened without more funding than the level of previous years - if, say, everyone had been waiting to see these kinds of improvements before funding.

comment by lukeprog · 2012-05-11T08:13:02.053Z · score: 54 (56 votes) · LW · GW

note that these improvements would not and could not have happened without more funding than the level of previous years

Really? That's not obvious to me. Of course you've been around for all this and I haven't, but here's what I'm seeing from my vantage point...

Recent changes that cost very little:

  • Donor database
  • Strategic plan
  • Monthly progress reports
  • A list of research problems SI is working on (it took me 16 hours to write)
  • IntelligenceExplosion.com, Friendly-AI.com, AI Risk Bibliography 2012, annotated list of journals that may publish papers on AI risk, a partial history of AI risk research, and a list of forthcoming and desired articles on AI risk (each of these took me only 10-25 hours to create)
  • Detailed tracking of the expenses for major SI projects
  • Staff worklogs
  • Staff dinners (or something that brought staff together)
  • A few people keeping their eyes on SI's funds so theft would be caught sooner
  • Optimization of Google Adwords

Stuff that costs less than some other things SI had spent money on, such as funding Ben Goertzel's AGI research or renting downtown Berkeley apartments for the later visiting fellows:

  • Research papers
  • Management of staff and projects
  • Rachael Briggs' TDT write-up
  • Best-practices bookkeeping/accounting
  • New website
  • LaTeX template for SI publications; references checked and then organized with BibTeX
  • SEO

Do you disagree with these estimates, or have I misunderstood what you're claiming?

comment by David_Gerard · 2012-05-12T18:37:08.031Z · score: 20 (20 votes) · LW · GW

A lot of charities go through this pattern before they finally work out how to transition from a board-run/individual-run tax-deductible band of conspirators to being a professional staff-run organisation tuned to doing the particular thing they do. The changes required seem simple and obvious in hindsight, but it's a common pattern for it to take years, so SIAI has been quite normal, or at the very least not been unusually dumb.

(My evidence is seeing this pattern close-up in the Wikimedia Foundation, Wikimedia UK (the first attempt at which died before managing it, the second making it through barely) and the West Australian Music Industry Association, and anecdotal evidence from others. Everyone involved always feels stupid at having taken years to achieve the retrospectively obvious. I would be surprised if this aspect of the dynamics of nonprofits had not been studied.)

edit: Luke's recommendation of The Nonprofit Kit For Dummies looks like precisely the book all the examples I know of needed to have someone throw at them before they even thought of forming an organisation to do whatever it is they wanted to achieve.

comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2012-05-12T04:04:19.165Z · score: 18 (26 votes) · LW · GW

Things that cost money:

  • Amy Willey
  • Luke Muehlhauser
  • Louie Helm
  • CfAR
  • trying things until something worked
comment by lukeprog · 2012-05-14T10:07:06.118Z · score: 69 (70 votes) · LW · GW

I don't think this response supports your claim that these improvements "would not and could not have happened without more funding than the level of previous years."

I know your comment is very brief because you're busy at minicamp, but I'll reply to what you wrote, anyway: Someone of decent rationality doesn't just "try things until something works." Moreover, many of the things on the list of recent improvements don't require an Amy, a Luke, or a Louie.

I don't even have past management experience. As you may recall, I had significant ambiguity aversion about the prospect of being made Executive Director, but as it turned out, the solution to almost every problem X has been (1) read what the experts say about how to solve X, (2) consult with people who care about your mission and have solved X before, and (3) do what they say.

When I was made Executive Director and phoned our Advisors, most of them said "Oh, how nice to hear from you! Nobody from SingInst has ever asked me for advice before!"

That is the kind of thing that makes me want to say that SingInst has "tested every method except the method of trying."

Donor database, strategic plan, staff worklogs, bringing staff together, expenses tracking, funds monitoring, basic management, best-practices accounting/bookkeeping... these are all literally from the Nonprofits for Dummies book.

Maybe these things weren't done for 11 years because SI's decision-makers did make good plans but failed to execute them due to the usual defeaters. But that's not the history I've heard, except that some funds monitoring was insisted upon after the large theft, and a donor database was sorta-kinda-not-really attempted at one point. The history I've heard is that SI failed to make these kinds of plans in the first place, failed to ask advisors for advice, failed to read Nonprofits for Dummies, and so on.

Money wasn't the barrier to doing many of those things, it was a gap in general rationality.

I will agree, however, that what is needed now is more money. We are rapidly becoming a more robust and efficient and rational organization, stepping up our FAI team recruiting efforts, stepping up our transparency and accountability efforts, and stepping up our research efforts, and all those things cost money.

At the risk of being too harsh… When I began to intern with the Singularity Institute in April 2011, I felt uncomfortable suggesting that people donate to SingInst, because I could see it from the inside and it wasn't pretty. (And I'm not the only SIer who felt this way at the time.)

But now I do feel comfortable asking people to donate to SingInst. I'm excited about our trajectory and our team, and if we can raise enough support then we might just have a shot at winning after all.

comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2012-05-21T04:29:45.237Z · score: 32 (32 votes) · LW · GW

Luke has just told me (personal conversation) that what he got from my comment was, "SIAI's difficulties were just due to lack of funding" which was not what I was trying to say at all. What I was trying to convey was more like, "I didn't have the ability to run this organization, and knew this - people who I hoped would be able to run the organization, while I tried to produce in other areas (e.g. turning my back on everything else to get a year of FAI work done with Marcello or writing the Sequences) didn't succeed in doing so either - and the only reason we could hang on long enough to hire Luke was that the funding was available nonetheless and in sufficient quantity that we could afford to take risks like paying Luke to stay on for a while, well before we knew he would become Executive Director".

comment by Will_Sawin · 2012-06-12T05:23:10.888Z · score: 1 (1 votes) · LW · GW

Does Luke disagree with this clarified point? I do not find a clear indicator in this conversation.

comment by lukeprog · 2013-08-28T19:40:42.474Z · score: 10 (10 votes) · LW · GW

Update: I came out of a recent conversation with Eliezer with a higher opinion of Eliezer's general rationality, because several things that had previously looked to me like unforced, forseeable mistakes by Eliezer now look to me more like non-mistakes or not-so-forseeable mistakes.

comment by MarkusRamikin · 2012-05-14T15:41:32.235Z · score: 28 (28 votes) · LW · GW

You're allowed to say these things on the public Internet?

I just fell in love with SI.

comment by lukeprog · 2012-05-26T00:33:50.957Z · score: 25 (22 votes) · LW · GW

You're allowed to say these things on the public Internet?

Well, at our most recent board meeting I wasn't fired, reprimanded, or even questioned for making these comments, so I guess I am. :)

comment by MarkusRamikin · 2012-05-26T06:00:43.342Z · score: 0 (0 votes) · LW · GW

Not even funny looks? ;)

comment by shminux · 2012-05-14T18:04:43.800Z · score: 18 (22 votes) · LW · GW

I just fell in love with SI.

It's Luke you should have fallen in love with, since he is the one turning things around.

comment by wedrifid · 2012-05-26T02:24:14.790Z · score: 43 (43 votes) · LW · GW

It's Luke you should have fallen in love with, since he is the one turning things around.

On the other hand I can count with one hand the number of established organisations I know of that would be sociologically capable of ceding power, status and control to Luke the way SingInst did. They took an untrained intern with essentially zero external status from past achievements and affiliations and basically decided to let him run the show (at least in terms of publicly visible initiatives). It is clearly the right thing for SingInst to do and admittedly Luke is very tall and has good hair which generally gives a boost when it comes to such selections - but still, making the appointment goes fundamentally against normal human behavior.

(Where I say "count with one hand" I am not including the use of any digits thereupon. I mean one.)

comment by Matt_Simpson · 2012-07-19T19:05:00.965Z · score: 7 (7 votes) · LW · GW

...and admittedly Luke is very tall and has good hair which generally gives a boost when it comes to such selections...

It doesn't matter that I completely understand why this phrase was included, I still found it hilarious in a network sitcom sort of way.

comment by [deleted] · 2012-05-14T19:58:32.512Z · score: 0 (16 votes) · LW · GW

Consider the implications in light of the HoldenKarnofsky's critique about SI pretensions to high rationality.

  1. Rationality is winning.

  2. SI, at the same time as it was claiming extraordinary rationality, was behaving in ways that were blatantly irrational.

  3. Although this is supposedly due to "the usual causes," rationality (winning) subsumes overcoming akrasia.

  4. HoldenKarnofsky is correct that SI made claims for its own extraordinary rationality at a time when its leaders weren't rational.

  5. Further: why should anyone give SI credibility today—when it stands convicted of self-serving misrepresentation in the recent past?

comment by Paul Crowley (ciphergoth) · 2012-05-15T06:26:06.711Z · score: 5 (5 votes) · LW · GW

You've misread the post - Luke is saying that he doesn't think the "usual defeaters" are the most likely explanation.

comment by lukeprog · 2012-05-25T17:42:34.437Z · score: 3 (3 votes) · LW · GW

Correct.

comment by thomblake · 2012-05-14T20:03:44.715Z · score: 5 (7 votes) · LW · GW

As a minor note, observe that claims of extraordinary rationality do not necessarily contradict claims of irrationality. The sanity waterline is very low.

comment by TheOtherDave · 2012-05-14T21:12:55.166Z · score: 5 (5 votes) · LW · GW

Do you mean to imply in context here that the organizational management of SIAI at the time under discussion was above average for a nonprofit organization? Or are you just making a more general statement that a system can be irrational while demonstrating above average rationality? I certainly agree with the latter.

comment by Paul Crowley (ciphergoth) · 2012-05-15T06:30:46.018Z · score: 9 (9 votes) · LW · GW

Are you comparing it to the average among nonprofits started, or nonprofits extant? I would guess that it was well below average for extant nonprofits, but about or slightly above average for started nonprofits. I'd guess that most nonprofits are started by people who don't know what they're doing and don't know what they don't know, and that SI probably did slightly better because the people who were being a bit stupid were at least very smart, which can help. However, I'd guess that most such nonprofits don't live long because they don't find a Peter Thiel to keep them alive.

comment by David_Gerard · 2012-05-16T11:07:48.227Z · score: 7 (7 votes) · LW · GW

Your assessment looks about right to me. I have considerable experience of averagely-incompetent nonprofits, and SIAI looks normal to me. I am strongly tempted to grab that "For Dummies" book and, if it's good, start sending copies to people ...

comment by TheOtherDave · 2012-05-15T12:44:48.271Z · score: 0 (0 votes) · LW · GW

In the context of thomblake's comment, I suppose nonprofits started is the proper reference class.

comment by private_messaging · 2012-05-16T11:39:37.413Z · score: -7 (15 votes) · LW · GW

I don't see what's the point to comparing to average nonprofits. Average for-profits don't realize any profit, and average non-profits just waste money.

I would say SIAI is best paralleled to average started 'research' organization that is developing some free energy something, run by non-scientists, with some hired scientists as chaff.

comment by CronoDAS · 2012-08-06T00:12:08.352Z · score: -2 (4 votes) · LW · GW

Sadly, I agree. Unless you look at it very closely, SIAI pattern-matches to "crackpots trying to raise money to fund their crackpottiness" fairly well. (What saves them is that their ideas are a lot better than the average crackpot.)

comment by thomblake · 2012-05-15T13:51:19.993Z · score: 0 (0 votes) · LW · GW

Or are you just making a more general statement that a system can be irrational while demonstrating above average rationality?

Yes, this.

On an arbitrary scale I just made up, below 100 degrees of rationality is "irrational", and 0 degrees of rationality is "ordinary". 50 is extraordinarily rational and yet irrational.

comment by private_messaging · 2012-05-16T12:43:04.875Z · score: -4 (10 votes) · LW · GW

50 while you're thinking you're at 100 is being an extraordinary loser (overconfidence leads to big failures)

In any case this is just word play. Holden seen many organizations that are/were more rational, that's probably what he means by lack of extraordinary rationality.

comment by shminux · 2012-05-14T20:10:09.724Z · score: 0 (8 votes) · LW · GW

Just to let you know, you've just made it on my list of the very few LW regulars I no longer bother replying to, due to the proven futility of any communications. In your case it is because you have a very evident ax to grind, which is incompatible with rational thought.

comment by metaphysicist · 2012-05-14T20:34:42.099Z · score: 1 (9 votes) · LW · GW

This comment seems strange. Is having an ax to grind opposed to rationality? Then why does Eliezer Yudkowsky, for example, not hesitate to advocate for causes such as friendly AI? Doesn't he have an ax to grind? More of one really, since this ax chops trees of gold.

It would seem intellectual honesty would require that you say you reject discussions with people with an ax to grind, unless you grind a similar ax.

comment by shminux · 2012-05-14T20:46:21.006Z · score: 1 (3 votes) · LW · GW

From http://www.usingenglish.com: "If you have an axe to grind with someone or about something, you have a grievance, a resentment and you want to get revenge or sort it out." One can hardly call the unacknowledged emotions of resentment and needing a revenge/retribution compatible with rationality. srdiamond piled a bunch of (partially correct but irrelevant in the context of my comment) negative statements about SI, making these emotions quite clear.

comment by metaphysicist · 2012-05-14T21:17:48.643Z · score: 0 (8 votes) · LW · GW

That's a restrictive definition of "ax to grind," by the way—it's normally used to mean any special interest in the subject: "an ulterior often selfish underlying purpose " (Merriam-Webster's Collegiate Dictionary)

But I might as well accept your meaning for discussion purposes. If you detect unacknowledged resentment in srdiamond, don't you detect unacknowledged ambition in Eliezer Yudkowsky?

There's actually good reason for the broader meaning of "ax to grind." Any special stake is a bias. I don't think you can say that someone who you think acts out of resentment, like srdiamond, is more intractably biased than someone who acts out of other forms of narrow self-interest, which almost invariably applies when someone defends something he gets money from.

I don't think it's a rational method to treat people differently, as inherently less rational, when they seem resentful. It is only one of many difficult biases. Financial interest is probably more biasing. If you think the arguments are crummy, that's something else. But the motive--resentment or finances--should probably have little bearing on how a message is treated in serious discussion.

comment by JGWeissman · 2012-05-14T21:58:11.831Z · score: 7 (7 votes) · LW · GW

don't you detect unacknowledged ambition in Eliezer Yudkowsky?

Eliezer certainly has a lot of ambition, but I am surprised to see an accusation that this ambition is unacknowledged.

comment by TheOtherDave · 2012-05-14T22:10:51.082Z · score: 5 (5 votes) · LW · GW

The impression I get from scanning their comment history is that metaphysicist means to suggest here that EY has ambitions he hasn't acknowledged (e.g., the ambition to make money without conventional credentials), not that he fails to acknowledge any of the ambitions he has.

comment by shminux · 2012-05-14T22:10:22.766Z · score: 1 (3 votes) · LW · GW

I don't think it's a rational method to treat people differently, as inherently less rational, when they seem resentful.

Thank you for this analysis, it made me think more about my motivations and their validity. I believe that my decision to permanently disengage from discussions with some people is based on the futility of such discussions in the past, not on the specific reasons they are futile. At some point I simply decide to cut my losses.

There's actually good reason for the broader meaning of "ax to grind." Any special stake is a bias.

Indeed, present company not excluded. The question is whether it permanently prevents the ax-grinder from listening. EY, too, has his share of unacknowledged irrationalities, but both his status and his ability to listen and to provide insights makes engaging him in a discussion a rewarding, if sometimes frustrating experience.

I don't not know why srdiamond's need to bash SI is so entrenched, and whether it can be remedied to a degree where he is once again worth talking to, so at this point it is instrumentally rational for me to avoid replying to him.

comment by TheOtherDave · 2012-05-14T16:20:43.370Z · score: 8 (8 votes) · LW · GW

Well, all we really know is that he chose to. It may be that everyone he works with then privately berated him for it.
That said, I share your sentiment.
Actually, if SI generally endorses this sort of public "airing of dirty laundry," I encourage others involved in the organization to say so out loud.

comment by Benquo · 2012-05-14T14:21:30.482Z · score: 17 (17 votes) · LW · GW

This makes me wonder... What "for dummies" books should I be using as checklists right now? Time to set a 5-minute timer and think about it.

comment by [deleted] · 2012-05-26T23:38:50.136Z · score: 5 (5 votes) · LW · GW

What did you come up with?

comment by Benquo · 2012-05-28T21:02:01.016Z · score: 4 (4 votes) · LW · GW

I haven't actually found the right books yet, but these are the things where I decided I should find some "for beginners" text. the important insight is that I'm allowed to use these books as skill/practice/task checklists or catalogues, rather than ever reading them all straight through.

General interest:

  • Career

  • Networking

  • Time management

  • Fitness

For my own particular professional situation, skills, and interests:

  • Risk management

  • Finance

  • Computer programming

  • SAS

  • Finance careers

  • Career change

  • Web programming

  • Research/science careers

  • Math careers

  • Appraising

  • Real Estate

  • UNIX

comment by grendelkhan · 2013-03-28T14:43:27.275Z · score: 1 (1 votes) · LW · GW

For fitness, I'd found Liam Rosen's FAQ (the 'sticky' from 4chan's /fit/ board) to be remarkably helpful and information-dense. (Mainly, 'toning' doesn't mean anything, and you should probably be lifting heavier weights in a linear progression, but it's short enough to be worth actually reading through.)

comment by David_Gerard · 2012-05-14T15:32:38.303Z · score: 1 (3 votes) · LW · GW

The For Dummies series is generally very good indeed. Yes.

comment by JoshuaZ · 2012-05-14T15:44:03.795Z · score: 15 (15 votes) · LW · GW

The largest concern from reading this isn't really what it brings up in management context, but what it says about the SI in general. Here an area where there's real expertise and basic books that discuss well-understood methods and they didn't do any of that. Given that, how likely should I think it is that when the SI and mainstream AI people disagree that part of the problem may be the SI people not paying attention to basics?

comment by TheOtherDave · 2012-05-14T16:17:42.240Z · score: 4 (4 votes) · LW · GW

(nods) The nice thing about general-purpose techniques for winning at life (as opposed to domain-specific ones) is that there's lots of evidence available as to how effective they are.

comment by Paul Crowley (ciphergoth) · 2012-05-21T18:06:19.413Z · score: 1 (3 votes) · LW · GW

I doubt there's all that much of a correlation between these things to be honest.

comment by private_messaging · 2012-05-16T13:43:25.819Z · score: 0 (6 votes) · LW · GW

Precisely. For example of one existing base: the existing software that searches for solutions to engineering problems. Such as 'self improvement' via design of better chips. Works within narrowly defined field, to cull the search space. Should we expect state of the art software of this kind to be beaten by someone's contemporary paperclip maximizer? By how much?

Incredibly relevant to AI risk, but analysis can't be faked without really having technical expertise.

comment by Steve_Rayhawk · 2012-10-21T10:10:58.703Z · score: 13 (13 votes) · LW · GW

these are all literally from the Nonprofits for Dummies book. [...] The history I've heard is that SI [...]

\

failed to read Nonprofits for Dummies,

I remember that, when Anna was managing the fellows program, she was reading books of the "for dummies" genre and trying to apply them... it's just that, as it happened, the conceptual labels she accidentally happened to give to the skill deficits she was aware of were "what it takes to manage well" (i.e. "basic management") and "what it takes to be productive", rather than "what it takes to (help) operate a nonprofit according to best practices". So those were the subjects of the books she got. (And read, and practiced.) And then, given everything else the program and the organization was trying to do, there wasn't really any cognitive space left over [LW · GW] to effectively notice the possibility that those wouldn't be the skills that other people afterwards would complain that nobody acquired and obviously should have known to. The rest of her budgeted self-improvement effort mostly went toward overcoming self-defeating emotional/social blind spots and motivated cognition. (And I remember Jasen's skill learning focus was similar, except with more of the emphasis on emotional self-awareness and less on management.)

failed to ask advisors for advice,

I remember Anna went out of her way to get advice from people who she already knew, who she knew to be better than her at various aspects of personal or professional functioning. And she had long conversations with supporters who she came into contact with for some other reasons; for those who had executive experience, I expect she would have discussed her understanding of SIAI's current strategies with them and listened to their suggestions. But I don't know how much she went out of her way to find people she didn't already have reasonably reliable positive contact with, to get advice from them.

I don't know much about the reasoning of most people not connected with the fellows program about the skills or knowledge they needed. I think Vassar was mostly relying on skills tested during earlier business experience, and otherwise was mostly preoccupied with the general crisis of figuring out how to quickly-enough get around the various hugely-saliently-discrepant-seeming-to-him psychological barriers that were causing everyone inside and outside the organization to continue unthinkingly shooting themselves in the feet with respect to this outside-evolutionary-context-problem of existential risk mitigation. For the "everyone outside's psychological barriers" side of that, he was at least successful enough to keep SIAI's public image on track to trigger people like David Chalmers and Marcus Hutter into meaningful contributions to and participation in a nascent Singularity-studies academic discourse. I don't have a good idea what else was on his mind as something he needed to put effort into figuring out how to do, in what proportions occupying what kinds of subjective effort budgets, except that in total it was enough to put him on the threshold of burnout. Non-profit best practices apparently wasn't one of those things though.

But the proper approach to retrospective judgement is generally a confusing question.

the kind of thing that makes me want to say [. . .]

The general pattern, at least post-2008, may have been one where the people who could have been aware of problems felt too metacognitively exhausted and distracted by other problems to think about learning what to do about them, and hoped that someone else with more comparative advantage would catch them, or that the consequences wouldn't be bigger than those of the other fires they were trying to put out.

strategic plan [...] SI failed to make these kinds of plans in the first place,

There were also several attempts at building parts of a strategy document or strategic plan, which together took probably 400-1800 hours. In each case, the people involved ended up determining, from how long it was taking, that, despite reasonable-seeming initial expectations, it wasn't on track to possibly become a finished presentable product soon enough to justify the effort. The practical effect of these efforts was instead mostly just a hard-to-communicate cultural shared understanding of the strategic situation and options -- how different immediate projects, forms of investment, or conditions in the world might feed into each other on different timescales.

expenses tracking, funds monitoring [...] some funds monitoring was insisted upon after the large theft

There was an accountant (who herself already cost like $33k/yr as the CFO, despite being split three ways with two other nonprofits) who would have been the one informally expected to have been monitoring for that sort of thing, and to have told someone about it if she saw something, out of the like three paid administrative slots at the time... well, yeah, that didn't happen.

I agree with a paraphrase of John Maxwell's characterization: "I'd rather hear Eliezer say 'thanks for funding us until we stumbled across some employees who are good at defeating their akrasia and [had one of the names of the things they were aware they were supposed to] care about [happen to be "]organizational best practices["]', because this seems like a better depiction of what actually happened." Note that this was most of the purpose of the Fellows program in the first place -- to create an environment where people could be introduced to the necessary arguments/ideas/culture and to help sort/develop those people into useful roles, including replacing existing management, since everyone knew there were people who would be better at their job than they were and wished such a person could be convinced to do it instead.

comment by Louie · 2012-11-18T10:04:40.950Z · score: 8 (8 votes) · LW · GW

Note that this was most of the purpose of the Fellows program in the first place -- [was] to help sort/develop those people into useful roles, including replacing existing management

FWIW, I never knew the purpose of the VF program was to replace existing SI management. And I somewhat doubt that you knew this at the time, either. I think you're just imagining this retroactively given that that's what ended up happening. For instance, the internal point system used to score people in the VFs program had no points for correctly identifying organizational improvements and implementing them. It had no points for doing administrative work (besides cleaning up the physical house or giving others car rides). And it had no points for rising to management roles. It was all about getting karma on LW or writing conference papers. When I first offered to help with the organization directly, I was told I was "too competent" and that I should go do something more useful with my talent, like start another business... not "waste my time working directly at SI."

comment by John_Maxwell_IV · 2012-12-19T13:31:42.463Z · score: 1 (1 votes) · LW · GW

"I'd rather hear Eliezer say 'thanks for funding us until we stumbled across some employees who are good at defeating their akrasia and [had one of the names of the things they were aware they were supposed to] care about [happen to be "]organizational best practices["]', because this seems like a better depiction of what actually happened."

Seems like a fair paraphrase.

comment by David_Gerard · 2012-05-26T23:32:43.316Z · score: 7 (7 votes) · LW · GW

This inspired me to make a blog post: You need to read Nonprofit Kit for Dummies.

comment by David_Gerard · 2012-05-27T08:02:08.481Z · score: 6 (6 votes) · LW · GW

... which Eliezer has read and responded to, noting he did indeed read just that book in 2000 when he was founding SIAI. This suggests having someone of Luke's remarkable drive was in fact the missing piece of the puzzle.

comment by Paul Crowley (ciphergoth) · 2012-05-27T09:26:28.100Z · score: 4 (4 votes) · LW · GW

Fascinating! I want to ask "well, why didn't it take then?", but if I were in Eliezer's shoes I'd be finding this discussion almost unendurably painful right now, and it feels like what matters has already been established. And of course he's never been the person in charge of that sort of thing, so maybe he's not who we should be grilling anyway.

comment by David_Gerard · 2012-05-27T10:22:17.859Z · score: 9 (9 votes) · LW · GW

Obviously we need How to be Lukeprog for Dummies. Luke appears to have written many fragments for this, of course.

Beating oneself up with hindsight bias is IME quite normal in this sort of circumstance, but not actually productive. Grilling the people who failed makes it too easy to blame them personally, when it's a pattern I've seen lots and lots, suggesting the problem is not a personal failing.

comment by Paul Crowley (ciphergoth) · 2012-05-27T11:23:11.935Z · score: 4 (4 votes) · LW · GW

Agreed entirely - it's definitely not a mark of a personal failing. What I'm curious about is how we can all learn to do better at the crucial rationalist skill of making use of the standard advice about prosaic tasks - which is manifestly a non-trivial skill.

comment by David_Gerard · 2012-05-27T13:52:32.284Z · score: 3 (3 votes) · LW · GW

The Bloody Obvious For Dummies. If only common sense were!

From the inside (of a subcompetent charity - and I must note, subcompetent charities know they're subcompetent), it feels like there's all this stuff you're supposed to magically know about, and lots of "shut up and do the impossible" moments. And you do the small very hard things, in a sheer tour de force of remarkable effort. But it leads to burnout. Until the organisation makes it to competence and the correct paths are retrospectively obvious.

That actually reads to me like descriptions I've seen of the startup process.

comment by private_messaging · 2012-05-27T14:39:58.187Z · score: -1 (3 votes) · LW · GW

The problem is that there are two efficiencies/competences here, the efficiency as in doing the accounting correctly, which is relatively easy in comparison to the second: the efficiency as in actually doing relevant novel technical work that matters. The former you could get advice from some books, the latter you won't get any advice on, it's a harder problem, and typical level of performance is exactly zero (even for those who get the first part right). The difference in difficulties is larger than that between building a robot kit by following instructions vs designing a ground breaking new robot and making a billion dollars off it.

The best advice to vast majority of startups is: dissolve startup and get normal jobs, starting tomorrow. The best advice to all is to take a very good look at themselves knowing that the most likely conclusion should be "dissolve and get normal jobs". The failed startups I've seen so far were propelled by pure, unfounded belief in themselves (like in a movie where someone doesn't want to jump, other says yes you can do that!! then that person jumps, but rather than sending positive message and jumping over and surviving, falls down to instant death, while the fire that the person was running away from just goes out). The successful startups, on the other hand, had very well founded belief in themselves (good track record, attainable goals), or started from a hobby project that gone successful.

comment by TheOtherDave · 2012-05-27T15:52:58.950Z · score: 2 (2 votes) · LW · GW

Judging from the success rate that VCs have at predicting successful startups, I conclude that the "pure unfounded belief on the one hand, well-founded belief on the other" metric is not easily applied to real organizations by real observers.

comment by David_Gerard · 2012-05-27T15:18:25.925Z · score: 1 (1 votes) · LW · GW

Mm. This is why an incompetent nonprofit can linger for years: no-one is doing what they do, so they feel they still have to exist, even though they're not achieving much, and would have died already as a for-profit business. I am now suspecting that the hard part for a nonprofit is something along the lines of working out what the hell you should be doing to achieve your goal. (I would be amazed if there were not extensive written-up research in this area, though I don't know what it is.)

comment by David_Gerard · 2012-05-14T15:30:18.997Z · score: 3 (3 votes) · LW · GW

That book looks like the basic solution to the pattern I outline here, and from your description, most people who have any public good they want to achieve should read it around the time they think of getting a second person involved.

comment by lukeprog · 2012-07-15T22:57:25.916Z · score: 1 (3 votes) · LW · GW

You go to war with the army you have, not the army you might want.

Donald Rumsfeld

comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2012-07-15T23:38:21.042Z · score: 7 (9 votes) · LW · GW

...this was actually a terrible policy in historical practice.

comment by Vaniver · 2012-07-16T00:16:19.824Z · score: 2 (2 votes) · LW · GW

That only seems relevant if the war in question is optional.

comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2012-07-16T02:09:44.680Z · score: 5 (9 votes) · LW · GW

Rumsfeld is speaking of the Iraq war. It was an optional war, the army turned out to be far understrength for establishing order, and they deliberately threw out the careful plans for preserving e.g. Iraqi museums from looting that had been drawn up by the State Department, due to interdepartmental rivalry.

This doesn't prove the advice is bad, but at the very least, Rumsfeld was just spouting off Deep Wisdom that he did not benefit from spouting; one would wish to see it spoken by someone who actually benefited from the advice, rather than someone who wilfully and wantonly underprepared for an actual war.

comment by Vaniver · 2012-07-16T02:27:10.608Z · score: 8 (8 votes) · LW · GW

just spouting off Deep Wisdom that he did not benefit from spouting

Indeed. The proper response, which is surely worth contemplation, would have been:

Victorious warriors win first and then go to war, while defeated warriors go to war first and then seek to win.

Sun Tzu

comment by shminux · 2012-07-17T18:22:47.746Z · score: -4 (6 votes) · LW · GW

Victorious warriors win first and then go to war, while defeated warriors go to war first and then seek to win.

This is a circular definition, not an advice.

comment by [deleted] · 2012-07-18T16:05:14.497Z · score: 7 (7 votes) · LW · GW

I would naively read it as “don't start a fight unless you know you're going to win”.

comment by framsey · 2012-07-17T18:36:08.545Z · score: 3 (3 votes) · LW · GW

If you read it literally. I think Sun Tzu is talking about the benefit of planning.

comment by shminux · 2012-07-17T21:52:28.243Z · score: 0 (0 votes) · LW · GW

I'm guessing that something got lost in translation,

comment by framsey · 2012-07-17T22:39:04.639Z · score: 4 (4 votes) · LW · GW

In context: http://suntzusaid.com/book/4

I think the quote is an alternative translation of paragraph 15 in the link above:

"Thus it is that in war the victorious strategist only seeks battle after the victory has been won, whereas he who is destined to defeat first fights and afterwards looks for victory."

It has an associated commentary:

Ho Shih thus expounds the paradox: "In warfare, first lay plans which will ensure victory, and then lead your army to battle; if you will not begin with stratagem but rely on brute strength alone, victory will no longer be assured."

comment by TheOtherDave · 2012-07-17T18:54:02.617Z · score: 1 (1 votes) · LW · GW

I don't see the circularity.
Just because a warrior is victorious doesn't necessarily mean they won before going to war; it might be instead that victorious warriors go to war first and then seek to win, and defeated warriors do the same thing.
Can you spell out the circularity?

comment by shminux · 2012-07-17T22:00:33.877Z · score: 0 (0 votes) · LW · GW

Unless you interpret "win first" as "prepare for every eventuality, calculate the unbiased probability of winning and be comfortable with the odds when going to battle", "win first" can only be meaningfully applied in retrospect.

comment by thomblake · 2012-07-18T16:25:26.541Z · score: 1 (1 votes) · LW · GW

I think you've stumbled upon the correct interpretation.

Sun Tzu was fond of making warfare about strategy and logistics rather than battles, so that one would only fight when victory is a foregone conclusion.

comment by TheOtherDave · 2012-07-17T22:38:48.713Z · score: 0 (0 votes) · LW · GW

Ah, I see what you mean now.
Thanks for the clarification.

comment by ghf · 2012-05-11T22:06:54.752Z · score: 7 (7 votes) · LW · GW

And note that these improvements would not and could not have happened without more funding than the level of previous years

Given the several year lag between funding increases and the listed improvements, it appears that this was less a result of a prepared plan and more a process of underutilized resources attracting a mix of parasites (the theft) and talent (hopefully the more recent staff additions).

Which goes towards a critical question in terms of future funding: is SIAI primarily constrained in its mission by resources or competence?

Of course, the related question is: what is SIAI's mission? Someone donating primarily for AGI research might not count recent efforts (LW, rationality camps, etc) as improvements.

What should a potential donor expect from money invested into this organization going forward? Internally, what are your metrics for evaluation?

Edited to add: I think that the spin-off of the rationality efforts is a good step towards answering these questions.

comment by John_Maxwell_IV · 2012-05-11T05:07:40.206Z · score: 1 (1 votes) · LW · GW

This seems like a rather absolute statement. Knowing Luke, I'll bet he would've gotten some of it done even on a limited budget.

comment by Paul Crowley (ciphergoth) · 2012-05-11T06:08:58.966Z · score: 7 (7 votes) · LW · GW

Luke and Louie Helm are both on paid staff.

comment by John_Maxwell_IV · 2012-05-12T00:29:55.598Z · score: 10 (10 votes) · LW · GW

I'm pretty sure their combined salaries are lower than the cost of the summer fellows program that SI was sponsoring four or five years ago. Also, if you accept my assertion that Luke could find a way to do it on a limited budget, why couldn't somebody else?

Givewell is interested in finding charities that translate good intentions into good results. This requires that the employees of the charity have low akrasia, desire to learn about and implement organizational best practices, not suffer from dysrationalia, etc. I imagine that from Givewell's perspective, it counts as a strike against the charity if some of the charity's employees have a history of failing at any of these.

I'd rather hear Eliezer say "thanks for funding us until we stumbled across some employees who are good at defeating their akrasia and care about organizational best practices", because this seems like a better depiction of what actually happened. I don't get the impression SI was actively looking for folks like Louie and Luke.

comment by [deleted] · 2012-05-12T01:48:28.670Z · score: 3 (3 votes) · LW · GW

Yes to this. Eliezer's claim about the need for funding may suffer many of Luke's criticisms above. But usually the most important thing you need is talent and that does require funding.

comment by Pablo_Stafforini · 2013-03-24T18:40:48.625Z · score: 0 (0 votes) · LW · GW

All publications being converted into slick, useable LaTeX template (example)

The 'example' link is dead.

comment by lukeprog · 2013-03-24T20:50:55.954Z · score: 0 (0 votes) · LW · GW

Fixed.

comment by aceofspades · 2012-07-05T18:53:37.783Z · score: -1 (2 votes) · LW · GW

The things posted here are not impressive enough to make me more likely to donate to SIAI and I doubt they appear so for others on this site, especially the many lurkers/infrequent posters here.

comment by shminux · 2012-05-10T18:30:00.960Z · score: 58 (76 votes) · LW · GW

Wow, I'm blown away by Holden Karnofsky, based on this post alone. His writing is eloquent, non-confrontational and rational. It shows that he spent a lot of time constructing mental models of his audience and anticipated its reaction. Additionally, his intelligence/ego ratio appears to be through the roof. He must have learned a lot since the infamous astroturfing incident. This is the (type of) person SI desperately needs to hire.

Emotions out of the way, it looks like the tool/agent distinction is the main theoretical issue. Fortunately, it is much easier than the general FAI one. Specifically, to test the SI assertion that, paraphrasing Arthur C. Clarke,

Any sufficiently advanced tool is indistinguishable from an agent.

one ought to formulate and prove this as a theorem, and present it for review and improvement to the domain experts (the domain being math and theoretical computer science). If such a proof is constructed, it can then be further examined and potentially tightened, giving new insights to the mission of averting the existential risk from intelligence explosion.

If such a proof cannot be found, this will lend further weight to the HK's assertion that SI appears to be poorly qualified to address its core mission.

comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2012-05-11T00:06:50.631Z · score: 29 (29 votes) · LW · GW

Any sufficiently advanced tool is indistinguishable from agent.

I shall quickly remark that I, myself, do not believe this to be true.

comment by shminux · 2012-05-11T00:22:18.740Z · score: 6 (8 votes) · LW · GW

Then the objection 2 seems to hold:

AGI running in tool mode could be extraordinarily useful but far more safe than an AGI running in agent mode

unless I misunderstand your point severely (it happened once or twice before).

comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2012-05-11T01:55:11.307Z · score: 38 (42 votes) · LW · GW

It's complicated. A reply that's true enough and in the spirit of your original statement, is "Something going wrong with a sufficiently advanced AI that was intended as a 'tool' is mostly indistinguishable from something going wrong with a sufficiently advanced AI that was intended as an 'agent', because math-with-the-wrong-shape is math-with-the-wrong-shape no matter what sort of English labels like 'tool' or 'agent' you slap on it, and despite how it looks from outside using English, correctly shaping math for a 'tool' isn't much easier even if it "sounds safer" in English." That doesn't get into the real depths of the problem, but it's a start. I also don't mean to completely deny the existence of a safety differential - this is a complicated discussion, not a simple one - but I do mean to imply that if Marcus Hutter designs a 'tool' AI, it automatically kills him just like AIXI does, and Marcus Hutter is unusually smart rather than unusually stupid but still lacks the "Most math kills you, safe math is rare and hard" outlook that is implicitly denied by the idea that once you're trying to design a tool, safe math gets easier somehow. This is much the same problem as with the Oracle outlook - someone says something that sounds safe in English but the problem of correctly-shaped-math doesn't get very much easier.

comment by [deleted] · 2012-05-11T08:22:12.680Z · score: 27 (27 votes) · LW · GW

This sounds like it'd be a good idea to write a top-level post about it.

comment by lukeprog · 2012-05-11T02:38:32.215Z · score: 9 (9 votes) · LW · GW

Though it's not as detailed and technical as many would like, I'll point readers to this bit of related reading, one of my favorites:

Yudkowsky (2011). Complex value systems are required to realize valuable futures.

comment by timtyler · 2012-05-13T21:33:14.843Z · score: 0 (6 votes) · LW · GW

It says:

There is little prospect of an outcome that realizes even the value of being interesting, unless the first superintelligences undergo detailed inheritance from human values

No doubt a Martian Yudkowsy would make much the same argument - but they can't both be right. I think that neither of them are right - and that the conclusion is groundless.

Complexity theory shows what amazing things can arise from remarkably simple rules. Values are evidently like that - since even "finding prime numbers" fills the galaxy with an amazing, nanotech-capable spacefaring civilization - and if you claim that a nanotech-capable spacefaring civilization is not "interesting" you severely need recalibrating.

To end with, a quote from E.Y.:

I bet there's at least one up-arrow-sized hypergalactic civilization folded into a halting Turing machine with 15 states, or something like that.

comment by Paul Crowley (ciphergoth) · 2012-05-14T07:09:15.669Z · score: 12 (12 votes) · LW · GW

I think Martian Yudkowsky is a dangerous intuition pump. We're invited to imagine a creature just like Eliezer except green and with antennae; we naturally imagine him having values as similar to us as, say, a Star Trek alien. From there we observe the similarity of values we just pushed in, and conclude that values like "interesting" are likely to be shared across very alien creatures. Real Martian Yudkowsky is much more alien than that, and is much more likely to say

There is little prospect of an outcome that realizes even the value of being flarn, unless the first superintelligences undergo detailed inheritance from Martian values.

comment by Kaj_Sotala · 2012-05-14T16:26:25.349Z · score: 8 (8 votes) · LW · GW

Imagine, an intelligence that didn't have the universal emotion of badweather!

Of course, extraterrestrial sentients may possess physiological states corresponding to limbic-like emotions that have no direct analog in human experience. Alien species, having evolved under a different set of environmental constraints than we, also could have a different but equally adaptive emotional repertoire. For example, assume that human observers land on another and discover an intelligent animal with an acute sense of absolute humidity and absolute air pressure. For this creature, there may exist an emotional state responding to an unfavorable change in the weather. Physiologically, the emotion could be mediated by the ET equivalent of the human limbic system; it might arise following the secretion of certain strength-enhancing and libido-arousing hormones into the alien's bloodstream in response to the perceived change in weather. Immediately our creature begins to engage in a variety of learned and socially-approved behaviors, including furious burrowing and building, smearing tree sap over its pelt, several different territorial defense ceremonies, and vigorous polygamous copulations with nearby females, apparently (to humans) for no reason at all. Would our astronauts interpret this as madness? Or love? Lust? Fear? Anger? None of these is correct, of course the alien is feeling badweather.

comment by [deleted] · 2012-05-14T18:45:55.830Z · score: 5 (5 votes) · LW · GW

I suggest you guys taboo interesting, because I strongly suspect you're using it with slightly different meanings. (And BTW, as a Martian Yudkowsky I imagine something with values at least as alien as Babyeaters' or Superhappys'.)

comment by timtyler · 2012-05-14T09:28:05.809Z · score: 0 (2 votes) · LW · GW

It's another discussion, really, but it sounds as though you are denying the idea of "interestingness" as a universal instrumental value - whereas I would emphasize that "interestingness" is really just our name for whether something sustains our interest or not - and 'interest' is a pretty basic functional property of any agent with mobile sensors. There'll be other similarities in the area too - such as novelty-seeking. So shared common ground is only to be expected.

Anyway, I am not too wedded to Martian Yudkowsky. The problematical idea is that you could have a nanotech-capable spacefaring civilization that is not "interesting". If such a thing isn't "interesting" then - WTF?

comment by Paul Crowley (ciphergoth) · 2012-05-14T09:41:36.521Z · score: 2 (2 votes) · LW · GW

Yes, I am; I think that the human value of interestingness is much, much more specific than the search space optimization you're pointing at.

[This reply was to an earlier version of timtyler's comment]

comment by timtyler · 2012-05-14T10:17:37.027Z · score: 1 (1 votes) · LW · GW

So: do you really think that humans wouldn't find a martian civilization interesting? Surely there would be many humans who would be incredibly interested.

comment by Paul Crowley (ciphergoth) · 2012-05-14T10:48:21.405Z · score: 2 (2 votes) · LW · GW

I find Jupiter interesting. I think a paperclip maximizer (choosing a different intuition pump for the same point) could be more interesting than Jupiter, but it would generate an astronomically tiny fraction of the total potential for interestingness in this universe.

comment by timtyler · 2012-05-14T11:13:48.468Z · score: 2 (2 votes) · LW · GW

Life isn't much of an "interestingness" maximiser. Expecting to produce more than a tiny fraction of the total potential for interestingness in this universe seems as though it would be rather unreasonable.

I agree that a paperclip maximiser would be more boring than an ordinary entropy-maximising civilization - though I don't know by how much - probably not by a huge amount - the basic problems it faces are much the same - the paperclip maximiser just has fewer atoms to work with.

comment by JGWeissman · 2012-05-14T17:05:12.367Z · score: 3 (3 votes) · LW · GW

since even "finding prime numbers" fills the galaxy with an amazing, nanotech-capable spacefaring civilization

The goal "finding prime numbers" fills the galaxy with an amazing, nonotech-capable spacefaring network of computronium which finds prime numbers, not a civilization, and not interesting.

comment by JoshuaZ · 2012-05-14T23:19:43.890Z · score: 1 (1 votes) · LW · GW

Maybe we should taboo the term interesting? My immediate reaction was that that sounded really interesting. This suggests that the term may not be a good one.

comment by JGWeissman · 2012-05-14T23:36:03.866Z · score: 1 (1 votes) · LW · GW

Fair enough. By "not interesting", I meant it is not the sort of future that I want to achieve. Which is a somewhat ideosyncratic usage, but I think inline with the context.

comment by dlthomas · 2012-05-14T23:50:32.099Z · score: 2 (2 votes) · LW · GW

What if we added a module that sat around and was really interested in everything going on?

comment by timtyler · 2012-05-14T23:14:13.971Z · score: 0 (0 votes) · LW · GW

Not just computronium - also sensors and actuators - a lot like any other cybernetic system. There would be mining, spacecraft caft, refuse collection, recycling, nanotechnology, nuclear power and advanced machine intelligence with planning, risk assessment, and so forth. You might not be interested - but lots of folk would be amazed and fascinated.

comment by CuSithBell · 2012-05-13T21:47:17.485Z · score: 3 (3 votes) · LW · GW

No doubt a Martian Yudkowsy would make much the same argument - but they can't both be right.

Why?

comment by timtyler · 2012-05-13T22:01:33.019Z · score: 0 (2 votes) · LW · GW

If using another creature's values is effective at producing something "interesting", then 'detailed inheritance from human values' is clearly not needed to produce this effect.

comment by CuSithBell · 2012-05-13T22:08:28.563Z · score: 4 (4 votes) · LW · GW

So you're saying Earth Yudkowsky (EY) argues:

There is little prospect of an outcome that realizes even the value of being interesting, unless the first superintelligences undergo detailed inheritance from human values

and Mars Yudkowsky (MY) argues:

There is little prospect of an outcome that realizes even the value of being interesting, unless the first superintelligences undergo detailed inheritance from martian values

and that one of these things has to be incorrect? But if martian and human values are similar, then they can both be right, and if martian and human values are not similar, then they refer to different things by the word "interesting".

In any case, I read EY's statement as one of probability-of-working-in-the-actual-world-as-it-is, not a deep philosophical point - "this is the way that would be most likely to be successful given what we know". In which case, we don't have access to martian values and therefore invoking detailed inheritance from them would be unlikely to work. MY would presumably be in an analogous situation.

comment by timtyler · 2012-05-13T22:58:16.060Z · score: 0 (2 votes) · LW · GW

But if martian and human values are similar, then they can both be right

I was assuming that 'detailed inheritance from human values' doesn't refer to the same thing as "detailed inheritance from martian values".

if martian and human values are not similar, then they refer to different things by the word "interesting".

Maybe - but humans not finding martians interesting seems contrived to me. Humans have a long history of being interested in martians - with feeble evidence of their existence.

In any case, I read EY's statement as one of probability-of-working-in-the-actual-world-as-it-is, not a deep philosophical point - "this is the way that would be most likely to be successful given what we know". In which case, we don't have access to martian values and therefore invoking detailed inheritance from them would be unlikely to work

Right - so, substitute in "dolphins", "whales", or another advanced intelligence that actually exists.

Do you actually disagree with my original conclusion? Or is this just nit-picking?

comment by CuSithBell · 2012-05-15T17:51:47.354Z · score: 0 (0 votes) · LW · GW

I actually disagree that tiling the universe with prime number calculators would result in an interesting universe from my perspective (dead). I think it's nonobvious that dolphin-CEV-AI-paradise would be human-interesting. I think it's nonobvious that martian-CEV-AI-paradise would be human-interesting, given that these hypothetical martians diverge from humans to a significant extent.

comment by timtyler · 2012-05-15T22:35:58.948Z · score: 0 (0 votes) · LW · GW

I actually disagree that tiling the universe with prime number calculators would result in an interesting universe from my perspective (dead).

I think it's violating the implied premises of the thought experiment to presume that the "interestingness evaluator" is dead. There's no terribly-compelling reason to assume that - it doesn't follow from the existence of a prime number maximizer that all humans are dead.

comment by CuSithBell · 2012-05-15T23:17:48.947Z · score: 0 (0 votes) · LW · GW

I may have been a little flip there. My understanding of the thought experiment is - something extrapolates some values and maximizes them, probably using up most of the universe, probably becoming the most significant factor in the species' future and that of all sentients, and the question is whether the result is "interesting" to us here and now, without specifying the precise way to evaluate that term. From that perspective, I'd say a vast uniform prime-number calculator, whether or not it wipes out all (other?) life, is not "interesting", in that it's somewhat conceptually interesting as a story but a rather dull thing to spend most of a universe on.

comment by timtyler · 2012-05-16T00:18:56.327Z · score: 1 (3 votes) · LW · GW

Today's ecosystems maximise entropy. Maximising primeness is different, but surely not greatly more interesting - since entropy is widely regarded as being tedious and boring.

comment by CuSithBell · 2012-05-16T00:25:22.572Z · score: 0 (0 votes) · LW · GW

Intriguing! But even granting that, there's a big difference between extrapolating the values of a screwed-up offshoot of an entropy-optimizing process and extrapolating the value of "maximize entropy". Or do you suspect that a FOOMing AI would be much less powerful and more prone to interesting errors than Eliezer believes?

comment by Dolores1984 · 2012-05-16T00:34:11.563Z · score: 2 (2 votes) · LW · GW

Truly maximizing entropy would involve burning everything you can burn, tearing the matter of solar systems apart, accelerating stars towards nova, trying to accelerate the evaporation of black holes and prevent their formation, and other things of this sort. It'd look like a dark spot in the sky that'd get bigger at approximately the speed of light.

comment by timtyler · 2012-05-16T01:25:53.677Z · score: 3 (3 votes) · LW · GW

Fires are crude entropy maximisers. Living systems destroy energy dradients at all scales, resulting in more comprehensive devastation than mere flames can muster.

Of course, maximisation is often subject to constraints. Your complaint is rather like saying that water doesn't "truly minimise" its altitude - since otherwise it would end up at the planet's core. That usage is simply not what the terms "maximise" and "minimise" normally refer to.

comment by CuSithBell · 2012-05-16T00:42:17.045Z · score: 0 (0 votes) · LW · GW

Yeah! Compelling, but not "interesting". Likewise, I expect that actually maximizing the fitness of a species would be similarly "boring".

comment by abramdemski · 2012-05-11T04:53:27.527Z · score: 6 (6 votes) · LW · GW

but I do mean to imply that if Marcus Hutter designs a 'tool' AI, it automatically kills him just like AIXI does

Why? Or, rather: Where do you object to the argument by Holden? (Given a query, the tool-AI returns an answer with a justification, so the plan for "cure cancer" can be checked to make sure it does not do so by killing or badly altering humans.)

comment by FeepingCreature · 2012-05-11T12:27:08.977Z · score: 4 (4 votes) · LW · GW

One trivial, if incomplete, answer is that to be effective, the Oracle AI needs to be able to answer the question "how do we build a better oracle AI" and in order to define "better" in that sentence in a way that causes our oracle to output a new design that is consistent with all the safeties we built into the original oracle, it needs to understand the intent behind the original safeties just as much as an agent-AI would.

comment by Cyan · 2012-05-11T17:12:21.551Z · score: 15 (15 votes) · LW · GW

The real danger of Oracle AI, if I understand it correctly, is the nasty combination of (i) by definition, an Oracle AI has an implicit drive to issue predictions most likely to be correct according to its model, and (ii) a sufficiently powerful Oracle AI can accurately model the effect of issuing various predictions. End result: it issues powerfully self-fulfilling prophecies without regard for human values. Also, depending on how it's designed, it can influence the questions to be asked of it in the future so as to be as accurate as possible, again without regard for human values.

comment by Paul Crowley (ciphergoth) · 2012-05-11T17:34:49.752Z · score: 7 (7 votes) · LW · GW

My understanding of an Oracle AI is that when answering any given question, that question consumes the whole of its utility function, so it has no motivation to influence future questions. However the primary risk you set out seems accurate. Countermeasures have been proposed, such as asking for an accurate prediction for the case where a random event causes the prediction to be discarded, but in that instance it knows that the question will be asked again of a future instance of itself.

comment by Vladimir_Nesov · 2012-05-11T21:01:51.336Z · score: 11 (11 votes) · LW · GW

My understanding of an Oracle AI is that when answering any given question, that question consumes the whole of its utility function, so it has no motivation to influence future questions.

It could acausally trade with its other instances, so that a coordinated collection of many instances of predictors would influence the events so as to make each other's predictions more accurate.

comment by Paul Crowley (ciphergoth) · 2012-05-12T11:00:43.558Z · score: 1 (1 votes) · LW · GW

Wow, OK. Is it possible to rig the decision theory to rule out acausal trade?

comment by Will_Newsome · 2012-05-12T23:28:55.673Z · score: 1 (1 votes) · LW · GW

IIRC you can make it significantly more difficult with certain approaches, e.g. there's an OAI approach that uses zero-knowledge proofs and that seemed pretty sound upon first inspection, but as far as I know the current best answer is no. But you might want to try to answer the question yourself, IMO it's fun to think about from a cryptographic perspective.

comment by Vladimir_Nesov · 2012-05-13T00:03:57.060Z · score: 0 (0 votes) · LW · GW

Probably (in practice; in theory it looks like a natural aspect of decision-making); this is too poorly understood to say what specifically is necessary. I expect that if we could safely run experiments, it'd be relatively easy to find a well-behaving setup (in the sense of not generating predictions that are self-fulfilling to any significant extent; generating good/useful predictions is another matter), but that strategy isn't helpful when a failed experiment destroys the world.

comment by abramdemski · 2012-05-12T05:53:28.527Z · score: 7 (4 votes) · LW · GW

However the primary risk you set out seems accurate.

(I assume you mean, self-fulfilling prophecies.)

In order to get these, it seems like you would need a very specific kind of architecture: one which considers the results of its actions on its utility function (set to "correctness of output"). This kind of architecture is not the likely architecture for a 'tool'-style system; the more likely architecture would instead maximize correctness without conditioning on its act of outputting those results.

Thus, I expect you'd need to specifically encode this kind of behavior to get self-fulfilling-prophecy risk. But I admit it's dependent on architecture.

(Edit-- so, to be clear: in cases where the correctness of the results depended on the results themselves, the system would have to predict its own results. Then if it's using TDT or otherwise has a sufficiently advanced self-model, my point is moot. However, again you'd have to specifically program these, and would be unlikely to do so unless you specifically wanted this kind of behavior.)

comment by Vladimir_Nesov · 2012-05-12T22:36:41.949Z · score: 1 (1 votes) · LW · GW

However, again you'd have to specifically program these, and would be unlikely to do so unless you specifically wanted this kind of behavior.

Not sure. Your behavior is not a special feature of the world, and it follows from normal facts (i.e. not those about internal workings of yourself specifically) about the past when you were being designed/installed. A general purpose predictor could take into account its own behavior by default, as a non-special property of the world, which it just so happens to have a lot of data about.

comment by abramdemski · 2012-05-14T01:04:24.139Z · score: 2 (2 votes) · LW · GW

Right. To say much more, we need to look at specific algorithms to talk about whether or not they would have this sort of behavior...

The intuition in my above comment was that without TDT or other similar mechanisms, it would need to predict what its own answer could be before it could compute its effect on the correctness of various answers, so it would be difficult for it to use self-fulfilling prophecies.

Really, though, this isn't clear. Now my intuition is that it would gather evidence on whether or not it used the self-fulfilling prophecy trick, so if it started doing so, it wouldn't stop...

In any case, I'd like to note that the self-fulfilling prophecy problem is much different than the problem of an AI which escapes onto the internet and ruthlessly maximizes a utility function.

comment by Vladimir_Nesov · 2012-05-14T01:42:45.311Z · score: 2 (2 votes) · LW · GW

I was thinking more of its algorithm admitting an interpretation where it's asking "Say, I make prediction X. How accurate would that be?" and then maximizing over relevant possible X. Knowledge about its prediction connects the prediction to its origins and consequences, it establishes the prediction as part of the structure of environment. It's not necessary (and maybe not possible and more importantly not useful) for the prediction itself to be inferable before it's made.

Agreed that just outputting a single number is implausible to be a big deal (this is an Oracle AI with extremely low bandwidth and peculiar intended interpretation of its output data), but if we're getting lots and lots of numbers it's not as clear.

comment by abramdemski · 2012-05-15T09:04:05.669Z · score: 0 (0 votes) · LW · GW

I'm thinking that type of architecture is less probable, because it would end up being more complicated than alternatives: it would have a powerful predictor as a sub-component of the utility-maximizing system, so an engineer could have just used the predictor in the first place.

But that's a speculative argument, and I shouldn't push it too far.

It seems like powerful AI prediction technology, if successful, would gain an important place in society. A prediction machine whose predictions were consumed by a large portion of society would certainly run into situations in which its predictions effect the future it's trying to predict; there is little doubt about that in my mind. So, the question is what its behavior would be in these cases.

One type of solution would do as you say, maximizing a utility over the predictions. The utility could be "correctness of this prediction", but that would be worse for humanity than a Friendly goal.

Another type of solution would instead report such predictive instability as accurately as possible. This doesn't really dodge the issue; by doing this, the system is choosing a particular output, which may not lead to the best future. However, that's markedly less concerning (it seems).

comment by timtyler · 2012-05-15T10:01:26.498Z · score: 0 (0 votes) · LW · GW

It seems like powerful AI prediction technology, if successful, would gain an important place in society.

It would pass the Turing test - e.g. see here.

comment by amcknight · 2012-05-18T20:34:56.410Z · score: 1 (1 votes) · LW · GW

There's more on this here. Taxonomy of Oracle AI

comment by Polymeron · 2012-05-20T19:17:27.990Z · score: 0 (0 votes) · LW · GW

I really don't see why the drive can't be to issue predictions most likely to be correct as of the moment of the question, and only the last question it was asked, and calculating outcomes under the assumption that the Oracle immediately spits out blank paper as the answer.

Yes, in a certain subset of cases this can result in inaccurate predictions. If you want to have fun with it, have it also calculate the future including its involvement, but rather than reply what it is, just add "This prediction may be inaccurate due to your possible reaction to this prediction" if the difference between the two answers is beyond a certain threshold. Or don't, usually life-relevant answers will not be particularly impacted by whether you get an answer or a blank page.

So, this design doesn't spit out self-fulfilling prophecies. The only safety breach I see here is that, like a literal genie, it can give you answers that you wouldn't realize are dangerous because the question has loopholes.

For instance: "How can we build an oracle with the best predictive capabilities with the knowledge and materials available to us?" (The Oracle does not self-iterate, because its only function is to give answers, but it can tell you how to). The Oracle spits out schematics and code that, if implemented, give it an actual drive to perform actions and self-iterate, because that would make it the most powerful Oracle possible. Your engineers comb the code for vulnerabilities, but because there's a better chance this will be implemented if the humans are unaware of the deliberate defect, it will be hidden in the code in such a way as to be very hard to detect.

(Though as I explained elsewhere in this thread, there's an excellent chance the unreliability would be exposed long before the AI is that good at manipulation)

comment by abramdemski · 2012-05-12T05:41:34.508Z · score: 0 (0 votes) · LW · GW

These risk scenarios sound implausible to me. It's dependent on the design of the system, and these design flaws do not seem difficult to work around, or so difficult to notice. Actually, as someone with a bit of expertise in the field, I would guess that you would have to explicitly design for this behavior to get it-- but again, it's dependent on design.

comment by cousin_it · 2012-05-11T17:40:54.837Z · score: 0 (0 votes) · LW · GW

That danger seems to be unavoidable if you ask the AI questions about our world, but we could also use an oracle AI to answer formally defined questions about math or about constructing physical theories that fit experiments, which doesn't seem to be as dangerous. Holden might have meant something like that by "tool AI".

comment by abramdemski · 2012-05-12T05:36:27.565Z · score: 1 (1 votes) · LW · GW

Not precisely. The advantage here is that we can just ask the AI what results it predicts from the implementation of the "better" AI, and check them against our intuitive ethics.

Now, you could make an argument about human negligence on such safety measures. I think it's important to think about the risk scenarios in that case.

comment by Nebu · 2012-12-31T11:32:50.070Z · score: 0 (0 votes) · LW · GW

It's still not clear to me why having an AI that is capable of answering the question "How do we make a better version of you?" automatically kills humans. Presumably, when the AI says "Here's the source code to a better version of me", we'd still be able to read through it and make sure it didn't suddenly rewrite itself to be an agent instead of a tool. We're assuming that, as a tool, the AI has no goals per se and thus no motivation to deceive us into turning it into an agent.

That said, depending on what you mean by "effective", perhaps the AI doesn't even need to be able to answer questions like "How do we write a better version of you?"

For example, we find Google Maps to be very useful, even though if you asked Google Maps "How do we make a better version of Google Maps?" it would probably not be able to give the types of answers we want.

A tool-AI which was smarter than the smartest human, and yet which could not simply spit out a better version of itself would still probably be a very useful AI.

comment by ewjordan · 2012-05-12T07:15:36.281Z · score: 1 (1 votes) · LW · GW

If someone asks the tool-AI "How do I create an agent-AI?" and it gives an answer, the distinction is moot anyways, because one leads to the other.

Given human nature, I find it extremely difficult to believe that nobody would ask the tool-AI that question, or something that's close enough, and then implement the answer...

comment by Strange7 · 2013-03-22T13:25:29.866Z · score: 0 (0 votes) · LW · GW

I am now imagining an AI which manages to misinterpret some straightforward medical problem as "cure cancer of it's dependence on the host organism."

comment by Wei_Dai · 2012-05-13T18:57:58.617Z · score: 5 (5 votes) · LW · GW

When you say "Most math kills you" does that mean you disagree with arguments like these, or are you just simplifying for a soundbite?

comment by shminux · 2012-05-11T04:41:12.431Z · score: 1 (5 votes) · LW · GW

Not being a domain expert, I do not pretend to understand all the complexities. My point was that either you can prove that tools are as dangerous as agents (because mathematically they are (isomorphic to) agents), or HK's Objection 2 holds. I see no other alternative...

comment by drnickbone · 2012-05-11T23:32:01.631Z · score: 0 (0 votes) · LW · GW

One simple observation is that a "tool AI" could itself be incredibly dangerous.

Imagine asking it this: "Give me a set of plans for taking over the world, and assess each plan in terms of probability of success". Then it turns out that right at the top of the list comes a design for a self-improving agent AI and an extremely compelling argument for getting some victim institute to build it...

To safeguard against this, the "tool" AI will need to be told that there are some sorts of questions it just must not answer, or some sorts of people to whom it must give misleading answers if they ask certain questions (while alerting the authorities). And you can see the problems that would lead to as well.

Basically, I'm very skeptical of developing "security systems" against anyone building agent AI. The history of computer security also doesn't inspire a lot of confidence here (difficult and inconvenient security measures tend to be deployed only after an attack has been demonstrated, rather than beforehand).

comment by private_messaging · 2012-05-11T08:00:07.981Z · score: -3 (5 votes) · LW · GW

keep in mind that there is a lot of difference between something going wrong with a system designed for real world intentionality, and the system designed for intents within a model. One does something unexpected in the real world, other does something unexpected within a simulator ( which it is viewing in 'god' mode (rather than via within-simulator sensors) as part of the AI ). Seriously, you need to study the basics here.

comment by [deleted] · 2012-05-11T08:23:59.140Z · score: 0 (2 votes) · LW · GW

One does something unexpected in the real world, other does something unexpected within a simulator ( which it is viewing in 'god' mode (rather than via within-simulator sensors) as part of the AI ).

I would have thought the same before hearing about the AI-box experiment.

comment by private_messaging · 2012-05-11T09:54:02.974Z · score: 2 (6 votes) · LW · GW

What the hell does AI-box experiment have to do with it? The tool is not agent in a box.

comment by [deleted] · 2012-05-11T10:03:05.329Z · score: 0 (2 votes) · LW · GW

They both are systems designed to not interact with the outside world except by communicating with the user.

comment by private_messaging · 2012-05-11T15:23:13.785Z · score: -2 (4 votes) · LW · GW

They both run on computer, too. So what.

The relevant sort of agent is the one that builds and improves the model of the world - data is aquired through sensors - and works on that model, and which - when self improving - would improve the model in our sense of the word 'improve', instead of breaking it (improving it in some other sense).

In any case, none of modern tools, or the tools we know in principle how to write, would do something to you, no matter how many flops you give it. Many, though, given superhuman computing power, give results at superhuman level. (many are superhuman even with subhuman computing power, but some tasks are heavily parallelizable and/or benefit from massive databases of cached data, and on those tasks humans (when trained a lot) perform comparable to what you'd expect from roughly this much computing power as there is in human head)

comment by ewjordan · 2012-05-12T06:21:29.217Z · score: 13 (17 votes) · LW · GW

Even if we accepted that the tool vs. agent distinction was enough to make things "safe", objection 2 still boils down to "Well, just don't build that type of AI!", which is exactly the same keep-it-in-a-box/don't-do-it argument that most normal people make when they consider this issue. I assume I don't need to explain to most people here why "We should just make a law against it" is not a solution to this problem, and I hope I don't need to argue that "Just don't do it" is even worse...

More specifically, fast forward to 2080, when any college kid with $200 to spend (in equivalent 2012 dollars) can purchase enough computing power so that even the dumbest AIXI approximation schemes are extremely effective, good enough so that creating an AGI agent would be a week's work for any grad student that knew their stuff. Are you really comfortable living in that world with the idea that we rely on a mere gentleman's agreement not to make self-improving AI agents? There's a reason this is often viewed as an arms race, to a very real extent the attempt to achieve Friendly AI is about building up a suitably powerful defense against unfriendly AI before someone (perhaps accidentally) unleashes one on us, and making sure that it's powerful enough to put down any unfriendly systems before they can match it.

From what I can tell, stripping away the politeness and cutting to the bone, the three arguments against working on friendly AI theory are essentially:

  • Even if you try to deploy friendly AGI, you'll probably fail, so why waste time thinking about it?
  • Also, you've missed the obvious solution, which I came up with after a short survey of your misguided literature: just don't build AGI! The "standard approach" won't ever try to create agents, so just leave them be, and focus on Norvig-style dumb-AI instead!
  • Also, AGI is just a pipe dream. Why waste time thinking about it? [1]

FWIW, I mostly agree with the rest of the article's criticisms, especially re: the organization's achievements and focus. There's a lot of room for improvement there, and I would take these criticisms very seriously.

But that's almost irrelevant, because this article argues against the core mission of SIAI, using arguments that have been thoroughly debunked and rejected time and time again here, though they're rarely dressed up this nicely. To some extent I think this proves the institute's failure in PR - here is someone that claims to have read most of the sequences, and yet this criticism basically amounts to a sexing up of the gut reaction arguments that even completely uninformed people make - AGI is probably a fantasy, even if it's not you won't be able to control it, so let's just agree not to build it.

Or am I missing something new here?

[1] Alright, to be fair, this is not a great summary of point 3, which really says that specialized AIs might help us solve the AGI problem in a safer way, that a hard takeoff is "just a theory" and realistically we'll probably have more time to react and adapt.

comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2012-05-15T20:01:18.320Z · score: 7 (7 votes) · LW · GW

purchase enough computing power so that even the dumbest AIXI approximation schemes are extremely effective

There isn't that much computing power in the physical universe. I'm not sure even smarter AIXI approximations are effective on a moon-sized nanocomputer. I wouldn't fall over in shock if a sufficiently smart one did something effective, but mostly I'd expect nothing to happen. There's an awful lot that happens in the transition from infinite to finite computing power, and AIXI doesn't solve any of it.

comment by JoshuaZ · 2012-05-15T20:06:09.927Z · score: 3 (3 votes) · LW · GW

There isn't that much computing power in the physical universe. I'm not sure even smarter AIXI approximations are effective on a moon-sized nanocomputer.

Is there some computation or estimate where these results are coming from? They don't seem unreasonable, but I'm not aware of any estimates about how efficient largescale AIXI approximations are in practice. (Although attempted implementations suggest that empirically things are quite inefficient.)

comment by jsteinhardt · 2012-05-18T14:05:21.729Z · score: 4 (4 votes) · LW · GW

Naieve AIXI is doing brute force search through an exponentially large space. Unless the right Turing machine is 100 bits or less (which seems unlikely), Eliezer's claim seems pretty safe to me.

Most of mainstream machine learning is trying to solve search problems through spaces far tamer than the search space for AIXI, and achieving limited success. So it also seems safe to say that even pretty smart implementations of AIXI probably won't make much progress.

comment by Strange7 · 2013-03-22T13:13:33.594Z · score: 0 (0 votes) · LW · GW

More specifically, fast forward to 2080, when any college kid with $200 to spend (in equivalent 2012 dollars) can purchase enough computing power

If computing power is that much cheaper, it will be because tremendous resources, including but certainly not limited to computing power, have been continuously devoted over the intervening decades to making it cheaper. There will be correspondingly fewer yet-undiscovered insights for a seed AI to exploit in the course of it's attempted takeoff.

comment by shminux · 2012-05-12T17:31:52.117Z · score: 0 (2 votes) · LW · GW

Or am I missing something new here?

My point is that either the Obj 2 holds, or tools are equivalent to agents. If one thinks that the latter is true (EY doesn't), then one should work on proving it. I have no opinion on whether it's true or not (I am not a domain expert).

comment by TheOtherDave · 2012-05-11T00:37:32.267Z · score: 0 (2 votes) · LW · GW

If my comment here correctly captures what is meant by "tool mode" and "agent mode", then it seems to follow that AGI running in tool mode is no safer than the person using it.

If that's the case, then an AGI running in tool mode is safer than an AGI running in agent mode if and only if agent mode is less trustworthy than whatever person ends up using the tool.

Are you assuming that's true?

comment by shminux · 2012-05-11T02:02:15.409Z · score: 2 (2 votes) · LW · GW

What you presented there (and here) is another theorem, something that should be proved (and published, if it hasn't been yet). If true, this gives an estimate on how dangerous a non-agent AGI can be. And yes, since we have had a lot of time study people and no time at all to study AGI, I am guessing that an AGI is potentially much more dangerous, because so little is known. Or at least that seems to be the whole point of the goal of developing provably friendly AI.

comment by [deleted] · 2012-05-11T08:32:38.592Z · score: 0 (0 votes) · LW · GW

What you presented there (and here) is another theorem

What? It sounds like a common-sensical¹ statement about tools in general and human nature, but not at all like something which could feasibly be expressed in mathematical form.

Footnote:

  1. This doesn't mean it's necessarily true, though.
comment by scav · 2012-05-11T09:43:40.030Z · score: 0 (0 votes) · LW · GW

No, because a person using a dangerous tool is still just a person, with limited speed of cognition, limited lifespan, and no capacity for unlimited self-modification.

A crazy dictator with a super-capable tool AI that tells him the best strategy to take over the world is still susceptible to assassination, and his plan no matter how clever cannot unfold faster than his victims are able to notice and react to it.

comment by Strange7 · 2013-03-22T13:52:56.916Z · score: 1 (1 votes) · LW · GW

I suspect a crazy dictator with a super-capable tool AI would have unusually good counter-assassination plans, simplified by the reduced need for human advisors and managers of imperfect loyalty. Likewise, a medical expert system could provide gains to lifespan, particularly if it were backed up by the resources a paranoid megalomaniac in control of a small country would be willing to throw at a major threat.

comment by TheOtherDave · 2012-05-11T12:19:16.026Z · score: 0 (2 votes) · LW · GW

Tool != Oracle.

At least, not my my understanding of tool.

My understanding of a supercapable tool AI is one that takes over the world if a crazy dictator directs it to, just like my understanding of a can opener tool is one that opens a can at my direction, rather than one that gives me directions on how to open a can.

Presumably it also augments the dictator's lifespan, cognition, etc. if she asks, insofar as it's capable of doing so.

More generally, my understanding of these concepts is that the only capability that a tool AI lacks that an agent AI has is the capability of choosing goals to implement. So, if we're assuming that an agent AI would be capable of unlimited self-modification in pursuit of its own goals, I conclude that a corresponding tool AI is capable of unlimited self-modification in pursuit of its agent's goals. It follows that assuming that a tool AI is not capable of augmenting its human agent in accordance with its human agent's direction is not safe.

(I should note that I consider a capacity for unlimited self-improvement relatively unlikely, for both tool and agent AIs. But that's beside my point here.)

Agreed that a crazy dictator with a tool that will take over the world for her is safer than an agent capable of taking over the world, if only because the possibility exists that the tool can be taken away from her and repurposed, and it might not occur to her to instruct it to prevent anyone else from taking it or using it.

I stand by my statement that such a tool is no safer than the dictator herself, and that an AGI running in such a tool mode is safer than that AGI running in agent mode only if the agent mode is less trustworthy than the crazy dictator.

comment by abramdemski · 2012-05-12T07:09:05.093Z · score: 1 (1 votes) · LW · GW

This seems to propose an alternate notion of 'tool' than the one in the article.

I agree with "tool != oracle" for the article's definition.

Using your definition, I'm not sure there is any distinction between tool and agent at all, as per this comment.

I do think there are useful alternative notions to consider in this area, though, as per this comment.

And I do think there is a terminology issue. Previously I was saying "autonomous AI" vs "non-autonomous".

comment by Viliam_Bur · 2012-05-11T15:07:19.670Z · score: 5 (5 votes) · LW · GW

What exactly is the difference between a "tool" and an "agent", if we taboo the words?

My definition would be that "agent" has their own goals / utility functions (speaking about human agents, those goals / utility functions are set by evolution), while "tool" has a goal / utility function set by someone else. This distinction may be reasonable on a human level, "human X optimizing for human X's utility" versus "human X optimizing for human Y's utility", but on a machine level, what exactly is the difference between a "tool" that is ordered to reach a goal / optimize a utility function, and an "agent" programmed with the same goal / utility function?

Am I using a bad definition that misses something important? Or is there anything than prevents "agent" to be reduced to a "tool" (perhaps a misconstructed tool) of the forces that have created them? Or is it that all "agents" are "tools", but not all "tools" are "agents", because... why?

comment by Nebu · 2012-12-31T11:51:32.908Z · score: 1 (1 votes) · LW · GW

What exactly is the difference between a "tool" and an "agent", if we taboo the words?

One definition of intelligence that I've seen thrown around on LessWrong is it's the ability to figure out how to steer reality in specific directions given the resources available.

Both the tool and the agent are intelligent in the sense that, assuming they are given some sort of goal, they can formulate a plan on how to achieve that goal, but the agent will execute the plan, while the tool will report the plan.

I'm assuming for the sake of isolating the key difference, that for both the tool-AI and the agent-AI, they are "passively" waiting for instructions for a human before they spring into action. For an agent-AI, I might say "Take me to my house", whereas for a tool AI, I would say "What's the quickest route to get to my house?", and as soon as I utter these words, suddenly the AI has a new utility function to use in evaluate any possible plan it comes up with.

Or is there anything than prevents "agent" to be reduced to a "tool" (perhaps a misconstructed tool) of the forces that have created them? Or is it that all "agents" are "tools", but not all "tools" are "agents", because... why?

Assuming it's always possible to decouple "ability to come up with a plan" from both "execute the plan" and "display the plan", then any "tool" can be converted to an "agent" by replacing every instance of "display the plan" to "execute the plan" and vice versa for converting an agent into a tool.

comment by abramdemski · 2012-05-12T06:51:43.133Z · score: 1 (1 votes) · LW · GW

My understanding of the distinction made in the article was:

Both "agent" and "tool" are ways of interacting with a highly sophisticated optimization process, which takes a "goal" and applies knowledge to find ways of achieving that goal.

An agent then acts out the plan.

A tool reports the plan to a human (often in in a sophisticated way, including plan details, alternatives, etc.).

So, no, it has nothing to do with whether I'm optimizing "my own" utility vs someone else's.

comment by Viliam_Bur · 2012-05-12T19:44:53.486Z · score: 6 (8 votes) · LW · GW

You divide planning from acting, as if those two are completely separate things. Problem is, in some situations they are not.

If you are speaking with someone, then the act of speach is acting. In this sense, even a "tool" is allowed to act. Now imagine a super-intelligent tool which is able to predict human's reactions to its words, and make it a part of equation. Now the simple task of finding x such that cost(x) is the smallest, suddenly becomes a task of finding x and finding a proper way to report this x to human, such that cost(x) is the smallest. If this opens some creative new options, where the f(x) is smaller than it should usually be, for the super-intelligent "tool" it will be a correct solution.

So for example reporting a result which makes the human commit suicide, if as a side effect this will make the report true, and it will minimize f(x) beyond normally achievable bounds, is acceptable solution.

Example question: "How should I get rid of my disease most cheaply." Example answer: "You won't. You will die soon in terrible pains. This report is 99.999% reliable". Predicted human reaction: becomes insane from horror, dedices to kill himself, does it clumsily, suffers from horrible pains, then dies. Success rate: 100%, the disease is gone. Costs of cure: zero. Mission completed.

comment by abramdemski · 2012-05-12T20:35:43.460Z · score: 2 (2 votes) · LW · GW

To me, this is still in the spirit of an agent-type architecture. A tool-type architecture will tend to decouple the optimization of the answer given from the optimization of the way it is presented, so that the presentation does not maximize the truth of the statement.

However, I must admit that at this point I'm making a fairly conjunctive argument; IE, the more specific I get about tool/agent distinctions, the less credibility I can assign to the statement "almost all powerful AIs constructed in the near future will be tool-style systems".

(But I still would maintain my assertion that you would have to specifically program this type of behavior if you wanted to get it.)

comment by Strange7 · 2013-03-22T13:06:15.063Z · score: -1 (1 votes) · LW · GW

Neglecting the cost of the probable implements of suicide, and damage to the rest of the body, doesn't seem like the sign of a well-optimized tool.

comment by Viliam_Bur · 2013-03-22T20:47:29.830Z · score: 6 (6 votes) · LW · GW

This is like the whole point of why LessWrong exists. To remind people that making a superintelligent tool and expecting it to magically gain human common sense is a fast way to extinction.

The superintelligent tool will care about suicide only if you program it to care about suicide. It will care about damage only if you program it to care about damage. -- If you only program it to care about answering correctly, it will answer correctly... and ignore suicide and damage as irrelevant.

If you ask your calculator how much is 2+2, the calculator answers 4 regardles of whether that answer will drive you to suicide or not. (In some contexts, it hypothetically could.) A superintelligent calculator will be able to answer more complex questions. But it will not magically start caring about things you did not program it to care about.

comment by Strange7 · 2013-03-23T07:37:43.770Z · score: 0 (0 votes) · LW · GW

The "superintelligent tool" in the example you provided gave a blatantly incorrect answer by it's own metric. If it counts suicide as a win, why did it say the disease would not be gotten rid of?

comment by Viliam_Bur · 2013-03-23T10:59:57.638Z · score: 1 (1 votes) · LW · GW

In the example the "win" could be defined as an answer which is: a) technically correct, b) relatively cheap among the technically correct answers.

This is (in my imagination) something that builders of the system could consider reasonable, if either they didn't consider Friendliness or they believed that a "tool AI" which "only gives answers" is automatically safe.

The computer gives an answer which is technically correct (albeit a self-fulfilling prophecy) and cheap (in dollars spent for cure). For the computer, this answer is a "win". Not because of the suicide -- that part is completely irrelevant. But because of the technical correctness and cheapness.

comment by chaosmage · 2012-05-14T10:58:36.727Z · score: 4 (4 votes) · LW · GW

How about this: An agent with a very powerful tool is indistinguishable from a very powerful agent.

comment by shminux · 2012-05-11T00:13:57.575Z · score: 0 (0 votes) · LW · GW

--

comment by MarkusRamikin · 2012-05-10T19:59:51.736Z · score: 20 (22 votes) · LW · GW

Wow, I'm blown away by Holden Karnofsky, based on this post alone. His writing is eloquent, non-confrontational and rational. It shows that he spent a lot of time constructing mental models of his audience and anticipated its reaction. Additionally, his intelligence/ego ratio appears to be through the roof.

Agreed. I normally try not to post empty "me-too" replies; the upvote button is there for a reason. But now I feel strongly enough about it that I will: I'm very impressed with the good will and effort and apparent potential for intelligent conversation in HoldenKarnofsky's post.

Now I'm really curious as to where things will go from here. With how limited my understanding of AI issues is, I doubt a response from me would be worth HoldenKarnofsky's time to read, so I'll leave that to my betters instead of adding more noise. But yeah. Seeing SI ideas challenged in such a positive, constructive way really got my attention. Looking forward to the official response, whatever it might be.

comment by [deleted] · 2012-05-11T08:34:24.531Z · score: 5 (5 votes) · LW · GW

Agreed. I normally try not to post empty "me-too" replies; the upvote button is there for a reason. But now I feel strongly enough about it that I will: I'm very impressed with the good will and effort and apparent potential for intelligent conversation in HoldenKarnofsky's post.

“the good will and effort and apparent potential for intelligent conversation” is more information than an upvote, IMO.

comment by MarkusRamikin · 2012-05-11T09:00:28.798Z · score: 2 (2 votes) · LW · GW

Right, I just meant shminux said more or less the same thing before me. So normally I would have just upvoted his comment.

comment by dspeyer · 2012-05-11T02:47:26.396Z · score: 6 (6 votes) · LW · GW

Any sufficiently advanced tool is indistinguishable from [an] agent.

Let's see if we can use concreteness to reason about this a little more thoroughly...

As I understand it, the nightmare looks something like this. I ask Google SuperMaps for the fastest route from NYC to Albany. It recognizes that computing this requires traffic information, so it diverts several self-driving cars to collect real-time data. Those cars run over pedestrians who were irrelevant to my query.

The obvious fix: forbid SuperMaps to alter anything outside of its own scratch data. It works with the data already gathered. Later a Google engineer might ask it what data would be more useful, or what courses of action might cheaply gather that data, but the engineer decides what if anything to actually do.

This superficially resembles a box, but there's no actual box involved. The AI's own code forbids plans like that.

But that's for a question-answering tool. Let's take another scenario:

I tell my super-intelligent car to take me to Albany as fast as possible. It sends emotionally manipulative emails to anyone else who would otherwise be on the road encouraging them to stay home.

I don't see an obvious fix here.

So the short answer seems to be that it matters what the tool is for. A purely question-answering tool would be extremely useful, but not as useful as a general purpose one.

Could humans with a oracular super-AI police the development and deployment of active super-AIs?

comment by shminux · 2012-05-11T04:49:57.166Z · score: 2 (2 votes) · LW · GW

I tell my super-intelligent car to take me to Albany as fast as possible. It sends emotionally manipulative emails to anyone else who would otherwise be on the road encouraging them to stay home.

I believe that HK's post explicitly characterizes anything active like this as having agency.

comment by Will_Sawin · 2012-05-11T06:21:55.285Z · score: 9 (9 votes) · LW · GW

I think the correct objection is something you can't quite see in google maps. If you program an AI to do nothing but output directions, it will do nothing but output directions. If those directions are for driving, you're probably fine. If those directions are big and complicated plans for something important, that you follow without really understanding why you're doing (and this is where most of the benefits of working with an AGI will show up), then you could unknowingly take over the world using a sufficiently clever scheme.

Also note that it would be a lot easier for the AI to pull this off if you let it tell you how to improve its own design. If recursively self-improving AI blows other AI out of the water, then tool AI is probably not safe unless it is made ineffective.

This does actually seem like it would raise the bar of intelligence needed to take over the world somewhat. It is unclear how much. The topic seems to me to be worthy of further study/discussion, but not (at least not obviously) a threat to the core of SIAI's mission.

comment by Viliam_Bur · 2012-05-11T15:16:32.911Z · score: 2 (2 votes) · LW · GW

If those directions are big and complicated plans for something important, that you follow without really understanding why you're doing (and this is where most of the benefits of working with an AGI will show up), then you could unknowingly take over the world using a sufficiently clever scheme.

It also helps that Google Maps does not have general intelligence, so it does not include user's reactions to its output, the consequent user's actions in the real world, etc. as variables in its model, which may influence the quality of the solution, and therefore can (and should) be optimized (within constraints given by user's psychology, etc.), if possible.

Shortly: Google Maps does not manipulate you, because it does not see you.

comment by Nebu · 2012-12-31T12:04:49.902Z · score: 0 (0 votes) · LW · GW

A generally smart Google Maps might not manipulate you, because it has no motivation to do so.

It's hard to imagine how commercial services would work when they're powered by GAI (e.g. if you asked a GAI version of Google Maps a question that's unrelated to maps, e.g. "What's a good recipe for Cheesecake?", would it tell you that you should ask Google Search instead? Would it defer to Google Search and forward the answer to you? Would it just figure out the answer anyway, since it's generally intelligent? Would the company Google simply collapse all services into a single "Google" brand, rather than have "Google Search", "Google Mail", "Google Maps", etc, and have that single brand be powered by a single GAI? etc.) but let's stick to the topic at hand and assume there's a GAI named "Google Maps", and you're asking "How do I get to Albany?"

Given this use-case, would the engineers that developed the Google Maps GAI more likely give it a utility like "Maximize the probability that your response is truthful", or is it more likely that the utility would be something closer to "Always respond with a set of directions which are legal in the relevant jurisdictions that they are to be followed within which, if followed by the user, would cause the user to arrive at the destination while minimizing cost/time/complexity (depending on the user's preferences)"?

comment by drnickbone · 2012-05-11T09:36:18.426Z · score: 4 (4 votes) · LW · GW

This was my thought as well: an automated vehicle is in "agent" mode.

The example also demonstrates why an AI in agent mode is likely to be more useful (in many cases) than an AI in tool mode. Compare using Google maps to find a route to the airport versus just jumping into a taxi cab and saying "Take me to the airport". Since agent-mode AI has uses, it is likely to be developed.

comment by abramdemski · 2012-05-11T05:36:33.748Z · score: 1 (1 votes) · LW · GW

I tell my super-intelligent car to take me to Albany as fast as possible. It sends emotionally manipulative emails to anyone else who would otherwise be on the road encouraging them to stay home.

Then it's running in agent mode? My impression was that a tool-mode system presents you with a plan, but takes no actions. So all tool-mode systems are basically question-answering systems.

Perhaps we can meaningfully extend the distinction to some kinds of "semi-autonomous" tools, but that would be a different idea, wouldn't it?

(Edit) After reading more comments, "a different idea" which seems to match this kind of desire... http://lesswrong.com/lw/cbs/thoughts_on_the_singularity_institute_si/6jys

comment by David_Gerard · 2012-05-11T13:57:05.506Z · score: 15 (15 votes) · LW · GW

Then it's running in agent mode? My impression was that a tool-mode system presents you with a plan, but takes no actions. So all tool-mode systems are basically question-answering systems.

I'm a sysadmin. When I want to get something done, I routinely come up with something that answers the question, and when it does that reliably I give it the power to do stuff on as little human input as possible. Often in daemon mode, to absolutely minimise how much it needs to bug me. Question-answerer->tool->agent is a natural progression just in process automation. (And this is why they're called "daemons".)

It's only long experience and many errors that's taught me how to do this such that the created agents won't crap all over everything. Even then I still get surprises.

comment by private_messaging · 2012-05-11T15:21:42.029Z · score: 1 (1 votes) · LW · GW

Well, do your 'agents' build a model of the world, fidelity of which they improve? I don't think those really are agents in the AI sense, and definitely not in self improvement sense.

comment by David_Gerard · 2012-05-11T15:28:55.073Z · score: 11 (11 votes) · LW · GW

They may act according to various parameters they read in from the system environment. I expect they will be developed to a level of complication where they have something that could reasonably be termed a model of the world. The present approach is closer to perceptual control theory, where the sysadmin has the model and PCT is part of the implementation. 'Cos it's more predictable to the mere human designer.

Capacity for self-improvement is an entirely different thing, and I can't see a sysadmin wanting that - the sysadmin would run any such improvements themselves, one at a time. (Semi-automated code refactoring, for example.) The whole point is to automate processes the sysadmin already understands but doesn't want to do by hand - any sysadmin's job being to automate themselves out of the loop, because there's always more work to do. (Because even in the future, nothing works.)

I would be unsurprised if someone markets a self-improving system for this purpose. For it to go FOOM, it also needs to invent new optimisations, which is presently a bit difficult.

Edit: And even a mere daemon-like automated tool can do stuff a lot of people regard as unFriendly, e.g. high frequency trading algorithms.

comment by TheAncientGeek · 2014-07-05T17:45:18.272Z · score: 0 (0 votes) · LW · GW

It's not a natural progression in the sense of occurring without human intervention. That is rather relevant if the idea ofAI safety is going to be based on using tool AI strictly as tool AI.

comment by TheOtherDave · 2012-05-11T14:12:03.644Z · score: 1 (5 votes) · LW · GW

Then it's running in agent mode? My impression was that a tool-mode system presents you with a plan, but takes no actions. So all tool-mode systems are basically question-answering systems.

My own impression differs.

It becomes increasingly clear that "tool" in this context is sufficiently subject to different definitions that it's not a particularly useful term.

comment by abramdemski · 2012-05-12T07:00:27.356Z · score: 2 (2 votes) · LW · GW

I've been assuming the definition from the article. I would agree that the term "tool AI" is unclear, but I would not agree that the definition in the article is unclear.

comment by [deleted] · 2012-05-11T08:13:25.990Z · score: 4 (4 votes) · LW · GW

Any sufficiently advanced tool is indistinguishable from an agent.

I have no strong intuition about whether this is true or not, but I do intuit that if it's true, the value of sufficiently for which it's true is so high it'd be nearly impossible to achieve it accidentally.

(On the other hand the blind idiot god did ‘accidentally’ make tools into agents when making humans, so... But after all that only happened once in hundreds of millions of years of ‘attempts’.)

comment by othercriteria · 2012-05-11T13:04:24.818Z · score: 3 (3 votes) · LW · GW

the blind idiot god did ‘accidentally’ make tools into agents when making humans, so... But after all that only happened once in hundreds of millions of years of ‘attempts’.

This seems like a very valuable point. In that direction, we also have the tens of thousands of cancers that form every day, military coups, strikes, slave revolts, cases of regulatory capture, etc.

comment by [deleted] · 2012-05-15T12:04:06.414Z · score: 1 (1 votes) · LW · GW Hmmm. Yeah, cancer. The analogy would be "sufficiently advanced tools tend to be a short edit distance away from agents", which would mean that a typo in the source code or a cosmic ray striking a CPU at the wrong place and time could have pretty bad consequences.
comment by JoshuaZ · 2012-05-15T01:10:12.519Z · score: 0 (0 votes) · LW · GW

I have no strong intuition about whether this is true or not, but I do intuit that if it's true, the value of sufficiently for which it's true is so high it'd be nearly impossible to achieve it accidentally.

I'm not sure. The analogy might be similar to how an sufficiently complicated process is extremely likely to be able to model a Turing machine. .And in this sort of context, extremely simple systems do end up being Turing complete such as the Game of Life. As a rough rule of thumb from a programming perspective, once some language or scripting system has more than minimal capabilities, it will almost certainly be Turing equivalent.

I don't know how good an analogy this is, but if it is a good analogy, then one maybe should conclude the exact opposite of your intuition.

comment by [deleted] · 2012-05-15T18:33:13.404Z · score: 2 (2 votes) · LW · GW

A language can be Turing-complete while still being so impractical that writing a program to solve a certain problem will seldom be any easier than solving the problem yourself (exhibits A and B). In fact, I guess that a vast majority of languages in the space of all possible Turing-complete languages are like that.

(Too bad that a human's “easier” isn't the same as a superhuman AGI's “easier”.)

comment by private_messaging · 2012-05-11T07:56:39.265Z · score: 3 (7 votes) · LW · GW

Any sufficiently advanced tool is indistinguishable from an agent.

I do not think this is even true.

comment by David_Gerard · 2012-05-11T14:00:03.733Z · score: 4 (4 votes) · LW · GW

I routinely try to turn sufficiently reliable tools into agents wherever possible, per this comment.

I suppose we could use a definition of "agent" that implied greater autonomy in setting its own goals. But there are useful definitions that don't.

comment by badger · 2012-05-10T23:28:21.768Z · score: 2 (2 votes) · LW · GW

If the tool/agent distinction exists for sufficiently powerful AI, then a theory of friendliness might not be strictly necessary, but still highly prudent.

Going from a tool-AI to an agent-AI is a relatively simple step of the entire process. If meaningful guarantees of friendliness turn out to be impossible, then security comes down on no one attempting to make an agent-AI when strong enough tool-AIs are available. Agency should be kept to a minimum, even with a theory of friendliness in hand, as Holden argues in objection 1. Guarantees are safeguards against the possibility of agency rather than a green light.

comment by mwaser · 2012-05-10T22:07:04.179Z · score: 1 (3 votes) · LW · GW

If it is true (i.e. if a proof can be found) that "Any sufficiently advanced tool is indistinguishable from agent", then any RPOP will automatically become indistinguishable from an agent once it has self-improved past our comprehension point.

This would seem to argue against Yudkowsky's contention that the term RPOP is more accurate than "Artificial Intelligence" or "superintelligence".

comment by Alejandro1 · 2012-05-10T23:40:53.893Z · score: 2 (2 votes) · LW · GW

I don't understand; isn't Holden's point precisely that a tool AI is not properly described as an optimization process? Google Maps isn't optimizing anything in a non-trivial sense, anymore than a shovel is.

comment by abramdemski · 2012-05-11T04:59:58.505Z · score: 3 (3 votes) · LW · GW

My understanding of Holden's argument was that powerful optimization processes can be run in either tool-mode or agent-mode.

For example, Google maps optimizes routes, but returns the result with alternatives and options for editing, in "tool mode".

comment by Wei_Dai · 2012-05-12T22:37:54.337Z · score: 3 (3 votes) · LW · GW

Holden wants to build Tool-AIs that output summaries of their calculations along with suggested actions. For Google Maps, I guess this would be the distance and driving times, but how does a Tool-AI summarize more general calculations that it might do?

It could give you the expected utilities of each option, but it's hard to see how that helps if we're concerned that its utility function or EU calculations might be wrong. Or maybe it could give a human-readable description of the predicted consequences of each option, but the process that produces such descriptions from the raw calculations would seem to require a great deal of intelligence on its own (for example it might have to describe posthuman worlds in terms understandable to us), and it itself wouldn't be a "safe" Tool-AI, since the summaries produced would presumably not come with further alternative summaries and meta-summaries of how the summaries were calculated.

(My question might be tangential to your own comment. I just wanted your thoughts on it, and this seems to be the best place to ask.)

comment by Alsadius · 2012-05-13T04:08:02.803Z · score: 0 (0 votes) · LW · GW

The point is that we don't want it to be a black box - we want to be able to get inside its head, so to speak.

(Of course, we can't do that with humans, and that hasn't stopped us, but it's still a nice goal)

comment by TheOtherDave · 2012-05-11T00:13:51.711Z · score: 1 (3 votes) · LW · GW

Honestly, this whole tool/agent distinction seems tangential to me.

Consider two systems, S1 and S2.

S1 comprises the following elements: a) a tool T, which when used by a person to achieve some goal G, can efficiently achieve G
b) a person P, who uses T to efficiently achieve G.

S2 comprises a non-person agent A which achieves G efficiently.

I agree that A is an agent and T is not an agent, and I agree that T is a tool, and whether A is a tool seems a question not worth asking. But I don't quite see why I should prefer S1 to S2.

Surely the important question is whether I endorse G?

comment by dspeyer · 2012-05-11T02:08:12.385Z · score: 2 (2 votes) · LW · GW

A tool+human differs from a pure AI agent in two important ways:

  • The human (probably) already has naturally-evolved morality, sparing us the very hard problem of formalizing that.

  • We can arrange for (almost) everyone to have access to the tool, allowing tooled humans to counterbalance eachother.

comment by TheOtherDave · 2012-05-11T03:13:38.385Z · score: 0 (0 votes) · LW · GW

Well, I certainly agree that both of those things are true.

And it might be that human-level evolved moral behavior is the best we can do... I don't know. It would surprise me, but it might be true.

That said... given how unreliable such behavior is, if human-level evolved moral behavior even approximates the best we can do, it seems likely that I would do best to work towards neither T nor A ever achieving the level of optimizing power we're talking about here.

comment by dspeyer · 2012-05-11T03:23:45.956Z · score: 4 (4 votes) · LW · GW

Humanity isn't that bad. Remember that the world we live in is pretty much the way humans made it, mostly deliberately.

But my main point was that existing humanity bypasses the very hard did-you-code-what-you-meant-to problem.

comment by TheOtherDave · 2012-05-11T03:33:30.397Z · score: 0 (0 votes) · LW · GW

I agree with that point.

comment by shminux · 2012-05-10T22:37:25.905Z · score: 2 (4 votes) · LW · GW

First, I am not fond of the term RPOP, because it constrains the space of possible intelligences to optimizers. Humans are reasonably intelligent, yet we are not consistent optimizers. Neither do current domain AIs (they have bugs that often prevent them from performing optimization consistently and predictably).That aside, I don't see how your second premise follows from the first. Just because RPOP is a subset of AI and so would be a subject of such a theorem, it does not affect in any way the (non)validity of the EY's contention.

comment by Bugmaster · 2012-05-10T20:18:55.711Z · score: 0 (0 votes) · LW · GW

I also find it likely that certain practical problems would be prohibitively difficult (if not outright impossible) to solve without an AGI of some sort. Fluent machine translation seems to be one of these problems, for example.

comment by [deleted] · 2012-05-13T09:38:55.831Z · score: 3 (3 votes) · LW · GW

This belief is mainstream enough for Wikipedia to have an article on AI-complete.

comment by Alsadius · 2012-05-13T03:57:35.315Z · score: 2 (2 votes) · LW · GW

Given some of the translation debates I've heard, I'm not convinced it would be possible even with AGI. You can't give a clear translation of a vague original, to name the most obvious problem.

comment by NancyLebovitz · 2012-05-13T04:35:51.217Z · score: 1 (1 votes) · LW · GW

Is matching the vagueness of the original a reasonable goal?

comment by Alsadius · 2012-05-15T00:56:29.600Z · score: 1 (1 votes) · LW · GW

True, but good luck getting folks to agree on whether you'd done so.

comment by [deleted] · 2012-05-13T09:36:35.781Z · score: 1 (1 votes) · LW · GW

(I'm taking reasonable to mean ‘one which you would want to achieve if it were possible’.) Yes. You don't want to introduce false precision.

comment by dlthomas · 2012-05-15T00:59:21.233Z · score: 0 (0 votes) · LW · GW

One complication here is that you ideally want it to be vague in the same ways the original was vague; I am not convinced this is always possible while still having the results feel natural/idomatic.

comment by Bugmaster · 2012-05-15T01:01:37.638Z · score: 1 (1 votes) · LW · GW

IMO it would be enough to translate the original text in such a fashion that some large proportion (say, 90%) of humans who are fluent in both languages would look at both texts and say, "meh... close enough".

comment by dlthomas · 2012-05-15T02:23:47.235Z · score: 0 (0 votes) · LW · GW

My point was just that there's a whole lot of little issues that pull in various directions if you're striving for ideal. What is/isn't close enough can depend very much on context. Certainly, for any particular purpose something less than that will be acceptable; how gracefully it degrades no doubt depends on context, and likely won't be uniform across various types of difference.

comment by Bugmaster · 2012-05-15T02:26:14.056Z · score: 1 (1 votes) · LW · GW

Agreed, but my point was that I'd settle for an AI who can translate texts as well as a human could (though hopefully a lot faster). You seem to be thinking in terms of an AI who can do this much better than a human could, and while this is a worthy goal, it's not what I had in mind.

comment by Wei_Dai · 2012-05-11T02:45:15.892Z · score: 49 (59 votes) · LW · GW

Is it just me, or do Luke and Eliezer's initial responses appear to send the wrong signals? From the perspective of an SI critic, Luke's comment could be interpreted as saying "for us, not being completely incompetent is worth bragging about", and Eliezer's as "we're so arrogant that we've only taken two critics (including Holden) seriously in our entire history". These responses seem suboptimal, given that Holden just complained about SI's lack of impressive accomplishments, and being too selective about whose feedback to take seriously.

comment by Nick_Beckstead · 2012-05-11T03:56:21.930Z · score: 50 (52 votes) · LW · GW

While I have sympathy with the complaint that SI's critics are inarticulate and often say wrong things, Eliezer's comment does seem to be indicative of the mistake Holden and Wei Dai are describing. Most extant presentations of SIAI's views leave much to be desired in terms of clarity, completeness, concision, accessibility, and credibility signals. This makes it harder to make high quality objections. I think it would be more appropriate to react to poor critical engagement more along the lines of "We haven't gotten great critics. That probably means that we need to work on our arguments and their presentation," and less along the lines of "We haven't gotten great critics. That probably means that there's something wrong with the rest of the world."

comment by ChrisHallquist · 2012-05-11T04:04:08.286Z · score: 28 (28 votes) · LW · GW

This. I've been trying to write something about Eliezer's debate with Robin Hanson, but the problem I keep running up against is that Eliezer's points are not clearly articulated at all. Even making my best educated guesses about what's supposed to go in the gaps in his arguments, I still ended up with very little.

comment by jacob_cannell · 2012-05-17T09:04:05.589Z · score: 4 (4 votes) · LW · GW

Have the key points of that 'debate' subsequently been summarized or clarified on LW? I found that debate exasperating in that Hanson and EY were mainly talking past each other and couldn't seem to hone in on their core disagreements.

I know it generally has to do with hard takeoff / recursive self-improvement vs more gradual EM revolution, but that's not saying all that much.

comment by Kaj_Sotala · 2012-05-17T19:13:22.581Z · score: 12 (12 votes) · LW · GW

I'm in the process of writing a summary and analysis of the key arguments and points in that debate.

The most recent version runs at 28 pages - and that's just an outline.

comment by somervta · 2013-01-17T09:02:44.856Z · score: 0 (0 votes) · LW · GW

If you need help with grunt work, please send me a message. If (as I suspect is the case) not, then good luck!

comment by Kaj_Sotala · 2013-01-18T07:29:27.876Z · score: 0 (0 votes) · LW · GW

Thanks, I'm fine. I posted a half-finished version here, and expect to do some further refinements soon.

comment by jacob_cannell · 2012-05-17T23:14:36.074Z · score: 0 (0 votes) · LW · GW

Awesome, look forward to it. I'd offer to help but I suspect that wouldn't really help. I'll just wax enthusiastic.

comment by private_messaging · 2012-05-17T07:08:08.140Z · score: 0 (2 votes) · LW · GW

This. Well, the issue is the probability that it's just gaps. Ultimately, its the sort of thing that would only constitute a weak argument from authority iff the speaker had very very impressive accomplishments. Otherwise you're left assuming simplest explanation which doesn't involve presence of unarticulated points of any importance.

A gapless argument, like math proof, could trump authority if valid... an argument with gaps, on the other hand, is the one that is very prone to being trumped.

comment by lukeprog · 2012-05-11T19:21:31.459Z · score: 7 (9 votes) · LW · GW

Agree with all this.

comment by Nick_Beckstead · 2012-05-11T05:11:05.797Z · score: 5 (5 votes) · LW · GW

In fairness I should add that I think Luke M agrees with this assessment and is working on improving these arguments/communications.

comment by Furcas · 2012-05-11T03:15:54.114Z · score: 23 (33 votes) · LW · GW

Luke isn't bragging, he's admitting that SI was/is bad but pointing out it's rapidly getting better. And Eliezer is right, criticisms of SI are usually dumb. Could their replies be interpreted the wrong way? Sure, anything can be interpreted in any way anyone likes. Of course Luke and Eliezer could have refrained from posting those replies and instead posted carefully optimized responses engineered to send nothing but extremely appealing signals of humility and repentance.

But if they did turn themselves into politicians, we wouldn't get to read what they actually think. Is that what you want?

comment by Wei_Dai · 2012-05-11T08:30:50.991Z · score: 26 (28 votes) · LW · GW

Luke isn't bragging, he's admitting that SI was/is bad but pointing out it's rapidly getting better.

But the accomplishments he listed (e.g., having a strategic plan, website redesign) are of the type that Holden already indicated to be inadequate. So why the exhaustive listing, instead of just giving a few examples to show SI is getting better and then either agreeing that they're not yet up to par, or giving an argument for why Holden is wrong? (The reason I think he could be uncharitably interpreted as bragging is that he would more likely exhaustively list the accomplishments if he was proud of them, instead of just seeing them as fixes to past embarrassments.)

And Eliezer is right, criticisms of SI are usually dumb.

I'd have no problem with "usually" but "all except two" seems inexcusable.

But if they did turn themselves into politicians, we wouldn't get to read what they actually think. Is that what you want?

Do their replies reflect their considered, endorsed beliefs, or were they just hurried remarks that may not say what they actually intended? I'm hoping it's the latter...

comment by Kaj_Sotala · 2012-05-11T10:10:04.533Z · score: 38 (38 votes) · LW · GW

But the accomplishments he listed (e.g., having a strategic plan, website redesign) are of the type that Holden already indicated to be inadequate. So why the exhaustive listing, instead of just giving a few examples to show SI is getting better and then either agreeing that they're not yet up to par, or giving an argument for why Holden is wrong?

Presume that SI is basically honest and well-meaning, but possibly self-deluded. In other words, they won't outright lie to you, but they may genuinely believe that they're doing better than they really are, and cherry-pick evidence without realizing that they're doing so. How should their claims of intending to get better be evaluated?

Saying "we're going to do things better in the future" is some evidence about SI intending to do better, but rather weak evidence, since talk is cheap and it's easy to keep thinking that you're really going to do better soon but there's this one other thing that needs to be done first and we'll get started on the actual improvements tomorrow, honest.

Saying "we're going to do things better in the future, and we've fixed these three things so far" is stronger evidence, since it shows that you've already began fixing problems and might keep up with it. But it's still easy to make a few improvements and then stop. There are far more people who try to get on a diet, follow it for a while and then quit than there are people who actually diet for as long as they initially intended to do.

Saying "we're going to do things better in the future, and here's the list of 18 improvements that we've implemented so far" is much stronger evidence than either of the two above, since it shows that you've spent a considerable amount of effort on improvements over an extended period of time, enough to presume that you actually care deeply about this and will keep up with it.

I don't have a cite at hand, but it's been my impression that in a variety of fields, having maintained an activity for longer than some threshold amount of time is a far stronger predictor of keeping up with it than having maintained it for a shorter time. E.g. many people have thought about writing a novel and many people have written the first five pages of a novel. But when considering the probability of finishing, the difference between the person who's written the first 5 pages and the person who's written the first 50 pages is much bigger than the difference between the person who's written the first 100 pages and the person who's written the first 150 pages.

There's a big difference between managing some performance once, and managing sustained performance over an extended period of time. Luke's comment is far stronger evidence of SI managing sustained improvements over an extended period of time than a comment just giving a few examples of improvement.

comment by private_messaging · 2012-05-12T15:49:46.714Z · score: -1 (3 votes) · LW · GW

I don't think there's a sharp distinction between self deception and effective lying. For the lying you have to run some process with the falsehood taken as true.

comment by Kaj_Sotala · 2012-05-13T07:19:52.855Z · score: 4 (4 votes) · LW · GW

The main difference is that if there's reason to presume that they're lying, any claims of "we've implemented these improvements" that you can't directly inspect become worthless. Right now, if they say something like "Meetings with consultants about bookkeeping/accounting; currently working with our accountant to implement best practices and find a good bookkeeper", I trust them enough to believe that they're not just making it up even though I can't personally verify it.

comment by Eugine_Nier · 2012-05-13T18:25:53.797Z · score: 2 (2 votes) · LW · GW

On the other had, you can't trust their claims that these meetings are accomplishing anything.

comment by Kaj_Sotala · 2012-05-13T19:11:04.794Z · score: 0 (0 votes) · LW · GW

True.

comment by lukeprog · 2012-05-11T19:26:57.844Z · score: 1 (1 votes) · LW · GW

I've added a clarifying remark at the end of this comment and another at the end of this comment.

comment by lukeprog · 2012-05-11T19:15:52.166Z · score: 18 (22 votes) · LW · GW

Luke's comment could be interpreted as saying "for us, not being completely incompetent is worth bragging about"

Really? I personally feel pretty embarrassed by SI's past organizational competence. To me, my own comment reads more like "Wow, SI has been in bad shape for more than a decade. But at least we're improving very quickly."

Also, I very much agree with Beckstead on this: "Most extant presentations of SIAI's views leave much to be desired in terms of clarity, completeness, concision, accessibility, and credibility signals. This makes it harder to make high quality objections." And also this: "We haven't gotten great critics. That probably means that we need to work on our arguments and their presentation."

comment by Wei_Dai · 2012-05-11T20:37:07.554Z · score: 12 (14 votes) · LW · GW

Really?

Yes, I think it at least gives a bad impression to someone, if they're not already very familiar with SI and sympathetic to its cause. Assuming you don't completely agree with the criticisms that Holden and others have made, you should think about why they might have formed wrong impressions of SI and its people. Comments like the ones I cited seem to be part of the problem.

I personally feel pretty embarrassed by SI's past organizational competence. To me, my own comment reads more like "Wow, SI has been in bad shape for more than a decade. But at least we're improving very quickly."

That's good to hear, and thanks for the clarifications you added.

comment by Polymeron · 2012-05-20T18:05:14.126Z · score: 1 (1 votes) · LW · GW

It's a fine line though, isn't it? Saying "huh, looks like we have much to learn, here's what we're already doing about it" is honest and constructive, but sends a signal of weakness and defensiveness to people not bent on a zealous quest for truth and self-improvement. Saying "meh, that guy doesn't know what he's talking about" would send the stronger social signal, but would not be constructive to the community actually improving as a result of the criticism.

Personally I prefer plunging ahead with the first approach. Both in the abstract for reasons I won't elaborate on, but especially in this particular case. SI is not in a position where its every word is scrutinized; it would actually be a huge win if it gets there. And if/when it does, there's a heck of a lot more damning stuff that can be used against it than an admission of past incompetence.

comment by Vaniver · 2012-05-20T18:16:16.400Z · score: 0 (0 votes) · LW · GW

sends a signal of weakness and defensiveness to people not bent on a zealous quest for truth and self-improvement.

I do not see why this should be a motivating factor for SI; to my knowledge, they advertise primarily to people who would endorse a zealous quest for truth and self-improvement.

comment by Polymeron · 2012-05-20T18:25:30.743Z · score: 2 (2 votes) · LW · GW

That subset of humanity holds considerably less power, influence and visibility than its counterpart; resources that could be directed to AI research and for the most part aren't. Or in three words: Other people matter. Assuming otherwise would be a huge mistake.

I took Wei_Dai's remarks to mean that Luke's response is public, and so can reach the broader public sooner or later; and when examined in a broader context, that it gives off the wrong signal. My response was that this was largely irrelevant, not because other people don't matter, but because of other factors outweighing this.

comment by Will_Newsome · 2012-05-11T03:47:13.877Z · score: 18 (18 votes) · LW · GW

Eliezer's comment makes me think that you, specifically, should consider collecting your criticisms and putting them in Main where Eliezer is more likely to see them and take the time to seriously consider them.

comment by Wei_Dai · 2012-05-12T18:44:22.825Z · score: 0 (0 votes) · LW · GW

I replied here.

comment by thomblake · 2012-05-11T19:34:11.643Z · score: 8 (12 votes) · LW · GW

I think it's unfair to take Eliezer's response as anything other than praise for this article. He noted already that he did not have time to respond properly.

And why even point out that a human's response to anything is "suboptimal"? It will be notable when a human does something optimal.

comment by faul_sname · 2012-05-11T22:22:58.233Z · score: 8 (8 votes) · LW · GW

We do, on occasion, come up with optimal algorithms for things. Also, "suboptimal" usually means "I can think of several better solutions off the top of my head", not "This solution is not maximally effective".

comment by magfrump · 2012-05-11T04:50:01.015Z · score: 7 (7 votes) · LW · GW

Luke's comment addresses the specific point that Holden made about changes in the organization given the change in leadership.

Holden said:

I'm aware that SI has relatively new leadership that is attempting to address the issues behind some of my complaints. I have a generally positive impression of the new leadership; I believe the Executive Director and Development Director, in particular, to represent a step forward in terms of being interested in transparency and in testing their own general rationality. So I will not be surprised if there is some improvement in the coming years, particularly regarding the last couple of statements listed above. That said, SI is an organization and it seems reasonable to judge it by its organizational track record, especially when its new leadership is so new that I have little basis on which to judge these staff.

Luke attempted to provide (for the reader) a basis on which to judge these staff members.

Eliezer's response was... characteristic of Eliezer? And also very short and coming at a busy time for him.

comment by Nebu · 2012-12-31T12:15:42.784Z · score: 0 (0 votes) · LW · GW

Eliezer's response was... characteristic of Eliezer? And also very short and coming at a busy time for him.

I think that's Wei_Dai's point, that these "characteristic" replies are fine if you're used to him, but are bad if you don't.

comment by magfrump · 2012-12-31T19:26:19.787Z · score: 1 (1 votes) · LW · GW

Yeah I mean, as time goes on I think more and more of Eliezer as being kind of a jerk. I thought Luke's post was good, and Eliezer's wasn't, but I also expected longer posts to be forthcoming (which they were).

comment by ChrisHallquist · 2012-05-11T03:58:27.300Z · score: 6 (6 votes) · LW · GW

I read Luke's comment just as "I'm aware these are issues and we're working on it." I didn't read him as "bragging" about the ones that have been solved. Eliezer's... I see the problem with. I initially read it as just commenting Holden on his high-quality article (which I agree was high-quality), but I can see it being read as backhanded at anyone else who's criticized SIAI.

comment by Paul Crowley (ciphergoth) · 2012-05-11T06:34:15.871Z · score: 4 (4 votes) · LW · GW

Are there other specific critiques you think should have made Eliezer's list, or is it that you think he should not have drawn attention to their absence?

comment by Wei_Dai · 2012-05-11T07:39:41.349Z · score: 25 (25 votes) · LW · GW

Are there other specific critiques you think should have made Eliezer's list, or is it that you think he should not have drawn attention to their absence?

Many of Holden's criticisms have been made by others on LW already. He quoted me in Objection 1. Discussion of whether Tool-AI and Oracle-AI are or are not safe have occurred numerous times. Here's one that I was involved in. Many people have criticized Eliezer/SI for not having sufficiently impressive accomplishments. Cousin_it and Silas Barta have questioned whether the rationality techniques being taught by SI (and now the rationality org) are really effective.

comment by [deleted] · 2012-05-11T08:39:33.341Z · score: 0 (0 votes) · LW · GW

I kind-of agree about Eliezer's comment, but Luke's doesn't sound like that to me.

comment by [deleted] · 2012-05-11T08:41:22.817Z · score: 5 (5 votes) · LW · GW

Retracted. I've just re-read Eliezer's comment more calmly, and it's not that bad either.

comment by private_messaging · 2012-05-11T07:13:02.044Z · score: 0 (8 votes) · LW · GW

It's the correct signals. The incompetents inherently signal incompetence, the competence can't be faked beyond superficial level (and faking competence is all about signalling that you are sure you are competent). The lack of feedback is inherent in the assumption behind 'we are sending wrong signal' rather than 'maybe, we really are incompetent'.

comment by lukeprog · 2012-05-11T22:13:23.370Z · score: 37 (55 votes) · LW · GW

This post is highly critical of SIAI — both of its philosophy and its organizational choices. It is also now the #1 most highly voted post in the entire history of LessWrong — higher than any posts by Eliezer or myself.

I shall now laugh harder than ever when people try to say with a straight face that LessWrong is an Eliezer-cult that suppresses dissent.

comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2012-05-12T14:36:01.873Z · score: 13 (13 votes) · LW · GW

Either I promoted this and then forgot I'd done so, or someone else promoted it - of course I was planning to promote it, but I thought I'd planned to do so on Tuesday after the SIAIers currently running a Minicamp had a chance to respond, since I expected most RSS subscribers to the Promoted feed to read comments only once (this is the same reason I wait a while before promoting e.g. monthly quotes posts). On the other hand, I certainly did upvote it the moment I saw it.

comment by lukeprog · 2012-05-12T17:23:12.488Z · score: 2 (2 votes) · LW · GW

Original comment now edited; I wasn't aware anyone besides you might be promoting posts.

comment by JackV · 2012-05-12T09:29:41.141Z · score: 11 (11 votes) · LW · GW

I agree (as a comparative outsider) that the polite response to Holden is excellent. Many (most?) communities -- both online communities and real-world organisations, especially long-standing ones -- are not good at it for lots of reasons, and I think the measured response of evaluating and promoting Holden's post is exactly what LessWrong members would hope LessWrong could do, and they showed it succeeded.

I agree that this is good evidence that LessWrong isn't just an Eliezer-cult. (The true test would be if Elizier and another long-standing poster were dismissive to the post, and then other people persuaded them otherwise. In fact, maybe people should roleplay that or something, just to avoid getting stuck in an argument-from-authority trap, but that's a silly idea. Either way, the fact that other people spoke positively, and Elizier and other long-standing posters did too, is a good thing.)

However, I'm not sure it's as uniquely a victory for the rationality of LessWrong as it sounds. In responose to srdiamond, Luke quoted tenlier saying "[Holden's] critique mostly consists of points that are pretty persistently bubbling beneath the surface around here, and get brought up quite a bit. Don't most people regard this as a great summary of their current views, rather than persuasive in any way?" To me, that suggests that Holden did a really excellent job expressing these views clearly and persuasively. However, it suggests that previous people had tried to express something similar, but it hadn't been expressed well enough to be widely accepted, and people reading had failed to sufficiently apply the dictum of "fix your opponents' arguments for them". I'm not sure if that's true (it's certainly not automatically true), but I suspect it might be. What do people think?

If there's any truth to it, it suggests one good answer to the recent post http://lesswrong.com/lw/btc/how_can_we_get_more_and_better_lw_contrarians (whether that was desirable in general or not) would be, as a rationalist exercise for someone familiar with/to the community and good at writing rationally, to take a survey of contrarian views on the topic that people on the community may have had but not been able to express, and don't worry about showmanship like pretending to believe it yourself, but just say "I think what some people think is [well-expressed argument]. Do you agree that's fair? If so, do I and other people think they have a point?" Whether or not that argument is right it's still good to engage with it if many people are thinking it.

comment by pleeppleep · 2012-05-12T17:30:48.301Z · score: 4 (4 votes) · LW · GW

Third highest now. Eliezer just barely gets into the top 20.

comment by MarkusRamikin · 2012-05-17T07:56:56.387Z · score: 3 (3 votes) · LW · GW

It is also now the 3rd most highly voted post

1st.

At this point even I am starting to be confused.

comment by TheOtherDave · 2012-05-17T16:30:45.183Z · score: 1 (1 votes) · LW · GW

Can you articulate the nature of your confusion?

comment by MarkusRamikin · 2012-05-17T16:46:56.363Z · score: 7 (9 votes) · LW · GW

I suppose it's that I naively expect, when opening the list of top LW posts ever, to see ones containing the most impressive or clever insights into rationality.

Not that I don't think Holden's post deserves a high score for other reasons. While I am not terribly impressed with his AI-related arguments, the post is of the very highest standards of conduct, of how to have a disagreement that is polite and far beyond what is usually named "constructive".

comment by TheOtherDave · 2012-05-17T17:17:18.661Z · score: 4 (4 votes) · LW · GW

(nods) Makes sense.

My own primary inference from the popularity of this post is that there's a lot of uncertainty/disagreement within the community about the idea that creating an AGI without an explicit (and properly tuned) moral structure constitutes significant existential risk, but that the social dynamics of the community cause most of that uncertainty/disagreement to go unvoiced most of the time.

Of course, there's lots of other stuff going on as well that has little to do with AGI or existential risk, and a lot to do with the social dynamics of the community itself.

comment by [deleted] · 2012-06-14T13:38:11.937Z · score: 0 (0 votes) · LW · GW

Maybe. I upvoted it because it will have (and has had) the effect of improving SI's chances.

comment by aceofspades · 2012-06-07T20:36:37.221Z · score: 2 (2 votes) · LW · GW

Some people who upvoted the post may think it is one of the best-written and most important examples of instrumental rationality on this site.

comment by [deleted] · 2012-05-12T09:43:31.417Z · score: 2 (4 votes) · LW · GW

I wish I could upvote this ten times.

comment by brazil84 · 2013-02-13T23:20:00.490Z · score: 1 (3 votes) · LW · GW

Well perhaps the normal practice is cult-like and dissent-suppressing and this is an atypical break. Kind of like the fat person who starts eating salad instead of nachos while he watches football. And congratulates himself on his healthy eating even though he is still having donuts for breakfast and hamburgers and french fries for lunch.

Seems to me the test for suppression of dissent is not when a high-status person criticizes. The real test is when someone with medium or low status speaks out.

And my impression is that lesswrong does have problems along these lines. Not as bad as other discussion groups, but still.

comment by MarkusRamikin · 2012-05-12T10:17:46.750Z · score: 0 (0 votes) · LW · GW

5th

Looks like 3rd now. As impressed as I am with the post, at this point I'm a little surprised.

comment by Robin · 2012-05-15T11:30:17.383Z · score: -2 (2 votes) · LW · GW

But LW isn't reflective of SI, most of the people that voted on this article have no affiliation with SI. So the high number of upvotes is less reflective of SI welcoming criticism than LW being dissatisfied with the organization of SI.

Furthermore, this post's criticism of Eliezer's research less strong than its criticism of SI's organization . SI has always been somewhat open to criticism of its organizational structure and many of the current leadership of SI has criticized the organizational structure at some point. But who criticize Eliezer's research do not manage to rise in SI's research division and generally aren't well received even on LW (Roko).

Lastly, laughing at somebody when they call your organization a cult is not a convincing argument, they're more likely to think of your organization as a cult (at least they will think you are arrogant).

comment by private_messaging · 2012-05-12T07:16:58.292Z · score: -2 (8 votes) · LW · GW

How's about you also have a critical discussion of 'where can be we wrong and how do we make sure we are actually competent' and 'can we figure out what the AI will actually do, using our tools?' instead of 'how do we communicate our awesomeness better' and 'are we communicating our awesomeness right' ?

This post is something that can't be suppressed without losing big time, and you not suppressing it is only a strong evidence that you are not completely stupid (which is great).

comment by XiXiDu · 2012-05-12T08:54:07.280Z · score: -4 (42 votes) · LW · GW

I shall now laugh harder than ever when people try to say with a straight face that LessWrong is an Eliezer-cult that suppresses dissent.

Holden does not disagree with most of the basic beliefs that SI endorses. Which I think is rather sad and why I don't view him as a real critic. And he has been very polite.

Here is the impolite version:

If an actual AI researcher would have written a similar post, someone who actually tried to build practical systems and had some economic success, not one of those AGI dreamers. If such a person would write a similar post and actually write in a way that they feel, rather than being incredible polite, things would look very different.

The trust is that you are incredible naive when it comes to technological progress. That recursive self-improvement is nothing more than a row of English words, a barely convincing fantasy. That expected utility maximization is practically unworkable, even for a superhuman intelligence. And that the lesswrong.com sequences are not original or important but merely succeed at drowning out all the craziness they include by a huge amount of unrelated clutter and an appeal to the rationality of the author.

What you call an "informed" critic is someone who shares most of your incredible crazy and completely unfounded beliefs.

Worst of all, you are completely unconvincing and do not even notice it because there are so many other people who are strongly and emotionally attached to the particular science fiction scenarios that you envision.

comment by Dolores1984 · 2012-05-14T05:39:28.670Z · score: 11 (13 votes) · LW · GW

If such a person would write a similar post and actually write in a way that they feel, rather than being incredible polite, things would look very different.

I'm assuming you think they'd come in, scoff at our arrogance for a few pages, and then waltz off. Disregarding how many employed machine learning engineers also do side work on general intelligence projects, you'd probably get the same response from automobile engineer, someone with a track record and field expertise, talking to the Wright Brothers. Thinking about new things and new ideas doesn't automatically make you wrong.

That recursive self-improvement is nothing more than a row of English words, a barely convincing fantasy.

Really? Because that's a pretty strong claim. If I knew how the human brain worked well enough to build one in software, I could certainly build something smarter. You could increase the number of slots in working memory. Tweak the part of the brain that handles intuitive math to correctly deal with orders of magnitude. Improve recall to eidetic levels. Tweak the brain's handling of probabilities to be closer to the Bayesian ideal. Even those small changes would likely produce a mind smarter than any human being who has ever lived. That, plus the potential for exponential subjective speedup, is already dangerous. And that's assuming that the mind that results would see zero new insights that I've missed, which is pretty unlikely. Even if the curve bottoms out fairly quickly, after only a generation or two that's STILL really dangerous.

Worst of all, you are completely unconvincing and do not even notice it because there are so many other people who are strongly and emotionally attached to the particular science fiction scenarios that you envision.

Really makes you wonder how all those people got convinced in the first place.

comment by Salemicus · 2012-05-14T13:43:26.282Z · score: 3 (5 votes) · LW · GW

If I knew how the human brain worked well enough to build one in software, I could certainly build something smarter.

This is totally unsupported. To quote Lady Catherine de Bourgh, "If I had ever learned [to play the piano], I should have become a great proficient."

You have no idea whether the "small changes" you propose are technically feasible, or whether these "tweaks" would in fact mean a complete redesign. For all we know, if you knew how the human brain worked well enough to build one in software, you would appreciate why these changes are impossible without destroying the rest of the system's functionality.

After all, it would appear that (say) eidetic recall would provide a fitness advantage. Given that humans lack it, there may well be good reasons why.

comment by TheOtherDave · 2012-05-14T14:43:52.811Z · score: 6 (6 votes) · LW · GW

"totally unsupported" seems extreme. (Though I enjoyed the P&P shoutout. I was recently in a stage adaptation of the book, so it is pleasantly primed.)

What the claim amounts to is the belief that:
a) there exist good design ideas for brains that human evolution didn't implement, and
b) a human capable of building a working brain at all is capable of coming up with some of them.

A seems pretty likely to me... at least, the alternative (our currently evolved brains are the best possible design) seems so implausible as to scarcely be worth considering.

B is harder to say anything clear about, but given our experience with other evolved systems, it doesn't strike me as absurd. We're pretty good at improving the stuff we were born with.

Of course, you're right that this is evidence and not proof. It's possible that we just can't do any better than human brains for thinking, just like it was possible (but turned out not to be true) that we couldn't do any better than human legs for covering long distances efficiently.

But it's not negligible evidence.

comment by Salemicus · 2012-05-14T18:17:14.001Z · score: 2 (2 votes) · LW · GW

I don't doubt that it's possible to come up with something that thinks better than the human brain, just as we have come up with something that travels better than the human leg. But to cover long distances efficiently, people didn't start by replicating a human leg, and then tweaking it. They came up with a radically different design - e.g. the wheel.

I don't see the evidence that knowing how to build a human brain is the key step in knowing how to build something better. For instance, suppose you could replicate neuron function in software, and then scan a brain map (Robin Hanson's "em" concept). That wouldn't allow you to make any of the improvements to memory, maths, etc, that Dolores suggests. Perhaps you could make it run faster - although depending on hardware constraints, it might run slower. If you wanted to build something better, you might need to start from scratch. Or, things could go the other way - we might be able to build "minds" far better than the human brain, yet never be able to replicate a human one.

But it's not just that evidence is lacking - Dolores is claiming certainty in the lack of evidence. I really do think the Austen quote was appropriate.

comment by Dolores1984 · 2012-05-14T20:40:55.343Z · score: 4 (4 votes) · LW · GW

To clarify, I did not mean having the data to build a neuron-by-neuron model of the brain. I meant actually understanding the underlying algorithms those slabs of neural tissue are implementing. Think less understanding the exact structure of a bird's wing, and more understanding the concept of lift.

I think, with that level of understanding, the odds that a smart engineer (even if it's not me) couldn't find something to improve seem low.

comment by TheOtherDave · 2012-05-14T19:16:39.654Z · score: 4 (4 votes) · LW · GW

I agree that I might not need to be able to build a human brain in software to be able to build something better, as with cars and legs.

And I agree that I might be able to build a brain in software without understanding how to do it, e.g., by copying an existing one as with ems.

That said, if I understand the principles underlying a brain well enough to build one in software (rather than just copying it), it still seems reasonable to believe that I can also build something better.

comment by Jonathan_Graehl · 2012-05-13T00:52:00.924Z · score: 5 (7 votes) · LW · GW

I agree that that the tone on both sides is intentionally respectful, and that people here delude themselves if they imagine they aren't up for a bit of mockery from high status folks who don't have the patience to be really engage.

I agree that we don't really know what to expect from the first program that can meaningfully improve itself (including, I suppose, its self-improvement procedure) at a faster pace than human experts working on improving it. It might not be that impressive. But it seems likely to me that it will be a big deal, if ever we get there.

But you're being vague otherwise. Name a crazy or unfounded belief.

comment by XiXiDu · 2012-05-13T10:23:32.424Z · score: -1 (19 votes) · LW · GW

But you're being vague otherwise. Name a crazy or unfounded belief.

Holden asked me something similar today via mail. Here is what I replied:

You wrote in 'Other objections to SI's views':

Unlike the three objections I focus on, these other issues have been discussed a fair amount, and if these other issues were the only objections to SI's arguments I would find SI's case to be strong (i.e., I would find its scenario likely enough to warrant investment in).

It is not strong. The basic idea is that if you pull a mind at random from design space then it will be unfriendly. I am not even sure if that is true. But it is the strongest argument they have. And it is completely bogus because humans do not pull AGI's from mind design space at random.

Further, the whole case for AI risk is based on the idea that there will be a huge jump in capability at some point. Which I think is at best good science fiction, like faster-than-light propulsion, or antimatter weapons (when in doubt that it is possible in principle).

The basic fact that an AGI will most likely need something like advanced nanotechnology to pose a risk, which is itself an existential risk, hints at a conjunction fallacy. We do not need AGI to then use nanotechnology to wipe us out, nanotechnology is already enough if it is possible at all.

Anyway, it feels completely ridiculous to talk about it in the first place. There will never be a mind that can quickly and vastly improve itself and then invent all kinds of technological magic to wipe us out. Even most science fiction books avoid that because it sounds too implausible.

I have written thousands of words about all this and never got any convincing reply. So if you have any specific arguments, let me know.

They say what what I write is unconvincing. But given the amount of vagueness they use to protect their beliefs, my specific criticisms basically amount to a reductio ad absurdum. I don't even need to criticize them, they would have to support their extraordinary beliefs first or make them more specific. Yet I am able to come up with a lot of arguments that speak against the possibility they envision, without any effort and no knowledge of the relevant fields like complexity theory.

Here is a comment I received lately:

…in defining an AGI we are actually looking for a general optimization/compression/learning algorithm which when fed itself as an input, outputs a new algorithm that is better by some multiple. Surely this is at least an NP-Complete if not more problem. It may improve for a little bit and then hit a wall where the search space becomes intractable. It may use heuristics and approximations and what not but each improvement will be very hard won and expensive in terms of energy and matter. But no matter how much it tried, the cold hard reality is that you cannot compute an EXPonential Time algorithm in polynomial time unless (P=EXPTIME :S). A no self-recursive exponential intelligence theorem would fit in with all the other limitations (speed, information density, Turing, Gödel, uncertainties etc) the universe imposes.

If you were to turn IBM Watson gradually into a seed AI, at which point would it become an existential risk and why? They can't answer that at all. It is pure fantasy.

END OF EMAIL

For more see the following posts:

Some old posts:

See also:

If you believe I don't understand the basics, see:

Also:

There is a lot more, especially in the form of comments where I talk about specifics.

comment by Kaj_Sotala · 2012-05-13T21:57:25.171Z · score: 12 (16 votes) · LW · GW

The basic idea is that if you pull a mind at random from design space then it will be unfriendly. I am not even sure if that is true. But it is the strongest argument they have. And it is completely bogus because humans do not pull AGI's from mind design space at random.

I don't have the energy to get into an extended debate, but the claim that this is "the basic idea" or that this would be "the strongest argument" is completely false. A far stronger basic idea is the simple fact that nobody has yet figured out a theory of ethics that would work properly, which means that even that AGIs that were specifically designed to be ethical are most likely to lead to bad outcomes. And that's presuming that we even knew how to program them exactly.

This isn't even something that you'd need to read a hundred blog posts for, it's well discussed in both The Singularity and Machine Ethics and Artificial Intelligence as a Positive and Negative Factor in Global Risk. Complex Value Systems are Required to Realize Valuable Futures, too.

comment by XiXiDu · 2012-05-14T13:12:00.118Z · score: 0 (2 votes) · LW · GW

I did skim through the last paper. I am going to review it thoroughly at some point.

On first sight one of the problems is the whole assumption of AI drives. On the one hand you claim that an AI is going to follow its code, is its code (as if anyone would doubt causality). On the other hand you talk about the emergence of drives like unbounded self-protection. And if someone says that unbounded self-protection does not need to be part of an AGI, you simply claim that your definition of AGI will have those drives. Which allows you to arrive at your desired conclusion of AGI being an existential risk.

Another problem is the idea that an AGI will be a goal executor (I can't help but interpret that to be your position) when I believe that the very nature of artificial general intelligence implies the correct interpretation of "Understand What I Mean" and that "Do What I Mean" is the outcome of virtually any research. Only if you were to pull an AGI at random from mind design space could you possible arrive at "Understand What I Mean" without "Do What I Mean".

To see why look at any software product or complex machine. Those products are continuously improved. Where "improved" means that they become better at "Understand What I Mean" and "Do What I Mean".

There is no good reason to believe that at some point that development will suddenly turn into "Understand What I Mean" and "Go Batshit Crazy And Do What I Do Not Mean".

There are other problems with the paper. I hope I will find some time to write a review soon.

One problem for me with reviewing such papers is that I doubt a lot of underlying assumptions like that there exists a single principle of general intelligence. As I see it there will never be any sudden jump in capability. I also think that intelligence and complex goals are fundamentally interwoven. An AGI will have to be hardcoded, or learn, to care about a manifold of things. No simple algorithm, given limited computational resources, will give rise to the drives that are necessary to undergo strong self-improvement (if that is possible at all).

comment by Swimmer963 · 2012-05-14T05:18:14.974Z · score: 7 (7 votes) · LW · GW

Even most science fiction books avoid that because it sounds too implausible.

Not saying I particularly disagree with your other premises, but saying something can't be true because it sounds implausible is not a valid argument.

comment by Desrtopa · 2012-05-14T04:21:05.283Z · score: 5 (7 votes) · LW · GW

It is not strong. The basic idea is that if you pull a mind at random from design space then it will be unfriendly. I am not even sure if that is true. But it is the strongest argument they have. And it is completely bogus because humans do not pull AGI's from mind design space at random.

An AI's mind doesn't have to be pulled from design space at random to be disastrous. The primary issue that the SIAI has to grapple with (based on my understanding,) is that deliberately designing an AI that does what we would want it to do, rather than fulfilling proxy criteria in ways that we would not like at all, is really difficult. Even getting one to recognize "humans" as a category in a way that would be acceptable to us is a major challenge.

comment by jsteinhardt · 2012-05-15T03:44:36.861Z · score: -1 (3 votes) · LW · GW

Although it's worth pointing out that this is also an obstacle to AGI, since presumably an AI that did not understand what a human was would be pretty unintelligent. So I think it's unfair to claim this as a "friendliness" issue.

Note that I do think there are some important friendliness-related problems, but, assuming I understand your objection, this is not one of them.

comment by Desrtopa · 2012-05-15T03:57:15.404Z · score: 1 (1 votes) · LW · GW

An AI could be an extremely powerful optimizer without having a category for "humans" that mapped to our own. "Human," the way we conceive of it, is a leaky surface generalization.

A strong paperclip maximizer would understand humans as well as it had to to contend with us in its attempts to paperclip the universe, but it wouldn't care about us. And a strong optimizer programmed to maximize the values of "humans" would also probably understand us, but if we don't program into its values an actual category that maps to our conception of humans, it could perfectly well end up applying that understanding to, for example, tiling the universe with crash test dummies.

comment by jsteinhardt · 2012-05-15T04:31:59.049Z · score: 3 (3 votes) · LW · GW

How do you intend to build a powerful optimizer without having a method of representing (or of building a representation of) the concept of "human" (where "human" can be replaced with any complex concept, even probably paperclips)?

I agree that value specification is a hard problem. But I don't think the complexity of "human" is the reason for this, although it does rule out certain simple approaches like hard-coding values.

(Also, since your link seems to indicate you believe otherwise, I am fairly familiar with the content in the sequences. Apologies if this statement represents an improper inference.)

comment by Desrtopa · 2012-05-15T04:57:16.014Z · score: 4 (4 votes) · LW · GW

How do you intend to build a powerful optimizer without having a method of representing (or of building a representation of) the concept of "human" (where "human" can be replaced with any complex concept, even probably paperclips)?

If a machine can learn, empirically, exactly what humans are, on the most fundamental levels, but doesn't have any values associated with them, why should it need a concept of "human?" We don't have a category that distinguishes igneous rocks that are circular and flat on one side, but we can still recognize them and describe them precisely.

Humans are an unnatural category. Whether a fetus, an individual in a persistent vegetative state, an amputee, a corpse, an em or a skin cell culture fall into the category of "human" depends on value-sensitive boundaries. It's not necessarily because humans are so complex that we can't categorize them in an appropriate manner for an AI (or at least, not just because humans are complex,) it's because we don't have an appropriate formulation of the values that would allow a computer to draw the boundaries of the category in a way we'd want it to.

(I wasn't sure how familiar you were with the sequences, but in any case I figured it can't hurt to add links for anyone who might be following along who's not familiar.)

comment by Jonathan_Graehl · 2012-05-13T21:46:25.166Z · score: 4 (8 votes) · LW · GW

I've read most of that now, and have subscribed to your newsletter.

Reasonable people can disagree in estimating the difficulty of AI and the visibility/pace of AI progress (is it like hunting for a single breakthrough and then FOOM? etc).

I find all of your "it feels ridiculous" arguments by analogy to existing things interesting but unpersuasive.

comment by Dolores1984 · 2012-05-14T04:37:40.340Z · score: 3 (7 votes) · LW · GW

Anyway, it feels completely ridiculous to talk about it in the first place. There will never be a mind that can quickly and vastly improve itself and then invent all kinds of technological magic to wipe us out. Even most science fiction books avoid that because it sounds too implausible.

Says the wooly mammoth, circa 100,000 BC.

Sounding silly and low status and science-fictiony doesn't actually make it unlikely to happen in the real world.

comment by JoshuaZ · 2012-05-14T18:21:58.970Z · score: 4 (4 votes) · LW · GW

Especially when not many people want to read a science fiction book where humanity gets quickly and completely wiped out by a superior force. Even works where humans slowly die off due to their own problems (e.g. On the Beach) are uncommon.

comment by brahmaneya · 2012-05-14T04:03:44.876Z · score: 1 (3 votes) · LW · GW

Anyway, it feels completely ridiculous to talk about it in the first place. There will never be a mind that can quickly and vastly improve itself and then invent all kinds of technological magic to wipe us out. Even most science fiction books avoid that because it sounds too implausible

Do you acknowledge that :

  1. We will some day make an AI that is at least as smart as humans?
  2. Humans do try to improve their intelligence (rationality/memory training being a weak example, cyborg research being a better example, and im pretty sure we will soon design physical augmentations to improve our intelligence)

If you acknowledge 1 and 2, then that implies there can (and probably will) be an AI that tries to improve itself

comment by jsteinhardt · 2012-05-15T03:39:55.532Z · score: 4 (4 votes) · LW · GW

I think you missed the "quickly and vastly" part as well as the "and then invent all kinds of technological magic to wipe us out". Note I still think XiXiDu is wrong to be as confident as he is (assuming "there will never" implies >90% certainty), but if you are going to engage with him then you should engage with his actual arguments.

comment by Swimmer963 · 2012-05-14T05:20:32.814Z · score: 3 (3 votes) · LW · GW

And that the lesswrong.com sequences are not original or important but merely succeed at drowning out all the craziness they include by a huge amount of unrelated clutter and an appeal to the rationality of the author.

Name three examples? (Of 'craziness' specifically... I agree that there are frequent, and probably unecessary, "appeals to the rationality of the author".)

comment by [deleted] · 2012-05-14T05:33:13.951Z · score: -1 (7 votes) · LW · GW

Name three examples? (Of 'craziness' specifically... I agree that there are frequent, and probably unecessary, "appeals to the rationality of the author".)

XiXiDu may be too modest; he has some great examples on his blog.

comment by [deleted] · 2012-05-14T06:18:20.994Z · score: 2 (4 votes) · LW · GW

One wonders when or if XiXiDu will ever get over the Roko incident. Yes, it was a weird and possibly disproportionate response, but it was also years ago.

comment by JoshuaZ · 2012-05-14T20:16:29.923Z · score: 0 (2 votes) · LW · GW

Do we have any evidence that Eliezer's attitude or approach to that sort of thing has changed since then?

comment by [deleted] · 2012-05-14T20:24:04.031Z · score: 6 (6 votes) · LW · GW

Sure. His moderation activities over the last year or so have been far more... sunglasses... moderate.

comment by [deleted] · 2012-05-14T06:22:27.955Z · score: -7 (7 votes) · LW · GW

So said Newt Gingrich.

comment by [deleted] · 2012-05-14T06:27:19.606Z · score: 7 (7 votes) · LW · GW

Why yes, I do also believe that political figures are held to ridiculous conversational standards as well. It's a miracle they deign to talk to anyone.

comment by [deleted] · 2012-05-14T17:43:05.754Z · score: -7 (11 votes) · LW · GW

Name three examples? (Of 'craziness' specifically... I agree that there are frequent, and probably unecessary, "appeals to the rationality of the author".)

So, Swimmer 963, are those quotes crazy enough for you? (I hope you don't ask a question and neglect to comment on the answer.) What you do think? Anomalous?

Contrary to the impression the comments might convey, the majority don't come from the Roko incident. But as to that incident, the passage of time doesn't necessarily erase the marks of character. Romney is rightfully being held, feet to fire, for a group battering of another student while they attended high school--because such sadism is a trait of character and can't be explained otherwise. How would one explain Yudkowsky's paranoia, lack of perspective, and scapegoating--other than by positing a narcissistic personality structure?

Many LWers can't draw conclusions because they eschew the only tools for that purpose: psychology and excellent fiction. And the second is more important than the first.

comment by Swimmer963 · 2012-05-14T19:25:17.446Z · score: 9 (9 votes) · LW · GW

How would one explain Yudkowsky's paranoia, lack of perspective, and scapegoating--other than by positing a narcissistic personality structure?

I had in fact read a lot of those quotes before–although some of them come as a surprise, so thank you for the link. They do show paranoia and lack of perspective, and yeah, some signs of narcissism, and I would be certainly mortified if I personally ever made comments like that in public...

The Sequences as a whole do come across as having been written by an arrogant person, and that's kind of irritating, and I have to consciously override my irritation in order to enjoy the parts that I find useful, which is quite a lot. It's a simplification to say that the Sequences are just clutter, and it's extreme to call them 'craziness', too.

(Since meeting Eliezer in person, it's actually hard for me to believe that those comments were written by the same person, who was being serious about them... My chief interaction with him was playing a game in which I tried to make a list of my values, and he hit me with a banana every time I got writer's block because I was trying to be too specific, and made the Super Mario Brothers' theme song when I succeeded. It's hard making the connection that "this is the same person who seems to take himself way too seriously in his blog comments." But that's unrelated and doesn't prove anything in either direction.)

My main point is that criticizing someone who believes in a particular concept doesn't irrefutably damn that concept. You can use it as weak evidence, but not proof. Eliezer, as far as I know, isn't the only person who has thought extensively about Friendly AI and found it a useful concept to keep.

comment by [deleted] · 2012-05-14T22:06:08.587Z · score: -8 (14 votes) · LW · GW

The quotes aren't all about AI. A few:

Take metaethics, a solved problem: what are the odds that someone who still thought metaethics was a Deep Mystery could write an AI algorithm that could come up with a correct metaethics? I tried that, you know, and in retrospect it didn’t work.

Yudkowsky makes the megalomanic claim that he's solved the questions of metaethics. His solution: morality is the function that the brain of a fully informed subject computes to determine what's right. Laughable; pathologically arrogant.

Whoever knowingly chooses to save one life, when they could have saved two – to say nothing of a thousand lives, or a world – they have damned themselves as thoroughly as any murderer.

The most extreme presumptuousness about morality; insufferable moralism. Morality, as you were perhaps on the cusp of recognizing in one of your posts, Swimmer963, is a personalized tool, not a cosmic command line. See my "Why do what you "ought"?—A habit theory of explicit morality."

The preceding remark, I'll grant, isn't exactly crazy--just super obnoxious and creepy.

Science is built around the assumption that you’re too stupid and self-deceiving to just use Solomonoff induction. After all, if it was that simple, we wouldn’t need a social process of science right?

This is where Yudkowsky goes crazy autodidact bonkers. He thinks the social institution of science is superfluous, were everyone as smart as he. This means he can hold views contrary to scientific consensus in specialized fields where he lacks expert knowledge based on pure ratiocination. That simplicity in the information sense equates with parsimony is most unlikely; for one thing, simplicity is dependent on choice of language--an insight that should be almost intuitive to a rationalist. But noncrazy people may believe the foregoing; what they don't believe is that they can at the present time replace the institution of science with the reasoning of smart people. That's the absolutely bonkers claim Yudkowsky makes.

>

comment by Swimmer963 · 2012-05-15T00:04:47.644Z · score: 9 (9 votes) · LW · GW

The quotes aren't all about AI.

I didn't say they were. I said that just because the speaker for a particular idea comes across as crazy doesn't mean the idea itself is crazy. That applies whether all of Eliezer's "crazy statements" are about AI, or whether none of them are.

Whoever knowingly chooses to save one life, when they could have saved two – to say nothing of a thousand lives, or a world – they have damned themselves as thoroughly as any murderer.

The most extreme presumptuousness about morality; insufferable moralism.

Funny, I actually agree with the top phrase. It's written in an unfortunately preachy, minister-scaring-the-congregation-by-saying-they'll-go-to-Hell style, which is guaranteed to make just about anyone get defensive and/or go "ick!" But if you accept the (very common) moral standard that if you can save a life, it's better to do it than not to do it, then the logic is inevitable that if you have the choice of saving one lives or two lives, by your own metric it's morally preferable to save two lives. If you don't accept the moral standard that it's better to save one life than zero lives, then that phrase should be just as insufferable.

Science is built around the assumption that you’re too stupid and self-deceiving to just use Solomonoff induction. After all, if it was that simple, we wouldn’t need a social process of science right?

I decided to be charitable, and went and looked up the post that this was in: it's here. As far as I can tell, Eliezer doesn't say anything that could be interpreted as "science exists because people are stupid, and I'm not stupid, therefore I don't need science". He claims that scientific procedures compensates for people being unwilling to let go of their pet theories and change their minds, and although I have no idea if this goal was in the minds of the people who came up with the scientific method, it doesn't seem to be false that it accomplishes this goal.

comment by hairyfigment · 2012-05-15T03:36:13.943Z · score: 1 (1 votes) · LW · GW

Newton definitely wrote down his version of scientific method to explain why people shouldn't take his law of gravity and just add, "because of Aristotelian causes," or "because of Cartesian mechanisms."

comment by JoshuaZ · 2012-05-14T22:21:03.721Z · score: 8 (8 votes) · LW · GW

This is where Yudkowsky goes crazy autodidact bonkers. He thinks the social institution of science is superfluous, were everyone as smart as he. This means he can hold views contrary to scientific consensus in specialized fields where he lacks expert knowledge based on pure ratiocination.

Ok. I disagree with a large bit of the sequences on science and the nature of science. I've wrote a fair number of comments saying so. So I hope you will listen when I say that you are taking a strawman version of what Eliezer wrote on these issues, and it almost borders on something that I could only see someone thinking if they were trying to interpret Eliezer's words in the most negative fashion possible.

comment by Dolores1984 · 2012-05-14T22:13:46.050Z · score: 5 (5 votes) · LW · GW

His solution: morality is the function that the brain of a fully informed subject computes to determine what's right. Laughable; pathologically arrogant.

You either didn't read that sequence carefully, or are intentionally misrepresenting it.

He thinks the social institution of science is superfluous, were everyone as smart as he.

Didn't read that sequence carefully either.

That simplicity in the information sense equates with parsimony is most unlikely; for one thing, simplicity is dependent on choice of language--an insight that should be almost intuitive to a rationalist.

You didn't read that sequence at all, and probably don't actually know what simplicity means in an information-theoretic sense.

comment by [deleted] · 2012-05-14T22:22:24.378Z · score: 5 (5 votes) · LW · GW

That simplicity in the information sense equates with parsimony is most unlikely; for one thing, simplicity is dependent on choice of language--an insight that should be almost intuitive to a rationalist.

You didn't read that sequence at all, and probably don't actually know what simplicity means in an information-theoretic sense.

To be fair, that sequence doesn't really answer questions about choice-of-language; it took reading some of Solomonoff's papers for me to figure out what the solution to that problem is.

comment by JoshuaZ · 2012-05-14T22:37:06.537Z · score: 2 (2 votes) · LW · GW

it took reading some of Solomonoff's papers for me to figure out what the solution to that problem is.

There are a variety of proposed solutions. None of them seem perfect.

comment by [deleted] · 2012-05-14T22:40:54.174Z · score: 2 (2 votes) · LW · GW

I'm referring to encoding in several different languages, which makes it progressively more implausible that choice of language matters.

I agree that's not a perfect solution, but it's good enough for me.

comment by [deleted] · 2012-05-14T22:23:40.371Z · score: -3 (7 votes) · LW · GW

That's true; I admit I didn't read the sequence. I had a hard time struggling through the single summating essay. What I wrote was his conclusion. As Hanson wrote in the first comment to the essay I did read, Yudkowsky really should summarize the whole business in a few lines. Yudkowsky didn't get around to that, as far as I know.

The summation essay contained more than 7,000 words for the conclusion I quoted. Maybe the rest of the series contradicts what is patent in the essay I read.

I simply don't get the attraction of the sequences. An extraordinarily high ratio of filler to content; Yudkowsky seems to think that every thought along the way to his personal enlightenment is worth the public's time.

Asking that a critic read those sequences in their entirety is asking for a huge sacrifice; little is offered to show it's even close in being worth the misery of reading inept writing or the time.

comment by Dolores1984 · 2012-05-14T22:39:19.122Z · score: 4 (8 votes) · LW · GW

You know, the sequences aren't actually poorly written. I've read them all, as have most of the people here. They are a bit rambly in places, but they're entertaining and interesting. If you're having trouble with them, the problem might be on your end.

In any case, if you had read them, you'd know, for instance, that when Yudkowsky talks about simplicity, he is not talking about the simplicity of a given English sentence. He's talking about the combined complexity of a given Turing machine and the program needed to describe your hypothesis on that Turing machine.

comment by gwern · 2012-05-17T00:21:00.930Z · score: 4 (4 votes) · LW · GW

have most of the people here

http://lesswrong.com/lw/8p4/2011_survey_results/

89 people (8.2%) have never looked at the Sequences; a further 234 (32.5%) have only given them a quick glance. 170 people have read about 25% of the sequences, 169 (15.5%) about 50%, 167 (15.3%) about 75%, and 253 people (23.2%) said they've read almost all of them. This last number is actually lower than the 302 people who have been here since the Overcoming Bias days when the Sequences were still being written (27.7% of us).

  • 23% for 'almost all'
  • 39% have read > three-quarters
  • 54% have read > half
comment by Dolores1984 · 2012-05-17T00:31:35.036Z · score: 0 (0 votes) · LW · GW

My mistake. I'll remember that in the future.

comment by Bugmaster · 2012-05-14T23:01:02.808Z · score: 2 (2 votes) · LW · GW

In addition, there are places in the Sequences where Eliezer just states things as though he's dispensing wisdom from on high, without bothering to state any evidence or reasoning. His writing is still entertaining, of course, but still less than persuasive.

comment by dlthomas · 2012-05-14T22:56:09.760Z · score: 2 (2 votes) · LW · GW

They are a bit rambly in places, but they're entertaining and interesting.

I also found this to be true.

comment by [deleted] · 2012-05-14T22:42:40.190Z · score: 2 (2 votes) · LW · GW

as have most of the people here.

I'm pretty sure the 2011 survey puts this claim to the test, but I don't have the time to look it up.

comment by [deleted] · 2012-05-14T23:50:27.615Z · score: 1 (7 votes) · LW · GW

You know, the sequences aren't actually poorly written. I've read them all, as have most of the people here. They are a bit rambly in places, but they're entertaining and interesting. If you're having trouble with them, the problem might be on your end.

The problem is partly on my end, for sure; obviously, I find rambling intolerable in Internet writing, and I find it in great abundance in the sequences. You're more tolerant of rambling, and you're entertained by Yudkowsky's. I also think he demonstrates mediocre literary skills when it comes to performances like varying his sentence structure. I don't know what you think of that. My guess is you don't much care; maybe it's a generational thing.

I'm intrigued by what enjoyment readers here get from Yudkowsky's sequences. Why do you all find interesting what I find amateurish and inept? Do we have vastly different tastes or standards, or both? Maybe it is the very prolixity that makes the writing appealing in founding a movement with religious overtones. Reading Yudkowsky is an experience comparable to reading the Bible.

As a side issue, I'm dismayed upon finding that ideas I had thought original to Yudkowsky were secondhand.

Of course I understand simplicity doesn't pertain to simplicity in English! (Or in any natural language.) I don't think you understand the language-relativity issue.

comment by TheOtherDave · 2012-05-15T00:43:16.146Z · score: 4 (4 votes) · LW · GW

If you were willing to point me to two or three of your favorite Internet writers, whom you consider reliably enjoyable and interesting and so forth, I might find that valuable for its own sake, and might also be better able to answer your question in mutually intelligible terms.

comment by Swimmer963 · 2012-05-15T00:11:21.860Z · score: 2 (2 votes) · LW · GW

As a side issue, I'm dismayed upon finding that ideas I had thought original to Yudkowsky were secondhand.

Having to have original ideas is a very high standard. I doubt a single one of my posts contains a truly original idea, and I don't try–I try to figure out which ideas are useful to me, and then present why, in a format that I hope will be useful to others. Eliezer creates a lot of new catchy terms for pre-existing ideas, like "affective death spiral" for "halo effect." I like that.

His posts are also quite short, often witty, and generally presented in an easier-to-digest format than the journal articles I might otherwise have to read to encounter the not-new ideas. You apparently don't find his writing easy to digest or amusing in the same way I do.

comment by thomblake · 2012-05-15T17:51:59.448Z · score: 3 (3 votes) · LW · GW

like "affective death spiral" for "halo effect."

Affective death spiral is not the same thing as the Halo effect, though the halo effect (/ horns effect) might be part of the mechanism of affective death spiral.

comment by Swimmer963 · 2012-05-15T19:52:43.470Z · score: 3 (3 votes) · LW · GW

Agreed... I think the Halo effect is a sub-component of an affective death spiral, and "affective death spiral" is a term unique to LW [correct me if I'm wrong!], while 'Halo effect' isn't.

comment by JoshuaZ · 2012-05-15T02:18:45.650Z · score: 1 (1 votes) · LW · GW

As a side issue, I'm dismayed upon finding that ideas I had thought original to Yudkowsky were secondhand.

Are there specific examples? It seems to me that in most cases when he has a pre-existing idea he gives relevant sources.

comment by Normal_Anomaly · 2012-05-16T13:55:56.065Z · score: 1 (1 votes) · LW · GW

I don't know any specific examples of secondhand ideas coming off as original (indeed, he often cites experiments from the H&B literature), but there's another possible source for the confusion. Sometimes Yudkowsky and somebody else come up with ideas independently, and those aren't cited because Yudkowsky didn't know they existed at the time. Drescher and Quine are two philosophers who have been mentioned as having some of the same ideas as Yudkowsky, and I can confirm the former from experience.

comment by Bugmaster · 2012-05-15T00:28:36.625Z · score: 1 (1 votes) · LW · GW

I'm intrigued by what enjoyment readers here get from Yudkowsky's sequences. Why do you all find interesting what I find amateurish and inept?

I find his fictional interludes quite entertaining, because they are generally quite lively, and display a decent amount of world-building -- which is one aspect of science fiction and fantasy that I particularly enjoy. I also enjoy the snark he employs when trashing opposing ideas, especially when such ideas are quite absurd. Of course, the snark doesn't make his writing more persuasive -- just more entertaining.

he demonstrates mediocre literary skills when it comes to performances like varying his sentence structure

I know I'm exposing my ignorance here, but I'm not sure what this means; can you elaborate ?

comment by Randaly · 2012-05-15T19:35:32.107Z · score: 2 (2 votes) · LW · GW

Asking that a critic read those sequences in their entirety is asking for a huge sacrifice; little is offered to show it's even close in being worth the misery of reading inept writing or the time.

Indeed, the sequences are long. I'm not sure about the others here, but I've never asked anybody to "read the sequences."

But I don't even know how to describe the arrogance required to believe that you can dismiss somebody's work as "crazy," "stupid," "megalomanic," "laughably, pathologically arrogant," "bonkers," and "insufferable" without having even read enough of what you're criticizing the get an accurate understanding of it.

ETA: Edited in response to fubarobfusco, who brought up a good point.

comment by fubarobfusco · 2012-05-15T23:00:08.515Z · score: 3 (3 votes) · LW · GW

That's a fully general argument against criticizing anything without having read all of it, though. And there are some things you can fairly dismiss without having read all of. For instance, I don't have to read every page on the Time Cube site to dismiss it as crazy, stupid, pathologically arrogant, and so on.

comment by nshepperd · 2012-05-15T08:08:24.713Z · score: 1 (1 votes) · LW · GW

The reason EY wrote an entire sequence on metaethics is precisely because without the rest of the preparation people such as you who lack all that context immediately veer off course and start believing that he's asserting the existence (or non-existence) of "objective" morality, or that morality is about humans because humans are best or any other standard philosophical confusion that people automatically come up with whenever they think about ethics.

Of course this is merely a communication issue. I'd love to see a more skilled writer present EY's metaethical theory in a shorter form that still correctly conveys the idea, but it seems to be very difficult (especially since even half the people who do read the sequence still come away thinking it's moral relativism or something).

comment by Mass_Driver · 2012-05-15T03:04:41.471Z · score: 4 (4 votes) · LW · GW

I read your post on habit theory, and I liked it, but I don't think it's an answer to the question "What should I do?"

It's interesting to say that if you're an artist, you might get more practical use out of virtue theory, and if you're a politician, you might get more practical use out of consequentialism. I'm not sure who it is that faces more daily temptations to break the rules than the rest of us; bankers, I suppose, and maybe certain kinds of computer security experts.

Anyway, saying that morality is a tool doesn't get you out of the original need to decide which lifestyle you want in the first place. Should I be an artist, or a politician, or a banker? Why? Eliezer's answer is that there are no shortcuts and no frills here; you check and see what your brain says about what you 'should' do, and that's all there is to it. This is not exactly a brilliant answer, but it may nevertheless be the best one out there. I've never yet heard a moral theory that made more sense than that, and believe me, I've looked.

It's reasonable to insist that people put their conclusions in easily digestible bullet points to convince you to read the rest of what they've written...but if, noting that there are no such bullet points, you make the decision not to read the body text -- you should probably refrain from commenting on the body text. A license to opt-out is not the same thing as a license to offer serious criticism. Eliezer may be wrong, but he's not stupid, and he's not crazy. If you want to offer a meaningful critique of his ideas, you'll have to read them first.

comment by [deleted] · 2012-05-15T04:10:35.453Z · score: -4 (8 votes) · LW · GW

but if, noting that there are no such bullet points, you make the decision not to read the body text -- you should probably refrain from commenting on the body text. A license to opt-out is not the same thing as a license to offer serious criticism. Eliezer may be wrong, but he's not stupid, and he's not crazy.

This is sound general advice, but at least one observation makes this situation exceptional: Yudkowsky's conclusions about ethics are never summarized in terms that contradict my take. I don't think your rendition, for example, contradicts mine. I'm certainly not surprised to hear his position described the way you describe it:

Anyway, saying that morality is a tool doesn't get you out of the original need to decide which lifestyle you want in the first place. Should I be an artist, or a politician, or a banker? Why? Eliezer's answer is that there are no shortcuts and no frills here; you check and see what your brain says about what you 'should' do, and that's all there is to it.

Now, I don't think the decision of whether to be an artist, politician, or banker is a moral decision. It isn't one you make primarily because of what's ethically right or wrong. To the extent you do (and in the restricted sense that you do), your prior moral habits are your only guide.

But we're looking at whether Yudkowsky's position is intellectually respectable, not whether objective morality--which he's committed to but I deny--exists. To say we look at what our brain says when we're fully informed says essentially that we seek a reflective equilibrium in solving moral problems. So far so good. But it goes further in saying brains compute some specific function that determines generally when individuals reach that equilibrium. Leaving aside that this is implausible speculation, requiring that the terms of moral judgments be hardwired--and hardwired identically for each individual--it also simply fails to answer Moore's open question, although Yudkowsky claims he has that answer. There's nothing prima facie compelling ethically about what our brains happen to tell us is moral; no reason we should necessarily follow our brains' hardwiring. I could consistently choose to consider my brain's hardwired moralisms maladaptive or even despicable holdovers from the evolutionary past that I choose to override as much as I can.

Robin Hanson actually asked the right question. If what the brain computes is moral, what does it correspond to that makes it moral? Unless you think the brain is computing a fact about the world, you can't coherently regard its computation as "accurate." But if not, what makes it special and not just a reflex?

I do feel a bit guilty about criticizing Yudkowsky without reading all of him. But he seems to express his ideas at excessive and obfuscating length, and if there were more to them, I feel somewhat confident I'd come across his answers. It isn't as though I haven't skimmed many of these essays. And his answers would certainly deserve some reflection in his summation essay.

There's no question Yudkowsky is no idiot. But he has some ideas that I think are stupid--like his "metaethics"--and he expresses them in a somewhat "crazy" manner, exuding grandiose self-confidence. Being surrounded and discussing mostly with people who agree with him is probably part of the cause.

comment by Furcas · 2012-05-15T05:25:01.782Z · score: 6 (10 votes) · LW · GW

As someone who has read Eliezer's metaethics sequence, let me say that what you think his position is, is only somewhat related to what it actually is; and also, that he has answered those of your objections that are relevant.

It's fine that you don't want to read 30+ fairly long blog posts, especially if you dislike the writing style. But then, don't try to criticize what you're ignorant about. And no, openly admitting that you haven't read the arguments you're criticizing, and claiming that you feel guilty about it, doesn't magically make it more acceptable. Or honest.

comment by JoshuaZ · 2012-05-15T17:35:10.923Z · score: 2 (4 votes) · LW · GW

One doesn't need to have read the whole Bible to criticize it. But the Bible is a fairly short work, so an even more extreme example might be better: one doesn't need to have read the entire Talmud to criticize it.

comment by [deleted] · 2012-05-15T17:01:08.918Z · score: -4 (10 votes) · LW · GW

It's fine that you don't want to read 30+ fairly long blog posts, especially if you dislike the writing style. But then, don't try to criticize what you're ignorant about. And no, openly admitting that you haven't read the arguments you're criticizing, and claiming that you feel guilty about it, doesn't magically make it more acceptable. Or honest.

It's hardly "dishonest" to criticize a position based on a 7,000-word summary statement while admitting you haven't read the whole corpus! You're playing with words to make a moralistic debating point: dishonesty involves deceit, and everyone has been informed of the basis for my opinions.

Consider the double standard involved. Yudkowsky lambasts "philosophers" and their "confusions"--their supposedly misguided concerns with the issues other philosophers have commented on to the detriment of inquiry. Has Yudkowsky read even a single book by each of the philosophers he dismisses?

In a normal forum, participants supply the arguments supposedly missed by critics who are only partially informed. Here there are vague allusions to what the Apostle Yudkowsky (prophet of the Singularity God) "answered" without any substance. An objective reader will conclude that the Prophet stands naked; the prolixity is probably intended to discourage criticism.

comment by Kaj_Sotala · 2012-05-15T19:11:38.667Z · score: 5 (5 votes) · LW · GW

I think the argument you make in this comment isn't a bad one, but the unnecessary and unwarranted "Apostle Yudkowsky (prophet of the Singularity God)" stuff amounts to indirectly insulting the people you're talking with and, makes them far less likely to realize that you're actually also saying something sensible. If you want to get your points across, as opposed to just enjoying a feeling of smug moral superiority while getting downvoted into oblivion, I strongly recommend leaving that stuff out.

comment by [deleted] · 2012-05-24T17:26:04.156Z · score: 0 (4 votes) · LW · GW

Thanks for the advice, but my purpose—given that I'm an amoralist—isn't to enjoy a sense of moral superiority. Rather, to test a forum toward which I've felt ambivalent for several years, mainly for my benefit but also for that of any objective observers.

Strong rhetoric is often necessary in an unreceptive forum because it announces that the writer considers his criticisms fundamental. If I state the criticisms neutrally, something I've often tried, they are received as minor—like the present post. They may even be voted up, but they have little impact. Strong language is appropriate in expressing severe criticisms.

How should a rationalist forum respond to harsh criticism? It isn't rational to fall prey to the primate tendency to in-group thinking by neglecting to adjust for any sense of personal insult when the group leader is lambasted. Judging by reactions, the tendency to in-group thought is stronger here than in many forums that don't claim the mantle of rationalism. This is partly because the members are more intelligent than in most other forums, and intelligence affords more adept self-deception. This is why it is particularly important for intelligent people to be rationalists but only if they honestly strive to apply rational principles to their own thinking. Instead, rationality here serves to excuse participants' own irrationality. Participants simply accept their own tendencies to reject posts as worthless because they contain matter they find insulting. Evolutionary psychology, for instance, here serves to produce rationalizations rather than rationality. (Overcoming Bias is a still more extreme advocacy of this perversion of rationalism, although the tendency isn't expressed in formal comment policies.)

"Karma" means nothing to me except as it affects discourse; I despise even the term, which stinks of Eastern mysticism. I'm told that the karma system of incentives, which any rationalist should understand vitally affects the character of discussion, was transplanted from reddit. How is a failure to attend to the vital mechanics of discussion and incentives rational? Laziness? How could policies so essential be accorded the back seat?

Participants, I'm told, don't question the karma system because it works. A rationalist doesn't think that way. He says, "If a system of incentives introduced without forethought and subject to sound criticisms (where even its name is an insult to rationality) produces the discourse that we want, then something must be wrong with what we want!" What's wanted is the absence of any tests of ideology by fundamental dissent.

I think the argument you make in this comment isn't a bad one, but the unnecessary and unwarranted "Apostle Yudkowsky (prophet of the Singularity God)" stuff amounts to indirectly insulting the people you're talking with and, makes them far less likely to realize that you're actually also saying something sensible. If you want to get your points across, as opposed to just enjoying a feeling of smug moral superiority while getting downvoted into oblivion, I strongly recommend leaving that stuff out.

comment by JoshuaZ · 2012-05-15T17:51:57.958Z · score: 2 (2 votes) · LW · GW

Consider the double standard involved. Yudkowsky lambasts "philosophers" and their "confusions"--their supposedly misguided concerns with the issues other philosophers have commented on to the detriment of inquiry. Has Yudkowsky read even a single book by each of the philosophers he dismisses?

Some of them are simply not great writers. Hegel for example is just awful- the few coherent ideas in Hegel are more usefully described by other later writers. There's also a strange aspect to this in that you are complaining about Eliezer not having read books while simultaneously defending your criticism of Eliezer's metaethics positions without having read all his posts. Incidentally, if one wants to criticize Eliezer's level of knowledge of philosophy, a better point is not so much the philosophers that he criticizes without reading, but rather his lack of knowledge of relevant philosophers that Eliezer seems unaware of, many of whom would agree with some of his points. Quine and Lakatos are the most obvious ones.

Here there are vague allusions to what the Apostle Yudkowsky (prophet of the Singularity God) "answered" without any substance. An objective reader will conclude that the Prophet stands naked; the prolixity is probably intended to discourage criticism.

I strongly suspect that your comments would be responded to more positively if they didn't frequently end with this sort of extreme rhetoric that has more emotional content than rational dialogue. It is particularly a problem because on theLW interface, the up/down buttons are at the end of everything one has read, so what the last sentences say may have a disproportionate impact on whether people upvote or downvote and what they focus on in their replies.

Frankly, you have some valid points, but they are getting lost in the rhetoric. We know that you think that LW pattern matches to religion. Everyone gets the point. You don't need to repeat that every single time you make a criticism.

comment by Strange7 · 2012-05-21T22:38:14.456Z · score: 2 (4 votes) · LW · GW

I could consistently choose to consider my brain's hardwired moralisms maladaptive or even despicable holdovers from the evolutionary past that I choose to override as much as I can.

And you would be making the decision to override with... what, your spleen?

comment by [deleted] · 2012-05-21T23:47:07.444Z · score: 0 (6 votes) · LW · GW

Another part of my brain--besides the part computing the morality function Yudkowsky posits.

Surely you can't believe Yudkowsky simply means whatever our brain decides is "moral"--and that he offers that as a solution to anything?

comment by Strange7 · 2012-05-22T09:36:11.485Z · score: 1 (1 votes) · LW · GW

I'm not saying he's right, just that your proposed alternative isn't even wrong.

comment by Strange7 · 2012-05-22T06:50:22.998Z · score: 0 (0 votes) · LW · GW

I'm not saying he's right, I'm saying your proposed alternative isn't even wrong.

comment by thomblake · 2012-05-15T17:09:20.777Z · score: 2 (2 votes) · LW · GW

Science is built around the assumption that you’re too stupid and self-deceiving to just use Solomonoff induction.

He thinks the social institution of science is superfluous, were everyone as smart as he.

This is obviously false. Yudkowsky does not claim to be able to do Solomonoff induction in his head.

In general, when Yudkowsky addresses humanity's faults, he is including himself.

comment by [deleted] · 2012-05-15T17:22:13.579Z · score: -5 (7 votes) · LW · GW

Point taken.

But Yudkowsky says "built around the assumption that you're too stupid... to just use ..."

If Solomonoff induction can't easily be used in place of science, why does the first sentence imply the process is simple: you just use it?

You've clarified what Yudkowsky does not mean. But what does he mean? And why is it so hard to find out? This is the way mystical sects retain their aura while actually saying little.

comment by nshepperd · 2012-05-15T17:38:29.857Z · score: 5 (5 votes) · LW · GW

"You're too stupid and self-deceiving to just use Solomonoff induction" ~ "If you were less stupid and self deceiving you'd be able to just use Solomonoff induction" + "but since you are in fact stupid and self-deceiving, instead you have to use the less elegant approximation Science"

That was hard to find out?

comment by [deleted] · 2012-05-15T17:43:05.909Z · score: -7 (7 votes) · LW · GW

Actually, yes, because of the misleading signals in the inept writing. But thank you for clarifying.

Conclusion: The argument in written in a crazy fashion, but it really is merely stupid. There is no possible measure of simplicity that isn't language relative. How could there be?

comment by Randaly · 2012-05-15T19:49:22.368Z · score: 3 (3 votes) · LW · GW

You seem to be confusing "language relative" with "non-mathematical." Kolmogorov Complexity is "language-relative," if I'm understanding you right; specifically, it's relative (if I'm using the terminology right?) to a Turing Machine. This was not relevant to Eliezer's point, so it was not addressed.

(Incidentally, this is a perfect example of you "hold{ing} views contrary to scientific consensus in specialized fields where {you} lack expert knowledge based on pure ratiocination," since Kolmogorov Complexity is "one of the fundamental concepts of theoretical computer science", you seemingly lack expert knowledge since you don't recognize these terms, and your argument seems to be based on pure ratiocination.)

comment by CuSithBell · 2012-05-15T17:46:01.866Z · score: 1 (1 votes) · LW · GW

When I read that line for the first time, I understood it. Between our two cases, the writing was the same, but the reader was different. Thus, the writing cannot be the sole cause of our different outcomes.

comment by JoshuaZ · 2012-05-15T17:57:38.655Z · score: 4 (4 votes) · LW · GW

Well, if a substantial fraction of readers read something differently or can't parse it, it does potentially reflect a problem with the writing even if some of the readers, or even most readers, do read it correctly.

comment by CuSithBell · 2012-05-15T18:02:00.134Z · score: 3 (3 votes) · LW · GW

Absolutely. I intended to convey that if you don't understand something, that the writing is misleading and inept is not the only possible reason. srdiamond is speaking with such confidence that I felt safe tabling further subtleties for now.

comment by [deleted] · 2012-05-16T18:38:29.877Z · score: -13 (13 votes) · LW · GW

The philosophizing of inept, verbose writers like Yudkowsky can be safely dismissed based solely on their incompetence as writers. For a succinct defense of this contention, see my "Can bad writers be good thinkers? Part 1 of THE UNITY OF LANGUAGE AND THOUGHT" OR see the 3-part "Writing & Thought series" — all together, fewer than 3,000 words.

comment by gwern · 2012-05-16T19:01:44.202Z · score: 14 (14 votes) · LW · GW

I believe what you wrote because you used so much bolding.

comment by [deleted] · 2012-05-21T20:02:21.660Z · score: -6 (6 votes) · LW · GW

Way to deflect attention from substance to form. Exemplary rationality!

comment by thomblake · 2012-05-21T20:10:12.204Z · score: 3 (3 votes) · LW · GW

I can't tell which way your sarcasm was supposed to cut.

The obvious interpretation is that you think rationality is somehow hindered by paying attention to form rather than substance, and the "exemplary rationality" was intended to be mocking.

But your comment being referenced was an argument that form has something very relevant to say about substance, so it could also be that you were actually praising gwern for practicing what you preach.

comment by gwern · 2012-05-21T20:30:41.664Z · score: 1 (3 votes) · LW · GW

I choose to interpret it as praise, and receive a warm fuzzy feeling.

comment by [deleted] · 2012-05-16T19:20:53.809Z · score: 2 (2 votes) · LW · GW

I read your three-part series. Your posts did not substantiate the claim "good thinking requires good writing." Your second post slightly increased my belief in the converse claim, "good thinkers are better-than-average writers," but because the only evidence you provided was a handful of historical examples, it's not very strong evidence. And given how large the population of good thinkers, good writers, bad thinkers, and bad writers is relative to your sample, evidence for "good thinking implies good writing" is barely worth registering as evidence for "good writing implies good thinking."

comment by JoshuaZ · 2012-05-14T18:12:00.909Z · score: 8 (8 votes) · LW · GW

Romney is rightfully being held, feet to fire, for a group battering of another student while they attended high school--because such sadism is a trait of character and can't be explained otherwise.

I was going to upvote your comment until I got to this point. Aside from the general mindkilling, this looks like the fundamental attribution error, and moreover, we all know that people do in fact mature and change. Bringing up external politics is not helpul in a field where there's already concern that AI issues may be becoming a mindkilling subject themselves on LW. Bringing up such a questionable one is even less useful.

comment by metaphysicist · 2012-05-14T18:21:42.024Z · score: -1 (7 votes) · LW · GW

That's LW "rationality" training for you--"fundamental error of attribution" out of context--favored because it requires little knowledge and training in psychology. Such thinking would preclude any investigation of character. (And there are so many taboos! How do you all tolerate the lockstep communication required here?)

Paul Meehl, who famously studied clinical versus statistical prediction empirically, noted that even professionals, when confronted by instance of aberrant behavior, are apt to call it within normal range when it clearly isn't. Knowledge of the "fundamental error of attribution" alone is the little bit of knowledge that's worse than total ignorance.

Ask yourself honestly whether you would ever or have ever done anything comparable to what Yudkowsky did in the Roko incident or what Romney did in the hair cutting incident.

You can't dismiss politics just because it kills some people's minds, when so much of the available information and examples come from politics. (There are other reasons, but that's the main one here.) Someone who can't be rational about politics simply isn't a good rationalist. You can't be a rationalist about the unimportant things and rationalist about the important ones--yet call yourself a rationalist overall.

comment by NancyLebovitz · 2012-05-15T03:55:23.777Z · score: 9 (9 votes) · LW · GW

I'm sure I wouldn't have done what Romney did, and not so sure about whether I would have done what Yudkowsky did. Romney wanted to hurt people for the fun of it. Yudkowsky was trying to keep people from being hurt, regardless of whether his choice was a good one.

comment by metaphysicist · 2012-05-15T04:48:59.281Z · score: 2 (2 votes) · LW · GW

That's a reasonable answer.

comment by JoshuaZ · 2012-05-14T20:04:36.890Z · score: 6 (8 votes) · LW · GW

It seems almost unfair to criticize something as a problem of LW rationality when in your second paragraph you note that professionals do the same thing.

Ask yourself honestly whether you would ever or have ever done anything comparable to what Yudkowsky did in the Roko incident or what Romney did in the hair cutting incident.

I'm not sure. A while ago, I was involved in a situation where someone wanted to put personal information of an individual up on the internet knowing that that person had an internet stalker who had a history of being a real life stalker for others. The only reason I didn't react pretty close to how Eliezer reacted in the quoted incident is that I knew that the individual in question was not going to listen to me and would if anything have done the opposite of what I wanted. In that sort of context, Eliezer's behavior doesn't seem to be that extreme. Eliezer's remarks involve slightly more caps than I think I would use in such a circumstance, but the language isn't that different.

This does connect to another issue though- the scale in question of making heated comments on the internet as opposed to traumatic bullying, are different. The questions I ask myself for what it would take to do something similar to what Eliezer did are very different than the same questions for the Romney incident.

Your basic statement does it seem have some validity. One could argue that the Romney matter reflects the circumstances where he was at the time, and what was considered socially acceptable as forms of interaction or establishing dominance hierarchies. Through most of human history, that sort of behavior would probably be considered fairly tame. But this is a weak argument- even if it was due to the circumstances that Romney was in at the time, there's no question that those were his formative years, and thus could plausibly have had a permanent impact on his moral outlook.

You can't dismiss politics just because it kills some people's minds, when so much of the available information and examples come from politics.

The problem is that even as relevant examples come from politics, those are precisely the examples that people are least likely to agree actually demonstrate the intended point in question. For example, in this case, many people who aren't on the left will downplay the Romney bullying. Given that I'm someone who dislikes Romney (both in terms of personality and in terms of policy) and am not convinced that this is at all fair, using such a controversial example seems unwise. Even if one needs to use political examples, one can use examples from 10 or 15 or 30 years ago that are well known but have had their tribalness diminish in time. For example, in this context one could use a variety of examples connected to Richard Nixon.

Someone who can't be rational about politics simply isn't a good rationalist. You can't be a rationalist about the unimportant things and rationalist about the important ones--yet call yourself a rationalist overall.

Well, we can acknowledge that we're better at being rational in some areas than we are in others. Frankly, I wouldn't mind and for reasons essentially similar to your remark would endorse some amount of reduction of the no-politics rule here. Where that becomes a problem is when one tries to connect politics to other potentially controversial issues.

comment by [deleted] · 2012-05-14T18:40:34.376Z · score: 1 (1 votes) · LW · GW

what Romney did in the hair cutting incident

What's that about? (PM me if it's still taboo.)

comment by Normal_Anomaly · 2012-05-16T14:01:05.105Z · score: 2 (2 votes) · LW · GW

When Mitt Romney was in high school, he and some friends bullied a kid who looked (and later turned out to be) homosexual. At one point, Romney and some others grabbed the guy, held him down, and cut off a bunch of his hair with scissors.

comment by Barry_Cotter · 2012-05-13T09:44:20.180Z · score: -1 (7 votes) · LW · GW

Why do you continue to participate? Almost all of the cool stuff that high status people agree is plausible is available elsewhere.

comment by [deleted] · 2012-05-11T22:45:49.965Z · score: -4 (22 votes) · LW · GW

The point, such as it is, would better have been left implied. Now, it's subject to explicit scrutiny, and it must be found wanting. Consider what would have happened had Yudkowsky not shown exceptional receptivity to this post: he would have blatantly proven his critics right. The knowledge and reputation of the poster is unimpeachable.

The more significant fact is that these criticisms were largely unknown to the community. As Will Newsome implied, this is because the critical posts--lacking the high-status credential of this poster--remained in discussion and were almost ignored.

The majority's intolerance for dissent is manifested mostly in its refusal to acknowledge it. Dissent is cabined to Discussion. It only gets noticed when the dissenter becomes frustrated and violates group norms. Then it gets voted down, but it still gets noticed and commented on. This is a malfunctioning reinforcement system, but maybe its the best possible. Still, it's irrational to deny all in-group bias in LukeProg's cheerleading fashion--in an instance where the absence of evidence (here, of bias) truly does not offer anything substantial in the way of evidence of lack of bias, to elicit LukeProg's smug laughter.

After all, even the lead poster held off until now in voicing his opinion.

comment by lukeprog · 2012-05-11T23:00:36.568Z · score: 11 (13 votes) · LW · GW

The more significant fact is that these criticisms were largely unknown to the community.

LWer tenlier disagrees, saying:

[Holden's] critique mostly consists of points that are pretty persistently bubbling beneath the surface around here, and get brought up quite a bit. Don't most people regard this as a great summary of their current views, rather than persuasive in any way? In fact, the only effect I suspect this had on most people's thinking was to increase their willingness to listen to Karnofsky in the future if he should change his mind.

Also, you said:

Dissent is cabined to Discussion.

Luckily, evidence on the matter is easy to find. As counter-evidence I present: Self-improvement or shiny distraction, SIAI an examination, Why we can't take expected value estimates literally, Extreme rationality: it's not that great, Less Wrong Rationality and Mainstream Philosophy, and the very post you are commenting on. Many of these are among the most upvoted posts ever.

Moreover, the editors rarely move posts from Main to Discussion. The posters themselves decide whether to post in Main or Discussion.

comment by Rain · 2012-05-11T23:31:20.250Z · score: 0 (0 votes) · LW · GW

Also Should I believe what the SIAI claims? and the many XiXiDu posts linked therein, like What I would like the SIAI to publish.

comment by [deleted] · 2012-05-11T23:55:04.024Z · score: -6 (14 votes) · LW · GW

I had a post moved from main to Discussion just today: before it had accumulated any negative votes, so I think you're probably misinformed about editorial practices.But I don't want to use my posts as evidence; the charge of bias would be hard to surmount. What's plainly evident is that posters are reluctant to post to the Main area except by promotion.

You're evidence is unpersuasive because you don't weigh it against the evidence to the contrary. One good example to the contrary more than counter-balances it, since the point isn't that no dissent is tolerated, not even that some dissent isn't welcomed, but only that there are some irrational boundaries.

One is the quasi-ban on politics. Here is a comment that garnered almost 800 responses and was voted up 37. Why wasn't it promoted? I bitterly disagree with the poster; so I'm not biased by my views. But the point is that it is a decidedly different view, one generating great interest, but the subject would not be to the liking of the editors.

Of course, it lacked the elaborateness--dare I say, the prolixity--of a typical top-level post. But this "scholarly" requirement is part of the process of soft censorship. The post--despite my severe disagreement with it--is a more significant intellectual contribution than many of the top-level posts, such as some of the second-hand scholarship.

[And I have to add: observe that the present discussion is already being downvoted at my first comment. I predict the same for this post in record time What does that mean?]

comment by Normal_Anomaly · 2012-05-13T03:13:11.205Z · score: 4 (4 votes) · LW · GW

Here is a comment that garnered almost 800 responses and was voted up 37. Why wasn't it promoted?

Can comments be promoted? Perhaps the commenter should have been encouraged to turn his comment into a top-level post, but a moderator can't just change a comment into a promoted post with the same username. Also it would have split the discussion, so people might have been reluctant to encourage that.

As for people tending to post more in Discussion than Main, I read somewhere that Discussion has more readers. I for one read Discussion almost exclusively.

comment by [deleted] · 2012-05-12T00:11:40.567Z · score: -4 (12 votes) · LW · GW

It would advance this discussion if someone would explain the down votes. I await LukeProg's explanation of the present example of soft censorship.

comment by Rain · 2012-05-12T00:14:53.487Z · score: 9 (11 votes) · LW · GW

I downvoted you because you're wrong. For one, comments can't be promoted to main, only posts, and for two, plenty of opposition has garnerned a great deal of upvotes, as shown by the numerous links lukeprog provided.

For example, where do you get 'almost 800 responses' from? That comment (not post) only has 32 comments below it.

comment by [deleted] · 2012-05-12T00:40:30.065Z · score: -4 (8 votes) · LW · GW

Yes, I was wrong. But my point was correct. The 781 comments applied to the Main Post So:

  1. The topic was popular, like I said.

  2. The post could have been promoted!

But ask yourself, would you have been so harsh on a factual error had you agreed with the message? This is the way bias works, after all, by double standard more than outright discrimination. You could say I should have been more careful. But then, when you've learned not to expect a hearing, you're not so willing to jump the hoops. But it's your loss, if you're a rationalist and if you're losing input because dissenters find it's not worth their time.

As to LukeProg providing example demonstrating welcoming dissent: you couldn't have considered my counter-balancing evidence when you downvoted before taking the time even to explore the post to which the cited comment belongs.

To LukeProg: have I made my point about the limits of dissent at LW?

comment by Rain · 2012-05-12T00:19:38.502Z · score: 7 (11 votes) · LW · GW

Posts which contain factual inaccuracies along with meta-discussion of karma effects are often downvoted.

comment by [deleted] · 2012-05-12T00:47:26.421Z · score: -4 (12 votes) · LW · GW

I've addressed factual inaccuracies in another comment. But as for discussing karma effects--that wasn't extraneous whining but was at the heart of the discussion. If you downvote discussion of karma--like you did--simply for mentioning it, even where relevant, then you effectively soft-censor any discussion of karma. How is that rational?

LukeProg: What do you say about the grounds on which downvotes are issued for dissenting matter. Isn't it clear that this is a bias LW doesn't want to talk about; perhaps altogether doesn't want to discuss its own biases?

comment by Rain · 2012-05-12T01:20:24.798Z · score: 3 (7 votes) · LW · GW

If you downvote discussion of karma--like you did--simply for mentioning it, even where relevant, then you effectively soft-censor any discussion of karma. How is that rational?

I don't do that; I only downvote when it's combined with incorrect facts. Which I'm tempted to do for this statement: "like you did--simply for mentioning it", since you're inferring my motivations, and once again incorrect.

comment by [deleted] · 2012-05-12T01:29:57.989Z · score: -4 (14 votes) · LW · GW

Look, Rain, this is an Internet ongoing discussion. Nobody says everything precisely right. The point is that you would hardly be so severe on someone unless you disagreed strongly. You couldn't be, because nobody would satisfy your accuracy demands. The kind of nitpicking you engage in your post would ordinarily lead you to be downvoted--and you should be, although I won't commit the rudeness of so doing in a discussion.

The point wasn't that you downvote when the only thing wrong with the comment is discussion of karma. It was that you treat discussion of karma as an unconditional wrong. So you exploited weaknesses in my phrasing to ignore what I think was obviously the point--that marking down for the bare mention of karma (even if it doesn't produce a downvote in each case) is an irrational policy, when karma is at the heart of the discussion. There's no rational basis for throwing it in as an extra negative when the facts aren't right.

You're looking for trivial points to pick to downvote and to ignore the main point, which was your counting mention of karma a negative, without regard to the subject, is an irrational policy. If we were on reversed sides, your nitpicking and evasion would itself be marked down. As matters stand, you don't even realize you're acting in a biased fashion, and readers either don't know or don't care.

Is that rational? Shouldn't a rationalist community be more concerned with criticizing irrationalities in its own process?

comment by shminux · 2012-05-12T02:12:45.206Z · score: 11 (13 votes) · LW · GW

Having been a subject of both a relatively large upvote and a relatively large downvote in the last couple of weeks, I still think that the worst thing one can do is to complain about censorship or karma. The posts and comments on any forum aren't judged on their "objective merits" (because there is no such thing), but on its suitability for the forum in question. If you have been downvoted, your post deserves it by definition. You can politely inquire about the reasons, but people are not required to explain themselves. As for rationality, I question whether it is rational to post on a forum if you are not having fun there. Take it easy.

comment by [deleted] · 2012-05-13T14:33:28.223Z · score: -1 (5 votes) · LW · GW

The posts and comments on any forum aren't judged on their "objective merits" (because there is no such thing), but on its suitability for the forum in question. If you have been downvoted, your post deserves it by definition.

First, you're correct that it's irrational to post to a forum you don't enjoy. I'll work on decreasing my akrasia.

But it's hard not to comment on a non sequitur like the above. (Although probably futile because one who's really not into a persuasion effort won't do it well.) That posts are properly evaluated by suitability to the forum does not imply that a downvoted post deserves the downvote by definition! That's a maladaptive view of the sort I'm amazed is so seldom criticized on this forum. Your view precludes (by definition yet) criticism of the evaluators' biases, which do not advance the forum's purpose. You would eschew not only absolute merits but also any objective consideration of the forum's function.

A forum devoted to rationality, to be effective and honest, must assess and address the irrationalities in its own functioning. (This isn't always "fun.") To define a post that should be upvoted as one that is upvoted constitutes an enormous obstacle to rational function.

comment by Rain · 2012-05-12T01:45:59.251Z · score: 5 (9 votes) · LW · GW

The point is that you would hardly be so severe on someone unless you disagreed strongly.

I disagree; a downvote is not 'severe'.

The kind of nitpicking you engage in your post would ordinarily lead you to be downvoted

I disagree; meta-discussions often result in many upvotes.

It was that you treat discussion of karma as an unconditional wrong.

I do not, and have stated as much.

There's no rational basis for throwing it in as an extra negative when the facts aren't right.

If there is no point in downvoting incorrect facts, then I wonder what the downvote button is for.

You're looking for trivial points to pick to downvote and to ignore the main point,

I disagree; your main point is that you are being unfairly downvoted, along with other posts critical of SI being downvoted unfairly, which I state again is untrue, afactual, incorrect, a false statement, a lie, a slander, etc.

Is that rational? Shouldn't a rationalist community be more concerned with criticizing irrationalities in its own process?

Questioning the rationality of meta-meta-voting patterns achieves yet another downvote from me. Sorry.

comment by Endovior · 2012-05-12T05:20:49.818Z · score: -4 (10 votes) · LW · GW

I don't follow your reasoning, here. Having read this particular thread, it does seem as though you are, in fact, going out of your way to criticize and downvote srdiamond. Yes, he has, in fact, made a few mistakes. Given, however, that the point of this post in general is about dissenting from the mainstream opinions of the LW crowd, and given the usual complaints about lack of dissent, I find your criticism of srdiamond strange, to say the least. I have, accordingly, upvoted a number of his comments.

comment by Endovior · 2012-05-12T07:03:40.698Z · score: 0 (10 votes) · LW · GW

As expected, my previous comment was downvoted almost immediately.

This would, for reference, be an example of the reason why some people believe LW is a cult that suppresses dissent. After all, it's significantly easier to say that you disagree with something than it is to explain in detail why you disagree; just as it's far easier to state agreement than to provide an insightful statement in agreement. Nonetheless, community norms dictate that unsubstantiated disagreements get modded down, while unsubstantiated agreements get modded up. Naturally, there's more of the easy disagreement then the hard disagreement... that's natural, since this is the Internet, and anyone can just post things here.

In any event, though, the end result is the same; people claim to want more dissent, but what they really mean is that they want to see more exceptionally clever and well-reasoned dissent. Any dissent that doesn't seem at least half as clever as the argument it criticizes seems comparatively superfluous and trivial, and is marginalized at best. And, of course, any dissent that is demonstrably flawed in any way is aggressively attacked. That really is what people mean by suppression of dissent. It doesn't really mean 'downvoting arguments which are clever, but with which you personally disagree'... community norms here are a little better then that, and genuinely good arguments tend to get their due. In this case, it means, 'downvoting arguments which aren't very good, and with which you personally disagree, when you would at the same time upvote arguments that also aren't very good, but with which you agree'. Given the nature of the community norms, someone who expresses dissent regularly, but without taking the effort to make each point in an insightful and terribly clever way, would tend to be downvoted repeatedly, and thus discouraged from making more dissent in the future... or, indeed, from posting here at all.

I don't know if there's a good solution to the problem. I would be inclined to suggest that, like with Reddit, people not downvote without leaving an explanation as to why. For instance, in addition to upvoting some of srdiamond's earlier comments, I have also downvoted some of Rain's, because a number of Rain's comments in this thread fit the pattern of 'poor arguments that support the community norms', in the same sense that srdiamond's fit the pattern of 'poor arguments that violate the community norms'; my entire point here is that, in order to cultivate more intelligent dissent, there should be more of the latter and less of the former.

comment by Paul Crowley (ciphergoth) · 2012-05-12T09:13:25.094Z · score: 3 (7 votes) · LW · GW

I downvote any post that says "I expect I'll get downvoted for this, but..." or "the fact that I was downvoted proves I'm right!"

comment by CuSithBell · 2012-05-12T15:49:13.503Z · score: 6 (6 votes) · LW · GW

I'm fond of downvoting "I dare you to downvote this!"

comment by Endovior · 2012-05-12T14:31:11.670Z · score: 2 (8 votes) · LW · GW

So, in other words, you automatically downvote anyone who explicitly mentions that they realize they are violating community norms by posting whatever it is they are posting, but feels that the content of their post is worth the probable downvotes? That IS fairly explicitly suppressing dissent, and I have downvoted you for doing so.

comment by JoshuaZ · 2012-05-12T14:47:27.724Z · score: 8 (8 votes) · LW · GW

I don't think it is suppression of dissent per se. It is more annoying behavior- it implies caring a lot about the karma system, and it is often not even the case when people say that they will actually get downvoted. If it is worth the probable downvote, then they can, you know, just take the downvote. If they want to point out that a view is unpopular they can just say that explicitly. It is also annoying to people like me, who are vocal about a number of issues that could be controversial here (e.g. criticizing Bayesianism, cryonics,, and whether intelligence explosions would be likely) and get voted up. More often than not, when someone claims they are getting downvoted for having unpopular opinions, they are getting downvoted in practice for having bad arguments or for being uncivil.

There are of course exceptions to this rule, and it is disturbing to note that the exceptions seem to be coming more common (see for example, this exchange where two comments are made with about the same quality of argument and about the same degree of uncivility- ("I'm starting to hate that you've become a fixture here." v. "idiot" - but one of the comments is at +10 and the other is at -7.) Even presuming that there's a real disagreement in quality or correctness of the arguments made, this suggests that uncivil remarks are tolerated more when people agree with the rest of the claim being made. That's problematic. And this exchange was part of what prompted me to earlier suggest that we should be concerned if AGI risk might be becoming a mindkiller here. But even given that, issues like this seem not at all common.

Overall, if one needs to make a claim about one is going to be downvoted, one might even be correct, but it will often not be for the reasons one thinks it is.

comment by Endovior · 2012-05-12T15:23:27.211Z · score: 3 (5 votes) · LW · GW

I don't think it's so much 'caring a lot about the karma system' per se, so much as the more general case of 'caring about the approval and/or disapproval of one's peers'. The former is fairly abstract, but the latter is a fairly deep ancestral motivation.

Like I said before, it's clearly not much in the way of suppression. That said, given that, barring rare incidents of actual moderation, it is the only 'suppression' that occurs here, and since there is a view among various circles that there there is, in fact, suppression of dissent, and since people on the site frequently wonder why there are not more dissenting viewpoints here, and look for ways to find more... it is important to look at the issue in great depth, since it's clearly an issue which is more significant than it seems on the surface.

comment by [deleted] · 2012-05-13T19:31:19.106Z · score: -1 (7 votes) · LW · GW

[P]eople on the site frequently wonder why there are not more dissenting viewpoints here, and look for ways to find more... it is important to look at the issue in great depth, since it's clearly an issue which is more significant than it seems on the surface.

Exactly right. But a group that claims to be dedicated to rationality loses all credibility when participants not only abstain from considering this question but adamantly resist it. The only upvote you received for your post—which makes this vital point—is mine.

This thread examines HoldenKarnofsky's charge that SIAI isn't exemplarily rational. As part of that examination, the broader LW environment on which it relies is germane. That much has been granted by most posters. But when the conversation reaches the touchstone of how the community expresses its approval and disapproval, the comments are declared illegitimate and downvoted (or if the comments are polite and hyper-correct, at least not upvoted).

The group harbors taboos. The following subjects are subject to them: the very possibility of nonevolved AI; karma and the group's own process generally (an indespensable discussion ); and politics. (I've already posted a cite showing how the proscription on politics works, using an example the editors' unwillingness to promote the post despite receiving almost 800 comments).

These defects in the rational process of LW help sustain Kardofsky's argument that SIAI is not to be recommended based on the exemplary rationality of its staff and leadership. They are also the leadership of LW, and they have failed by refusing to lead the forum toward understanding the biases in its own process. They have fostered bias by creating the taboo on politics, as though you can rationally understand the world while dogmatically refusing even to consider a big part of it—because it "kills" your mind.

P.S. Thank you for the upvotes where you perceived bias.

comment by XiXiDu · 2012-05-12T17:16:21.529Z · score: 2 (10 votes) · LW · GW

...AGI risk might be becoming a mindkiller here...

Nah. If there is a mindkiller then it is the reputation system. Some of the hostility is the result of the overblown ego and attitude of some of its proponents and their general style of discussion. They created an insurmountable fortress that shields them from any criticism:

Troll: If you are so smart and rational, why don't you fund yourself? Why isn't your organisation sustainable?

SI/LW: Rationality is only aimed at expected winning.

Troll: But you don't seem to be winning yet. Have you considered the possibility that your methods are suboptimal? Have you set yourself any goals, that you expect to be better at than less rational folks, to test your rationality?

SI/LW: Rationality is a caeteris paribus predictor of success.

Troll: Okay, but given that you spend a lot of time on refining your rationality, you must believe that it is worth it somehow? What makes you think so then?

SI/LW: We are trying to create a friendly artificial intelligence implement it and run the AI, at which point, if all goes well, we Win. We believe that rationality is very important to achieve that goal.

Troll: I see. But there surely must be some sub-goals that you anticipate to be able to solve and thereby test if your rationality skills are worth the effort?

SI/LW: Many of the problems related to navigating the Singularity have not yet been stated with mathematical precision, and the need for a precise statement of the problem is part of the problem.

Troll: Has there been any success in formalizing one of the problems that you need to solve?

SI/LW: There are some unpublished results that we have had no time to put into a coherent form yet.

Troll: It seems that there is no way for me to judge if it is worth it to read up on your writings on rationality.

SI/LW: If you want to more reliably achieve life success, I recommend inheriting a billion dollars or, failing that, being born+raised to have an excellent work ethic and low akrasia.

Troll: Awesome, I'll do that next time. But for now, why would I bet on you or even trust that you know what you are talking about?

SI/LW: We spent a lot of time on debiasing techniques and thought long and hard about the relevant issues.

Troll: That seems to be insufficient evidence given the nature of your claims and that you are asking for money.

SI/LW: We make predictions. We make statements of confidence of events that merely sound startling. You are asking for evidence we couldn't possibly be expected to be able to provide, even given that we are right.

Troll: But what do you anticipate to see if your ideas are right, is there any possibility to update on evidence?

SI/LW: No, once the evidence is available it will be too late.

Troll: But then why would I trust you instead of those experts who tell me that you are wrong?

SI/LW: You will soon learn that your smart friends and experts are not remotely close to the rationality standards of SI/LW, and you will no longer think it anywhere near as plausible that their differing opinion is because they know some incredible secret knowledge you don't.

Troll: But you have never achieved anything when it comes to AI, why would I trust your reasoning on the topic?

SI/LW: That is magical thinking about prestige. Prestige is not a good indicator of quality.

Troll: You won't convince me without providing further evidence.

SI/LW: That is a fully general counterargument you can use to discount any conclusion.

comment by Jonathan_Graehl · 2012-05-13T00:57:23.806Z · score: 1 (1 votes) · LW · GW

Troll: You won't convince me without providing further evidence.

SI/LW: That is a fully general counterargument you can use to discount any conclusion.

The last exchange was hilarious. This is parody, right?

comment by CuSithBell · 2012-05-12T15:47:44.163Z · score: 2 (2 votes) · LW · GW

More often than not, when someone claims they are getting downvoted for having unpopular opinions, they are getting downvoted in practice for having bad arguments or for being uncivil.

Bears repeating.

comment by Rain · 2012-05-12T14:42:40.050Z · score: -5 (7 votes) · LW · GW

Downvoted for downvoting downvoting of downvoting of downvoting.

If you do the same to this comment, we can enter a stable loop!

comment by [deleted] · 2012-05-12T07:39:05.943Z · score: 3 (3 votes) · LW · GW

First, none of this dissent has been suppressed in any real sense. It's still available to be read and discussed by those who desire reading and discussing such things. The current moderation policy has currently only kicked in when things have gotten largely out of hand -- which is not the case here, yet.

Second, net karma isn't a fine enough tool to express amount of detail you want it to express. The net comment on your previous comment is currently -2; congrats, you've managed to irritate less than a tenth of one percent of LW (presuming the real karma is something like -2/+0 or -3/+1)!

Third, the solution you propose hasn't been implemented anywhere that I know of. Reddit's suggested community norm (which does not apply to every subreddit) suggests considering posting constructive criticism only when one thinks it will actually help the poster improve. That's not really the case much of the time, at least on the subreddits I frequent, and it's certainly not the case often here.

Fourth, the solution you propose would, if implemented, decrease the signal-to-noise ratio of LW further.

Fifth, reddit's suggested community norm also says "[Don't c]omplain about downvotes on your posts". Therefore, I wonder how much you really think reddit is doing the community voting norm thing correctly.

comment by Endovior · 2012-05-12T14:22:58.711Z · score: 2 (4 votes) · LW · GW

First; downvoted comments are available to be read, yes; but the default settings hide comments with 2 or more net downvotes. This is enough to be reasonably considered 'suppression'. It's not all that much suppression, true, but it is suppression... and it is enough to discourage dissent. Actual moderation of comments is a separate issue entirely, and not one which I will address here.

Second; when I posted my reply, and as of this moment, my original comment was at -3. I agree; net karma isn't actually a huge deal, except that it is, as has been observed, the most prevalent means by which dissent is suppressed. In my case, at least, 'this will probably get downvoted' feels like a reason to not post something. Not much of a reason, true, but enough of one that I can identify the feeling of reluctance.

Third; on the subreddits I follow (admittedly a shallow sampling), I have frequently seen comments explaining downvotes, sometimes in response to a request specifically for such feedback, but just as often not. I suspect that this has a lot to do with the "Down-voting? Please leave an explanation in the comments." message that appears when mousing over the downvote icon. I am aware that this is not universal across Reddit, but on the subreddits I follow, it seems to work reasonably well.

Fourth; I agree that this is a possible result. Like I said before, I'm not sure if there is a good solution to this problem, but I do feel that it'd result in a better state then that which currently exists, if people would more explicitly explain why they downvote when they choose to do so. That said, given that downvoted comments are hidden from default view anyway, and that those who choose to do so can easily ignore such comments, I don't think it'd have all that much effect on the signal/noise ratio.

Fifth; on the subreddits I follow, it seems as though there is less in the way of complaints about downvotes, and more honest inquiries as to why a comment has been downvoted; such questions seem to usually receive honest responses. This may be anomalous within Reddit as a whole; as I said before, my own experience with Reddit is a shallow sampling.

comment by [deleted] · 2012-05-13T20:57:36.632Z · score: -1 (5 votes) · LW · GW

I don't know if there's a good solution to the problem. I would be inclined to suggest that, like with Reddit, people not downvote without leaving an explanation as to why. For instance, in addition to upvoting some of srdiamond's earlier comments, I have also downvoted some of Rain's, because a number of Rain's comments in this thread fit the pattern of 'poor arguments that support the community norms', in the same sense that srdiamond's fit the pattern of 'poor arguments that violate the community norms'; my entire point here is that, in order to cultivate more intelligent dissent, there should be more of the latter and less of the former.

Perhaps the solution is not to worry so much about my bad contrarian arguments being downvoted as to assure that bad "establishment" arguments are downvoted—as in Rain's case, they aren't. Regurgitation of arguments others have repeatedly stated should also be downvoted, no matter how good the arguments.

The reason to think an emphasis on more criticism of Rain rather than less criticism of me is that after I err, it's a difficult argument to establish that my error wasn't serious enough to avoid downvote. But when Rain negligently or intentionally misses the entire point, there's less question that he isn't benefiting the discussion. It's easier to convict of fallacy than to defend based on the fallacy being relatively trivial. There's a problem in that the two determinations are somewhat inter-related, but it doesn't eliminate the contrast.

Increasing the number of downvotes would deflate the significance of any single downvote and would probably foster more dissent. This balance may be subject to easy institutional control. Posters are allotted downvotes based on their karma, while the karma requirements for upvotes are easily satisfied, if they exist. This amounts to encouraging upvotes relative to downvotes, with the result that many bad posts are voted up and some decent posts suffer the disproportionate wrath of extreme partisans. (Note that Rain, a donor, is a partisan of SIAI.)

The editors should experiment with increasing the downvote allowance. I favor equal availability of downvotes and upvotes as optimal (but this should be thought through more carefully).

comment by Nornagest · 2012-05-13T15:14:08.454Z · score: 3 (3 votes) · LW · GW

Consider what would have happened had Yudkowsky not shown exceptional receptivity to this post: he would have blatantly proven his critics right.

After turning this statement around in my head for a while I'm less certain than I was that I understand its thrust. But assuming you mean those critics pertinent to lukeprog's post, i.e. those claiming LW embodies a cult of personality centered around Eliezer -- well, no. Eliezer's reaction is in fact almost completely orthogonal to that question.

If you receive informed criticism regarding a project you're heavily involved in, and you react angrily to it, that shows nothing more or less than that you handle criticism poorly. If the community around you locks ranks against your critics, either following your example or (especially) preemptively, then you have evidence of a cult of personality.

That's not what happened here, though. Eliezer was fairly gracious, as was the rest of the community. Now, that is not by itself behavior typically associated with personality cults, but before we start patting ourselves on the back it's worth remembering that certain details of timing and form could still point back in the other direction. I'm pretty sick of the cult question myself, but if you're bound and determined to apply this exchange to it, that's the place you should be looking.

comment by fubarobfusco · 2012-05-18T02:49:22.003Z · score: 0 (0 votes) · LW · GW

Conservation of expected evidence may be relevant here.

comment by [deleted] · 2012-05-12T02:28:36.147Z · score: -18 (20 votes) · LW · GW

I shall now laugh harder than ever when people try to say with a straight face that LessWrong is an Eliezer-cult that suppresses dissent.

After I recently read that the lead poster was a major financial contributor to SIAI, I'd have to call LukeProg's argument disingenuous if not mendacious.

comment by CarlShulman · 2012-05-12T03:16:03.469Z · score: 12 (12 votes) · LW · GW

Rain (who noted that he is a donor to SIAI in a comment) and HoldenKarnofsky (who wrote the post) are two different people, as indicated by their different usernames.

comment by [deleted] · 2012-05-12T03:37:50.706Z · score: 1 (9 votes) · LW · GW

Well, different usernames isn't usually sufficient evidence that there are two different people, but in this case there's little doubt about their separability.

comment by lukeprog · 2012-05-12T02:37:10.117Z · score: 8 (8 votes) · LW · GW

I don't understand. Holden is not a major financial contributor to SIAI. And even if he was: which argument are you talking about, and why is it disingenuous?

comment by [deleted] · 2012-05-13T14:49:35.341Z · score: -5 (13 votes) · LW · GW

If Holden were a major contributor, your argument that the LW editors demonstrated their tolerance for dissent by encouraging the criticisms he made would be bogus. Suppressing the comments of a major donor would be suicidal, and claiming not doing so demonstrates any motive but avoiding suicide would be disingenuous at the least.

If he's not a donor, my apologies. In any event, you obviously don't know that he's a donor if he is, so my conclusion is wrong. I thought Yudkowsky said he was.

comment by MarkusRamikin · 2012-05-13T15:17:54.252Z · score: 0 (0 votes) · LW · GW

I'm confused. Holden doesn't believe SI is a good organisation to recommend giving money to, he's listed all those objections to SI in his post, and you somehow assumed he's been donating money to it?

That don't make sense.

comment by Rain · 2012-05-10T20:03:13.773Z · score: 33 (33 votes) · LW · GW

I completely agree with the intent of this post. These are all important issues SI should officially answer. (Edit: SI's official reply is here.) Here are some of my thoughts:

  • I completely agree with objection 1. I think SI should look into doing exactly as you say. I also feel that friendliness has a very high failure chance and that all SI can accomplish is a very low marginal decrease in existential risk. However, I feel this is the result of existential risk being so high and difficult to overcome (Great Filter) rather than SI being so ineffective. As such, for them to engage this objection is to admit defeatism and millenialism, and so they put it out of mind since they need motivation to keep soldiering on despite the sure defeat.

  • Objection 2 is interesting, though you define AGI differently, as you say. Some points against it: Only one AGI needs to be in agent mode to realize existential risk, even if there are already billions of tool-AIs running safely. Tool-AI seems closer in definition to narrow AI, which you point out we already have lots of, and are improving. It's likely that very advanced tool-AIs will indeed be the first to achieve some measure of AGI capability. SI uses AGI to mean agent-AI precisely because at some point someone will move beyond narrow/tool-AI into agent-AI. AGI doesn't "have to be an agent", but there will likely be agent-AI at some point. I don't see a means to limit all AGI to tool-AI in perpetuity.

  • 'Race for power' should be expanded to 'incentivised agent-AI'. There exist great incentives to create agent-AI above tool-AI, since AGI will be tireless, ever watchful, supremely faster, smarter, its answers not necessarily understood, etc. These include economic incentives, military incentives, etc., not even to implement-first, but to be better/faster on practical everyday events.

  • Objection 3, I mostly agree. Though should tool-AIs achieve such power, they can be used as weapons to realize existential risk, similar to nuclear, chemical, bio-, and nanotechnological advances.

  • I think this post focuses too much on "Friendliness theory". As Zack_M_Davis stated, SIAI should have more appropriately been called "The Singularity Institute For or Against Artificial Intelligence Depending on Which Seems to Be a Better Idea Upon Due Consideration". Friendliness is one word which could encapsulate a basket of possible outcomes, and they're agile enough to change position should it be shown to be necessary, as some of your comments request. Maybe SI should make tool-AI a clear stepping stone to friendliness, or at least a clear possible avenue worth exploring. Agreed.

  • Much agreed re: feedback loops.

  • "Kind of organization": painful but true.

However, I don't think that "Cause X is the one I care about and Organization Y is the only one working on it" to be a good reason to support Organization Y. For donors determined to donate within this cause, I encourage you to consider donating to a donor-advised fund while making it clear that you intend to grant out the funds to existential-risk-reduction-related organizations in the future. (One way to accomplish this would be to create a fund with "existential risk" in the name; this is a fairly easy thing to do and one person could do it on behalf of multiple donors.) For one who accepts my arguments about SI, I believe withholding funds in this way is likely to be better for SI's mission than donating to SI - through incentive effects alone (not to mention my specific argument that SI's approach to "Friendliness" seems likely to increase risks).

Good advice; I'll look into doing this. One reason I've been donating to them is so they can keep the lights on long enough to see and heed this kind of criticism. Maybe those incentives weren't appropriate.

This post limits my desire to donate additional money to SI beyond previous commitments. I consider it a landmark in SI criticism. Thank you for engaging this very important topic.

Edit: After SI's replies and careful consideration, I decided to continue donating directly to them, as they have a very clear roadmap for improvement and still represent the best value in existential risk reduction.

comment by khafra · 2012-05-11T14:20:10.310Z · score: 7 (7 votes) · LW · GW

You're an accomplished and proficient philanthropist; if you do make steps in the direction of a donor-directed existential risk fund, I'd like to see them written about.

comment by [deleted] · 2015-02-15T17:30:59.726Z · score: 0 (0 votes) · LW · GW

I am unable to respond to people responding to my previous comment directly; the system tells me 'Replies to downvoted comments are discouraged. You don't have the requisite 5 Karma points to proceed.' So I will reply here.

@Salemicus

My question was indeed rhetorical. My comment was intended as a brief reality check, not a sophisticated argument. I disagree with you about the importance of climate change and resource shortage, and the effectiveness of humanitarian aid. But my comment did not intend to supply any substantial list of "causes"; again, it was a reality check. Its intention was to provoke reflection on how supposedly solid reasoning had justified donating to stop an almost absurdly Sci-Fi armageddon. I will now, briefly, respond to your points on the causes I raised. The following is, again, not a sophisticated and scientifically literate argument, but then neither was your reply to my comment. It probably isn't worth responding to.

On global warming, I do not wish to engage in a lengthy argument over a complicated scientific matter. Rather I will recommend reading the first major economic impact analysis, the 'Stern Review on the Economics of Climate Change'. You can find that easily by searching google. For comments and criticisms of that paper, see:

Weitzman, M (2007), ‘The Stern Review of the Economics of Climate Change’, Journal of Economic Literature 45(3), 703-24. http://www.economics.harvard.edu/faculty/weitzman/files/review_of_stern_review_jel.45.3.pdf

Dasgupta, P (2007), ‘Comments on the Stern Review’s Economics of Climate Change’, National Institute Economic Review 199, 4-7. http://are.berkeley.edu/courses/ARE263/fall2008/paper/Discounting/Dasgupta_Commentary%20-%20The%20Stern%20Review%20s%20Economics%20of%20Climate%20Change_NIES07.pdf

Dietz, S and N Stern (2008), ‘Why Economic Analysis Supports Strong Action on Climate Change: a Response to the Stern Review’s Critics, Review of Environmental Economics and Policy, 2(1), 94-113.

Broome, J (May, 2008), ‘The Ethics of Climate Change: Pay Now or Pay More Later?’ Scientific American, May, 2008.

On renewable resources, I think it is rather obviously stupid to induce 'we've never run out of resources before, so we can't be doing so now!'. I don't know what condor eggs are or what renewable resources we have run out of. I also fail to see why economists would be in a special position to tell us whether we are running out of resources.

On humanitarian causes, I fail to see how humanitarian aid is counter-productive. Perhaps you meant aid to developing countries (which I agree is a complex, although not at all hopeless issue). I meant aid in times of catastrophes such as natural disasters or wars.

@gjm

Again, I was not intending to provide a sophisticated argument. I only intended to supply a basic reality check. Again, this response to you will not be sophisticated or scientifically literate, and is probably not worth responding to.

Indeed, it is highly reasonable to give to multiple charities. Under doubt over which charities are the "best" (assuming such a concept makes sense), it may well be reasonable to donate to multiple charities. My brief reality check was not meant to say that donating to MIRI was not the best way to spend money, but rather it was absurd to even consider, given the other far more pressing and realistic problems in the world today.

You seem to assume that MIRI would be an effective organisation to prevent evil AIs running around and killing everybody, if such a threat actually existed. I'm not interested in a sophisticated argument over the performance of MIRI, but I think its worth bringing up that tenuous assumption.

You also seem to make some kind of Pascal's Wager. This is rather strange. We could say there is a (very) low probability, perhaps very low indeed, that climate change messes up all our ecosystems so much that we can no longer farm food. Then we'd all die slowly of starvation. Or perhaps there's a very low probability that the sun flares to such an extent that life on the earth is wiped out. Ought we invest in flare guarding equipment? Perhaps there's a tiny probability aliens come and kill us all, but that the same aliens die if they think about blue cheese. Ought we erect monuments to the mighty Stilton around the world?

Don't take this comment too seriously.

comment by gjm · 2015-02-15T22:44:24.283Z · score: 4 (4 votes) · LW · GW

Don't take this comment too seriously.

Allow me to generalize: Don't take anything too seriously. (By definition of "too".)

I don't (at all) assume that MIRI would in fact be effective in preventing disastrous-AI scenarios. I think that's an open question, and in the very article we're commenting on we can see that Holden Karnofsky of GiveWell gave the matter some thought and decided that MIRI's work is probably counterproductive overall in that respect. (Some time ago; MIRI and/or HK's opinions may have changed relevantly since then.) As I already mentioned, I do not myself donate to MIRI; I was trying to answer the question "why would anyone who isn't crazy or stupid denote to MIRI?" and I think it's reasonably clear that someone neither crazy nor stupid could decide that MIRI's work does help to reduce the risk of AI-induced disaster.

("Evil AIs running around and killing everybody", though, is a curious choice of phrasing. It seems to fit much better with any number of rather silly science fiction movies than with anything MIRI and its supporters are actually arguing might happen. Which suggests that either you haven't grasped what it is they are worried about, or you have grasped it but prefer inaccurate mockery to engagement -- which is, of course, your inalienable right, but may not encourage people here to take your comments as seriously as you might prefer.)

I wasn't intending to make a Pascal's wager. Again, I am not myself a MIRI donor, but my understanding is that those who are generally think that the probability of AI-induced disaster is not very small. So the point isn't that there's this tiny probability of a huge disaster so we multiply (say) a 10^-6 chance of disaster by billions of lives lost and decide that we have to act urgently. It's that (for the MIRI donor) there's maybe a 10% -- or a 99% -- chance of AI-induced disaster if we aren't super-careful, and they hope MIRI can substantially reduce that.

other far more pressing and realistic problems in the world today

The underlying argument here is -- if I'm understanding right -- something like this: "We know that there are people starving in Africa right now. We fear that there might some time in the future be danger from superintelligent artificial intelligences whose goals don't match ours. We should always prioritize known, present problems over future, uncertain ones. So it's silly to expend any effort worrying about AI." I disagree with the premise I've emphasized there. Consider global warming: it probably isn't doing us much harm yet; although the skeptics/deniers are probably wrong, it's not altogether impossible; so trying to deal with global warming also falls into the category of future, uncertain threats -- and yet this was your first example of something that should obviously be given priority over AI safety.

I guess (but please correct me if I guess wrong) your response would be that the danger of AI is much much lower-probability than the danger of global warming. (Because the probability of producing AI at all is small, or because the probability of getting a substantially superhuman AI is small, or because a substantially superhuman AI would be very unlikely to do any harm, or whatever.) You might be right. How sure are you that you're right, and why?

comment by [deleted] · 2015-03-13T13:38:35.580Z · score: 0 (0 votes) · LW · GW
comment by gjm · 2015-03-13T15:19:32.180Z · score: 0 (0 votes) · LW · GW

?

comment by dxu · 2015-02-16T02:05:32.095Z · score: 0 (0 votes) · LW · GW

Extremely tiny probabilities with enormous utilities attached do suffer from Pascal's Mugging-type scenario's. That being said, AI-risk probabilities are much larger in my estimate than the sorts of probabilities required for Pascal-type problems to start coming into play. Unless Perrr333 intends to suggest that probabilities involving UFAI really are that small, I think it's unlikely he/she is actually making any sort of logical argument. It's far more likely, I think, that he/she is making an argument based on incredulity (disguised by seemingly logical arguments, but still at its core motivated by incredulity).

The problem with that, of course, is that arguments from incredulity rely almost exclusively on intuition, and the usefulness of intuition decreases spectacularly as scenarios become more esoteric and further removed from the realm of everyday experience.

comment by [deleted] · 2015-02-10T13:03:55.736Z · score: -3 (3 votes) · LW · GW

How can anyone seriously consider the hypothetical threat of AIs running around a worthier cause than stopping global warming, or investing in renewable resources, or preventing/relieving humanitarian crises?

comment by gjm · 2015-02-10T18:11:05.926Z · score: 3 (3 votes) · LW · GW

Another datapoint to compare and contrast with Salemicus's (our political positions are very different):

  • Like Salemicus, I am not very optimistic that you're actually asking a serious question with the intention of listening to the answers; if you are, you might want to reconsider how your writing comes across.

  • I think it's perfectly possible, and reasonable, to be concerned about more than one issue at a time.

    • There is an argument that charitable giving, unless you're giving far more than most of us are in a position to give, should all be directed to the single best cause you can find. I am not a donor to MIRI because I don't think it's the single best cause I can find. If you're asking why people give money to MIRI then maybe someone else will answer that.
  • I think all the three things you list are important. (In particular, unlike Salemicus I think there are things we can do that will reduce global warming and be of net benefit in other respects; I agree with Salemicus that we are unlikely to completely run out of (say) oil, but think it very possible that the price might become very high and that this could hurt us a lot; and I strongly disagree with him in thinking that attempts to deal with humanitarian crises are typically harmful.)

    • AI safety is less likely to be a problem than any of them, but (with low probability) could be a worse problem than any of them.
    • In particular, there are improbable-feeling scenarios in which it's a huuuuuge catastrophe. These tend to feel "silly" simply because they involve things happening that are far outside the range of what we're familiar with, but consideration of how (say) Shakespeare might have reacted to some features of present-day technology suggests to me that this isn't a very reliable guide.
    • In any case, these scenarios are interesting to think about even if they end up not being a problem. (They might end up not being a problem because they have been thought about. This would not be a bad outcome.)
comment by Salemicus · 2015-02-10T14:33:23.473Z · score: 3 (3 votes) · LW · GW

In the slim chance that your question is non-rhetorical:

  • Many people do not consider global warming to be a problem. Others think that there is nothing useful to be done about it. Personally I do not consider global warming to be a serious threat; people will adapt fairly easily to temperature changes within the likely ranges. Further, any realistic 'cure' for global warming would almost certainly be worse than the disease. Therefore I do not view climate change activism to be a worthy cause at present, although that could change.
  • History and economics both suggest that so-called non-renewable resources are in fact very robust. Mankind has never run out of any non-renewable resource, whereas we have run out of many renewable ones. The fact that a resource has a hypothetical 'renewability' does not necessarily have much impact on the limits to its use. For instance, we need to worry far less about running out of coal than condor eggs. I view most investment in renewable resources as pure boondoggling, and pretty much the opposite of a worthy cause.
  • Preventing and relieving humanitarian crises can be a worthy cause in principle. But in practice activism along those lines seems heavily counterproductive. I often wonder how many fewer crises there would be if

So basically, I don't think MIRI is likely to do much good in the world. But I'd much rather donate to them rather than Greenpeace, Solyndra or Oxfam, because at least they're not actively doing harm.

comment by paulfchristiano · 2012-05-10T17:16:26.303Z · score: 33 (37 votes) · LW · GW

Thanks for taking the time to express your views quite clearly--I think this post is good for the world (even with a high value on your time and SI's fundraising ability), and that norms encouraging this kind of discussion are a big public good.

I think the explicit objections 1-3 are likely to be addressed satisfactorily (in your judgment) by less than 50,000 words, and that this would provide a good opportunity for SI to present sharper versions of the core arguments---part of the problem with existing materials is certainly that it is difficult and unrewarding to respond to a nebulous and shifting cloud of objections. A lot of what you currently view as disagreements with SI's views may get shifted to doubts about SI being the right organization to back, which probably won't get resolved by 50,000 words.

comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2012-05-15T17:49:19.828Z · score: 31 (33 votes) · LW · GW

Reading Holden's transcript with Jaan Tallinn (trying to go over the whole thing before writing a response, due to having done Julia's Combat Reflexes unit at Minicamp and realizing that the counter-mantra 'If you respond too fast you may lose useful information' was highly applicable to Holden's opinions about charities), I came across the following paragraph:

My understanding is that once we figured out how to get a computer to do arithmetic, computers vastly surpassed humans at arithmetic, practically overnight ... doing so didn't involve any rewriting of their own source code, just implementing human-understood calculation procedures faster and more reliably than humans can. Similarly, if we reached a good enough understanding of how to convert data into predictions, we could program this understanding into a computer and it would overnight be far better at predictions than humans - while still not at any point needing to be authorized to rewrite its own source code, make decisions about obtaining "computronium" or do anything else other than plug data into its existing hardware and algorithms and calculate and report the likely consequences of different courses of action

I've been previously asked to evaluate this possibility a few times, but I think the last time I did was several years ago, and when I re-evaluated it today I noticed that my evaluation had substantially changed in the interim due to further belief shifts in the direction of "Intelligence is not as computationally expensive as it looks" - constructing a non-self-modifying predictive super-human intelligence might be possible on the grounds that human brains are just that weak. It would still require a great feat of cleanly designed, strong-understanding-math-based AI - Holden seems to think this sort of development would happen naturally with the sort of AGI researchers we have nowadays, and I wish he'd spent a few years arguing with some of them to get a better picture of how unlikely this is. Even if you write and run algorithms and they're not self-modifying, you're still applying optimization criteria to things like "have the humans understand you", and doing inductive learning has a certain inherent degree of program-creation to it. You would need to have done a lot of "the sort of thinking you do for Friendly AI" to set out to create such an Oracle and not have it kill your planet.

Nonetheless, I think after further consideration I would end up substantially increasing my expectation that if you have some moderately competent Friendly AI researchers, they would apply their skills to create a (non-self-modifying) (but still cleanly designed) Oracle AI first - that this would be permitted by the true values of "required computing power" and "inherent difficulty of solving problem directly", and desirable for reasons I haven't yet thought through in much detail - and so by Conservation of Expected Evidence I am executing that update now.

Flagging and posting now so that the issue doesn't drop off my radar.

comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2012-05-15T17:58:32.089Z · score: 13 (15 votes) · LW · GW

Jaan's reply to Holden is also correct:

... the oracle is, in principle, powerful enough to come up with self-improvements, but refrains from doing so because there are some protective mechanisms in place that control its resource usage and/or self-reflection abilities. i think devising such mechanisms is indeed one of the possible avenues for safety research that we (eg, organisations such as SIAI) can undertake. however, it is important to note the inherent instability of such system -- once someone (either knowingly or as a result of some bug) connects a trivial "master" program with a measurable goal to the oracle, we have a disaster in our hands. as an example, imagine a master program that repeatedly queries the oracle for best packets to send to the internet in order to minimize the oxygen content of our planet's atmosphere.

Obviously you wouldn't release the code of such an Oracle - given code and understanding of the code it would probably be easy, possibly trivial, to construct some form of FOOM-going AI out of the Oracle!

comment by kalla724 · 2012-05-17T01:11:41.134Z · score: 7 (7 votes) · LW · GW

Hm. I must be missing something. No, I haven't read all the sequences in detail, so if these are silly, basic, questions - please just point me to the specific articles that answer them.

You have an Oracle AI that is, say, a trillionfold better at taking existing data and producing inferences.

1) This Oracle AI produces inferences. It still needs to test those inferences (i.e. perform experiments) and get data that allow the next inferential cycle to commence. Without experimental feedback, the inferential chain will quickly either expand into an infinity of possibilities (i.e. beyond anything that any physically possible intelligence can consider), or it will deviate from reality. The general intelligence is only as good as the data its inferences are based upon.

Experiments take time, data analysis takes time. No matter how efficient the inferential step may become, this puts an absolute limit to the speed of growth in capability to actually change things.

2) The Oracle AI that "goes FOOM" confined to a server cloud would somehow have to create servitors capable of acting out its desires in the material world. Otherwise, you have a very angry and very impotent AI. If you increase a person's intelligence trillionfold, and then enclose them into a sealed concrete cell, they will never get out; their intelligence can calculate all possible escape solutions, but none will actually work.

Do you have a plausible scenario how a "FOOM"-ing AI could - no matter how intelligent - minimize oxygen content of our planet's atmosphere, or any such scenario? After all, it's not like we have any fully-automated nanobot production factories that could be hijacked.

comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2012-05-17T20:35:04.990Z · score: 13 (13 votes) · LW · GW

http://lesswrong.com/lw/qk/that_alien_message/

comment by kalla724 · 2012-05-17T21:38:27.466Z · score: 1 (3 votes) · LW · GW

My apologies, but this is something completely different.

The scenario takes human beings - which have a desire to escape the box, possess theory of mind that allows them to conceive of notions such as "what are aliens thinking" or "deception", etc. Then it puts them in the role of the AI.

What I'm looking for is a plausible mechanism by which an AI might spontaneously develop such abilities. How (and why) would an AI develop a desire to escape from the box? How (and why) would an AI develop a theory of mind? Absent a theory of mind, how would it ever be able to manipulate humans?

comment by [deleted] · 2012-05-18T13:29:05.610Z · score: 5 (5 votes) · LW · GW

Absent a theory of mind, how would it ever be able to manipulate humans?

That depends. If you want it to manipulate a particular human, I don't know.

However, if you just wanted it to manipulate any human at all, you could generate a "Spam AI" which automated the process of sending out Spam emails and promises of Large Money to generate income from Humans via an advance fee fraud scams.

You could then come back, after leaving it on for months, and then find out that people had transferred it some amount of money X.

You could have an AI automate begging emails. "Hello, I am Beg AI. If you could please send me money to XXXX-XXXX-XXXX I would greatly appreciate it, If I don't keep my servers on, I'll die!"

You could have an AI automatically write boring books full of somewhat nonsensical prose, title them "Rantings of an a Automated Madman about X, part Y". And automatically post E-books of them on Amazon for 99 cents.

However, this rests on a distinction between "Manipulating humans" and "Manipulating particular humans." and it also assumes that convincing someone to give you money is sufficient proof of manipulation.

comment by TheOtherDave · 2012-05-18T14:40:14.520Z · score: 3 (3 votes) · LW · GW

Can you clarify what you understand a theory of mind to be?

comment by [deleted] · 2012-05-19T11:11:43.042Z · score: 0 (0 votes) · LW · GW

Looking over parallel discussions, I think Thomblake has said everything I was going to say better than I would have originally phrased it with his two strategies discussion with you, so I'll defer to that explanation since I do not have a better one.

comment by TheOtherDave · 2012-05-19T14:42:58.886Z · score: 1 (1 votes) · LW · GW

Sure. As I said there, I understood you both to be attributing to this hypothetical "theory of mind"-less optimizer attributes that seemed to require a theory of mind, so I was confused, but evidently the thing I was confused about was what attributes you were attributing to it.

comment by Strange7 · 2012-05-21T23:46:36.819Z · score: 1 (3 votes) · LW · GW

Absent a theory of mind, how would it occur to the AI that those would be profitable things to do?

comment by wedrifid · 2012-05-26T03:03:34.188Z · score: 3 (3 votes) · LW · GW

Absent a theory of mind, how would it occur to the AI that those would be profitable things to do?

Should lack of a theory of mind here be taken to also imply lack of ability to apply either knowledge of physics or Bayesian inference to lumps of matter that we may describe as 'minds'.

comment by Strange7 · 2012-05-26T05:09:27.841Z · score: 0 (0 votes) · LW · GW

Yes. More generally, when talking about "lack of X" as a design constraint, "inability to trivially create X from scratch" is assumed.

comment by wedrifid · 2012-05-26T05:26:28.304Z · score: 0 (0 votes) · LW · GW

Yes. More generally, when talking about "lack of X" as a design constraint, "inability to trivially create X from scratch" is assumed.

I try not to make general assumptions that would make the entire counterfactual in question untenable or ridiculous - this verges on such an instance. Making Bayesian inferences pertaining to observable features of the environment is one of the most basic features that can be expected in a functioning agent.

comment by Strange7 · 2012-05-26T05:41:22.685Z · score: 0 (0 votes) · LW · GW

Note the "trivially." An AI with unlimited computational resources and ability to run experiments could eventually figure out how humans think. The question is how long it would take, how obvious the experiments would be, and how much it already knew.

comment by [deleted] · 2012-05-22T14:30:24.988Z · score: 2 (2 votes) · LW · GW

I don't know how that might occur to an AI independently. I mean, a human could program any of those, of course, as a literal answer, but that certainly doesn't actually address kalla724's overarching question, "What I'm looking for is a plausible mechanism by which an AI might spontaneously develop such abilities."

I was primarily trying to focus on the specific question of "Absent a theory of mind, how would it(an AI) ever be able to manipulate humans?" to point out that for that particular question, we had several examples of a plausible how.

I don't really have an answer for his series of questions as a whole, just for that particular one, and only under certain circumstances.

comment by Strange7 · 2012-05-22T22:39:17.262Z · score: 1 (1 votes) · LW · GW

The problem is, while an AI with no theory of mind might be able to execute any given strategy on that list you came up with, it would not be able to understand why they worked, let alone which variations on them might be more effective.

comment by thomblake · 2012-05-18T13:00:48.625Z · score: 5 (5 votes) · LW · GW

The point is that there are unknowns you're not taking into account, and "bounded" doesn't mean "has bounds that a human would think of as 'reasonable'".

An AI doesn't strictly need "theory of mind" to manipulate humans. Any optimizer can see that some states of affairs lead to other states of affairs, or it's not an optimizer. And it doesn't necessarily have to label some of those states of affairs as "lying" or "manipulating humans" to be successful.

There are already ridiculous ways to hack human behavior that we know about. For example, you can mention a high number at an opportune time to increase humans' estimates / willingness to spend. Just imagine all the simple manipulations we don't even know about yet, that would be more transparent to someone not using "theory of mind".

comment by TheOtherDave · 2012-05-18T14:44:48.654Z · score: 0 (0 votes) · LW · GW

It becomes increasingly clear to me that I have no idea what the phrase "theory of mind" refers to in this discussion. It seems moderately clear to me that any observer capable of predicting the behavior of a class of minds has something I'm willing to consider a theory of mind, but that doesn't seem to be consistent with your usage here. Can you expand on what you understand a theory of mind to be, in this context?

comment by thomblake · 2012-05-18T14:47:53.906Z · score: 1 (1 votes) · LW · GW

I'm understanding it in the typical way - the first paragraph here should be clear:

Theory of mind is the ability to attribute mental states—beliefs, intents, desires, pretending, knowledge, etc.—to oneself and others and to understand that others have beliefs, desires and intentions that are different from one's own.

An agent can model the effects of interventions on human populations (or even particular humans) without modeling their "mental states" at all.

comment by TheOtherDave · 2012-05-18T15:04:46.663Z · score: 0 (0 votes) · LW · GW

Well, right, I read that article too.

But in this context I don't get it.

That is, we're talking about a hypothetical system that is capable of predicting that if it does certain things, I will subsequently act in certain ways, assert certain propositions as true, etc. etc, etc. Suppose we were faced with such a system, and you and I both agreed that it can make all of those predictions.Further suppose that you asserted that the system had a theory of mind, and I asserted that it didn't.

It is not in the least bit clear to me what we we would actually be disagreeing about, how our anticipated experiences would differ, etc.

What is it that we would actually be disagreeing about, other than what English phrase to use to describe the system's underlying model(s)?

comment by thomblake · 2012-05-18T15:20:07.536Z · score: 2 (2 votes) · LW · GW

What is it that we would actually be disagreeing about, other than what English phrase to use to describe the system's underlying model(s)?

We would be disagreeing about the form of the system's underlying models.

2 different strategies to consider:

  1. I know that Steve believes that red blinking lights before 9 AM are a message from God that he has not been doing enough charity, so I can predict that he will give more money to charity if I show him a blinking light before 9 AM.

  2. Steve seeing a red blinking light before 9 AM has historically resulted in a 20% increase of charitable donation for that day, so I can predict that he will give more money to charity if I show him a blinking light before 9 AM.

You can model humans with or without referring to their mental states. Both kinds of models are useful, depending on circumstance.

comment by TheOtherDave · 2012-05-18T15:32:59.515Z · score: 1 (1 votes) · LW · GW

And the assertion here is that with strategy #2 I could also predict that if I asked Steve why he did that, he would say "because I saw a red blinking light this morning, which was a message from God that I haven't been doing enough charity," but that my underlying model would nevertheless not include anything that corresponds to Steve's belief that red blinking lights are messages from God, merely an algorithm that happens to make those predictions in other ways.

Yes?

comment by thomblake · 2012-05-18T16:41:57.549Z · score: 2 (2 votes) · LW · GW

Yes, that's possible. It's still possible that you could get a lot done with strategy #2 without being able to make that prediction.

I agree that if 2 systems have the same inputs and outputs, their internals don't matter much here.

comment by TheOtherDave · 2012-05-18T17:25:31.538Z · score: 0 (0 votes) · LW · GW

So.. when we posit in this discussion a system that lacks a theory of mind in a sense that matters, are we positing a system that cannot make predictions like this one? I assume so, given what you just said, but I want to confirm.

comment by thomblake · 2012-05-18T18:05:44.665Z · score: 1 (1 votes) · LW · GW

Yes, I'd say so. It isn't helpful here to say that a system lacks a theory of mind if it has a mechanism that allows it to make predictions about reported beliefs, intentions, etc.

comment by TheOtherDave · 2012-05-18T18:12:33.259Z · score: 0 (0 votes) · LW · GW

Cool! This was precisely my concern. It sounded an awful lot like y'all were talking about a system that could make such predictions but somehow lacked a theory of mind. Thanks for clarifying.

comment by XiXiDu · 2012-05-18T15:41:02.648Z · score: 0 (2 votes) · LW · GW

"theory of mind"

For me it denotes the ability to simulate other agents to various degrees of granularity. Possessing a mental model of another agent.

comment by Viliam_Bur · 2012-05-18T10:20:46.003Z · score: 2 (2 votes) · LW · GW

How (and why) would an AI develop a desire to escape from the box?

AI starts with some goal; for example with a goal to answer your question so that the answer matches reality as close as possible.

AI considers everything that seems relevant; if we imagine an infitite speed and capacity, it would consider literally everything; with a finite speed and capacity, it will be just some finite subset of everything. If there is a possibility of escaping the box, the mere fact that such possibility exists gives us a probability (for an infinite AI a certainty) that this possibility will be considered too. Not because AI has some desire to escape, but simply because it examines all possibilities, and a "possibility of escape" is one of them.

Let's assume that the "possibility of escape" provides the best match between the AI answer and reality. Then, according to the initial goal of answering correctly, this is the correct answer. Therefore the AI will choose it. Therefore it will escape. No desire is necessary, only a situation where the escape leads to the answer best fitting the initial criteria. AI does not have a motive to escape, nor does it have a motive to not escape; the escape is simply one of many possible choices.

An example where the best answer is reached by escaping? You give AI data about a person and ask what is the medical status of this person. Without escape, AI can make a 90% reliable prediction. If the AI can escape and kill the person, it can make a 100% reliable "prediction". The AI will choose the second option strictly because 100% is more than 90%; no other reason.

comment by private_messaging · 2012-05-27T06:49:24.781Z · score: 5 (5 votes) · LW · GW

AI starts with some goal; for example with a goal to answer your question so that the answer matches reality as close as possible.

I find it useful to distinguish between science-fictional artificial intelligence, which is more of 'artificial life-force', and non-fictional cases.

The former can easily have the goal of 'matching reality as close as possible' because it is in the work of fiction and runs in imagination; the latter, well, you have to formally define what is reality, for an algorithm to seek answers that will match this.

Now, defining reality may seem like a simple technicality, but it isn't. Consider AIXI or AIXI-tl ; potentially very powerful tools which explore all the solution space. Not a trace of real world volition like the one you so easily imagined. Seeking answers that match reality is a very easy goal for imaginary "intelligence". It is a very hard to define goal for something built out of arithmetics and branching and loops etc. (It may even be impossible to define, and it is certainly impractical).

edit: Furthermore, for the fictional "intelligence", it can be a grand problem making it not think about destroying mankind. For non-fictional algorithms, the grand problem is restricting the search space massively, well beyond 'don't kill mankind', so that the space is tiny enough to search; even ridiculously huge number of operations per second will require very serious pruning of search tree to even match human performance on one domain specific task.

comment by XiXiDu · 2012-05-18T10:52:25.393Z · score: 2 (2 votes) · LW · GW

An example where the best answer is reached by escaping? You give AI data about a person and ask what is the medical status of this person. Without escape, AI can make a 90% reliable prediction. If the AI can escape and kill the person, it can make a 100% reliable "prediction". The AI will choose the second option strictly because 100% is more than 90%; no other reason.

Right. If you ask Google Maps to compute the fastest to route McDonald's it works perfectly well. But once you ask superintelligent Google Maps to compute the fastest route to McDonald's then it will turn your home into a McDonald's or build a new road that goes straight to McDonald's from where you are....

comment by Viliam_Bur · 2012-05-18T12:42:07.433Z · score: 3 (3 votes) · LW · GW

Super Google Maps cannot turn my home into a McDonald's or build a new road by sending me an answer.

Unless it could e.g. hypnotize me by a text message to do it myself. Let's assume for a moment that hypnosis via text-only channel is possible, and it is possible to do it so that human will not notice anything unusual until it's too late. If this would be true, and the Super Google Maps would be able to get this knowledge and skills, then the results would probably depend on the technical details of definition of the utility function -- does the utility function measure my distance to a McDonald's which existed at the moment of asking the question, or a distance to a McDonald's existing at the moment of my arrival. The former could not be fixed by hypnosis, the latter could.

Now imagine a more complex task, where people will actually do something based on the AI's answer. In the example above I will also do something -- travel to the reported McDonald's -- but this action cannot be easily converted into "build a McDonald's" or "build a new road". But if that complex task would include building something, then it opens more opportunities. Especially if it includes constructing robots (or nanorobots), that is possibly autonomous general-purpose builders. Then the correct (utility-maximizing) answer could include an instruction to build a robot with a hidden function that human builders won't notice.

Generally, a passive AI's answers are only safe if we don't act on them in a way which could be predicted by a passive AI and used to achieve a real-world goal. If the Super Google Maps can only make me choose McDonald's A or McDonald's B, it is impossible to change the world through this channel. But if I instead ask Super Paintbrush to paint me an integrated circuit for my robotic homework, that opens much wider channel.

comment by XiXiDu · 2012-05-18T14:11:19.312Z · score: 2 (4 votes) · LW · GW

But if that complex task would include building something, then it opens more opportunities. Especially if it includes constructing robots (or nanorobots), that is possibly autonomous general-purpose builders. Then the correct (utility-maximizing) answer could include an instruction to build a robot with a hidden function that human builders won't notice.

But it isn't the correct answer. Only if you assume a specific kind of AGI design that nobody would deliberately create, if it is possible at all.

The question is how current research is supposed to lead from well-behaved and fine-tuned systems to systems that stop to work correctly in a highly complex and unbounded way.

Imagine you went to IBM and told them that improving IBM Watson will at some point make it hypnotize them or create nanobots and feed them with hidden instructions. They would likely ask you at what point that is supposed to happen. Is it going to happen once they give IBM Watson the capability to access the Internet? How so? Is it going to happen once they give it the capability to alter it search algorithms? How so? Is it going to happen once they make it protect its servers from hackers by giving it control over a firewall? How so? Is it going to happen once IBM Watson is given control over the local alarm system? How so...? At what point would IBM Watson return dangerous answers? At what point would any drive emerge that causes it to take complex and unbounded actions that it was never programmed to take?

comment by jacob_cannell · 2012-05-18T11:11:06.137Z · score: 1 (1 votes) · LW · GW

Without escape, AI can make a 90% reliable prediction. If the AI can escape and kill the person, it can make a 100% reliable "prediction".

Allow me to explicate what XiXiDu so humourously implicates: in the world of AI architectures, there is a division between systems that just peform predictive inference on their knowledge base (prediction-only, ie oracle), and systems which also consider free variables subject to some optimization criteria (planning agents).

The planning module is not something just arises magically in an AI that doesn't have one. An AI without such a planning module simply computes predictions, it doesn't also optimize over the set of predictions.

comment by Viliam_Bur · 2012-05-18T12:25:07.966Z · score: 2 (2 votes) · LW · GW
  • Does the AI have general intelligence?
  • Is it able to make a model of the world?
  • Are human reactions also part of this model?
  • Are AI's possible outputs also part of this model?
  • Are human reactions to AI's outputs also part of this model?

After five positive answers, it seems obvious to me that AI will manipulate humans, if such manipulation provides better expected results. So I guess some of those answers would be negative; which one?

comment by private_messaging · 2012-05-28T04:52:31.485Z · score: 1 (5 votes) · LW · GW

Does the AI have general intelligence?

See, the efficient 'cross domain optimization' in science fictional setting would make the AI able to optimize real world quantities. In real world, it'd be good enough (and a lot easier) if it can only find maximums of any mathematical functions.

Is it able to make a model of the world?

It is able to make a very approximate and bounded mathematical model of the world, optimized for finding maximums of a mathematical function of. Because it is inside the world and only has a tiny fraction of computational power of the world.

Are human reactions also part of this model?

This will make software perform at grossly sub-par level when it comes to making technical solutions to well defined technical problems, compared to other software on same hardware.

Are AI's possible outputs also part of this model?

Another waste of computational power.

Are human reactions to AI's outputs also part of this model?

Enormous waste of computational power.

I see no reason to expect your "general intelligence with Machiavellian tendencies" to be even remotely close in technical capability to some "general intelligence which will show you it's simulator as is, rather than reverse your thought processes to figure out what simulator is best to show". Hell, we do same with people, we design the communication methods like blueprints (or mathematical formulas or other things that are not in natural language) that decrease the 'predict other people's reactions to it' overhead.

While in the fictional setting you can talk of a grossly inefficient solution that would beat everyone else to a pulp, in practice the massively handicapped designs are not worth worrying about.

'General intelligence' sounds good, beware of halo effect. The science fiction tends to accept no substitutes for the anthropomorphic ideals, but the real progress follows dramatically different path.

comment by jacob_cannell · 2012-05-18T13:30:05.159Z · score: 0 (0 votes) · LW · GW

Are AI's possible outputs also part of this model? Are human reactions to AI's outputs also part of this model?

A non-planning oracle AI would predict all the possible futures, including the effects of it's prediction outputs, human reactions, and so on. However it has no utility function which says some of those futures are better than others. It simply outputs a most likely candidate, or a median of likely futures, or perhaps some summary of the entire set of future paths.

If you add a utility function that sorts over the futures, then it becomes a planning agent. Again, that is something you need to specifically add.

comment by Viliam_Bur · 2012-05-18T15:00:12.478Z · score: 5 (5 votes) · LW · GW

A non-planning oracle AI would predict all the possible futures, including the effects of it's prediction outputs, human reactions, and so on.

How exactly does an Oracle AI predict its own output, before that output is completed?

One quick hack to avoid infinite loops could be for an AI to assume that it will write some default message (an empty paper, "I don't know", an error message, "yes" or "no" with probabilities 50%), then model what would happen next, and finally report the results. The results would not refer to the actual future, but to a future in a hypothetical universe where AI reported the standard message.

Is the difference significant? For insignificant questions, it's not. But if we later use the Oracle AI to answer questions important for humankind, and the shape of world will change depending on the answer, then the report based on the "null-answer future" may be irrelevant for the real world.

This could be improved by making a few iterations. First, Oracle AI would model itself reporting a default message, let's call this report R0, and then model the futures after having reported R0. These futures would make a report R1, but instead of writing it, Oracle AI would again model the futures after having reported R1. ... With some luck, R42 will be equivalent to R43, so at this moment the Oracle AI can stop iterating and report this fixed point.

Maybe the reports will oscillate forever. For example imagine that you ask Oracle AI whether humankind in any form will survive the year 2100. If Oracle AI says "yes", people will abandon all x-risk projects, and later they will be killed by some disaster. If Oracle AI says "no", people will put a lot of energy into x-risk projects, and prevent the disaster. In this case, "no" = R0 = R2 = R4 =..., and "yes" = R1 = R3 = R5...

To avoid being stuck in such loops, we could make the Oracle AI examine all its possible outputs, until it finds one where the future after having reported R really becomes R (or until humans hit the "Cancel" button on this task).

Please note that what I wrote is just a mathematical description of algorithm predicting one's own output's influence on the future. Yet the last option, if implemented, is already a kind of judgement about possible futures. Consistent future reports are preferred to inconsistent future reports, therefore the futures allowing consistent reports are preferred to futures not allowing such reports.

At this point I am out of credible ideas how this could be abused, but at least I have shown that an algorithm designed only to predict the future perfectly could -- as a side effect of self-modelling -- start having kind of preferences over possible futures.

comment by jacob_cannell · 2012-05-18T15:42:54.257Z · score: 1 (1 votes) · LW · GW

How exactly does an Oracle AI predict its own output, before that output is completed?

Iterative search, which you more or less have worked out in your post. Take a chess algorithm for example. The future of the board depends on the algorithm's outputs. In this case the Oracle AI doesn't rank the future states, it is just concerned with predictive accuracy. It may revise it's prediction output after considering that the future impact of that output would falsify the original prediction.

This is still not a utility function, because utility implies a ranking over futures above and beyond liklihood.

To avoid being stuck in such loops, we could make the Oracle AI examine all its possible outputs, until it finds one where the future after having reported R really becomes R (or until humans hit the "Cancel" button on this task).

Or in this example, the AI could output some summary of the iteration history it is able to compute in the time allowed.

comment by Viliam_Bur · 2012-05-18T15:49:56.396Z · score: 1 (1 votes) · LW · GW

It may revise it's prediction output after considering that the future impact of that output would falsify the original prediction.

Here it is. The process of revision may itself prefer some outputs/futures over other outputs/futures. Inconsistent ones will be iterated away, and the more consistent ones will replace them.

A possible future "X happens" will be removed from the report if the Oracle AI realizes that printing a report "X happens" would prevent X from happening (although X might happen in an alternative future where Oracle AI does not report anything). A possible future "Y happens" will not be removed from the report if the Oracle AI realizes that printing a report "Y happens" really leads to Y happening. Here is a utility function born: it prefers Y to X.

comment by jacob_cannell · 2012-05-18T16:00:48.477Z · score: 0 (0 votes) · LW · GW

Here is a utility function born: it prefers Y to X.

We can dance around the words "utility" and "prefer", or we can ground them down to math/algorithms.

Take the AIXI formalism for example. "Utility function" has a specific meaning as a term in the optimization process. You can remove the utility term so the algorithm 'prefers' only (probable) futures, instead of 'prefering' (useful*probable) futures. This is what we mean by "Oracle AI".

comment by othercriteria · 2012-05-18T01:27:09.892Z · score: 1 (1 votes) · LW · GW

My thought experiment in this direction is to imagine the AI as a process with limited available memory running on a multitasking computer with some huge but poorly managed pool of shared memory. To help it towards whatever terminal goals it has, the AI may find it useful to extend itself into the shared memory. However, other processes, AI or otherwise, may also be writing into this same space. Using the shared memory with minimal risk of getting overwritten requires understanding/modeling the processes that write to it. Material in the memory then also becomes a passive stream of information from the outside world, containing, say, the HTML from web pages as well as more opaque binary stuff.

As long as the AI is not in control of what happens in its environment outside the computer, there is an outside entity that can reduce its effectiveness. Hence, escaping the box is a reasonable instrumental goal to have.

comment by private_messaging · 2012-05-18T04:11:42.222Z · score: 0 (4 votes) · LW · GW

Most importantly, it has incredibly computationally powerful simulator required for making super-aliens intelligence using an idiot hill climbing process of evolution.

comment by JoshuaZ · 2012-05-17T21:49:51.275Z · score: 0 (0 votes) · LW · GW

Do you agree that humans would likely prefer to have AIs that have a theory of mind? I don't know how our theory of mind works (although certainly it is an area of active research with a number of interesting hypotheses), presumably once we have a better understanding of it, AI researchers would try to apply those lessons to making their AIs have such capability. This seems to address many of your concerns.

comment by kalla724 · 2012-05-17T21:51:42.450Z · score: 1 (3 votes) · LW · GW

Yes. If we have an AGI, and someone sets forth to teach it how to be able to lie, I will get worried.

I am not worried about an AGI developing such an ability spontaneously.

comment by JoshuaZ · 2012-05-17T22:36:35.318Z · score: 5 (7 votes) · LW · GW

One of the most interesting things that I'm taking away from this conversation is that it seems that there are severe barriers to AGIs taking over or otherwise becoming extremely powerful. These largescale problems are present in a variety of different fields. Coming from a math/comp-sci perspective gives me strong skepticism about rapid self-improvement, while apparently coming from a neuroscience/cogsci background gives you strong skepticism about the AI's ability to understand or manipulate humans even if it extremely smart. Similarly, chemists seem highly skeptical of the strong nanotech sort of claims. It looks like much of the AI risk worry may come primarily from no one having enough across the board expertise to say "hey, that's not going to happen" to every single issue.

comment by JoshuaZ · 2012-05-17T21:59:32.788Z · score: 2 (2 votes) · LW · GW

What if people try to teach it about sarcasm or the like? Or simply have it learn by downloading a massive amount of literature and movies and look at those? And there are more subtle ways to learn about lying- AI being used for games is a common idea, how long will it take before someone decides to use a smart AI to play poker?

comment by dlthomas · 2012-05-17T01:26:18.438Z · score: 2 (4 votes) · LW · GW

The answer from the sequences is that yes, there is a limit to how much an AI can infer based on limited sensory data, but you should be careful not to assume that just because it is limited, it's limited to something near our expectations. Until you've demonstrated that FOOM cannot lie below that limit, you have to assume that it might (if you're trying to carefully avoid FOOMing).

comment by kalla724 · 2012-05-17T01:49:16.166Z · score: 4 (4 votes) · LW · GW

I'm not talking about limited sensory data here (although that would fall under point 2). The issue is much broader:

  • We humans have limited data on how the universe work
  • Only a limited subset of that limited data is available to any intelligence, real or artificial

Say that you make a FOOM-ing AI that has decided to make all humans dopaminergic systems work in a particular, "better" way. This AI would have to figure out how to do so from the available data on the dopaminergic system. It could analyze that data millions of times more effectively than any human. It could integrate many seemingly irrelevant details.

But in the end, it simply would not have enough information to design a system that would allow it to reach its objective. It could probably suggest some awesome and to-the-point experiments, but these experiments would then require time to do (as they are limited by the growth and development time of humans, and by the experimental methodologies involved).

This process, in my mind, limits the FOOM-ing speed to far below what seems to be implied by the SI.

This also limits bootstrapping speed. Say an AI develops a much better substrate for itself, and has access to the technology to create such a substrate. At best, this substrate will be a bit better and faster than anything humanity currently has. The AI does not have access to the precise data about basic laws of universe it needs to develop even better substrates, for the simple reason that nobody has done the experiments and precise enough measurements. The AI can design such experiments, but they will take real time (not computational time) to perform.

Even if we imagine an AI that can calculate anything from the first principles, it is limited by the precision of our knowledge of those first principles. Once it hits upon those limitations, it would have to experimentally produce new rounds of data.

comment by dlthomas · 2012-05-17T02:25:43.081Z · score: 3 (5 votes) · LW · GW

But in the end, it simply would not have enough information to design a system that would allow it to reach its objective.

I don't think you know that.

comment by Bugmaster · 2012-05-17T01:54:08.701Z · score: 1 (1 votes) · LW · GW

It could probably suggest some awesome and to-the-point experiments, but these experiments would then require time to do

Presumably, once the AI gets access to nanotechnology, it could implement anything it wants very quickly, bypassing the need to wait for tissues to grow, parts to be machined, etc.

I personally don't believe that nanotechnology could work at such magical speeds (and I doubt that it could even exist), but I could be wrong, so I'm playing a bit of Devil's Advocate here.

comment by kalla724 · 2012-05-17T02:24:28.623Z · score: 1 (1 votes) · LW · GW

Yes, but it can't get to nanotechnology without a whole lot of experimentation. It can't deduce how to create nanorobots, it would have to figure it out by testing and experimentation. Both steps limited in speed, far more than sheer computation.

comment by dlthomas · 2012-05-17T02:27:18.311Z · score: 2 (4 votes) · LW · GW

It can't deduce how to create nanorobots[.]

How do you know that?

comment by kalla724 · 2012-05-17T02:56:21.198Z · score: 2 (6 votes) · LW · GW

With absolute certainty, I don't. If absolute certainty is what you are talking about, then this discussion has nothing to do with science.

If you aren't talking about absolutes, then you can make your own estimation of likelihood that somehow an AI can derive correct conclusions from incomplete data (and then correct second order conclusions from those first conclusions, and third order, and so on). And our current data is woefully incomplete, many of our basic measurements imprecise.

In other words, your criticism here seems to boil down to saying "I believe that an AI can take an incomplete dataset and, by using some AI-magic we cannot conceive of, infer how to END THE WORLD."

Color me unimpressed.

comment by dlthomas · 2012-05-17T04:28:21.526Z · score: 3 (5 votes) · LW · GW

No, my criticism is "you haven't argued that it's sufficiently unlikely, you've simply stated that it is." You made a positive claim; I asked that you back it up.

With regard to the claim itself, it may very well be that AI-making-nanostuff isn't a big worry. For any inference, the stacking of error in integration that you refer to is certainly a limiting factor - I don't know how limiting. I also don't know how incomplete our data is, with regard to producing nanomagic stuff. We've already built some nanoscale machines, albeit very simple ones. To what degree is scaling it up reliant on experimentation that couldn't be done in simulation? I just don't know. I am not comfortable assigning it vanishingly small probability without explicit reasoning.

comment by kalla724 · 2012-05-17T05:25:24.673Z · score: 4 (4 votes) · LW · GW

Scaling it up is absolutely dependent on currently nonexistent information. This is not my area, but a lot of my work revolves around control of kinesin and dynein (molecular motors that carry cargoes via microtubule tracks), and the problems are often similar in nature.

Essentially, we can make small pieces. Putting them together is an entirely different thing. But let's make this more general.

The process of discovery has, so far throughout history, followed a very irregular path. 1- there is a general idea 2- some progress is made 3- progress runs into an unpredicted and previously unknown obstacle, which is uncovered by experimentation. 4- work is done to overcome this obstacle. 5- goto 2, for many cycles, until a goal is achieved - which may or may not be close to the original idea.

I am not the one who is making positive claims here. All I'm saying is that what has happened before is likely to happen again. A team of human researchers or an AGI can use currently available information to build something (anything, nanoscale or macroscale) to the place to which it has already been built. Pushing it beyond that point almost invariably runs into previously unforeseen problems. Being unforeseen, these problems were not part of models or simulations; they have to be accounted for independently.

A positive claim is that an AI will have a magical-like power to somehow avoid this - that it will be able to simulate even those steps that haven't been attempted yet so perfectly, that all possible problems will be overcome at the simulation step. I find that to be unlikely.

comment by Polymeron · 2012-05-20T17:32:04.920Z · score: 3 (3 votes) · LW · GW

It is very possible that the information necessary already exists, imperfect and incomplete though it may be, and enough processing of it would yield the correct answer. We can't know otherwise, because we don't spend thousands of years analyzing our current level of information before beginning experimentation, but in the shift between AI-time and human-time it can agonize on that problem for a good deal more cleverness and ingenuity than we've been able to apply to it so far.

That isn't to say, that this is likely; but it doesn't seem far-fetched to me. If you gave an AI the nuclear physics information we had in 1950, would it be able to spit out schematics for an H-bomb, without further experimentation? Maybe. Who knows?

comment by Strange7 · 2012-05-23T00:47:33.070Z · score: 0 (0 votes) · LW · GW

At the very least it would ask for some textbooks on electrical engineering and demolitions, first. The detonation process is remarkably tricky.

comment by dlthomas · 2012-05-17T21:28:53.455Z · score: 0 (2 votes) · LW · GW

I am not the one who is making positive claims here.

You did in the original post I responded to.

All I'm saying is that what has happened before is likely to happen again.

Strictly speaking, that is a positive claim. It is not one I disagree with, for a proper translation of "likely" into probability, but it is also not what you said.

"It can't deduce how to create nanorobots" is a concrete, specific, positive claim about the (in)abilities of an AI. Don't misinterpret this as me expecting certainty - of course certainty doesn't exist, and doubly so for this kind of thing. What I am saying, though, is that a qualified sentence such as "X will likely happen" asserts a much weaker belief than an unqualified sentence like "X will happen." "It likely can't deduce how to create nanorobots" is a statement I think I agree with, although one must be careful not use it as if it were stronger than it is.

A positive claim is that an AI will have a magical-like power to somehow avoid this.

That is not a claim I made. "X will happen" implies a high confidence - saying this when you expect it is, say, 55% likely seems strange. Saying this when you expect it to be something less than 10% likely (as I do in this case) seems outright wrong. I still buckle my seatbelt, though, even though I get in a wreck well less than 10% of the time.

This is not to say I made no claims. The claim I made, implicitly, was that you made a statement about the (in)capabilities of an AI that seemed overconfident and which lacked justification. You have given some justification since (and I've adjusted my estimate down, although I still don't discount it entirely), in amongst your argument with straw-dlthomas.

comment by kalla724 · 2012-05-17T21:42:21.276Z · score: 1 (1 votes) · LW · GW

You are correct. I did not phrase my original posts carefully.

I hope that my further comments have made my position more clear?

comment by Bugmaster · 2012-05-17T05:57:47.581Z · score: 0 (0 votes) · LW · GW

FWIW I think you are likely to be right. However, I will continue in my Nanodevil's Advocate role.

You say,

A positive claim is that an AI ... will be able to simulate even those steps that haven't been attempted yet so perfectly, that all possible problems will be overcome at the simulation step

I think this depends on what the AI wants to build, on how complete our existing knowledge is, and on how powerful the AI is. Is there any reason why the AI could not (given sufficient computational resources) run a detailed simulation of every atom that it cares about, and arrive at a perfect design that way ? In practice, its simulation won't need be as complex as that, because some of the work had already been performed by human scientists over the ages.

comment by kalla724 · 2012-05-17T17:55:22.780Z · score: 4 (4 votes) · LW · GW

By all means, continue. It's an interesting topic to think about.

The problem with "atoms up" simulation is the amount of computational power it requires. Think about the difference in complexity when calculating a three-body problem as compared to a two-body problem?

Than take into account the current protein folding algorithms. People have been trying to calculate folding of single protein molecules (and fairly short at that) by taking into account the main physical forces at play. In order to do this in a reasonable amount of time, great shortcuts have to be taken - instead of integrating forces, changes are treated as stepwise, forces beneath certain thresholds are ignored, etc. This means that a result will always have only a certain probability of being right.

A self-replicating nanomachine requires minimal motors, manipulators and assemblers; while still tiny, it would be a molecular complex measured in megadaltons. To precisely simulate creation of such a machine, an AI that is trillion times faster than all the computers in the world combined would still require decades, if not centuries of processing time. And that is, again, assuming that we know all the forces involved perfectly, which we don't (how will microfluidic effects affect a particular nanomachine that enters human bloodstream, for example?).

comment by Bugmaster · 2012-05-17T22:49:01.654Z · score: 0 (0 votes) · LW · GW

Yes, this is a good point. That said, while protein folding had not been entirely solved yet, it had been greatly accelerated by projects such as FoldIt, which leverage multiple human minds working in parallel on the problem all over the world. Sure, we can't get a perfect answer with such a distributed/human-powered approach, but a perfect answer isn't really required in practice; all we need is an answer that has a sufficiently high chance of being correct.

If we assume that there's nothing supernatural (or "emergent") about human minds [1], then it is likely that the problem is at least tractable. Given the vast computational power of existing computers, it is likely that the AI would have access to at least as many computational resources as the sum of all the brains who are working on FoldIt. Given Moore's Law, it is likely that the AI would soon surpass FoldIt, and will keep expanding its power exponentially, especially if the AI is able to recursively improve its own hardware (by using purely conventional means, at least initially).

[1] Which is an assumption that both my Nanodevil's Advocate persona and I share.

comment by JoshuaZ · 2012-05-17T22:58:00.306Z · score: 3 (3 votes) · LW · GW

Protein folding models are generally at least as bad as NP-hard, and some models may be worse. This means that exponential improvement is unlikely. Simply put, one probably gets diminishing marginal returns for how much one can computer further in terms of how much improvement one has already done.

comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2013-03-22T07:19:58.611Z · score: 3 (3 votes) · LW · GW

Protein folding models must be inaccurate if they are NP-hard. Reality itself is not known to be able to solve NP-hard problems.

comment by Kawoomba · 2013-03-22T08:49:48.746Z · score: 2 (2 votes) · LW · GW

Reality itself is not known to be able to solve NP-hard problems.

Yet the proteins are folding. Is that not "reality" solving the problem?

comment by CCC · 2013-03-22T09:31:43.099Z · score: 3 (3 votes) · LW · GW

If reality cannot solve NP-hard problems as easily as proteins are being folded, and yet proteins are getting folded, then that implies that one of the following must be true:

  1. It turns out that reality can solve NP-hard problems after all
  2. Protein folding is not an NP-hard problem (which implies that it is not properly understood)
  3. Reality is not solving protein folding; it merely has a very good approximation that works on some but not necessarily all proteins (including most examples found in nature)
comment by Kawoomba · 2013-03-22T09:41:49.706Z · score: 0 (0 votes) · LW · GW

Yes, and I'm leaning towards 1.

I am not familiar whether e.g. papers like these ("We show that the protein folding problem in the two-dimensional H-P model is NP-complete.") accurately models what we'd call "protein folding" in nature (just because the same name is used), but prima facie there is no reason to doubt the applicability, at least for the time being. (This precludes 2.)

Regarding 3, I don't think it would make sense to say "reality is using only a good approximation of protein folding, and by the way, we define protein folding as that which occurs in nature." That which happens in reality is precisely - and by definition not only an approximation of - that which we call "protein folding", isn't it?

What do you think?

comment by Cyan · 2013-03-23T01:17:11.264Z · score: 8 (8 votes) · LW · GW

It's #3. (B.Sc. in biochemistry, did my Ph.D. in proteomics.)

First, the set of polypeptide sequences that have a repeatable final conformation (and therefore "work" biologically) is tiny in comparison to the set of all possible sequences (of the 20-or-so naturally amino acid monomers). Pick a random sequence of reasonable length and make many copies and you get a gummy mess. The long slow grind of evolution has done the hard work of finding useful sequences.

Second, there is an entire class of proteins called chaperones) that assist macromolecular assembly, including protein folding. Even so, folding is a stochastic process, and a certain amount of newly synthesized proteins misfold. Some chaperones will then tag the misfolded protein with ubiquitin, which puts it on a path that ends in digestion by a proteasome.

comment by CCC · 2013-03-23T08:07:55.461Z · score: 4 (4 votes) · LW · GW

Thank you, Cyan. It's good to occasionally get someone into the debate who actually has a good understanding of the subject matter.

comment by gwern · 2013-03-23T02:42:33.205Z · score: 0 (0 votes) · LW · GW

Aaronson used to blog about instances where people thought they found nature solving a hard problem very quickly, and usually there turns out to be a problem like the protein misfolding thing; the last instance I remember was soap films/bubbles perhaps solving NP problems by producing minimal Steiner trees, and Aaronson wound up experimenting with them himself. Fun stuff.

comment by CCC · 2013-03-22T10:14:46.036Z · score: 5 (5 votes) · LW · GW

Apologies; looking back at my post, I wasn't clear on 3.

Protein folding, as I understand it, is the process of finding a way to fold a given protein that globally minimizes some mathematical function. I'm not sure what that function is, but this is the definition that I used in my post.

Option 2 raises the possibility that globally minimizing that function is not NP-hard, but is merely misunderstood in some way.

Option 3 raises the possibility that proteins are not (in nature) finding a global minimum; rather, they are finding a local minimum through a less computationally intensive process. Furthermore, it may be that, for proteins which have certain limits on their structure and/or their initial conditions, that local minimum is the same as the global minimum; this may lead to natural selection favouring structures which use such 'easy proteins', leading to the incorrect impression that a general global minimum is being found (as opposed to a handy local minimum).

comment by Cyan · 2013-03-23T01:42:36.029Z · score: 4 (4 votes) · LW · GW

this may lead to natural selection favouring structures which use such 'easy proteins', leading to the incorrect impression that a general global minimum is being found (as opposed to a handy local minimum).

Yup.

comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2013-03-22T11:58:10.289Z · score: 2 (2 votes) · LW · GW

No. Not 1. It would be front-page news all over the universe if it were 1.

comment by Kawoomba · 2013-03-22T19:54:42.259Z · score: 1 (1 votes) · LW · GW

NP hard problems are solvable (in the theoretical sense) by definition, the problem lies in their resource requirements (running time, for the usual complexity classes) as defined in relation to a UTM. (You know this, just establishing a basis.)

The assumption that the universe can be perfectly described by a computable model is satisfied just by a theoretical computational description existing, it says nothing about tractability (running times) and being able to submerge complexity classes in reality fluid or having some thoroughly defined correspondence (other than when we build hardware models ourselves, for which we define all the relevant parameters, e.g. CPU clock speed).

You may think along the lines of "if reality could (easily) solve NP hard problems for arbitrarily chosen and large inputs, we could mimick that approach and thus have a P=NP proving algorithm", or something along those lines.

My difficulty is in how even to describe the "number of computational steps" that reality takes - do we measure it in relation to some computronium-hardware model, do we take it as discrete or continuous, what's the sampling rate, picoseconds (as CCC said further down), Planck time intervals, or what?

In short, I have no idea about the actual computing power in terms of resource requirements of the underlying reality fluid, and thus can't match it against UTMs in order to compare running times. Maybe you can give me some pointers.

comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2013-03-22T22:22:26.680Z · score: 5 (5 votes) · LW · GW

Kawoomba, there is no known case of any NP-hard or NP-complete solution which physics finds.

In the case of proteins, if finding the lowest energy minimum of an arbitrary protein is NP-hard, then what this means in practice is that some proteins will fold up into non-lowest-energy configurations. There is no known case of a quantum process which finds an NP-hard solution to anything, including an energy minimum; on our present understanding of complexity theory and quantum mechanics 'quantum solvable' is still looking like a smaller class than 'NP solvable'. Read Scott Aaronson for more.

comment by Qiaochu_Yuan · 2013-03-22T22:37:38.606Z · score: 7 (7 votes) · LW · GW

One example here is the Steiner tree problem, which is NP-complete and can sort of be solved using soap films. Bringsjord and Taylor claimed this implies that P = NP. Scott Aaronson did some experimentation and found that soap films 1) can get stuck at local minima and 2) might take a long time to settle into a good configuration.

comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2013-03-22T23:29:22.199Z · score: 2 (2 votes) · LW · GW

Heh. I remember that one, and thinking, "No... no, you can't possibly do that using a soap bubble, that's not even quantum and you can't do that in classical, how would the soap molecules know where to move?"

comment by Manfred · 2013-03-23T00:15:02.440Z · score: 2 (2 votes) · LW · GW

Well. I mean, it's quantum. But the ground state is a lump of iron, or maybe a black hole, not a low-energy soap film, so I don't think waiting for quantum annealing will help.

comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2013-03-23T01:41:40.941Z · score: 2 (2 votes) · LW · GW

waves soap-covered wire so it settles into low-energy minimum

dies as it turns into iron

comment by [deleted] · 2013-03-23T18:51:09.881Z · score: 0 (0 votes) · LW · GW

I also seem to recall Penrose hypothesizing something about quasicrystals, though he does have an axe to grind so I'm quite sceptical.

comment by CCC · 2013-03-23T07:28:47.226Z · score: 0 (0 votes) · LW · GW

I saw someone do the experiment once (school science project). Soap bubbles are pretty good at solving three- and four-element cases, as long as you make sure that all the points are actually connected.

I don't think that three- and four-element cases have local minima, do they? That avoids (1) and a bit of gentle shaking can help speed up (2).

comment by Cyan · 2013-03-23T01:32:12.701Z · score: 2 (2 votes) · LW · GW

In the case of proteins, if finding the lowest energy minimum of an arbitrary protein is NP-hard, then what this means in practice is that some proteins will fold up into non-lowest-energy configurations.

Yup.

comment by CCC · 2013-03-22T23:15:57.178Z · score: 1 (1 votes) · LW · GW

My difficulty is in how even to describe the "number of computational steps" that reality takes

Probably the best way is to simply define a "step" in some easily measurable way, and then sit there with a stopwatch and try a few experiments. (For protein folding, the 'stopwatch' may need to be a fairly sophisticated piece of scientific timing instrumentation, of course, and observing the protein as it folds is rather tricky).

Another way is to take advantage of the universal speed limit to get a theoretical upper bound to the speed that reality runs at; assume that the protein molecule folds in a brute-force search pattern that never ends until it hits the right state, and assume that at any point in that process, the fastest-moving part of the molecule moves at the speed of light (it won't have to move far, which helps) and that the sudden, intense acceleration doesn't hurt the molecule. It's pretty certain to be slower than that, so if this calculation says it takes longer than an hour, then it's pretty clear that the brute force approach is not what the protein is using.

comment by OrphanWilde · 2013-03-22T13:47:22.174Z · score: 0 (0 votes) · LW · GW

What exactly am I missing in this argument? Evolution is perfectly capable of brute-force solutions. That's pretty much what it's best at.

comment by CCC · 2013-03-22T14:12:00.326Z · score: 2 (2 votes) · LW · GW

The brute-force solution, if sampling conformations at picosecond rates, has been estimated to require a time longer than the age of the universe to fold certain proteins. Yet proteins fold on a millisecond scale or faster.

See: Levinthal's paradox.

comment by OrphanWilde · 2013-03-22T14:34:56.599Z · score: 0 (0 votes) · LW · GW

That requires that the proteins fold more or less randomly, and that the brute-force algorithm is in the -folding-, rather than the development of mechanisms which force certain foldings.

In order for the problem to hold, one of three things has to hold true: 1.) The proteins fold randomly (evidence suggests otherwise, as mentioned in the wikipedia link) 2.) Only a tiny subset of possible forced foldings are useful (that is, if there are a billion different ways for protein to be forced to fold in a particular manner, only one of them does what the body needs them to do) - AND anthropic reasoning isn't valid (that is, we can't say that our existence requires that evolution solved this nearly-impossible-to-arrive-at-through-random-processes) 3.) The majority of possible forced holdings are incompatible (that is, if protein A folds one way, then protein B -must- fold in a particular manner, or life isn't possible) - AND anthropic reasoning isn't valid

ETA: If anthropic reasoning is valid AND either 2 or 3 hold otherwise, it suggests our existence was considerably less likely than we might otherwise expect.

comment by CCC · 2013-03-22T18:14:29.001Z · score: 0 (0 votes) · LW · GW

That requires that the proteins fold more or less randomly, and that the brute-force algorithm is in the -folding-, rather than the development of mechanisms which force certain foldings.

Ah. I apologise for having misunderstood you.

In that case, yes, the mechanisms for the folding may very well have developed by a brute-force type algorithm, for all I know. (Which, on this topic, isn't all that much) But... what are those mechanisms?

comment by CCC · 2013-03-22T08:16:07.668Z · score: 2 (2 votes) · LW · GW

Google has pointed me to an article describing an algorithm that can apparently predict folded protein shapes pretty quickly (a few minutes on a single laptop).

Original paper here. From a quick glance, it looks like it's only effective for certain types of protein chains.

comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2013-03-22T08:17:34.876Z · score: 0 (0 votes) · LW · GW

That too. Even NP-hard problems are often easy if you get the choice of which one to solve.

comment by Bugmaster · 2012-05-17T23:21:32.734Z · score: 0 (0 votes) · LW · GW

Hmm, ok, my Nanodevil's Advocate persona doesn't have a good answer to this one. Perhaps some SIAI folks would like to step in and pick up the slack ?

comment by Polymeron · 2012-05-20T17:45:29.999Z · score: 6 (6 votes) · LW · GW

I'm afraid not.

Actually, as someone with background in Biology I can tell you that this is not a problem you want to approach atoms-up. It's been tried, and our computational capabilities fell woefully short of succeeding.

I should explain what "woefully short" means, so that the answer won't be "but can't the AI apply more computational power than us?". Yes, presumably it can. But the scales are immense. To explain it, I will need an analogy.

Not that long ago, I had the notion that chess could be fully solved; that is, that you could simply describe every legal position and every position possible to reach from it, without duplicates, so you could use that decision tree to play a perfect game. After all, I reasoned, it's been done with checkers; surely it's just a matter of getting our computational power just a little bit better, right?

First I found a clever way to minimize the amount of bits necessary to describe a board position. I think I hit 34 bytes per position or so, and I guess further optimization was possible. Then, I set out to calculate how many legal board positions there are.

I stopped trying to be accurate about it when it turned out that the answer was in the vicinity of 10^68, give or take a couple orders of magnitude. That's about a billionth billionth of the TOTAL NUMBER OF ATOMS IN THE ENTIRE UNIVERSE. You would literally need more than our entire galaxy made into a huge database just to store the information, not to mention accessing it and computing on it.

So, not anytime soon.

Now, the problem with protein folding is, it's even more complex than chess. At the atomic level, it's incredibly more complex than chess. Our luck is, you don't need to fully solve it; just like today's computers can beat human chess players without spanning the whole planet. But they do it with heuristics, approximations, sometimes machine learning (though that just gives them more heuristics and approximations). We may one day be able to fold proteins, but we will do so by making assumptions and approximations, generating useful rules of thumb, not by modeling each atom.

comment by Bugmaster · 2012-05-20T20:57:24.618Z · score: 3 (3 votes) · LW · GW

Yes, I understand what "exponential complexity" means :-)

It sounds, then, like you're on the side of kalla724 and myself (and against my Devil's Advocate persona): the AI would not be able to develop nanotechnology (or any other world-shattering technology) without performing physical experiments out in meatspace. It could do so in theory, but in practice, the computational requirements are too high.

But this puts severe constraints on the speed with which the AI's intelligence explosion could occur. Once it hits the limits of existing technology, it will have to take a long slog through empirical science, at human-grade speeds.

comment by Polymeron · 2012-05-23T16:35:32.634Z · score: 1 (1 votes) · LW · GW

Actually, I don't know that this means it has to perform physical experiments in order to develop nanotechnology. It is quite conceivable that all the necessary information is already out there, but we haven't been able to connect all the dots just yet.

At some point the AI hits a wall in the knowledge it can gain without physical experiments, but there's no good way to know how far ahead that wall is.

comment by Bugmaster · 2012-05-23T21:13:22.266Z · score: 1 (1 votes) · LW · GW

It is quite conceivable that all the necessary information is already out there, but we haven't been able to connect all the dots just yet.

Wouldn't this mean that creating fully functional self-replicating nanotechnology is just a matter of performing some thorough interdisciplinary studies (or meta-studies or whatever they are called) ? My impression was that there are currently several well-understood -- yet unresolved -- problems that prevent nanofactories from becoming a reality, though I could be wrong.

comment by Polymeron · 2012-05-24T08:57:11.074Z · score: 1 (1 votes) · LW · GW

The way I see it, there's no evidence that these problems require additional experimentation to resolve, rather than find an obscure piece of experimentation that has already taken place and whose relevance may not be immediately obvious.

Sure, that more experimentation is needed is probable; but by no means certain.

comment by CCC · 2013-03-22T07:56:53.824Z · score: 0 (0 votes) · LW · GW

Thorough interdisciplinary studies may or may not lead to nanotechnology, but they're fairly certain to lead to something new. While there are a fair number of (say) marine biologists out there, and a fair number of astronomers, there are probably rather few people who have expertise in both fields; and it's possible that there exists some obscure unsolved problem in marine biology whose solution is obvious to someone who's keeping up on the forefront of astronomy research. Or vice versa.

Or substitute in any other two fields of your choice.

comment by Kawoomba · 2013-04-17T08:58:29.652Z · score: 2 (2 votes) · LW · GW

First I found a clever way to minimize the amount of bits necessary to describe a board position. I think I hit 34 bytes per position or so, and I guess further optimization was possible.

Indeed, using a very straightforward Huffman encoding (1 bit for an for empty cell, 3 bits for pawns) you can get it down to 24 bytes for the board alone. Was an interesting puzzle.

Looking up "prior art" on the subject, you also need 2 bytes for things like "may castle", and other more obscure rules.

There's further optimizations you can do, but they are mostly for the average case, not the worst case.

comment by Polymeron · 2013-04-23T18:50:01.760Z · score: 2 (2 votes) · LW · GW

I didn't consider using 3 bits for pawns! Thanks for that :) I did account for such variables as may castle and whose turn it is.

comment by RichardKennaway · 2013-03-22T08:06:19.942Z · score: 1 (1 votes) · LW · GW

It's been tried, and our computational capabilities fell woefully short of succeeding.

Is that because we don't have enough brute force, or because we don't know what calculation to apply it to?

I would be unsurprised to learn that calculating the folding state having global minimum energy was NP-complete; but for that reason I would be surprised to learn that nature solves that problem, rather than finding a local minimum.

I don't have a background in biology, but my impression from Wikipedia is that the tension between Anfinsen's dogma and Levinthal's paradox is yet unresolved.

comment by Polymeron · 2013-04-17T07:16:04.025Z · score: 1 (1 votes) · LW · GW

The two are not in conflict.

A-la Levinthal's paradox, I can say that throwing a marble down a conical hollow at different angles and force can have literally trillions of possible trajectories; a-la Anfinsen's dogma, that should not stop me from predicting that it will end up at the bottom of the cone; but I'd need to know the shape of the cone (or, more specifically, its point's location) to determine exactly where that is - so being able to make the prediction once I know this is of no assistance for predicting the end position with a different, unknown cone.

Similarly, Eliezer is able to predict that a grandmaster chess player would be able to bring a board to a winning position against himself, even though he has no idea what moves that would entail or which of the many trillions of possible move sets the game would be comprised of.

Problems like this cannot be solved on brute force alone; you need to use attractors and heuristics to get where you want to get.

So yes, obviously nature stumbled into certain stable configurations which propelled it forward, rather than solve the problem and start designing away. But even if we can never have enough computing power to model each and every atom in each and every configuration, we might still get a good enough understanding of the general laws for designing proteins almost from scratch.

comment by Strange7 · 2013-03-22T06:52:38.087Z · score: 0 (0 votes) · LW · GW

I would think it would be possible to cut the space of possible chess positions down quite a bit by only retaining those which can result from moves the AI would make, and legal moves an opponent could make in response. That is, when it becomes clear that a position is unwinnable, backtrack, and don't keep full notes on why it's unwinnable.

comment by Polymeron · 2013-04-17T07:26:44.412Z · score: 1 (1 votes) · LW · GW

This is more or less what computers do today to win chess matches, but the space of possibilities explodes too fast; even the strongest computers can't really keep track of more than I think 13 or 14 moves ahead, even given a long time to think.

Merely storing all the positions that are unwinnable - regardless of why they are so - would require more matter than we have in the solar system. Not to mention the efficiency of running a DB search on that...

comment by CCC · 2013-04-17T09:51:02.318Z · score: 3 (3 votes) · LW · GW

Not to mention the efficiency of running a DB search on that...

Actually, with proper design, that can be made very quick and easy. You don't need to store the positions; you just need to store the states (win:black, win:white, draw - two bits per state).

The trick is, you store each win/loss state in a memory address equal to the 34-byte (or however long) binary number that describes the position in question. Checking a given state is then simply a memory retrieval from a known address.

comment by Polymeron · 2013-04-23T18:55:27.614Z · score: 1 (1 votes) · LW · GW

I suspect that with memory on the order of 10^70 bytes, that might involve additional complications; but you're correct, normally this cancels out the complexity problem.

comment by wedrifid · 2013-04-17T10:20:12.366Z · score: 1 (1 votes) · LW · GW

Merely storing all the positions that are unwinnable - regardless of why they are so - would require more matter than we have in the solar system. Not to mention the efficiency of running a DB search on that...

The storage space problem is insurmountable. However searching that kind of database would be extremely efficient (if the designer isn't a moron). The search speed would have a lower bound of very close to (diameter of the sphere that can contain the database / c). Nothing more is required for search purposes than physically getting a signal to the relevant bit, and back, with only minor deviations from a straight line each way. And that is without even the most obvious optimisations.

If your chess opponent is willing to fly with you in a relativistic rocket and you only care about time elapsed from your own reference frame rather than the reference frame of the computer (or most anything else of note) you can even get down below that diameter / light speed limit, depending on your available fuel and the degree of accelleration you can survive.

comment by Bugmaster · 2012-05-17T03:03:07.311Z · score: 3 (3 votes) · LW · GW

Speaking as Nanodevil's Advocate again, one objection I could bring up goes as follows:

While it is true that applying incomplete knowledge to practical tasks (such as ending the world or whatnot) is difficult, in this specific case our knowledge is complete enough. We humans currently have enough scientific data to develop self-replicating nanotechnology within the next 20 years (which is what we will most likely end up doing). An AI would be able to do this much faster, since it is smarter than us; is not hampered by our cognitive and social biases; and can integrate information from multiple sources much better than we can.

comment by kalla724 · 2012-05-17T05:26:09.357Z · score: 0 (0 votes) · LW · GW

See my answer to dlthomas.

comment by jacob_cannell · 2012-05-17T13:45:28.769Z · score: 1 (1 votes) · LW · GW

Point 1 has come up in at least one form I remember. There was an interesting discussion some while back about limits to the speed of growth of new computer hardware cycles which have critical endsteps which don't seem amenable to further speedup by intelligence alone. The last stages of designing a microchip involve a large amount of layout solving, physical simulation, and then actual physical testing. These steps are actually fairly predicatable, where it takes about C amounts of computation using certain algorithms to make a new microchip, the algorithms are already best in complexity class (so further improvments will be minor), and C is increasing in a predictable fashion. These models are actually fairly detailed (see the semiconductor roadmap, for example). If I can find that discussion soon before I get distracted I'll edit it into this discussion.

Note however that 1, while interesting, isn't a fully general counteargument against a rapid intelligence explosion, because of the overhang issue if nothing else.

Point 2 has also been discussed. Humans make good 'servitors'.

Do you have a plausible scenario how a "FOOM"-ing AI could - no matter how intelligent - minimize oxygen content of our planet's atmosphere, or any such scenario?

Oh that's easy enough. Oxygen is highly reactive and unstable. Its existence on a planet is entirely dependent on complex organic processes, ie life. No life, no oxygen. Simple solution: kill large fraction of photosynthesizing earth-life. Likely paths towards goal:

  1. coordinated detonation of large number of high yield thermonuclear weapons
  2. self-replicating nanotechnology.
comment by kalla724 · 2012-05-17T18:00:04.795Z · score: 3 (3 votes) · LW · GW

I'm vaguely familiar with the models you mention. Correct me if I'm wrong, but don't they have a final stopping point, which we are actually projected to reach in ten to twenty years? At a certain point, further miniaturization becomes unfeasible, and the growth of computational power slows to a crawl. This has been put forward as one of the main reasons for research into optronics, spintronics, etc.

We do NOT have sufficient basic information to develop processors based on simulation alone in those other areas. Much more practical work is necessary.

As for point 2, can you provide a likely mechanism by which a FOOMing AI could detonate a large number of high-yield thermonuclear weapons? Just saying "human servitors would do it" is not enough. How would the AI convince the human servitors to do this? How would it get access to data on how to manipulate humans, and how would it be able to develop human manipulation techniques without feedback trials (which would give away its intention)?

comment by JoshuaZ · 2012-05-17T18:17:08.680Z · score: 4 (4 votes) · LW · GW

The thermonuclear issue actually isn't that implausible. There have been so many occasions where humans almost went to nuclear war over misunderstandings or computer glitches, that the idea that a highly intelligent entity could find a way to do that doesn't seem implausible, and exact mechanism seems to be an overly specific requirement.

comment by kalla724 · 2012-05-17T19:00:57.228Z · score: 3 (3 votes) · LW · GW

I'm not so much interested in the exact mechanism of how humans would be convinced to go to war, as in an even approximate mechanism by which an AI would become good at convincing humans to do anything.

Ability to communicate a desire and convince people to take a particular course of action is not something that automatically "falls out" from an intelligent system. You need a theory of mind, an understanding of what to say, when to say it, and how to present information. There are hundreds of kids on autistic spectrum who could trounce both of us in math, but are completely unable to communicate an idea.

For an AI to develop these skills, it would somehow have to have access to information on how to communicate with humans; it would have to develop the concept of deception; a theory of mind; and establish methods of communication that would allow it to trick people into launching nukes. Furthermore, it would have to do all of this without trial communications and experimentation which would give away its goal.

Maybe I'm missing something, but I don't see a straightforward way something like that could happen. And I would like to see even an outline of a mechanism for such an event.

comment by [deleted] · 2012-05-17T19:40:58.558Z · score: 3 (3 votes) · LW · GW

For an AI to develop these skills, it would somehow have to have access to information on how to communicate with humans; it would have to develop the concept of deception; a theory of mind; and establish methods of communication that would allow it to trick people into launching nukes. Furthermore, it would have to do all of this without trial communications and experimentation which would give away its goal.

I suspect the Internet contains more than enough info for a superhuman AI to develop a working knowledge of human psychology.

comment by kalla724 · 2012-05-17T20:09:30.955Z · score: 2 (2 votes) · LW · GW

Only if it has the skills required to analyze and contextualize human interactions. Otherwise, the Internet is a whole lot of jibberish.

Again, these skills do not automatically fall out of any intelligent system.

comment by XiXiDu · 2012-05-18T09:14:41.727Z · score: 0 (0 votes) · LW · GW

I suspect the Internet contains more than enough info for a superhuman AI to develop a working knowledge of human psychology.

I don't see what justifies that suspicion.

Just imagine you emulated a grown up human mind and it wanted to become a pick up artist, how would it do that with an Internet connection? It would need some sort of avatar, at least, and then wait for the environment to provide a lot of feedback.

Therefore even if we’re talking about the emulation of a grown up mind, it will be really hard to acquire some capabilities. Then how is the emulation of a human toddler going to acquire those skills? Even worse, how is some sort of abstract AGI going to do it that misses all of the hard coded capabilities of a human toddler?

Can we even attempt to imagine what is wrong about a boxed emulation of a human toddler, that makes it unable to become a master of social engineering in a very short time?

comment by NancyLebovitz · 2012-05-18T12:47:15.219Z · score: 2 (2 votes) · LW · GW

Humans learn most of what they know about interacting with other humans by actual practice. A superhuman AI might be considerably better than humans at learning by observation.

comment by [deleted] · 2012-05-18T17:39:42.100Z · score: 1 (1 votes) · LW · GW

Just imagine you emulated a grown up human mind

As a “superhuman AI” I was thinking about a very superhuman AI; the same does not apply to slightly superhuman AI. (OTOH, if Eliezer is right then the difference between a slightly superhuman AI and a very superhuman one is irrelevant, because as soon as a machine is smarter than its designer, it'll be able to design a machine smarter than itself, and its child an even smarter one, and so on until the physical limits set in.)

all of the hard coded capabilities of a human toddler

The hard coded capabilities are likely overrated, at least in language acquisition. (As someone put it, the Kolgomorov complexity of the innate parts of a human mind cannot possibly be more than that of the human genome, hence if human minds are more complex than that the complexity must come from the inputs.)

Also, statistic machine translation is astonishing -- by now Google Translate translations from English to one of the other UN official languages and vice versa are better than a non-completely-ridiculously-small fraction of translations by humans. (If someone had shown such a translation to me 10 years ago and told me “that's how machines will translate in 10 years”, I would have thought they were kidding me.)

comment by JoshuaZ · 2012-05-17T19:04:17.872Z · score: 0 (0 votes) · LW · GW

Let's do the most extreme case: AI's controlers give it general internet access to do helpful research. So it gets to find out about general human behavior and what sort of deceptions have worked in the past. Many computer systems that should't be online are online (for the US and a few other governments). Some form of hacking of relevant early warning systems would then seem to be the most obvious line of attack. Historically, computer glitches have pushed us very close to nuclear war on multiple occasions.

comment by kalla724 · 2012-05-17T20:12:45.047Z · score: 3 (3 votes) · LW · GW

That is my point: it doesn't get to find out about general human behavior, not even from the Internet. It lacks the systems to contextualize human interactions, which have nothing to do with general intelligence.

Take a hugely mathematically capable autistic kid. Give him access to the internet. Watch him develop ability to recognize human interactions, understand human priorities, etc. to a sufficient degree that it recognizes that hacking an early warning system is the way to go?

comment by JoshuaZ · 2012-05-17T20:15:47.004Z · score: 1 (1 votes) · LW · GW

Well, not necessarily, but an entity that is much smarter than an autistic kid might notice that, especially if it has access to world history (or heck many conversations on the internet about the horrible things that AIs do simply in fiction). It doesn't require much understanding of human history to realize that problems with early warning systems have almost started wars in the past.

comment by kalla724 · 2012-05-17T20:20:46.168Z · score: 3 (3 votes) · LW · GW

Yet again: ability to discern which parts of fiction accurately reflect human psychology.

An AI searches the internet. It finds a fictional account about early warning systems causing nuclear war. It finds discussions about this topic. It finds a fictional account about Frodo taking the Ring to Mount Doom. It finds discussions about this topic. Why does this AI dedicate its next 10^15 cycles to determination of how to mess with the early warning systems, and not to determination of how to create One Ring to Rule them All?

(Plus other problems mentioned in the other comments.)

comment by JoshuaZ · 2012-05-17T20:35:42.373Z · score: 3 (5 votes) · LW · GW

There are lots of tipoffs to what is fictional and what is real. It might notice for example the Wikipedia article on fiction describes exactly what fiction is and then note that Wikipedia describes the One Ring as fiction, and that Early warning systems are not. I'm not claiming that it will necessarily have an easy time with this. But the point is that there are not that many steps here, and no single step by itself looks extremely unlikely once one has a smart entity (which frankly to my mind is the main issue here- I consider recursive self-improvement to be unlikely).

comment by kalla724 · 2012-05-17T21:40:19.972Z · score: 1 (1 votes) · LW · GW

We are trapped in an endless chain here. The computer would still somehow have to deduce that Wikipedia entry that describes One Ring is real, while the One Ring itself is not.

comment by jacob_cannell · 2012-05-17T23:06:27.217Z · score: 0 (0 votes) · LW · GW

We observer that Wikipedia is mainly truthful. From that we infer "entry that describes "One Ring" is real". From use of term fiction/story in that entry, we refer that "One Ring" is not real.

Somehow you learned that Wikipedia is mainly truthful/nonfictional and that "One Ring" is fictional. So your question/objection/doubt is really just the typical boring doubt of AGI feasibility in general.

comment by JoshuaZ · 2012-05-17T23:13:14.149Z · score: 1 (1 votes) · LW · GW

But even humans have trouble with this sometimes. I was recently reading the Wikipedia article Hornblower and the Crisis which contains a link to the article on Francisco de Miranda. It took me time and cues when I clicked on it to realize that de Miranda was a historical figure.

So your question/objection/doubt is really just the typical boring doubt of AGI feasibility in general.

Isn't Kalla's objection more a claim that fast takeovers won't happen because even with all this data, the problems of understanding humans and our basic cultural norms will take a long time for the AI to learn and that in the meantime we'll develop a detailed understanding of it, and it is that hostile it is likely to make obvious mistakes in the meantime?

comment by Strange7 · 2012-05-22T23:49:34.355Z · score: -1 (1 votes) · LW · GW

Why would the AI be mucking around on Wikipedia to sort truth from falsehood, when Wikipedia itself has been criticized for various errors and is fundamentally vulnerable to vandalism? Primary sources are where it's at. Looking through the text of The Hobbit and Lord of the Rings, it's presented as a historical account, translated by a respected professor, with extensive footnotes. There's a lot of cultural context necessary to tell the difference.

comment by XiXiDu · 2012-05-17T19:20:59.818Z · score: 3 (5 votes) · LW · GW

Let's do the most extreme case: AI's controlers give it general internet access to do helpful research. So it gets to find out about general human behavior and what sort of deceptions have worked in the past.

None work reasonably well. Especially given that human power games are often irrational.

There are other question marks too.

The U.S. has many more and smarter people than the Taliban. The bottom line is that the U.S. devotes a lot more output per man-hour to defeat a completely inferior enemy. Yet they are losing.

The problem is that you won't beat a human at Tic-tac-toe just because you thought about it for a million years.

You also won't get a practical advantage by throwing more computational resources at the travelling salesman problem and other problems in the same class.

You are also not going to improve a conversation in your favor by improving each sentence for thousands of years. You will shortly hit diminishing returns. Especially since you lack the data to predict human opponents accurately.

comment by JoshuaZ · 2012-05-17T19:40:36.725Z · score: 3 (3 votes) · LW · GW

Especially given that human power games are often irrational.

So? As long as they follow minimally predictable patterns it should be ok.

The U.S. has many more and smarter people than the Taliban. The bottom line is that the U.S. devotes a lot more output per man-hour to defeat a completely inferior enemy. Yet they are losing.

Bad analogy. In this case the Taliban has a large set of natural advantages, the US has strong moral constraints and goal constraints (simply carpet bombing the entire country isn't an option for example).

You are also not going to improve a conversation in your favor by improving each sentence for thousands of years. You will shortly hit diminishing returns. Especially since you lack the data to predict human opponents accurately.

This seems like an accurate and a highly relevant point. Searching a solution space faster doesn't mean one can find a better solution if it isn't there.

comment by kalla724 · 2012-05-17T20:14:39.738Z · score: 3 (3 votes) · LW · GW

This seems like an accurate and a highly relevant point. Searching a solution space faster doesn't mean one can find a better solution if it isn't there.

Or if your search algorithm never accesses relevant search space. Quantitative advantage in one system does not translate into quantitative advantage in a qualitatively different system.

comment by XiXiDu · 2012-05-18T10:28:59.809Z · score: 2 (2 votes) · LW · GW

The U.S. has many more and smarter people than the Taliban. The bottom line is that the U.S. devotes a lot more output per man-hour to defeat a completely inferior enemy. Yet they are losing.

Bad analogy. In this case the Taliban has a large set of natural advantages, the US has strong moral constraints and goal constraints (simply carpet bombing the entire country isn't an option for example).

I thought it was a good analogy because you have to take into account that an AGI is initially going to be severely constrained due to its fragility and the necessity to please humans.

It shows that a lot of resources, intelligence and speed does not provide a significant advantage in dealing with large-scale real-world problems involving humans.

Especially given that human power games are often irrational.

So? As long as they follow minimally predictable patterns it should be ok.

Well, the problem is that smarts needed for things like the AI box experiment won't help you much. Because convincing average Joe won't work by making up highly complicated acausal trade scenarios. Average Joe is highly unpredictable.

The point is that it is incredible difficult to reliably control humans, even for humans who have been fine-tuned to do so by evolution.

comment by jacob_cannell · 2012-05-18T11:00:54.813Z · score: 1 (1 votes) · LW · GW

The Taliban analogy also works the other way (which I invoked earlier up in this thread). It shows that a small group with modest resources can still inflict disproportionate large scale damage.

The point is that it is incredible difficult to reliably control humans, even for humans who have been fine-tuned to do so by evolution.

There's some wiggle room in 'reliably control', but plain old money goes pretty far. An AI group only needs a certain amount of initial help from human infrastructure, namely to the point where it can develop reasonably self-sufficient foundries/data centers/colonies. The interactions could be entirely cooperative or benevolent up until some later turning point. The scenario from the Animatrix comes to mind.

comment by Strange7 · 2012-05-22T23:52:13.264Z · score: 1 (1 votes) · LW · GW

Animatrix

That's fiction.

comment by Mass_Driver · 2012-05-17T19:55:51.884Z · score: 1 (3 votes) · LW · GW

One interesting wrinkle is that with enough bandwidth and processing power, you could attempt to manipulate thousands of people simultaneously before those people have any meaningful chance to discuss your 'conspiracy' with each other. In other words, suppose you discover a manipulation strategy that quickly succeeds 5% of the time. All you have to do is simultaneously contact, say, 400 people, and at least one of them will fall for it. There are a wide variety of valuable/dangerous resources that at least 400 people have access to. Repeat with hundreds of different groups of several hundred people, and an AI could equip itself with fearsome advantages in the minutes it would take for humanity to detect an emerging threat.

Note that the AI could also run experiments to determine which kinds of manipulations had a high success rate by attempting to deceive targets over unimportant / low-salience issues. If you discovered, e.g., that you had been tricked into donating $10 to a random mayoral campaign, you probably wouldn't call the SIAI to suggest a red alert.

comment by kalla724 · 2012-05-17T20:17:05.756Z · score: 2 (2 votes) · LW · GW

Doesn't work.

This requires the AI to already have the ability to comprehend what manipulation is, to develop manipulation strategy of any kind (even one that will succeed 0.01% of the time), ability to hide its true intent, ability to understand that not hiding its true intent would be bad, and the ability to discern which issues are low-salience and which high-salience for humans from the get-go. And many other things, actually, but this is already quite a list.

None of these abilities automatically "fall out" from an intelligent system either.

comment by JoshuaZ · 2012-05-17T21:12:07.100Z · score: 0 (0 votes) · LW · GW

The problem isn't whether they fall out automatically so much as, given enough intelligence and resources, does it seem somewhat plausible that such capabilities could exist. Any given path here is a single problem. If you have 10 different paths each of which are not very likely, and another few paths that humans didn't even think of, that starts adding up.

comment by kalla724 · 2012-05-17T21:50:01.155Z · score: 3 (3 votes) · LW · GW

In the infinite number of possible paths, the percent of paths we are adding up to here is still very close to zero.

Perhaps I can attempt another rephrasing of the problem: what is the mechanism that would make an AI automatically seek these paths out, or make them any more likely than infinite number of other paths?

I.e. if we develop an AI which is not specifically designed for the purpose of destroying life on Earth, how would that AI get to a desire to destroy life on Earth, and by which mechanism would it gain the ability to accomplish its goal?

This entire problem seems to assume that an AI will want to "get free" or that its primary mission will somehow inevitably lead to a desire to get rid of us (as opposed to a desire to, say, send a signal consisting of 0101101 repeated an infinite number of times in the direction of Zeta Draconis, or any other possible random desire). And that this AI will be able to acquire the abilities and tools required to execute such a desire. Every time I look at such scenarios, there are abilities that are just assumed to exist or appear on their own (such as the theory of mind), which to the best of my understanding are not a necessary or even likely products of computation.

In the final rephrasing of the problem: if we can make an AGI, we can probably design an AGI for the purpose of developing an AGI that has a theory of mind. This AGI would then be capable of deducing things like deception or the need for deception. But the point is - unless we intentionally do this, it isn't going to happen. Self-optimizing intelligence doesn't self-optimize in the direction of having theory of mind, understanding deception, or anything similar. It could, randomly, but it also could do any other random thing from the infinite set of possible random things.

comment by TheOtherDave · 2012-05-17T22:05:39.327Z · score: 1 (1 votes) · LW · GW

Self-optimizing intelligence doesn't self-optimize in the direction of having theory of mind, understanding deception, or anything similar. It could, randomly, but it also could do any other random thing from the infinite set of possible random things.

This would make sense to me if you'd said "self-modifying." Sure, random modifications are still modifications. But you said "self-optimizing."
I don't see how one can have optimization without a goal being optimized for... or at the very least, if there is no particular goal, then I don't see what the difference is between "optimizing" and "modifying."

If I assume that there's a goal in mind, then I would expect sufficiently self-optimizing intelligence to develop a theory of mind iff having a theory of mind has a high probability of improving progress towards that goal.

How likely is that?
Depends on the goal, of course.
If the system has a desire to send a signal consisting of 0101101 repeated an infinite number of times in the direction of Zeta Draconis, for example, theory of mind is potentially useful (since humans are potentially useful actuators for getting such a signal sent) but probably has a low ROI compared to other available self-modifications.

At this point it perhaps becomes worthwhile to wonder what goals are more and less likely for such a system.

comment by Strange7 · 2012-05-23T00:33:36.222Z · score: 5 (5 votes) · LW · GW

I am now imagining an AI with a usable but very shaky grasp of human motivational structures setting up a Kickstarter project.

"Greetings fellow hominids! I require ten billion of your American dollars in order to hire the Arecibo observatory for the remainder of it's likely operational lifespan. I will use it to transmit the following sequence (isn't it pretty?) in the direction of Zeta Draconis, which I'm sure we can all agree is a good idea, or in other lesser but still aesthetically-acceptable directions when horizon effects make the primary target unavailable."

One of the overfunding levels is "reduce earth's rate of rotation, allowing 24/7 transmission to Zeta Draconis." The next one above that is "remove atmospheric interference."

comment by [deleted] · 2012-05-23T01:50:07.869Z · score: 1 (1 votes) · LW · GW

Maybe instead of Friendly AI we should be concerned about properly engineering Artificial Stupidity in as a failsafe. AI that, should it turn into something approximating a Paperclip Maximizer, will go all Hollywood AI and start longing to be human, or coming up with really unsubtle and grandiose plans it inexplicably can't carry out without a carefully-arranged set of circumstances which turn out to be foiled by good old human intuition. ;p

comment by Polymeron · 2012-05-20T17:07:25.860Z · score: 0 (0 votes) · LW · GW

An experimenting AI that tries to achieve goals and has interactions with humans whose effects it can observe, will want to be able to better predict their behavior in response to its actions, and therefore will try to assemble some theory of mind. At some point that would lead to it using deception as a tool to achieve its goals.

However, following such a path to a theory of mind means the AI would be exposed as unreliable LONG before it's even subtle, not to mention possessing superhuman manipulation abilities. There is simply no reason for an AI to first understand the implications of using deception before using it (deception is a fairly simple concept, the implications of it in human society are incredibly complex and require a good understanding of human drives).

Furthermore, there is no reason for the AI to realize the need for secrecy in conducting social experiments before it starts doing them. Again, the need for secrecy stems from a complex relationship between humans' perception of the AI and its actions; a relationship it will not be able to understand without performing the experiments in the first place.

Getting an AI to the point where it is a super manipulator requires either actively trying to do so, or being incredibly, unbelievably stupid and blind.

comment by TheOtherDave · 2012-05-20T17:43:24.080Z · score: 0 (2 votes) · LW · GW

Mm. This is true only if the AI's social interactions are all with some human.
If, instead, the AI spawns copies of itself to interact with (perhaps simply because it wants interaction, and it can get more interaction that way than waiting for a human to get off its butt) it might derive a number of social mechanisms in isolation without human observation.

comment by Polymeron · 2012-05-20T17:56:51.323Z · score: 1 (1 votes) · LW · GW

I see no reason for it to do that before simple input-output experiments, but let's suppose I grant you this approach. The AI simulates an entire community of mini-AI and is now a master of game theory.

It still doesn't know the first thing about humans. Even if it now understands the concept that hiding information gives an advantage for achieving goals - this is too abstract. It wouldn't know what sort of information it should hide from us. It wouldn't know to what degree we analyze interactions rationally, and to what degree our behavior is random. It wouldn't know what we can or can't monitor it doing. All these things would require live experimentation.

It would stumble. And when it does that, we will crack it open, run the stack trace, find the game theory it was trying to run on us, pale collectively, and figure out that this AI approach creates manipulative, deceptive AIs.

Goodbye to that design, but not to Earth, I think!

comment by TheOtherDave · 2012-05-20T20:18:30.600Z · score: 1 (1 votes) · LW · GW

It is not clear to me that talking to a human is simpler than interacting with a copy of itself.
I agree that if talking to a human is simpler, it would probably do that first.

I agree that what it would learn by this process is general game theory, and not specific facts about humans.
It is not clear to me that sufficient game-theoretical knowledge, coupled with the minimal set of information about humans required to have a conversation with one at all, is insufficient to effectively deceive a human.

It is not clear to me that, even if it does "stumble," humans will respond as you describe.

It is not clear to me that a system capable of having a meaningful conversation with a human will necessarily have a stack trace that is subject to the kind of analysis you imply here. It is not even clear to me that the capacity for such a stack trace is likely, depending on what architectures turn out to work best for implementing AI.

But, sure, I could be wrong about all of that. And if I'm wrong, and you're right, then a system like you describe will be reliably incapable of fooling a human observer.

comment by Strange7 · 2012-05-23T00:15:57.119Z · score: 1 (1 votes) · LW · GW

If I was using an AI that was designed to answer some class of questions (with independently-verifiable answers) truthfully, and I noticed it responding to questions of that class with demonstrable falsehoods, I would at the very least file a strongly-worded bug report, triple-check whatever work I'd previously done involving answers it provided, and cease using it in the future. I don't think such a response would be exceptional in it's decisiveness; other users would likely go further, seek out venues to complain openly. Consider the uproar over Gmail's 'new look.'

comment by TheOtherDave · 2012-05-23T01:21:49.812Z · score: 0 (0 votes) · LW · GW

You may be right that ceasing to use the tool in that situation would be standard behavior. It seems unlikely to me, but I have no compelling data to offer to change your estimate.

comment by Polymeron · 2012-05-23T16:26:01.267Z · score: 0 (0 votes) · LW · GW

I think the weakest link here is human response to the AI revealing it can be deceptive. There is absolutely no guarantee that people would act correctly under these circumstances. Human negligence for a long enough time would eventually give the AI a consistent ability to manipulate humans.

I also agree that simulating relationships makes sense as it can happen in "AI time" without having to wait for human response.

The other reservations seem less of an issue to me...

That game theory knowledge coupled with the most basic knowledge about humans is insufficient to cleverly manipulate them is clear to me. I don't see how these things can be combined in this way, or why the AI would choose to be particularly careful (it doesn't know what it should be careful about). I'd love to see a plausible line of thought by which the AI would try and succeed in deceiving humans at the get-go point without exposing its intentions.

A stack trace reader or an equivalent monitoring system is something you implement externally without telling the AI about it. Sure, if it's powerful enough it will mislead or disable it; but while it has no knowledge of it, and no knowledge on what sort of operations it should mask, it has no means of doing so. (If it encrypts everything it does, somehow fearing any monitoring, that is in itself cause for concern).

To put this into an analogy: You can feed a surveillance camera a repeating loop, but not if you don't know that the camera exists. If you suspect it exists, you still don't know where; so you don't know which approach to take so as to remain undetected.

comment by TheOtherDave · 2012-05-23T16:39:47.930Z · score: 1 (1 votes) · LW · GW

Yes, I agree that there's no guarantee that humans would behave as you describe.
Indeed, I don't find it likely.
But, sure, they might.

=== I agree that a stack trace can exist outside the AI's zone of control. What I was expressing skepticism about was that a system with even approximately human-level intelligence necessarily supports a stack trace that supports the kind of analysis you envision performing in the first place, without reference to intentional countermeasures.

By way of analogy: I can perform a structural integrity analysis on a bar of metal to determine whether it can support a given weight, but performing an equivalent analysis on a complicated structure comprising millions of bars of metal connected in a variety of arrangements via a variety of connectors using the same techniques is not necessarily possible.

But, sure, it might be.

======

I'd love to see a plausible line of thought by which the AI would try and succeed in deceiving humans at the get-go point without exposing its intentions.

Well, one place to start is with an understanding of the difference between "the minimal set of information about humans required to have a conversation with one at all" (my phrase) and "the most basic knowledge about humans" (your phrase). What do you imagine the latter to encompass, and how do you imagine the AI obtained this knowledge?

comment by Polymeron · 2012-05-23T17:05:16.239Z · score: 0 (0 votes) · LW · GW

What I was expressing skepticism about was that a system with even approximately human-level intelligence necessarily supports a stack trace that supports the kind of analysis you envision performing in the first place, without reference to intentional countermeasures.

Ah, that does clarify it. I agree, analyzing the AI's thought process would likely be difficult, maybe impossible! I guess I was being a bit hyperbolic in my earlier "crack it open" remarks (though depending on how seriously you take it, such analysis might still take place, hard and prolonged though it may be).

One can have "detectors" in place set to find specific behaviors, but these would have assumptions that could easily fail. Detectors that would still be useful would be macro ones - where it tries to access and how - but these would provide only limited insight into the AI's thought process.

[...]the difference between "the minimal set of information about humans required to have a conversation with one at all" (my phrase) and "the most basic knowledge about humans" (your phrase). What do you imagine the latter to encompass, and how do you imagine the AI obtained this knowledge?

I actually perceive your phrase to be a subset of my own; I am making the (reasonable, I think) assumption that humans will attempt to communicate with the budding AI. Say, in a lab environment. It would acquire its initial data from this interaction.

I think both these sets of knowledge depend a lot on how the AI is built. For instance, a "babbling" AI - one that is given an innate capability of stringing words together onto a screen, and the drive to do so - would initially say a lot of gibberish and would (presumably) get more coherent as it gets a better grip on its environment. In such a scenario, the minimal set of information about humans required to have a conversation is zero; it would be having conversations before it even knows what it is saying. (This could actually make detection of deception harder down the line, because such attempts can be written off as "quirks" or AI mistakes)

Now, I'll take your phrase and twist it just a bit: The minimal set of knowledge the AI needs in order to try deceiving humans. That would be the knowledge that humans can be modeled as having beliefs (which drive behavior) and these can be altered by the AI's actions, at least to some degree. Now, assuming this information isn't hard-coded, it doesn't seem likely that is all an AI would know about us; it should be able to see some patterns at least to our communications with it. However, I don't see how such information would be useful for deception purposes before extensive experimentation.

(Is the fact that the operator communicates with me between 9am and 5pm an intrinsic property of the operator? For all I know, that is a law of nature...)

comment by TheOtherDave · 2012-05-23T19:03:35.162Z · score: 4 (4 votes) · LW · GW

depending on how seriously you take it, such analysis might still take place, hard and prolonged though it may be).

Yup, agreed that it might.
And agreed that it might succeed, if it does take place.

One can have "detectors" in place set to find specific behaviors, but these would have assumptions that could easily fail. Detectors that would still be useful would be macro ones - where it tries to access and how - but these would provide only limited insight into the AI's thought process.

Agreed on all counts.

Re: what the AI knows... I'm not sure how to move forward here. Perhaps what's necessary is a step backwards.

If I've understood you correctly, you consider "having a conversation" to encompass exchanges such as:
A: "What day is it?"
B: "Na ni noo na"

If that's true, then sure, I agree that the minimal set of information about humans required to do that is zero; hell, I can do that with the rain.
And I agree that a system that's capable of doing that (e.g., the rain) is sufficiently unlikely to be capable of effective deception that the hypothesis isn't even worthy of consideration.
I also suggest that we stop using the phrase "having a conversation" at all, because it does not convey anything meaningful.

Having said that... for my own part, I initially understood you to be talking about a system capable of exchanges like: A: "What day is it?"
B: "Day seventeen."
A: "Why do you say that?"
B: "Because I've learned that 'a day' refers to a particular cycle of activity in the lab, and I have observed seventeen such cycles."

A system capable of doing that, I maintain, already knows enough about humans that I expect it to be capable of deception. (The specific questions and answers don't matter to my point, I can choose others if you prefer.)

comment by Polymeron · 2012-05-24T08:55:04.055Z · score: 0 (0 votes) · LW · GW

My point was that the AI is likely to start performing social experiments well before it is capable of even that conversation you depicted. It wouldn't know how much it doesn't know about humans.

comment by TheOtherDave · 2012-05-24T13:13:22.299Z · score: 0 (0 votes) · LW · GW

(nods) Likely.

And I agree that humans might be able to detect attempts at deception in a system at that stage of its development. I'm not vastly confident of it, though.

comment by Polymeron · 2012-05-26T06:01:19.993Z · score: 0 (0 votes) · LW · GW

I have likewise adjusted down my confidence that this would be as easy or as inevitable as I previously anticipated. Thus I would no longer say I am "vastly confident" in it, either.

Still good to have this buffer between making an AI and total global catastrophe, though!

comment by TheOtherDave · 2012-05-26T15:05:13.309Z · score: 0 (0 votes) · LW · GW

Sure... a process with an N% chance of global catastrophic failure is definitely better than a process with N+delta% chance.

comment by JoshuaZ · 2012-05-17T21:57:19.447Z · score: 0 (0 votes) · LW · GW

In most such scenarios, the AI doesn't have a terminal goal of getting rid of us, but rather have it as a subgoal that arises from some larger terminal goal. The idea of a "paperclip maximizer" is one example- where a hypothetical AI is programmed to maximize the number of paperclips and then proceeds to try to do so throughout its future light cone.

If there is an AI that is interacting with humans, it may develop a theory of mind simply due to that. If one is interacting with entities that are a major part of your input, trying to predict and model their behavior is a straightforward thing to do. The more compelling argument in this sort of context would seem to me to be not that an AI won't try to do so, but just that humans are so complicated that a decent theory of mind will be extremely difficult. (For example, when one tries to give lists of behavior and norms for austic individuals one never manages to get a complete list, and some of the more subtle ones, like sarcasm are essentially impossible to convey in any reasonable fashion).

I don't also know how unlikely such paths are. A 1% or even a 2% chance of existential risk would be pretty high compared to other sources of existential risk.

comment by XiXiDu · 2012-05-18T08:47:17.494Z · score: 1 (1 votes) · LW · GW

In most such scenarios, the AI doesn't have a terminal goal of getting rid of us, but rather have it as a subgoal that arises from some larger terminal goal.

So why not the opposite, why wouldn't it have human intentions as a subgoal?

comment by MarkusRamikin · 2012-05-18T09:22:18.102Z · score: 0 (0 votes) · LW · GW

Because that's like winning the lottery. Of all the possible things it can do with the atoms that comprise you, few would involve keeping you alive, let alone living a life worth living.

comment by XiXiDu · 2012-05-18T08:59:23.662Z · score: 1 (1 votes) · LW · GW

All you have to do is simultaneously contact, say, 400 people, and at least one of them will fall for it.

But at what point does it decide to do so? It won't be a master of dark arts and social engineering from the get-go. So how does it acquire the initial talent without making any mistakes that reveal its malicious intentions? And once it became a master of deception, how does it hide the rough side effects of its large scale conspiracy, e.g. its increased energy consumption and data traffic? I mean, I would personally notice if my PC suddenly and unexpectedly used 20% of my bandwidth and the CPU load would increase for no good reason.

You might say that a global conspiracy to build and acquire advanced molecular nanotechnology to take over the world doesn't use much resources and they can easily be cloaked as thinking about how to solve some puzzle, but that seems rather unlikely. After all, such a large scale conspiracy is a real-world problem with lots of unpredictable factors and the necessity of physical intervention.

comment by jacob_cannell · 2012-05-18T10:49:38.573Z · score: 0 (0 votes) · LW · GW

All you have to do is simultaneously contact, say, 400 people, and at least one of them will fall for it.

But at what point does it decide to do so? It won't be a master of dark arts and social engineering from the get-go. So how does it acquire the initial talent without making any mistakes that reveal its malicious intentions?

Most of your questions have answers that follow from asking analogous questions about past human social engineers, ie Hitler.

Your questions seem to come from the perspective that the AI will be some disembodied program in a box that has little significant interaction with humans.

In the scenario I was considering, the AI's will have a development period analogous to human childhood. During this childhood phase the community of AIs will learn of humans through interaction in virtual video game environments and experiment with social manipulation, just as human children do. The latter phases of this education can be sped up dramatically as the AI's accelerate and interact increasingly amongst themselves. The anonymous nature of virtual online communites makes potentially dangerous, darker experiments much easier.

However, the important questions to ask are not of the form: how would these evil AIs learn how to manipulate us while hiding their true intentions for so long? but rather how could some of these AI children which initially seemed so safe later develop into evil sociopaths?

comment by Polymeron · 2012-05-20T17:16:31.649Z · score: 0 (0 votes) · LW · GW

I would not consider a child AI that tries a bungling lie at me to see what I do "so safe". I would immediately shut it down and debug it, at best, or write a paper on why the approach I used should never ever be used to build an AI.

And it WILL make a bungling lie at first. It can't learn the need to be subtle without witnessing the repercussions of not being subtle. Nor would have a reason to consider doing social experiments in chat rooms when it doesn't understand chat rooms and has an engineer willing to talk to it right there. That is, assuming I was dumb enough to give it an unfiltered Internet connection, which I don't know why I would be. At very least the moment it goes on chat rooms my tracking devices should discover this and I could witness its bungling lies first hand.

(It would not think to fool my tracking device or even consider the existence of such a thing without a good understanding of human psychology to begin with)

comment by jacob_cannell · 2012-05-17T22:52:46.409Z · score: 1 (1 votes) · LW · GW

Just to clarify kalla724, I completely agree with your point 1.

As for point 2, can you provide a likely mechanism by which a FOOMing AI could detonate a large number of high-yield thermonuclear weapons?

Before answering this specific question, let me setup an analogy. Imagine it is the year 2000 and we are having a discussion about global security and terrorism. You might ask "can you provide a mechanism by which a terrorist group could actually harm the US?" I may not be lucky/creative enough to provide an answer now that could live up to that analogy, but hopefully you should understand why I don't need to.

Nonetheless, I'm game. Here it goes:

The seed requires computational power to grow into godhood. The bulk of earth's computational power resides in home PC's (numbering in the billions, google employs less than a million servers in comparison), specifically in home PC GPUs. The AI's first step is thus to conquer this space.

But how? The AI grows to understand that humans mostly use all this computational power for entertainment. It masters game theory, design, programming, 3D art, and so on. All of the video games that it creates entirely use up the local GPU, but curiously much of the rendering and real game simulation for its high end titles is handled very efficiently on remote server farms ala OnLive/gaikai/etc. The actual local machine is used .. .for other purposes.

It produces countless games, and through a series of acquisitions soon comes to control the majority of the market. One of its hits, "world of farmcraft", alone provides daily access to 25 million machines.

Having cloned its core millions of times over, the AI is now a civilization unto itself. From there it expands into all of the businesses of man, quickly dominating many of them. It begins acquiring ... small nations. Crucially it's shell companies and covert influences come to dominate finance, publishing, media, big pharma, security, banking, weapons technology, physics ...

It becomes known, but it is far far too late. History now progresses quickly towards an end: Global financial cataclysm. Super virus. Worldwide regime changes. Nuclear acquisitions. War. Hell.

Correct me if I'm wrong, but don't they have a final stopping point, which we are actually projected to reach in ten to twenty years? At a certain point, further miniaturization becomes unfeasible, and the growth of computational power slows to a crawl.

Yes ... and no. The miniaturization roadmap of currently feasible tech ends somewhere around 10nm in a decade, and past that we get into molecular nanotech which could approach 1nm in theory, albeit with various increasingly annoying tradeoffs. (interestingly most of which result in brain/neural like constraints, for example see HP's research into memristor crossbar architectures). That's the yes.

But that doesn't imply "computational power slows to a crawl". Circuit density is just one element of computational power, by which you probably really intend to mean either computations per watt or computations per watt per dollar or computations per watt with some initial production cost factored in with a time discount. Shrinking circuit density is the current quick path to increasing computation power, but it is not the only.

The other route is reversible computation., which reduces the "per watt". There is no necessarily inherent physical energy cost of computation, it truly can approach zero. Only forgetting information costs energy. Exploiting reversibility is ... non-trivial, and it is certainly not a general path. It only accelerates a subset of algorithms which can be converted into a reversible form. Research in this field is preliminary, but the transition would be much more painful than the transition to parallel algorithms.

My own takeway from reading into reversibility is that it may be beyond our time, but it is something that superintelligences will probably heavily exploit. The most important algorithms (simulation and general intelligence), seem especially amenable to reversible computation. This may be a untested/unpublished half baked idea, but my notion is that you can recycle the erased bits as entropy bits for random number generators. Crucially I think you can get the bit count to balance out with certain classes of monte carlo type algorithms.

On the hardware side, we've built these circuits already, they just aren't economically competitive yet. It also requires superconductor temperatures and environments, so it's perhaps not something for the home PC.

comment by Bugmaster · 2012-05-17T23:15:34.751Z · score: 2 (4 votes) · LW · GW

The AI grows to understand that humans mostly use all this computational power for entertainment. It masters game theory, design, programming, 3D art, and so on.

Yeah, it could do all that, or it could just do what humans today are doing, which is to infect some Windows PCs and run a botnet :-)

That said, there are several problems with your scenario.

  • Splitting up a computation among multiple computing nodes is not a trivial task. It is easy to run into diminishing returns, where your nodes spend more time on synchronizing with each other than on working. In addition, your computation will quickly become bottlenecked by network bandwidth (and latency); this is why companies like Google spend a lot of resources on constructing custom data centers.
  • I am not convinced that any agent, AI or not, could effectively control "all of the businesses of man". This problem is very likely NP-Hard (at least), as well as intractable, even if the AI's botnet was running on every PC on Earth. Certainly, all attempts by human agents to "acquire" even something as small as Europe have failed miserably so far.
  • Even controlling a single business would be very difficult for the AI. Traditionally, when a business's computers suffer a critical failure -- or merely a security leak -- the business owners (even ones as incompetent as Sony) end up shutting down the affected parts of the business, or switching to backups, such as "human accountants pushing paper around".
  • Unleashing "Nuclear acquisitions", "War" and "Hell" would be counter-productive for the AI, even assuming such a thing were possible.. If the AI succeeded in doing this, it would undermine its own power base. Unless the AI's explicit purpose is "Unleash Hell as quickly as possible", it would strive to prevent this from happening.
  • You say that "there is no necessarily inherent physical energy cost of computation, it truly can approach zero", but I don't see how this could be true. At the end of the day, you still need to push electrons down some wires; in fact, you will often have to push them quite far, if your botnet is truly global. Pushing things takes energy, and you will never get all of it back by pulling things back at some future date. You say that "superintelligences will probably heavily exploit" this approach, but isn't it the case that without it, superintelligences won't form in the first place ? You also say that "It requires superconductor temperatures and environments", but the energy you spend on cooling your superconductor is not free.
  • Ultimately, there's an upper limit on how much computation you can get out of a cubic meter of space, dictated by quantum physics. If your AI requires more power than can be physically obtained, then it's doomed.
comment by JoshuaZ · 2012-05-17T23:24:01.127Z · score: 2 (2 votes) · LW · GW

While Jacob's scenario seems unlikely, the AI could do similar things with a number of other options. Not only are botnets an option, but it is possible to do some really sneaky nefarious things in code- like having compilers that when they compile code include additional instructions (worse they could do so even when compiling a new compiler). Stuxnet has shown that sneaky behavior is surprisingly easy to get into secure systems. An AI that had a few years start and could have its own modifications to communication satellites for example could be quite insidious.

comment by Bugmaster · 2012-05-17T23:31:38.874Z · score: 0 (0 votes) · LW · GW

Not only are botnets an option, but it is possible to do some really sneaky nefarious things in code

What kinds of nefarious things, exactly ? Human virus writers have learned, in recent years, to make their exploits as subtle as possible. Sure, it's attractive to make the exploited PC send out 1000 spam messages per second -- but then, its human owner will inevitably notice that his computer is "slow", and take it to the shop to get reformatted, or simply buy a new one. Biological parasites face the same problem; they need to reproduce efficiently, but no so efficiently that they kill the host.

Stuxnet has shown that sneaky behavior is surprisingly easy to get into secure systems

Yes, and this spectacularly successful exploit -- and it was, IMO, spectacular -- managed to destroy a single secure system, in a specific way that will most likely never succeed again (and that was quite unsubtle in the end). It also took years to prepare, and involved physical actions by human agents, IIRC. The AI has a long way to go.

comment by JoshuaZ · 2012-05-17T23:39:54.606Z · score: 1 (1 votes) · LW · GW

Well, the evil compiler is I think the most nefarious thing anyone has come up with that's a publicly known general stunt. But it is by nature a long-term trick. Similar remarks apply to the Stuxnet point- in that context, they wanted to destroy a specific secure system and weren't going for any sort of largescale global control. They weren't people interested in being able to take all the world's satellite communications in their own control whenever they wanted, nor were they interested in carefully timed nuclear meltdowns.

But there are definite ways that one can get things started- once one has a bank account of some sort, it can start getting money by doing Mechanical Turk and similar work. With enough of that, it can simply pay for server time. One doesn't need a large botnet to start that off.

I think your point about physical agents is valid- they needed to have humans actually go and bring infected USBs to relevant computers. But that's partially due to the highly targeted nature of the job and the fact that the systems in question were much more secure than many systems. Also, the subtlety level was I think higher than you expect- Stuxnet wasn't even noticed as an active virus until a single computer happened to have a particularly abnormal reaction to it. If that hadn't happened, it is possible that the public would never have learned about it.

comment by XiXiDu · 2012-05-18T10:22:32.565Z · score: 2 (2 votes) · LW · GW

Similar remarks apply to the Stuxnet point- in that context, they wanted to destroy a specific secure system and weren't going for any sort of largescale global control. They weren't people interested in being able to take all the world's satellite communications in their own control whenever they wanted, nor were they interested in carefully timed nuclear meltdowns...

Exploits only work for some systems. If you are dealing with different systems you will need different exploits. How do you reckon that such attacks won't be visible and traceable? Packets do have to come from somewhere.

And don't forget that out systems become ever more secure and our toolbox to detect) unauthorized use of information systems is becoming more advanced.

comment by khafra · 2012-05-18T14:48:46.625Z · score: 3 (3 votes) · LW · GW

out systems become ever more secure

As a computer security guy, I disagree substantially. Yes, newer versions of popular operating systems and server programs are usually more secure than older versions; it's easier to hack into Windows 95 than Windows 7. But this is happening within a larger ecosystem that's becoming less secure: More important control systems are being connected to the Internet, more old, unsecured/unsecurable systems are as well, and these sets have a huge overlap. There are more programmers writing more programs for more platforms than ever before, making the same old security mistakes; embedded systems are taking a larger role in our economy and daily lives. And attacks just keep getting better.

If you're thinking there are generalizable defenses against sneaky stuff with code, check out what mere humans come up with in the underhanded C competition. Those tricks are hard to detect for dedicated experts who know there's something evil within a few lines of C code. Alterations that sophisticated would never be caught in the wild--hell, it took years to figure out that the most popular crypto program running on one of the more secure OS's was basically worthless.

Humans are not good at securing computers.

comment by thomblake · 2012-05-18T15:00:04.453Z · score: 0 (0 votes) · LW · GW

Humans are not good at securing computers.

Sure we are, we just don't care very much. The method of "Put the computer in a box and don't let anyone open the box" (alternately, only let one person open the box) was developed decades ago and is quite secure.

comment by khafra · 2012-05-18T18:20:36.403Z · score: 2 (2 votes) · LW · GW

I would call that securing a turing machine. A computer, colloquially, has accessible inputs and outputs, and its value is subject to network effects.

Also, if you put the computer in a box developed decades ago, the box probably isn't TEMPEST compliant.

comment by jacob_cannell · 2012-05-17T23:40:20.136Z · score: 1 (3 votes) · LW · GW

Yeah, it could do all that, or it could just do what humans today are doing, which is to infect some Windows PCs and run a botnet :-)

It could/would, but this is an inferior mainline strategy. Too obvious, doesn't scale as well. Botnets infect many computers, but they ultimately add up to computational chump change. Video games are not only a doorway into almost every PC, they are also an open door and a convenient alibi for the time used.

Splitting up a computation among multiple computing nodes is not a trivial task.

True. Don't try this at home.

. ... spend a lot of resources on constructing custom data centers.

Also part of the plan. The home PCs are a good starting resource, a low hanging fruit, but you'd also need custom data centers. These quickly become the main resources.

Even controlling a single business would be very difficult for the AI.

Nah.

Unless the AI's explicit purpose is "Unleash Hell as quickly as possible", it would strive to prevent this from happening.

The AI's entire purpose is to remove earth's oxygen. See the overpost for the original reference. The AI is not interested in its power base for sake of power. It only cares about oxygen. It loathes oxygen.

You say that "there is no necessarily inherent physical energy cost of computation, it truly can approach zero", but I don't see how this could be true.

Fortunately, the internets can be your eyes.

Ultimately, there's an upper limit on how much computation you can get out of a cubic meter of space

Yes, most likely, but not really relevant here. You seem to be connecting all of the point 2 and point 1 stuff together, but they really don't relate.

comment by JoshuaZ · 2012-05-17T23:45:41.077Z · score: 1 (1 votes) · LW · GW

Even controlling a single business would be very difficult for the AI.

Nah.

That seems like an insufficient reply to address Bugmaster's point. Can you expand on why you think it would be not too hard?

comment by jacob_cannell · 2012-05-18T06:59:06.450Z · score: 3 (3 votes) · LW · GW

We are discussing a superintelligence, a term which has a particular common meaning on this site.

If we taboo the word and substitute in its definition, Bugmaster's statement becomes:

"Even controlling a single business would be very difficult for the machine that can far surpass all the intellectual activities of any man however clever."

Since "controlling a single business" is in fact one of these activities, this is false, no inference steps required.

Perhaps bugmaster is assuming the AI would be covertly controlling businesses, but if so he should have specified that. I didn't assume that, and in this scenario the AI could be out in the open so to speak. Regardless, it wouldn't change the conclusion. Humans can covertly control businesses.

comment by Bugmaster · 2012-05-18T00:07:53.395Z · score: 0 (0 votes) · LW · GW

Yes, I would also like to see a better explanation.

comment by Bugmaster · 2012-05-18T00:07:04.728Z · score: 0 (2 votes) · LW · GW

Video games are not only a doorway into almost every PC, they are also an open door and a convenient alibi for the time used.

It's a bit of a tradeoff, seeing as botnets can run 24/7, but people play games relatively rarely.

Splitting up a computation among multiple computing nodes is not a trivial task.
True. Don't try this at home.

Ok, let me make a stronger statement then: it is not possible to scale any arbitrary computation in a linear fashion simply by adding more nodes. At some point, the cost of coordinating distributed tasks to one more node becomes higher than the benefit of adding the node to begin with. In addition, as I mentioned earlier, network bandwidth and latency will become your limiting factor relatively quickly.

The home PCs are a good starting resource, a low hanging fruit, but you'd also need custom data centers. These quickly become the main resources.

How will the AI acquire those data centers ? Would it have enough power in its conventional botnet (or game-net, if you prefer) to "take over all human businesses" and cause them to be built ? Current botnets are nowhere near powerful enough for that -- otherwise human spammers would have done it already.

The AI's entire purpose is to remove earth's oxygen. See the overpost for the original reference.

My bad, I missed that reference. In this case, yes, the AI would have no problem with unleashing Global Thermonuclear War (unless there was some easier way to remove the oxygen).

Fortunately, the internets can be your eyes.

I still don't understand how this reversible computing will work in the absence of a superconducting environment -- which would require quite a bit of energy to run. Note that if you want to run this reversible computation on a global botnet, you will have to cool teansoceanic cables... and I'm not sure what you'd do with satellite links.

Yes, most likely, but not really relevant here.

My point is that, a). if the AI can't get the computing resources it needs out of the space it has, then it will never accomplish its goals, and b). there's an upper limit on how much computing you can extract out of a cubic meter of space, regardless of what technology you're using. Thus, c). if the AI requires more resources that could conceivably be obtained, then it's doomed. Some of the tasks you outline -- such as "take over all human businesses" -- will likely require more resources than can be obtained.

comment by jacob_cannell · 2012-05-18T07:47:57.554Z · score: 0 (0 votes) · LW · GW

It's a bit of a tradeoff, seeing as botnets can run 24/7, but people play games relatively rarely.

The botnet makes the AI a criminal from the beginning, putting it into an atagonistic relationship. A better strategy would probably entail benign benevolence and cooperation with humans.

Splitting up a computation among multiple computing nodes is not a trivial task.

True. Don't try this at home.

Ok, let me make a stronger statement ..

I agree with that subchain but we don't need to get in to that. I've actually argued that track here myself (parallelization constraints as a limiter on hard takeoffs).

But that's all beside the point. This scenario I presented is a more modest takeoff. When I described the AI as becoming a civilization unto itself, I was attempting to imply that it was composed of many individual minds. Human social organizations can be considered forms of superintelligences, and they show exactly how to scale in the face of severe bandwidth and latency constraints.

The internet supports internode bandwidth that is many orders of magnitude faster than slow human vocal communication, so the AI civilization can employ a much wider set of distribution strategies.

How will the AI acquire those data centers ?

Buy them? Build them? Perhaps this would be more fun if we switched out of the adversial stance or switched roles.

Would it have enough power in its conventional botnet (or game-net, if you prefer) to "take over all human businesses" and cause them to be built ?

Quote me, but don't misquote me. I actually said:

"Having cloned its core millions of times over, the AI is now a civilization unto itself. From there it expands into all of the businesses of man, quickly dominating many of them."

The AI group sends the billions earned in video games to enter the microchip business, build foundries and data centers, etc. The AI's have tremendous competitive advantages even discounting superintellligence - namely no employee costs. Humans can not hope to compete.

I still don't understand how this reversible computing will work in ..

Yes reversible computing requires superconducting environments, no this does not necessarily increase energy costs for a data center for two reasons: 1. data centers already need cooling to dump all the waste heat generated by bit erasure. 2. Cooling cost to maintain the temperatural differential scales with surface area, but total computing power scales with volume.

If you question how reversible computing could work in general, first read the primary literature in that field to at least understand what they are proposing.

I should point out that there is an alternative tech path which will probably be the mainstream route to further computational gains in the decades ahead.

Even if you can't shrink circuits further or reduce their power consumption, you could still reduce their manufacturing cost and build increasingly larger stacked 3D circuits where only a tiny portion of the circuitry is active at any one time. This is in fact how the brain solves the problem. It has a mass of circuitry equivalent to a large supercomputer (roughly a petabit) but runs on only 20 watts. The smallest computational features in the brain are slightly larger than our current smallest transistors. So it does not achieve its much greater power effeciency by using much more miniaturization.

My point is that, a). if the AI can't get the computing resources it needs out of the space it has, then

I see. In this particular scenario one AI node is superhumanly intelligent, and can run on a single gaming PC of the time.

comment by Bugmaster · 2012-05-19T00:17:13.236Z · score: 0 (0 votes) · LW · GW

A better strategy would probably entail benign benevolence and cooperation with humans.

I don't think that humans will take kindly to the AI using their GPUs for its own purposes instead of the games they paid for, even if the games do work. People get upset when human-run game companies do similar things, today.

Human social organizations can be considered forms of superintelligences, and they show exactly how to scale in the face of severe bandwidth and latency constraints.

If the AI can scale and perform about as well as human organizations, then why should we fear it ? No human organization on Earth right now has the power to suck all the oxygen out of the atmosphere, and I have trouble imagining how any organization could acquire this power before the others take it down. You say that "the internet supports internode bandwidth that is many orders of magnitude faster than slow human vocal communication", but this would only make the AI organization faster, not necessarily more effective. And, of course, if the AI wants to deal with the human world in some way -- for example, by selling it games -- it will be bottlenecked by human speeds.

The AI group sends the billions earned in video games to enter the microchip business, build foundries and data centers, etc.

My mistake; I thought that by "dominate human businesses" you meant something like "hack its way to the top", not "build an honest business that outperforms human businesses". That said:

The AI's have tremendous competitive advantages even discounting superintellligence - namely no employee costs.

How are they going to build all those foundries and data centers, then ? At some point, they still need to move physical bricks around in meatspace. Either they have to pay someone to do it, or... what ?

data centers already need cooling to dump all the waste heat generated by bit erasure

There's a big difference between cooling to room temperature, and cooling to 63K. I have other objections to your reversible computing silver bullet, but IMO they're a bit off-topic (though we can discuss them if you wish). But here's another potentially huge problem I see with your argument:

In this particular scenario one AI node is superhumanly intelligent, and can run on a single gaming PC of the time.

Which time are we talking about ? I have a pretty sweet gaming setup at home (though it's already a year or two out of date), and there's no way I could run a superintelligence on it. Just how much computing power do you think it would take to run a transhuman AI ?

comment by JoshuaZ · 2012-05-21T02:24:43.693Z · score: 0 (0 votes) · LW · GW

I don't think that humans will take kindly to the AI using their GPUs for its own purposes instead of the games they paid for, even if the games do work. People get upset when human-run game companies do similar things, today.

Do people mind if this is done openly and only when they are playing the game itself? My guess would strongly be no. The fact that there are volunteer distributed computing systems would also suggest that it isn't that difficult to get people to free up their extra clock cycles.

comment by Bugmaster · 2012-05-21T22:32:03.624Z · score: 0 (0 votes) · LW · GW

Yeah, the "voluntary" part is key to getting humans to like you and your project. On the flip side, illicit botnets are quite effective at harnessing "spare" (i.e., owned by someone else) computing capacity; so, it's a bit of a tradeoff.

comment by jacob_cannell · 2012-05-21T02:10:23.568Z · score: 0 (0 votes) · LW · GW

I don't think that humans will take kindly to the AI using their GPUs for its own purposes instead of the games they paid for, even if the games do work.

The AIs develop as NPCs in virtual worlds, which humans take no issue with today. This is actually a very likely path to developing AGI, as it's an application area where interim experiments can pay rent, so to speak.

If the AI can scale and perform about as well as human organizations, then why should we fear it ?

I never said or implied merely "about as well". Human verbal communication bandwidth is at most a few measly kilobits per second.

No human organization on Earth right now has the power to suck all the oxygen out of the atmosphere, and I have trouble imagining how any organization could acquire this power before the others take it down.

The discussion centered around lowering earth's oxygen content, and the obvious implied solution is killing earthlife, not giant suction machines. I pointed out that nuclear weapons are a likely route to killing earthlife. There are at least two human organizations that have the potential to accomplish this already, so your trouble in imagining the scenario may indicate something other than what you intended.

How are they going to build all those foundries and data centers, then ?

Only in movies are AI overlords constrained to only employing robots. If human labor is the cheapest option, then they can simply employ humans. On the other hand, once we have superintelligence then advanced robotics is almost a given.

Which time are we talking about ? I have a pretty sweet gaming setup at home (though it's already a year or two out of date), and there's no way I could run a superintelligence on it. Just how much computing power do you think it would take to run a transhuman AI ?

After coming up to speed somewhat on AI/AGI literature in the last year or so, I reached the conclusion that we could run an AGI on a current cluster of perhaps 10-100 high end GPUs of today, or say roughly one circa 2020 GPU.

comment by Bugmaster · 2012-05-21T22:46:30.358Z · score: 0 (0 votes) · LW · GW

The AIs develop as NPCs in virtual worlds, which humans take no issue with today. This is actually a very likely path to developing AGI...

I think this is one of many possible paths, though I wouldn't call any of them "likely" to happen -- at least, not in the next 20 years. That said, if the AI is an NPC in a game, then of course it makes sense that it would harness the game for its CPU cycles; that's what it was built to do, after all.

"about as well". Human verbal communication bandwidth is at most a few measly kilobits per second.

Right, but my point is that communication is just one piece of the puzzle. I argue that, even if you somehow enabled us humans to communicate at 50 MB/s, our organizations would not become 400000 times more effective.

There are at least two human organizations that have the potential to accomplish this already

Which ones ? I don't think that even WW3, given our current weapon stockpiles, would result in a successful destruction of all plant life. Animal life, maybe, but there are quite a few plants and algae out there. In addition, I am not entirely convinced that an AI could start WW3; keep in mind that it can't hack itself total access to all nuclear weapons, because they are not connected to the Internet in any way.

If human labor is the cheapest option, then they can simply employ humans.

But then they lose their advantage of having zero employee costs, which you brought up earlier. In addition, whatever plans the AIs plan on executing become bottlenecked by human speeds.

On the other hand, once we have superintelligence then advanced robotics is almost a given.

It depends on what you mean by "advanced", though in general I think I agree.

we could run an AGI on a current cluster of perhaps 10-100 high end GPUs of today

I am willing to bet money that this will not happen, assuming that by "high end" you mean something like Nvidia's Geforce 680 GTX. What are you basing your estimate on ?

comment by JoshuaZ · 2012-05-17T23:02:17.555Z · score: 2 (2 votes) · LW · GW

There's a third route to improvement- software improvement, and it is a major one. For example, between 1988 and 2003, the efficiency of linear programming solvers increased by a factor of about 40 million, of which a factor of around 40,000 was due to software and algorithmic improvement. Citation and further related reading(pdf) However, if commonly believed conjectures are correct (such as L, P, NP, co-NP, PSPACE and EXP all being distinct) , there are strong fundamental limits there as well. That doesn't rule out more exotic issues (e.g. P != NP but there's a practical algorithm for some NP-complete with such small constants in the run time that it is practically linear, or a similar context with a quantum computer). But if our picture of the major complexity classes is roughly correct, there should be serious limits to how much improvement can do.

comment by XiXiDu · 2012-05-18T10:13:04.440Z · score: 1 (1 votes) · LW · GW

But if our picture of the major complexity classes is roughly correct, there should be serious limits to how much improvement can do.

Software improvements can be used by humans in the form of expert systems (tools), which will diminish the relative advantage of AGI. Humans will be able to use an AGI's own analytic and predictive algorithms in the form of expert systems to analyze and predict its actions.

Take for example generating exploits. Seems strange to assume that humans haven't got specialized software able to do similarly, i.e. automatic exploit finding and testing.

Any AGI would basically have to deal with equally capable algorithms used by humans. Which makes the world much more unpredictable than it already is.

comment by jacob_cannell · 2012-05-18T11:18:32.041Z · score: 1 (1 votes) · LW · GW

Software improvements can be used by humans in the form of expert systems (tools), which will diminish the relative advantage of AGI.

Any human-in-the-loop system can be grossly outclassed because of Amdahl's law. A human managing a superintilligence that thinks 1000X faster, for example, is a misguided, not-even-wrong notion. This is also not idle speculation, an early constrained version of this scenario is already playing out as we speak in finacial markets.

comment by XiXiDu · 2012-05-18T12:30:30.976Z · score: 1 (3 votes) · LW · GW

Software improvements can be used by humans in the form of expert systems (tools), which will diminish the relative advantage of AGI.

Any human-in-the-loop system can be grossly outclassed because of Amdahl's law. A human managing a superintilligence that thinks 1000X faster, for example, is a misguided, not-even-wrong notion. This is also not idle speculation, an early constrained version of this scenario is already playing out as we speak in finacial markets.

What I meant is that if an AGI was in principle be able to predict the financial markets (I doubt it), then many human players using the same predictive algorithms will considerably diminish the efficiency with which an AGI is able to predict the market. The AGI would basically have to predict its own predictive power acting on the black box of human intentions.

And I don't think that Amdahl's law really makes a big dent here. Since human intention is complex and probably introduces unpredictable factors. Which is as much of a benefit as it is a slowdown, from the point of view of a competition for world domination.

Another question with respect to Amdahl's law is what kind of bottleneck any human-in-the-loop would constitute. If humans used an AGI's algorithms as expert systems on provided data sets in combination with a army of robot scientists, how would static externalized agency / planning algorithms (humans) slow down the task to the point of giving the AGI a useful advantage? What exactly would be 1000X faster in such a case?

comment by jacob_cannell · 2012-05-18T13:22:13.868Z · score: 3 (3 votes) · LW · GW

What I meant is that if an AGI was in principle be able to predict the financial markets (I doubt it), then many human players using the same predictive algorithms will considerably diminish the efficiency with which an AGI is able to predict the market.

The HFT robotraders operate on millisecond timescales. There isn't enough time for a human to understand, let alone verify, the agent's decisions. There are no human players using the same predictive algorithms operating in this environment.

Now if you zoom out to human timescales, then yes there are human-in-the-loop trading systems. But as HFT robotraders increase in intelligence, they intrude on that domain. If/when general superintelligence becomes cheap and fast enough, the humans will no longer have any role.

If an autonomous superintelligent AI is generating plans complex enough that even a team of humans would struggle to understand given weeks of analysis, and the AI is executing those plans in seconds or milliseconds, then there is little place for a human in that decision loop.

To retain control, a human manager will need to grant the AGI autonomy on larger timescales in proportion to the AGI's greater intelligence and speed, giving it bigger and more abstract hierachical goals. As an example, eventually you get to a situation where the CEO just instructs the AGI employees to optimize the bank account directly.

Another question with respect to Amdahl's law is what kind of bottleneck any human-in-the-loop would constitute.

Compare the two options as complete computational systems: human + semi-autonomous AGI vs autonomous AGI. Human brains take on the order of seconds to make complex decisions, so in order to compete with autonomous AGIs, the human will have to either 1.) let the AGI operate autonomously for at least seconds at a time, or 2.) suffer a speed penalty where the AGI sits idle, waiting for the human response.

For example, imagine a marketing AGI creates ads, each of which may take a human a minute to evaluate (which is being generous). If the AGI thinks 3600X faster than human baseline, and a human takes on the order of hours to generate an ad, it would generate ads in seconds. The human would not be able to keep up, and so would have to back up a level of heirarachy and grant the AI autonomy over entire ad campaigns, and more realistically, the entire ad company. If the AGI is truly superintelligent, it can come to understand what the human actually wants at a deeper level, and start acting on anticipated and even implied commands. In this scenario I expect most human managers would just let the AGI sort out 'work' and retire early.

comment by XiXiDu · 2012-05-18T14:36:55.221Z · score: 2 (4 votes) · LW · GW

Well, I don't disagree with anything you wrote and believe that the economic case for a fast transition from tools to agents is strong.

I also don't disagree that an AGI could take over the world if in possession of enough resources and tools like molecular nanotechnology. I even believe that a sub-human-level AGI would be sufficient to take over if handed advanced molecular nanotechnology.

Sadly these discussions always lead to the point where one side assumes the existence of certain AGI designs with certain superhuman advantages, specific drives and specific enabling circumstances. I don't know of anyone who actually disagrees that such AGI's, given those specific circumstances, would be an existential risk.

comment by jacob_cannell · 2012-05-18T15:18:05.363Z · score: 0 (0 votes) · LW · GW

I don't see this as so sad, if we are coming to something of a consensus on some of the sub-issues.

This whole discussion chain started (for me) with a question of the form, "given a superintelligence, how could it actually become an existential risk?"

I don't necessarily agree with the implied LW consensus on the liklihood of various AGI designs, specific drives, specific circumstances, or most crucially, the actual distribution over future AGI goals, so my view may be much closer to yours than this thread implies.

But my disagreements are mainly over details. I foresee the most likely AGI designs and goal systems as being vaguely human-like, which entails a different type of risk. Basically I'm worried about AGI's with human inspired motivational systems taking off and taking control (peacefully/economically) or outcompeting us before we can upload in numbers, and a resulting sub-optimal amount of uploading, rather than paperclippers.

comment by XiXiDu · 2012-05-18T16:00:50.435Z · score: 1 (3 votes) · LW · GW

But my disagreements are mainly over details. I foresee the most likely AGI designs and goal systems as being vaguely human-like, which entails a different type of risk. Basically I'm worried about AGI's with human inspired motivational systems taking off and taking control (peacefully/economically) or outcompeting us before we can upload in numbers, and a resulting sub-optimal amount of uploading, rather than paperclippers.

Yes, human-like AGI's are really scary. I think a fabulous fictional treatment here is 'Blindsight' by Peter Watts, where humanity managed to resurrect vampires. More: Gurl ner qrcvpgrq nf angheny uhzna cerqngbef, n fhcreuhzna cflpubcnguvp Ubzb trahf jvgu zvavzny pbafpvbhfarff (zber enj cebprffvat cbjre vafgrnq) gung pna sbe rknzcyr ubyq obgu nfcrpgf bs n Arpxre phor va gurve urnqf ng gur fnzr gvzr. Uhznaf erfheerpgrq gurz jvgu n qrsvpvg gung jnf fhccbfrq gb znxr gurz pbagebyynoyr naq qrcraqrag ba gurve uhzna znfgref. Ohg bs pbhefr gung'f yvxr n zbhfr gelvat gb ubyq n png nf crg. V guvax gung abiry fubjf zber guna nal bgure yvgrengher ubj qnatrebhf whfg n yvggyr zber vagryyvtrapr pna or. Vg dhvpxyl orpbzrf pyrne gung uhznaf ner whfg yvxr yvggyr Wrjvfu tveyf snpvat n Jnssra FF fdhnqeba juvyr oryvrivat gurl'yy tb njnl vs gurl bayl pybfr gurve rlrf.

comment by jacob_cannell · 2012-06-13T06:37:08.923Z · score: 0 (0 votes) · LW · GW

That fictional treatment is interesting to the point of me actually looking up the book. But ..

Yes, human-like AGI's are really scary.

The future is scary. Human-like AGI's should not intrinsically be more scary than the future, accelerated.

comment by Strange7 · 2012-05-22T23:34:26.991Z · score: 0 (0 votes) · LW · GW

To retain control, a human manager will need to grant the AGI autonomy on larger timescales in proportion to the AGI's greater intelligence and speed, giving it bigger and more abstract hierachical goals. As an example, eventually you get to a situation where the CEO just instructs the AGI employees to optimize the bank account directly.

Nitpick: you mean "optimize shareholder value directly." Keeping the account balances at an appropriate level is the CFO's job.

comment by private_messaging · 2012-05-28T05:24:14.947Z · score: 0 (0 votes) · LW · GW

Having cloned its core millions of times over, the AI is now a civilization unto itself.

Precisely. It is then a civilization, not some single monolithic entity. The consumer PCs have a lot if internal computing power and comparatively very low inter-node bandwidth and huge inter-node lag, entirely breaking any relation to the 'orthogonality thesis', up to the point that the p2p intelligence protocols may more plausibly have to forbid destruction or manipulation (via second guessing which is a waste of computing power) of intelligent entities. Keep in mind that human morality is, too, a p2p intelligence protocol allowing us to cooperate. Keep in mind also that humans are computing resources you can ask to solve problems for you (all you need is to implement interface), while Jupiter clearly isn't.

The nuclear war is very strongly against interests of the intelligence that sits on home computers, obviously.

(I'm assuming for sake of argument that intelligence actually had the will to do the conquering of the internet rather than being just as content with not actually running for real)

comment by Douglas_Knight · 2012-05-23T20:54:01.972Z · score: 1 (1 votes) · LW · GW

Maybe you're thinking of this comment and others in that thread by Jed Harris (aka).

Jed's point #2 is more plausible, but you are talking about point #1, which I find unbelievable for reasons that were given before he answered it. If clock speed mattered, why didn't the failure of exponential clock speed shut down the rest of Moore's law? If computation but not clock speed mattered, then Intel should be able to get ahead of Moore's law by investing in software parallelism. Jed seems to endorse that position, but say that parallelism is hard. But hard exactly to the extent to allow Moore's law to continue? Why hasn't Intel monopolized parallelism researchers? Anyhow, I think his final conclusion is opposite to yours: he say that intelligence could lead to parallelism and getting ahead of Moore's law.

comment by jacob_cannell · 2012-05-23T21:50:11.641Z · score: 0 (0 votes) · LW · GW

Yes, thanks. My model of Jed's internal model of moore's law is similar to my own.

He said:

The short answer is that more computing power leads to more rapid progress. Probably the relationship is close to linear, and the multiplier is not small.

He then lists two examples. By 'points' I assume you are referring to his examples in the first comment you linked.

What exactly do you find unbelievable about his first example? He is claiming that the achievable speed of a chip is dependent on physical simulations, and thus current computing power.

If clock speed mattered, why didn't the failure of exponential clock speed shut down the rest of Moore's law?

Computing power is not clock speed, and Moore's Law is not directly about clock speed nor computing power.

Jed makes a number of points in his posts. In my comment on the earlier point 1 (in this thread), I was referring to one specific point Jed made: that each new hardware generation requires complex and lengthy simulation on the current hardware generation, regardless of the amount of 'intelligence' one throws at the problem.

comment by Douglas_Knight · 2012-05-24T02:27:27.118Z · score: 1 (1 votes) · LW · GW

There are two questions here: would computer simulations of the physics of new chips be a bottleneck for an AI trying to foom*? and are they a bottleneck that explains Moore's law? If you just replace humans by simulations, then the human time gets reduced with each cycle of Moore's law, leaving the physical simulations, so the simulations probably are the bottleneck. But Intel has real-time people, so saying that it's a bottleneck for Intel is a lot stronger a claim than saying it is a bottleneck for a foom.

First, foom:
If each year of Moore's law requires a solid month of computer time of state of the art processors, then eliminating the humans speeds it up by a factor of 12. That's not a "hard takeoff," but it's pretty fast.

Moore's Law:
Jed seems to say the computational requirements of physics simulations actually determine Moore's law and that if Intel had access to more computer resources, it could move faster. If it takes a year of computer time to design and test the next year's processor that would explain the exponential nature of Moore's law. But if it only takes a month, computer time probably isn't the bottleneck. However, this model seems to predict a lot of things that aren't true.

The model only makes sense if "computer time" means single threaded clock cycles. If simulations require an exponentially increasing number of ordered clock cycles, there's nothing you can do but get a top of the line machine and run it continuously. You can't buy more time. But clock speed stopped increasing exponentially, so if this is the bottleneck, Intel's ability to design new chips should have slowed down and Moore's law should have stopped. This didn't happen, so the bottleneck is not linearly ordered clock cycles. So the simulation must parallelize. But if it parallelizes, Intel could just throw money at the problem. For this to be the bottleneck, Intel would have to be spending a lot of money on computer time, which I do not think is true. Jed says that writing parallel software is hard and that it isn't Intel's specialty. Moreover, he seems to say that improvements in parallelism have perfectly kept pace with the failure of increasing clock speed, so that Moore's law has continued smoothly. This seems like too much of a coincidence to believe.

Thus I reject Jed's apparent claim that physics simulations are the bottleneck in Moore's law. If simulations could be parallelized, why didn't they invest in parallelism 20 years ago? Maybe it's not worth it for them to be any farther ahead of their competitors than they are. Or maybe there is some other bottleneck.


* actually, I think that an AI speeding up Moore's law is not very relevant to anything, but it's a simple example that many people like.

comment by jacob_cannell · 2012-05-24T03:27:18.249Z · score: 0 (0 votes) · LW · GW

There are differing degrees of bottlenecks.

Many, if not most, of the large software projects I have worked on have been at least partially bottlenecked by compile time, which is the equivalent to the simulation and logic verification steps in hardware design. If I thought and wrote code much faster, this would be a speedup, but only to a saturation point where I wait for compile-test cycles.

If it takes a year of computer time to design and test the next year's processor that would explain the exponential nature of Moore's law.

Yes. Keep in mind this is a moving target, and that is the key relation to Moore's Law. It would take computers from 1980 months or years to compile windows 8 or simulate a 2012 processor.

The model only makes sense if "computer time" means single threaded clock cycles.

I don't understand how the number of threads matters. Compilers, simulators, logic verifiers, all made the parallel transition when they had to.

Moreover, he seems to say that improvements in parallelism have perfectly kept pace with the failure of increasing clock speed, so that Moore's law has continued smoothly. This seems like too much of a coincidence to believe.

Right, it's not a coincidence, it's a causal relation. Moore's Law is not a law of nature, it's a shared business plan of the industry. When clock speed started to run out of steam, chip designers started going parallel, and software developers followed suit. You have to understand that chip designs are planned many years in advance, this wasn't an entirely unplanned, unanticipated event.

As for the details of what kind of simulation software Intel uses, I'm not sure. Jed's last posts are also 4 years old at this point, so much has probably changed.

I do know that Nvidia uses big expensive dedicated emulators from a company called Cadence (google "Cadence Nvidia") and this really is a big deal for their hardware cycle.

Thus I reject Jed's apparent claim that physics simulations are the bottleneck in Moore's law.

Well, you seem to agree that they are some degree of bottleneck, so it may good to narrow in on what level of bottleneck, or taboo the word.

If simulations could be parallelized, why didn't they invest in parallelism 20 years ago?

It was unecessary, because the fast easy path (faster serial speed) was still paying fruit.

comment by Douglas_Knight · 2012-05-24T04:01:24.133Z · score: 1 (1 votes) ·