Should we postpone AGI until we reach safety? 2020-11-18T15:43:51.846Z
Help wanted: feedback on research proposals for FHI application 2020-10-08T14:42:50.033Z
otto.barten's Shortform 2020-09-19T10:34:32.453Z
Looking for non-AI people to work on AGI risks 2019-12-30T20:41:43.625Z


Comment by otto.barten (otto-barten) on Should we postpone AGI until we reach safety? · 2020-12-11T08:55:30.112Z · LW · GW

Richard, thanks for your reply. Just for reference, I think this goes under argument 5, right?

It's a powerful argument, but I think it's not watertight. I would counter it as follows:

  1. As stated above, I think the aim should be an ideally global treaty were no country is allowed to go beyond a certain point of research. The countries should then enforce the treaty on all research institutes/companies within their borders. You're right that in this case, a criminal or terrorist group will have an edge. But seeing how hard it currently is for legally allowed and indeed heavily funded groups to develop AGI, I'm not convinced that terrorist or criminal groups could easily do this. For reference, I read this paper by a lawyer this week on an actual way to implement such a treaty. I think signing such a treaty will not affect countries without effective AGI research capabilities, so they won't have a reason not to sign it, and will benefit from the increased existential safety. The ones likely least inclined to sign up will be the countries that are trying to develop AGI now. So effectively, I think a global treaty and a US/China deal would amount to roughly the same thing.
  2. You could make the same argument for tax, (not profitable) climate action, R&D, defense spending against a common enemy, and probably many other issues. Does that mean we have zero tax, climate action, R&D, or defense? No, because at some point countries realize it's better to not be the relative winner, than to all loose. In many cases this is then formalized in treaties, with varying but nonzero success. I think that could work in this case as well. Your argument is indeed a problem in all of the fields I mention, so you have a point. But I think, fortunately, it's not a decisive point.
Comment by otto.barten (otto-barten) on otto.barten's Shortform · 2020-12-11T08:31:26.874Z · LW · GW

Minimum hardware leads to maximum security. As a lab or a regulatory body, one can increase safety of AI prototypes by reducing the hardware or amount of data researchers have access to.

Comment by otto.barten (otto-barten) on Should we postpone AGI until we reach safety? · 2020-11-24T20:41:01.055Z · LW · GW

My response to counterargument 3 is summarized in this plot, for reference:

Basically, this would only be an issue if postponement cannot be done until risks are sufficiently low, and if take-off would be slow without postponement intervention.

Comment by otto.barten (otto-barten) on Should we postpone AGI until we reach safety? · 2020-11-23T20:09:36.837Z · LW · GW

Interesting line of thought. I don't know who and how, but I still think we should already think about if it would be a good idea in principle.

Can I restate your idea as 'we have a certain amount of convinced manpower, we should use it for the best purpose, which is AI safety'? I like the way of thinking, but I still think we should use some of them for looking into postponement. Arguments:

- The vast majority of people is unable to contribute meaningfully to AI safety research. Of course all these people could theoretically do whatever makes most money and then donate to AI safety research. But most will not do that in practice. I think many of these people could be used for the much more generic task of convincing others about AI risks, and also arguing for postponement. As an example, I saw a project once with the goal of teaching children about AI safety which claimed they could not continue for lack of 5000$ of funding. I think there's a vast sea of resource-constrained possibility out there once we make the decision that telling everyone about AI risk is officially a good idea.

- Postponement weirdly seems to be a neglected topic within the AI safety community (for dislike of regulation, I guess), but also outside the community (for lack of AI risk insight). I think it's a lot more neglected at this point than technical AI safety, which is perhaps also niche, but does have its own institutes already looking at it. Since it looks important and neglected, I think an hour spent on postponement is probably better spent than an hour on AI safety, perhaps unless you're a talented AI safety researcher.

Comment by otto.barten (otto-barten) on Should we postpone AGI until we reach safety? · 2020-11-22T13:39:59.198Z · LW · GW

Thanks for that comment! I didn't know Bill McKibben, but I read up on his 2019 book 'Falter: Has the Human Game Begun to Play Itself Out?' I'll post a review as a post later. I appreciate your description of what the scene was like back in the 90s or so, that's really insightful. Also interesting to read about nanotech, I never knew these concerns were historically so coupled.

But having read McKibben's book, I still can't find others on my side of the debate. McKibben is indeed the first one I know who both recognizes AGI danger, and does not believe in a tech fix, or at least does not consider this a good outcome. However, I would expect that he would cite others on his side of the debate. Instead, in the sections on AGI, he cites people like Bostrom and Omohundro, which are not postponists in any way. Therefore I'm still guessing at this moment that a 'postponement side' of this debate is now absent, and it's just that McKibben happened to know Kurzweil who got him personally concerned about AGI risk. If that's not true and there are more voices out there exploring AGI postponement options, I'd still be happy to hear about it. Also if you could find links to old discussions, I'm interested!

Comment by otto.barten (otto-barten) on Should we postpone AGI until we reach safety? · 2020-11-20T19:30:22.986Z · LW · GW

Thanks for your insights Adam. If every AGI researcher is in some sense for halting AGI research, I'd like to get more confirmation on that. What are their arguments? Would they also work for non-AGI researchers?

I can imagine the combination of Daniel's point 1 and 2 stops AGI researchers from speaking out on this. But for non-AGI researchers, why not explore something that looks difficult, but may have existential benefits?

Comment by otto.barten (otto-barten) on Should we postpone AGI until we reach safety? · 2020-11-20T05:18:41.718Z · LW · GW

I agree and thanks for bringing some nuance in the debate. I think that would be a useful path to explore.

Comment by otto.barten (otto-barten) on Should we postpone AGI until we reach safety? · 2020-11-19T17:47:44.362Z · LW · GW

I'm imagining an international treaty, national laws, and enforcement from police. That's a serious proposal.

Comment by otto.barten (otto-barten) on Should we postpone AGI until we reach safety? · 2020-11-19T17:43:10.314Z · LW · GW

I think a small group of thoughtful, committed, citizens can change the world. Indeed, it is the only thing that ever has.

Comment by otto.barten (otto-barten) on Should we postpone AGI until we reach safety? · 2020-11-19T17:31:38.731Z · LW · GW

I appreciate the effort you took in writing a detailed response. There's one thing you say in which I'm particularly interested, for personal reasons. You say 'I've been in or near this debate since the 1990s'. That suggests there are many people with my opinion. Who? I would honestly love to know, because frankly it feels lonely. All people I've met, so far without a single exception, are either not afraid of AI existential risk at all, or believe in a tech fix and are against regulation. I don't believe in the tech fix, because as an engineer, I've seen how much of engineering is trial and error (and science even more). People have ideas, try them, it says boom and then they try something else. Until they get there. If we do that with AGI, I think it's sure to go wrong. That's why I think at least some kind of policy intervention is mandatory, not optional. And yes it will be hard. But no argument I've heard so far has convinced me that it's impossible. Or that it's counterproductive.

I think we should first answer the question: is postponement until safety a good idea if it would be implementable. What's your opinion on that one?

Also, I'm serious: who else is on my side of this debate? You would really help me personally to let me talk to them, if they exist.

Comment by otto.barten (otto-barten) on Should we postpone AGI until we reach safety? · 2020-11-18T21:07:23.813Z · LW · GW

That goes under Daniel's point 2 I guess?

Comment by otto.barten (otto-barten) on Should we postpone AGI until we reach safety? · 2020-11-18T21:05:39.859Z · LW · GW

I'm not in favour of nuclear war either :)

Comment by otto.barten (otto-barten) on Should we postpone AGI until we reach safety? · 2020-11-18T21:00:36.648Z · LW · GW

Why do you think we need to find out 1 before trying? I would say, if it is indeed a good idea to postpone, then we can just start trying to postpone. Why would we need to know beforehand how effective that will be? Can't we find that out by trial and error if needed? Worst case, we would be postponing less. That is of course, as long as the flavor of postponement does not have serious negative side effects.

Or rephrased, why do these brakes need to be carefully calibrated?

Comment by otto.barten (otto-barten) on Should we postpone AGI until we reach safety? · 2020-11-18T20:21:12.011Z · LW · GW

Thanks for your comments, these are interesting points. I agree that these are hard questions and that it's not clear that policymakers will be good at answering them. However, I don't think AI researchers themselves are any better, which you seem to imply. I've worked as an engineer myself and I've seen that when engineers or scientists are close to their own topic, their judgement of any risks/downsides of this topic does not become more reliable, but less. AGI safety researchers will be convinced about AGI risk, but I'm afraid their judgement of their own remedies will also not be the best judgment available. You're right, these risk estimates may be technical and politicians will not have the opportunity to look into the details. What I would have in mind is more a governmental body. We have an environmental planning agency in The Netherlands for example, helping politicians with technical climate questions. Something like that for AGI - with knowledgeable people, but not tied to AI research themselves - that's how close you can come to a good risk estimate I think.

You might also say that any X-risk above a certain threshold, say 1%, is too high. Then perhaps it doesn't even matter whether it's 10% or 15%. Although I still think it's important impartial experts in service of the public find out.

Comment by otto.barten (otto-barten) on Should we postpone AGI until we reach safety? · 2020-11-18T15:59:24.945Z · LW · GW

Thanks for your thoughts. Of course we don't know whether AGI will harm or help us. However I'm making the judgement that the harm could plausibly be so big (existential), that it outweighs the help (reduction in suffering for the time until safe AGI, and perhaps reduction of other existential risks). You seem to be on board with this, is that right?

Why exactly do you think interference would fail? How certain are you? I'm acknowledging it would be hard, but not sure how optimist/pessimist to be on this.

Comment by otto.barten (otto-barten) on The date of AI Takeover is not the day the AI takes over · 2020-11-01T12:58:04.477Z · LW · GW

Well sure, why not. I'll send you a PM.

Comment by otto.barten (otto-barten) on The date of AI Takeover is not the day the AI takes over · 2020-10-29T09:44:19.573Z · LW · GW

I have kind of a strong opinion in favor of policy intervention because I don't think it's optional. I think it's necessary. My main argument is as follows:

I think we have two options to reduce AI extinction risk:

1) Fixing it technically and ethically (I'll call the combination of both working out the 'tech fix'). Don't delay.

2) Delay until we can work out 1. After the delay, AGI may or may not still be carried out, depending mainly on the outcome of 1.

If option 1 does not work, of which there is a reasonable chance (it hasn't worked so far and we're not necessarily close to a safe solution), I think option 2 is our only chance to reduce the AI X-risk to acceptable levels. However, AI academics and corporations are both strongly opposed to option 2. It would therefore take a force at least as powerful as those two groups combined to still pursue this option. The only option I can think of is a popular movement. Lobbying and think tanking may help, but corporations will be better funded and therefore the public interest is not likely to prevail. Wonkery could be promising as well. I'm happy to be convinced of more alternative options.

If the tech fix works, I'm all for it. But currently, I think the risks are way too big and it may not work at all. Therefore I think it makes sense to apply the precautionary principle here and start with policy interventions, until it can be demonstrated that X-risk for AGI has fallen to an acceptable level. As a nice side effect, this should dramatically increase AI Safety funding, since suddenly corporate incentives are to fund this first in order to reach allowed AGI.

I'm aware that this is a strong minority opinion on LW, since:

1) Many people here have affinity with futurism which would love an AGI revolution

2) Many people have backgrounds in AI academia, and/or AI corporations, which both have incentives to continue working on AGI

3) It could be wrong of course. :) I'm open for arguments which would change the above line of thinking.

So I'm not expecting a host of upvotes, but as rationalists, I'm sure you appreciate the value of dissent as a way to move towards a careful and balanced opinion. I do at least. :)

Comment by otto.barten (otto-barten) on The date of AI Takeover is not the day the AI takes over · 2020-10-29T09:17:55.545Z · LW · GW

I wouldn't say less rational, but more bipartisan, yes. But you're right I guess that European politics is less important in this case. Also don't forget Chinese politics, which has entirely different dynamics of course.

I think you have a good point as well that wonkery, think tankery, and lobbying are also promising options. I think they, and starting a movement, should be on a little list of policy intervention options. I think each will have its own merits and issues. But still, we should have a group of people actually starting to work on this, whatever the optimal path turns out to be.

Comment by otto.barten (otto-barten) on The date of AI Takeover is not the day the AI takes over · 2020-10-27T23:42:04.770Z · LW · GW

It's funny, I heard that opinion a number of times before, mostly from Americans. Maybe it has to do with your bipartisan flavor of democracy. I think Americans are also much more skeptical of states in general. You tend to look to companies for solving problems, Europeans tend to look to states (generalized). In The Netherlands we have a host of parties, and although there are still a lot of pointless debates, I wouldn't say it's nearly as bad as what you describe. I can't imagine e.g. climate change solved without state intervention (the situation here is now that the left is calling for renewables, the right for nuclear - not so bad).

For AI Safety, even with a bipartisan debate, the situation now is that both parties implicitly think AI Safety is not an issue (probably because they have never heard of it, or at least not given it serious thought). After politicization, worst case at least one of the parties will think it's a serious issue. That would mean that roughly 50% of the time, if party #1 wins, we get a fair chance of meaningful intervention such as appropriate funding, hopefully helpful regulation efforts (that's our responsibility too - we can put good regulation proposals out there), and even cooperation with other countries. If party #2 wins, there will perhaps be zero effort or some withdrawal. I would say this 50% solution easily beats the 0% solution we have now. In a multi-party system such as we have, the outcome could even be better.

I think we should prioritize getting the issue out there. The way I see it, it's the only hope for state intervention, which is badly needed.

Comment by otto.barten (otto-barten) on The date of AI Takeover is not the day the AI takes over · 2020-10-26T23:06:33.089Z · LW · GW

Don't get me wrong, I think institutes like FHI are doing very useful research. I think there should be a lot more of them, at many different universities. I just think what's missing in the whole X-risk scene is a way to take things out of this still fairly marginal scene and into the mainstream. As long as the mainstream is not convinced that this is an actual problem, efforts are always enormously going to lag mainstream AI efforts, with predictable results.

Comment by otto.barten (otto-barten) on The date of AI Takeover is not the day the AI takes over · 2020-10-26T19:19:16.736Z · LW · GW

I know their work and I'm pretty sure there's no list on how to convince governments and corporations that AI risk is an actual thing.. PhDs are not the kind of people inclined to take any concrete action I think.

Comment by otto.barten (otto-barten) on The date of AI Takeover is not the day the AI takes over · 2020-10-26T17:17:30.944Z · LW · GW

I agree and I think books such as Superintelligence have definitely decreased the x-risk chance. I think 'convincing governments and corporations that this is a real risk' would be a great step forward. What I haven't seen anywhere, is a coherent list of options how to achieve that, preferably ranked by impact. A protest might be up there, but probably there are better ways. I think making that list would be a great first step. Can't we do that here somewhere?

Comment by otto.barten (otto-barten) on The date of AI Takeover is not the day the AI takes over · 2020-10-26T12:33:01.157Z · LW · GW

That makes sense and I think it's important that this point gets made. I'm particularly interested by the political movement that you refer to. Could you explain this concept in more detail? Is there anything like such a political movement already being built at the moment? If not, how would you see this starting?

Comment by otto.barten (otto-barten) on otto.barten's Shortform · 2020-10-15T11:33:57.939Z · LW · GW

I think actually 1+1 = ? is not really an easy enough goal, since it's not 100% sure that the answer is 2. Getting to 100% certainty (including what I actually meant with that question) could still be nontrivial. But let's say the goal is 'delete filename.txt'? Could be the trick is in the language..

Comment by otto.barten (otto-barten) on otto.barten's Shortform · 2020-10-15T11:30:27.786Z · LW · GW

Thanks again for your reply. I see your point that the world is complicated and a utility maximizer would be dangerous, even if the maximization is supposedly trivial. However, I don't see how an achievable goal has the same problem. If my AI finds the answer of 2 before a meteor hits it, I would say it has solidly landed at 100% and stops doing anything. Your argument would be true if it decides to rule out all possible risks first, before actually starting to look for the answer of the question, which would otherwise quickly be found. But since ruling out those risks would be much harder to achieve than finding the answer, I can't see my little agent doing that.

I think my easy goals come closest to what you call other-izers. Any more pointers for me to find that literature?

Thanks for your help, it helps me to calibrate my thoughts for sure!

Comment by otto.barten (otto-barten) on otto.barten's Shortform · 2020-10-13T11:36:33.902Z · LW · GW

Thanks for your insights. I don't really understand 'setting [easy] goals is an unsolved problem'. If you set a goal: "tell me what 1+1 is", isn't that possible? And once completed ("2!"), the AI would stop to self-improve, right?

I think this may contribute to just a tiny piece of the puzzle, however, because there will always be someone setting a complex or, worse, non-achievable goal ("make the world a happy place!"), and boom there you have your existential risk again. But in a hypothetical situation where you have your AGI in the lab, no-one else has, and you want to play around safely, I guess easy goals might help?

Curious about your thoughts, and also, I can't imagine this is an original idea. Any literature already on the topic?

Comment by otto.barten (otto-barten) on Help wanted: feedback on research proposals for FHI application · 2020-10-12T08:54:00.430Z · LW · GW

Thanks Charlie! :)

They are asking for only one proposal, so I will have to choose one and am planning to work out that one. So I'm mostly asking about which idea you find most interesting, rather than about which one is the strongest proposal now - that will be worked out. But thanks a lot for your feedback so far - that helps!

Comment by otto.barten (otto-barten) on otto.barten's Shortform · 2020-09-19T10:49:08.788Z · LW · GW

AGI is unnecessary for an intelligence explosion

Many arguments state that it would require an AGI to have an intelligence explosion. However, it seems to me that the critical point for achieving this explosion is that an AI can self-improve. Which skills are needed for that? If we have hardware overhang, it probably comes down to the type of skills an AI researcher uses: reading papers, combining insights, doing computer experiments until new insights emerge, writing papers about them. Perhaps an AI PhD can weigh in on the actual skills needed. I'm however making the argument that far from all mental skills humans have are needed for AI research. Appreciating art? Not needed. Intelligent conversation about non-AI topics? Not needed. Motor skills? Not needed.

I think the skills needed most for AI research (and therefore self-improvement) are the skills at which a computer may be relatively strong: methodical thinking, language processing, coding. Therefore I would expect that we reach an intelligence explosion significantly earlier than developing actual AGI with all human skills. This should be important for the timeline discussion.

Comment by otto.barten (otto-barten) on otto.barten's Shortform · 2020-09-19T10:34:32.711Z · LW · GW

Tune AGI intelligence by easy goals

If an AGI is provided an easily solvable utility function ("fetch a coffee"), it will lack the incentive to self-improve indefinitely. The fetch-a-coffee-AGI will only need to become as smart as a hypothetical simple-minded waiter. By creating a certain easiness for a utility function, we can therefore tune the intelligence level we want an AGI to achieve using self-improvement. The only way to achieve an indefinite intelligence explosion (until e.g. material boundaries) would be to program a utility function maximizing something. Therefore this type of utility function will be most dangerous.

Could we create AI safety by prohibiting maximizing-type utility functions? Could we safely experiment with AGIs just a little smarter than us, by using moderately hard goals?

Comment by otto.barten (otto-barten) on Reaching out to people with the problems of friendly AI · 2020-09-12T12:10:01.813Z · LW · GW

This is the exact topic I'm thinking a lot about, thanks for the link! I've wrote my own essay for a general audience but it seems ineffective. I knew about the Wait but why blog post, but there must be better approaches possible. What I find hard to understand is that there have been multiple best-selling books about the topic, but still no general alarm is raised and the topic is not discussed in e.g. politics. I would be interested in why this paradox exists, and also how to fix it.

Is there any more information about reaching out to a general audience on Lesswrong? I've not been able to find it using the search function etc.

The reason I'm interested is twofold:

1) If we convince a general audience that we face an important and understudied issue, I expect them to fund research into it several orders of magnitude more generously, which should help enormously in reducing the X-risk (I'm not working in the field myself).

2) If we convince a general audience that we face an important and understudied issue, they may convince governing bodies to regulate, which I think would be wise.

I've heard the following counterarguments before, but didn't find them convincing. If someone would want to convince me that convincing the public about AGI risk is not a good idea, these are places to start:

1) General audiences might start pressing for regulation which could delay AI research in general and/or AGI. That's true and indeed a real problem, since all the potential positive aspects of AI/AGI (which may be enormous) cannot be applied yet. However, in my opinion the argument is not sufficient because:

A) AGI existential risk is so high and important that reducing it is more important than AI/AGI delay, and

B) Increased knowledge of AGI will also increase general AI interest, and this effect could outweigh the delay that regulation might cause.

2) AGI worries from the general public could make AI researchers more secretive and less cooperative in working together with AI Safety research. My problem with this argument is the alternative: I think currently, without e.g. politicians discussing this issue, the investments in AI Safety are far too small to have a realistic shot at actually solving the issue timely. Finally, AI Safety may well not be solvable at all, in which case regulation gets more important.

Would be super to read your views and get more information!

Comment by otto.barten (otto-barten) on Looking for non-AI people to work on AGI risks · 2020-01-01T16:58:51.203Z · LW · GW

Also another thought. (Partially) switching careers comes with a large penalty, since you don't have as much previous knowledge, experience, credibility, and network for the new topic. The only reason I'm thinking about it, is that I think AGI risk is a lot more important to work on than climate risk. If you're moving in the opposite direction:

1) Do you agree that such moving comes with a penalty?

2) Do you think that climate risk is a lot more important to work on than AGI risk?

If so, only one of us can be right. It would be nice to know who that is, so we don't make silly choices.

Comment by otto.barten (otto-barten) on Looking for non-AI people to work on AGI risks · 2020-01-01T15:45:26.691Z · LW · GW

Hi Brian, thanks for your reply! I think we would not need very special qualifications for this, it's more a matter of reading up on the main status of AI and safe AI, cite the main conclusions from academia and make sure they get presented well to both policy makers and normal people. You say you'd expect countless others to want to work on this too, but I didn't find them yet. I'm still hopeful they may exist somewhere, and if you find people already doing this, I'd love to get in contact with them. Else, we should start ourselves.

Interesting observation! I'm thinking that your second front is especially interesting/worrying where AI improvement tasks are automated. For a positive feedback loop to occur, making AI get smarter very fast, many imagine an AGI is necessary. However, I'm thinking, what's improving AI now? Which skills are required? I think it's partially hardware improvement: academia and industry working together to keep Moore's law going. The other part is software/algorithm improvements, also done by academics and companies such as Deep Mind etc. So if the tasks of those researchers would be automated, that would be the point at which the singularity could take off. Their jobs tend to be analytical and focused on a single task, more than generically human and social, which I guess means that AI would find them easier. That in turn means the singularity (there should be a less scifi name for this) could happen sooner than AGI, if policy doesn't intervene. So also a long winded I agree.

So how should we go about organizing this, if no one is doing it yet? Any thoughts?

Thanks again for your reply, as I said above it's heartening that there are people out there who are on more or less the same page!

Comment by otto.barten (otto-barten) on Looking for non-AI people to work on AGI risks · 2020-01-01T14:16:15.951Z · LW · GW

Hi WH, thank you for the reply! I find it really heartening and encouraging to learn what others are thinking.

Could you explain what hardware you think would be needed? It's kind of the first time I'm hearing someone talk about that, so I'm curious of course to learn what you think it would take.

I agree with your point that understanding risks of AI projects is a good way of framing things. Given the magnitude of AGI risks (as I understand it now, human extinction), an alarmist tone of a policy report would still be justified in my opinion. I also agree that we should keep an open mind: I see the benefits of AI, and even more the benefits of AGI, which would be biblical if we could control the risks. Climate adaptation could indeed be carried out a lot better, as could many other tasks. However, I think that we will not be able to control AGI, and we may therefore go extinct if we still develop it. But agreed: let's keep an open mind about the developments.

Do you know any reliable overview of AGI risks? It would be great to have a kind of IPCC equivalent that's as uncontroversial as possible to convince people that this problem needs attention. Or papers stating that there is a nonzero chance of human extinction, from a reliable source. Any such information would be great!

If I can help you by the way with ideas on how to fight the climate crisis, let me know!