Desired articles on AI risk?

"Why If Your AGI Doesn't Take Over The World, Somebody Else's Soon Will"

i.e. however good your safeguards are, it doesn't help if:

• another team can take your source code and remove safeguards (and why they might have incentives to do so)
• Multiple discovery means that your AGI invention will soon be followed by 10 independent ones, at least one of which will lack necessary safeguards

EDIT: "safeguard" here means any design feature put in to prevent the AGI obtaining singleton status.

Something on singletons: desirability, plausibility, paths to various kinds (strongly relates to stable attractors)

"Hell Futures - When is it better to be extinct?" (not entirely serious)

I'd be interested to see a critique of Hanson's em world, but within the same general paradigm (i.e. not "that won't happen because intelligence explosion").

e.g.

• ems would respect our property rights why exactly?
• how useful is analysis given "ems behave just like fast copyable humans" assumption probably won't be valid for long?

A Survey of Mathematical Ethics which covers work in multiple disciplines. I'd love to know what parts of ethics have been formalized enough to be written mathematically and, for example, any impossibility results that have been shown.

Something on logical uncertainty

Why the hard problems of AI (mainly, how to represent the world) are likely to ever be solved.

What are your requirements for the desired articles? Is it sufficient that, say, the request respondent read the abstracts of all relevant papers, and then summarizes and cites them? If so, I can knock out a few of these soon.

"What Would AIXI Do With Infinite Computing Power and a Halting Oracle?"

This is a fun one, but even easier would be "What would AIXI do with Aladdin's genie?"

"Experiments we could run today on a laptop which might tell us something about AI risk"

It might be very short, haha. But if not, it would be interesting!

It seems that no one is working in papers about the convergence of values. In a scale of difficult, math problems seens the priority, but the value and preferences disagreement impose a constrain in the implementantion. More specifically, in the "programmer writing code with black boxes" part.

How does making LW posts about these compare to writing papers with a more academic focus?

What is the proportionality thesis in the context of Intelligence Explosion?

The one I googled says something about the worst punishments for the worst crimes.

"AI Will Be Maleficent By Default"

seems like an a priori predetermined conclusion (bad science, of the "I want this to be true" kind), rather than a research result (good problem statement for AGI risk research). A better title would be rephrased as a research question:

"Will AI Be Maleficent By Default?"

"Autonomous Technology and the Greater Human Good" by Steve Omohundro

Summary slides: http://selfawaresystems.files.wordpress.com/2012/12/autonomous-technology-and-the-greater-human-good.pdf

"What Would AIXI Do With Infinite Computing Power and a Halting Oracle?"

Is this problem well-posed? Doesn't the answer depend completely on the reward function?

I'd like to hear futther commentary on "Stable Attractors of Technologically Advanced Civilizations" for a few reasons. 1) If it relates to cultural evolution, it relates to my masters 2) Seems like a sociology/memetics problem, areas I studied for a while. 3) Whoever would like to tackle it, I'd like to cooperate.

I wish I were confident enough in my skills, knowledge and rationality to actually work on some of these papers. "Self-Modification and Löb's Theorem" is exactly what comes up when I query my brain for "awesome time spent in a cave dedicated entirely to solving X". All the delicious mind-bending recursion.

Hopefully in a few years I'll look back on this and chuckle. "Hah, to think that back then I thought of simple things like self-modification and Löb's Theorem as challenges! How much stronger I've become, now."

More on topic, I feel like there's a need for something addressing the specific AGI vs Specialized AI questions/issues. Among things that pop to mind, why a self-modifying "specialized" AI will, given enough time and computing power, just end up becoming a broken paperclip-maximizing AGI anyway - even if it's "grown" from weak Machine Learning algorithms.

A Survey of Mathematical Ethics which covers work in multiple disciplines. I'd love to know what parts of ethics have been formalized enough to be written mathematically and, for example, any impossibility results that have been shown.