Learning to get things right first time

Having been in the industry for longer than most people around here, I am fairly confident that what you are suggesting is, for all practical purposes, impossible. I mean, it's possible to train and set up prizes, it's impossible to write bug-free code of any complexity.

If you look through the history of such attempts (e.g. Dijkstra's formal verification, various procedural and functional languages specifically developed for this purpose), even the successful ones failed at the stated goals of producing bug-free code.

As Gram_Stone mentioned earlier, the reason is nearly always the same: humans are incapable of producing bug-free non-trivial formal requirements. And that you cannot teach without being a fraud, because human mind itself is too weak and buggy to analyze a complex enough problem.

I don't know if this is commercially feasible, but I do like this idea from the perspective of building civilizational competence at getting things right on the first try.

Maybe I'm missing the point, but it seems that good old-fashioned formal verification would largely mitigate the issue of bugs in the classical sense, which seems to be what MIRI already intends to use. The greatest problem lies outside of the issue of verifying that the program functions in the obvious sense; it lies in ensuring that the specification is referring to that which we want it to refer: the full complement of human values. This is a philosophical and scientific problem that is likely beyond the purview of formal verification or best programming practices.

For more than a decade I have been systematically identifying error-prone programming habits—by reviewing the literature, by analyzing other people’s mistakes, and by analyzing my own mistakes—and redesigning my programming environment to eliminate those habits. For example, “escape” mechanisms, such as backslashes in various network protocols and % in printf, are error-prone: it’s too easy to feed “normal” strings to those functions and forget about the escape mechanism.

I switched long ago to explicit tagging of “normal” strings; the resulting APIs are wordy but no longer error-prone. The combined result of many such changes has been a drastic reduction in my own error rate. Starting in 1997, I offered $500 to the first person to publish a verifiable security hole in the latest version of qmail, my Internet-mail transfer agent; see http://cr.yp.to/qmail/guarantee.html. There are now more than a million Internet SMTP servers running qmail. Nobody has attempted to claim the $500. Starting in 2000, I made a similar $500 offer for djbdns, my DNS software; see http://cr.yp.to/djbdns/guarantee.html. This program is now used to publish the IP addresses for two million .com domains: citysearch.com, for example, and lycos.com. Nobody has attempted to claim the $500.

There were several non-security bugs in qmail, and a few in djbdns. My error rate has continued to drop since then. I’m no longer surprised to whip up a several-thousand-line program and have it pass all its tests the first time.

Bug-elimination research, like other user-interface research, is highly nonmathematical.

The goal is to have users, in this case programmers, make as few mistakes as possible in achieving their desired effects. We don’t have any way to model this—to model human psychology—except by experiment. We can’t even recognize mistakes without a human’s help. (If you can write a program to recognize a class of mistakes, great—we’ll incorporate your program into the user interface, eliminating those mistakes—but we still won’t be able to recognize the remaining mistakes.) I’ve seen many mathematicians bothered by this lack of formalization; they ask nonsensical questions like “How can you prove that you don’t have any bugs?” So I sneak out of the department, take off my mathematician’s hat, and continue making progress towards the goal.

http://cr.yp.to/cv/activities-20050107.pdf (apparently this guy's accomplishments are legendary in crypto circles)

Giving an artificial intelligence good values may be a particularly important challenge, and one where we need to be correct first time.

This view is probably completely mistaken, for two separate reasons:

1. we can test AI architectures at different levels of scaling. A human brain is just a scaled up primate brain, which suggests that all the important features of how value acquisition works, empathy, altruism, value alignment, whatever - all of those features can be tested first in AGI that is near human level.

2. We already have encountered numerous large-scale 'one-shot' engineering challenges, and there is already an extremely effective general solution. If you have a problem that you have to get right the first try, you change this into an iterative problem by creating a simulation framework. Doing that for AGI may involve creating the Matrix, more or less, but that isn't necessarily anymore complex than creating AGI in the first place.

This is a really cool idea. The things you need to do to get things right the first time in a novel situation varies heavily according to what the problem is. Programming would be useful for training yourself on many tasks since many situations need you to consider a decision and then decide on a series of actions that follow logically from the situation.

However, there are many other skills. To successfully complete some tasks, you may need an extremely good memory so that you can remember to follow all the steps. Others may require you to be free from biases, understand people and understand risk. For many situations, the best way to make decisions is actually intuitive and here the programming practise will be much less useful. But, it'd still be helpful for many situations.

From http://www.infoq.com/news/2015/05/provably-correct-software :

Companies like ASML and Philips Healthcare report 40% or greater reduction in costs and time-to-market using formal methods to deliver highly reliable software control products. [...]

NASA uses the Spin model checker to develop and verify Mars rover control software – google "mars code". Toyota, after its debacle with Camry unintended acceleration, adopted the Altran SPARK formal-methods tools to develop reliable software (toyota ada spark).

I think there may be an unfounded assumption here - that an unfriendly AI would be the results of some sort of bug, or coding errors that could be identified ahead of time and fixed.

I rather suspect those sorts of stuff would not result in "unfriendly", they would result in crash/nonsense/non-functional AI.

Presumably part of the reason the whole friendly/non-friendly thing is an issue is because our models of cognition are crude, and a ton of complex high-order behavior is a result of emergent properties in a system, not from explicit coding. I would expect the sort of error that accidentally turns an AI into a killer robot would be subtle enough that it is only comprehensible in hindsight, if then. (Note this does not mean intentionally making a hostile AI is all that hard. I can make hostility, or practical outcomes identical to it, without AI at all, so it stands to reason that could carry over)

Meta: I'd love to know whether the downvotes are because people don't like the presentation of undeveloped ideas like this, or because they don't think the actual idea is a good one.

(The first would put me off posting similar things in the future, the second would encourage me as a feedback mechanism.)

I would suggest that software is not a good domain to start this project in because it is by nature able to be replicated in various forms. (Ctrl-c, CTRL-V for one).

If you consider other areas; for example - refining a piece of wood. In fact in any other domain where working with a rare natural formation that has value due to its form; (Rare minerals or crystals; Ceramics where once fired you must start again if it doesn't work, pure substances where once contaminated will need to be purified again, )

even simpler models of a perfection process are possible - peeling an egg perfectly, shelling a nut.

Its a matter of putting extra value on a perfect finish product. Where a premium diamond is worth a lot more than a 99% pure one.

Side note: the advantage of software is that it is easily scaleable to teach many people at once.

Most challenges we can approach with trial-and-error, so many of our habits and social structures are set up to encourage this.

This hasn't always been the case. Throughout history leader have had to get it right the first time. Especially in war-times. I'd bet that someone more versed in history than I could give lots of examples. Granted in our current society which is saturated with complexity and the limits of human comprehension trial-and-error seems like a viable way. But hierachical processes seem to work quite well for major human endeavors like th manhatten project.