Brainstorming additional AI risk reduction ideas

Establish a scholarship to collect information on young talent

Related: Reaching young math/compsci talent

Create a merit scholarship for the type of young talent that SI wants to attract – this can reveal valuable information about this group of people, and can potentially be used a targeted publicity tool if handled well.

Information that could be collected from applications

• Basic personal details (age, location, contact methods, etc.)
• Education (past and future)
• Academic interests
• Career goals
• Awards and competition results
• Third-party reviews (i.e., letters of recommendation)
• Basic personality assessment (see previous LW discussion on correlates with Big Five personality traits: [1], [2], [3])
• Ideas about and attitudes toward x-risks/FAI/SI/FHI (these could be responses to prompts – as a bonus, applicants are introduced to the content in the prompts)
• ... Pretty much anything else (personal anecdote: I've revealed things about myself in college and scholarship applications that I have never expressed to anyone else)

Uses of this information

• Check whether SI is effectively reaching the right people with its current plans.
• The compiled list of young talent could be directly used to advertise things like SPARC to the right people.
• General survey tool.

Potential problems and difficulties

• Its use as an information gathering tool could be seen negatively.
• Legal issues?
• Publicity. The scholarship has to be made known to the relevant people, and this has to be done in such a way that SI is seen as a reputable institute. However, a scholarship does open up new avenues for publicity.
• Cost and manpower.

Is anyone else doing this?

As with many ideas, we ought to be cautious if we see no one else doing something similar. Indeed, I cannot think of any high school scholarship that is used primarily to collect information for the sponsoring organization (is this really the case?). However, there is good reason for this – no one else is interested in reaching the same group of high school students. SI is the only organization I know of who wants to reach high school students for their research group.

FHI had a competition that could be an attempt to collect information, but I'm not sure.

High school scholarships

It would be wise to consult current high school scholarships, and AoPS has a good list.

Thoughts on the idea of this thread go here.

Check for an AI breakout in a toy model

Without deliberately stacking the deck, setup a situation in which an AI has a clear role to play in a toy model. But make the toy model somewhat sloppy, and give the AI great computer power, in the hope that it will "escape" and achieve its goals in unconventional way. If it doesn't, that useful information; if it does, that's even more useful, and we can get some info by seeing how it did that.

Then instead of the usual "paperclip maximiser goes crazy", we could point to this example as a canonical model of misbehaviour. Not something that is loaded with human terms and seemingly vague sentiments about superintelligences, but more like "how do you prevent the types of behaviour that agent D-f55F showed in the factorising Fibonacci number in the FCS world? After all, interacting with humans and the outside world throws up far more vulnerabilities than the specific ones D-f55F took advantage of in that problem. What are you doing to formally rule out exploitation of these vulnerabilities?"

(if situations like this have happened before, then no need to recreate them, but they should be made more prominent).

Grow the Cognitive Surplus-Powered AI Risks Research Community

Luke already identified funding Friendliness-related academic papers as a way to purchase AI risk reduction. But being an academic is not a necessary condition for producing useful research.

There already seems to be a community of amateurs who think about decision theory issues and other issues related to Friendliness in their spare time on Less Wrong. Some simple ideas that might grow/encourage this community: Offer cash prizes for useful insights, or feature useful insights on the Singularity Institute blog. Respond to the work of amateurs, as a way of recognizing its legitimacy. Improve recommended reading lists for understanding/contributing to FAI related topics and increase their visibility. Sponsor a "FAI programmer wannabe" mailing list/reading group.

Publish AI Research Guidelines

The Singularity Institute has argued that the pace of friendliness research should outpace that of general purpose AI research if we want to have a positive singularity. But not all general purpose AI research is created equal. Some might be relevant to friendliness. Some might be useful in architecting a FAI. And some might only be useful for architecting a potential UFAI.

It seems possible that persuading AI researchers to change the course of their research is much easier than persuading them to quit altogether. By publishing a set of research recommendations, SI could potentially shape whatever AI research is done towards maximally Friendly ends.

Costs: Someone would have to understand all major AI research fronts in enough depth to evaluate their relevance for Friendliness, and have a good idea of the problems that need to be solved for Friendliness and what form a Friendly architecture might take. Psychological research on persuasion might additionally be beneficial. For example, is it best to explicitly identify lines of research that seem especially dangerous or just leave them out of the document altogether?

Persuade Grantmaking Organizations that Certain Lines of Research are Dangerous

I've never been involved in academia, but my vague idea of how things work is that researchers apply for grants from organizations that sponsor their research. If these organizations could be persuaded to change the criteria they used to assign grants, it's possible the progress of AI research could be shaped.

Hire a High-Profile AI Researcher

SI says they've succeeded in convincing a few high-profile AI researchers that AGI research is dangerous. If one of these researchers could be hired as a SI staff member, they could lend their expertise to the development of Friendliness theories and also enhance SI's baseline credibility in the AI research community in general.

A related idea is to try to get these AI researchers to sign a petition making a statement about AI dangers.

Both of these ideas risk potentially unwanted publicity.

Note that both of these ideas are on SI's radar; I mention them here so folks can comment.

Make a "moral expert system" contest

Have a set of moral dilemmas, and

1) Through an online form, humans say what choice they would make in that situation

2) There's a contest to write a program that would choose like a human in those situations.

(Or alternatively, a program that given some of the choices that a human made, guesses which other choices he made in other situations)

Sponsor a "morality turing test" contest

From Prolegomena to any future artificial moral agent:

A Moral Turing Test (MTT ) might similarly be proposed to bypass disagreements about ethical standards by restricting the standard Turing Test to conversations about morality. If human interrogators ’ cannot identify the machine at above chance accuracy, then the machine is, on this criterion, a moral agent . [...] To shift the focus from conversational ability to action, an alternative MTT could be structured in such a way that theinterrogator ’ is given pairs of descriptions of actual, morally-significant actions of a human and an AMA, purged of all references that would identify the agents. If the interrogator correctly identify es the machine at a level above chance, then the machine has failed the test. A problem for this version of the MTT is that distinguishability is the wrong criterion because the machine might be recognizable for acting in ways that are consistently better than a human in the same situation. So instead, the interrogator might be asked to assess whether one agent is less moral than the other. If the machine is not reported as responding less morally than the human, it will have passed the test. This test is called the ‘comparative MTT’"

The rules may have to be tweaked a bit more, but it sounds like something that might get various AI students or wannabe AI programmers interested in morality.

Lobby the Government for security regulations for AI

Just like there are security regulations on bioengineering, chemicals, nuclear power, weapons, etc. - there could be regulations on AI, with official auditing of risks, etc. This would create more demand for officially recognized "AI Risk" experts; will force projects to pay more attention to those issues (even if it's only for coming up with rationalizations for why their project is safe), etc.

This doesn't have to mean banning "unsafe" research; the existence of a "safe AI" certification means it might become a prerequisite for certain grants, or a marketing argument (even if the security standards for "safe AI" are not sufficient to actually guarantee safety).

Create a Direct Friendly AGI Investment Vehicle

Currently most AGI research is primarily sponsored by private angel investors and VCs towards private ends.

I have savings I would like to invest in friendly AGI, and I don't see a clear vehicle to enable that purchase at the moment. I've brought this up before at LW meetings and believe there is a growing market for such a vehicle.

The idea could take the form of a public accessible angel group or community chest type organization that allows investors to get exposure to a large pool of relatively safe AGI designs. The SIAI could take a lead role as a safety rating agency, connecting to John Maxwell's Safety Guideline ideas elsewhere in this thread.

AGI research is taking off regardless, the idea here is to differentially accelerate the safer designs.

Make a "Friendly AI" programming challenge in a toy simulated society

(warning: this is pretty half-baked)

Step one (to prepare the contest) would be making interesting simulated societies, in which agents evolve social norms.

A simulated society would be a population of agents, and each turn some of them would be picked randomly to participate in a game, and as a result some agents may be killed, or modified (tagged "evil", "pregnant", "hungry", etc.), or new agents may be created, etc.

Agents would each have a set of "norms" ("in situation X, doing Y is good") that together would effectively amount to the agent's utility function. These norms would also work as the agents' genome, so when new agents are created, their norms are derived from those of their parent(s).

In effect, this would amount to evolving the most appropriate utility function for the simulation's specific rules. Several sets of rules could be devised to make different interesting societies.

Step two would be to have a contest where programmers have to write a "God" program that would be analogous to a superhuman AI in our world. I'm not quite sure of the best way to do that, but in all cases the program should be given some of the evolved norms as input.

One possibility would be to have the "God" program be a utility function, after which the whole simulated world is optimized according to that utility function.

Another would be to have the "God" program be an agent that participates in all games with many more choices available.

A twist could be that the "God" program is given only the norms on which some agents disagree, those on which everybody always agree "go without saying" (such as "rocks don't count as sentient beings"). Another is that the same "'God" program would be used in simulations with different rules.

(A big problem with this is that in the simulations, unlike in reality, agents are ontologically basic objects, so a lot of "dreams of friendly AI" would actually work in the simulation. Still, it should be possible to devise a simulation where the "God" program doesn't have access to the agents as ontologically basic objects)

A contest like that may allow people to realize that some of their ideas on value extrapolation etc. do not work.

Someone sent me this anonymous suggestion:

Well, let’s consider AIXI-tl properly, mathematically, without the 'what would I do in it’s shoes’ idiocy and without incompetent “let’s just read the verbal summary”. The AIXI-tl

1: looks for a way to make the button pressed.

2: actually, not even that; it does not relate itself to the representation of itself inside it’s representation of the world, and can’t model world going on without itself. It can’t understand death. It’s internal model is dualist.

It is an AI that won’t stop you from shutting it down. If you try to resolve 2, then you hit another very hard problem, wireheading.

Those two problems naturally stay in the way of creation of AI that kills everyone, or AI that wants to bring about heaven on earth, but they are entirely irrelevant to the creation of useful AI in general. Thus the alternative approach to AI risk reduction is to withdraw all funding from SI or any other organization working on philosophy of mind for AI, as those organizations create the risk of AGI that solves those two very hard problems which prevent arbitrary useful AI from killing us all.