Announcement: AI alignment prize round 2 winners and next round

It seems like maybe there should be an archive page for past rounds.

I might post something else later this month, but if not, my submission is my new Prisoners' Dilemma thing [LW · GW].

Congrats everyone! Really good to see this sort of thing happening.

I submit "Shaping Economic Incentives for Collaborative AGI [LW · GW]" for the next prize.

My submission is whitelisting [LW · GW]!

Hey there!

I don't know if you'll consider these reward-function-learning posts as candidates, or whether you'll see them are rephrasing of old ideas. There are some new results in there - the different value functions and the fact that Q-learning agents can learn un-riggable learning processes is new.

https://www.lesswrong.com/posts/55hJDq5y7Dv3S4h49/reward-function-learning-the-value-function

https://www.lesswrong.com/posts/upLot6eG8cbXdKiFS/reward-function-learning-the-learning-process

In any case, thanks for running these competitions!

I made some notes as I read the winner. Is Tom Everitt on here? Should I send these somewhere?

2.2: The description of ξ is a bit odd - it seems like w_ν is your actual prior over models.

2.4: I'm not totally sold on some parts where it seems like you equate agents and policies. I think the thing we want to compare is not the difference in utility given the same policy, but the difference in true utility between policies that are chosen.

If the human and the AI have the same utility-maximizing policy, they can have nonzero "misalignment" if they have different standard deviations in utility. Meanwhile, even if there is zero misalignment on policy π, that might not be the utility-maximizing policy for the AI.

Should footnote 1 read CA rather than MA?

3.5: It seems like an agent is incentivized to create the extra observation systems, and then encode the important facts about the world within delusional high-reward states of the original observation channel, even assuming it still has to use the original observation and action channels. But it might not even require that facade - a corruption-aware system might notice that a policy that involves changing its hardware and how it uses it will get it larger rewards even according to the original utility function over percepts - beneficial corruption, if you will.

5.5: Decoupled data is nice, but I'm concerned that it's not fully addressing the issue it gets brought up to address, which is fundamentally the issue of induction - how to predict reward for novel situations. An agent can learn as much as it wants about human preferences over normal states of the world, you can even tell it "stories" about unseen states, but it can still be misaligned on novel states never mentioned by humans. The issue is not just about the data the agent sees, but about how it interprets it.

5.5.3: For the counterfactual "default" policy, there are still some unsolved problems with matching human intuition. The classic example is an agent whose default policy is no output - and so it thinks that the default state of the world is the programmers acting confused and trying to fix the agent, because that's what happens with the default policy.

5.6: (thinking about fig. 5.2) I'm hesitant to call any of these paths aligned, and I think it has to do with the difference between good agent design and successful learning of human values. This is a bit related to the inductive problem that I don't think decoupled data fully addresses - it's possible for one to code up something that tries to learn from humans, and successfully learns and faithfully maximizes some value function, but that value function is not the human value function. But this problem is quite ill-understood - most current work on it focuses on "corrigibility" in the intuitive sense of being willing to defer to humans even after you think you've learned human preferences.

I'm not sure I see the point to awarding an already-in-the-works 67 page paper that happened to be released at the time of the competition if the goal of the prize is to stimulate AI work that otherwise would not have happened.

I wonder if it's possible to help with this in a tax-advantaged way. Maybe set up donation to MIRI earmarked for this kind of thing.

Congrats to all winners! :-) I've read about half the winning submissions, and am looking forward to looking through the other half.

Hello!

Here are my submissions for this time. They are all strategy related.

The first one is a project for popularisation AI safety topics. This is not a technical text by its content but the project itself is still technological.

https://medium.com/threelaws/proposal-for-executable-and-interactive-simulations-of-ai-safety-failure-scenarios-7acab7015be4

As a bonus I would add a couple of non-technical ideas about possible economic or social partial solutions for slowing down AI race (which would enable having more time for solving the AI alignment) :

https://medium.com/threelaws/making-the-tax-burden-of-robot-usage-equal-to-the-tax-burden-of-human-labour-c8e97df751a1

https://medium.com/threelaws/starting-a-human-self-sufficiency-movement-the-handicap-principle-eb3a14f7f5b3

The latter text is not totally new - it is a distilled and edited version of one of my other old texts, that was originally multiple times longer and had a narrower goal than the new one.

Regards:

Roland

My new submission is "Dangerous AI is possible before 2030" which stable version is here https://philpapers.org/rec/TURPOT-5

The draft was first discussed on LW on April 3.

I also going to submit another text later which I will link here.