I agree. I imagine a disagreement where the person says "you can't prove my proposal won't work" and the alignment judge says "you can't prove your proposal will work", and then there's some nasty, very-hard-to-resolve debate about whether AGI is dangerous by default or not, involving demands for more detail, and then the person says "I can't provide more detail because we don't have AGI yet", etc.

I think of the disagreement between Paul and Eliezer about whether IDA was a promising path to safe AGI (back when Paul was more optimistic about it); both parties there were exceptionally smart and knowledgeable about alignment, and they still couldn't reconcile.

Giving people a more narrow problem (e.g. ELK) would help in some ways, but there could still be disagreement over whether solving that particular problem in advance (or at all) is in fact necessary to avert AGI catastrophe.

I don't think well specified problem is the way to go. 

Instead chose a group of judges, and if the majority of them think something is progress, then that counts a winn. This are the rules, and no argument afterwards.

But to make this fair, the judges should be available to discuss possible research directions, and what they would and would not concider progress, before and during the 3 months of reaserch. Or if that's not possible, due to the scale, maybe some group of people who are good at predicting what the real judges would say.

Besides, giving out the money, and not stay in touch during the 3 months, seems like a bad idea anyway.

My yes-and thought here is "okay, can we do this in a way that the various labs will feel is exciting, or win-win?". Rather than making the deal with individuals, somehow frame it as a partnership with the companies.

Leadership will by default be angry and work hard to discredit you and perhaps threaten to fire anyone who accepts your deal.

"Why is management so insistent I don't go do this alignment thing if it won't actually change my view?" 

This sounds like an wonderful way to invoke the Streisand effect and cause an internal political upheaval. I would love to see DeepMind leadership overreact to this kind of outreach and attempt to suppress a bunch of Google AI researcher's moral compunctions through heavy-handed threats. Just imagine someone doing this if Google researchers had concerns about racial bias in their algorithms. It really seems like the dream scenario.

Thinking out loud about how to address this issue:

It doesn't have to be the same 3 months for everyone. You could stagger this over the course of 2 years, with an 8th of the relevant researchers taking a sabbatical to give this challenge a shot, at any given time.

This also means that they can build on each other's work, in serial, if they're in fact making progress, and can see how others have failed to make progress, if they're not.

Maybe alternatively a big EA actor could just buy some big AGI labs?

That wouldn't mess up schedules that badly.

I expect this proposal would be more likely to receive funding, though normally in funding applications you submit different proposals with various different levels.

For the record I think this would also be valuable! If as an alignment researcher your arguments don't survive the scrutiny of skeptics, you should probably update away from them. I think maybe what you're highlighting here is the operationalization of "shoot down", which I wholeheartedly agree is the actual problem.

Re: the quantities of funding, I know you're being facetious, but just to point it out, the economic value of of "capabilities researchers being accidentally too optimistic about alignment" and "alignment researchers being too pessimistic about alignment" are asymmetric.

Hmmm, I wonder if there's a version like this that would actually be decent at 'getting people to grapple with' the ideas? Like, if you got Alignment Skeptic Alice to judge the ELK prize submissions with Paul Christiano and Mark Xu (presumably by paying her a bunch of money), would Alice come away from the experience thinking "actually ELK is pretty challenging" or would she come away from it thinking "Christiano and Xu are weirdly worried about edge cases that would never happen in reality"?

God, absolutely, yes, do I get to talk to the sceptic in question regularly over the three months? 

Given three months of dialogue with someone who thinks like me about computers and maths, and where we both promise to take each other's ideas seriously, if I haven't changed his mind far enough to convince him that there are serious reasons to be scared, he will have changed mine. 

I have actually managed this with a couple of sceptic friends, although the three months of dialogue has been spread out over the last decade!

And I don't know what I'm talking about. Are you seriously saying that our best people can't do this?! Eliezer used to make a sport of getting people to let him out of his box. And has always been really really good at explaining complicated thoughts persuasively.

Maybe our arguments aren't worth listening to. Maybe we're just wrong.

Give me this challenge!! Nobody needs to pay me, I will try to do this for fun and curiosity with anyone on the other side who is open-minded enough to commit to regular chats. An hour every evening?

In person would be better, so Cambridge or maybe London? I can face the afternoon train for this.

I don't think that's an unpopular opinion! That's just true.

Money-wise, we have a Cassandra problem. No one believes us. So the question is, how can we get everyone competent enough to end the world to realize that they're about to end the world? Is it possible to use money to do this? Apparently there is money, but not so much time.

just like parenting

Strongly encourage you to make this comment its own post; the work of laying out “here’s SOTA and here’s why it doesn’t work” in precise and accessible language is invaluable to incentivizing outsiders to work on alignment.

They were wanking on endlessly about quantum consciousness and how any agent sufficiently intelligent to escape its box would be intelligent enough to figure out that the utility function we'd given it wasn't actually what we wanted.

I've been going on about 'AI will kill us all' to anyone who'll listen for about ten years now. Everyone I know thinks I've got another stupid science-fiction bee in my bonnet to replace my previous idiotic nerd-concerns about overpopulation and dysgenics and grey goo and engineered plagues and factory farming and nuclear accidents and nuclear proliferation and free speech and ubiquitous surveillance and whatever the hell else it was that used to seem important. 

Almost all my friends are graduates of the University of Cambridge, and roughly half of them studied real subjects and can program computers.

Goddamn it, I know actual AI safety researchers, people much cleverer and more capable than me, who just looked at AlphaZero and shrugged, apparently completely unable to see that building a really good general reinforcement learner might have real-world implications beyond chess.

There's some bunch of academic parasites in Cambridge who were originally supposed to be worrying about existential risks and really seemed quite serious at first, but recently they mostly seem interested in global warming and the disparate impact of COVID.

Also, whaddya mean, 15 years ago? IJ Good pointed out that we were all doomed about 70 years ago, and really it's blindingly obvious to anyone who thinks about it, which is where the inspiration for things like the Terminator movies come from. Or Frankenstein, or the Sorcerers' Apprentice, or the Golem or the Monkey's Paw.

You were fucked, and it was knowable that you were fucked, before you were born, and so was I. That's what it's like to be born into a world of lunatics.

But you have a genuine case to hate me in particular, if you like, because I knew. I really did, in spite of the fact that everyone was telling me I was wrong all the time. And I did nothing. Because it didn't look like there was anything to be done and I already knew that I was no good at research.

Learn from me, and use your anger. You may still have 15 years, and more and more people are coming round to the conclusion that the things that are obviously true might actually be true. Go down fighting.

Or not. Getting drunk on a beach and trying to have as much of a life as possible before the end is also a good use of your time. That was the only deal that was ever on offer, really.

This actually brings up an important consideration. It would be bad to incentivize AGI research by paying people who work inside it huge sums, the same way paying slaveowners to free their slaves is a poor abolitionists' strategy. In reality people aren't Homo Economicus, so I think we could work around it, but we'd need to be careful.