Has anyone actually tried to convince Terry Tao or other top mathematicians to work on alignment?

Thank you for posting about this here so that you can get feedback, and so that other people can know how much people are doing this sort of thing (and by the same token it could be good for people who've already done this sort of thing to say so).

I have a bit of a sinking feeling reading your draft; I'll try to say concrete things about it, but I don't think I'll capture all of what's behind the feeling. I think part of the feeling is about, this just won't work.

Part of it is like, the email seems to come from a mindset that doesn't give weight to curiosity and serious investigation (what Tao does with his time).

I know that to you it isn't the most interesting problem to think about[1] but it really, actually is a very very important and urgent completely open problem. It isn’t simply a theoretical concern, if Demis’ predictions of 10 to 20 years to AGI are anywhere near correct, it will deeply affect you and your family (and everyone else).

I think there's a sort of violence or pushiness here that's anti-helpful. It doesn't acknowledge that Tao doesn't have good reason to trust your judgements about what's "very very important and urgent", and people who go around telling other people what things are "very very important and urgent" in contexts without trust in judgement are often trying to coerce people into doing things they don't want to do. It doesn't acknowledge that people aren't utility maximizing machines, but instead have momentum and joy and specialization and context. (Not to say that Tao doesn't deserve to be informed about the consequences of events happening in the world and his possible effect on those consequences, and not to say that stating beliefs is bad, and not to say that Tao might just be curious and learn about AI risk if it's shown to him in the right way.)  

Another thing is the sources recommendation. The links should be to technical arguments about problems in AI alignment and technical descriptions of the overall problem, the sort of thing X-risk and AI-risk thinkers say to each other, not to  material prepared with introductoriness in mind.

It is not once but twice that I have heard leaders of AI research orgs say they want you to work on AI alignment.

This is kind of weird and pushy. On the face of it, it looks like either you're confused and think that big high status people to you are also big high status people to Tao and therefore should be able to give him orders about what to work on, and that Tao is even the sort of entity that takes orders; or at least, it looks like you yourself are trying to take orders from big high status people, propagating perceived urgency from them to whoever else, without regard to individual agents's local/private information about what's good for them to do. Like, it looks like you got scared, flailed and grasped for whatever the high status people said they think might be cool, and then wanted to push that. (I'm being blunt here, but to be clear, if something like this is happening, that's very empathizable-with; I don't think "you're bad" or anything like that, and doing stuff that seems like it would have good consequences is generally good.)

If you are ever interested you can start by reading

This is sort of absurd: 1. if Tao were interested, he could likely have lots of conversations with competent AI alignment thinkers, which would be a much better use of his time, and 2. frankly, it seems like you're posturing as someone giving orders to Tao.

I think that to a non-trivial extent, we have a limited supply of such efforts. The more times Terry has been contacted about this, the less likely he is to respond positively.

And to some extent I suppose this is true about reaching out to such figures more generally. Ie. maybe word gets out that we've been doing such outreach and by the time we contact John Doe, even if it's our first time contacting John Doe, we may have exhausted our supply of John Doe's patience.

So then, I don't think such an action is as low cost as it may seem. It costs more than the time it takes to write the email.

What makes more sense to me is to try to traverse through social networks and reach him that way. Figure out which nodes are close to him who he listens to. Note that they might be bloggers like Scott Alexander or someone like Dan Luu. From there think about which of those nodes make sense to pursue. Maybe one, maybe multiple. Then backtrack and think about how we can utilize our current connections to reach those nodes.

I also think it'd be worth brainstorming more creative solutions with a bunch of yoda timers [? · GW]. I'll try one right now.

• Billboard ad.
• Well put together video.
• Figure out what sort of people have the skillset for this and pay them as consultants.
• Be like Hermione and read books to educate ourselves first. Especially because it signals competence. Signals of incompetence could make someone like Terry be a lot more likely to reject us.
• Research more directly whether stuff like this has been done before. Talk to people who have done it to see what advice they have.
• Put a bounty on it. $50k to get a response from him.
• Attach money to it. $50k to Terry to sit down and discuss this.
• Start with (way) less prestigious people. Presumably they're easier to reach and perhaps to convince. Then with a bunch of them working on it, higher prestige people would start to notice and be more easily convinced.

Some of these ideas seem pretty solid. My sense is that the best path forward is:

1. More brainstorming from the community.
2. Get in touch with various organizations (MIRI, CEA...) to see where they are at with this stuff.
3. Education. Figure out what academic fields/topics are relevant. Learn about them. Probably do some sort of write-up. Nothing too crazy, but I think the low hanging fruits should be addressed.
4. Decide on a path forward. I suspect that the initiatives should come from an organization like MIRI or CEA, because I assume people like Terry would be more likely to respond to representatives of decently prestigious organizations.

This is a little rambly. Sorry. I'll end here.

Hi, long-time lurker but first-time poster with a background in math here. I personally agree that it would be a good idea if we were to at least try to get some extremely talented mathematicians to think about alignment. Even if they decide not to, it might still be interesting to see what kinds of objections they have to working on it (e.g. is it because they think it's futile and doomed to failure, because they think AGI is never going to happen, because they think alignment will not be an issue, because they feel they have nothing to contribute, or because it's not technically interesting enough?).

However, I would also like to second TekhneMakre's concerns about the format and content of the email. If you sample some comments on posts on Terry Tao's blog, you will find that there are a number of commenters who would probably best be described as cranks who indefatigably try to convince Terry that their theories are worth paying attention to, that Terry is currently not wisely spending his time, etc. He (sensibly) ignores these comments, and has probably learned for the sake of sanity not to engage with anyone who seems to fit this bill. I am concerned that the email outlined in your post will set off the same response and thus be ignored. AI safety is still a rather fringe idea amongst academics, at least partly because it is speculative and lacking concreteness. It took me years as an academic-adjacent person to be even somewhat convinced that it could be a problem (I still am not totally convinced, but I am convinced it is at least worth looking into). I do not think an email appealing to emotion and anecdotes is likely to convince someone from that background encountering this problem.

I have three alternative suggestions; I'm not sure how good they are, so take them each with a grain of salt:

Firstly, note that Scott Aaronson has said here https://scottaaronson.blog/?p=6288#comment-1928043 that he would provisionally be willing to think about alignment for a year. This seems like it would have several advantages (1) He has already signaled interest, provisionally, so it would be easier to convince him that it might be worth working on, (2) He is already acquainted with many of the arguments for taking AGI seriously, so could start working on the problem more immediately, (3) He is well acquainted with the rationalist community and so would not be put off by rationalist norms or affiliated ideas such as EA (which I believe accounts for the skepticism of at least some academics), (4) Scott's area of work is CS theory, which seems like it would be more relevant to alignment than Tao's fields of interest. 

Secondly, there are some academics who take AI safety arguments seriously. Jacob Steinhardt comes to mind, but I'm sure there are a decent number of others, especially given recent progress on AI. If these academics were to contact other top academics asking them to consider working on AI safety, the request would come across as much more credible. They would also know how to frame the problem in such a way to pique the interest of top mathematicians/computer scientists. 

Thirdly, note that there are many academics who are open to working on big policy problems that do not directly concern their primary research interests. Terry Tao, I believe, is one of them, as evidenced by https://newsroom.ucla.edu/dept/faculty/professor-terence-tao-named-to-president-bidens-presidents-council-of-advisors-on-science-and-technology . I'm not sure to what extent this is an easier problem or a desirable course of action, but if you could convince some people in politics that this problem is worth taking seriously, it is possible that the government might directly ask these scientists to think about it.  

This last point is not a suggestion, but I would like to add one note. Eliezer claims that he was told that you cannot pay top mathematicians to work on problems. I believe this is somewhat false. There are many examples of very talented professors and PhD students leaving academia to work at hedge funds. One example is Abhinav Kumar, who a few years ago was 1 of 4 coauthors on a paper solving the long-open problem on optimal sphere packings in 24 dimensions. He left an Associate Professorship at MIT to work at Renaissance Technologies (a hedge fund). Not exactly in the same vein, but Huawei has recruited 4 Fields medalists to work with them (e.g. see https://www.ihes.fr/en/laurent-lafforgue-en/ for one example) although I'm not certain whether they are working on applied problems. I cannot say whether money is a motivating factor in any given one of these cases, but there are more examples like this, and I think it is fair to say that at least some substantial fraction of all such people involved might have been motivated at least partly by money.

My concern is less your email, and more the precedent. Having the rationality community model and encourage obviously undesired forms of contact with high-prestige figures seems like it could lead to intrusions of privacy. One person sending an email is ignorable. If emails, phone calls, unsolicited office visits, etc. start piling up under the banner of “AI risk,” it could feel quite invasive to those on the receiving end. My concern in particular is that people doing as you’re doing may not have the capacity to coordinate their actions. We may not even know whether or how much “randomly emailing Terry Tao about X risk” is going on.

I think Maxim Kontsevich might be a better candidate for an elite mathematician to try to recruit. Check out this 2014 panel [LW · GW] with him, Tao and some other eminent mathematicians -- he alone said that he thought HLAI(in math) is plausible in our lifetimes, but also that working on it might be immoral(!) He also mentioned an AI forecast by Kolmogorov that I had never heard of before, so it seems he has some pre-existing interest in the area.

Um. If you want to convince a mathematician like Terry Tao to be interested in AI alignment, you will need to present yourself as a reasonably competent mathematician or related expert and actually formulate an AI problem in such a way so that someone like Terry Tao would be interested in it. If you yourself are not interested in the problem, then Terry Tao will not be interested in it either.

Terry Tao is interested in random matrix theory (he wrote the book on it), and random matrix theory is somewhat related to my approach to AI interpretability and alignment. If you are going to send these problems to a mathematician, please inform me about this before you do so.

My approach to alignment: Given matrices $A_1,\dots ,A_r;B_1,\dots ,B_r$, define a superoperator $\Gamma (A_1,\dots ,A_r;B_1,\dots ,B_r)$ by setting 

$\Gamma (A_1,\dots ,A_r;B_1,\dots ,B_r)\left(X\right)={\sum }_{k=1}^r{A}_{k}X{B}_k^{\ast }$, and define $\Phi (A_1,\dots ,A_r)=\Gamma (A_1,\dots ,A_r;A_1,\dots ,A_r)$. Define the $L_2$-spectral radius of $A_1,\dots ,A_r$ as ${\rho }_2(A_1,\dots ,A_r)=\rho \left(\Phi \right(A_1,\dots ,A_r){)}^{1/2}$. Here, $\rho \left(A\right)={lim}_{n\to \infty }\parallel A^n{\parallel }^{1/n}$ is the usual spectral radius.

Define ${\rho }_{2,d}^K(A_1,\dots ,A_r)=max\{\frac{\rho \left(\Gamma \right(A_1,\dots ,A_r;X_1,\dots ,X_r\left)\right)}{{\rho }_2(X_1,\dots ,X_r)}:X_1,\dots ,X_r\in M_d(K\left)\right\}$. Here, $K$ is either the field of reals, field of complex numbers, or division ring of quaternions.

Given matrices $A_1,\dots ,A_r;B_1,\dots ,B_r$, define

$\parallel (A_1,\dots ,A_r)\simeq (B_1,\dots ,B_r)\parallel =\frac{\Gamma (A_1,\dots ,A_r;B_1,\dots ,B_r)}{{\rho }_2(A_1,\dots ,A_r){\rho }_2(B_1,\dots ,B_r)}$. The value $\parallel (A_1,\dots ,A_r)\simeq (B_1,\dots ,B_r)\parallel$ is always a real number in the interval $[0,1]$ that is a measure of how jointly similar the tuples $(A_1,\dots ,A_r),(B_1,\dots ,B_r)$ are. The motivation behind ${\rho }_{2,d}^K(A_1,\dots ,A_r)$ is that $\frac{{\rho }_{2,d}^K(A_1,\dots ,A_r)}{{\rho }_2(A_1,\dots ,A_r)}$ is always a real number in $[0,1]$ (well except when the denominator is zero) that measures how well $A_1,\dots ,A_r$ can be approximated by $d\times d$-matrices. Informally, $\frac{{\rho }_{2,d}^K(A_1,\dots ,A_r)}{{\rho }_2(A_1,\dots ,A_r)}$ measures how random $A_1,\dots ,A_r$ are where a lower value of $\frac{{\rho }_{2,d}^K(A_1,\dots ,A_r)}{{\rho }_2(A_1,\dots ,A_r)}$ indicates a lower degree of randomness.

A better theoretical understanding of ${\rho }_{2,d}^K(A_1,\dots ,A_r)$ would be great. If $X_1,\dots ,X_r\in M_d\left(K\right)$ and $\frac{\rho \left(\Gamma \right(A_1,\dots ,A_r;X_1,\dots ,X_r\left)\right)}{{\varphi }_2(X_1,\dots ,X_r)}$ is locally maximized, then we say that $(X_1,\dots ,X_r)$ is an LSRDR of $(A_1,\dots ,A_r)$. Said differently, $(X_1,\dots ,X_r)\in M_d\left(K\right)$ is an LSRDR of $(A_1,\dots ,A_r)$ if the similarity $\parallel (A_1,\dots ,A_r)\simeq (X_1,\dots ,X_r)\parallel$ is maximized.

Here, the notion of an LSRDR is a machine learning notion that seems to be much more interpretable and much less subject to noise than many other machine learning notions. But a solid mathematical theory behind LSRDRs would help us understand not just what LSRDRs do, but the mathematical theory would help us understand why they do it.

Problems in random matrix theory concerning LSRDRs:

1. Suppose that $U_1,\dots ,U_r$ are random matrices (according to some distribution). Then what are some bounds for ${\rho }_{2,d}^K(U_1,\dots ,U_r)$.
2. Suppose that $U_1,\dots ,U_r$ are random matrices and $A_1,\dots ,A_r$ are non-random matrices. What can we say about the spectrum of $\Gamma (A_1,\dots ,A_r;U_1,\dots ,U_r)$? My computer experiments indicate that this spectrum satisfies the circular law, and the radius of the disc for this circular law is proportional to ${\rho }_2(A_1,\dots ,A_r)$, but a proof of this circular law would be nice.
3. Tensors can be naturally associated with collections of matrices. Suppose now that $U_1,\dots ,U_r$ are the matrices associated with a random tensor. Then what are some bounds for ${\rho }_{2,d}^K(U_1,\dots ,U_r)$.

P.S. By massively downvoting my posts where I talk about mathematics that is clearly applicable to AI interpretability and alignment, the people on this site are simply demonstrating that they need to do a lot of soul searching before they annoy people like Terry Tao with their lack of mathematical expertise.

P.P.S. Instead of trying to get a high profile mathematician like Terry Tao to be interested in problems, it may be better to search for a specific mathematician who is an expert in a specific area related to AI alignment since it may be easier to contact a lower profile mathematician, and a lower profile mathematician may have more specific things to say and contribute. You are lucky that Terry Tao is interested in random matrix theory, but this does not mean that Terry Tao is interested in anything in the intersection between alignment and random matrix theory. It may be better to search harder for mathematicians who are interested in your specific problems.

P.P.P.S. To get more mathematicians interested in alignment, it may be a good idea to develop AI systems that behave much more mathematically. Neural networks currently do not behave very mathematically since they look like the things that engineers would come up with instead of mathematicians. 

P.P.P.P.S. I have developed the notion of an LSRDR for cryptocurrency research because I am using this to evaluate the cryptographic security of cryptographic functions.

I have heard about the thing where you commit to a $100m reward for any ML or mathmetician who solves alignment, and simultaneously pay 100 top ML and mathmeticians $1m over the course of a year to do nothing but pursue a solution to alignment (pursuing the bounty in the process). Even if all 100 of them fail, you still selected the best 100 out of every mathmetician who applied for those positions, so a large proportion of them might pursue the problem on their own afterwards in pursuit of the ongoing $100 million bounty. One way or another, many of these influential people will be convinced that the problem is significant and tell their friends, or even contract their friends as consultants to help with the problem.

There's plenty of trust issues, going both ways, but I'm not a grantmaker or lawyer and I think some smart, experienced people could probably figure out how to mitigate most of them.

I really want this to happen.

And why stop at Terry Tao? We could also email other top mathematicians and physicists.

I would put that in Google doc, it will be easier to suggest changes etc