Pay Risk Evaluators in Cash, Not Equity

I raised a similar proposal to various people a while ago.

The strongest objection I'm aware of is something like:

You actually want evaluators to have as much skin in the game as other employees so that when they take actions that might shut the company down or notably reduce the value of equity, this is a costly signal.

Further, it's good if evaluators are just considered normal employees and aren't separated out in any specific way. Then, other employees at the company will consider these evaluators to be part of their tribe and will feel some loyalty. (Also reducing the chance that risk evaluators feel like they are part of rival "AI safety" tribe.) This probably has a variety of benefits in terms of support from the company. For example, when evaluators make a decision with is very costly for the company, it is more likely to respected by other employees.

A key question is how much of the problem comes from evaluators within the company sounding the alarm versus the rest of the company (and the world) not respecting this alarm. And of course, how much the conflict of interest (and tribal-ish affiliation with the company) will alter the judgment of evaluators in a problematic way.

If I was confident that all risk evaluators were incorruptible, high integrity, total altruists (to the world or something even more cosmopolitan), then I think the case for them getting compensated normally with equity is pretty good. This would allow them to perform a costly signal later and would have other benefits. Though perhaps the costly signal is less strong if it is clear these people don't care about money (but this itself might lend some credibility).

Given my understanding of the actual situation at AI companies, I think AI companies should ideally pay risk evaluators in cash.

I have more complex views about the situation at each of the different major AI labs and what is highest priority for ensuring good risk evaluations.

Your third paragraph mentions "all AI company staff" and the last refers to "risk evaluators" (i.e. "everyone within these companies charged with sounding the alarm"). Are these groups roughly the same or is the latter subgroup significantly smaller?

It's a good point that this is an extreme conflict of interest. But isn't one obvious reason why these companies design their incentive structures this way because they don't have the cash lying around to "simply" compensate people in cash instead?

E.g. here's ChatGPT:

Q: Why do companies like OpenAI partly pay their employees in equity?

ChatGPT: Companies like OpenAI partly pay employees in equity to align the employees' incentives with the company's long-term success. By offering equity, companies encourage employees to take ownership of the business's outcomes, fostering a sense of responsibility and motivation to contribute to its growth.

Additionally, equity compensation can attract top talent by offering potential future financial rewards tied to the company’s success, especially for startups or high-growth firms. It also helps conserve cash, which is valuable for companies in early stages or those heavily investing in research and development.

It would indeed be good if AI safety research was paid for in cash, not equity. But doesn't this problem just reduce to there being far less interest in (and cash for) safety research relative to capability research than we would need for a good outcome?

Early discussion of AI risk often focused on debating the viability of various elaborate safety schemes humanity might someday devise—designing AI systems to be more like “tools” than “agents,” for example, or as purely question-answering oracles locked within some kryptonite-style box. These debates feel a bit quaint now, as AI companies race to release agentic models they barely understand directly onto the internet.

Why do you call current AI models "agentic"? It seems to me they are more like tool AI or oracle AI...

The LessWrong Review [? · GW] runs every year to select the posts that have most stood the test of time. This post is not yet eligible for review, but will be at the end of 2025. The top fifty or so posts are featured prominently on the site throughout the year.

Hopefully, the review is better than karma at judging enduring value. If we have accurate prediction markets on the review results, maybe we can have better incentives on LessWrong today. Will this post make the top fifty?

I think you're right that the incentive structure around AI safety is important for getting those doing the work to do it as well as they can. I think there might be something to be said for the suggestion of moving to a cash payment over equity but think that needs a lot more development.

For instnace, if everyone is paid up front for the work they are doing to protect the world from some AI takeover in the future, then they are no longer tied to that future in terms of their current state. That might not produce any better results than equity that could decline in value in the future.

Ultimately the goal has to be that those able to and doing the work have a pretty tight personal interests stance on future state of AI. It might even be the case that such a research effort alignment is only loosely connected to compensation. 

Additionally, as you note, it's not entirely those working to limit a bad outcome for humans in general from AGI but also what the incentives are for the companies as a whole. Here I think the discussion regarding AI liabilities and insurance might matter more. Which also opens up a whole question about corporate law. Years ago, pre 1930s, banking law used to hold the bankers liable for twice the losses from bank failures to make them better at risk management with other peoples' money. That seems to have been a special case that didn't apply to other businesses even if they were largely owned by outsiders. Perhaps making those who are in conrtol of AI development and deplyment, or are largely the ones financing the efforts, personally responsibile might be a better incentive structure. 

All of these are difficult to work though to get a good, and fair, structure in place. I don't think any one approach will ultimately be the solution but all or some combination of them might be. But I also think it's a given that the risk will always remain. So figuring out just what level of risk as acceptable is also needed, and problematic in its own way.