Can you describe what changed / what made you start feeling that the problem is solvable / what your new attack is, in short?

Value extrapolation is thus necessary for AI alignment. It is also almost [LW · GW] sufficient [LW · GW], since it allows AIs to draw correct conclusions from imperfectly defined human data.

I am missing something... The idea of correctly extrapolating human values is basically the definition of the Eliezer's original proposal, CEV. In fact, it's right there in the name. What is the progress over the last decade?

Aligned AI is a benefit corporation dedicated to solving the alignment problem

Is this a UK or US public-benefit corporation?

Who are the other founders?

Who and how much are you capitalized for?

Best of luck!

Hmm, the only overlap I can see between your recent work and this description (including optimism about very-near-term applications) is the idea of training an ensemble of models on the same data, and then if the models disagree with each other on a new sample, then we're probably out of distribution (kinda like the Yarin Gal dropout ensemble thing and much related work).

And if we discover that we are in fact out of distribution, then … I don't know. Ask a human for help?

If that guess is at all on the right track (very big "if"!), I endorse it as a promising approach well worth fleshing out further (and I myself put a lot of hope on things in that vein working out [LW · GW]). I do, however, think there are AGI-specific issues to think through, and I'm slightly worried that y'all will get distracted by the immediate deployment issues and not make as much progress on AGI-specific stuff. But I'm inclined to trust your judgment :)

Unmentioned but large comparative advantage of this: it's not based in the Bay Area.

The typical alignment pitch of: "Come and work on this super-difficult problem you may or may not be well suited for at all" Is a hard enough sell for already-successful people (which intelligent people often are) without adding: "Also you have to move to this one specific area of California which has a bit of a housing and crime problem and very particular culture"

If you think this is financially viable, then I'm fairly keen on this, especially if you provide internships and development opportunities for aspiring safety researchers.

So we need a way to have alignment deployed throughout the algorithmic world before anyone develops AGI. To do this, we'll start by offering alignment as a service for more limited AIs.

I'm tentatively fairly excited about some version of this, so I'll suggest some tweaks that can hopefully be helpful for your success (or for the brainstorming of anyone else who's thinking about doing something similar in the future).

We will refine and develop this deployment plan, depending on research results, commercial opportunities, feedback, and suggestions.

I suspect there'd be much better commercial/scaling opportunities for a somewhat similar org that offered a more comprehensive, high-quality package of "trustworthy AI services"--e.g., addressing bias, privacy issues, and other more mainstream concerns along with safety/alignment concerns. Then there'd be less of a need to convince companies about paying for some new service--you would mostly just need to convince them that you're the best provider of services that they're already interested in. (Cf. ethical AI consulting companies that already exist.)

(One could ask: But wouldn't the extra price be the same, whether you're offering alignment in a package or separately? Not necessarily--IP concerns and transaction costs incentivize AI companies to reduce the number of third parties they share their algorithms with.)

As an additional benefit, a more comprehensive package of "trustworthy AI services" would be directly competing for consumers with companies like the AI consulting company mentioned above. This might pressure those companies to start offering safety/alignment services--a mechanism for broadening adoption that isn't available to an org that only provides alignment services.

[From the website] We are hiring AI safety researchers, ML engineers and other staff.

Relatedly to the earlier point, given that commercial opportunities are a big potential bottleneck (in other words, given that selling limited alignment services might be as much of a communications and persuasion challenge as it is a technical challenge), my intuition would be to also put significant emphasis into hiring people who will kill it at the persuasion: people who are closely familiar with the market and regulatory incentives faced by relevant companies, people with sales and marketing experience, people with otherwise strong communications skills, etc. (in addition to the researchers and engineers).

Given that there's a lot of variation in how humans extrapolate values, whose extrapolation process do you intend to use?

To do this, we'll start by offering alignment as a service for more limited AIs. Value extrapolation scales down as well as up: companies value algorithms that won't immediately misbehave in new situations, algorithms that will become conservative and ask for guidance when facing ambiguity.

What are examples of AIs you think you can currently align and how much (order of magnitude, say) would it cost to have you align one for me? If I have a 20B parameter language model, can you align it for me?

For some time, I have planned to make a post calling for more people to actually try to solve the problem of alignment. I haven't studied Stuart's work in detail (something to be rectified soon), I always say June Ku's metaethical.ai is the most advanced scheme we have, but as an unapologetic fan of CEV, this talk of value extrapolation seems on the right track. I do wonder to what extent a solution to alignment for autonomous superhuman AI can lead (in advance) to spinoff for narrower and less powerful systems - superhuman alignment seems to require a determination of the full "human utility function", or something similar; I suppose the extrapolation part might be relevant for lesser AI, even if the full set of human values are not - but we shall learn more as Stuart's scheme unfolds.

I will add that I am personally interested in contributing to this kind of research (paid work would be most empowering, but absent that, I will still keep doing what I can, when I can, until we run out of time), but my circumstances are a little unusual, and might be incompatible with what some organizations require. So for now I'll just mention my interest.

I'm encouraged by your optimism, and wish you the best of luck (British, and otherwise), but I hope you're not getting much of your intuition from the "Humans have demonstrated a skill with value extrapolation..." part. I don't think we have good evidence for this in a broad enough range of circumstances for it to apply well to the AGI case.

We know humans do pretty 'well' at this - when surrounded by dozens of other similar agents, in a game-theoretical context where it pays to cooperate, it pays to share values with others, and where extreme failure modes usually lead to loss of any significant power before they can lead to terrible abuse of that power.

Absent such game-theoretic constraints, I don't think we know much at all about how well humans do at this.

Further, I don't think I know what it means to do value extrapolation well - beyond something like "you're doing it well if you're winning" (what would it look like for almost all humans to do it badly?). That's fine for situations where cooperation with humans is the best way to win. Not so much where it isn't.

But with luck I'm missing something!

I’m really glad to see this. I can’t say I fully grasp your particular approach, but what you’ve written about model fragments has really resonated.

My intuition around value extrapolation is that if we extrapolate the topic itself it’ll eventually turn into creating fine models of nervous system dynamics. Will be curious to see how your work intersects and what it assumes about neuroscience, and also what sort of neuroscience progress you think might make your work easier.

Good luck!

All the best on this new venture!

Regarding 'value extrapolation', I wrote a little on grounding the acquisition of the right priors by 'learning to value learn' last year in Motivations, Natural Selection, and Curriculum Engineering [AF · GW] (section Transmissible Accumulation). It basically just has seeds of ideas, but you may be interested.

I think value extrapolation is more tractable than many assume, even for vary powerful systems. I think this because I expect AI systems to strongly prefer a small number of general explanations over many shallow explanations [LW · GW]. I expect such general explanations for human values are more likely to extend to unusual situations than more shallow explanations.

One approach that seems really underexplored is to directly generate data on how human preferences extend to extreme situations or very capable AIs. OpenAI was able to greatly improve the alignment of current language models by learning a reward model from text examples of current language models following human instructions, ranked by how well the AI’s output followed the human’s instruction. We should be able to generate a similar values data set, but for much AIs much stronger than current language models. See here for a more extended discussion [LW · GW].

To do this, we'll start by offering alignment as a service for more limited AIs.

Interesting move! Will be interesting to see how you will end up packaging and positioning this alignment as a service, compared to the services offered by more general IT consulting companies. Good luck!

How do you know when you have solved the value extrapolation problem? 

One hypothesis I have for what you might say is something like "a training scheme solves the value extrapolation problem when the sequence of inputs that will be seen in deployment by the AI produced by that training scheme leads to outputs which lead to positive outcomes by human lights" though from what I can tell, that's basically the same as having a training scheme that leads to an "impact aligned [LW · GW]" AI*.

If it isn't this, how is your answer different?

*[ETA: the definition of impact alignment that Evan gives in the linked post technically only refers to an AI "which doesn't take actions that we would judge to be bad/problematic/dangerous/catastrophic," but in my comment above, I meant to refer to what I think is the more relevant property for an AI to have, which I'll call (impact aligned)_Jack: an agent is (impact aligned)_Jack to the degree that, by human lights, it doesn't take bad actions and does take good actions." I think that this is more relevant because Evan's definition doesn't distinguish between a rock and an intuitively aligned AI.]