It seems to me all the more basic desires ("values"), e.g. the lower layers of Maslow's hierarchy of needs, are mainly determined by heritable factors. Because they are relatively stable across cultures. So presumably you talk about "higher" values being a function of "data sources in the world"? I.e. of nurture rather than nature?

I agree there probably are some heritable values, though my big difference here is that I think that the set of primitive values is quite a bit smaller than you might think.

Though be warned, heritability doesn't actually answer our question, because the way it's interpreted by laymen is pretty wrong:

https://www.lesswrong.com/posts/YpsGjsfT93aCkRHPh/what-does-knowing-the-heritability-of-a-trait-tell-me-in [LW · GW]

I probably should have separated formal ethical theories that people are describing, which you call morals and what their values actually are more.

I was always referring to values when I was talking about morals.

You are correct that someone describing a moral theory doesn't mean that they actually agree with or implement the theory.

I still think that if you had the amount of control over a human that an ML person had over an AI today, you could brainwash them to value ~arbitrary values with a lot of control, and it would be the central technology of political and social situations, which is a lot.

Personal AI Assistant Ideas

When I imagine having a personal AI assistant with approximately current levels of capability I have a variety of ideas of what I'd like it to do for me.

Auto-background-research

I like to record myself rambling about my current ideas while walking my dog. I use an app that automatically saves a mediocre transcription of the recording. Ideally, my AI assistant would respond to a new transcription by doing background research to find academic literature related to the ideas mentioned within the transcript. That way, when I went to look at my old transcript, it would already be annotated with links to prior work done on the topic, and analysis of the overlap between my ideas and the linked literature.

Also, ideally, the transcription would be better quality. Context clues about the topic of the conversation should be taken into account when making guesses about transcribing unclear words. Where things are quite unclear, there should be some sort of indicator in the text of that.

Various voice modes, and ability to issue verbal commands to switch between them

Receptive Listening Voice-mode

Also for when I want to ramble about my ideas, I'd like having my AI assistant act as a receptive listener, just saying things like 'that makes sense', or 'could you explain more about what you mean about <idea>?'. Ideally, this would feel relatively natural, and not too intrusive. Asking for clarification just where the logic didn't quite follow, or I used a jargon term in some non-standard seeming way. The conversation in this mode would be one-sided, with the AI assistant just helping to draw out my ideas, not contributing much. Occasionally it might say, 'Do you mean <idea> in the same sense that <famous thinker> means <similar sounding idea>?' And I could explain similarity or differences.

Question Answering Voice-mode

Basically a straightforward version of the sort of thing Perplexity does, which is try to find academic sources to answer a technical question. I'd want to be able to ask, and get a verbal answer of summaries of the sources, but also have a record saved of the conversation and the sources (ideally, downloaded and saved to a folder). This would be mostly short questions from me, and long responses from the model.

Discussion Voice-mode

More emphasis on analysis of my ideas, extrapolating, pointing out strengths and weaknesses. Something like what I get when discussing ideas with Claude Sonnet 3.5 after having told it to act as a philosopher or scientist who is critically examining my ideas. This would be a balanced back-and-forth, with roughly equal length conversational turns between me and the model.

Coding Project Collaboration

I would want to be able to describe a coding project, and as I do so have the AI assistant attempt to implement the pieces. If I were using a computer, then an ongoing live output of the effects of the code could be displayed. If I were using voice-mode, then the feedback would be occasional updates about errors or successful runs and the corresponding outputs. I could ask for certain metrics to be reported against certain datasets or benchmarks, also general metrics like runtime (or iterations per second where appropriate), memory usage, and runtime complexity estimates.

Anthropic Wishlist
 

Anthropic is currently my favorite model supplier, in addition to being my most trusted and approved of lab in regards to AI safety and also user data privacy. So, when I fantasize about what I'd like future models to be like, my focus tends to be on ways I wish I could improve Anthropic's offering.

Most of these could be implemented as features in a wrapper app. But I'd prefer them to be implemented directly by Anthropic, so that I didn't have to trust an additional company with my data.

In order from most desired to less desired:

1. Convenience feature: Ability to specify a system prompt which gets inserted at the beginning of every conversation. Mine would say something like, "Act like a rational scientist who gives logical critiques. Keep praise to a minimum, no sycophancy, no apologizing. Avoid filler statements like, 'let me know if you have further questions.'
   
    
2. Ability to check up-to-date documents and code for publicly available code libraries. This doesn't need to be a web search! You could have a weekly scraper check for changes to public libraries, like python libraries. So many of the issues I run into with LLMs generating code that doesn't work is because of outdated calls to libraries which have since been updated.
   
    
3. Voice Mode, with appropriate interruptions, tone-of-voice detection, and prosody matching. Basically, like what OpenAI is working on.
   
    
4. Academic citations. This doesn't need to be a web search! This could just be from searching an internal archive of publicly available open access scientific literature, updated once a week or so.
   
    
5. Convenience feature: A button to enable 'summarize this conversation, and start a new conversation with that summary as a linked doc'.
   
    
6. Ability to test out generated code to make sure it at least compiles/runs before showing it to me. This would include the ability to have a code file which we were collaborating on, which got edited as the conversation went on. Instead of giving responses intended to modify just a specific function within a code file, where I need to copy/paste it in, and then rerun the code to check if it works.
   
    
7. Ability to have some kind of personalization of the model which went deeper than a simple system prompt. Some way for me to give feedback. Some way for me to select some of my conversations to be 'integrated', such that the mode / tone / info from that conversation were incorporated more into future fresh conversations. Sometimes I feel like I've 'made progress' in a conversation, gotten into a better pattern, and it's frustrating to have to start over from scratch every time.

I updated the title with one Eliezer seemed fine with (after poking Robby). Not my top choice, but better than the previous one.

I found this a particularly helpful lens. 

I had had individual thoughts like "of course in this case the alien woman is, you know, bad for kidnapping him and torturing him", but the particular ecosystem frame feels probably-useful for generating followup questions. (It's also thematically resonant with the themes in the book!)

This thread did motivate me to add an additional disclaimer to the post.

It seems to me all the more basic desires ("values"), e.g. the lower layers of Maslow's hierarchy of needs, are mainly determined by heritable factors. Because they are relatively stable across cultures. So presumably you talk about "higher" values being a function of "data sources in the world"? I.e. of nurture rather than nature?

Another point I'd like to raise is that values (in the sense of desires/goals) are arguably quite different from morals. First, morals are more general than desires. Extraterrestrials could also come up with a familiar theory of, say, preference utilitarianism, while not sharing several of our desires, e.g. for eating chocolate or for having social contacts. Indeed, theories of ethics like utilitarianism or Kantian deontology "abstract away" from specific desires by coming up with more general principles which are independent of concrete things individuals may want. Second, it is clearly possible and consistent for someone (e.g. a psychopath) to want X without believing that X is morally right. Conversely, philosophers arguing for some theory of ethics don't necessarily adhere perfectly to the principles of this system, in the same way in which a philosopher arguing for a theory of rationality isn't necessarily perfectly rational himself.

I just edited this into the OP.

We'd need to get more quantitative here about how much AI labor we can use for alignment before it's too dangerous, and my answer is that we could get about 1-2 OOMs smarter than humans at inference where we could be confident in using them safely, and IMO close to an arbitrary number of copies of that AI, conditional on good control techniques being used.

To address 2 comments:

and I have high confidence that control measures will not be used consistently and correctly in practice.

Yeah, this seems pretty load-bearing for the plan, and a lot of the reason I don't have probabilities of extinction below 0.1-1% is because I am actually worried about labs not doing control measures consistently.

I assign more moderate probabilities than you do, in that I think both the scenarios of labs not doing control properly and doing control properly are both somewhat plausible to me now, but yeah it would really be high-value for labs to prepare themselves to do control work properly.

To address this:

I just don't think that extends to ASI

Maybe initially, but critically, I think the evidence we will get for pre-AGI levels will heavily constrain our expectations of what an ASI will do re it's alignment, and that we will learn a lot more about both alignment and control techniques when we get human-level models, and I think we can trust a lot of the evidence to generalize at least 2 OOMs up.

So I think a lot of the uncertainty will start becoming removed as AI scales up.

I agree with this:

I think almost anything that counts as AGI is very nearly ASI by default (not because of RSI, just because of hardware scaling ability)

Even without recursive self-improvement, it's pretty easy to scale by several OOMs, and while there are enough bottlenecks to prevent FOOM, they are not enough to slow it down by 1 decade except in tail scenarios

It is defecting against cooperate-bot.

It was me. I initially suggested "Bernard Arnault won't give you $77" as the title, Eliezer said "don't bury the lead, just say 'ASI will not leave just a little sunlight for Earth'". After reading this thread I was thinking about alternate titles and was thinking about ones that would both convey the right thing and feel like a reasonably succinct/aesthetic/etc.

There's also the question of the non-helper as an instance of a class, and you as an instance of a class, and the resulting implied ecology. Or to say it a different way: apply TDT to the shadowmoth / meal question. To say it a third way: if people like me react to situations like this--involving some relationship with someone or something--in such-and-such a way, then what trophic niche are we opening up, i.e. what sort of food are we making available for what sort of predator?