AIXI isn't a model of how an AGI might work inside, it's a model of how an AGI might behave if it is acting optimally. A real AGI would not be expected to act like AIXI, but it would be expected to act somewhat more like AIXI the smarter it is. Since not acting like that is figuratively leaving money on the table. 

The point of the whole utility maximization framing isn't that we necessarily expect AIs to have an explicitly represented utility function internally^[1]. It's that as the AI gets better at getting what it wants and working out the conflicts [LW · GW] between its various desires, its behavior [LW · GW] will be increasingly well-predicted [LW · GW] as optimizing some utility function. 

If a utility function can't accurately summarise your desires, that kind of means they're mutually contradictory. Not in the sense of "I value X, but I also value Y", but in the sense of "I sometimes act like I want X and don't care about Y, other times like I want Y and don't care about X."

Having contradictory desires is kind of a problem if you want to Pareto optimize for those desires well. You risk sabotaging your own plans and running around in circles. You're better off if you sit down and commit to things like "I will act as if I valued both X and Y at all times." If you're smart, you do this a lot. The more contradictions you resolve like this, the more coherent your desires will become, and the closer the'll be to being well described as a utility function.  

I think you can observe simple proto versions of this in humans sometimes, where people move from optimizing for whatever desire feels salient in the moment when they're kids (hunger, anger, joy, etc.), to having some impulse control and sticking to a long-term plan, even if it doesn't always feel good in the moment. 

Human adults are still broadly not smart enough to be well described as general utility maximizers. Their desires are a lot more coherent than those of human kids or other animals, but still not that coherent in absolute terms. The point where you'd roughly expect AIs to become well-described as utility maximizers more than humans are would come after they're broadly smarter than humans are. Specifically, smarter at long-term planning and optimization. 

This is precisely what LLMs are still really bad at. Though efforts to make them better at it are ongoing, and seem to be among the highest priorities for the labs. Precisely because long-term consequentialist thinking is so powerful, and most of the really high-value economic activities require it.

1. ^{^}

   Though you could argue that at some superhuman level of capability, having an explicit-ish representation stored somewhere in the system would be likely, even if the function may no actually be used much for most minute-to-minute processing. Knowing what you really want seems handy, even if you rarely actually call it to mind during routine tasks.

Thank you.

I suppose that's true, but this kind of confirms my intuition that there's something funky going on here that isn't accounted for by rationalist-empiricist-reductionism. Like why are time translations so much more important for our general work than space translations? I guess because the sun bombards the earth with a steady stream of free energy, and earth has life which continuously uses this sunlight to stay out of equillbrium. In a lifeless solar system, time-translations just let everything spin, which isn't that different from space-translations.

> could one replace the energy-first formulations of quantum mechanics with momentum-first formulations?

Momentum is to space what energy is to time. Precisely, energy generates (in the Lie group sense) time-translations, whereas momentum generates spatial translations. So any question about ways in which energy and momentum differ is really a question about how time and space differ.

In ordinary quantum mechanics, time and space are treated very differently: $t$ is a coordinate whereas $x$ is a dynamical variable (which happens to be operator-valued). The equations of QM tell us how $x$ evolves as a function of $t$.

But ordinary QM was long-ago replaced by quantum field theory, in which time and space are on a much more even footing: they are both coordinates, and the equations of QFT tell us how a third thing (the field $\varphi (x,t)$) evolves as a function of $x$ and $t$. Now, the only difference between time and space is that there is only one dimension of the former but three of the latter (there may be some other very subtle differences I'm glossing over here, but I wouldn't be surprised if they ultimately stem from this one).

All of this is to say: our best theory of how nature works (QFT), is neither formulated as "energy-first" nor as "momentum-first". Instead, energy and momentum are on fairly equal footing.

Thank you zhukeepa [LW · GW]  for your thoughtful post. I've been also recently interested in the contribution that religions potentially can bring to the rationality enterprise as a whole.

My perspective is that even though trapped priors consistently pose epistemological barriers to effective reasoning, there is a superficial layer of those trapped priors which are accessible via concentration, breath control and ritual, in other words, to meditation. 

I mention meditation as a generalized concept which can depict many different variations of those three points I mention: concentration, breath control and ritual. This should definitely include all Yoga traditions as well as Christian / Jewish prayer, and many modern forms of therapeutic practice.

The experience you describe of connecting with a distant memory via a reenactment of it and vocal guidance of a therapist is a good example of meditation, in that sense. 

Meditation can produce outstanding results when it comes to creativity,  productivity as well as mental and physical health.  And fundamentally, meditation is a technique, something which was shaped through many iterations to become more and more effective, that is, less and and less wrong, pardon the pun. 

One of the strongest arguments you make in favor of the religious experience is on the timely persistence, or, as you put it, the fact that those traditions have been substantially time-tested, and thus probably have something relevant to say.  

This is something relevant on Moral Philosophy, as well as something relevant on the period that produced any target religion we want to analyze, and also, in the meditation front, something relevant on technical management of cognitive faculties via concentration, breath and ritual.

Technique only thrives in time via survival, and that's how we get such impressive results from more than a thousand year old meditation techniques. We can argue on the premises which are used to enact the specific ritual that a given meditation practice rely on, but we can hardly argue with the results it produces.
 

We recently had a security incident where an attacker used an old AWS access key to generate millions of tokens from various Claude models via AWS Bedrock. While we don't have any specific reason to think that any user data was accessed (and some reasons^[1] to think it wasn't), most possible methods by which this key could have been found by an attacker would also have exposed our database credentials to the attacker. We don't know yet how the key was leaked, but we have taken steps to reduce the potential surface area in the future and rotated relevant credentials. This is a reminder that LessWrong does not have Google-level security and you should keep that in mind when using the site.

1. ^{^}

   The main reason we don't think any user data was accessed is because this attack bore several signs of being part of a larger campaign, and our database also contains other LLM API credentials which would not have been difficult to find via a cursory manual inspection. Those credentials don't seem have been used by the attackers. Larger hacking campaigns like this are mostly automated, and for economic reasons the organizations conducting those campaigns don't usually sink time into manually inspecting individual targets for random maybe-valuable stuff that isn't part of their pipeline.

Any agent which thinks it is at risk of being seen as cooperate-bot and thus fine to defect against in the future will be more wary of trusting that ASI.

It seems to me all the more basic desires ("values"), e.g. the lower layers of Maslow's hierarchy of needs, are mainly determined by heritable factors. Because they are relatively stable across cultures. So presumably you talk about "higher" values being a function of "data sources in the world"? I.e. of nurture rather than nature?

I agree there probably are some heritable values, though my big difference here is that I think that the set of primitive values is quite a bit smaller than you might think.

Though be warned, heritability doesn't actually answer our question, because the way it's interpreted by laymen is pretty wrong:

https://www.lesswrong.com/posts/YpsGjsfT93aCkRHPh/what-does-knowing-the-heritability-of-a-trait-tell-me-in [LW · GW]

I probably should have separated formal ethical theories that people are describing, which you call morals and what their values actually are more.

I was always referring to values when I was talking about morals.

You are correct that someone describing a moral theory doesn't mean that they actually agree with or implement the theory.

I still think that if you had the amount of control over a human that an ML person had over an AI today, you could brainwash them to value ~arbitrary values with a lot of control, and it would be the central technology of political and social situations, which is a lot.

Personal AI Assistant Ideas

When I imagine having a personal AI assistant with approximately current levels of capability I have a variety of ideas of what I'd like it to do for me.

Auto-background-research

I like to record myself rambling about my current ideas while walking my dog. I use an app that automatically saves a mediocre transcription of the recording. Ideally, my AI assistant would respond to a new transcription by doing background research to find academic literature related to the ideas mentioned within the transcript. That way, when I went to look at my old transcript, it would already be annotated with links to prior work done on the topic, and analysis of the overlap between my ideas and the linked literature.

Also, ideally, the transcription would be better quality. Context clues about the topic of the conversation should be taken into account when making guesses about transcribing unclear words. Where things are quite unclear, there should be some sort of indicator in the text of that.

Various voice modes, and ability to issue verbal commands to switch between them

Receptive Listening Voice-mode

Also for when I want to ramble about my ideas, I'd like having my AI assistant act as a receptive listener, just saying things like 'that makes sense', or 'could you explain more about what you mean about <idea>?'. Ideally, this would feel relatively natural, and not too intrusive. Asking for clarification just where the logic didn't quite follow, or I used a jargon term in some non-standard seeming way. The conversation in this mode would be one-sided, with the AI assistant just helping to draw out my ideas, not contributing much. Occasionally it might say, 'Do you mean <idea> in the same sense that <famous thinker> means <similar sounding idea>?' And I could explain similarity or differences.

Question Answering Voice-mode

Basically a straightforward version of the sort of thing Perplexity does, which is try to find academic sources to answer a technical question. I'd want to be able to ask, and get a verbal answer of summaries of the sources, but also have a record saved of the conversation and the sources (ideally, downloaded and saved to a folder). This would be mostly short questions from me, and long responses from the model.

Discussion Voice-mode

More emphasis on analysis of my ideas, extrapolating, pointing out strengths and weaknesses. Something like what I get when discussing ideas with Claude Sonnet 3.5 after having told it to act as a philosopher or scientist who is critically examining my ideas. This would be a balanced back-and-forth, with roughly equal length conversational turns between me and the model.

Coding Project Collaboration

I would want to be able to describe a coding project, and as I do so have the AI assistant attempt to implement the pieces. If I were using a computer, then an ongoing live output of the effects of the code could be displayed. If I were using voice-mode, then the feedback would be occasional updates about errors or successful runs and the corresponding outputs. I could ask for certain metrics to be reported against certain datasets or benchmarks, also general metrics like runtime (or iterations per second where appropriate), memory usage, and runtime complexity estimates.

Anthropic Wishlist
 

Anthropic is currently my favorite model supplier, in addition to being my most trusted and approved of lab in regards to AI safety and also user data privacy. So, when I fantasize about what I'd like future models to be like, my focus tends to be on ways I wish I could improve Anthropic's offering.

Most of these could be implemented as features in a wrapper app. But I'd prefer them to be implemented directly by Anthropic, so that I didn't have to trust an additional company with my data.

In order from most desired to less desired:

1. Convenience feature: Ability to specify a system prompt which gets inserted at the beginning of every conversation. Mine would say something like, "Act like a rational scientist who gives logical critiques. Keep praise to a minimum, no sycophancy, no apologizing. Avoid filler statements like, 'let me know if you have further questions.'
   
    
2. Ability to check up-to-date documents and code for publicly available code libraries. This doesn't need to be a web search! You could have a weekly scraper check for changes to public libraries, like python libraries. So many of the issues I run into with LLMs generating code that doesn't work is because of outdated calls to libraries which have since been updated.
   
    
3. Voice Mode, with appropriate interruptions, tone-of-voice detection, and prosody matching. Basically, like what OpenAI is working on.
   
    
4. Academic citations. This doesn't need to be a web search! This could just be from searching an internal archive of publicly available open access scientific literature, updated once a week or so.
   
    
5. Convenience feature: A button to enable 'summarize this conversation, and start a new conversation with that summary as a linked doc'.
   
    
6. Ability to test out generated code to make sure it at least compiles/runs before showing it to me. This would include the ability to have a code file which we were collaborating on, which got edited as the conversation went on. Instead of giving responses intended to modify just a specific function within a code file, where I need to copy/paste it in, and then rerun the code to check if it works.
   
    
7. Ability to have some kind of personalization of the model which went deeper than a simple system prompt. Some way for me to give feedback. Some way for me to select some of my conversations to be 'integrated', such that the mode / tone / info from that conversation were incorporated more into future fresh conversations. Sometimes I feel like I've 'made progress' in a conversation, gotten into a better pattern, and it's frustrating to have to start over from scratch every time.

I updated the title with one Eliezer seemed fine with (after poking Robby). Not my top choice, but better than the previous one.