All AGI Safety questions welcome (especially basic ones) [~monthly thread]

I don't know why other people say it, but I can explain why it's nice to say it.

• log P(x) behaves nicely in comparison to P(x) when it comes to placing iterated bets. When you maximize P(x), you're susceptible to high risk high reward scenarios, even when they lead to failure with probability arbitrarily close to 1. The same is not true when maximizing log P(x). I'm cheating here since this only really makes sense when big-P refers to "principal" (i.e., the thing growing or shrinking with each bet) rather than "probability".
• p(x) doesn't vary linearly with the controls we typically have, so calculus intuition tends to break down when used to optimize p(x). Log p(x) does usually vary linearly with the controls we typically have, so we can apply more calculus intuition to optimizing it. I think this happens because of the way we naturally think of "dimensions of" and "factors contributing to" a probability and the resulting quirks of typical maximum entropy distributions.
• Log p(x) grows monotonically with p(x) whenever x is possible, so the result is the same whether you argmax log p(x) or p(x).
• p(x) is usually intractable to calculate, but there's a slick trick to approximate it using the Evidence Based Lower Bound, which requires dealing with log p(x) rather than p(x) directly. Saying log p(x) calls that trick to mind more easily than saying just p(x).
• All the cool papers do it.

Paul's comment here [LW(p) · GW(p)] is relevant, but I'm also confused.

1. We don't die in literally all branches of the future, and I want to make my future selves proud even if they're a tiny slice of probability.
2. I am a mind with many shards of value, many of which can take joy from small and local things, like friends and music. Expecting that the end is coming changes longer term plans, but not the local actions and joys too much.

Use the dignity heuristic as reward shaping [LW · GW]

“There's another interpretation of this, which I think might be better where you can model people like AI_WAIFU as modeling timelines where we don't win with literally zero value. That there is zero value whatsoever in timelines where we don't win. And Eliezer, or people like me, are saying, 'Actually, we should value them in proportion to how close to winning we got'. Because that is more healthy... It's reward shaping! We should give ourselves partial reward for getting partially the way. He says that in the post, how we should give ourselves dignity points in proportion to how close we get.

And this is, in my opinion, a much psychologically healthier way to actually deal with the problem. This is how I reason about the problem. I expect to die. I expect this not to work out. But hell, I'm going to give it a good shot and I'm going to have a great time along the way. I'm going to spend time with great people. I'm going to spend time with my friends. We're going to work on some really great problems. And if it doesn't work out, it doesn't work out. But hell, we're going to die with some dignity. We're going to go down swinging.”

Even if the world is probably doomed, there's a certain sense of - I don't call it joy, more like a mix of satisfaction and determination, that comes from fighting until the very end with everything we have.  Poets and writers have argued since the dawn of literature that the pursuit of a noble cause is more meaningful than any happiness we can achieve, and while I strongly disagree with the idea of judging cool experiences by meaning, it does say a lot about how historically strong the vein of humans deriving value from working for what you believe in is.

In terms of happiness itself - I practically grew up a transhumanist, and my ideals of joy are bounded pretty damn high.  Deviating from that definitely took a lot of getting used to, and I don't claim to be done in that regard either.  I remember someone once saying that six months after it really "hit" them for the first time, their hedonic baseline had oscillated back to being pretty normal.  I don't know if that'll be the same for everyone, and maybe it shouldn't be - the end of the world and everything we love shouldn't be something we learn to be okay with.  But we fight against it happening, so maybe between all the fire and brimstone, we're able to just enjoy living while we are.  It isn't constant, but you do learn to live in the moment, in time.  The world is a pretty amazing place, filled with some of the coolest sentient beings in history.

I am not an AI researcher, but it seems analogous to the acceptance of mortality for most people. Throughout history, almost everyone has had to live with the knowledge that they will inevitably die, perhaps suddenly. Many methods of coping have been utilized, but at the end of the day it seems like something that human psychology is just... equipped to handle. x-risk is much worse than personal mortality, but you know, failure to multiply and all that.

Living in the moment helps. There’s joy and beauty and life right here, right now, and that’s worth enjoying.

I wrote a relevant comment elsewhere. [LW(p) · GW(p)] Basically I think that finding joy in the face of AGI catastrophe is roughly the same task as finding joy in the face of any catastrophe. The same human brain circuitry is at work in both cases.

In turn, I think that finding joy in the face of a catastrophe is not much different than finding joy in "normal" circumstances, so my advice would be standard stuff like aerobic exercise, healthy food, sufficient sleep, supportive relationships, gratitude journaling, meditation, etc (of course, this advice is not at all tailored to you and could be bad advice for you).

Also, AGI catastrophe is very much impending. It's not here yet. In the present moment, it doesn't exist. It might be worth reflecting on the utility of worry and how much you'd like to worry each day.

Firstly, it's essential to remember that you can't control the situation; you can only control your reaction to it. By focusing on the elements you can influence and accepting the uncertainty of the future, it becomes easier to manage the anxiety that may arise from contemplating potentially catastrophic outcomes. This mindset allows AGI safety researchers to maintain a sense of purpose and motivation in their work, as they strive to make a positive difference in the world.

Another way to find joy in this field is by embracing the creative aspects of exploring AI safety concerns. There are many great examples of fiction based on problems with AI safety. E.g. 

• "Runaround" by Isaac Asimov (1942)
• "The Lifecycle of Software Objects" by Ted Chiang (2010)
• "Cat Pictures Please" by Naomi Kritzer (2015)
•  The Matrix (1999) - Directed by the Wachowskis 
• The Terminator (1984) - Directed by James Cameron
• 2001: A Space Odyssey (1968) - Directed by Stanley Kubrick
• Blade Runner (1982) - Directed by Ridley Scott
• etc, etc.

Engaging in creative storytelling not only provides a sense of enjoyment, but it can also help to spread awareness about AI safety issues and inspire others to take action.

In summary, finding joy in the world of AGI safety research and enthusiasm involves accepting what you can and cannot control, and embracing the creative aspects of exploring potential AI safety concerns. By focusing on making a positive impact and engaging in imaginative storytelling, individuals in this field can maintain a sense of fulfillment and joy in their work, even when faced with the possibility of a seemingly doomed future.

Based on my understanding of grokking, it is just something that will always occur in a suitable regime: sufficient model capacity (overcomplete) combined with proper complexity regularization. If you optimize on the test hard enough the model will first memorize the test set and get to near zero predictive error, but non zero regularization error. At that point if you continue optimizing there is only one way the system can further improve the loss function, and that is by maintaining low/zero prediction error while also reducing the regularization penalty error and thus model complexity. It seems inevitable given the right setup and sufficient optimization time (although typical training regimes may not suffice in practice).

From Comment on decision theory [LW · GW]:

We aren't working on decision theory in order to make sure that AGI systems are decision-theoretic, whatever that would involve. We're working on decision theory because there's a cluster of confusing issues here (e.g., counterfactuals, updatelessness, coordination) that represent a lot of holes or anomalies in our current best understanding of what high-quality reasoning is and how it works.

From On the purposes of decision theory research [LW · GW]:

I think decision theory research is useful for:

1. Gaining information about the nature of rationality (e.g., is “realism about rationality [LW · GW]” true?) and the nature of philosophy (e.g., is it possible to make real progress in decision theory, and if so what cognitive processes are we using to do that?), and helping to solve the problems of normativity [LW · GW], meta-ethics [LW · GW], and metaphilosophy [LW · GW].
2. Better understanding potential AI safety failure modes that are due to flawed decision procedures implemented in or by AI.
3. Making progress on various seemingly important intellectual puzzles that seem directly related to decision theory, such as [LW · GW] free will, anthropic reasoning, logical uncertainty, Rob's examples of counterfactuals, updatelessness, and coordination, and more.
4. Firming up [LW(p) · GW(p)] the foundations of human rationality.

Decision theory is a term used in mathematical statistics and philosophy. In applied AI terms, a decision theory is the algorithm used by an AI agent to compute what action to take next. The nature of this algorithm is obviously relevant to alignment. That being said, philosophers like to argue among themselves about different decision theories and how they relate to certain paradoxes and limit cases, and they conduct these arguments using a terminology that is entirely disconnected from that being used in most theoretical and applied AI research. Not all AI alignment researchers believe that these philosophical arguments are very relevant to moving forward AI alignment research.

AI’s may make decisions in ways that we find counterintuitive. These ways are likely to be shaped to be highly effective (because being effective is instrumentally useful to many goals), so understanding the space of highly effective decision theories is one way to think about how advanced AI’s will reason without having one in front of us right now.

My take is centred more on current language models, which are also non-optimizers, so I'm afraid this won't be super relevant if you're already familiar with the rest of this and were asking specifically about the context of image systems.

Language models are simulators [LW · GW] of worlds sampled from the prior representing our world (insofar as the totality of human text is a good representation of our world), and doesn't have many of the properties we would associate with "optimizeriness".  They do, however, have the potential to form simulacra that are themselves optimizers, such as GPT modelling humans (with pretty low fidelity right now) when making predictions.  One danger from this kind of system that isn't itself an optimizer, is the possibility of instantiating deceptive simulacra that are powerful enough to act in ways that are dangerous to us (I'm biased here, but I think this section [LW · GW] from one of my earlier posts does a not-terrible job of explaining this).

There's also the possibility of these systems becoming optimizers, as you mentioned.  This could happen either during training (where the model at some point during training becomes agentic and starts to deceptively act like a non-optimizer simulator would - I describe this scenario in another section [LW · GW] from the same post), or could happen later, as people try to use RL on it for downstream tasks.  I think what happens here mechanistically at the end could be one of a number of things - the model itself completely becoming an optimizer, an agentic head on top of the generative model that's less powerful than the previous scenario at least to begin with, a really powerful simulacra that "takes over" the computational power of the simulation, etc.

I'm pretty uncertain on numbers I would assign to either outcome, but the latter seems pretty likely (although I think the former might still be a problem), especially with the application of powerful RL for tasks that benefit a lot from consequentialist reasoning.  This post [LW · GW] by the DeepMind alignment team goes into more detail on this outcome, and I agree with its conclusion that this is probably the most likely path to AGI (modulo some minor details I don't fully agree with that aren't super relevant here).

Here's why I personally think solving AI alignment is more effective than generally slowing tech progress

• If we had aligned AGI and coordinated in using it for the right purposes, we could use it to make the world less vulnerable to other technologies
• It's hard to slow down technological progress in general and easier to steer the development of a single technology, namely AGI
• Engineered pandemics and nuclear war are very unlikely to lead to unrecoverable societal collapse if they happen (see this report [EA · GW]) whereas AGI seems relatively likely (>1% chance)
• Other more dangerous technology (like maybe nano-tech) seems like it will be developed after AGI so it's only worth worrying about those technologies if we can solve AGI

So the same question? :P

I'd imagine the AGI Safety Fundamentals introductory governance curriculum (Google doc) is somewhat optimized for getting people up to speed quickly.

If you ask oracle AGI "What code should I execute to achieve goal X?" the result, with very high probability, is agentic AGI.

You can read this [LW · GW] and this [LW · GW]

For the same reason that a chainsaw isn't safe, just massively scaled up. Maybe Chornobyl would be a better example of an unsafe tool? That's assuming that by tool AGI you mean something that isn't agentic. If you let it additionally be agentic, then you're back to square one, and all you have is a digital slave.

An oracle is nice in that it's not trying to enforce its will upon the world. The problem with that is differentiating between it and an AGI that is sitting in a box and giving you (hopefully) good ideas, but with a hidden agenda. Which brings you back to the eternal question of how to get good advisors that are like Gandalf, rather than Wormtongue. 

Check the tool AI [? · GW] and oracle AI [? · GW] tags for more info.

One answer is the concept of "mesa-optimizers" - that is, if a machine learning algorithm is trained to answer questions well, it's likely that in order to do that, it will build an internal optimizer that's optimizing for something else other than answering questions - and that thing will have the same dangers as a non tool/oracle AI. Here's the AI safety forum tag page: https://www.alignmentforum.org/tag/mesa-optimization

In order for a Tool/Oracle to be highly capable/useful and domain-general, I think it would need to perform some kind of more or less open-ended search or optimization. So the boundary between "Tool", "Oracle", and "Sovereign" (etc.) AI seems pretty blurry to me. It might be very difficult in practice to be sure that (e.g.) some powerful "tool" AI doesn't end up pursuing instrumentally convergent goals (like acquiring resources for itself). Also, when (an Oracle or Tool is) facing a difficult problem and searching over a rich enough space of solutions, something like "consequentialist agents" seem to be a convergent thing to stumble upon and subsequently implement/execute.

Suggested reading: https://www.lesswrong.com/posts/kpPnReyBC54KESiSn/optimality-is-the-tiger-and-agents-are-its-teeth [LW · GW]

Yes to safety orgs, the Stampy UI has one based on this post [LW · GW]. We aim for it to be a maintained living document. I don't know of one with capabilities orgs, but that would be a good addition.

Stampy recommends The Alignment Problem and a few others, but none are exactly textbook flavored.

A high-quality continuously updated textbook would be a huge boon for the field, but given the rate that things are changing and the many paradigms it would be a lot of work. The closest thing is probably the AGI Safety Fundamentals course.

The classical example (now outdated) is Superintelligence by Nick Bostrom.  For something closer to an updated introductory-style text on the field, I would probably recommend the AGI Safety Fundamentals curriculum.

When one uses mathematics to clarify many AI alignment solutions, or even just to clarify Monte Carlo tree search as a decision making process, then the mathematical structures one finds can often best be interpreted as being mathematical counterfactuals, in the Pearl causal model sense. This explains the interest into counterfactual machine reasoning among many technical alignment researchers.

To explain this without using mathematics: say that we want to command a very powerful AGI agent to go about its duties while acting as if it cannot successfully bribe or threaten any human being. To find the best policy which respects this 'while acting as if' part of the command, the AGI will have to use counterfactual machine reasoning.

I think the issue might be that the ELK head (the system responsible for eliciting another system's latent knowledge) might itself be deceptively aligned. So if we don't solve deceptive alignment our ELK head won't be reliable.

Both explainable AI and interpretable AI are pronouns that are being used to have different meanings in different contexts. It really depends on the researcher what they mean by it.

Independent research is an option, but you'd have a much better chance of being funded if you did one of the in-person training courses like SERI MATS or Refine. You could apply for a remote position, but most orgs are in-person and US/UK. Not all though, there is one relevant org in Prague, and likely a few others popping up soon.

Currently, we don't know how to make a smarter-than-us AGI obey any particular set of instructions whatsoever, at least not robustly in novel circumstances once they are deployed and beyond our ability to recall/delete. (Because of the Inner Alignment Problem). So we can't just type in stipulations like that. If we could, we'd be a lot closer to safety. Probably we'd have a lot more stipulations than just that. Speculating wildly, we might want to try something like: "For now, the only thing you should do is be a faithful imitation of Paul Christiano but thinking 100x faster." Then we can ask our new Paul-mimic to think for a while and then come up with better ideas for how to instruct new versions of the AI.

What is "killing"? What is "harming"? What is "jeopardizing"? What is "living"? What is "human"? What is the difference between "I cause future killing/harming/jeopardizing" and "future killing/harming/jeopardizing will be in my lightcone"? How to explain all of this to AI? How to check if it understood everything correctly?

We don't know.

It is definitely advisable to build a paper-clip maximiser that also needs to respect a whole bunch of additional stipulations about not harming people. The worry among many alignment researchers is that it might be very difficult to make these stipulations robust enough to deliver the level of safety we ideally want, especially in the case of AGIs that might get hugely intelligent or hugely powerful. As we are talking about not-yet-invented AGI technology, nobody really knows how easy or hard it will be to build robust-enough stipulations into it. It might be very easy in the end, but maybe not. Different researchers have different levels of optimism, but in the end nobody knows, and the conclusion remains the same no matter what the level of optimism is. The conclusion is to warn people about the risk and to do more alignment research with the aim to make it easier build robust-enough stipulations into potential future AGIs.

In addition to Daniel's point, I think an important piece is probabilistic thinking - the AGI will execute not based on what will happen but on what it expects to happen. What probability is acceptable? If none, it should do nothing.

Here's a more detailed writeup about this: https://www.lesswrong.com/posts/rP66bz34crvDudzcJ/decision-theory-does-not-imply-that-we-get-to-have-nice [LW · GW] 

Yes, a lot of it has been informed by economics. Some authors emphasize the relation, others de-emphasize it.

The relation goes beyond alignment and safety research. The way in which modern ML research defines its metric of AI agent intelligence is directly based on utility theory, which was developed by Von Neumann and Morgenstern to describe games and economic behaviour.

But it could make AGIs less dangerous by causing them to make exploitable mistakes, or fail to learn facts or techniques that would make them too powerful.

There is in fact class of AI safety proposals that try to make the AI unaware of certain things, such as not being aware that there is a shutdown button for it.

One of the issues with these types of proposals is that as the AI gets smarter/more powerful, it has to come up with increasingly crazy hypotheses about the world to ignore the facts that all the evidence is pointing towards (such as the fact that there's a conspiracy, or an all powerful being, or something doing this).  This could, in the long term, cause it to be very dangerous in it's unpredictability.

It depends. But yes, incorrect epistemics can make an AGI safer, if it is the right and carefully calibrated kind of incorrect. A goal-directed AGI that incorrectly believes that its off switch does not work will be less resistant to people using it. So the goal is here to design an AGI epistemics that is the right kind of incorrect.

Note: designing an AGI epistemics that is the right kind of incorrect seems to go against a lot of the principles that aspiring rationalists seem to hold dear, but I am not an aspiring rationalist. For more technical info on such designs, you can look up my sequence on counterfactual planning.

I would say that the idea of superintelligence is important for the idea that AGI is hard to control (because we likely can't outsmart it).

I would also say that there will not be any point at which AGIs are "as smart as humans". The first AGI may be dumber than a human, and it will be followed (perhaps immediately) by something smarter than a human, but "smart as a human" is a nearly impossible target to hit because humans work in ways that are alien to computers. For instance, humans are very slow and have terrible memories; computers are very fast and have excellent memories (when utilized, or no memory at all if not programmed to remember something, e.g. GPT3 immediately forgets its prompt and its outputs).

This is made worse by the impatience of AGI researchers, who will be trying to create an AGI "as smart as a human adult" in a time span of 1 to 6 months, because they're not willing to spend 18 years on each attempt, and so if they succeed, they will almost certainly have invented something smarter than a human over a longer training interval. c.f. my own 5-month-old human

If by intelligence spectrum you mean variations in capability across different generally intelligent minds, such that there can be minds that are dramatically more capable (and thus more dangerous): yes, it's pretty important.

If it were impossible to make an AI more capable than the most capable human no matter what software or hardware architectures we used, and no matter how much hardware we threw at it, AI risk would be far less concerning.

But it really seems like AI can be smarter than humans. Narrow AIs (like MuZero) already outperform all humans at some tasks, and more general AIs like large language models are making remarkable and somewhat spooky progress.

Focusing on a very simple case, note that using bigger, faster computers tends to let you do more. Video games are a lot more realistic than they used to be. Physics simulators can simulate more. Machine learning involves larger networks. Likewise, you can run the same software faster. Imagine you had an AGI that demonstrated performance nearly identical to that of a reasonably clever human. What happens when you use enough hardware that it runs 1000 times faster than real time, compared to a human? Even if there are no differences in the quality of individual insights or the generality of its thought, just being able to think fast alone is going to make it far, far more capable than a human.

Mass-self-copying is one of the mechanisms by which an infrahuman AGI might scale up its compute and turn itself into a superintelligence. Fine-tuning its own weights to produce a more-powerful successor is another strategy it might use. So if you can deny it the ability to do these things, this potentially buys a little time.

This only helps if AGI spends significant time in the infrahuman domain, if one of these strategies would been the first-choice strategy for how to foom (as opposed to social engineering or nanobots or training a successor without using its own weights), and if humanity can do something useful with time that's bought. So it's probably a little bit helpful, but in the best case it's still only a small modifier on a saving throw, not a solution.

Pretty significant, and probably one big component of limiting the damage, though doesn't rule out all failure modes.

Reducing internet usage and limiting the amount of data available to AI companies might seem like a feasible approach to regulate AI development. However, implementing such measures would likely face several obstacles.  E.g.

• AI companies purchase internet access like any other user, which makes it challenging to specifically target them for data reduction without affecting other users. One potential mechanism to achieve this goal could involve establishing regulatory frameworks that limit the collection, storage, and usage of data by AI companies. However, these restrictions might inadvertently affect other industries that rely on data processing and analytics.
• A significant portion of the data utilized by AI companies is derived from open-source resources like Common Crawl and WebText2. These companies have normally already acquired copies of this data for local use, meaning that limiting internet usage would not directly impact their access to these datasets.
• If any country were to pass a law limiting the network data available to AI-based companies, it is likely that these companies would relocate to other countries with more lenient regulations. This would render such policies ineffective on a global scale, while potentially harming the domestic economy and innovation in the country implementing the restrictions.

In summary, while the idea of reducing the amount of data AI companies have to work with might appear feasible, practical implementation faces significant hurdles. A more effective approach to regulating AI development could involve establishing international standards and ethical guidelines, fostering transparency in AI research, and promoting cross-sector collaboration among stakeholders. This would help to ensure the responsible and beneficial growth of AI technologies without hindering innovation and progress.

It's definitely something Stampy would want to link to, and if those authors wanted to maintain their list on Stampy rather than LessWrong that would be welcome, though I could imagine them wanting to retain editing restrictions.

Edit: Added a link to Stampy.

Yes. If it turns out that alignment is trivial or unneeded whenever AGIs appear. The same as with all excessive safety requirements. The problem here being discerning between excessive and necessary. An example would be hand sanitizers etc. during the pandemic. In the beginning it wasn't stupid, as it wasn't known how the virus spread. Later, when it became obvious that it was airborne, then keeping antibacterial gels everywhere in the name of ending the pandemic was stupid (at least in the context of Covid) and wasteful. We're currently in the phase of having no idea about how safe AGI will or won't be, so it's a good idea to check for potential pitfalls.

It comes down to uncertainty. It seems wiser to go slowly and carefully, even if that will result in loss, if doing so can limit the potential dangers of AGI. Which could potentially involve the eradication of everything you hold dear.

I'm not sure how literally you treat the present - if you only care about, say the next 5 years, then yes, the safety overhead is harmful in that it takes resources from other stuff. If you care about the next 30 years, then the overhead is more justified, given that many people expect to have an AGI by then.  

The main problems with CEV are that there is no method for it in practice, and no proof that it could work in principle.

I'm lazy, so I'll just link to this https://www.lesswrong.com/tag/coherent-extrapolated-volition [? · GW]

Why can there only be one? Isn't X-risk the risk of an extinction event, rather than the event itself?

It's a matter of expected value. A comet strike that would sterilize earth is very unlikely - in the last ~500 million years there have been 6 mass extinctions (including the current one). So it seems that naturally occurring X-risks exist, but aren't that likely (ignoring black swans, problems of induction etc, of course).

On the other hand, if you believe that creating a god is possible and even likely, you then have to consider whether it's more likely for you to summon a demon than to get something that is helpful.

So if your prior is that there is like 5% probability of an X-risk in the next 100 years (these numbers are meaningless - I chose them to make things easier), that there is 10% probability of AGI, and that it's only 50% probability that the AGI wouldn't destroy everything, then virtually all of your probability mass is that it's the AGI that will be the X-risk. And the obvious action is to focus most of your efforts on that.