All AGI Safety questions welcome (especially basic ones) [May 2023]

What are the most promising plans for automating alignment research as mentioned in for example OpenAI's approach to alignment and by others?

Suppose we train a model on the sum of all human data, using every sensory modality ordered by timestamp, like a vastly more competent GPT (For the sake of argument, assume that a competent actor with the right incentives is training such a model). Such a predictive model would build an abstract world model of human concepts, values, ethics, etc., and be able to predict how various entities would act based on such a generalised world model. This model would also "understand" almost all human-level abstractions about how fictional characters may act, just like GPT does. My question is: if we used such a model to predict how an AGI, aligned with our CEV [? · GW], would act, in what way could it be misaligned? What failure modes are there for pure predictive systems without a reward function that can be exploited or misgeneralised? It seems like the most plausible mental model I have for aligning intelligent systems without them pursuing radically alien objectives. 

Even without ensuring inner alignment, is it possible to reliably train the preferences of an AGI to be more risk averse, to be bounded and to discount the future more? For example, just by using rewards in RL with those properties and even if the AGI misgeneralizes the objective or the objective is not outer aligned, the AGI might still internalize the intended risk aversion and discounting. How likely is it to do so? Or, can we feasibly hardcode risk aversion, bounded preferences and discounting into today's models without reducing capabilities too much?

My guess is that such AGIs would be safer. Given that there are some risks to the AGI that the AGI will be caught and shut down if it tries to take over, takeover attempts should be relatively less attractive. But could it make much difference?

Do the concepts behind AGI safety only make sense if you have roughly the same worldview as the top AGI safety researchers - secular atheism and reductive materialism/physicalism and a computational theory of mind?

Hi, I have a few questions that I'm hoping will help me clarify some of the fundamental definitions. I totally get that these are problematic questions in the absence of consensus around these terms -- I'm hoping to have a few people weigh in and I don't mind if answers are directly contradictory or my questions need to be re-thought.

• If it turns out that LLMs are a path to the first "true AGI" in the eyes of, say, the majority of AI practitioners, what would such a model need to be able to do, and at what level, to be considered AGI, that GPT-4 can't currently do?
• If we look at just "GI", rather than "AGI", do any animals have GI? If so, where is the cutoff between intelligent and unintelligent animals? Is GPT-4 considered more or less intelligent than, say, an amoeba, or a gorilla, or an octopus etc?
• When we talk about "alignment", are today's human institutions and organisations aligned to a lesser or greater extent we would want AGI aligned? Are religions, governments, corporations, organised crime syndicates, serial killers, militaries, fossil fuel companies etc considered aligned, and to what extent? Does the term alignment have meaning when talking about individuals, or groups of humans? Does it have meaning for animal populations, or inanimate objects?
• If I spoke to a group of AI researchers of the pre-home-computer era, and described a machine capable of producing poetry, books, paintings, computer programs, etc in such a way that the products could not be distinguished from top 10% human examples of those things, that could pass entrance exams into 90% of world universities, could score well over 100 in typical IQ tests of that era, and could converse in dozens of languages via both text and voice, would they consider that to meet some past definition of AGI? Has AGI ever had enough of a consensus definition for that to be a meaningful question?
• If we peer into the future, do we expect commodity compute power and software to get capable enough for an average technical person on an average budget to build and/or train their own AGI from the "ground up"? If so, will there not be a point where a single rogue human or small informal group can intentionally build non-aligned unsafe AGI? And if so, how far away is that point?

Apologies if these more speculative/thought-experimenty questions are off the mark for this thread, happy to be pointed to a more appropriate place for them!

How feasible is it to build a powerful AGI with little agency but that's good at answering questions accurately?

Could we solve alignment with one, e.g. getting it to do research, or using it to verify the outputs of other AGIs?

In the context of Deceptive Alignment, would the ultimate goal of an AI system appear random and uncorrelated with the training distribution's objectives from a human perspective? Or would it be understandable to humans that the goal is somewhat correlated with the objectives of the training distribution?

For instance, in the article below, it is written that "the model just has some random proxies that were picked up early on, and that's the thing that it cares about." To what extent does it learn random proxies?

https://www.lesswrong.com/posts/A9NxPTwbw6r6Awuwt/how-likely-is-deceptive-alignment

If an AI system pursues ultimate goals such as power or curiosity, there seems to be a pseudocorrelation regardless of what the base objective is.

On the other hand, can it possibly learn to pursue a goal completely unrelated to the context of the training distribution, such as mass-producing objects of a peculiar shape?

In the context of Deceptive Alignment, would the ultimate goal of an AI system appear random and uncorrelated with the training distribution's objectives from a human perspective? Or would it be understandable to humans that the goal is somewhat correlated with the objectives of the training distribution?

For instance, in the article below, it is written that "the model just has some random proxies that were picked up early on, and that's the thing that it cares about." To what extent does it learn random proxies?

https://www.lesswrong.com/posts/A9NxPTwbw6r6Awuwt/how-likely-is-deceptive-alignment

If an AI system pursues ultimate goals such as power or curiosity, there seems to be a pseudocorrelation regardless of what the base objective is.

On the other hand, can it possibly learn to pursue a goal completely unrelated to the context of the training distribution, such as mass-producing objects of a peculiar shape?

In the context of Deceptive Alignment, would the ultimate goal of an AI system appear random and uncorrelated with the training distribution's objectives from a human perspective? Or would it be understandable to humans that the goal is somewhat correlated with the objectives of the training distribution?

This question is in the spirit of "I think I'm doing something dumb / obviously wrong -- help me see why" but it's maybe too niche for this thread. (Answers that redirect me to a better place to ask are welcome.)

I recently read Paul Christiano, Eric Neyman and Mark Xu's "Formalizing the presumption of independence" (https://arxiv.org/pdf/2211.06738.pdf). My understanding is that they aim to formalise some types of reasonable (but defeasible) “hand-waving” in otherwise formal proofs, in a way that maintains the underlying deductive structure of a formal proof and responds appropriately to new information / arguments. They're particularly interested in heuristic estimators that presume the independence of random variables so long as we have no reason to think the variables aren't independent and so long as we can adjust the estimate appropriately if we learn about their dependencies.

To that end, suppose we want to estimate $f(X_1,\dots ,X_n)$, where $\{X_i{\}}_{1\le n}$ is a set of real-valued random variables, $f:R^n\to R$, and we have a collection of deductively proved (in)equalities $\pi =\pi (X_1,\dots ,X_n)$ about $\{X_i{\}}_{1\le n}$. Then a natural heuristic estimator could be:

$\tilde{E}\left(f\right(X_1,\dots ,X_n\left)|\pi \right)=E\left(f\right(X_1^{\prime },\dots ,X_n^{\prime }\left)|{\pi }^{\prime }\right)$

where each $X_i^{\prime }$ has the same marginal distributions as $X_i$, ${\pi }^{\prime }=\pi (X_1^{\prime },\dots ,X_n^{\prime })$ (i.e. ${\pi }^{\prime }$ is equal to $\pi$ but with each instance of $X_i$ replaced by $X_i^{\prime }$), and where the $X_i^{\prime }$ are conditionally independent given ${\pi }^{\prime }$. This formalises the idea that we assume we've thought of all the dependencies between the variables of interest and that they're independent, conditional on everything we've thought of so far -- but we can revise this estimate by conditioning on new information and dependencies later.

Before considering any information relating the $X_i$ to each other, $\tilde{E}$ assumes that they are unconditionally independent. As we condition on information about them, we update the estimate to account for this and maintain that the variables are conditionally independent, given the information considered so far. E.g. in the twin primes example, we can initially assume that $\left\{x\text{is prime}\right\}$ and $\{x+2\text{is prime}\}$ are independent, and then condition on the fact that if $x$ is prime, then $x+2$ is odd (this can be operationalised by considering the appropriate indicator function and conditioning on it taking value $1$) to adjust the estimate and assume (for now) that there are no further dependencies.

We always have $\tilde{E}\left(XY\right)=E\left(X^{\prime }{Y}^{\prime }\right)=E\left(X^{\prime }\right)E\left(Y^{\prime }\right)$. In fact, we always have $\tilde{E}\left(XY|\pi \right)=E\left(X^{\prime }|{\pi }^{\prime }\right)E\left(Y^{\prime }|{\pi }^{\prime }\right)$. If we further have that $\pi$ doesn't relate $X$ and $Y$ (i.e. doesn't include a formula containing both $X$ and $Y$), then I think we have $E\left(X^{\prime }|{\pi }^{\prime }\right)=E\left(X|\pi \right)$ and $E\left(Y^{\prime }|{\pi }^{\prime }\right)=E\left(Y|\pi \right)$, giving $\tilde{E}\left(XY|\pi \right)=E\left(X|\pi \right)E\left(Y|\pi \right)$ (i.e. without the primes).

My suggested heuristic estimator apparently has lots of nice properties thanks to being an expectation, including some of the informal properties listed in the paper, which can be stated formally (e.g. if ${\pi }_j$ doesn't have an instance of any of the $X_i$, then conditioning on it won't change the heuristic estimate).

My suggested estimator jumped out to me pretty quickly as capturing (to my understanding) what the authors want, but I'd expect myself to be much worse at this than the authors, who will have spent a while longer thinking about it. So my estimator seems "too good to be true" and I think it's likely I'm pretty confused or missing something obvious and/or important. Please help me see what I'm missing! A couple of hypotheses:

• There's something wrong / incoherent about my suggested heuristic estimator
• My suggested heuristic estimator is too general to be useful
  • The paper mainly considers very specific special cases with specific algorithms for heuristic estimators rather than something as general as this, which might be difficult to implement in practice

I would like to know about the history of the term "AI alignment". I found an article written by Paul Christiano in 2018. Did the use of the term start around this time? Also, what is the difference between AI alignment and value alignment?

https://www.alignmentforum.org/posts/ZeE7EKHTFMBs8eMxn/clarifying-ai-alignment

How slow humans perception comparing to AI? Is it a pure difference of "signal speed of neurons" and "signal speed of copper/aluminum"?

I really appreciate these. 

1. Why do some people think that alignment will be easy/easy enough? 
2. Is there such thing as 'aligned enough to help solve alignment research'? 

Are there ways to make a utility maximizer impervious to Pascal's mugging?

Given an aligned AGI, to what extent are people ok with letting the AGI modify us? Examples of such modifications include (feel free to add to the list):

• Curing aging/illnesses
• Significantly altering our biological form
• Converting us to digital life forms
• Reducing/Removing the capacity to suffer
• Giving everyone instant jhanas/stream entry/etc.
• Altering our desires to make them easier to satisfy
• Increasing our intelligence (although this might be an alignment risk?)
• Decreasing our intelligence
• Refactoring our brains entirely

What exact parts of being "human" do we want to preserve?

Does current AI technology possess the power to work its way up to AGI? For example, if the world all of the sudden put a permanent halt on all AI advancements and we were left with GPT-4, would it, given an infinite amount of time of use, achieve AGI through RLHF and access to the internet? This assumes that GPT-4 is not already an AGI system.

i.e.: Do AI systems need to forgo changes to their actual architecture to become smarter, or do is it possible that they get (significantly) smarter through simply enough usage?