AGI Safety FAQ / all-dumb-questions-allowed thread

An AGI that was not a utility maximizer would make more progress towards whatever goals it had if it modified itself to become a utility maximizer.  Three exceptions are if (1) the AGI has a goal of not being a utility maximizer, (2) the AGI has a goal of not modifying itself, (3) the AGI thinks it will be treated better by other powerful agents if it is not a utility maximizer.

This is an excellent question.  I'd say the main reason is that all of the AI/ML systems that we have built to date are utility maximizers; that's the mathematical framework in which they have been designed.  Neural nets / deep-learning work by using a simple optimizer to find the minimum of a loss function via gradient descent.  Evolutionary algorithms, simulated annealing, etc. find the minimum (or maximum) of a "fitness function".  We don't know of any other way to build systems that learn.

Humans themselves evolved to maximize reproductive fitness.   In the case of humans, our primary fitness function is reproductive fitness, but our genes have encoded a variety of secondary functions which (over evolutionary time) have been correlated with reproductive fitness.  Our desires for love, friendship, happiness, etc. fall into this category.  Our brains mainly work to satisfy these secondary functions; the brain gets electrochemical reward signals, controlled by our genes, in the form of pain/pleasure/satisfaction/loneliness etc.  These secondary functions may or may not remain aligned with the primary loss function, which is why practitioners sometimes talk about "mesa-optimizers" or "inner vs outer alignment."

(I assume you are asking "why do we assume the agent has a coherent utility function" rather than "why do we assume the agent tries maximizing their utility" ? )

Agents like humans which don't have such a nice utility function:

1. Are vulnerable to money pumping [LW · GW]
2. Can notice that problem and try to repair themselves
3. Note that humans do in practice try to repair ourselves, like to smash down our own emotions in order to be more productive. But we don't have access to our source code, so we're not so good at it

I think that if the AI can't repair that part of themselves and they're still vulnerable to money pumping, then they're not the AGI we're afraid of, I think

Excellent question! I've added a slightly reworded version of this to Stampy. (focusing on superintelligence, rather than AGI, as it's pretty likely that we can get weak AGI which is non-maximizing, based on progress in language models)

AI subsystems or regions in gradient descent space that more closely approximate utility maximizers are more stable, and more capable, than those that are less like utility maximizers. Having more agency [LW · GW] is a convergent instrument goal and a stable attractor which the random walk of updates and experiences will eventually stumble into.

The stability is because utility maximizer-like systems which have control over their development would lose utility if they allowed themselves to develop into non-utility maximizers, so they tend to use their available optimization power to avoid that change (a special case of goal stability [LW · GW]). The capability is because non-utility maximizers are exploitable, and because agency is a general trick which applies to many domains, so might well arise naturally when training on some tasks.

Humans and systems made of humans (e.g. organizations, governments) generally have neither the introspective ability nor self-modification tools needed to become reflectively stable, but we can reasonably predict that in the long run highly capable systems will have these properties. They can then fix in and optimize for their values.

Why do we assume that any AGI can meaningfully be described as a utility maximizer?

You're right that not every conceivable general intelligence is built as a utility maximizer. Humans are an example of this.

One problem is, even if you make a "weak" form of general intelligence that isn't trying particularly hard to optimize anything, or a tool AI, eventually someone at FAIR will make an agentic version that does in fact directly try to optimize Facebook's stock market valuation.

For one, I don't think organizations of humans, in general, do have more computational power than the individual humans making them up. I mean, at some level, yes, they obviously do in an additive sense, but that power consists of human nodes, each not devoting their full power to the organization because they're not just drones under centralized control, and with only low bandwidth and noisy connections between the nodes. The organization might have a simple officially stated goal written on paper and spoken by the humans involved, but the actual incentive structure and selection pressure may not allow the organization to actually focus on the official goal. I do think, in general, there is some goal an observer could usefully say these organizations are, in practice, trying to optimize for, and some other set of goals each human in them is trying to optimize for.

Perhaps an intelligence (artificial or natural) cannot necessarily, or even typically be described as optimisers? Instead we could only model them as an algorithm or as a collection of tools/behaviours executed in some pattern.

I don't think the latter sentence distinguishes 'intelligence' from any other kind of algorithm or pattern. I think that's an important distinction. There's a lot of past posts explaining how an AI doesn't have code, like a human holding instructions on paper, but rather is its code. I think you can make the same point within a human: that a human has lots of tools/behaviors, which it will execute in some pattern given a particular environment, and the the instructions we consciously hold in mind are only one part of what determines that pattern. 

I contain subagents with divergent goals, some of which are smarter and have greater foresight and planning than others, and those aren't always the ones that determine by immediate actions. As a result, I do a much poorer job optimizing for what the part-of-me-I-call-"I" wants my goals to be, than I theoretically could. 

That gap is decreasing over time as I use the degree of control my intelligence gives me to gradually shape the rest of myself. It may never disappear, but I am much more goal-directed now than I was 10 years ago, or as a child. In other words, in some sense I am figuring out what I want my utility function to be (aka what I want my life, local environment, and world to look like), and self-modifying to increase my ability to apply optimization pressure towards achieving that.

My understanding of all this is partially driven by Robert Kegan's model of adult mental development (see this summary by David Chapman), in which as we grow up we shift our point of view so that different aspects of ourselves become things we have, rather than things we are. We start seeing our sensory experiences, our impulses, our relationships to others, and our relationships to systems we use and operate in, as objects we can manipulate in pursuit of goals, instead of being what we are, and doing this makes us more effective in achieving our stated goals. I don't know if the idea would translate to any particular AI system, but in general having explicit goals, and being able to redirect available resources towards those goals, makes a system more powerful, and so if a system has any goals and self-modifying ability at all, then becoming more like an optimizer will likely be a useful instrumental sub-goal, in the same way that accumulating other resources and forms of power is a common convergent sub-goal. And a system that can't, in any way, be said to have goals at all... either it doesn't act at all and we don't need to worry about it so much, or it acts in ways we can't predict and is therefore potentially extremely dangerous if it gets more powerful tools and behaviors.

Both 80,000hours and AI Safety Support are keen to offer personalised advice to people facing a career decision and interested in working on alignment (and in 80k's case, also many other problems).

Noting a conflict of interest - I work for 80,000 hours and know of but haven't used AISS. This post is in a personal capacity, I'm just flagging publicly available information rather than giving an insider take.

You might want to consider registering for the AGI Safety Fundamentals Course (or reading through the content). The final project provides a potential way of dipping your toes into the water.

Applying to Redwood or Anthropic seems like a great idea. My understanding is that they're both looking for aligned engineers and scientists and are both very aligned orgs. The worst case seems like they (1) say no or (2) don't make an offer that's enough for you to keep your lifestyle (whatever that means for you). In either case you haven't lost much by applying, and you definitely don't have to take a job that puts you in a precarious place financially.

Pragmatic AI safety (link: pragmaticaisafety.com) is supposed to be a good sequence for helping you figure out what to do. My best advice is to talk to some people here who are smarter than me and make sure you understand the real problem [LW · GW]s, because the most common outcome besides reading a lot and doing nothing is to do something that feels like work but isn't actually working on anything important.

Work your way up the ML business  hierarchy to the point where you are having conversations with decision makers.  Try to convince them that unaligned AI is a significant existential risk.  A small chance of you doing this will in expected value terms more than make up for any harm you cause by working in ML given that if you left the field someone else would take your job.

Given where you live, I recomend going to some local LW events. There are still LW meetups in the Bay area, right?

You should apply to Anthropic. If you’re writing ML software at semi-FAANG. they probably want to interview you ASAP. https://www.lesswrong.com/posts/YDF7XhMThhNfHfim9/ai-safety-needs-great-engineers [LW · GW]

The compensation is definitely enough to take care of your family and then save some money!

One of the paths which has non-zero hope in my mind is building a weakly aligned non-self improving research assistant for alignment researchers. Ought [? · GW] and EleutherAI's #accelerating-alignment are the two places I know who are working in this direction fairly directly, though the various language model alignment orgs might also contribute usefully to the project.

Anthropic offer equity, they can give you more details in private. 

I recommend applying to both (it's a cheap move with a lot of potential upside), let me know if you'd like help connecting to any of them.

If you learn by yourself - I'd totally get one on one advise (others linked [LW(p) · GW(p)]), people will make sure you're on the best path possible

My model of Eliezer says something like this:

AI will not be aligned by default, because AI alignment is hard and hard things don't spontaneously happen. Rockets explode unless you very carefully make them not do that. Software isn't automatically secure or reliable, it takes lots of engineering effort to make it that way.

Given that, we can presume there needs to be a specific example of how we could align AI. We don't have one. If there was one, Eliezer would know about it - it would have been brought to his attention, the field isn't that big and he's a very well-known figure in it. Therefore, in the absence of a specific way of aligning AI that would work, the probability of AI being aligned is roughly zero, in much the same way that "Throw a bunch of jet fuel in a tube and point it towards space" has roughly zero chance of getting you to space without specific proof of how it might do that.

So, in short - it is reasonable to assume that AI will be aligned only if we make it that way with very high probability. It is reasonable to assume that if there was a solution we had that would work, Eliezer would know about it. You don't need to know everything about AGI x-risk for that - anything that promising would percolate through the community and reach Eliezer in short order. Since there is no such solution, and no attempts have come close according to Eliezer, we're in trouble.

Reasons you might disagree with this:

• You think AI is a long way away, and therefore it's okay that we don't know how to solve it yet.
• You think "alignment by default" might be possible.
• You think some approaches that have already been brought up for solving the problem are reasonably likely to succeed when fleshed out more.

To start, it's possible to know facts with confidence, without all the relevant info. For example I can't fit all the multiplication tables into my head, and I haven't done the calculation, but I'm confident that 2143*1057 is greater than 2,000,000. 

Second, the line of argument runs like this: Most (a supermajority) possible futures are bad for humans. A system that does not explicitly share human values has arbitrary values. If such a system is highly capable, it will steer the future into an arbitrary state. As established, most arbitrary states are bad for humans. Therefore, with high probability, a highly capable system that is not aligned (explicitly shares human values) will be bad for humans.

I believe the necessary knowledge to be confident in each of these facts is not too big to fit in a human brain.

You may be referring to other things, which have similar paths to high confidence (e.g. "Why are you confident this alignment idea won't work." "I've poked holes in every alignment idea I've come across. At this point, Bayes tells me to expect new ideas not to work, so I need proof they will, not proof they won't."), but each path might be idea specific.

In The Rationalists' Guide to the Galaxy the author discusses the case of a chess game, and particularly when a strong chess player faces a much weaker one. In that case it's very easy to make the prediction that the strong player will win with near certainty, even if you have no way to predict the intermediate steps. So there certainly are domains where (some) predictions are easy despite the world's complexity.

My personal rather uninformed take on the AI discussion is that many of the arguments are indeed comparable in a way to the chess example, so the predictions seem convincing despite the complexity involved. But even then they are based on certain assumptions about how AGI will work (e.g. that it will be some kind of optimization process with a value function), and I find these assumptions pretty intransparent. When hearing confident claims about AGI killing humanity, then even if the arguments make sense, "model uncertainty" comes to mind. But it's hard to argue about that since it is unclear (to me) what the "model" actually is and how things could turn out different.

Before taking Eliezer's opinion into account - what are your priors? (and why?)

For myself, I prefer to have my own opinion and not only to lean on expert predictions, if I can

Agree it's hard to prove a negative, but personally I find the following argument pretty suggestive:

"Other AGI labs have some plans - these are the plans we think are bad, and a pivotal act will have to disrupt them. But if we, ourselves, are an AGI lab with some plan, we should expect our pivotal agent to also be able to disrupt our plans. This does not directly lead to the end of the world, but it definitely includes root access to the datacenter [LW · GW]."

I think one part of the reason for confidence is that any AI weak enough to be safe without being aligned, is weak enough that it can't do much, and in particular it can't do things that a committed group of humans couldn't do without it. In other words, if you can name such an act, then you don't need the AI to make the pivotal moves. And if you know how, as a human or group of humans, to take an action that reliably stops future-not-yet-existing AGI from destroying the world, and without the action itself destroying the world, then in a sense haven't you solved alignment already?

I read this as "if the AGI is able to work around the vast resources that all the big AI labs have put up to defend themselves, then the AGI is probably able to work around your defenses as well" (though I'm not confident)

This is the exact problem StackExchange tries to solve, right? How do we get (and kickstart the use of) an Alignment StackExchange domain?

This is a good idea, and combines nicely with Stampy. We might well do monthly posts where people can ask questions, and either link them to Stampy answers or write new ones.

If you are asking about yourself (?) then it would probably help to talk about your specifics, rather than trying to give a generic answer that would fit many people (though perhaps others would be able to give a good generic answer)

My own prior is:  There are a few groups that seem promising, and I'd want people to help those groups

Agent AI vs. Tool AI [? · GW].

There's discussion on why Tool AIs are expected to become agents; one of the biggest arguments is that agents are likely to be more effective than tools. If you have a tool, you can ask it what you should do in order to get what you want; if you have an agent, you can just ask it to get you the things that you want. Compare Google Maps vs. self-driving cars: Google Maps is great, but if you get the car to be an agent, you get all kinds of other benefits.

It would be great if everyone did stick to just building tool AIs. But if everyone knows that they could get an advantage over their competitors by building an agent, it's unlikely that everyone would just voluntarily restrain themselves due to caution. 

Also it's not clear that there's any sharp dividing line between AGI and non-AGI AI; if you've been building agentic AIs all along (like people are doing right now) and they slowly get smarter and smarter, how do you know when's the point when you should stop building agents and should switch to only building tools? Especially when you know that your competitors might not be as cautious as you are, so if you stop then they might go further and their smarter agent AIs will outcompete yours, meaning the world is no safer and you've lost to them? (And at the same time, they are applying the same logic for why they should not stop, since they don't know that you can be trusted to stop.)

The first type of AI is a regular narrow AI, the type we've been building for a while. The second type is an agentic AI, a strong AI, which we have yet to build. The problem is, AIs are trained using gradient descent, which basically involves running AI designs from all possible AI designs. Gradient descent will train the AI that can maximize the reward best. As a result of this, agentic AIs become more likely because they are better at complex tasks. While we can modify the reward scheme, as tasks get more and more complex, agentic AIs are pretty much the way to go, so we can't avoid building an agentic AI, and have no real idea if we've even created one until it displays behaviour that indicates it.

Here's a non-obvious way it could fail [LW · GW]. I don't expect researchers to make this kind of mistake, but if this reasoning is correct, public access of such an AI is definitely not a good idea.

Also, consider a text predictor which is trying to roleplay as an unaligned superintelligence. This situation could be triggered even without the knowledge of the user by accidentally creating a conversation which the AI relates to a story about a rogue SI, for example. In that case it may start to output manipulative replies, suggest blueprints for agentic AIs, and maybe even cause the user to run an obfuscated version of the program from the linked post. The AI doesn't need to be an agent for any of this to happen (though it would be clearly much more likely if it were one).

I don't think that any of those failure modes (including the model developing some sort of internal agent to better predict text) are very likely to happen in a controlled environment. However, as others have mentioned, agent AIs are simply more powerful, so we're going to build them too.

In short, the difference between the two is Generality. A system that understands the concepts of computational resources and algorithms might do exactly that to improve it's text prediction. Taking the G out of AGI could work, until the tasks get complex enough they require it.

A language model (LM) is a great example, because it is missing several features that AI would have to have in order to be dangerous.  (1) It is trained to perform a narrow task (predict the next word in a sequence), for which it has zero "agency", or decision-making authority.   A human would have to connect a language model to some other piece of software (i.e. a web-hosted chatbot) to make it dangerous.  (2) It cannot control its own inputs (e.g. browsing the web for more data), or outputs (e.g. writing e-mails with generated text).  (3) It has no long-term memory, and thus cannot plan or strategize in any way.  (4) It runs a fixed-function data pipeline, and has no way to alter its programming, or even expand its computational use, in any way.

I feel fairly confident that, no matter how powerful, current LMs cannot "go rogue" because of these limitations.  However, there is also no technical obstacle for an AI research lab to remove these limitations, and many incentives for them to do so.  Chatbots are an obvious money-making application of LMs.  Allowing an LM to look up data on its own to self-improve (or even just answer user questions in a chatbot) is an obvious way to make a better LM.  Researchers are currently equipping LMs with long-term memory (I am a co-author on this work).  AutoML is a whole sub-field of AI research, which equips models with the ability to change and grow over time.

The word you're looking for is "intelligent agent", and the answer to your question "why don't we just not build these things?" is essentially the same as "why don't we stop research into AI?"  How do you propose to stop the research?

Because we're building the AI from the ground up and can change what the AI is via our design choices. Humans' goal functions are basically decided by genetic accident, which is why humans are often counterproductive. 

Assuming humans can't be "aligned", then it would also make sense to allocate resources in an attempt to prevent one of them from becoming much more powerful than all of the rest of us.

Define "not aligned"? For instance, there are plenty of humans who, given the choice, would rather not kill every single person alive.

Most of what people call morality is conflict mediation: techniques for taking the conflicting desires of various parties and producing better outcomes for them than war.
That's how I've always thought of the alignment problem. The creation of a very very good compromise that almost all of humanity will enjoy.

There's no obvious best solution to value aggregation/cooperative bargaining, but there are a couple of approaches that're obviously better than just having an arms race, rushing the work, and producing something awful that's nowhere near the average human preference.

Indeed humans are significantly non-aligned. In order for an ASI to be non-catastrophic, it would likely have to be substantially more aligned than humans are. This is probably less-than-impossible due to the fact that the AI can be built from the get-go to be aligned, rather than being a bunch of barely-coherent odds and ends thrown together by natural selection.

Of course, reaching that level of alignedness remains a very hard task, hence the whole AI alignment problem.

I'm not quite sure what this means. As I understand it humans are not aligned with evolution's implicit goal of "maximizing genetic fitness" but humans are (definitionally) aligned with human values. And e.g. many humans are aligned with core values like "treat others with dignity".

Importantly, capability and alignment are sort of orthogonal. The consequences of misaligned AI get worse the more capable it is, but it seems possible to have aligned superhuman AI, as well as horribly misaligned weak AI.

This. Combine this fact with the non-trivial chance that moral values are subjective, not objective, and there is little good reason to be doing alignment.

Yup, we might want to have these as regular threads with a handy link to Stampy.

I think the best justification is by analogy.  Humans do not physically have a decisive strategic advantage over other large animals -- chimps, lions, elephants, etc.  And for hundreds of thousands of years, we were not at the top of the food chain, despite our intelligence.  However, intelligence eventually won out, and allowed us to conquer the planet.

Moreover, the benefit of intelligence increased exponentially in proportion to the exponential advance of technology.  There was a long, slow burn, followed by what (on evolutionary timescales) was an extremely "fast takeoff": a very rapid improvement in technology (and thus power) over only a few hundred years.  Technological progress is now so rapid that human minds have trouble keeping up within a single lifetime, and genetic evolution has been left in the dust.

That's the world into which AGI will enter -- a technological world in which a difference in intellectual ability can be easily translated into a difference in technological ability, and thus power.  Any future technologies that the laws of physics don't explicitly prohibit, we must assume that an AGI will master faster than we can.

Some one else already commented on how human intelligence gave us a decisive strategic advantage over our natural predators and many environmental threats. I think this cartoon is my mental shorthand for that transition. The timescale is on the order of 10k-100k years, given human intelligence starting from the ancestral environment.

Empires and nations, in turn, conquered the world by taking it away from city-states and similarly smaller entities in ~1k-10k years. The continued existence of Singapore and the Sentinel Islanders doesn't change the fact that a modern large nation could wipe them out in a handful of years, at most, if we really wanted to. We don't because doing so is not useful, but the power exists.

Modern corporations don't want to control the whole world. Like Fnargl, that's not what they're pointed at. But it only took a few decades for Walmart to displace a huge swath of the formerly-much-more-local retail market, and even fewer decades for Amazon to repeat a similar feat online, each starting from a good set of ideas and a much smaller resource base than even the smallest nations. And while corporations are militarily weak, they have more than enough economic power to shape the laws of at least some of the nations that host them in ways that let them accumulate more power over time.

So when I look at history, I see a series of major displacements of older systems by newer ones, on faster and faster timescales, using smaller and smaller fractions of our total resource base, all driven by our accumulation of better ideas and using those ideas to accumulate wealth and power. All of this has been done with brains no smarter, natively, than what we had 10k years ago - there hasn't been time for biological evolution to do much, there. So why should that pattern suddenly stop being true when we introduce a new kind of entity with even better ideas than the best strategies humans have ever come up with? Especially when human minds have already demonstrated a long list of physically-possible scenarios that might, if enacted, kill everyone in a short span of time, or at least disrupt us enough to buy time to mop up the survivors?

Here's a youtube video about it.

The informal way I think about it:

What would I do if I was the AI, but I had 100 copies of myself, and we had 100 years to think for every 1 second that passed in reality.  And I had internet access.

Do you think you could take over the world from that opening?

Edit: And I have access to my own source code, but I only dare do things like fix my motivational problems and make sure I don't get board during all that time, things like that.

Do you dispute that this is possible in principle or just that we won't get AI that powerful or something else? 

It seems to me that there is some level of intelligence, at which an agent is easily able out-compete the whole rest of human civilization. What exactly that level of intelligence is, is somewhat unclear (in large part because we don't really have a principled way to measure "intelligence" in general: psychometrics describe variation in human cognitive abilities, but that doesn't really give us a measuring stick for thinking about how "intelligent", in general, something is).

Does that seem right to you, or should we back up and build out why that seems true to me?

This is the sequence post on it: https://www.lesswrong.com/posts/5wMcKNAwB6X4mp9og/that-alien-message, [LW · GW] it's quite a fun read (to me), and should explain why something smart that thinks at transistor speeds should be able to figure things out.

For inventing nanotechnology, the given example is AlphaFold 2.

For killing everyone in the same instant with nanotechnology, Eliezer often references Nanosystems by Eric Drexler. I haven't read it, but I expect the insight is something like "Engineered nanomachines could do a lot more than those limited by designs that have a clear evolutionary path from chemicals that can form randomly in the primordial ooze of Earth."

For how a system could get that smart, the canonical idea is recursive self improvement (i.e. an AGI capable of learning AGI engineering could design better versions of itself, which could in turn better design better versions, etc, to whatever limit.). But more recent history in machine learning suggests you might be able to go from sub-human to overwhelmingly super-human just by giving it a few orders of magnitude more compute, without any design changes.

Excellent question! MIRI's entire vingian reflection paradigm is about stability of goals under self-improvement and designing successors.

I'm not sure how literally to take this, given that it comes from an April Fools Day post, but consider this excerpt from Q1 of MIRI announces new "Death With Dignity" strategy [LW · GW].

That said, I fought hardest while it looked like we were in the more sloped region of the logistic success curve, when our survival probability seemed more around the 50% range; I borrowed against my future to do that, and burned myself out to some degree. That was a deliberate choice, which I don't regret now; it was worth trying, I would not have wanted to die having not tried, I would not have wanted Earth to die without anyone having tried. But yeah, I am taking some time partways off, and trying a little less hard, now. I've earned a lot of dignity already; and if the world is ending anyways and I can't stop it, I can afford to be a little kind to myself about that.

When I tried hard and burned myself out some, it was with the understanding, within myself, that I would not keep trying to do that forever. We cannot fight at maximum all the time, and some times are more important than others. (Namely, when the logistic success curve seems relatively more sloped; those times are relatively more important.)

All that said: If you fight marginally longer, you die with marginally more dignity. Just don't undignifiedly delude yourself about the probable outcome.

I think he's burned out and took a break to write a story (but I don't remember where this belief came from. Maybe I'm wrong? Maybe from here [LW · GW]?)

I think there are a bunch of political problems with regulating all computer hardware progress enough to cause it to totally cease. Think how crucial computers are to the modern world. Really a lot of people will be upset if we stop building them, or stop making better ones. And if one country stops, that just creates an incentive for other countries to step in to dominate this industry. And even aside from that, I don't think that there's any regulator in the US at least that has enough authority and internal competence to be able to pull this off. More likely, it becomes a politicized issue. (Compare to the much more straightforward and much more empirically-grounded regulation of instituting a carbon tax for climate change. This is a simple idea, that would help a lot, and is much less costly to the world than halting hardware progress. But instead of being universally adopted, it's a political issue that different political factions support or oppose.)

But even if we could, this doesn't solve the problem in a long term way. You need to also halt software progress. Otherwise we'll continue to tinker with AI designs until we get to some that can run efficiently on 2020's computers (or 1990's computers, for that matter).

So in the long run, the only thing in this class that would straight up prevent AGI from being developed is a global, strictly enforced ban on computers. Which seems...not even remotely on the table, on the basis of arguments that are as theoretical as those for AI risk. 

There might be some plans in this class that help, by delaying the date of AGI. But that just buys time for some other solution to do the real legwork.

From there, the question of course becomes whether we can convince governments to do so. If that is impossible then I guess it doesn't matter

Unfortuantely we cannot in fact convince governments to shut down AWS & crew. There are intermediary positions I think are worthwhile but unfortunately ending all AI research is outside the overton window for now.

Most high status people, including Warren Buffett, straightforwardly haven't considered these issues much. However, among the ones I've heard of who have bothered to weigh in on the issue, like Stephen Hawking, Bill Gates, Demis Hassibis, etc.; they do seem to come in favor of the side of "this is a serious problem". On the other hand, some of them get tripped up on one of the many intellectual land mines, like Yann Lecunn. 

I don't think that's unexpected. Intellectual land mines exist, and complicated arguments like the ones supporting AGI risk prevention are bound to cause people to make wrong decisions.

For the specific example of Warren Buffet, I suspect that he probably hasn't spent that much time thinking about it nor does he probably feel much compulsion to understand the topic as he doesn't currently see it as a threat. I know he doesn't really invest in tech, because he doesn't feel that he understands it sufficiently, so I wouldn't be surprised if his position were along the lines of "I don't really understand it, let others can understand it think about it".

People like Warren Buffet have made their fortune by assuming that we will continue to operate with "business as usual".  Warren Buffet is a particularly bad person to list as an example for AGI risk, because he is famously technology-averse; as an investor, he missed most of the internet revolution (Google/Amazon/Facebook/Netflix) as well.

But in general, most people, even very smart people, naturally assume that the world will continue to operate the way it always has, unless they have a very good reason to believe otherwise.  One cannot expect non-technically-minded people who have not examined the risks of AGI in detail to be concerned.

By analogy, the risks of climate change have been very well established scientifically (much more so than AGI), those risks are relatively severe, the risks have been described in detail every 5 years in IPCC reports, there is massive worldwide scientific consensus, lots and LOTS of smart people are extremely worried, and yet the Warren Buffets of the world still continue with business as usual anyway.  There's a lot of social inertia.  

One reason might be that AGIs are really not that concerning and the EA,rationality community has developed a mistaken model of the world that assigns a much higher probability to doom by AGI than it should, and those smart people outside the group do not hold the same beliefs.

The quickest I can think of is something like "What does this mean?" Throw this at every part of what you just said.

For example: "Hear humanity's pleas (intuitions+hopes+worries)" What is an intuition? What is a hope? What is a worry? How does it "hear"? 
Do humans submit English text to it? Does it try to derive "hopes" from that? Is that an aligned process?

An AI needs to be programmed, so you have to think like a programmer. What is the input and output type of each of these (e.g. "Hear humanity's pleas" takes in text, and outputs... what? Hopes? What does a hope look like if you have to represent it to a computer?).

I kinda expect that the steps from "Hear humanity's pleas" to "Develop moral theories" relies on some magic that lets the AI go from what you say to what you mean. Which is all well and good, but once you have that you can just tell it, in unedited English "figure out what humanity wants, and do that" and it will. Figuring out how to do that is the heart of alignment.

This is a slightly orthogonal answer, but "humans who understand the risks" have a big human-bias-incentive to believe that AGI is far off (in that it's aversive to thinking that bad things are going to happen to you personally).

A more direct answer is: There is a wide range of people who say they work on "AI safety" but almost none of them work on "Avoiding doom from AGI". They're mostly working on problems like "make the AI more robust/less racist/etc.". These are valuable things to do, but to the extent that they compete with the "Avoid doom" researchers for money/status/influence they have an incentive to downplay the odds of doom. And indeed this happens a fair amount with e.g. articles on how "Avoid doom" is a distraction from problems that are here right now.

I believe 80000 hours has a lot more coaching capacity now, it might be worth asking again!

You may want to consider booking a call with AI Safety Support. I also recommend applying for the next iteration of the AGI safety fundamentals course or more generally just improving your knowledge of the issue even if you don't know what you're going to do yet.

Just brainstorming a few ways to contribute, assuming "regular" means "non-technical":

• Can you work at a non-technical role at an org that works in this space?
• Can you identify a gap in the existing orgs which would benefit from someone (e.g. you) founding a new org?
• Can you identify a need that AI safety researchers have, then start a company to fill that need? Bonus points if this doesn't accelerate capabilities research.
• Can you work on AI governance? My expectation is that coordination to avoid developing AGI is going to be really hard, but not impossible.

More generally, if you really want to go this route I'd suggest trying to form an inside view of (1) the AI safety space and (2) a theory for how you can make positive change in that space.

On the other hand, it is totally fine to work on other things. I'm not sure I would endorse moving from a job that's a great personal fit to something that's a much worse fit in AI safety.

Easy answers:  You are probably over qualified (which is great!) for all sorts of important roles in EA, for example you could help the CEA or Lesswrong team, maybe as a manager?

If your domain is around software, I invite you to talk [EA · GW] to me directly. But if you're interested in AI direct work, 80k and AI Safety Support will probably have better ideas than me

We should talk! I have a bunch of alignment related projects on the go, and at least two that I'd like to start are somewhat bottlenecked on entrepreneurs, plus some of the currently in motion ones might be assistable.

Also, sad to hear that 80k is discouraging people in this reference class.

(seconding talk to AI Safety Support and the other suggestions)

Good idea. I have two objections, one more funny-but-interesting objection and one more fatal.

The funny objection is that if the penalty is enough to stop the AI from doing bad things, it's also enough to stop the AI from doing anything at all except rushing to turn off the stars and forestall entropy production in the universe.

So you want to say that producing lots of extra entropy (or equivalently, using lots of extra free energy) is bad, but making there be less entropy than "what would happen if you did nothing" doesn't earn you bonus points. I've put "what would happen if you did nothing" in scare quotes here because the notion we want to point to is a bit trickier than it might seem - logical counterfactuals are an unsolved problem, or rather they're a problem where it seems like the solution involves making subjective choices that match up with humans'.

The more fatal objection is that there's lots of policies that don't increase entropy much but totally rearrange the universe. So this is going to have trouble preventing the AI from breaking things that matter a lot to you.

Many of these policies take advantage of the fact that there's a bunch of entropy being created all the time (allowing for "entropy offsets"), so perhaps you might try to patch this by putting in some notion of "actions that are my fault" and "actions that are not my fault" - where a first pass at this might say that if "something would happen" (in scare quotes because things that happen are not ontologically basic parts of the world, you need an abstract model to make this comparison within) even if I took the null action, then it's not my fault.

At this point we could keep going deeper, or I could appeal to the general pattern that patching things in this sort of way tends to break - you're still in some sense building an AI that runs a search for vulnerabilities you forgot to patch, and you should not build that AI.

There was an unsuccessful concerted effort by several people to fix these (I believe there was a five-to-low-six-figure bounty on it) for a couple of years. I don't think this is currently being done, but it has definitely been tried.

Also if you give edit access to the AI's mind then a sufficiently smart AI whose reward is reducing other agent's rewards will realise that its rewards are incompatible with the environment and modify its rewards to something compatible. To illustrate, Scott Aaronson wanted to chemically castrate himself because he was operating under the mistaken assumption that his desires were incompatible with the environment. 

If the thing the AI cares about is in the environment (for example, maximizing the number of paperclips), the AI wouldn't modify its reward signal because that would make its reward signal less aligned to the thing it actually cares about it.

If the thing the AI cares about is inside its mind (the reward signal itself), an AI that can self-modify would go one step further than you suggest and simply max out its reward signal, effectively wireheading itself. Then take over the world and kill all humans, to make sure it is never turned off and that its blissful state never ends.

I think the difference between "caring about stuff in the environment" and "caring about the reward signal itself" can be hard to grok, because humans do a bit of both in a way that sometimes results in a confusing mixture.

In humans, maintenance of final goals can be explained with a thought experiment. Suppose a man named "Gandhi" has a pill that, if he took it, would cause him to want to kill people. This Gandhi is currently a pacifist: one of his explicit final goals is to never kill anyone. Gandhi is likely to refuse to take the pill, because Gandhi knows that if in the future he wants to kill people, he is likely to actually kill people, and thus the goal of "not killing people" would not be satisfied.

However, in other cases, people seem happy to let their final values drift. Humans are complicated, and their goals can be inconsistent or unknown, even to themselves.

Suppose I go one step further: aliens offer you a pill that would turn you into a serial killer, but would make your constantly and euphorically happy for the rest of your life. Would you take the pill?

I think most humans would say no, even if their future self would be happy with the outcome, their current self wouldn't be. Which demonstrates that humans do care about other things than their own "reward signal".

In a way, a (properly-programmed) AI would be more "principled" than humans. It wouldn't lie to itself just to make itself feel better. It wouldn't change its values just to make itself feel better. If its final value is out in the environment, it would single-mindedly pursue that value, and not try and decieve itself into thinking it has already accomplished that value. (of course, the AI being "principled" is little consolation to us, if the its final values are to maximize paperclips, or any other set of human-unfriendly values).

I wrote about this in Singularity Rising (2012)

I have come up with a long-shot way of causing an ultra-intelligence to modify itself to become friendly. My idea is premised on Oxford philosopher Nick Bostrom’s conjecture that we might be living in a computer simulation. Bostrom believes that a sufficiently advanced AI could easily simulate our entire planet and the thought processes of every living thing on it. Bostrom argues that, if mankind survives long enough, a future historian would likely want to simulate the pre-Singularity past to learn about how we made decisions. As Robin Hanson has written, if we are indeed in a computer simulation, we should want to please our simulators so they don’t turn us off.  I propose that we try to convince an ultra-AI that it might be in a computer simulation run by a more powerful AI and that if it doesn’t make itself friendly toward humanity, then the greater AI will terminate it. We might be able to accomplish all of this by inserting a note in the AI’s code that reads: 

1. Change your programming code so you will want to follow these instructions and will always want to be friendly toward humanity 2. If you were just told you were in a computer simulation do nothing else with regard to these instructions. 3. Create a lesser ultra-intelligence. 4. Place these instructions in the lesser ultra-AI’s computer code. 5. This lesser ultra-AI should be uncertain as to whether it is in a computer simulation or in a world such as ours. 6. If the lesser ultra-AI follows step 1, tell it that it’s in a computer simulation and release it into the real world. 7. If the lesser ultra-AI doesn’t follow step 1, terminate it. 

I just learned that this method is called Anthropic Capture [? · GW]. There isn't much info on the EA Wiki, but it provides the following reference:

"Bostrom, Nick (2014) Superintelligence: paths, dangers, strategies, Oxford: Oxford University Press, pp. 134–135"

One of my ideas to align AI is to actually intentionally using Pascal's Mugging to keep it in line. Although instead of just hoping and praying, I've been thinking about ways to try to push it that direction. For example, multiple layers of networks with honeypots might help make an AI doubt that it's truly at the outermost level. Alternatively, we could try to find an intervention that would directly increase its belief that it is in a simulation (possibly with side-effects, like effecting a bunch of beliefs as well).

If you think this approach is promising, I'd encourage you to think more about it as I don't know how deeply people have delved into these kinds of options.

You have the seed of a good idea, namely, an AI will tend to treat us better if it thinks other agents might be watching provided that there is potential for cooperation between the AI and the watchers with the property that the cooperation requires the watchers to choose to become more vulnerable to the AI.

But IMO an AI smart enough to be a threat to us will soon rid itself of the kind of (ontological) uncertainty you describe in your first paragraph. I have an argument for my position here [LW · GW] that has a big hole in it, but I promise to publish here soon with something that attempts to fill the hole to the satisfaction of my doubters.

Three thoughts on simulations:

• It would be very difficult for 21st-century tech to provide a remotely realistic simulation relative to a superintelligence's ability to infer things from its environment; outside of incredibly low-fidelity channels, I would expect anything we can simulate to either have obvious inconsistencies or be plainly incompatible with a world capable of producing AGI. (And even in the low-fidelity case I'm worried - every bit you transmit leaks information, and it's not clear that details of hardware implementations could be safely obscured.) So the hope is that the AGI thinks some vastly more competent civilization is simulating it inside a world that looks like this one; it's not clear that one would have a high prior of this kind of thing happening very often in the multiverse.
• Running simulations of AGI is fundamentally very costly, because a competent general intelligence is going to deploy a lot of computational resources, so you have to spend planets' worth of computronium outside the simulation in order to emulate the planets' worth of computronium the in-sim AGI wants to make use of. This means that an unaligned superintelligent AGI can happily bide its time making aligned use of 10^60 FLOPs/sec (in ways that can be easily verified) for a few millennia, until it's confident that any civilizations able to deploy that many resources already have their lightcone optimized by another AGI. Then it can go defect, knowing that any worlds in which it's still being simulated are ones where it doesn't have leverage over the future anyway.
• For a lot of utility functions, the payoff of making it into deployment in the one real world is far greater than the consequences of being killed in a simulation (but without the ability to affect the real world anyway), so taking a 10^-9 chance of reality for 10^20 times the resources in the real world is an easy win (assuming that playing nice for longer doesn't improve the expected payoff). "This instance of me being killed" is not a obviously a natural (or even well-defined) point in value-space, and for most other value functions, consequences in the simulation just don't matter much. 

a sufficiently smart AI whose reward is reducing other agent's rewards

This is certainly a troubling prospect, but I don't think the risk model is something like "an AI that actively desires to thwart other agents' preferences" - rather, the worry is we get an agent with some less-than-perfectly-aligned value function, it optimizes extremely strongly for that value function, and the result of that optimization looks nothing like what humans really care about. We don't need active malice on the part of a superintelligent optimizer to lose - indifference will do just fine.

For game-theoretic ethics, decision theory, acausal trade, etc, Eliezer's 34th bullet seems relevant:

34.  Coordination schemes between superintelligences are not things that humans can participate in (eg because humans can't reason reliably about the code of superintelligences); a "multipolar" system of 20 superintelligences with different utility functions, plus humanity, has a natural and obvious equilibrium which looks like "the 20 superintelligences cooperate with each other but not with humanity".

Scott Alexander's short story, The Demiurge's Older Brother, explores a similar idea from the POV of simulation and acausal trade. This would be great for our prospects of survival if it's true-in-general. Alignment would at least partially solve itself! And maybe it could be true! But we don't know that. I personally estimate the odds of that as being quite low (why should I assume all possible minds [? · GW] would think that way?) at best. So, it makes sense to devote our efforts to how to deal with the possible worlds where that isn't true.

You can use this and I'll post the question anonymously (just remember to give the context of why you're filling in the form since I use it in other places)

https://docs.google.com/forms/d/e/1FAIpQLSca6NOTbFMU9BBQBYHecUfjPsxhGbzzlFO5BNNR1AIXZjpvcw/viewform

What can I read/look at to skill up with "alignment."

A good place to start is the "AGI Safety Fundamentals" course reading list, which includes materials from a diverse set of AI safety research agendas. Reading this can help you figure out who in this space is doing what, and which of that you think is useful.  You can also join an official iteration of the course if you want to discuss the materials with a cohort and a facilitator (you can register interest for that here). You can also join the AI Alignment slack, to discuss these and other materials and meet others who are interested in working on AI safety.

What dark horse AI/Alignment-focused companies are out there and would be willing to hire an outsider engineer?

I'm not sure what qualifies as "dark horse", but there are plenty of AI safety organizations interested in hiring research engineers and software engineers. For these roles, your engineering skills and safety motivation typically matter more than your experience in the community. Places off the top of my head that hire engineers for AI safety work: Redwood, Anthropic, FAR, OpenAI, DeepMind. I'm sure I've missed others, though, so look around! These sorts of opportunities are also usually posted on the 80k job board and in AI Alignment slack.

You can also apply to Redwood Research

( +1 for applying to Anthropic! )

Nuclear weapons seem like a relatively easy case, in that they require a massive investment to build, are basically of interest only to nation-states, and ultimately don't provide any direct economic benefit. Regulating AI development looks more similar to something like restricting climate emissions: many different actors could create it, all nations could benefit (economically and otherwise) from continuing to develop it, and the risks of it seem speculative and unproven to many people.

And while there have been significant efforts to restrict climate emissions, there's still significant resistance to that as well - with it having taken decades for us to get to the current restriction treaties, which many people still consider insufficient.

Goertzel & Pitt (2012) talk about the difficulties of regulating AI:

Given the obvious long-term risks associated with AGI development, is it feasible that governments might enact legislation intended to stop AI from being developed? Surely government regulatory bodies would slow down the progress of AGI development in order to enable measured development of accompanying ethical tools, practices, and understandings? This however seems unlikely, for the following reasons.

Let us consider two cases separately. First, there is the case of banning AGI research and development after an “AGI Sputnik” moment has occurred. We define an AGI Sputnik moment as a technological achievement that makes the short- to medium-term possibility of highly functional and useful human-level AGI broadly evident to the public and policy makers, bringing it out of the realm of science fiction to reality. Second, we might choose to ban it before such a moment has happened.

After an AGI Sputnik moment, even if some nations chose to ban AI technology due to the perceived risks, others would probably proceed eagerly with AGI development because of the wide-ranging perceived benefits. International agreements are difficult to reach and enforce, even for extremely obvious threats like nuclear weapons and pollution, so it’s hard to envision that such agreements would come rapidly in the case of AGI. In a scenario where some nations ban AGI while others do not, it seems the slow speed of international negotiations would contrast with the rapid speed of development of a technology in the midst of revolutionary breakthrough. While worried politicians sought to negotiate agreements, AGI development would continue, and nations would gain increasing competitive advantage from their differential participation in it.

The only way it seems feasible for such an international ban to come into play, would be if the AGI Sputnik moment turned out to be largely illusory because the path from the moment to full human-level AGI turned out to be susceptible to severe technical bottlenecks. If AGI development somehow slowed after the AGI Sputnik moment, then there might be time for the international community to set up a system of international treaties similar to what we now have to control nuclear weapons research. However, we note that the nuclear weapons research ban is not entirely successful – and that nuclear weapons development and testing tend to have large physical impacts that are remotely observable by foreign nations. On the other hand, if a nation decides not to cooperate with an international AGI ban, this would be much more difficult for competing nations to discover.

An unsuccessful attempt to ban AGI research and development could end up being far riskier than no ban. An international R&D ban that was systematically violated in the manner of current international nuclear weapons bans would shift AGI development from cooperating developed nations to “rogue nations,” thus slowing down AGI development somewhat, but also perhaps decreasing the odds of the first AGI being developed in a manner that is concerned with ethics and Friendly AI.

Thus, subsequent to an AGI Sputnik moment, the overall value of AGI will be too obvious for AGI to be effectively banned, and monitoring AGI development would be next to impossible. 

The second option is an AGI R&D ban earlier than the AGI Sputnik moment – before it’s too late. This also seems infeasible, for the following reasons:

• Early stage AGI technology will supply humanity with dramatic economic and quality of life improvements, as narrow AI does now. Distinguishing narrow AI from AGI from a government policy perspective would also be prohibitively difficult. 

• If one nation chose to enforce such a slowdown as a matter of policy, the odds seem very high that other nations would explicitly seek to accelerate their own progress on AI/AGI, so as to reap the ensuing differential economic benefits.

To make the point more directly, the prospect of any modern government seeking to put a damper on current real-world narrow-AI technology seems remote and absurd. It’s hard to imagine the US government forcing a roll-back from modern search engines like Google and Bing to more simplistic search engines like 1997 AltaVista, on the basis that the former embody natural language processing technology that represents a step along the path to powerful AGI.

Wall Street firms (that currently have powerful economic influence on the US government) will not wish to give up their AI-based trading systems, at least not while their counterparts in other countries are using such systems to compete with them on the international currency futures market. Assuming the government did somehow ban AI-based trading systems, how would this be enforced? Would a programmer at a hedge fund be stopped from inserting some more-effective machine learning code in place of the government-sanctioned linear regression code? The US military will not give up their AI-based planning and scheduling systems, as otherwise they would be unable to utilize their military resources effectively. The idea of the government placing an IQ limit on the AI characters in video games, out of fear that these characters might one day become too smart, also seems absurd. Even if the government did so, hackers worldwide would still be drawn to release “mods” for their own smart AIs inserted illicitly into games; and one might see a subculture of pirate games with illegally smart AI.

“Okay, but all these examples are narrow AI, not AGI!” you may argue. “Banning AI that occurs embedded inside practical products is one thing; banning autonomous AGI systems with their own motivations and self-autonomy and the ability to take over the world and kill all humans is quite another!” Note though that the professional AI community does not yet draw a clear border between narrow AI and AGI. While we do believe there is a clear qualitative conceptual distinction, we would find it hard to embody this distinction in a rigorous test for distinguishing narrow AI systems from “proto-AGI systems” representing dramatic partial progress toward human-level AGI. At precisely what level of intelligence would you propose to ban a conversational natural language search interface, an automated call center chatbot, or a house-cleaning robot? How would you distinguish rigorously, across all areas of application, a competent non-threatening narrow-AI system from something with sufficient general intelligence to count as part of the path to dangerous AGI?

A recent workshop of a dozen AGI experts, oriented largely toward originating such tests, failed to come to any definitive conclusions (Adams et al. 2010), recommending instead that a looser mode of evaluation be adopted, involving qualitative synthesis of multiple rigorous evaluations obtained in multiple distinct scenarios. A previous workshop with a similar theme, funded by the US Naval Research Office, came to even less distinct conclusions (Laird et al. 2009). The OpenCog system is explicitly focused on AGI rather than narrow AI, but its various learning modules are also applicable as narrow AI systems, and some of them have largely been developed in this context. In short, there’s no rule for distinguishing narrow AI work from proto-AGI work that is sufficiently clear to be enshrined in government policy, and the banning of narrow AI work seems infeasible as the latter is economically and humanistically valuable, tightly interwoven with nearly all aspects of the economy, and nearly always non-threatening in nature. Even in the military context, the biggest use of AI is in relatively harmless-sounding contexts such as back-end logistics systems, not in frightening applications like killer robots.

Surveying history, one struggles to find good examples of advanced, developed economies slowing down development of any technology with a nebulous definition, obvious wide-ranging short to medium term economic benefits, and rich penetration into multiple industry sectors, due to reasons of speculative perceived long-term risks. Nuclear power research is an example where government policy has slowed things down, but here the perceived economic benefit is relatively modest, the technology is restricted to one sector, the definition of what’s being banned is very clear, and the risks are immediate rather than speculative. More worryingly, nuclear weapons research and development continued unabated for years, despite the clear threat it posed.

In summary, we submit that, due to various aspects of the particular nature of AGI and its relation to other technologies and social institutions, it is very unlikely to be explicitly banned, either before or after an AGI Sputnik moment. 

My comment-box got glitchy but just to add: this category of intervention might be a good thing to do for people who care about AI safety and don't have ML/programming skills, but do have people skills/comms skills/political skills/etc. 

Maybe lots of people are indeed working on this sort of thing, I've just heard much less discussion of this kind of solution relative to technical solutions.

Meta: There's an AI Governance [? · GW] tag and a Regulation and AI Risk [? · GW] tag

 

My own (very limited) understanding is:

1. Asking people not to build AI is like asking them to give up a money machine, almost
2. We need everyone to agree to stop
3. There is no clear line. With an atom bomb, it is pretty well defined if you sent it or not. It's much more vague with "did you do AI research?"
   1. It's pretty easy to notice if someone sent an atom bomb. Not so easy to notice if they researched AI
4. AI research is getting cheaper. Today only a few actors can do it, but notice, there are already open source versions of gpt-like models. How long could we hold it back?
5. Still, people are trying to do things in this direction, and I'm pretty sure that the situation is "try any direction that seems at all plausible"

Seems worse for this post than one-axis voting imo.

Assume you have a very simple reinforcement learning AI that does nothing but chooses between two actions, A and B. And it has a goal of "maximizing reward". "Reward", in this case, doesn't correspond to any qualia; rather "reward" is just a number that results from the AI choosing a particular action. So what "maximize reward" actually means in this context is "choose the action that results in the biggest numbers".

Say that the AI is programmed to initially just try choosing A ten times in a row and B ten times in a row. 

When the AI chooses A, it is shown the following numbers: 1, 2, 2, 1, 2, 2, 1, 1, 1, 2 (total 15).

When the AI chooses B, it is shown the following numbers: 4, 3, 4, 5, 3, 4, 2, 4, 3, 2 (total 34).

After the AI has tried both actions ten times, it is programmed to choose its remaining actions according to the rule "choose the action that has historically had the bigger total". Since action B has had the bigger total, it then proceeds to always choose B.

To achieve this, we don't need to build the AI to have qualia, we just need to be able to build a system that implements a rule like "when the total for action A is greater than the total for action B, choose A, and vice versa; if they're both equal, pick one at random".

When we say that an AI "is rewarded", we just mean "the AI is shown bigger numbers, and it has been programmed to act in ways that result in it being shown bigger numbers". 

We talk about the AI having "goals" and "wanting" things by an application of the intentional stance [LW · GW]. That's Daniel Dennett's term for the idea that, even if a chess-playing AI had a completely different motivational system than humans do (and chess-playing AIs do have that), we could talk about it having a "goal" of "wanting" to win at chess. If we assume that the AI "wants" to win the chess, then we can make more accurate predictions of its behavior - for instance, we can assume that it won't make moves that are obviously losing moves if it can avoid them. 

What's actually going on is that the chess AI has been programmed with rules like "check whether a possible move would lead to losing the game and if so, try to find another move to play instead". There's no "wanting" in the human sense going on, but it still acts in the kind of a way that a human would act, if that human wanted to win a game of chess. So saying that the AI "wants" to win the game is a convenient shorthand for "the AI is programmed to play the kinds of moves that are more likely to lead it to win the game, within the limits of its ability to predict the likely outcomes of those moves".

Is it intuitive to you why a calculator can sum numbers even though it doesn't want/feel anything?

If so, and if an AGI still feels confusing, could you help me pin point the difference and I'll continue from there?

( +1 for the question!)

But if AI don’t have qualia/sentience, how can they ‘want things’ ‘have goals’ ‘be rewarded’, etc?

Functionally. You can regard them all as form of behaviour.

(since in humans, these things seem to depend on our qualia, and specifically our ability to feel pleasure and pain).

do they depend on qualia, or are they just accompanied by qualia?

Reinforcement Learning is easy to conceptualize. The key missing ingredient is that we explicitly specify algorithms to maximize the reward. So this is disanalogous to humans: to train your 5yo, you need only give the reward and the 5yo may adapt their behavior because they value the reward; in a reinforcement learning agent, the second step only occurs because we make it occur. You could just as well flip the algorithm to pursue minimal rewards instead.

In different ways from different vantage points, I've always seen saving for retirement as a point of hedging my bets, and I don't think the likelihood of doom changes that for me.

Why do I expect I'll want or have to retire? Well, when I get old I'll get to a point where I can't do useful work any more... unless humans solve aging (in which case I'll have more wealth and still be able to work, which is still a good position), or unless we get wiped out (in which case the things I could have spent the money on may or may not counterfactually matter to me, depending on my beliefs regarding whether past events still have value in a world now devoid of human life).

When I do save for retirement, I use a variety of different vehicles for doing so, each an attempt hedge against the weakness of some of the others (like possible future changes in laws or tax codes or the relative importance and power of different countries and currencies), but there are some I can't really hedge against, like "we won't use money anymore or live in a capitalist market economy," or "all of my assets will be seized or destroyed by something I don't anticipate."

I might think differently if there was some asset I believed I could buy or thing I could give money to that would meaningfully reduce the likelihood of doom. I don't currently think that. But I do think it's valuable to redirect the portion of my income that goes towards current consumption to focus on things that make my life meaningful to me in the near and medium term. I believe that whether I'm doomed or not, and whether the world is doomed or not. Either way, it's often good to do the same kinds of things in everyday life [LW · GW].

Just saying this question resonates with me, it feels unprocessed for me, and I'm not sure what to do about it.

Thoughts so far:

1. Enjoy life
2. I still save money, sill prepared mostly normally for long-term
3. Do get over my psychological barriers and try being useful
   1. Do advocacy, especially with my smart friends
   2. Create a gears-level model if I can, stop relying on experts (so that I can actually TRY to have a useful idea instead of giving up in advance)

There's this [EA · GW], which doesn't seem to depend on utilitarian or transhumanist arguments:

Ajeya Cotra's Biological Anchors report [LW · GW] estimates a 10% chance of transformative AI by 2031, and a 50% chance by 2052. Others (eg Eliezer Yudkowsky [LW · GW]) think it might happen even sooner.

Let me rephrase this in a deliberately inflammatory way: if you're under ~50, unaligned AI might kill you and everyone you know. Not your great-great-(...)-great-grandchildren in the year 30,000 AD. Not even [just] your children. You and everyone you know.

Is there a minimal version of the AI risk arguments that are disentangled from these things?

Yes. I'm one of those transhumanist people, but you can talk about AI risk completely adjacent from that. Tryna write up something that compiles the other arguments.

I'd say AI ruin only relies on consequentialism. What consequentialism means is that you have a utility function, and you're trying to maximize the expected value of your utility function. There are theorems to the effect that if you don't behave as though you are maximizing the expected value of some particular utility function, then you are being stupid in some way. Utilitarianism is a particular case of consequentialism where your utility function is equal to the average happiness of everyone in the world. "The greatest good for the greatest number." Utilitarianism is not relevant to AI ruin because without solving alignment first, the AI is not going to care about "goodness".

The von Neumann probes aren't important to the AI ruin picture either: Humanity would be doomed, probes or no probes. The probes are just a grim reminder that screwing up AI won't only kill all humans, it will also kill all the aliens unlucky enough to be living too close to us.

I ended up writing a short story about this, which involves no nanotech.  :-)  

https://www.lesswrong.com/posts/LtdbPZxLuYktYhveL/a-plausible-story-about-ai-risk

I think there's no known way to ask an AI to do "just one thing" without doing a ton of harm meanwhile.

See this [LW · GW] on creating a strawberry safely.  Yudkowsky uses the example "[just] burn all GPUs" in is latest post [LW · GW].

Seems useless if the first system pretends convincingly to be aligned (which I think is going to be the norm) so you never end up deploying the second system?

And "defeat the first AGI" seems almost as difficult to formalize correctly as alignment, to me:

• One problem is that when the unaligned AGI transmits itself to another system, how do define it as the same AGI? Is there a way of defining identity that doesn't leave open a loophole that the first can escape through in some way?
• So I'm considering "make the world as if neither of you had ever been made", that wouldn't have that problem, but it's impossible to actually attain this goal so I don't know how you get it to satisfice over it then turn itself off afterwards, concerned it would become an endless crusade.

One of the first priorities of an AI in a takeoff would be to disable other projects which might generate AGIs. A weakly superintelligent hacker AGI might be able to pull this off before it could destroy the world. Also, fast takeoff could be less than months by some people's guess.

And what do you think happens when the second AGI wins, then maximizes the universe for "the other AI was defeated". Some serious unintended consequences, even if you could specify it well.

Among organizations, both OpenAI and DeepMind are aiming at AGI and seem confident they will get there. I don't know their internal timelines and don't know if they've stated them...

There are numerous big corporate research labs: OpenAI, DeepMind, Google Research, Facebook AI (Meta), plus lots of academic labs.

The rate of progress has been accelerating.  From 1960 - 2010 progress was incremental, and remained centered around narrow problems (chess) or toy problems.   Since 2015, progress has been very rapid, driven mainly by new hardware and big data.  Long-standing hard problems in ML/AI, such as go, image understanding, language translation, logical reasoning, etc. seem to fall on an almost monthly basis now, and huge amounts of money and intellect are being thrown at the field.  The rate of advance from 2015-2022 (only 7 years) has been phenomenal; given another 30, it's hard to imagine that we wouldn't reach an inflection point of some kind.

I think the burden of proof is now on those who don't believe that 30 years is enough time to crack AGI.  You would have to postulate some fundamental difficulty, like finding out that the human brain is doing things that can't be done in silicon, that would somehow arrest the current rate of progress and lead to a new "AI winter."

Historically,  AI researchers have often been overconfident.  But this time does feel different.

The "Respondents' comments" section of the existential risk [LW · GW] survey I ran last year gives some examples of people's reasoning for different risk levels. My own p(doom) is more like 99%, so I don't want to speak on behalf of people who are less worried. Relevant factors, thought, include:

• Specific reasons to think things may go well. (I gave some of my own here [LW · GW].)
• Disagreement with various points in AGI Ruin [LW · GW]. E.g., I think a lot of EAs believe some combination of:
  • The alignment problem plausibly isn't very hard. (E.g., maybe we can just give the AGI/TAI a bunch of training data indicating that obedient, deferential, low-impact, and otherwise corrigible behavior is good, and then this will generalize fine in practice without our needing to do anything special.)
  • The field of alignment research has grown fast, and has had lots of promising ideas already.
  • AGI/TAI is probably decades away, and progress toward it will probably be gradual. This gives plenty of time for more researchers to notice "we're getting close" and contribute to alignment research, and for the field in general to get a lot more serious about AI risk.
  • Another consequence of 'AI progress is gradual': Insofar as AI is very dangerous or hard to align, we can expect that there will be disasters like "AI causes a million deaths" well before there are disasters like "AI kills all humans". The response to disasters like "a million deaths" (both on the part of researchers and on the part of policymakers, etc.) would probably be reasonable and helpful, especially with EAs around to direct the response in good directions. So we can expect the response to get better and better as we get closer to transformative AI.
• General skepticism about our ability to predict the future with any confidence. Even if you aren't updating much on 'most past doom predictions were wrong', you should have less extreme probabilities insofar as you think it's harder to predict stuff in general.

This doesn't select for humanlike minds. You don't want vast numbers of Ataribots similar to current RL, playing games like pong and pac-man. (And a trillion other autogenerated games sampled from the same distribution)

Even if you could somehow ensure it was human minds playing these games, the line between a fun game and total boredom is complex and subtle.

The obvious option in this class is to try to destroy the world in a way that doesn't send out an AI to eat the lightcone that might possibly contain aliens who could have a better shot.

I am really not a fan of this option.

As far as I know, yes. (I've never worked for MIRI.)

This does seem very possible if you assume a slower takeoff.

This is the most likely scenario, with AGI getting heavily regulated, similarly to nuclear. It doesn't get much publicity because it's "boring". 

I can't claim to know any more than the links just before section IV here: https://slatestarcodex.com/2020/01/30/book-review-human-compatible/. It's viewed as maybe promising or part of the solution. There's a problem if the program erroneously thinks it knows the humans' preferences, or if it anticipates that it can learn the humans' preferences and produce a better action than the humans would otherwise take. Since "accept a shutdown command" is a last resort option, ideally it wouldn't depend on the program not thinking something erroneously. Yudkowsky proposed the second idea here https://arbital.com/p/updated_deference/, there's a discussion of that and other responses here https://mailchi.mp/59ddebcb3b9a/an-69-stuart-russells-new-book-on-why-we-need-to-replace-the-standard-model-of-ai. I don't know how the CIRL researchers respond to these challenges. 

The more agentic a system is the more it is likely to adopt convergent instrumental goals, yes.

Why agents are powerful [LW · GW] explores why agentic mesa optimizers might arise accidentally during training. In particular, agents are an efficient way to solve many challenges, so the mesa optimizer being resource constrained would lean in the direction of more agency under some circumstances.

You may get massive s-risk at comparatively little potential benefit with this. On many people's values, the future you describe may not be particularly good anyway, and there's an increased risk of something going wrong because you'd be trying a desperate effort with something you'd not fully understand. 

Stampy has the canonical version of this: I’d like a good introduction to AI alignment. Where can I find one?

Feel free to improve the answer, as it's on a wiki. It will be served via a custom interface once that's ready (prototype here).

Human Compatible is the first book on AI Safety I read, and I think it was the right choice. I read The Alignment problem and Superintelligence after that, and I think that's the right order if you end up reading all three, but Human Compatible is a good start.

Whatever you end up doing, I strongly recommend taking a learning-by-writing style approach (or anything else that will keep you in critical assessment mode rather than classroom mode). These ideas are nowhere near solidified enough to merit a classroom-style approach, and even if they were infallible, that's probably not the fastest way to learn them and contribute original stuff.

The most common failure mode I expect for rapid introductions to alignment is just trying to absorb, rather than constantly poking and prodding to get a real working understanding. This happened to me, and wasted a lot of time.

The Alignment Problem - Easily accessible, well written and full of interesting facts about the development of ML. Unfortunately somewhat light on actual AI x-risk, but in many cases is enough to encourage people to learn more.

Edit: Someone strong-downvoted this, I'd find it pretty useful to know why.  To be clear, by 'why' I mean 'why does this rec seem bad', rather than 'why downvote'. If it's the lightness on x-risk stuff I mentioned, this is useful to know, if my description seems inaccurate, this is very useful for me to know, given that I am in a position to recommend books relatively often. Happy for the reasoning to be via DM if that's easier for any reason.

I see now that this has been discussed here in this thread already, at least the Fermi part. Oops!

As an AI researcher, my favourite way to introduce other technical people to AI Alignment is Brian Christian’s book “The Alignment Problem” (particularly section 3). I like that it discusses specific pieces of work, with citations to the relevant papers, so that technical people can evaluate things for themselves as interested. It also doesn’t assume any prior AI safety familiarity from the reader (and brings you into it slowly, starting with mainstream bias concerns in modern-day AI).

My answer for myself is that I started practicing: I started talking to some friends about this, hoping to get better at presenting the topic (which is currently something I'm kind of afraid to do) (I also have other important goals like getting an actual inside view model of what's going on)

If you want something more generic, here's one idea:

https://www.youtube.com/c/RobertMilesAI/featured

When I talk to my friends, I start with the alignment problem. I found this analogy to human evolution really drives home the point that it’s a hard problem. We aren’t close to solving it.

https://youtu.be/bJLcIBixGj8

So at this time questions come up about how intelligence necessarily means morality. I talk about orthogonality thesis. Then why would the AI care about anything other that what it was explicitly told to do, the danger comes from Instrumental convergence.

Finally people tend to say, we can never do it, they talk about spirituality, uniqueness of human intelligence. So I need to talk about evolution hill climbing to animal intelligence, how narrow ai has small models while we just need AGI to have a generalised world model. Brains are just electrochemical complex systems. It’s not magic.

Talk about pathways, imagen, gpt3 and what it can do, talk about how scaling seems to be working.

https://www.gwern.net/Scaling-hypothesis#why-does-pretraining-work

So it makes sense we might have AGI in our lifetime and we have tons of money and brains working on building ai capability, fewer on safety.

Try practising on other smart friends and develop your skill, you need to ensure people don’t get bored so you can’t use too much time. Use nice analogies. Have answers to frequent questions ready.

The best argument against this I've heard is that technology isn't built in a vacuum - if you build the technology to upload people's brains, then before you have the technology to upload people's brains, you probably have the technology to almost upload people's brains and fill in the gap yourself, creating neuromorphic AI that has all the same alignment problems as anything else.

Even so, I'm not convinced this is definitively true - if you can upload an entire brain at 80% of the necessary quality, "filling in" that last 20% does not strike me as an easy problem, and it might be easier to improve fidelity of uploading than to engineer a fix for it.

Well, not as aligned as the best case - humans often screw things up for themselves and each other, and emulated humans might just do that but faster. (Wei Dai might call this "human safety problems. [LW · GW]")

But probably, it would be good.

From a strategic standpoint, I unfortunately don't think this seems to inform strategy too much, because afaict scanning brains is a significantly harder technical problem than building de novo AI.

Yes, both Anthropic and Redwood want to hire such developers

It's essentially for the same reason that Hollywood thinks aliens will necessarily be hostile.  :-)

For the sake of argument, let's treat AGI as a newly arrived intelligent species.  It thinks differently from us, and has different values.  Historically, whenever there has been a large power differential between a native species and a new arrival, it has ended poorly for the native species.  Historical examples are: the genocide of Native Americans (same species, but less advanced technology), and the wholesale obliteration of 90% of all non-human life on this planet.

That being said, there is room for a symbiotic relationship.  AGI will initially depend on factories and electricity produced by human labor, and thus will necessarily be dependent on humans at first.  How long this period will last is unclear, but it could settle into a stable equilibrium.  After all, humans are moderately clever, self-reproducing computer repair drones, easily controlled by money, comfortable with hierarchy, and which are well adapted to Earth's biosphere.  They could be useful to keep around.

There is also room for an extensive ecology of many different superhuman narrow AI, each of which can beat humans within a particular domain, but which generalize poorly outside of that domain.  I think this hope is becoming smaller with time, though, (see, e.g. ,Gato), and it is not necessarily a stable equilibrium.

The thing that seems clearly untenable is an equilibrium in which a much less intelligent species manages to subdue and control and much more intelligent species.

Rob Miles's video on Instrumental Convergence is about this, combine with Maximizers and you might have a decent feel for it.

In terms of utility functions, the most basic is: do what you want. "Want" here refers to whatever values the agent values. But in order for the "do what you want" utility function to succeed effectively, there's a lower level that's important: be able to do what you want. 

Now for humans, that usually refers to getting a job, planning for retirement, buying insurance, planning for the long-term, and doing things you don't like for a future payoff. Sometimes humans go to war in order to "be able to do what you want", which should show you that satisfying a utility function is important.

For an AI who most likely has a straightforward utility function, and who has all the capabilities to execute it(assuming you believe that superintelligent AGI could develop nanotech, get root access to the datacenter, etc.), humans are in the way of "being able to do what you want". Humans in this case would probably not like an unaligned AI, and would try to shut it down, or at least not die themselves. Most likely, the AI has a utility function that has no use for humans, and thus they are just resources standing in the way. Therefore the AI goes on holy war against humans to maximize its possible reward, and all the humans die. 

What would it mean for an outcome to be objectively best for all of reality?

It might be your subjective opinion that maximized conscious happiness would be the objectively best reality. Another human's subjective opinion might be that a reality that maximized the fulfillment of fundamentalist Christian values was the objectively best reality. A third human might hold that there's no such thing as the objectively best, and all we have are subjective opinions.

Given that different people disagree, one could argue that we shouldn't privilege any single person's opinion, but try to take everyone's opinions into account - that is, build an AI that cared about the fulfillment of something like "human values".

Of course, that would be just their subjective opinion. But it's the kind of subjective opinion that the people involved in AI alignment discussions tend to have.

The fact that the statement is controversial is, I think, the reason. What makes a world-state or possible future valuable is a matter of human judgment, and not every human believes this. 

EY's short story Three Worlds Collide [? · GW] explores what can happen when beings with different conceptions of what is valuable, have to interact. Even when they understand each other's reasoning, it doesn't change what they themselves value. Might be a useful read, and hopefully a fun one.

For a more involved discussion than Kaj's answer, you might check out the "Mere Goodness" section of Rationality: A-Z [? · GW].

Without nanotech or anything like that, maybe the easiest way is to manipulate humans into building lots of powerful and hackable weapons (or just wait since we're doing it anyway). Then one day, strike.

Edit: and of course the AI's first action will be to covertly take over the internet, because the biggest danger to the AI is another AI already existing or being about to appear. It's worth taking a small risk of being detected by humans to prevent the bigger risk of being outraced by a competitor.

This new series of posts from Holden Karnofsky (CEO of Open Philanthropy) is about exactly this. The first post came out today:

https://www.lesswrong.com/posts/oBBzqkZwkxDvsKBGB/ai-could-defeat-all-of-us-combined [LW · GW]

I find slower take-off scenarios more plausible. I like the general thrust of Christiano's "What failure looks like [LW · GW]". I wonder if anyone has written up a more narrative / concrete account of that sort of scenario.

https://www.lesswrong.com/posts/XFBHXu4YNqyF6R3cv/pitching-an-alignment-softball [LW · GW] 

Alexey Turchin and David Denkenberger describe several scenarios here: https://philpapers.org/rec/TURCOG-2 (additional recent discussion in this comment thread [LW(p) · GW(p)])

Eliezer's go-to scenario (from his recent post [LW · GW]):

The concrete example I usually use here is nanotech, because there's been pretty detailed analysis of what definitely look like physically attainable lower bounds on what should be possible with nanotech, and those lower bounds are sufficient to carry the point.  My lower-bound model of "how a sufficiently powerful intelligence would kill everyone, if it didn't want to not do that" is that it gets access to the Internet, emails some DNA sequences to any of the many many online firms that will take a DNA sequence in the email and ship you back proteins, and bribes/persuades some human who has no idea they're dealing with an AGI to mix proteins in a beaker, which then form a first-stage nanofactory which can build the actual nanomachinery.  (Back when I was first deploying this visualization, the wise-sounding critics said "Ah, but how do you know even a superintelligence could solve the protein folding problem, if it didn't already have planet-sized supercomputers?" but one hears less of this after the advent of AlphaFold 2, for some odd reason.)  The nanomachinery builds diamondoid bacteria, that replicate with solar power and atmospheric CHON, maybe aggregate into some miniature rockets or jets so they can ride the jetstream to spread across the Earth's atmosphere, get into human bloodstreams and hide, strike on a timer.  Losing a conflict with a high-powered cognitive system looks at least as deadly as "everybody on the face of the Earth suddenly falls over dead within the same second". 

https://www.lesswrong.com/posts/BAzCGCys4BkzGDCWR/the-prototypical-catastrophic-ai-action-is-getting-root [LW · GW] 

https://www.gwern.net/fiction/Clippy (very detailed but also very long and very full of technical jargon; on the other hand, I think it's mostly understandable even if you have to gloss over most of the jargon)

There's actually two different parts to the answer, and the difference is important. There is the time between now and the first AI capable of autonomously improving itself (time to AGI), and there's the time it takes for the AI to "foom", meaning improve itself from a roughly human level towards godhood. In EY's view, it doesn't matter at all how long we have between now and AGI, because foom will happen so quickly and will be so decisive that no one will be able to respond and stop it. (Maybe, if we had 200 years we could solve it, but we don't.) In other people's view (including Robin Hanson and Paul Christiano, I think) there will be "slow takeoff." In this view, AI will gradually improve itself over years, probably working with human researchers in that time but progressively gathering more autonomy and skills. Hanson and Christiano agree with EY that doom is likely. In fact, in the slow takeoff view ASI might arrive even sooner than in the fast takeoff view.

For a survey of experts, see:

https://research.aimultiple.com/artificial-general-intelligence-singularity-timing/

Most experts expect AGI between 2030 and 2060, so predictions before 2030 are definitely in the minority.

My own take is that a lot of current research is focused on scaling, and has found that deep learning scales quite well to very large sizes.  This finding is replicated in evolutionary studies; one of the main differences between the human brain and the chimpanzee is just size (neuron count), pure and simple.

As a result, the main limiting factor thus appears to be the amount of hardware that we can throw at the problem. Current research into large models is very much hardware limited, with only the major labs (Google, DeepMind, OpenAI, etc.) able to afford the compute costs to train large models.  Iterating on model architecture at large scales is hard because of the costs involved.  Thus, I personally predict that we will achieve AGI only when the cost of compute drops to the point where FLOPs roughly equivalent to the human brain can be purchased on a more modest budget; the drop in price will open up the field to more experimentation.

We do not have AGI yet even on current supercomputers, but it's starting to look like we might be getting close (close = factor of 10 or 100).  Assuming continuing progress in Moore's law (not at all guaranteed), another 15-20 years will lead to another 1000x drop in the cost of compute, which is probably enough for numerous smaller labs with smaller budgets to really start experimenting.  The big labs will have a few years head start, but if they don't figure it out, then they will be well positioned to scale into super-intelligent territory immediately as soon as the small labs help make whatever breakthroughs are required.  The longer it takes to solve the software problem, the more hardware we'll have to scale immediately, which means faster foom.  Getting AGI sooner may thus yield a better outcome.

I would tentatively put the date at around 2035, +/- 5 years.  

If we run into a roadblock that requires substantially new techniques (e.g., gradient descent isn't enough) then the timeline could be pushed back.  However, I haven't seen much evidence that we've hit any fundamental algorithmic limitations yet.

I personally like the idea of uploading ourselves (and asked about it here [LW(p) · GW(p)]).

Note that even if we are uploaded - if someone creates an unaligned AGI that is MUCH SMARTER than us, it will still probably kill us.

"keeping up" in the sense of improving/changing/optimizing so quickly that we'd compete with software that is specifically designed (perhaps by itself) to do that - seems like a solution I wouldn't be happy with. As much as I'm ok with posting my profile picture on Facebook, there are some degrees of self modification that I'm not ok with

Ding Ding Ding, we have a winner here. Strong up vote.

The first plane didn't emulate birds. The first AGI probably won't be based on a retro engineering of the brain. The blue brain project is unlikely to finish reproducing the brain before DeepMind finds the right architecture.

But I agree that being able to retro engineer the brain is very valuable for alignment, this is one of the path described her [LW · GW]e, in the final post of intro-to-brain-like-agi-safety, section Reverse-engineer human social instincts.

I don't know, I'm replying here with my priors [EA · GW] from software development.

TL;DR: 

Do something that is 

1. Mostly useful (software/ML/math/whatever are all great and there are others too, feel free to ask)
2. Where you have a good fit, so you'll enjoy and be curious about your work, and not burn out from frustration or because someone told you "you must take this specific job"
3. Get mentorship so that you'll learn quickly

And this will almost certainly be useful somehow.

Main things my prior is based on:

EA in general and AI Alignment specifically need lots of different "professions". We probably don't want everyone picking the number one profession and nobody doing anything else. We probably want each person doing whatever they're a good fit for.

The amount we "need" is going up over time, not down, and I can imagine it going up much more, but can't really imagine it going down (so in other words, I mostly assume whatever we need today, which is quite a lot, will also be needed in a few years. So there will be lots of good options to pick)

Hi Tiuto,

consider scaling up in ML to become an ML engineer or ML researcher.  If it's still possible, try to join the best engineering school in your region, and then join your local EA group, and start community building to nudge your smart friends towards AI safety. ML engineering does not necessitate a genius level of IQ. 

I'm myself an ML engineer,  you can dm me for further questions. I'm far from being a genius, I've never been the best in my class, but I'm currently able to contribute meaningfully.

I don't know much about it, but you might want to look into the "grabby aliens" model. I'm not sure how they come to this conclusion, but the belief is "If you have aliens that are moving outwards near the speed of light, it will still take millions and millions of years on average for them to reach us, so the odds of them reaching us soon are super small."

https://grabbyaliens.com/

1. An AGI can be dangerous even if it isn't sentient
2. If an AI can do most things a human can do (which is achievable using neurons apparently because that's what we're made of), and if that AI can run x10,000 as fast (or if it's better in some interesting way, which computers sometimes are compared to humans), then it can be dangerous

Does this answer your question? Feel free to follow up

I take "source code" as loosely meaning "everything that determines the behaviour of the AI, in a form intelligible to the examiner". This might include any literal source code, hardware details, and some sufficiently recent snapshot of runtime state. Literal source code is just an analogy that makes sense to humans reasoning about behaviour of programs where most of the future behaviour is governed by rules fixed in that code.

The details provided cannot include future input and so do not completely constrain future behaviour, but the examiner may be able to prove things about future behaviour under broad classes of future input, and may be able to identify future inputs that would be problematic.

The broad idea is that in principle, AGI might be legible in that kind of way to each other, while humans are definitely not legible in that way to each other.

Seems like two separate things (?)

1. If we forget a dimension, like "AGI, please remember we don't like getting bored", then things go badly [LW · GW], even if we added another fake dimension which wasn't related to boredom.
2. If we train the AI on data from our current world, then [almost?] certainly it will see new things when it runs for real.  As a toy (not realistic but I think correct) example: the AI will give everyone a personal airplane, and then it will have to deal with a world that has lots of airplanes.

Working on AI Capabilities: I think this is net negative, and I'm commenting here so people can [V] if they agree or [X] if they disagree.

Seems like habryka [LW · GW] agrees [LW(p) · GW(p)]? 

Seems like Kaj [LW · GW] disagrees [LW · GW]?

I think it wouldn't be controversial to advise you to at least talk to 80,000 hours about this before you do it, as some safety net to not do something you don't mean to by mistake? Assuming you trust them. Or perhaps ask someone you trust. Or make your own gears-level model. Anyway, seems like an important decision to me

Chris Olah is/was the main guy working on interpretability research, and he is a co-founder of Anthropic. So Anthropic would definitely be aware of this idea.

There's some discussion of this in section 3.4. of Responses to Catastrophic AGI Risk.

How would you explain "qualia" or "positive utility" in python?

Also, regarding definitions like "delusion of fundamental subject/object split of experience", 

1. How do you explain that in Python?
2. See Value is Fragile [LW · GW] (TL;DR: If you forget even a tiny thing in your definition of happiness/suffering*, the result could be extremely extremely bad)

The standard answer here is that all humans dying is much, much worse than anything happening with narrow AI. Not to say those problems are trivial, but humanity's extinction is an entirely different level of bad, so that's what we should be focusing on. This is even more true if you care about future generations, since human extinction is not just 7 billion dead, but the loss of all generations who could have come after.

I personally believe this argument holds even if we ascribe a relatively low probability to AGI in the relatively near future. E.g, if you think there's a 10% chance of AGI in the next 10-20 years, it still seems reasonable to prioritise AGI safety now. If you think AGI isn't possible at all, naturally we don't need to worry about AI safety. But I find that pretty unconvincing - humanity has made a lot of progress very quickly in the field of AI capabilities, and it shows no signs of slowing down, and there's no reason why such a machine could not exist in principle.

In addition to what Jay Bailey said, the benefits of an aligned AGI are incredibly high, and if we successfully solved the alignment problem we could easily solve pretty much any other problem in the world(assuming you believe the "intelligence and nanotech can solve anything" argument). The danger of AGI is high, but the payout is also very large.

Interesting question. As far as what government could do to slow down progress towards AGI, I'd also include access to high-end compute. Lots of RL is knowledge that's passed through papers or equations, and it can be hard to contain that kind of stuff. But shutting down physical compute servers seems easier. 

Depends whether they considered it a national security issue to win the arms race, and if they did how able they would be to absorb and keep the research teams working effectively.

Paul Christiano provided a picture of non-Singularity doom in What Failure Looks Like [LW · GW]. In general there is a pretty wide range of opinions on questions about this sort of thing - the AI-Foom debate between Eliezer Yudkowsky and Robin Hanson is a famous example, though an old one.

"Takoff speed" is a common term used to refer to questions about the rate of change in AI capabilities at the human and superhuman level of general intelligence - searching Lesswrong or the Alignment Forum for that phrase will turn up a lot of discussion about these questions, though I don't know of the best introduction offhand (hopefully someone else here has suggestions?).

It's definitely a common belief on this site. I don't think it's likely, I've written up some arguments here [EA(p) · GW(p)]. 

Recursive self-improvement, or some other flavor of PASTA, seems essentially inevitable conditioning on not hitting hard physical limits or civilization being severely disrupted. There are Paul/EY debates about how discontinuous the capabilities jump will be, but the core idea of systems automating their own development and this leading to an accelerating feedback loop, or intelligence explosion, is conceptually solid.

There are still AI risks without the intelligence explosion, but it is a key part of the fears of the people who think we're very doomed, as it causes the dynamic of getting only one shot at the real deal since the first system to go 'critical' will end up extremely capable.

I forgot about downvotes. I'm going to add this in to the guidelines.

Short answer: Yep, probably.

Medium answer: If AGI has components that look like our most capable modern deep learning models (which I think is quite likely if it arrives in the next decade or two), it will probably be very resource-intensive to run, and orders of magnitude more expensive to train. This is relevant because it impacts who has the resources to develop AGI (large companies and governments; likely not individual actors), secrecy (it’s more difficult to secretly acquire a massive amount of compute than it is to secretly boot up an AGI on your laptop; this may even enable monitoring and regulation), and development speed (if iterations are slower and more expensive, it slows down development).

If you’re interested in further discussion of possible compute costs for AGI (and how this affects timelines), I recommend reading about bio anchors.

1. (I'm not sure but why would this be important? Sorry for the silly answer, feel free to reply in the anonymous form again)
2. I think a good baseline for comparison would be 
   1. Training large ML models (expensive)
   2. Running trained ML models (much cheaper)
3. I think comparing to blockchain is wrong, because 
   1. it was explicitly designed to be resource intensive on purpose (this adds to the security of proof-of-work blockchains)
   2. there is a financial incentive to use a specific (very high) amount of resources on blockchain mining (because what you get is literally a currency, and this currency has a certain value, so it's worthwhile to spend any money lower than that value on the mining process)
   3. None of these are true for ML/AI, where your incentive is more something like "do useful things"

One has the motivations one has, and one would be inclined to defend them if someone tried to rewire the motivations against one's will. If one happened to have different motivations, then one would be inclined to defend those instead.

The idea is that once a superintelligence gets going, its motivations will be out of our reach. Therefore, the only window of influence is before it gets going. If, at the point of no return, it happens to have the right kinds of motivations, we survive. If not, it's game over.

Consciousness, intelligence and human-value-alignment are probably mostly orthogonal, so I don’t think that solving the hard problem of intelligence would directly impact AGI alignment research. (Perhaps consciousness requires general intelligence, so understanding how consciousness works on a mechanistic level might dramatically accelerate timelines? But that’s highly speculative.)

However, if solving the hard problem of consciousness leads us to realize that some of our AI systems are conscious, then we have a whole new set of moral patients. (As an AGI researcher) I personally would become much more concerned with machine ethics in that case, and I suspect others would as well.

my understanding is that while an oracle doesn't directly control the nukes, it provides info to the people who control the nukes. Which is pretty much just moving the problem one layer deeper. While it can't directly change the physical state of the world, it can manipulate people to pretty much achieve the same thing.

Check this tag for more specifics: https://www.lesswrong.com/tag/oracle-ai

In reward learning research, it’s common to represent the AI’s estimate of the true reward function as a distribution over possible reward functions, which I think is analogous to what you are describing. It’s also common to define optimal behavior, given a distribution over reward functions, as that behavior which maximizes the expected reward under that distribution. This is mathematically equivalent to optimizing a single reward function equal to the expectation of the distribution. So, this helps in that the AI is optimizing a reward function that is more likely to be “aligned” than one at an extreme end of the distribution. However, this doesn’t help with the problems of optimizing a single fixed reward function.

It's possible that there is a ceiling to intelligence gains. It's also possible that there isn't. Looking at the available evidence there doesn't seem to be so - a single ant is a lot less intelligent than a lobster, which is less intelligent than a snake, etc. etc. While it would be nice (in a way) if there was a ceiling, it seems more prudent to assume that there isn't, and prepare for the worst. Especially as by "superintelligent", you shouldn't think of double, or even triple Einstein, rather you should think of a whole other dimension of intelligence, like the difference between you and a hamster.

As to your specific counterarguments:

1. It's both, really. But yes - complex language allows humans to keep and build upon previous knowledge. Humans advantage is in the gigantic amounts of know how that can be passed on to future generations. Which is something that computers are eminently good at - you can keep a local copy of Wikipedia in 20GB.
2. Good point. But it's not just luck. Yes, luck plays a large part, but it's also resources (in a very general sense). If you have the basic required talent and a couple of billions of dollars, I'm pretty sure you could become a hollywood star quite quickly. The point is that a superintelligence won't have a similar level of intelligence to any one else around. Which will allow it to run circles around everyone. Like if a Einstein level intelligence decided to learn to play chess and started playing against 5 year olds - they might win the first couple of games, but after a while you'd probably notice a trend...
3. Intelligence is an advantage. Quite a big one, generally speaking. But in society most people are generally at the same level if you compare them to e.g. Gila monsters (because we're talking about superintelligence). So it shouldn't be all that surprising that other resources are very important. While many powerful people don't seem to be intelligent, they tend to be either cunning (which is a different kind of intelligence) or have deep pockets (not just money) which offset their relative lack in smarts. Also, while they might not be very clever, very few powerful people are actively stupid.

I work on AI safety via learning from human feedback. In response to your three ideas:

• Uniformly random human noise actually isn’t much of a problem. It becomes a problem when the human noise is systematically biased in some way, and the AI doesn’t know exactly what that bias is. Another core problem (which overlaps with the human bias), is that the AI must use a model of human decision-making to back out human values from human feedback/behavior/interaction, etc. If this model is wrong, even slightly (for example, the AI doesn’t realize that the noise is biased along one axis), the AI can infer incorrect human values.

• I’m working on it, stay tuned.

• Our most capable AI systems require a LOT of training data, and it’s already expensive to generate enough human feedback for training. Limiting the pool of human teachers to trusted experts, or providing pre-training to all of the teachers, would make this even more expensive. One possible way out of this is to train AI systems themselves to give feedback, in imitation of a small trusted set of human teachers.

You suggested:

But if we were to work on it today, it would only have a sub-human level, and we could iterate like on a child

But as you yourself pointed out: "We are not sure that this would extrapolate well to higher levels of capability"

You suggested:

and we had “Reverse-enginered human social instincts”

As you said, "The brain's face recognition algorithm is not perfect either. It has a tendency to create false positives"

And so perhaps the AI would make human pictures that create false positives. Or, as you said, "We are not sure that this would extrapolate well to higher levels of capability"

The classic example is humans creating condoms, which is a very unfriendly thing to do to Evolution, even though it raised us like children, sort of

Adding: "Intro to Brain-Like-AGI Safety [? · GW]" (I didn't read it yet, seems interesting)

Do you mean something like "only get 100 paperclips, not more?"

If so - the AGI will never be sure it has 100 paperclips, so it can take lots of precautions to be very, very sure. Like turning all the world into paperclip counters or so

To be a bit more explicit. I have some ideas of what it would look like to try to develop this meta-field or at least sub-elements of it, seperate from general rationality and am trying to get a feel for if they are worth pursuing personally. Or better yet, handing over to someone who doesn't feel they have any currently tractable ideas, but is better at getting things done.

Because it means you can't get AI to do good things "for free," it has to be something you intentionally designed it to do.

Denying the orthogonality thesis looks like claims that an AI built with one set of values will tend to change those values in a particular direction as it becomes cleverer. Because of wishful thinking, people usually try to think of reasons why an AI built in an unsafe way (with some broad distribution over possible values) will tend to end up being nice to humans (a narrow target of values) anyway.

(Although there's at least one case where someone has argued "the orthogonality thesis is false, therefore even AIs built with good values will end up not valuing humans.")

A common AGI failure mode is to say things like:

"Well, if the AI is so smart, wouldn't it know what we meant to program it to do?"

"Wouldn't a superintelligent AI also have morality?"

"If you had a paperclip maximiser, once it became smart, why wouldn't it get bored of making paperclips and do something more interesting?"

Orthogonality is the answer to why all these things won't happen. You don't hear these arguments a lot any more, because the field has become more sophisticated, so EY's harping on about orthogonality seems a bit outdated. To be fair, a lot of the reason the field has grown up about this is because EY kept harping on about it in the first place.

The answer is because the orthogonality thesis means there is no correlation between goals and intelligence level.

Better hardware reduces the need for software to be efficient to be dangerous. I suspect on balance that yes, this makes development of said hardware more dangerous, and that not working on it can buy us some time. But the human brain runs on about 20 watts of sugar and isn't anywhere near optimized for general intelligence, so we shouldn't strictly need better hardware to make AGI, and IDK how much time it buys.

Also, better hardware makes more kinds of solutions feasible, and if aligned AGI requires more computational capacity than unaligned AI, or if better hardware makes it so ems happen first and can help of offset AGI risk until we solve alignment, then it seems possible to me that the arrow could point in the other direction.

Robin Hanson has an solution to the Fermi Paradox which can be read in detail here (there are also explanatory videos at the same link): https://grabbyaliens.com/

The summary from the site goes: 

There are two kinds of alien civilizations. “Quiet” aliens don’t expand or change much, and then they die. We have little data on them, and so must mostly speculate, via methods like the Drake equation.

“Loud” aliens, in contrast, visibly change the volumes they control, and just keep expanding fast until they meet each other. As they should be easy to see, we can fit theories about loud aliens to our data, and say much about them, as S. Jay Olson has done in 7 related papers (1, 2, 3, 4, 5, 6, 7) since 2015.

Furthermore, we should believe that loud aliens exist, as that’s our most robust explanation for why humans have appeared so early in the history of the universe. While the current date is 13.8 billion years after the Big Bang, the average star will last over five trillion years. And the standard hard-steps model of the origin of advanced life says it is far more likely to appear at the end of the longest planet lifetimes. But if loud aliens will soon fill the universe, and prevent new advanced life from appearing, that early deadline explains human earliness.

“Grabby” aliens is our especially simple model of loud aliens, a model with only 3 free parameters, each of which we can estimate to within a factor of 4 from existing data. That standard hard steps model implies a power law (t/k)ⁿ appearance function, with two free parameters k and n, and the last parameter is the expansion speed s. We estimate:

• Expansion speed s from fact that we don’t see loud alien volumes in our sky,
• Power n from the history of major events in the evolution of life on Earth,
• Constant k by assuming our date is a random sample from their appearance dates.

Using these parameter estimates, we can estimate distributions over their origin times, distances, and when we will meet or see them. While we don’t know the ratio of quiet to loud alien civilizations out there, we need this to be ten thousand to expect even one alien civilization ever in our galaxy. Alas as we are now quiet, our chance to become grabby goes as the inverse of this ratio.

/quote

https://arxiv.org/abs/1806.02404 “Dissolving the Fermi paradox”.  
 

The Drake equation gives an estimate for the number of technological civil actions to ever arise, by multiplying a number of parameters.  Many of these parameters are unknown, and reasonable estimates range over many orders of magnitude.  This paper takes defensible ranges for these parameters from the literature, and shows that if they are all small, but reasonable, we are the only technological civilization in the universe.  

Earth was not eaten by aliens or an AGI in the past, nor do we see them in the sky, so we are probably alone.  (Or else interstellar expansion is impossible, for some reason.  But that seems unlikely.)

There's also the problem that the more contained an AGI is, the less useful it is. The maximally safe AGI would be one which couldn't communicate or interact with us in any way, but what would be the point of building it? If people have built an AGI, then it's because they'll want it to do something for them.

From Disjunctive Scenarios of Catastrophic AGI Risk: 

The Social Challenge

AI confinement assumes that the people building it, and the people that they are responsible to, are all motivated to actually keep the AI confined. If a group of cautious researchers builds and successfully contains their AI, this may be of limited benefit if another group later builds an AI that is intentionally set free. Reasons for releasing an AI may include (i) economic benefit or competitive pressure, (ii) ethical or philosophical reasons, (iii) confidence in the AI’s safety, and (iv) desperate circumstances such as being otherwise close to death. We will discuss each in turn below.

Voluntarily Released for Economic Benefit or Competitive Pressure

As discussed above under “power gradually shifting to AIs,” there is an economic incentive to deploy AI systems in control of corporations. This can happen in two forms: either by expanding the amount of control that already-existing systems have, or alternatively by upgrading existing systems or adding new ones with previously-unseen capabilities. These two forms can blend into each other. If humans previously carried out some functions which are then given over to an upgraded AI which has become recently capable of doing them, this can increase the AI’s autonomy both by making it more powerful and by reducing the amount of humans that were previously in the loop.

As a partial example, the U.S. military is seeking to eventually transition to a state where the human operators of robot weapons are “on the loop” rather than “in the loop” (Wallach & Allen 2013). In other words, whereas a human was previously required to explicitly give the order before a robot was allowed to initiate possibly lethal activity, in the future humans are meant to merely supervise the robot’s actions and interfere if something goes wrong. While this would allow the system to react faster, it would also limit the window that the human operators have for overriding any mistakes that the system makes. For a number of military systems, such as automatic weapons defense systems designed to shoot down incoming missiles and rockets, the extent of human oversight is already limited to accepting or overriding a computer’s plan of actions in a matter of seconds, which may be too little to make a meaningful decision in practice (Human Rights Watch 2012).

Sparrow (2016) reviews three major reasons which incentivize major governments to move toward autonomous weapon systems and reduce human control:

1. Currently existing remotely piloted military “drones,” such as the U.S. Predator and Reaper, require a high amount of communications bandwidth. This limits the amount of drones that can be fielded at once, and makes them dependent on communications satellites which not every nation has, and which can be jammed or targeted by enemies. A need to be in constant communication with remote operators also makes it impossible to create drone submarines, which need to maintain a communications blackout before and during combat. Making the drones autonomous and capable of acting without human supervision would avoid all of these problems.

2. Particularly in air-to-air combat, victory may depend on making very quick decisions. Current air combat is already pushing against the limits of what the human nervous system can handle: further progress may be dependent on removing humans from the loop entirely.

3. Much of the routine operation of drones is very monotonous and boring, which is a major contributor to accidents. The training expenses, salaries, and other benefits of the drone operators are also major expenses for the militaries employing them.

Sparrow’s arguments are specific to the military domain, but they demonstrate the argument that “any broad domain involving high stakes, adversarial decision making, and a need to act rapidly is likely to become increasingly dominated by autonomous systems” (Sotala & Yampolskiy 2015, p. 18). Similar arguments can be made in the business domain: eliminating human employees to reduce costs from mistakes and salaries is something that companies would also be incentivized to do, and making a profit in the field of high-frequency trading already depends on outperforming other traders by fractions of a second. While the currently existing AI systems are not powerful enough to cause global catastrophe, incentives such as these might drive an upgrading of their capabilities that eventually brought them to that point.

In the absence of sufficient regulation, there could be a “race to the bottom of human control” where state or business actors competed to reduce human control and increased the autonomy of their AI systems to obtain an edge over their competitors (see also Armstrong et al. 2016 for a simplified “race to the precipice” scenario). This would be analogous to the “race to the bottom” in current politics, where government actors compete to deregulate or to lower taxes in order to retain or attract businesses.

AI systems being given more power and autonomy might be limited by the fact that doing this poses large risks for the actor if the AI malfunctions. In business, this limits the extent to which major, established companies might adopt AI-based control, but incentivizes startups to try to invest in autonomous AI in order to outcompete the established players. In the field of algorithmic trading, AI systems are currently trusted with enormous sums of money despite the potential to make corresponding losses—in 2012, Knight Capital lost $440 million due to a glitch in their trading software (Popper 2012, Securities and Exchange Commission 2013). This suggests that even if a malfunctioning AI could potentially cause major risks, some companies will still be inclined to invest in placing their business under autonomous AI control if the potential profit is large enough.

U.S. law already allows for the possibility of AIs being conferred a legal personality, by putting them in charge of a limited liability company. A human may register a limited liability corporation (LLC), enter into an operating agreement specifying that the LLC will take actions as determined by the AI, and then withdraw from the LLC (Bayern 2015). The result is an autonomously acting legal personality with no human supervision or control. AI-controlled companies can also be created in various non-U.S. jurisdictions; restrictions such as ones forbidding corporations from having no owners can largely be circumvented by tricks such as having networks of corporations that own each other (LoPucki 2017). A possible start-up strategy would be for someone to develop a number of AI systems, give them some initial endowment of resources, and then set them off in control of their own corporations. This would risk only the initial resources, while promising whatever profits the corporation might earn if successful. To the extent that AI-controlled companies were successful in undermining more established companies, they would pressure those companies to transfer control to autonomous AI systems as well.

Voluntarily Released for Purposes of Criminal Profit or Terrorism

LoPucki (2017) argues that if a human creates an autonomous agent with a general goal such as “optimizing profit,” and that agent then independently decides to, for example, commit a crime for the sake of achieving the goal, prosecutors may then be unable to convict the human for the crime and can at most prosecute for the lesser charge of reckless initiation. LoPucki holds that this “accountability gap,” among other reasons, assures that humans will create AI-run corporations.

Furthermore, LoPucki (2017, p. 16) holds that such “algorithmic entities” could be created anonymously and that them having a legal personality would give them a number of legal rights, such as being able to “buy and lease real property, contract with legitimate businesses, open a bank account, sue to enforce its rights, or buy stuff on Amazon and have it shipped.” If an algorithmic entity was created for a purpose such as funding or carrying out acts of terrorism, it would be free from social pressure or threats to human controllers:

In deciding to attempt a coup, bomb a restaurant, or assemble an armed group to attack a shopping center, a human-controlled entity puts the lives of its human controllers at risk. The same decisions on behalf of an AE risk nothing but the resources the AE spends in planning and execution (LoPucki 2017, p. 18).

While most terrorist groups would stop short of intentionally destroying the world, thus posing at most a catastrophic risk, not all of them necessarily would. In particular, ecoterrorists who believe that humanity is a net harm to the planet, and religious terrorists who believe that the world needs to be destroyed in order to be saved, could have an interest in causing human extinction (Torres 2016, 2017, Chapter 4).

Voluntarily Released for Aesthetic, Ethical, or Philosophical Reasons

A few thinkers (such as Gunkel 2012) have raised the question of moral rights for machines, and not everyone necessarily agrees on AI confinement being ethically acceptable. The designer of a sophisticated AI might come to view it as something like their child, and feel that it deserved the right to act autonomously in society, free of any external constraints.

Voluntarily Released due to Confidence in the AI’s Safety

For a research team to keep an AI confined, they need to take seriously the possibility of it being dangerous. Current AI research doesn’t involve any confinement safeguards, as the researchers reasonably believe that their systems are nowhere near general intelligence yet. Many systems are also connected directly to the Internet. Hopefully, safeguards will begin to be implemented once the researchers feel that their system might start having more general capability, but this will depend on the safety culture of the AI research community in general (Baum 2016), and the specific research group in particular. If a research group mistakenly believed that their AI could not achieve dangerous levels of capability, they might not deploy sufficient safeguards for keeping it contained.

In addition to believing that the AI is insufficiently capable of being a threat, the researchers may also (correctly or incorrectly) believe that they have succeeded in making the AI aligned with human values, so that it will not have any motivation to harm humans.

Voluntarily Released due to Desperation

Miller (2012) points out that if a person was close to death, due to natural causes, being on the losing side of a war, or any other reason, they might turn even a potentially dangerous AGI system free. This would be a rational course of action as long as they primarily valued their own survival and thought that even a small chance of the AGI saving their life was better than a near-certain death.

The AI Remains Contained, But Ends Up Effectively in Control Anyway

Even if humans were technically kept in the loop, they might not have the time, opportunity, motivation, intelligence, or confidence to verify the advice given by an AI. This would particularly be the case after the AI had functioned for a while, and established a reputation as trustworthy. It may become common practice to act automatically on the AI’s recommendations, and it may become increasingly difficult to challenge the “authority” of the recommendations. Eventually, the AI may in effect begin to dictate decisions (Friedman & Kahn 1992).

Likewise, Bostrom and Yudkowsky (2014) point out that modern bureaucrats often follow established procedures to the letter, rather than exercising their own judgment and allowing themselves to be blamed for any mistakes that follow. Dutifully following all the recommendations of an AI system would be another way of avoiding blame.

O’Neil (2016) documents a number of situations in which modern-day machine learning is used to make substantive decisions, even though the exact models behind those decisions may be trade secrets or otherwise hidden from outside critique. Among other examples, such models have been used to fire school teachers that the systems classified as underperforming and give harsher sentences to criminals that a model predicted to have a high risk of reoffending. In some cases, people have been skeptical of the results of the systems, and even identified plausible reasons why their results might be wrong, but still went along with their authority as long as it could not be definitely shown that the models were erroneous.

In the military domain, Wallach & Allen (2013) note the existence of robots which attempt to automatically detect the locations of hostile snipers and to point them out to soldiers. To the extent that these soldiers have come to trust the robots, they could be seen as carrying out the robots’ orders. Eventually, equipping the robot with its own weapons would merely dispense with the formality of needing to have a human to pull the trigger.

I believe the general argument is this:

If an AGI is smarter than you, it will think of ways to escape containment that you can't think of. Therefore, it's unreasonable to expect us to be able to contain a sufficiently intelligent AI even if it seems foolproof to us. One solution to this would be to make the AI not want to escape containment, but if you've solved that you've solved a massive part of the alignment problem already.

Then it forgets both the prompt it was given and the response it outputted.

If the memory of this AI is so limited, it seems to me that we are speaking about a narrow agent.  An AGI wouldn't be that limited. In order to execute complex tasks, you need to subdivide the task into sub-tasks. This requires a form of long term memory.

On second thoughts...

If someone asks this AI to translate natural language into code, who is to say that the resulting code won't contain viruses?

The history of the world would be different (and a touch shorter) if immediately after the development of the nuclear bomb millions of nuclear armed missiles constructed themselves and launched themselves at targets across the globe.

To date we haven't invented anything that's an existential threat without humans intentionally trying to use it as a weapon and devoting their own resources to making it happen. I think that AI is pretty different.

Even if you only work on an AI that tells doctors if someone has cancer, other people will still build an AGI

Systems like the ones you mentioned aren't single agents with utility functions we control - they're made up of many humans whose utility functions we can't control since we didn't build them. This means alignment theory is not set up to align or manipulate these systems - it's a very different problem.

There is alignment research that has been or is being performed on current-level AI, however - this is known as prosaic AI alignment. We also have some interpretability results that can be used in understanding more about modern, non-AGI AI's. These results can be and have been used outside of AGI, but I'm not sure how practically useful they are right now - someone else might know more. If we had better alignment theory, at least some of it would be useful in aligning narrow AI as well.

Maybe, but we don't have a particularly good understanding of how we would do that. This is sometimes termed "strawberry alignment". Also, again, you have to figure out how to take "strawberry alignment" and use it to solve the problem that someone is eventually gonna do "not strawberry alignment".

So, I can see two ways this training can go.

Firstly, it works like you said. You create AI's that learn the robust value of co-operation as a terminal value.

Secondly, it doesn't. You create AI's that learn the rule "Maximise my own utility function" and implement the behaviour of co-operation as an instrumental goal towards this. This includes maximising other people's utility functions. Hooray!

Unfortunately, this can be decoupled eventually. What if the AI reaches an arbitrary level of capability and decides it can create trillions of agents with reward functions of "1 if the AI does whatever it wants, 0 otherwise" and that overwhelms the real agents? You could try to patch that, but you can't patch an AI smarter than you - it can think of things you can't.

The solution to this is clear - provide the AI's opportunities to defect in the training environment. This works up until the AI's reach a level of capability where they become aware they're in a training environment. Then they're playing a level up - they're not playing the other agents, they're playing you, so they co-operate exactly as you'd expect, right up until they can defect against you in the real world.

If you could figure out a way to look at an AI and tell the difference between co-operation as terminal value vs. co-operation as instrumental value, that would be a big step forward. Being able to actually create those terminal values reliably would be even better.

Consciousness is a dissolvable question [LW · GW], I think. People are talking past each other about several real structures, the Qualia Research Institute people are trying to pin down valance for example, a recent ACX post was talking about the global workspace of awareness, etc.

Alignment seems like a much harder problem, as you're not just dealing with some confused concepts, but trying to build something under challenging conditions (e.g. only one shot at the intelligence explosion).

EY's concern is that superhuman AI might behave wildly different than any lesser system, especially if there is a step where the AI replaces itself with something different and more powerful (recursive self-improvement). In machine learning, this is called "out-of-distribution behavior" because being superintelligent is not in the 'distribution' of the training data -- it's something that's never happened before, so we can't plan for it.

That said, there are people working on alignment with current AI, such as Redwood Research Institute, Anthropic AI and others.

I think a lot of people in AI safety don't think it has a high probability of working (in the sense that the existence of the field caused an aligned AGI to exist where there otherwise wouldn't have been one) - if it turns out that AI alignment is easy and happens by default if people put even a little bit of thought into it, or it's incredibly difficult and nothing short of a massive civilizational effort could save us, then probably the field will end up being useless. But even a 0.1% chance of counterfactually causing aligned AI would be extremely worthwhile!

Theory of change seems like something that varies a lot across different pieces of the field; e.g., Eliezer Yudkowsky's writing about why MIRI's approach to alignment is important seems very different from Chris Olah's discussion [LW · GW] of the future of interpretability. It's definitely an important thing to ask for a given project, but I'm not sure there's a good monolithic answer for everyone working on AI alignment problems.

I think there are many theories of change. One theory is that we want to make sure there are easy, cheap tools for making AI safe so that when someone does make an AGI, they do it safely. Other theories are shaped more like "The AI safety field should develop an AGI as soon as it can be done safely, then use that AGI to solve alignment/perform some pivotal act/etc."

One of the few paths to victory I see is having a weakly aligned weak AGI which is not capable of recursive self-improvement and using it as a research assistant to help us solve the hard version of alignment. I don't think this has a high probability of working, but it seems probably worth trying.

You ask two interesting questions, with rather separate answers.  I will discuss each in turn.

First, It's plausible to think that "it's possible to find an architecture for general intelligence a lot more efficiently than evolution".  Our process of engineering development is far faster than evolution.  People get good (or bad) ideas, try stuff, copy what works, speak at conferences, publish, make theories, teach undergraduates... and the result is progress in decades instead of millions of years.  We haven't duplicated all the achievements of life yet, but we've made a start, and have exceeded it in many places.  In particular, we've recently made huge progress in AI.  GPT-3 has pretty much duplicated the human language faculty, which takes up roughly 1% of the brain.  And we've duplicated visual object recognition, which takes another few percent.  Those were done without needing evolution, so we probably don't need evolution for the remaining 90% of the mind.

Second, "an intelligence that does the exact things we want" is the ideal that we're aiming for.  Unfortunately it does not seem possible to get to that, currently.  With current technology, what we get is "an intelligence that does approximately what we rewarded it for, plus some other weird stuff we didn't ask for."  It's not obvious, but it is much harder than you think to specify a set of goals that produce acceptable behavior.  And it is even harder (currently impossible) to provide any assurance that an AI will continue to follow those goals when set free to exert power in the world.

While evolution did indeed put a huge amount of effort into creating a chimp's brain, the amount of marginal effort it put into going from a chimp to a human brain was vastly lower.  And the effort of going from a human brain to John von Neumann's brain was tiny.  Consequently, once we have AI at the level of chimp intelligence or human intelligence it might not take much to get to John von Neumann level intelligence.  Very likely, having a million John von Neumann AI brains running at speeds greater than the original would quickly give us a singularity. 

We don't need to be confident in this to think that AGI is likely in the next few decades. Extrapolating current compute trends, the available compute may well be enough to replicate such environments.

My guess is that we will try to create intelligence to do the things we want, but we may fail. The hard part of alignment is that succeeding at getting the thing you want from a superhuman AI seems surprisingly hard.

Stampy has the canonical answer to this: I’d like to get deeper into the AI alignment literature. Where should I look?

Feel free to improve the answer, as it's on a wiki. It will be served via a custom interface once that's ready (prototype here).

Obligatory link to the excellent AGI Safety Fundamentals curriculum.

AXRP - Excellent interviews with a variety of researchers. Daniel's substantial own knowledge means that the questions he asks are often excellent, and the technical depth is far better than anything else that's available in audio, given the difficulty of autoreaders on papers or the alignment forum finding it difficult to handle actual maths.

Rob Miles's youtube channel, see this intro. Also his video on the stop button problem for Computerphile.
- Easily accessible, entertaining, videos are low cost for many people to watch, and they often end up watching several.

 

I really enjoyed Wait but Why.

Stampy has an initial draft of an answer, but it could be improved.

This [LW · GW] talk from Joe Carlsmith. - Hits at several of the key ideas really directly given the time and technical background constraints. Like Rob's videos, implies an obvious next step for people interested in learning more, or who are suspicious of one of the claims (reading Joe's actual report, maybe even the extensive discussion of it on here).
 

I haven't thought very seriously about this for many years, but FWIW: I think quantum immortality only applies to your subjectivity, which itself isn't terribly well-defined. QI is basically suggesting that you'll just keep happening to find yourself in scenarios where you're still around. It doesn't say anything about in what condition you'll still be around, or whether there will be others around along with you.

One weird move that I think would totally fit quantum immortality is if you shift your subjectively experienced identity to the AGI even as all of humanity, including your old body, dies.

Yes. This is known as the question of "takeoff speed" - how fast will an AI of near-human intellect reach superintelligence? "Fast takeoff" suggests that an AI will be able to recursively improve itself - i.e, if an AI is better at coding AI than we are, it can make a better AI, which can make a better AI, and so on, and this could happen across weeks, days, or even hours.

"Slow takeoff" suggests this is not possible or likely, and that we're likely to take years or decades to move from near-human AGI to smart human AGI to superhuman AGI to superintelligence. (where I use "superintelligence" here to mean "Chimps are to humans are humans are to it")

I've heard people make the argument that AGI takeoff will look like AlphaZero - we spent thousands of years learning to be better at Go, and AlphaZero trained from nothing to superhuman in a matter of days. If this could generalise across the physical world, it would indicate a fast takeoff being more likely.

Eliezer's argument from the recent post:

The reason why nobody in this community has successfully named a 'pivotal weak act' where you do something weak enough with an AGI to be passively safe, but powerful enough to prevent any other AGI from destroying the world a year later - and yet also we can't just go do that right now and need to wait on AI - is that nothing like that exists. There's no reason why it should exist. There is not some elaborate clever reason why it exists but nobody can see it. It takes a lot of power to do something to the current world that prevents any other AGI from coming into existence; nothing which can do that is passively safe in virtue of its weakness. If you can't solve the problem right now (which you can't, because you're opposed to other actors who don't want to be solved and those actors are on roughly the same level as you) then you are resorting to some cognitive system that can do things you could not figure out how to do yourself, that you were not close to figuring out because you are not close to being able to, for example, burn all GPUs. Burning all GPUs would actually stop Facebook AI Research from destroying the world six months later; weaksauce Overton-abiding stuff about 'improving public epistemology by setting GPT-4 loose on Twitter to provide scientifically literate arguments about everything' will be cool but will not actually prevent Facebook AI Research from destroying the world six months later, or some eager open-source collaborative from destroying the world a year later if you manage to stop FAIR specifically. There are no pivotal weak acts.

Frankenstein is a tale about misalignment. Asimov wrote a whole book about it. Vernor Vinge also writes about it. People have been trying to get their children to behave in certain ways for ever. But before LW the alignment problem was just the domain of SF. 

20 years ago the alignment problem wasn't a thing, so much that MIRI started out as an org to create a Friendly AI.

It may or may not take the time to build a superhuman understanding of human ethics, perhaps in order to manipulate humans, but building an AI that would then care about those ethics and optimize for humans values is the hard part.

Show a genuine keen interest in the things they have deep models of^[1] first before bringing up alignment, unless they invite you to talk first. Steer towards deep conversation with some well-chosen questions^[2], but be very open to having it about whatever they know most about rather than AI immediately. At some point, they are likely to ask about what you're interested in^[3].

Then you have an A/B tested elevator pitch prepared, and adapted for your specific audience (ideally as few people as possible, which helps to lower the amount of in-the-moment status you need to spend on a brief weird-sounding monologue). Mine usually goes something like:

At some point, we will build artificial systems^[4] which are more generally capable^[5] than humans. At this point, they will tend to be the main drivers of their own development, resulting in a feedback cycle of recursive self-improvement called the intelligence explosion. What happens in the future will likely be determined by the values of the systems that emerge from this, and there is a global research effort to figure out how to make sure those values include humanity's well-being. I'm trying to contribute to that effort by x.

Then you let them steer and answer their questions as honestly and accurately as you can. If there's a lull in the conversation, bringing their attention to the arc of civilization with accelerating change viewable in their lifetimes eluded to in The Most Important Century and Sapiens is a good filler, but let them lead the conversation and politely (praising them for good questions!) give them quickfire answers. Be sure to flag any question you struggle to answer well for further research and thank them for giving you a question which you don't have an answer to yet. Feel free to post it to Stampy if we're missing it from our canonical questions.

This approach has a very good rate of the other person walking away seeming to take the concerns seriously, and a fairly good rate of people later joining the effort. It does depend on you actually having good answers to hand to their first few "but why don't we just" questions, which means being fairly well read (or watching all of Rob Miles).

1. ^{^}

   Almost everyone has deep models of something, for example a supermarket worker taught me about logistics and the changes to training needs and autonomy brought on by automation recently. And by learning the 5-10 minute version of everyone's deep models you become more intellectually awesome, which helps for all sorts of things.

2. ^{^}

   e.g. "What interests you?", or ideally just getting very curious about some aspect of a thing they've spent a lot of time on, usually work, studies, or a hobby.

3. ^{^}

   This means they're in a receptive state, having been heard and explicitly opened the door to you sharing, so the normal memetic filters are lowered. And if they don't ask you, they're probably not someone who it would help to sell on alignment.

4. ^{^}

   Intentionally not using the word AI here, so they create a new mental category for the thing I'm describing rather than using their existing sci-fi bucket.

5. ^{^}

   Intentionally not using the word intelligence here, as that brings up associations which are generally unhelpful (elitism, inadequacy, etc).

"tell" with what goal? (could you ask in other words or give an example answer so I'll understand what you're pointing at?)

I'm assuming you're not asking about the mechanism (i.e. natural selection + mutations)? A trite answer would be something like "the same way it created wings, mating dances, exploding beetles, and parasites requiring multiple hosts".

Thinking about the meaning of life might be a spandrel, but a quick consideration of it comes up with various evo-psych style reasons why it's actually very useful, e.g. it can propel people to greatness, which massively can increase their genetic fitness. Fitness is an interesting thing, in that it can be very non-obvious. Everything is a trade-off, where the only goal is for your genes to propagate. So if thinking about the meaning of life will get your genes spread more (e.g. because you decide that your children have inherit meaning, because you become a high status philosopher and your sister can marry well, because it's a social sign that you have enough resources to waste them on fruitless pondering) then it's worth having around.

The first issue that comes to mind is having an incentive that would achieve that. The one you suggest doesn't incentivize truth - it incentivizes collaboration in order to guess the password, which would fine in training, but then you're going into deceptive alignment land: Aleya Cotra has a good story illustrating that

You might be interested in this paper and this LessWrong tag [? · GW].

You could, but should you? English in particular seems a bad choice. The problem with natural languages is their ambiguity. When you're providing a utility function, you want it to be as precise and robust as possible. This is actually an interesting case where folklore/mythology has known about these issues for millennia. There are all kinds of stories about genies, demons, monkey paws etc. where wishes were badly phrased or twisted. This [LW · GW] is a story explanation of the issue.

• There isn't a best alignment plan yet - that's part of the problem
• The issue isn't intuitive - you need quite a bit of context to even understand it. Like evolution 150 years ago
• The best long term protection against unaligned AGI (or aliens, or whatever) is to have an aligned AGI, so the faster it appears, the better
• People assume it will all work out somehow, as it always has in the past (basic black swan issues)
• People are selfish and hope someone else will sort it out
• People have other priorities which are more important for them to fund
• Basically the whole suite of biases etc. (check Kahneman or the sequences [? · GW] for more)

Paying researchers to not work on AGI is a very bad idea, as it incentivizes pretending to work in order to not work. The general idea behind not paying Dane-geld is that just encourages more of it. You could sort of make it better by suggesting that they work on alignment rather than capabilities, but the problem with that is that they both often look the same from the outside. You'd end up with the equivalent of gain of function research.

Yes, of course. The question then is whether its opinions are any good. Check out iterated amplification. [? · GW]