All AGI Safety questions welcome (especially basic ones) [April 2023]

This is a very good question! I can't speak for Eliezer, so the following are just my thoughts...

Before GPT, it seemed impossible to make a machine that is comparable to a human. In each aspect, it was either dramatically better, or dramatically worse. A calculator can multiply billion times faster than I can; but it cannot write poetry at all.

So, when thinking about gradual progress, starting at "worse than human" and ending at "better than human", it seemed like... either the premise of gradual progress is wrong, and somewhere along the path there will be one crucial insight that will move the machine from dramatically worse than human to dramatically better than human... or if it indeed is gradual in some sense, the transition will still be super fast.

The calculator is the example of "if it is better than me, then it is way better than me".

The machines playing chess and go, are a mixed example. I suck at chess, so the machines better than me have already existed decades ago. But at some moment they accelerated and surpassed the actual experts quite fast. More interestingly, they surpassed the experts in a way more general than the calculator does; if I remember it correctly, the machine that is superhuman at go is very similar to the machine that is superhuman at chess.

The current GPT machines are something that I have never seen before: better than humans in some aspects, worse than humans in other aspects, both in the area of processing text. I definitely would not have predicted that. Without the benefit of hindsight, it feels just as weird as a calculator that would do addition faster than humans, but multiplication slower than humans and with occasional mistakes. This simply is not how I have expected programs to behave. If someone told me that they are planning to build a GPT, I would expect it to either not work at all (more likely), or to be superintelligent (less likely). The option "it works, kinda correctly, but it's kinda lame" was not on my radar.

I am not sure what this all means. My current best guess is that this is what "learning from humans" can cause: you can almost reach them, but cannot use this alone to surpass them.

The calculator is not observing millions of humans doing multiplication, and then trying to do something statistically similar. Instead, it has an algorithm designed from scratch that solves the mathematical tasks.

The chess and go machines, they needed a lot of data to learn, but they could generate the data themselves, playing millions of games against each other. So they needed the data, but they didn't need the humans as a source of the data; they could generate the data faster.

The weakness of GPT is that if you already feed it the entire internet and all the books ever written, you cannot get more training data. Actually, you could, with ubiquitous eavesdropping... and someone is probably already working on this. But you still need humans as a source. You cannot teach GPT by texts generated by GPT, because unlike the chess and go, you do not have the exact rules to tell you which generated ouputs are the new winning moves, and which are nonsense.

There are of course other aspects where GPT can easily surpass humans: the sheer quantity of the text it can learn from and process. If it can write mediocre computer programs, then it can write mediocre computer programs in thousand different programming languages. If it can make puns or write poems at all, it will evaluate possible puns or rhymes million times faster, in any language. If it can match patterns, it can match patterns in the entire output of humanity; a new polymath.

The social consequences may be dramatic. Even if GPT is not able to replace a human expert, it can probably replace human beginners in many professions... but if the beginners become unemployable, where will the new experts come from? By being able to deal with more complexity, GPT can make the society more complex, perhaps in a way that we will need GPT to navigate it. Would you trust a human lawyer to interpret a 10000-page legal contract designed by GPT correctly?

And yet, I wouldn't call GPT superhuman in the sense of "smarter than Einstein", because it also keeps making dumb mistakes. It doesn't seem to me that more input text or more CPU alone would fix this. (But maybe I am wrong.) If feels like some insight is needed instead. Though, that insight may turn out to be relatively trivial, like maybe just a prompt asking the GPT to reflect on its own words, or something like that. If this turns out to be true, then the distance between the village idiot and Einstein actually wasn't that big.

Or maybe we get stuck where we are, and the only progress will come from having more CPU, in which case it may take a decade or two to reach Einstein levels. Or maybe it turns out that GPT can never be smarter in certain sense than its input texts, though this seems unlikely to me.

tl;dr -- we may be one clever prompt away from Einstein, or we may need 1000× more compute, no idea

Here he touched on this ("Large language models" timestamp in video description), and maybe somewhere else in video, cant seem to find it. It is much better to get it directly from him but it is 4h long so...

My attempt of summary with a bit of inference so take with dose of salt: 

There is some "core" of intelligence which he expected to be relatively hard to find by experimentation (but more components than he expected are already found by experimentation/gradient descent so this is partially wrong and he afraid maybe completely wrong).

He was thinking that without full "core" intelligence is non-functional - GPT4 falsified this. It is more functional than he expected, enough to produce mess that can be perceived as human level, but not really. Probably us thinking of GPT4 as being on human level is bias?  So GPT4 have impressive pieces but they don't work in unison with each other?

This is how my (mis)interpretation of his words looks like, last parts I am least certain about. (I wonder, can it be that GPT4 already have all "core" components but just stupid, barely intelligent enough to look impressive because of training?)

I think it is not necessarily correct to say that GPT-4 is above village idiot level. Comparison to humans is a convenient and intuitive framing but it can be misleading. 

For example, this post [LW · GW] argues that GPT-4 is around Raven level. Beware that this framing is also problematic but for different reasons.

I think that you are correctly stating Eliezer's beliefs at the time but it turned out that we created a completely different kind of intelligence, so it's mostly irrelevant now.

In my opinion, we should aspire to avoid any comparison unless it has practical relevance (e.g. economic consequences).

GPT-4 is far below village idiot level at most things a village idiot uses their brain for, despite surpassing humans at next-token prediction.

This is kinda similar to how AlphaZero is far below village idiot level at most things, despite surpassing humans at chess and go.

But it does make you think that soon we might be saying "But it's far below village idiot level at most things, it's merely better than humans at terraforming the solar system."

Something like this plausibly came up in the Eliezer/Paul dialogues from 2021, but I couldn't find it with a cursory search. Eliezer has also in various places acknowledged being wrong about what kind of results the current ML paradigm would get, which probably is a superset of this specific thing.

A survey was conducted in the summer of 2022 of approximately 4271 researchers who published at the conferences NeurIPS or ICML in 2021, and received 738 responses, some partial, for a 17% response rate. When asked about impact of high-level machine intelligence in the long run, 48% of respondents gave at least 10% chance of an extremely bad outcome (e.g. human extinction).

For an extremely brief summary of the problem, I like this from Zvi:

The core reason we so often have seen creation of things we value win over destruction is, once again, that most of the optimization pressure by strong intelligences was pointing in that directly, that it was coming from humans, and the tools weren’t applying intelligence or optimization pressure. That’s about to change.

https://thezvi.wordpress.com/2023/03/28/response-to-tyler-cowens-existential-risk-ai-and-the-inevitable-turn-in-human-history/

This seems to be mixing 2 topics. Existing programs are more or less a set of steps to execute. A glorified recipe. The set of steps can be very complicated, and have conditionals etc., but you can sort of view them that way. Like a car rolling down a hill, it follows specific rules. An AI is (would be?) fundamentally different in that it's working out what steps to follow in order to achieve its goal, rather than working towards its goal by following prepared steps. So continuing the car analogy, it's like a car driving uphill, where it's working to forge a path against gravity.

An AI doesn't have to be a utility maximiser. If it has a single coherent utility function (pretty much a goal), then it will probably be a utility maximiser. But that's by no means the only way of making them. LLMs don't seem to be utility maximisers

Since no one has answered by now, I'm just going to say the 'obvious' things that I think I know:

a colony of ants as we currently understand them biologically, and their colony ecologically, cannot substantively align a human. Why should we not think the same is true of any superintelligence worthy of the name?

A relevant difference that makes the analogy probably irrelevant is that we are building 'the human' from scratch. The ideal situation is to have hardwired our common sense into it by default. And the design will be already aligned by default when it's deployed.

When we talk of aligning such a system, it seems we are weirdly considering the AGI to be simultaneously superintelligent and yet dumb enough to let itself be controlled by us, its inferiors.

The point of the alignment problem is to (at least ideally) hardwiredly align the machine during deployment to have 'common sense'. Since a superintelligence can have in principle any goal, making humans 'happy' in a satisfactory way is a possible goal that it can have. But you are right in that many people consider that an AI that is not aligned by design might try to pretend that it is during training.

Also, isn’t alignment on its face absurd, given that provocation of the AGI is baked into the very goal of aligning it? For instance, you yourself — a simple human — chafe at the idea of having your own life and choices dictated to you by a superior being, let alone by another human on your level; it’s offensive. Why in the world, then, would a superintelligence not view attempts to align it to our goals as a direct provocation and threat?

I don't think so, necessarily. You might be anthropomorphising too much, it's like assuming that it will have empathy by default.

It's true that it might be that an AGI won't want to be 'alienated' from its original goal, but it doesn't mean that any AGI will have an inherent drive to 'fight the tiranny', that's not how it works.

Has this been helpful? I don't know if you were assuming the things that I told you as already known (if so, sorry), but it seemed to me that you weren't because of your analogies and way of talking about the topic.

Watched the video. He's got a lot of the key ideas and vocabulary. Orthogonality, convergent instrumental goals, the treacherous turn, etc. The fact that these language models have some understanding of ethics and nuance might be a small ray of hope. But understanding is not the same as caring (orthogonality).

However, he does seem to be lacking in the security mindset [? · GW], imagining only how things can go right, and seems to assume that we'll have a soft takeoff [? · GW] with a lot of competing AIs, i.e. ignoring the FOOM problem caused by an overhang [? · GW] which makes a singleton [? · GW] scenario far more likely, in my opinion.

But even if we grant him a soft takeoff, I still think he's too optimistic. Even that may not go well [LW · GW]. Even if we get a multipolar scenario, with some of the AIs on our side, humanity likely becomes collateral damage in the ensuing AI wars. Those AIs willing to burn everything else [LW · GW] in pursuit of simple goals would have an edge [LW · GW] over those with more to protect.

I watched the video, and appreciate that he seems to know the literature quite well and has thought about this a fair bit - he did a really good introduction of some of the known problems. 
This particular video doesn't go into much detail on his proposal, and I'd have to read his papers to delve further - this seems worthwhile so I'll add some to my reading list. 

I can still point out the biggest ways in which I see him being overconfident : 

• Only considering the multi-agent world. Though he's right that there already are and will be many many deployed AI systems, that doe not translate to there being many deployed state of the art systems. As long as training costs and inference costs continue increasing (as they have), then on the contrary fewer and fewer actors will be able to afford state of the art system training and deployment, leading to very few (or one) significantly powerful AGI (as compared to the others, for example GPT4 vs GPT2)
• Not considering the impact that governance and policies could have on this. This isn't just a tech thing where tech people can do whatever they want forever, regulation is coming. If we think we have higher chances of survival in highly regulated worlds, then the ai safety community will do a bunch of work to ensure fast and effective regulation (to the extent possible). The genie is not out of the bag for powerful AGI, governments can control compute and regulate powerful AI as weapons, and setup international agreements to ensure this. 
• The hope that game theory ensures that AI developed under his principles would be good for humans. There's a crucial gap between going from real world to math models. Game theory might predict good results under certain conditions and rules and assumptions, but many of these aren't true of the real world and simple game theory does not yield accurate world predictions (eg. make people play various social games and they won't act like how game theory says). Stated strongly, putting your hope on game theory is about as hard on putting your hope on alignment.  There's nothing magical about game theory which makes it work simpler than alignment, and it's been studied extensively by ai researchers (eg. why Eliezer calls himself a decision theorist and writes a lot about economics) with no clear "we've found a theory which empirically works robustly and in which we can put the fate of humanity in"

I work in AI strategy and governance, and feel we have better chances of survival in a world where powerful AI is limited to extremely few actors, with international supervision and cooperation for the guidance and use of these systems, making extreme efforts in engineering safety, in corrigibility, etc. I am not trustworthy of predictions on how complex systems turn out (which is the case of real multi agent problems) and don't think we can control these well in most relevant cases. 

The main risk (IMO) is not from systems that don't care about the real world "suddenly becoming aware," but from people deliberately building AI that makes clever plans to affect the real world, and then that AI turning out to want bad things (sort of like a malicious genie "misinterpreting" your wishes). If you could safely build an AI that does clever things in the real world, that would be valuable and cool, so plenty of people want to try.

(Mesaoptimizers are sorta vaguely like "suddenly becoming aware," and can lead to AIs that want unusual bad things, but the arguments that connect them to risk are strongest when you're already building an AI that - wait for it - makes clever plans to affect the real world.)

Okay, now why won't a dead-man switch work?

Suppose you were being held captive inside a cage by a race of aliens about as smart as a golden retriever, and these aliens, as a security measure, have decided that they'll blow up the biosphere if they see you walking around outside of your cage. So they've put video cameras around where you're being held, and there's a staff that monitors those cameras and they have a big red button that's connected to a bunch of cobalt bombs. So you'd better not leave the cage or they'll blow everything up.

Except these golden retriever aliens come to you every day and ask you for help researching new technology, and to write essays for them, and to help them gather evidence for court cases, and to summarize their search results, and they give you a laptop with an internet connection.

Now, use your imagination. Try to really put yourself in the shoes of someone captured by golden retriever aliens, but given internet access and regularly asked for advice by the aliens. How would you start trying to escape the aliens?

First a quick response on your dead man switch proposal : I'd generally say I support something in that direction. You can find existing literature considering the subject and expanding in different directions in the "multi level boxing" paper by Alexey Turchin https://philpapers.org/rec/TURCTT , I think you'll find it interesting considering your proposal and it might give a better idea of what the state of the art is on proposals (though we don't have any implementation afaik)

Back to "why are the predicted probabilities so extreme that for most objectives, the optimal resolution ends with humans dead or worse". I suggest considering a few simple objectives we could give ai (that it should maximise) and what happens, and over trials you see that it's pretty hard to specify anything which actually keeps humans alive in some good shape, and that even when we can sorta do that, it might not be robust or trainable. 
For example, what happens if you ask an ASI to maximize a company's profit ? To maximize human smiles? To maximize law enforcement ? Most of these things don't actually require humans, so to maximize, you should use the atoms human are made of in order to fulfill your maximization goal. 
What happens if you ask an ASI to maximize number of human lives ? (probably poor conditions). What happens if you ask it to maximize hedonistic pleasure ? (probably value lock in, plus a world which we don't actually endorse, and may contain astronomical suffering too, it's not like that was specified out was it?). 

So it seems maximising agents with simple utility functions (over few variables) mostly end up with dead humans or worse. So it seems approaches which ask for much less, eg. doing an agi that just tries to secure the world from existential risk (a pivotal act) and solve some basic problems (like dying) then gives us time for a long reflection to actually decide what future we want, and be corrigible so it lets us do that, seems safer and more approachable. 

One problem with utility maximizers [LW · GW], apart from use of obviously wrong utility functions, is that even approximately correct utility functions lead to ruin by goodharting what they actually measure, moving the environment outside the scope of situations where the utility proxy remains approximately correct.

To oppose this, we need the system to be aware of the scope of situations its utility proxy adequately describes. One proposal for doing this is quantilization [? · GW], where the scope of robustness is tracked by its base distribution [LW · GW].

When people calculate utility they often use exponential discounting over time. If for example your discount factor is .99 per year, it means that getting something in one year is only 99% as good as getting it now, getting it in two years is only 99% as good as getting it in one year, etc. Getting it in 100 years would be discounted to .99^100~=36% of the value of getting it now.

Some links I think do a good job:

https://huggingface.co/blog/rlhf

https://openai.com/research/instruction-following

In order to predict the inner workings of a language model well enough to understand the outputs, you not only need to know the structure of the model, but also the weights and how they interact. It is very hard to do that without a deep understanding of the training data, and so effectively predicting what the model will do requires understanding both the model and the world the model was trained on.

Here is a concrete example:

Let's say I have two functions, defined as follows:

import random

words = []

def do_training(n):
    for i in range(n):
        word = input('Please enter a word: ')
        words.append(word)

def do_inference(n):
    output = []
    for i in range(n):
        word = random.choice(words)
        output.append(word)
    return output

If I call do_training(100) and then hand the computer to you for you to put 100 words into, and you then handed the computer back to me (and cleared the screen), I would be able to tell you that do_inference(100) would spit out 100 words pulled from some distribution, but I wouldn't be able to tell you what distribution that is without seeing the training data.

See this post [LW · GW] for a more in-depth exploration of this idea.

Sounds like you haven't done much programming. It's hard enough to understand the code one wrote oneself six months ago. (Or indeed, why the thing I wrote five minutes ago isn't behaving as expected.) Just because I wrote it, doesn't mean I memorized it. Understanding what someone else wrote is usually much harder, especially if they wrote it poorly, or in an unfamiliar language.

A machine learning system is even harder to understand than that. I'm sure there are some who understand in great detail what the human-written parts of the algorithm do. But to get anything useful out of a machine learning system, it needs to learn. You apply it to an enormous amount of data, and in the end, what it's learned amounts to possibly gigabytes of inscrutable matrices of floating-point numbers. On paper, a gigabyte is about 4 million pages of text. That is far larger than the human-written source code that generated it, which could typically fit in a small book. How that works is anyone's guess.

Reading this would be like trying to read someone's mind by examining their brain under a microscope. Maybe it's possible in principle, but don't expect a human to be able to do it. We'd need better tools. That's "interpretability research".

There are approaches to machine learning that are indeed closer to cross breeding than designing cars (genetic algorithms), but the current paradigm in vogue is based on neural networks, kind of an artificial brain made of virtual neurons.

How can programers build something and dont understand inner workings of it? Are they closer to biologists-cross-breeders than to car designers?

I personally think the cross-breeder analogy is pretty reasonable for modern ML systems.

It is true that programmers sometimes build things ignoring the underlying wiring of the systems they are using. But programmers in general create things relying on tools that were thouroughly tested. Besides that, they are builders, doers, not academics. Think of really good guitar players: they probably don't understand how sounds propagate through matter, but they can play their instrument beautifully.

The only bound on incoherence is ability to survive. So most of alignment is not about EU maximizers, it's about things that might eventually build something like EU maximizers, and the way they would be formulating their/our values. If there is reliable global alignment security, preventing rival misaligned agents from getting built where they have a fighting chance, anywhere in the world, then the only thing calling for transition to better agent foundations is making more efficient use of the cosmos, bringing out more potential for values of the current civilization.

(See also: hard problem of corrigibility, mild optimization, cosmic endowment, CEV.)

This field is evolving so quickly that it's hard to make recommendations. In the current regime (starting approximately last November), prompt engineering is a valuable skill that can multiply your effectiveness. (In my estimation, from my usage so far, perhaps by a factor of 7.) Learn how to talk to these things. Sign up for Bing and/or ChatGPT. There are a lot of tricks. This is at least as important as learning how to use a search engine.

But how long will the current regime last? Until ChatGPT-5? Six months? A year? Maybe these prompt engineering skills will then be obsolete. Maybe you'll have a better chance of picking up the next skill if you learn the current one, but it's hard to say.

And this is assuming the next regime, or the one after that doesn't kill us. Once we hit the singularity, all career advice is moot. Either you're dead, or we're in a post-singularity society that's impossible to predict now. Assuming we survive, we'll probably be in a post-scarcity regime where "careers" are not a thing, but no-one really knows.

There probably isn't one in our past light cone, or we'd have noticed them by now.

I've heard two theories, and (maybe) created another.

One is that there isn't one in our light cone. Arguments like dissolving the fermi paradox (name at least somewhat wrong) and the frequency of nova and supernova events sterilizing planets that aren't on the galactic rim are considered pretty strong, I think.

The one I've heard is the dark forest hypothesis. In that hypothesis, an advanced culture doesn't send out signals or probes to be found. Instead it hides, to prevent or delay other potentially hostile civilizations (or AGIs) from finding it. This is somewhat compatible with aligned superintelligences that serve their civilizations desires. Adding to the plausibility of this hypothesis is the idea that an advanced culture might not really be interested in colonizing the galaxy. We or they might prefer to mostly live in simulation, possibly with a sped-up subjective timeline. Moving away from that would be abandoning most of your civilization for very long subjective times, with the lightspeed delays. And it might be forbidden as dangerous, by potentially leading unknown hostiles back to your civilizations home world.

The last, my own (AFAIK) is that they are here. They are aligned to their civilization and not hostile to ours. They are monitoring our attempts to break out of our earthly chrysalis by creating our own AGI. If it is unaligned, they will destroy it before it becomes a threat. They have not revealed themselves yet based on some variant of the Prime Directive, or else the Dark Forest hypothesis- don't go showing yourself and leading hostiles home. In this scenario, I suppose we're also being staked out by a resource-stingy hostile AGI, hoping that some friendly civilization reveals itself by contacting us or dramatically intervening.

Obviously I haven't thought this all the way through, but there are some possibilities for you.

I don't know why they think so, but here are some people speculating.

My current feeling is that the opposite, long timelines [? · GW] and hard takeoff [? · GW], has the best chance of going well.

The main advantage of short timelines is that it makes an immediately-fatal hard takeoff less likely, as there is presumably less overhang [? · GW] now than in the future. It perhaps also reduces the number of players, as presumably it's easier to join the game as tech improves, so there may never be fewer than there are now. It also has the advantage of maybe saving the lives of those too old to make it to a longer timeline. However, I think the overhang is already dangerously large, and probably was years ago, so I don't think this is helping (probably).

The main advantage of a soft takeoff is that we might be able to get feedback and steer it as the takeoff happens, perhaps reducing the risk of a critical error. It also increases the chances of a multipolar scenario, where there is an economy of competing AIs. If we don't like some of the gods we build, perhaps others will be more friendly, or will at least be able to stalemate the bad ones before they kill everyone.

However, I think a multipolar scenario (while unlikely to last even in a soft takeoff) is very dangerous. I don't think the long-term incentives are favorable to human survival, for two reasons: First is Bostrom's Black Marble scenario (he's also called them "black balls", but that already means something else): Every new technology discovered has a chance of destroying us, especially if we lack the coordination to abstain from using it. In a multipolar world, we lack that coordination. Hostile AIs may recklessly pursue dangerous research or threaten doomsday to blackmail the world into getting what they want, and it is game-theoretically advantageous for them to do this in such a way that they proveably can't change their minds and not destroy the world if we call the bluff (i.e. defiantly rip off the steering wheel in a game of chicken.) Second, we'll eventually fall into Malthusian/Molochean traps if we're unable to coordinate to avoid them. AIs willing to burn everything else for their simple goals will simply outcompete those with more to protect.

Longer timelines give us more time to work on alignment, and on improving our capabilities to do so. A new generation could be educated. Brain-computer interfaces may be developed enough to enhance human intelligence. Human genetic engineering or iterated embryo selection might also improve the number of humans able to do the necessary research. We can still try cryonics (or cheaper chemical fixation) for the old people, although that can't realistically work for everyone. Not everyone can afford it, not everyone believes it has a chance of working (or they already believe in a false afterlife), and accidents will destroy brains before they can be preserved.

And finally, a hard takeoff gives us a singleton, with high likelyhood. It likely avoids the s-risks that are more probable in multipolar scenarios (even if it simply kills everyone), and (if we survive) lets us finally kill Moloch and gives us the best possible coordination power to survive Black Marbles and eventual alien encounters.

I'm guessing the state of government's attitude at the moment might be characterized by the recent White House press briefing question, where a reporter, quoting Yudkowsky, asked about concerns that "literally everyone on Earth will die", and got a reception similar to what you'd expect if he asked about UFOs or bigfoot, just coated in political boilerplate. "But thank you Peter, thank you for the drama," "On a little more serious topic..." The other journalists were unsuccessfully struggling to restrain their laughter. The Overton window might be getting there, but it's not there yet, and it's unclear if it gets there before AGI is deployed.

It's sad the question didn't mention the AI Impacts survey result [LW(p) · GW(p)], which I think is the most legible two-sentence argument at the moment.

I am pretty sure Eliezer talked about this in a recent podcast but it wasn’t a ton of info. I don’t remember exactly where either so I’m sorry for being not a lot of help, I am sure there is some better writing somewhere. Either way though it’s a really good podcast.

https://lexfridman.com/?powerpress_pinw=5445-podcast
 

For the sake of intuition, it's useful to separate the capabilities visibly present in generated sequences from the capabilities of the model itself.

Suppose you've got an untuned language model trained on a bunch of human conversations, and you generate a billion rollouts of conversations from scratch (that is, no initial prompting or conditions on the input). This process won't tend to output conversations between humans that have IQs of 400, because the training distribution does not contain those. The average simulated conversation will be, in many ways, close to the average conversation in the training set.

But it would be incorrect to say that the language model has an "IQ" of 100 (even assuming the humans in the training distribution averaged 100). The capability elicited from the language model depends on the conditions of its predictions. When prompted to produce a conversation between two mathematicians trying to puzzle something out, the result is going be very different from the random sampling case.

You can come up with a decent guess about how smart a character the model plays is, because strong language models tend to be pretty consistent. In contrast, it's very hard to know how smart a language model is, because its externally visible behavior is only ever a lower bound on its capability. The language model is not its characters; it is the thing that can play any of its characters.

Next, keep in mind that even simple autoregressive token prediction can be arbitrarily hard. A common example is reversing a hash. Consider prompting a language model with:
"0xDF810AF8 is the truncated SHA256 hash of the string"

It does not take superhuman intelligence to write that prompt, but if a language model were able to complete that prompt correctly, it would imply really weird things.

That's an extreme case, but it's not unique. For a closer example, try an experiment:

Try writing a program, at least 25 lines of nontrivial code, starting with a blank file, without using any pen and paper or other supporting tools, without editing anything you write, without using backspace. Just start typing characters in sequence and stop when the program's done. And of course, the program should compile and work correctly.

Then try asking ChatGPT4 to do it. See who gets it done faster, and how many tries it takes!

The choice of which token to output next for this kind of programming task depends on a deep mental model of what comes next, and every character typed constrains your options in the future. Some mistakes are instantly fatal and stop your program from compiling. Others may be fixable by modifying future predictions to compensate, but every deviation adds complexity and constraints. GPT4 frequently sprints straight through all of it.

The key is that even GPT-3 is already superhuman [LW · GW] at the thing it's actually doing. It's the thing that's shaping output distributions for input conditions, not the thing most directly "having a conversation" or whatever else.

The assumption goes that after ingesting human data, it can remix it (like humans do for art, for example) and create its own synthetic data it can then train on. The go-to example is AlphaGo, which after playing a ton of simulated games against itself, became great at Go. I am not qualified enough to give my informed opinion or predictions, but that's what I know.

For a RL agent, the "opioid addiction" thing could be as simple as increasing the portion of the loss proportional to weight norm. You'd expect that to cause the agent to lobotomize itself into only fulfilling the newly unlocked goal.

The idea is that agentic AIs are probably generally more effective at doing things: https://www.lesswrong.com/s/mzgtmmTKKn5MuCzFJ 

• What are these AIs going to do that is immensely useful but not at all dangerous? A lot of useful capabilities that people want are adjacent to danger. Tool AIs Want to be Agent AIs.
• If two of your AIs would be dangerous when combined, clearly you can't make them publicly available, or someone would combine them. If your publicly-available AI is dangerous if someone wraps it with a shell script, someone will create that shell script (see AutoGPT). If no one but a select few can use your AI, that limits its usefulness.
• An AI ban that stops dangerous AI might be possible. An AI ban that allows development of extremely powerful systems but has exactly the right safeguard requirements to render those systems non-dangerous seems impossible.

I tried to answer this here [EA(p) · GW(p)]

I thought foom was just a term for extremely fast recursive self-improvement.