If GPT-6 is human-level AGI but costs $200 per page of output, what would happen?

post by Daniel Kokotajlo (daniel-kokotajlo) · 2020-10-09T12:00:36.814Z · LW · GW · 13 comments

This is a question post.

Contents

  Answers
    Dirichlet-to-Neumann
    ryan_b
    Daniel Kokotajlo
    ChristianKl
    ofer
    oceaninthemiddleofanisland
    Daniel Kokotajlo
    AnthonyC
    Zian
None
13 comments

I'm imagining a scenario in which OpenAI etc. continue to scale up their language models, and eventually we get GPT-6, which has the following properties:

--It can predict random internet text better than the best humans

--It can answer correctly questions which seem to require long chains of reasoning to answer

--With appropriate prompts it can write novel arguments, proofs, code, etc. of quality about equal to the stuff it has read on the internet. (The best stuff, if the prompt is designed correctly)

--With appropriate prompts it can give advice about arbitrary situations, including advice about strategies and plans. Again, the advice is about as good as the stuff it read on the internet, or the best stuff, if prompted correctly.

--It costs $200 per page of output, because just running the model requires a giant computing cluster.

My question is, how does this transform the world? I have the feeling that the world would be transformed pretty quickly. At the very least, the price of running the model would drop by orders of magnitude over the next few years due to algorithmic and hardware improvements, and then we'd see lots of jobs getting automated. But I'm pretty sure stuff would go crazy even before then. How?

(CONTEXT: I'm trying to decide whether "Expensive AGI" is meaningfully different from the usual AGI scenarios. If we get AGI but it costs $200 per page instead of $2, and thus isn't economically viable for most jobs, does that matter? EDIT: What if it costs $2,000 or $20,000 per page? Do things go FOOM soon even in that case?)

Answers

answer by Dirichlet-to-Neumann · 2020-10-09T12:32:17.135Z · LW(p) · GW(p)

$200 per page of quality proof output is super cheap - typical Fermi calculation shows a  typical mathematician cost about $100 000 a year, and output about 100* pages of peer-reviewed papers per years at best, so about $1000 per page for pure maths (applied maths are maybe half as expensive ?). 

So 1st consequence: every single theoretical scientist get fired. Computer scientist are more expensive and get fired even earlier. Also journalists, lawyers, accountants and basically any job that requires high writing skills. 

 

*wages may be about  $200 000 and output as high as 200 pages depending of country/talent/field but the order of magnitude is the same.

** first post here ;)

comment by Daniel Kokotajlo (daniel-kokotajlo) · 2020-10-09T18:31:52.455Z · LW(p) · GW(p)

Good point--some humans make more than $200 per page of output. Maybe I should have said $2000, or maybe not that it could produce the best-quality stuff, but instead just average-quality stuff.

answer by ryan_b · 2020-10-13T14:37:18.427Z · LW(p) · GW(p)

I believe the central impact will be a powerful compression of knowledge and a flood of legibility, which will be available to institutions and leadership first. Examples include:

  • Speechwriting
  • Report summarization
  • Report generation

Even the higher number, like $20,000 per page, is a good deal for something like Wikipedia, where the page is available to millions of readers, or for things like the Stanford Encyclopedia of Philosophy. This will have a big impact on:

  • Online encyclopedias
  • Online textbooks

While this could easily be used to generate high-quality propaganda, I feel like it still weighs much more heavily in favor of the truth. This is because bullshit's biggest advantage is that is is fast, cheap and easy to vary, whereas reality is inflexible and we comprehend it slowly. But under the proposed conditions, advanced bullshit and the truth cost the same amount, and have a similar speed. This leaves reality's inflexible pressure on every dimension of every problem a decisive advantage in favor of the truth. This has a big impact on things like:

  • Any given prediction
  • Project proposals

Especially if it is at the lower end of the price scale, it becomes trivial to feed it multiple prompts and get multiple interpretations of the same question. This will give us a lot of information both in terms of compression and also in terms of method, which will cause us to be able to redirect resources into the most successful methods, and also to drop inefficient ones. I further expect this to be very transparent very quickly, though mechanisms like:

  • Finance
  • Sports betting

It will see heavy use by the intelligence community. A huge problem we have in the United States is our general lack of language capability; for example if GPT-6 knows Mandarin as well as any Mandarin speaker, and translates to English as well as any translator, then suddenly we get through the bottleneck and gain access to good information about Chinese attitudes. I expect this same mechanism will make foreign investment much more attractive almost universally, since domestic and foreign firms will now be working on an almost level playing field in any country with widespread internet access. If this prediction holds, I expect a large boom in investment in otherwise underdeveloped countries, because the opportunities will finally be legible.

Another interesting detail is that if GPT-6 can provide the best summaries of the available knowledge, this means that most of the world's institutions will then be working from a much more uniform knowledge base than we do currently. My initial reaction was that this is clearly for the best because the biggest roadblock to coordination is getting on the same page with the other stakeholders, but it also occurs to me that it makes transparent to everyone the cases where certain stakeholders have an untenable position. I suspect this in turn makes it more likely that some parties get the screws put to them, and further when they a) understand their own position and b) understand that everyone else understands it, they are more likely to try something radical to shift the outcome. Consider North Korea, for example.

answer by Daniel Kokotajlo · 2020-10-09T12:08:16.856Z · LW(p) · GW(p)

Someone retrains the model using reinforcement learning to be more of an agent. Maybe a chatbot that tries to convince people of things, or give good advice, or make good predictions, or some combination thereof. This unlocks its hidden intelligence, so to speak, since it no longer thinks it is predicting other people's text. It's now basically a human-level AGI; it's expensive, but if they make it bigger and train it for longer maybe they can make a new version which is superhuman, and then things will go off the rails, because a superhuman AGI is well worth $20,000 per page of output.

However, this would take at least a few more months, probably more than a year, to achieve. I suspect that a lot of important things would happen sooner than that.

answer by ChristianKl · 2020-10-09T13:56:08.255Z · LW(p) · GW(p)

I think a GPT-6 that reaches AGI levels of ability while having high costs, it would likely have a larger attention amount and use part of it to have a working memory [LW · GW]. There's little to be gained by replacing the average human by a $200 per page AGI.

This means it can actually do long chains of reasoning by manipulating what's in his short term memory similar to how humans do long chains of reasoning by operating on what's in their working memory.

The ability to do reasoning the means that the quality isn't very dependent on what can be found on the internet. 

An AGI that's human level for the average problem likely has problems where it outperforms humans. 

comment by Donald Hobson (donald-hobson) · 2020-10-10T08:53:21.933Z · LW(p) · GW(p)

The ability to do reasoning the means that the quality isn't very dependent on what can be found on the internet. 

An AGI that's human level for the average problem likely has problems where it outperforms humans. 

The AI described isn't trying to outperform humans, its been optimised to imitate humans. Of course, there is a potential for mesa-optimization, but I don't think that would lead to a system that produced better text. (It might lead to the system producing strange or subtly manipulative text.)

comment by ChristianKl · 2020-10-10T14:44:15.946Z · LW(p) · GW(p)

The AGI has access to a bunch of background knowledge that humans don't have. No human mathematician will have read as much different math tools as the AGI.

There's the potential for the working memory to be comparatively larger then humans working memory in some cases. The AGI has a more explicit ability to add and remove items from it's working memory. 

Even if the AGI doesn't try to outperform humans it's different enough from humans that it won't have the same performance at every task as humans and likely be worse at some tasks while being better at others. 

comment by Donald Hobson (donald-hobson) · 2020-10-10T21:21:03.159Z · LW(p) · GW(p)

Take arithmetic. Lets assume that given the computational resources available, it would be utterly trivial to do perfect arithmetic. Lets also assume that the training data was written by people who were somewhat innumerate. Lets say that many of the arithmetical statements that appear in the training dataset are wrong. 

You give it the prompt "2+2=". The training data contained "2+2=7" as often as "2+2=4". The AI is only being selected towards the sort of text strings that exist in the training dataset. It has no concept that by "+" you mean addition and not something else. 

Of course, if humans give the correct answer 10% of the time, and 90% of the time give a wrong answer, but any particular wrong answer appears <1% of the time, you could find the right answer by taking the mode. 

comment by ChristianKl · 2020-10-11T19:25:00.716Z · LW(p) · GW(p)

It has no concept that by "+" you mean addition and not something else. 

If a lot of the trainings example say "2+2=7" then of cause the AI will not think that + means addition because it doesn't. If however people use + to mean addition GPT3 is already capable enough to learn the concept and use it to add numbers that aren't in it's training corpus. 

To have human level cognition you need the ability to use multiple concepts together in a new way. Knowing more mathematical concepts then the average mathematician might lead to better performance given that mathematical proofs are a lot about needing to know the concepts that are required for a given proof.

I also think that for GPT-x to reach AGI-hood it will need a large enough attention field to use part of that attention field as memory which means it can do reasoning in additional ways. 

comment by Donald Hobson (donald-hobson) · 2020-10-12T10:14:33.165Z · LW(p) · GW(p)

If however people use + to mean addition GPT3 is already capable enough to learn the concept and use it to add numbers that aren't in it's training corpus. 

Yes, but it will still be about as good as its training corpus.

One way of looking at this is that GPT-X is trying to produce text that looks just like human written text. Given two passages of text, there should be no easy way to tell which was written by a human, and which wasn't. 

GPT-X has expertise in all subjects, in a sense. Each time it produces text, it is sampling from the distribution of human competence. Detailed information about anteaters is in there somewhere, every now and again, it will sample an expert on them, but most of the time it will act like a person who doesn't know much about anteaters. 

answer by ofer · 2020-10-09T15:50:03.079Z · LW(p) · GW(p)

But I'm pretty sure stuff would go crazy even before then. How?

We can end up with an intelligence explosion via automated ML research. One of the tasks that could be automated by the language model is "brainstorming novel ML ideas". So you'll be able to pay $200 and get a text, that could have been written by a brilliant ML researcher, containing novel ideas that allow you to create a more efficient/capable language model. (Though I expect that this specific approach won't be competitive with fully automated approaches that do stuff like NAS.)

answer by oceaninthemiddleofanisland · 2020-10-09T22:18:54.505Z · LW(p) · GW(p)

'Predicting random text on the internet better than a human' already qualifies it as superhuman, as dirichlet-to-neumann pointed out. If you look at any given text, there's a given ratio of cognitive work needed to produce the text, per word-count. "Superhuman" only requires asking it to replicate the work of multiple people collaborating together, or processes which need a lot of human labour like putting together a business strategy or writing a paper. Assuming it's mediocre in some aspects, the clearest advantage GPT-6 would have would be an interdisciplinary one - pooling together domain knowledge from disparate areas to produce valuable new insights.

answer by Daniel Kokotajlo · 2020-10-11T07:12:42.752Z · LW(p) · GW(p)

Some tasks require much less than a page of output: --Brainstorming ideas, e.g. for product/character names, plot twists, etc. --Providing advice, e.g. as a therapist or financial advisor or lawyer --Answering questions, e.g. as a financial advisor or lawyer Perhaps there would be an economic niche for automating away some of these tasks.

answer by AnthonyC · 2020-10-25T16:45:40.658Z · LW(p) · GW(p)

Put it in the hands of inventors and researchers writing up business plans (with relevant metadata and project/company outcomes), and in the hands of VC, CVC, and other investors (including internally at larger companies). This would eliminate an enormous amount of friction, and energy wasted on both easily foreseeable mistakes and charades all parties have to play into today. That alone is sufficient to increase the rate of long-term economic and technological growth. By increasing ROR on investment, it will also increase total investment economy-wide. I'd expect this to become very common even at >$20k/page, since the expected values for success can be 8-9 figures and the vast majority of companies and projects fail.

answer by Zian · 2020-10-09T17:36:18.105Z · LW(p) · GW(p)

Based on "Why Tool AIs Want to Be Agent AI's" by Gwern, I would expect an AGI level GPT-6 to self improve and become a world gobbling AI.

The moment it gets a hint that it could answer better by getting (unknown bit of data from the Internet, extra memory, some other resource), the software's own utility function will push the machine in that direction.

comment by Daniel Kokotajlo (daniel-kokotajlo) · 2020-10-09T18:34:20.665Z · LW(p) · GW(p)

OK, but in this case I'm trying to imagine something that's not significantly smarter than humans. So it probably can't think of any self-improvement ideas that an AI scientist wouldn't have thought of already, and even if it did, it wouldn't have the ability to implement them without first getting access to huge supercomputers to re-train itself. Right?

comment by Zian · 2020-10-10T07:23:22.311Z · LW(p) · GW(p)

I worry that I'm splitting hairs now because it seems that the AI only needs to be clever enough to generate the following in response to a query :

The answer to your question will be provided more quickly if you provide 1 GB of RAM. (rinse and repeat until we get to an AI box)

13 comments

Comments sorted by top scores.

comment by rohinmshah · 2020-10-09T20:35:54.787Z · LW(p) · GW(p)

Seems to me like existing answers are assuming that this AI system outputs a page that is as good as the pages that the best humans at any particular intellectual job would output, which seems wild. I expect the world to be transformed before this point, and I'd call that a superintelligent AI system, not a "human-level" AI system.

(I also think that for many intellectual tasks, a page of output from the best humans would cost more than $200.)

comment by Daniel Kokotajlo (daniel-kokotajlo) · 2020-10-10T06:04:31.548Z · LW(p) · GW(p)

Makes sense. If you'd call that a superintelligent system, what would a human-level system be? Something which was as good as the best humans at some things, but only as good as the average human on most things? (just like many humans?)

Do you think making it $2,000 or $20,000 would make a difference?

Maybe you have thoughts on my purpose in asking this question: Is "Expensive AGI" a meaningfully different category of scenario from the usual AGI scenarios? Or is price a pretty unimportant variable? Maybe price is unimportant at this level of capability and generality, but very important at lower levels of capability and generality?

comment by rohinmshah · 2020-10-10T14:51:39.526Z · LW(p) · GW(p)

If you'd call that a superintelligent system, what would a human-level system be? Something which was as good as the best humans at some things, but only as good as the average human on most things?

Seems roughly correct, but most likely it will be superhuman at several things and worse than humans at many other things. It's hard to be more precise than this unfortunately.

Do you think making it $2,000 or $20,000 would make a difference?

Yeah at $20,000 I start having trouble imagining when this might be used.

Is "Expensive AGI" a meaningfully different category of scenario from the usual AGI scenarios?

It doesn't feel that different? Like, if that happened, probably the world doesn't change much, we get the cost down, and then we have non-expensive AGI. Seems like a standard continuous takeoff world. Presumably there can't be any hardware overhangs in such a world, though other kinds of overhangs (e.g. from recursive improvement) seem as likely as they are in other scenarios.

It's probably better for safety because we get more experience working with powerful AI systems before they go about transforming the world.

comment by Daniel Kokotajlo (daniel-kokotajlo) · 2020-10-11T12:08:58.294Z · LW(p) · GW(p)

I think the scenario you describe sounds sufficiently different to me to count as different. Like you say, we get more experience working with powerful AI systems because there's a substantial period where they exist but are too expensive to transform the world.

My current view is that things will be crazier than this. Were we to get expensive AGI, I think the world would be transformed (in the relevant sense, i.e. it being too late for us to change the course of history) before the usual cost decreases kick in. Like, usually it takes at least a year for the price to drop by an order of magnitude, right? My intuition is that if we had GPT-6 costing $20,000 it would take less than two years for us to reach the point at which it's too late.

As often happens, the responses to my question are helping me rethink how I should have phrased the question...

comment by ofer · 2020-10-09T15:50:45.607Z · LW(p) · GW(p)

Some quick thoughts/comments:

--It can predict random internet text better than the best humans

I wouldn't use this metric. I don't see how to map between it and anything we care about. If it's defined in terms of accuracy when predicting the next word, I won't be surprised if existing language models already outperform humans.

Also, I find the term "human-level AGI" confusing. Does it exclude systems that are super-human on some dimensions? If so, it seems too narrow to be useful. For the purpose of this post, I propose using the following definition: A system that is able to generate text in a way that allows to automatically perform any task that humans can perform by writing text.

comment by gwern · 2020-10-09T16:35:38.990Z · LW(p) · GW(p)

I wouldn't use this metric. I don't see how to map between it and anything we care about.

Nevertheless, it works. That's how self-supervised training/pretraining works.

If it's defined in terms of accuracy when predicting the next word, I won't be surprised if existing language models already outperforms humans.

They don't. GPT-3 is still, as far as I can tell, about twice as bad in an absolute sense as humans in text prediction: https://www.gwern.net/newsletter/2020/05#fn17

comment by ofer · 2020-10-09T18:35:04.912Z · LW(p) · GW(p)

Nevertheless, it works. That's how self-supervised training/pretraining works.

Right, I'm just saying that I don't see how to map that metric to things we care about in the context of AI safety. If a language model outperforms humans at predicting the next word, maybe it's just due to it being sufficiently superior at modeling low-level stuff (e.g. GPT-3 may be better than me at predicting you'll write "That's" rather than "That is".)

(As an aside, in the linked footnote I couldn't easily spot any paper that actually evaluated humans on predicting the next word.)

comment by gwern · 2020-10-09T19:44:43.402Z · LW(p) · GW(p)

(As an aside, in the linked footnote I couldn't easily spot any paper that actually evaluated humans on predicting the next word.)

Third paragraph:

GPT-2 was benchmarked at 43 perplexity on the 1 Billion Word (1BW) benchmark vs a (highly extrapolated) human perplexity of 12

https://www.gwern.net/docs/ai/2017-shen.pdf

The LAMBADA dataset was also constructed using humans to predict the missing words, but GPT-3 falls far short of perfection there, so while I can't numerically answer it (unless you trust OA's reasoning there), it is still very clear that GPT-3 does not match or surpass humans at text prediction.

comment by ofer · 2020-10-10T17:23:53.614Z · LW(p) · GW(p)

GPT-2 was benchmarked at 43 perplexity on the 1 Billion Word (1BW) benchmark vs a (highly extrapolated) human perplexity of 12

I wouldn't say that that paper shows a (highly extrapolated) human perplexity of 12. It compares human-written sentences to language model generated sentences on the degree to which they seem "clearly human" vs "clearly unhuman" as judged by humans. Amusingly, for every 8 human-written sentences that were judged as "clearly human", one human-written sentence was judged as "clearly unhuman". And that 8:1 ratio is the thing from which human perplexity is being derived from. This doesn't make sense to me.

If the human annotators in this paper had never annotated human-written sentences as "clearly unhuman", this extrapolation would have shown human perplexity of 1! (As if humans can magically predict an entire page of text sampled from the internet.)

The LAMBADA dataset was also constructed using humans to predict the missing words, but GPT-3 falls far short of perfection there, so while I can't numerically answer it (unless you trust OA's reasoning there), it is still very clear that GPT-3 does not match or surpass humans at text prediction.

If the comparison here is on the final LAMBADA dataset, after examples were filtered out based on disagreement between humans (as you mentioned in the newsletter), then it's an unfair comparison. The examples are selected for being easy for humans.

BTW, I think the comparison to humans on the LAMBADA dataset is indeed interesting in the context of AI safety (more so than "predict the next word in a random internet text"); because I don't expect the perplexity/accuracy to depend much on the ability to model very low-level stuff (e.g. "that's" vs "that is").

comment by Daniel Kokotajlo (daniel-kokotajlo) · 2020-10-09T19:46:07.715Z · LW(p) · GW(p)

OK, fair enough.

Yeah, human-level is supposed to mean not strongly superhuman at anything important, while also not being strongly subhuman in anything important.

comment by ofer · 2020-10-10T17:33:34.879Z · LW(p) · GW(p)

Yeah, human-level is supposed to mean not strongly superhuman at anything important, while also not being strongly subhuman in anything important.

I think that's roughly the concept Nick Bostrom used in Superintelligence when discussing takeoff dynamics. (The usage of that concept is my only major disagreement with that book.) IMO it would be very surprising if the first ML system that is not strongly subhuman at anything important would not be strongly superhuman at anything important (assuming this property is not optimized for).

comment by Daniel Kokotajlo (daniel-kokotajlo) · 2020-10-10T18:22:34.483Z · LW(p) · GW(p)

Yeah, I think I agree with that. Nice.

comment by ChristianKl · 2020-10-09T20:31:12.231Z · LW(p) · GW(p)

The most capable humans are often much more capable then the average and thus not superhuman. I remember the example of a hacker who gave a talk at the CCC about how he was in vacation in Taiwan and hacked their electronic payment system on the side. If you could scale him up 10,000 or 100,000 times the kind of cyberwar you could wage would be enormous.