When to assume neural networks can solve a problem

post by George (George3d6) · 2020-03-27T17:52:45.208Z · score: 13 (4 votes) · LW · GW · 12 comments

This is a link post for https://blog.cerebralab.com/When_to_assume_neural_networks_can_solve_a_problem

Contents

  A pragmatic guide
  The Various hues of AI risk
    1. The Bostromian position
    2. The Standard Position
    3. The misinformed position
  How people misunderstand what neural networks can do
  When to assume a neural network can solve a problem
  1.A neural network can almost certainly solve a problem if another ML algorithm has already succeeded.
    2. A neural network can almost certainly solve a problem very similar to ones already solved
  3. A neural network can solve problems that a human can solve with small-sized datapoints and little to no context
  4. A neural network might solve a problem when we are reasonably  sure it’s deterministic, we provide any relevant context as part of the  input data, and the data is reasonably small
  In conclusion
None
12 comments

Note: the original article has been split into two since I think the two points were only vaguely related, I will leave it as is here, since I'd rather not re-post stuff and I think the audience on LW might see the "link" between the two separate ideas presented here.

A pragmatic guide

Let’s begin with a gentle introduction in to the field of AI risk - possibly unrelated to the broader topic, but it’s what motivated me to write about the matter; it’s also a worthwhile perspective to start the discussion from. I hope for this article to be part musing on what we should assume machine learning can do and why we’d make those assumptions, part reference guide for “when not to be amazed that a neural network can do something”.

The Various hues of AI risk

I’ve often had a bone to pick against “AI risk” or, as I’ve referred to it, “AI alarmism”. When evaluating AI risk, there are multiple views on the location of the threat and the perceived warning signs.

1. The Bostromian position

I would call one of these viewpoints the “Bostromian position”, which seems to be mainly promoted by MIRI, philosophers like Nick Bostrom and on forums such as AI Alignment.

It’s hard to summarize without apparently straw-man arguments, e.g. “AIX + Moore’s law means that all powerful superhuman intelligence is dangerous, inevitable and close.” That’s partly because I’ve never seen a consistent top-to-bottom reasoning for it. Its proponents always seem to start by assuming things which I wouldn’t hold as given about the ease of data collection, the cost of computing power, the usefulness of intelligence.

I’ve tried to argue against this position, the summary of my view can probably be found in “Artificial general intelligence is here, and it's useless”. Whilst - for the reasons mentioned there - I don’t see it as particularly stable, I think it’s not fundamentally flawed; I could see myself arguing pro or con.

2. The Standard Position

Advocated by people ranging from my friends, to politicians, to respectable academics, to CEOs of large tech companies. It is perhaps best summarized in Stuart Russell’s book Human Compatible: Artificial Intelligence and the Problem of Control.

This viewpoint is mainly based around real-world use cases for AI (where AI can be understood as “machine learning”). People adopting this perspective are not wrong in being worried, but rather in being worried about the wrong thing.

It’s wrong to be upset by Facebook or Youtube using an algorithm to control and understand user preferences and blaming it on “AI”, rather than on people not being educated enough to use TOR, install a tracking blocker, use Ublock Origin and not center their entire life around conspiracy videos in their youtube feed or in anti-vaccination facebook groups.

It’s wrong to be alarmed by Amazon making people impulse-buy via a better understanding of their preferences , and thus getting them into inescapable debt, rather than by the legality of providing unethically leveraged debt so easily.

It’s wrong to fuss about automated trading being able to cause sudden large dips in the market, rather than about having markets so unstable and so focused on short-term trading as to make this the starting point of the whole argument.

It’s wrong to worry about NLP technology being used to implement preventive policing measures, rather than about governments being allowed to steal their citizens’ data, to request backdoors into devices and to use preventive policing to begin with.

It’s wrong to worry about the Chinese Communist Party using facial recognition and tracking technology to limit civil rights; Worry instead about CCP ruling via a brutal dictatorship that implements such measures without anybody doing something against it.

But I digress, though I ought to give a full rebuttal of this position at some point.

3. The misinformed position

A viewpoint distinct from the previous two. It stems from misunderstanding what machine learning systems can already do. It basically consists in panicking over “new developments” which actually have existed for decades.

This view is especially worth fighting against, since it’s based on misinformation. Whereas with categories 1 and 2 I can see valid arguments arising for regulating or better understanding machine learning systems (or AI systems in general), people in the third category just don’t understand what’s going on, so they are prone to adopt any view out of sheer fear or need of belonging, without truly understanding the matter.

Until recently I thought this kind of task was better left to PBS. In hindsight, I’ve seen otherwise smart individuals being amazed that “AI” can solve a problem which anyone that has actually worked with machine learning would have been able to tell you is obviously solvable and has been since forever.

Furthermore, I think addressing this viewpoint is relevant, as it’s actually challenging and interesting. The question of “What are the problems we should assume can be solved with machine learning?”, or even narrower and more focused on current developments “What are the problems we should assume a neural network should be able to solve?”, is one I haven’t seen addressed much.

There are theories like PAC learning and AIX which at a glance seem to revolve around this, as it pertains to machine learning in general, but if actually tried in practice won’t yield any meaningful answer.

How people misunderstand what neural networks can do

Let’s look at the general pattern of fear generated by misunderstanding the machine learning capabilities we’ve had for decades.

An example that originally motivated this article is Scott Alexander’s article post about being amazed that GPT-2 is able to learn how to play chess, poorly.

It seems to imply that GPT-2 playing chess well enough not to lose very badly against a medicore opponent (the author) is impressive and surprising.

Actually, the fact that a 1,500,000,000-parameter model designed for sequential inputs can be trained to kind of play chess is rather unimpressive, to say the least. I would have been astonished if GPT-2 were unable to play chess. Fully connected models a hundred times smaller ( https://github.com/pbaer/neural-chess) could do that more than 2 years ago.

The successful training of GPT-2 is not a feat because if a problem like chess has been already solved using various machine learning models we can assume it can be done with a generic neural network architecture (e.g. any given FC net or a FC net with a few attention layers) hundreds or thousands of times larger in terms of parameters.

When to assume a neural network can solve a problem

In the GPT-2 example, transformers (i.e. the BERT-like models inspired by the “Attention is all you need” paper’s proposed design) are pretty generic as far as NN architectures go. Not as generic as a fully connected net, arguably; they seem to perform more efficiently (in terms of training time and model size) on many tasks, and they are much better on most sequential input tasks.

So when should we assume that such generic NN architectures can solve a problem?

The answer might ease uniformed awe and might be relevant to actua problems – the kind for which “machine learning” might have been considered, but with doubt whether it’s worth bothering.

Playing chess decently is also a problem already solved. It can be done using small (compared to GPT-2) decision trees and a few very simple heuristics (see for example: https://github.com/AdnanZahid/Chess-AI-TDD). If a much smaller model can learn how to play “decently”, we should assume that a fairly generic, exponentially larger neural network can do the same.

The rule of thumb is:

1.A neural network can almost certainly solve a problem if another ML algorithm has already succeeded.

Given a problem that can be solved by an existing ML technique, we can assume that a somewhat generic neural network, if allowed to be significantly larger, can also solve it.

This assumption doesn’t always hold because:

As we are focusing mainly on generalizable neural network architectures (e.g. a fully connected net, which is what most people think of initially when they hear “neural network”), point a) is pretty irrelevant.

Given that most heuristics are applied equally well to any model, even for something like chess, and that size can sometimes be enough for the network to be able to just learn the heuristic, this rule basically holds almost every time.

I can’t really think of a counter example here… Maybe some specific types of numeric projections?

This is a rather boring first rule, yet worth stating as a starting point to build up from.

2. A neural network can almost certainly solve a problem very similar to ones already solved

Let’s say you have a model for predicting the risk of a given creditor based on a few parameters, e.g. current balance, previous credit record, age, driver license status, criminal record, yearly income, length of employment, {various information about current economic climate}, marital status, number of children, porn websites visited in the last 60 days.

Let’s say this model “solves” your problem, i.e. it predicts risk better than 80% of your human analysts.

But GDPR rolls along and you can no longer legally spy on some of your customers’ internet history by buying that data. You need to build a new model for those customers.

Your inputs are now truncated after and the customer’s online porn history is no longer available (or rather admittedly usable).

Is it safe to assume you can still build a reasonable model to solve this problem ?

The answer is almost certainly “yes; given our knowledge of the world, we can safely assume someone’s porn browsing history is not that relevant to their credit rating as some of those other parameters.

Another example: assume you know someone else is using a model, but their data is slightly different from yours.

You know a US-based snake-focused pet shop that uses previous purchases to recommend products and they’ve told you it’s done quite well for their bottom line. You are a UK-based parrot-focused pet shop. Can you trust their model or a similar one to solve your problem, if trained on your data ?

Again, the right answer is probably “yes”, because the data is similar enough. That’s why building a product recommendation algorithm was a hot topic 20 years ago, but nowadays everyone and their mom can just get a wordpress plugin for it and get close to Amazon’s level.

Or, to get more serious, let’s say you have a given algorithm for detecting breast cancer that - if trained on 100,000 images with follow-up checks to confirm the true diagnostics - performs better than an average radiologist.

Can you assume that, given the ability to make it larger, you can build a model to detect cancer in other types of soft tissue, also better than a radiologist ?

Once again, the answer is yes. The argument here is longer, because we aren’t so certain, mainly because of the lack of data. I’ve spent more or less a whole article arguing that the answer would still be yes.

In NLP the exact same neural network architectures seem to be decently good at doing translation or text generation in any language, as long as it belongs to the Indo European family and there is a significant corpus of data for it (i.e. equivalent to that used for training the extant models for English).

Modern NLP techniques seem to be able to tackle all language families, and they are doing so with less and less data. To some extent, however, the similarity of the data and the amount of training examples are tightly linked to the ability of a model in quickly generalizing for many languages.

Or looking at image recognition and object detection/boxing models, the main bottleneck consists in large amounts of well-labeled data, not the contents of the image. Edge cases exist, but generally all types of objects and images can be recognized and classified if enough examples are fed into an architecture originally designed for a different image task (e.g. a conv residual network designed for imagenet).

Moreover, given a network trained on imagenet, we can keep the initial weights and biases (essentially what the network “has learned”) instead of starting from scratch, and it will be able to “learn” on different datasets much faster from that starting point.

3. A neural network can solve problems that a human can solve with small-sized datapoints and little to no context

Let’s say we have 20x20px black and white images of two objects never seen before; they are “obviously different”, but not known to us . It’s reasonable to assume that, given a bunch of training examples, humans would be reasonably good at distinguishing the two.

It is also reasonable to assume, given a bunch of examples (let’s say 100), that almost any neural network of millions of parameters would ace this problem like a human.

You can visualize this in terms of amounts of information to learn. In this case, we have 400 pixels of 255 values each, so it’s reasonable to assume every possible pattern could be accounted for with a few million parameters in our equation.

But what “small datapoints” means here is the crux of this definition.

In short, “small” is a function of:

Take a classic image classification task like MNIST. Although a few minor improvements have been made, the state-of-the-art for MNIST hasn’t progressed much. The last 8 years have yielded an improvement from ~98.5% to ~99.4%, both of which are well within the usual “human error range”.

Compare that to something much bigger in terms of input and output size, like ImageNet, where the last 8 years have seen a jump from 50% to almost 90%.

Indeed, even with pre-CNN techniques, MNIST is basically solveable.

But even having defined “small” as a function of the above, we don’t have the formula for the actual function. I think that is much harder, but we can come up with a “cheap” answer that works for most cases - indeed, it’s all we need:

This might sound like a silly heuristic, but it holds surprisingly well for most “easy” machine learning problems. For instance, the reason many NLP tasks are now more advanced than most “video” tasks is size, despite the tremendous progress on images in terms of network architecture (which are much closer to the realm of video). The input & output size for meaningful tasks on videos is much larger; on the other hand, even though NLP is in a completely different domain, it’s much closer size-wise to image processing.

Then, what does “little to no context” mean ?

This is a harder one, but we can rely on examples with “large” and “small” amounts of context.

You can try to predict the stock market based purely on indicators about the stock market, but this is not the way most humans are solving the problem.

In that sense, an ML algorithm would likely produce results similar to a mathematician in solving the equation, since the context would be basically non-existent for the human.

There are certainly some limits. Unless we test our machine at 4,000 C the algorithm has no way of knowing that the yield will be 0 because the machine will melt; an engineer might suspect that.

So, I can formulate this 3rd principle as:

A generic neural network can probably solve a problem if:

Feel free to change my mind (with examples).

However, this still requires evaluating against human performance. But a lot of applications of machine learning are interesting precisely because they can solve problems humans can’t. Thus, I think we can go even deeper.

4. A neural network might solve a problem when we are reasonably sure it’s deterministic, we provide any relevant context as part of the input data, and the data is reasonably small

Here I’ll come back to one of my favorite examples - protein folding. One of the few problems in science where data is readily available, where interpretation and meaning are not confounded by large amounts of theoretical baggage, and where the size of a datapoint is small enough based on our previous definition. You can boil down the problem to:

This is one example. Like most NLP problems, where “size” becomes very subjective, we could easily argue one-hot-encoding is required for this type of inputs; then the size suddenly becomes 40,000 (there’s 20 proteinogenic amino acids that can be encoded by DNA) or 42,000 (if you care about selenoproteins and 44,000 if you care about niche proteins that don’t appear in eukaryotes).

It could also be argued that the input & output size is much smaller, since in most cases proteins are much smaller and we can mask & discard most of inputs & outputs for most cases.

Still, there are plenty of tasks that go from an, e.g. 255x255 pixel image to generate another 255x255 pixel image (style alternation, resolution enhancement, style transfer, contour mapping… etc). So based on this I’d posite the protein folding data is reasonably small and has been for the last few years.

Indeed, resolution enhancement via neural networks and protein folding via neural networks came about at around the same time (with every similar architecture, mind you). But I digress; I’m mistaking a correlation for the causal process that supposedly generated it. Then again, that’s the basis of most self-styled “science” nowadays, so what is one sin against the scientific method added to the pile ?

Based on my own fooling around with the problem, it seems that even a very simple model, simpler than something like VGG, can learn something ”meaningful” about protein folding. It can make guesses better than random and often enough come within 1% of the actual position of the atoms, if given enough (135 millions) parameters and half a day of training on an RTX2080. I can’t be sure about the exact accuracy, since apparently the exact evaluation criterion here is pretty hard to find and/or understand and/or implement for people that aren’t domain experts… or I am just daft, also a strong possibility.

To my knowledge the first widely successful protein folding network AlphaFold, whilst using some domain-specific heuristics, did most of the heavy lifting using a residual CNN, an architecture designed for categorizing images, something as widely unrelated with protein folding as one can think of.

That is not to say any architecture could have tackled this problem as well. It rather means we needn’t build a whole new technique to approach this type of problem. It’s the kind of problem a neural network can solve, even though it might require a bit of looking around for the exact network that can do it.

The other important thing here is that the problem seems to be deterministic. Namely:

The issue arises when thinking about b), that is to say, we know that the universe can deterministically fold peptides; we know amino acids are enough to accurately describe a peptide. However, the universe doesn’t work with “amino acids”, it works with trillions of interactions between much smaller particles.

So while the problem is deterministic and self-contained, there’s no guarantee that learning to fold proteins doesn’t entail learning a complete model of particle physics that is able to break down each amino acid into smaller functional components. A few million parameters wouldn’t be enough for that task.

This is what makes this 4th most generic definition the hardest to apply.

Some other examples here are things like predictive maintenance where machine learning models are being actively used to tackle problems human can’t, at any rate not without mathematical models. For these types of problems, there’s strong reasons to assume, based on the existing data, that the problems are partially (mostly?) deterministic.

There are simpler examples here, but I can’t think of any that, at the time of their inception, didn’t already fall into the previous 3 categories. At least, none that aren’t considered reinforcement learning.

The vast majority of examples fall within reinforcement learning, where one can solve an impressive amount of problems once they are able to simulate them.

People can find optimal aerodynamic shapes, design weird antennas to provide more efficient reception/coverage, beat video games like DOT and Starcraft which are exponentially more complex (in terms of degrees of freedom) than chess or Go.

The problem with RL is that designing the actual simulation is often much more complicated than using it to find a meaningful answer. RL is fun to do but doesn’t often yield useful results. However, edge cases do exist where designing the simulation does seem to be easier than extracting inferences out of it. Besides that, the more simulations advance based on our understanding of efficiently simulating physics (in itself helped by ML), the more such problems will become ripe for the picking.

In conclusion

I’ve attempted to provide a few simple heuristics for answering the question “When should we expect that a neural network can solve a problem ?”. That is to say, to what problems should you apply neural networks, in practice, right now. What problems should leave you “unimpressed” when solved by a neural network ? For which problems should our default hypothesis include their solvability, given enough architecture searching and current GPU capabilities.

I think this is fairly useful - not only for not getting impressed when someone shows us a party trick and tells us it’s AGI - but also for helping us quickly classify a problem as “likely solvable via ML” and “unlikely to be solved by ML”

To recap, neural networks can probably solve your problem:

  1. [Almost certainty] If other ML models already solved the problem.
  2. [Very high probability] If a similar problem has already been solved by an ML algorithm, and the differences between that and your problem don’t seem significant.
  3. [High probability] If the inputs & outputs are small enough to be comparable in size to those of other working ML models AND if we know a human can solve the problem with little context besides the inputs and outputs.
  4. [Reasonable probability] If the inputs & outputs are small enough to be comparable in size to those of other working ML models AND we have a high certainty about the deterministic nature of the problem (that is to say, about the inputs being sufficient to infer the outputs).

I am not certain about any of these rules, but this comes back to the problem of being able to say something meaningful. PACL can give us almost perfect certainty and is mathematically valid but it breaks down beyond simple classification problems.

Coming up with this kind of rules doesn’t provide an exact degree of certainty and they are derived from empirical observations. However, I think they can actually be applied to real world problems.

Indeed, these are to some extent the rules I do apply to real world problems, when a customer or friend asks me if a given problem is “doable”. These seem to be pretty close to the rules I’ve noticed other people using when thinking about what problems can be tackled.

So I’m hoping that this could serve as an actual practical guide for newcomers to the field, or for people that don’t want to get too involved in ML itself, but have some datasets they want to work on.

12 comments

Comments sorted by top scores.

comment by Vaniver · 2020-03-27T22:33:05.132Z · score: 5 (2 votes) · LW(p) · GW(p)

That’s partly because I’ve never seen a consistent top-to-bottom reasoning for it.

I think it's difficult to find a consistent top-to-bottom story because the overall argument is disjunctive.

That is, a conjunction is the intersection of different events ("the sidewalk is wet and it's raining" requires it to both be true that "the sidewalk is wet" and "it's raining"), whereas a disjunction is the union of different (potentially overlapping) events ("the sidewalk is wet" can be reached by either "the sidewalk is wet and it's raining" and "the sidewalk is wet and the fire hydrant is leaking").

So if you have a conclusion, like "autonomous vehicles will be commercially available in 2030", the more different ways there are for it to be true, the more likely it is. But also, the more different ways there are for it to be true, the less it makes sense to commit to any particular way. "Autonomous cars are commercially available in 2030 because Uber developed them" has more details, but those details are burdensome [LW · GW].

Also, it seems important to point out that the Bostromian position is about the future. That is, the state of autonomous vehicles today can tell you about whether or not they'll be commercially available in 2030, but there's no hard evidence and it requires careful reasoning to just get provisional conclusions. 

And thus, just like the state of neural networks in 2010 was only weakly informative about what would be possible in 2020, it seems reasonable to expect the state of things in 2020 will be only weakly informative and about will be possible in 2030. Which is a very different question from how you should try to solve practical problems now.

comment by George (George3d6) · 2020-03-27T22:37:08.969Z · score: 5 (2 votes) · LW(p) · GW(p)

I will probably be stealing the perspective of the view being disjunctive as a way to look at why it's hard to pin down.

And thus, just like the state of neural networks in 2010 was only weakly informative about what would be possible in 2020, it seems reasonable to expect the state of things in 2020 will be only weakly informative and about will be possible in 2030.

This statement I would partially disagree with.

I think the idea of training on a GPU was coming to the forefront by 2010 and also the idea of CNNs for image recognition: https://hal.inria.fr/inria-00112631/document (see both in that 2006 paper)y K. et al. (2006)

I'd argue it's fairly easy to look at today' landscape and claim that by 2030 the things that are likely to happen include:

  • ML playing any possible game better than humans assuming a team actually works on that specific game (maybe even if one doesn't), with huma-like inputs and human-like limitations in terms of granularity of taking inputs and giving outputs.
  • ML achieving all the things we can do with 2d images right now for 3d images and short (e.g. < 5 minute) videos.
  • Algorithms being able to write e.g. articles summarizing various knowledge it gathers from given sources and possibly even find relevant sources via searching based on keywords (so you could just say "Write an article about Peru's economic climate in 2028, rather than feed a bunch of articles about Peru's economy in 2028)... the second part is already doable, but I'm mentioning them together since I assume people will be more impressed with the final product
  • Algorithms being able to translate from and to almost any language about as well as human, but still not well enough to translate sources which require a lot of interpretation (e.g. yes for translating a biology paper from english to hindi or vice versa, no for translating a phenomenology paper from english to hindi or vice versa)
  • Controlling mechanical systems (e.g. robotic arms) via networks trained using RL.
  • Generally speaking, algorithms being used in areas where they already out-perform humans but where regulations and systematic inefficiencies combined with issues of stake don't currently allow them to be used (e.g. accounting, risk analysis, setting insurance policies, diagnosis, treatment planning). Algorithms being jointly used to help in various scientific fields by replacing the need for humans to use classical statistics and or manually fitting equations in order to model certain processes.

I'd wager points 1 to 4 are basically a given, point 5 is debatable since it depends on human regulators and cultural acceptance for the most part.

I'd also wager than, other than audio processing, there won't be much innovation beyond those 5 points that will create load of hype by 2030. You might have ensembles of those 4 things building up to something bigger, but those 5 things will be at the core of it.

But that's just my intuition, partially based on the kind of heuristics above about what is easily doable and what isn't. But alas, the point of the article was to talk about what's doable in the present, rather than what to expect from the future, so it's not really that related.

comment by maximkazhenkov · 2020-03-29T19:35:50.664Z · score: -2 (2 votes) · LW(p) · GW(p)
ML playing any possible game better than humans assuming a team actually works on that specific game (maybe even if one doesn't), with huma-like inputs and human-like limitations in terms of granularity of taking inputs and giving outputs.

I disagree with this point in particular. I'm assuming you're basing this prediction on the recent successes of AlphaStar and OpenAI5, but there are obvious cracks upon closer inspection.

The "any possible game" part, though, is the final nail in the coffin to me since you can conceive plenty of games that are equivalent or similar to the Turing test, which is to say AGI-complete.

(Although I guess AGI-completeness is a much smaller deal to you)

comment by George (George3d6) · 2020-03-29T19:53:02.522Z · score: 1 (1 votes) · LW(p) · GW(p)
Turing test, which is to say AGI-complete

You are aware chatbots have been "beating" the original Turing test since 2014, right? (And arguably even before)

Also, AGI-complete == fools 1/3 of human judges in an x minute conversation via text? Ahm, no, just no.

That statement is meaningless unless you define the Turing test and keeps being meaningless even if you define the turing test, there is literally no definition for "AGI complete". AGI is more of a generic term used to mean "kinda like a human", but it's not very concrete.


On the whole, yes, some games might prove too difficult for RL to beat... but I can't think of any in particular. I think the statement hold for basically any popular competitive game (e.g. one where there are currently cash prizes above > 1000$ to be won). I'm sure one could design an adversarial game specifically designed to not be beaten by RL but doable by a human... but that's another story. Turing test, which is to say AGI-complete

comment by maximkazhenkov · 2020-03-29T23:50:54.923Z · score: 1 (1 votes) · LW(p) · GW(p)
You are aware chatbots have been "beating" the original Turing test since 2014, right?

Yes, I was in fact. Seeing where this internet argument is going, I think it's best to leave it here.

comment by George (George3d6) · 2020-03-30T01:54:01.477Z · score: 1 (1 votes) · LW(p) · GW(p)
Yes, I was in fact. Seeing where this internet argument is going, I think it's best to leave it here.

So, in that case.

If your original chain of logic is:

1. An RL-based algorithm that could play any game could pass the turing test

2. An algorithm that can pass the Turing test is "AGI complete", thus it is unlikely that (1) will happen soon

And you agree with the statement:

3. An algorithm did pass the Turing test in 2014

You either:

a) Have a contradiction

b) Must have some specific definition of the Turing test under which 3 is untrue (and more generally, no known algorithm can pass the Turing test)

I assume your position here is b and I'd love to hear it.

I'd also love to hear the causal reasoning behind 2. (maybe explained by your definition of the Turing test ?)

If your definitions differ from commonly accepted definitions and your you rely on causality which is not widely implied, you must at least provide your versions of the definitions and some motivation behind the causality.

comment by Daniel Kokotajlo (daniel-kokotajlo) · 2020-03-27T22:39:05.678Z · score: 4 (3 votes) · LW(p) · GW(p)

Thanks for writing this and posting it here. I for one am a big fan of the "Bostromian position" as you call it, and moreover I think Stuart Russell is too. ("Human Compatible" is making basically the same points as "Superintelligence," only in a dumbed-down and streamlined manner, with lots of present-day examples to illustrate.) So I don't think your dismissal of positions 1 and 2 is fair. But I'm glad to see dialogue happening between the likes of me and the likes of you.

Moreover, I think you are actually right about Scott Alexander's GPT-2 chess thingy. As you've explained, we knew neural nets could do this sort of thing already, and so we shouldn't be too surprised if GPT-2 can do it too with a little retraining.

I suppose, in Scott's defense, perhaps he wasn't surprised but rather just interested, and using the fact that GPT-2 can play chess to argue for some further claim about how useful neural nets are in general and how soon AGI will appear. But I currently prefer your take.

comment by rohinmshah · 2020-03-28T17:03:40.515Z · score: 10 (4 votes) · LW(p) · GW(p)
"Human Compatible" is making basically the same points as "Superintelligence," only in a dumbed-down and streamlined manner, with lots of present-day examples to illustrate.

I do not agree with this. I think the arguments in Human Compatible are more convincing than the ones in Superintelligence (mostly because they make fewer questionable assumptions).

(I agree that Stuart probably does agree somewhat with the "Bostromian position".)

comment by Daniel Kokotajlo (daniel-kokotajlo) · 2020-03-28T17:48:56.084Z · score: 5 (3 votes) · LW(p) · GW(p)

Interesting. Well, I imagine you don't have the time right now, but I just want to register that I'd love to hear more about this. What questionable assumptions does Superintelligence make, that aren't made by Human Compatible? (This request for info goes out to everyone, not just Rohin)

comment by George (George3d6) · 2020-03-28T09:56:18.815Z · score: 1 (1 votes) · LW(p) · GW(p)
moreover I think Stuart Russell is too

Yes, I guess I should have made the clarification about that, I don't think Stuart Russell is necessarily much divergent from Bostrom in his views. Rather, he's most poniente arguments seem not to be very related to that view, so I think his books is a good guide for what I labeled as the second view in the article.

But he certainly tries to uphold both.

However the article was already too long and going into that would have made it even longer.... in hindsight I've decided to just split it into two, but the version here I shall leave as is.

comment by Daniel Kokotajlo (daniel-kokotajlo) · 2020-03-28T11:08:50.970Z · score: 1 (2 votes) · LW(p) · GW(p)
It’s hard to summarize without apparently straw-man arguments, e.g. “AIX + Moore’s law means that all powerful superhuman intelligence is dangerous, inevitable and close.” That’s partly because I’ve never seen a consistent top-to-bottom reasoning for it. Its proponents always seem to start by assuming things which I wouldn’t hold as given about the ease of data collection, the cost of computing power, the usefulness of intelligence.

I object to pretty much everything in this quote. I think the straw-man argument you give is pretty obviously worse than many other summaries you could give, e.g. Stuart Russell's "Look, humans have a suite of mental abilities that gives them dominance over all other life forms on this planet. The goal of much AI research is to produce something which is better in those mental abilities than humans. What if we succeed? We'd better figure out how to prevent history from repeating itself, and we'd better do it before it's too late."

Also no one in the AI safety sphere thinks that all powerful superhuman intelligence is dangerous; otherwise what would be the point of AI alignment research?

Also if you read almost anything on the subject, people will be constantly saying how they don't think superhuman intelligence is inevitable or close. Have you even read Superintelligence?

What do you mean, you've never seen a consistent top-to-bottom reasoning for it? This is not a rhetorical question, I am just not sure what you mean here. If you are accusing e.g. Bostrom of inconsistency, I am pretty sure you are wrong about that. If you are just saying he hasn't got an argument in premise-conclusion form, well, that seems true but not very relevant or important. I could make one for you if you like.

I don't know what assumptions you think the case for AI safety depends on -- ease of data collection? Cost of computing power? Usefulness of intelligence? -- but all three of these things seem like things that people have argued about at length, not assumed. Also the case for AI safety doesn't depend on these things being probable, only on them being not extremely unlikely.

comment by George (George3d6) · 2020-03-28T11:50:59.682Z · score: 1 (1 votes) · LW(p) · GW(p)
Also if you read almost anything on the subject, people will be constantly saying how they don't think superhuman intelligence is inevitable or close

If it's "meaningfully close enough to do something about it" I will take that as being 'close". I don't think Bostrom puts a number on it, or I don't remember him doing so, but he seems to address a real possibility rather than a hypothetical that is hundreds or thousands of years away.

What do you mean, you've never seen a consistent top-to-bottom reasoning for it? This is not a rhetorical question, I am just not sure what you mean here. If you are accusing e.g. Bostrom of inconsistency, I am pretty sure you are wrong about that.

I mean, I don't see a chain of conclusions that leads to the theory being "correct" , Vaniver [LW · GW] mentioned bellow how this is not the correct perspective to adopt and I agree with that.... or I would, assuming that the hypothesis would be Popperian (i.e. that one could do something to disprove AI being a large risk in the relative near future).

If you are just saying he hasn't got an argument in premise-conclusion form, well, that seems true but not very relevant or important. I could make one for you if you like.

If you could make such a premise-conclusion case I'd be more then welcome to hear it out.


ease of data collection? Cost of computing power? Usefulness of intelligence? -- but all three of these things seem like things that people have argued about at length, not assumed

Well, I am yet to see the arguments

Also the case for AI safety doesn't depend on these things being probable, only on them being not extremely unlikely.

It depends on you being able to put number on those probabilities though, otherwise you are in a Pascal wager's scenario, where any event that is not almost certainly ruled out should be taken into account with an amount of seriousness proportional to it's fictive impact.