Course recommendations for Friendliness researchers
post by Louie · 2013-01-09T14:33:50.300Z · LW · GW · Legacy · 112 commentsContents
113 comments
When I first learned about Friendly AI, I assumed it was mostly a programming problem. As it turns out, it's actually mostly a math problem. That's because most of the theory behind self-reference, decision theory, and general AI techniques haven't been formalized and solved yet. Thus, when people ask me what they should study in order to work on Friendliness theory, I say "Go study math and theoretical computer science."
But that's not specific enough. Should aspiring Friendliness researchers study continuous or discrete math? Imperative or functional programming? Topology? Linear algebra? Ring theory?
I do, in fact, have specific recommendations for which subjects Friendliness researchers should study. And so I worked with a few of my best interns at MIRI to provide recommendations below:
- University courses. We carefully hand-picked courses on these subjects from four leading universities — but we aren't omniscient! If you're at one of these schools and can give us feedback on the exact courses we've recommended, please do so.
- Online courses. We also linked to online courses, for the majority of you who aren't able to attend one of the four universities whose course catalogs we dug into. Feedback on these online courses is also welcome; we've only taken a few of them.
- Textbooks. We have read nearly all the textbooks recommended below, along with many of their competitors. If you're a strongly motivated autodidact, you could learn these subjects by diving into the books on your own and doing the exercises.
Have you already taken most of the subjects below? If so, and you're interested in Friendliness research, then you should definitely contact me or our project manager Malo Bourgon (malo@intelligence.org). You might not feel all that special when you're in a top-notch math program surrounded by people who are as smart or smarter than you are, but here's the deal: we rarely get contacted by aspiring Friendliness researchers who are familiar with most of the material below. If you are, then you are special and we want to talk to you.
Not everyone cares about Friendly AI, and not everyone who cares about Friendly AI should be a researcher. But if you do care and you might want to help with Friendliness research one day, we recommend you consume the subjects below. Please contact me or Malo if you need further guidance. Or when you're ready to come work for us.
|
Cognitive Science
|
If you're endeavoring to build a mind, why not start by studying your own? It turns out we know quite a bit: human minds are massively parallel, highly redundant, and although parts of the cortex and neocortex seem remarkably uniform, there are definitely dozens of special purpose modules in there too. Know the basic details of how the only existing general purpose intelligence currently functions. |
|
Heuristics and Biases
|
While cognitive science will tell you all the wonderful things we know about the immense, parallel nature of the brain, there's also the other side of the coin. Evolution designed our brains to be optimized at doing rapid thought operations that work in 100 steps or less. Your brain is going to make stuff up to cover up that its mostly cutting corners. These errors don't feel like errors from the inside, so you'll have to learn how to patch the ones you can and then move on.
|
|
Functional Programing
|
There are two major branches of programming: Functional and Imperative. Unfortunately, most programmers only learn imperative programming languages (like C++ or python). I say unfortunately, because these languages achieve all their power through what programmers call "side effects". The major downside for us is that this means they can't be efficiently machine checked for safety or correctness. The first self-modifying AIs will hopefully be written in functional programming languages, so learn something useful like Haskell or Scheme. |
|
Discrete Math
|
Much like programming, there are two major branches of mathematics as well: Discrete and continuous. It turns out a lot of physics and all of modern computation is actually discrete. And although continuous approximations have occasionally yielded useful results, sometimes you just need to calculate it the discrete way. Unfortunately, most engineers squander the majority of their academic careers studying higher and higher forms of calculus and other continuous mathematics. If you care about AI, study discrete math so you can understand computation and not just electricity.
|
|
Linear Algebra
|
Linear algebra is the foundation of quantum physics and a huge amount of probability theory. It even shows up in analyses of things like neural networks. You can't possibly get by in machine learning (later) without speaking linear algebra. So learn it early in your scholastic career. |
|
Set Theory
|
Like learning how to read in mathematics. But instead of building up letters into words, you'll be building up axioms into theorems. This will introduce you to the program of using axioms to capture intuition, finding problems with the axioms, and fixing them. |
|
Mathematical Logic
|
The mathematical equivalent of building words into sentences. Essential for the mathematics of self-modification. And even though Sherlock Holmes and other popular depictions make it look like magic, it's just lawful formulas all the way down. |
|
Efficient Algorithms and Intractable Problems
|
Like building sentences into paragraphs. Algorithms are the recipes of thought. One of the more amazing things about algorithm design is that it's often possible to tell how long a process will take to solve a problem before you actually run the process to check it. Learning how to design efficient algorithms like this will be a foundational skill for anyone programming an entire AI, since AIs will be built entirely out of collections of algorithms. |
|
Numerical Analysis
|
There are ways to systematically design algorithms that only get things slightly wrong when the input data has tiny errors. And then there's programs written by amateur programmers who don't take this class. Most programmers will skip this course because it's not required. But for us, getting the right answer is very much required. |
|
Computability and Complexity
|
This is where you get to study computing at it's most theoretical. Learn about the Church-Turing thesis, the universal nature and applicability of computation, and how just like AIs, everything else is algorithms... all the way down. |
|
Quantum Computing
|
It turns out that our universe doesn't run on Turing Machines, but on quantum physics. And something called BQP is the class of algorithms that are actually efficiently computable in our universe. Studying the efficiency of algorithms relative to classical computers is useful if you're programming something that only needs to work today. But if you need to know what is efficiently computable in our universe (at the limit) from a theoretical perspective, quantum computing is the only way to understand that. |
|
Parallel Computing
|
There's a good chance that the first true AIs will have at least some algorithms that are inefficient. So they'll need as much processing power as we can throw at them. And there's every reason to believe that they'll be run on parallel architectures. There are a ton of issues that come up when you switch from assuming sequential instruction ordering to parallel processing. There's threading, deadlocks, message passing, etc. The good part about this course is that most of the problems are pinned down and solved: You're just learning the practice of something that you'll need to use as a tool, but won't need to extend much (if any). |
|
Automated Program Verification
|
Remember how I told you to learn functional programming way back at the beginning? Now that you wrote your code in functional style, you'll be able to do automated and interactive theorem proving on it to help verify that your code matches your specs. Errors don't make programs better and all large programs that aren't formally verified are reliably *full* of errors. Experts who have thought about the problem for more than 5 minutes agree that incorrectly designed AI could cause disasters, so world-class caution is advisable. |
|
Combinatorics and Discrete Probability
|
Life is uncertain and AIs will handle that uncertainty using probabilities. Also, probability is the foundation of the modern concept of rationality and the modern field of machine learning. Probability theory has the same foundational status in AI that logic has in mathematics. Everything else is built on top of probability. |
|
Bayesian Modeling and Inference
|
Now that you've learned how to calculate probabilities, how do you combine and compare all the probabilistic data you have? Like many choices before, there is a dominant paradigm (frequentism) and a minority paradigm (Bayesianism). If you learn the wrong method here, you're deviating from a knowably correct framework for integrating degrees of belief about new information and embracing a cadre of special purpose, ad-hoc statistical solutions that often break silently and without warning. Also, quite embarrassingly, frequentism's ability to get things right is bounded by how well it later turns out to have agreed with Bayesian methods anyway. Why not just do the correct thing from the beginning and not have your lunch eaten by Bayesians every time you and them disagree? |
|
Probability Theory
|
No more applied probability: Here be theory! Deep theories of probabilities are something you're going to have to extend to help build up the field of AI one day. So you actually have to know why all the things you're doing are working inside out. |
|
Machine Learning
|
Now that you chose the right branch of math, the right kind of statistics, and the right programming paradigm, you're prepared to study machine learning (aka statistical learning theory). There are lots of algorithms that leverage probabilistic inference. Here you'll start learning techniques like clustering, mixture models, and other things that cache out as precise, technical definitions of concepts that normally have rather confused or confusing English definitions. |
|
Artificial Intelligence
|
We made it! We're finally doing some AI work! Doing logical inference, heuristic development, and other techniques will leverage all the stuff you just learned in machine learning. While modern, mainstream AI has many useful techniques to offer you, the authors will tell you outright that, "the princess is in another castle". Or rather, there isn't a princess of general AI algorithms anywhere -- not yet. We're gonna have to go back to mathematics and build our own methods ourselves. |
|
Incompleteness and Undecidability
|
Probably the most celebrated results is mathematics are the negative results by Kurt Goedel: No finite set of axioms can allow all arithmetic statements to be decided as either true or false... and no set of self-referential axioms can even "believe" in its own consistency. Well, that's a darn shame, because recursively self-improving AI is going to need to side-step these theorems. Eventually, someone will unlock the key to over-coming this difficulty with self-reference, and if you want to help us do it, this course is part of the training ground. |
|
Metamathematics
|
Working within a framework of mathematics is great. Working above mathematics -- on mathematics -- with mathematics, is what this course is about. This would seem to be the most obvious first step to overcoming incompleteness somehow. Problem is, it's definitely not the whole answer. But it would be surprising if there were no clues here at all. |
|
Model Theory
|
One day, when someone does side-step self-reference problems enough to program a recursively self-improving AI, the guy sitting next to her who glances at the solution will go "Gosh, that's a nice bit of Model Theory you got there!"
|
|
Category Theory
|
Category theory is the precise way that you check if structures in one branch of math represent the same structures somewhere else. It's a remarkable field of meta-mathematics that nearly no one knows... and it could hold the keys to importing useful tools to help solve dilemmas in self-reference, truth, and consistency. |
Outside recommendations |
|
|
|
|
Harry Potter and the Methods of Rationality
|
Highly recommended book of light, enjoyable reading that predictably inspires people to realize FAI is an important problem AND that they should probably do something about that.
|
|
|
Global Catastrophic Risks
|
A good primer on xrisks and why they might matter. SPOILER ALERT: They matter.
|
|
|
The Sequences
|
Rationality: the indispensable art of non-self-destruction! There are manifold ways you can fail at life... especially since your brain is made out of broken, undocumented spaghetti code. You should learn more about this ASAP. That goes double if you want to build AIs.
|
|
|
Good and Real
|
A surprisingly thoughtful book on decision theory and other paradoxes in physics and math that can be dissolved. Reading this book is 100% better than continuing to go through your life with a hazy understanding of how important things like free will, choice, and meaning actually work.
|
|
|
MIRI Research Papers
|
MIRI has already published 30+ research papers that can help orient future Friendliness researchers. The work is pretty fascinating and readily accessible for people interested in the subject. For example: How do different proposals for value aggregation and extrapolation work out? What are the likely outcomes of different intelligence explosion scenarios? Which ethical theories are fit for use by an FAI? What improvements can be made to modern decision theories to stop them from diverging from winning strategies? When will AI arrive? Do AIs deserve moral consideration? Even though most of your work will be more technical than this, you can still gain a lot of shared background knowledge and more clearly see where the broad problem space is located.
|
|
|
Universal Artificial Intelligence
|
A useful book on "optimal" AI that gives a reasonable formalism for studying how the most powerful classes of AIs would behave under conservative safety design scenarios (i.e., lots and lots of reasoning ability).
|
Free Online
Do also look into: Formal Epistemology, Game Theory, Decision Theory, and Deep Learning.
112 comments
Comments sorted by top scores.
comment by lukeprog · 2013-01-09T18:56:02.277Z · LW(p) · GW(p)
Allow me to add another clarification.
Earlier, I explained to someone that most of the problems in Eliezer's forthcoming Open Problems in Friendly AI sequence are still at the stage of being philosophy problems. Why, then, do Louie and I talk about FAI being "mostly a math problem, not a programming problem"?
The point we're trying to make is that Friendly AI, as we understand it, isn't chiefly about hiring programmers and writing code. Instead, it's mostly about formalizing problems (in reflective reasoning, decision theory, etc.) into math problems, and then solving those math problems. The formalization step itself will likely require the invention of new math — not so much clever programming tricks (though those may be required at a later stage).
Most of the "open FAI problems" are still at the stage of philosophy because, as Louie says, "most of the theory behind self-reference, decision theory, and general AI techniques haven't been formalized.... yet." But we think those philosophical problems will be formalized into math problems, sometimes requiring new math.
So we're not (at this stage) looking to hire great programmers. We're looking to hire great mathematicians; even though most of the problems are still at the "philosophy" stage of formalization.
The specific practice of turning philosophy into math is a key feature of the field called formal epistemology, but Louie and I couldn't find any "standard" classes or textbooks on formal epistemology that we would strongly recommend, so we had to leave it off the list for now.
Examples of turning philosophy into math include VNM's formalization of "rationality" into axiomatic utility theory, and Shannon's formalization of information concepts with his information theory.
Replies from: Vladimir_Nesov↑ comment by Vladimir_Nesov · 2013-01-09T19:56:12.548Z · LW(p) · GW(p)
will likely require the invention of new fundamental math
I wish you'd stop saying that (without justification or clarification). Modern math seems quite powerful enough to express most problems, so the words "new" and "fundamental" sound somewhat suspicious. Is this "new fundamental math" something like the invention of category theory? Probably not. Clarifying the topic of Friendly AI would almost certainly involve nontrivial mathematical developments, but in the current state of utter confusion it seems premature to characterize these developments as "fundamental".
We don't know how it turns out, what we know is that only a mathematical theory would furnish an accurate enough understanding of the topic, and so it seems to be a good heuristic to have mathematicians work on the problem, because non-mathematicians probably won't be able to develop a mathematical theory. In addition, we have some idea about the areas where additional training might be helpful, such as logic, type theory, formal languages, probability and computability.
Replies from: lukeprog, JoshuaFox↑ comment by lukeprog · 2013-01-09T23:28:18.364Z · LW(p) · GW(p)
You're right, the word "fundamental" might suggest the wrong kinds of things. I'm not at all confident that Friendly AI will require the invention of something like category theory. So, I've removed the word "fundamental" from the above comment.
comment by Qiaochu_Yuan · 2013-01-09T20:38:31.880Z · LW(p) · GW(p)
Should I contact you if I'm familiar with some of this material (mostly the purer mathematics, leaning away from computer science and towards category theory, plus MoR and the Sequences) and willing to learn the rest? Does SI offer summer internships?
Replies from: Louie↑ comment by Louie · 2013-01-10T06:14:42.426Z · LW(p) · GW(p)
Yep, SI has summer internships. You're already in Berkeley, right?
Drop me an email with the dates you're available and what you'd want out of an internship. My email and Malo's are both on our internship page:
http://singularity.org/interns/
Look forward to hearing from you.
comment by badger · 2013-01-09T22:36:36.802Z · LW(p) · GW(p)
The list doesn't include anything in the way of game theory, social choice, or mechanism design, which is going to be crucial for an AI that interacts with other agents or tries to aggregate preferences.
Relevant book recommendations (all available at links as pdfs):
- Essentials of Game Theory by Leyton-Brown and Shoham. (Gated, might be accessible through a university connection, otherwise easily searchable)
- A Course in Game Theory by Osbourne and Rubinstein (requires free registration)
- Multi-agent Systems: Algorithmic, Game-Theoretic, and Logical Foundations by Shoham and Leyton-Brown.
- Algorithmic Game Theory edited by Nisan, Roughgarden, Tardis, and Varizani.
↑ comment by DaFranker · 2013-01-10T17:36:14.819Z · LW(p) · GW(p)
On this note, perhaps very time-sensitively-relevant, Coursera's online course on Game Theory has just started. It's a 7-week course that claims to cover all the strong basics. It's quite interactive, too, and seems more optimized for efficient learning than traditional university courses.
If anyone's interested, you can still jump right in (probably until the 13th or 14th-ish), the Week 1 material is probably easy to cover in two hours (or maybe three depending on how carefully and rigorously you want to do the problem set) for most LWers.
The course description and syllabus should tell you everything else I haven't said (see second link above).
Replies from: Benito↑ comment by Ben Pace (Benito) · 2013-01-13T17:01:21.286Z · LW(p) · GW(p)
I've joined - thanks.
↑ comment by Kaj_Sotala · 2013-01-10T15:46:27.766Z · LW(p) · GW(p)
Agree with this. I haven't read those books, but I know that Public Choice III also contains 100> pages of thorough discussion about how the choice of a voting system influences the way that preferences get aggregated under those systems. Something like that seems like a must-read for Friendliness researchers.
↑ comment by Ben Pace (Benito) · 2013-01-13T16:39:08.309Z · LW(p) · GW(p)
The first link isn't free. It costs $5.
Replies from: badgercomment by jimrandomh · 2013-01-09T15:48:56.230Z · LW(p) · GW(p)
There are two major branches of programming: Functional and Imperative. Unfortunately, most programmers only learn imperative programming languages (like C++ or python). I say unfortunately, because these languages achieve all their power through what programmers call "side effects". The major downside for us is that this means they can't be efficiently machine checked for safety or correctness. The first self-modifying AIs will hopefully be written in functional programming languages, so learn something useful like Haskell or Scheme.
Please be careful about exposing programmers to ideology; it frequently turns into politics kills their minds. This piece in particular is a well-known mindkiller, and I have personally witnessed great minds acting very stupid because of it. The functional/imperative distinction is not a real one, and even if it were, it's less important to provability than languages' complexity, the quality of their type systems and the amount of stupid lurking in their dark corners.
Replies from: Louie, earthwormchuck163, siodine↑ comment by Louie · 2013-01-09T16:37:56.668Z · LW(p) · GW(p)
The functional/imperative distinction is not a real one
How is the distinction between functional and imperative programming languages "not a real one"? I suppose you mean that there's a continuum of language designs between purely functional and purely imperative. And I've seen people argue that you can program functionally in python or emulate imperative programming in Haskell. Sure. That's all true. It doesn't change the fact that functional-style programming is manifestly more machine checkable in the average (and best) case.
it's less important to provability than languages' complexity, the quality of their type systems and the amount of stupid lurking in their dark corners.
Agreed. The most poorly programmed functional programs will be harder to machine check than the mostly skillfully designed imperative programs. But I think for typical programming scenarios or best case scenarios, functional-style programming makes it hands-down more natural to write the correct kind of structures that can be reliably machine checked and imperative programming languages just don't.
The entry level functional programming course is going to focus on all the right things: type theory, model theory, deep structure. The first imperative programming course at most universities is going to teach you how to leverage side-effects, leverage side-effects more, and generally design your code in a way that makes it less tractable for verification and validation later on.
Replies from: fiddlemath, Kawoomba, jimrandomh, OrphanWilde↑ comment by fiddlemath · 2013-01-09T19:13:45.154Z · LW(p) · GW(p)
How is the distinction between functional and imperative programming languages "not a real one"?
"Not a real one" is sort of glib. Still, I think Jim's point stands.
The two words "functional" and "imperative" do mean different things. The problem is that, if you want to give a clean definition of either, you wind up talking about the "cultures" and "mindsets" of the programmers that use and design them, rather than actual features of the language. Which starts making sense, really, when you note "functional vs. imperative" is a perennial holy war, and that these terms have become the labels for sides in that war, rather than precise technical positions.
I mean, I am somewhat partisan in that war, and rather agree that, e.g., we should point new programmers to Scheme rather than Python or Java. But presenting "functional" vs. "imperative" as the major division in thinking about programming languages is epistemically dirty, when there's so many distinctions between languages that matter just as much, and describe things more precisely.
(Jim: fair rephrasing?)
Replies from: loup-vaillant↑ comment by loup-vaillant · 2013-01-14T12:01:29.262Z · LW(p) · GW(p)
I've wrote about the difference between imperative programs and functional ones (here, trying to explain why an imperative programmer might find functional programming difficult).
The main differences (do vs be, reversed order of reading, and heavy use of first class functions), make a quite sharp divide. Way sharper than other divides, such as "Object Oriented" versus the rest (though Alan Kay's original vision is probably more distinguishable than the current C++/Java vision).
Now when talking about programming an FAI, the most probable course of action will be to translate the math of FAI into a program. One thing I noticed about math formalism outside the specific field of computer programming, is that most formulations are either stateless, or explicit about state. When at some point we say X = some long expression, we know it won't change until the end of the assignment. The math I know tend to be functional by default.
↑ comment by Kawoomba · 2013-01-09T18:01:07.353Z · LW(p) · GW(p)
Pretty much by definition all (Turing-complete) programming languages can in principle be transformed into each other, it's not even very hard: just take the ASM code and build some rudimentary reverse compiler that creates some strange looking code in your desired goal language.
For practical purposes, machine checkability is easier for functional languages, but it's a difference in degree, not one in kind. (Corrections welcome!)
Replies from: JGWeissman↑ comment by JGWeissman · 2013-01-09T18:35:00.517Z · LW(p) · GW(p)
that creates some strange looking code in your desired goal language.
The code wouldn't just look strange, it would likely be more complex than code written directly in the language using its standard style at a high level to solve the same problem. The reversed compiled code would be harder to analyze, and harder to know what would be useful to prove about it, than code written directly in the target language with the reasoning behind why the programmer expects it to work serving as a guide to construct the proof.
Replies from: Kawoomba↑ comment by Kawoomba · 2013-01-09T18:46:55.090Z · LW(p) · GW(p)
Yes, however from what I recall about such proving methods, they were quite formulaic and about an equal amount of pain for each line (I should dig out some references some time), so the difference may not be as pronounced. Be that as it may, I agree that there is a difference in degree, which does matter for practical purposes.
Replies from: JGWeissman↑ comment by JGWeissman · 2013-01-09T19:02:03.247Z · LW(p) · GW(p)
I expect it is more than a difference in degree. When the programmer can tell the automated theorem prover that the return value of a certain function always satisfies some condition that other functions rely on, the theorem prover can try to prove that condition holds and use that fact for other proofs without being distracted by other possible conditions that are not relevant. This makes the search space much smaller.
The point is that the programmer having some understanding of the proof checker, and designing the code to be not just correct, but provably correct in a reasonable amount of time makes a big difference from the general problem of proving things about arbitrary code where you run into the halting problem.
Replies from: Kawoomba↑ comment by Kawoomba · 2013-01-09T19:14:52.237Z · LW(p) · GW(p)
I agree with both your example ("making the search space much smaller") and your explanation ("makes a big difference"), we just differ on whether thus changing the resource requirements constitutes a change in kind, or one in degree.
Also, I took your "code would likely be more complex" as "be less human-understandable", as opposed to its actual Kolmogorov complexity, which afaik would be identical.
↑ comment by jimrandomh · 2013-01-09T19:02:06.220Z · LW(p) · GW(p)
How is the distinction between functional and imperative programming languages "not a real one"? I suppose you mean that there's a continuum of language designs between purely functional and purely imperative.
Not exactly. There is a functional/imperative distinction, but I don't think it's located in the language; it's more a matter of style and dialect. The majority of the difference between functional style and imperative style is in how you deal with collections. In functional style, you use map, filter, fold and the like, mostly treat them as immutable and create new ones, and use a lot of lambda expressions to support this. In imperative style, you emulate the collection operators using control flow constructs. Most major programming languages today support both syles, and the two styles act as dialects. (The main exceptions are non-garbage-collected languages, which can't use the functional style because it interacts badly with object ownership, and Java, which lacks lambda expressions as a symptom of much bigger problems).
These styles are less different than they appear. A lot of use of mutation is illusory; it matches to a palette of a dozen or so recognizable patterns which could just as easily be written in functional form. In fact, ReSharper can automatically translate a lot of C# between these two styles, in a provably-correct fashion; and if you want to prove complicated things about programs, the infrastructure to make that sort of thing easy is part of the price of admission.
But there's a catch. Programmers who start in a functional language and avoid the imperative style don't learn the palette of limited mutations, and to them, imperative-style code is more inscrutable than to programmers who learned both styles. And while I'm much less certain of this, I think there may be an order-dependence, where programmers who learn imperative style first and then functional do better than those who learn them in reverse order. And I don't think it's possible to get to the higher skill levels without a thorough grasp of both.
Replies from: Louie, siodine, IlyaShpitser↑ comment by Louie · 2013-01-09T19:23:51.793Z · LW(p) · GW(p)
Well, I figure I don't really want to recommend a ton of programming courses anyway. I'm already recommending what I presume is more than a bachelor's degree worth of course when pre-reqs and outside requirements at these universities are taken into account.
So if someone takes one course, they can learn so much more that helps them later in this curriculum from the applied, function programming course than its imperative counterpart. And the normal number of functional programming courses that people take in a traditional math or CS program is 0. So I have to make a positive recommendation here to correct this. I couldn't make people avoid imperative programming courses anyway, even if I tried. So people will oversample them (and follow your implied recommendation) relative to my core recommendations anyway.
So in practice, most people will follow your advice, by following mine and actually studying some functional programming instead of none and then study a ton of imperative programming no matter what anyone says.
↑ comment by siodine · 2013-01-10T21:42:04.829Z · LW(p) · GW(p)
I don't think you understand functional programming. What background are you coming from?
Replies from: loup-vaillant↑ comment by loup-vaillant · 2013-01-14T12:11:22.370Z · LW(p) · GW(p)
Where did that come from? I didn't spot anything wrong in his comment, and I'm pretty knowledgeable myself (I'm no authority, but I believe my grasp of FP is quite comprehensive).
(Edit: Retracted: I now see it did came from somewhere: several comments, including the top one)
Replies from: siodine↑ comment by siodine · 2013-01-15T03:59:27.331Z · LW(p) · GW(p)
Functional programming isn't an idiomatic approach to container manipulation, it's a paradigm that avoids mutable state and data. Write a GUI in Haskell using pure functions to see how different the functional approach is and what it is at its core. Or just compare a typical textbook on imperative algorithms with one on functional algorithms. Container manipulation with functions is just an idiom.
And sure you can write functional code in C++, for example (which by the way has map, filter, fold, and so on), but you can also write OO code in C. But few people do, and for a reason: the language makes it near impossible or, at the very least, undesirable for humans. That's close enough for the distinction being located in the language.
Replies from: loup-vaillant↑ comment by loup-vaillant · 2013-01-15T11:35:49.077Z · LW(p) · GW(p)
Functional programming isn't an idiomatic approach to container manipulation, it's a paradigm that avoids mutable state and data.
I'm not sure you and jimrandomh actually disagree on that point. I mean, avoiding mutable state is bound to change your approach to container manipulation. You described the root cause, he described the surface differences.
Also,
And sure you can write functional code in C++, for example (which by the way has map, filter, fold, and so on), but you can also write OO code in C. But few people do, and for a reason: […]
jimrandomh knows that:
(The main exceptions are non-garbage-collected languages, which can't use the functional style because it interacts badly with object ownership, and Java, which lacks lambda expressions as a symptom of much bigger problems).
I personally tried functional style in C++, and did feel the pain ( is unusable before C++11). There are ways however to limit the damage. And languages such as Python and Javascript do not discourage the functional style so much. Their libraries do.
Now sure, languages do favour a style over the other, like Ocaml vs Lua. It does let us classify them meaningfully. On the other hand, it is quite easy to go against the default. I'm pretty sure both you and jimrandomh agree with that as well.
From the look of it, you just talked past each other, while there was no real disagreement to begin with…
Replies from: siodine↑ comment by siodine · 2013-01-15T16:03:00.381Z · LW(p) · GW(p)
In light of the following comment by jim, I think we do disagree:
Please be careful about exposing programmers to ideology; it frequently turns into politics kills their minds. This piece in particular is a well-known mindkiller, and I have personally witnessed great minds acting very stupid because of it. The functional/imperative distinction is not a real one, and even if it were, it's less important to provability than languages' complexity, the quality of their type systems and the amount of stupid lurking in their dark corners.
And while I would normally interpret jim's nearest comment in this thread charitably (i.e., mostly in agreement with me), it's more reasonable to interpret in light of quoted comment.
I think he probably doesn't or didn't understand the functional paradigm. If he did, I think he would know about its usefulness in concurrent or parallel programming, and consequently know that it is not just a mind-killing ideology like US political parties, but a paradigm with real advantages and real disadvantages over other paradigms. I don't think he would have written his first comment if he really knew that. I think he's probably confusing the functional idiomatic approach/style/dialect/whatever with the functional paradigm. I mean he says "The majority of the difference between functional style and imperative style is in how you deal with collections." And remember this thread was created in reference to a comment about a textbook on functional programming (not functional "style" -- maybe he's backpedaling or charitably he means fp).
(also c++ is a non-garbage-collected language. And more importantly I don't mean to shit on jim. I'm more worried about how many people thought it was a comment worth being at the top of the comment section in a thread about course recommendations for FAI researchers. I would have been fine ignoring it otherwise)
Replies from: loup-vaillant↑ comment by loup-vaillant · 2013-01-15T21:13:26.773Z · LW(p) · GW(p)
Let's see:
The functional/imperative distinction is not a real one
Oops.
I think he's probably confusing the functional idiomatic approach/style/dialect/whatever with the functional paradigm. I mean he says "The majority of the difference between functional style and imperative style is in how you deal with collections."
If I got it, you are saying that he perceived the surface features (dealing with collections), but not their deeper cause (avoid mutable state). Sounds about right. Re-oops, I guess.
I don't mean to shit on jim.
Now it occurred to me that I may have forced you to. Sorry.
↑ comment by IlyaShpitser · 2013-01-10T19:03:06.591Z · LW(p) · GW(p)
There is a functional/imperative distinction, but I don't think it's located in the language
It can be -- Haskell is purely functional, I would say early FORTRAN is purely imperative.
and Java, which lacks lambda expressions as a symptom of much bigger problems
Java has true closures (anonymous inner classes). But you don't want to use them. Or Java for that matter :).
I used to like functional programming much more when younger, before I realized the entire point of functional programming is to hide the causality of the program from the human, and the human needs the causality of the program to reason about the program and to debug. Sure, functional programs are easier for computers to handle, but human time is more precious.
Debugging non-trivial functional program projects requires the reinvention of imperative (causal) style (if you are very smart, this happens on the fly in your head, and the program still looks functional).
That this article wrote about the functional/imperative distinction in the way it did is a reminder that lesswrong is an attractor for holy war prone young people :). Keep religion out of science please!
Replies from: siodine↑ comment by siodine · 2013-01-10T22:03:25.513Z · LW(p) · GW(p)
the entire point of functional programming is to hide the causality of the program from the human
Why? I would say it's the opposite (and really the causality being clear and obvious is just a corollary of referential transparency). The difficulty of reasoning about concurrent/parallel code in an imperative language, for example, is one of the largest selling points of functional programming languages like erlang and haskell.
Replies from: IlyaShpitser↑ comment by IlyaShpitser · 2013-01-11T08:59:48.208Z · LW(p) · GW(p)
The causality in a functional language is far from obvious. Consider Haskell, a language that is both purely functional and lazy, and is considered somewhat of a beautiful poster child of the functional approach. Say you write a program and it has a bug -- it's not doing what it's supposed to. How would you debug it? Some alternatives:
(a) Use a debugger to step through a program until it does something it's not supposed to (this entails dealing with a causal order of evaluation of statements -- something Haskell as a lazy and functional language is explicitly hiding from you until you start a debugger).
(b) Use good ol' print statements. These will appear in a very strange order because of lazy evaluation. Again, Haskell hides the true order -- the true order has nothing to do with the way the code appears on the page. This makes it difficult to build a causal model of what's going on in your program. A causal model is what you need if you want to use print statements to debug.
(c) Intervene in a program by changing some intermediate runtime value to see what would happen to the output. As a functional language, Haskell does not allow you to change state (ignoring monads which are a very complicated beast, and at any rate would not support a straightforward value change while debugging anyways).
My claim is that causality is so central to how human beings think about complex computer programs that it is not possible to write and debug large programs written in functional style without either building a causal model of your program (something most functional language will fight with you about to the extent that they are functional), or mostly sticking to an imperative "causal" style, and only use simple functional idioms that you know work and that do not require further thinking (like map and reduce, and simple closure use). Note that even Haskell, a language committed to "wearing the hair shirt" (http://research.microsoft.com/en-us/um/people/simonpj/papers/haskell-retrospective/haskellretrospective.pdf) of functional programming, has given concessions to the imperative/causal writing style by providing the "do" shorthand.
Personally, I love functional idioms, and I think functional code with heavy recursion use is often quite beautiful. But I don't think the functional approach is well suited to writing complex code precisely because it is violating a principal rule of computer science -- make things easy for the human at the expense of the machine.
Replies from: siodine↑ comment by siodine · 2013-01-11T16:31:13.237Z · LW(p) · GW(p)
Laziness can muddy the waters, but it's also optional in functional programming. People using haskell in a practical setting usually avoid it and are coming up with new language extensions to make strict evaluation the default (like in records for example).
What you're really saying is the causal link between assembly and the language is less obvious, which is certainly true as it is a very high level language. However, if we're talking about the causality of the language itself, then functional languages enforce a more transparent causal structure of the code itself.
You can be certain that a function that isn't tainted by IO in haskell, for example, isn't going to involve dozens of different causal structures. An imperative function like AnimalFactory.create("dog") could involve dozens of different dependencies (e.g. through singletons or dependency injection) making the dependency graph (and causal structure) obfuscated. This lack of transparent guarantees about state and dependencies in imperative languages makes concurrent/parallelprogramming (and even plain code) very difficult to reason about and test.
Moreover, the concessions that haskell has given way to are probably temporary. Haskell is a research language and functional solutions to problems like IO and event driven programs have been put forward but are not yet widely accepted. And even ignoring these solutions, you still have a basic paradigm where you have top level imperative style code with everything else being functional.
And while it can be more difficult to debug functional programs, they're easier to test, and they're less prone to runtime bugs. And really, the debugging problem is one of laziness and difficult to use debuggers. Debugging F# with visual studio's debugger isn't that difficult.
(Note: that when I talk about functional programming, I'm talking about a paradigm that avoids mutable state and data rather than idiomatic approaches to container manipulation)
↑ comment by OrphanWilde · 2013-01-09T18:00:51.375Z · LW(p) · GW(p)
Functional-style programming doesn't make it any more natural, it just forbids you from doing things any other way. I spend most of my time when dealing with functional-style programming (primarily in XSLT) trying to figure out ways around the constraints imposed by the language rather than actually solving the problem I'm working on.
In XSLT I once copied a chunk of code 8 times, replacing its recursive function calls with its own code, because it was blowing up the stack; and it's not like I could use mutable variables and skip the recursion, it was literally the only implementation possible. And it had to call itself in multiple places of its own code; it couldn't be phrased in a tail-recursion friendly fashion. Meaning that for that code, no functional language could have resolved the stack explosion issue. -That's- functional programming to me; dysfunctional.
[ETA: Apparently there is a pattern which would overcome the tail stack issue, and compilers exist which can take advantage of it, so my statement that "No functional language could have resolved the stack explosion issue" was false.]
Replies from: JGWeissman, gwern↑ comment by JGWeissman · 2013-01-09T18:23:05.786Z · LW(p) · GW(p)
no functional language could have resolved the stack explosion issue
A functional language could compile to code that uses references to continuations (functions to call, with parameters) instead of the physical stack. See continuation passing style. (The language wouldn't need to use continuation passing style itself, the conversion would straightforward for the compiler to do.)
Replies from: OrphanWilde↑ comment by OrphanWilde · 2013-01-09T18:56:47.742Z · LW(p) · GW(p)
That, to me, falls under "Trying to figure out ways around the constraints imposed by the language."
I suspect it would have made for even messier code, as well, provided it would have even worked for that particular problem. I was parsing a deliberately and ingeniously obfuscated document, and I was updating particular pieces of information based on information obtained in recursive references which, needless to say, were obfuscated, including false flag references which would get me incorrect information if followed; the falsity couldn't be determined locally, either, and required complex conditional comparisons to information contained in parent references. Some of the references were cyclical, to boot.
Replies from: JGWeissman↑ comment by JGWeissman · 2013-01-09T19:11:01.922Z · LW(p) · GW(p)
That, to me, falls under "Trying to figure out ways around the constraints imposed by the language."
Functional languages do not have inherent stack limits. A stack limit is imposed by a particular implementation of the language, and what I described is how a different implementation of the language could have a much larger stack limit, (constrained by total memory rather than memory initially allocated to the physical stack), with no difference in the source code because the compiler can make the transformation to continuation passing style for you.
The point is that this problem you had with XSLT does not generalize to all possible functional languages the way you think it does.
Replies from: OrphanWilde↑ comment by OrphanWilde · 2013-01-09T19:19:50.011Z · LW(p) · GW(p)
Hm. Granted.
In fairness, I think it does generalize to most implementations.
↑ comment by gwern · 2013-01-09T18:11:53.609Z · LW(p) · GW(p)
In XSLT...That's functional programming to me; dysfunctional.
I think I see the problem here.
Perlis comes to mind:
Replies from: OrphanWildeBeware of the Turing tar-pit in which everything is possible but nothing of interest is easy.
↑ comment by OrphanWilde · 2013-01-09T18:21:34.127Z · LW(p) · GW(p)
The parts of XSLT that cause me problems are the functional programming parts; not every problem is well-suited to functional programming, especially when it comes to, as in that case, parsing complex interconnected documents to find information (ETA: Particularly when that information isn't guaranteed to be in any one particular place, and you have to conditionally check multiple locations, each of which itself may be referenced in one of multiple locations that have to be conditionally checked).
Functional programming takes a mathematical approach to problem-solving. There are some problems it is exceedingly elegant at solving. The issue is that it isn't any more elegant than a well-written alternative in another language, and it makes that elegance mandatory, which causes severe problems when you're dealing with an inelegant problem.
I'm not hired to solve elegant problems. I'm hired to solve the problems that companies have spent 20 million dollars to fail to solve.
Replies from: gwern↑ comment by gwern · 2013-01-09T18:32:24.151Z · LW(p) · GW(p)
The parts of XSLT that cause me problems are the functional programming parts; not every problem is well-suited to functional programming, especially when it comes to, as in that case, parsing complex interconnected documents to find information
So your example of how 'functional programming fails' is to use a vague personal anecdote about possibly the worst 'functional' language in the world, many versions of which don't even have higher-order functions which is a basic key functional feature dating back literally to the 1960s, and of which people have published research papers just to prove it is Turing-complete?
Do you understand why no one is going to find your claim that "functional programming sucks because I once wrote a bad program in XSLT" even remotely convincing? Even if you do boast about yourself that
Replies from: OrphanWildeI'm not hired to solve elegant problems. I'm hired to solve the problems that companies have spent 20 million dollars to fail to solve.
↑ comment by OrphanWilde · 2013-01-09T19:08:56.100Z · LW(p) · GW(p)
The problem was done under an NDA, so I can't get too specific. The shortest explanation, however, is "Parsing obfuscated document." It was a problem which was intentionally created by somebody to be as difficult as possible to solve in a programming language.
Functional programming doesn't suck because of that problem; it would have been difficult in ANY language. Functional programming sucks because it deliberately prevents you from doing things. As a rule, any time anybody says "You shouldn't be doing that," it is because they lack imagination as to why you would need to do that. Functional programming is designed around the principle that "You shouldn't be doing that."
↑ comment by earthwormchuck163 · 2013-01-10T08:18:19.843Z · LW(p) · GW(p)
I have personally witnessed great minds acting very stupid because of it.
I'm curious. Can you give a specific example?
↑ comment by siodine · 2013-01-10T22:15:41.549Z · LW(p) · GW(p)
This is a really stupid comment for how many upvotes it's getting. I don't mind the criticism of functional programming, but I do mind that this person is essentially saying "this is bad because I say so" and gets to the top of the comments.
Replies from: jimrandomh, OrphanWilde↑ comment by jimrandomh · 2013-01-11T01:57:04.488Z · LW(p) · GW(p)
What criticism of functional programming?
The grandparent comment (and my other comment in the thread) said nothing whatsoever about whether functional programming is good or bad. I only said that it was bad to present it as ideology - as opposed to, say, teaching in SML and leaving the whole functional/imperative thing unremarked.
↑ comment by OrphanWilde · 2013-01-10T22:19:42.904Z · LW(p) · GW(p)
Are you confusing the content of the quote with the content of the comment?
Replies from: siodine↑ comment by siodine · 2013-01-10T22:34:13.766Z · LW(p) · GW(p)
No. Jimrandomh just says functional programming, imperative programming, ect are "ideologies" (importing the negative connotation). Just says it kills minds. Just says it's a well-known mindkiller. Just says it's not a real distinction. Just puts it in a dichotomy between being more or less important than "languages' complexity, the quality of their type systems and the amount of stupid lurking in their dark corners." What Louie says is more reasonable given that it's a fairly standard position within academia and because it's a small part of a larger post. (I'd rather Louie had sourced what he said, though.)
Replies from: OrphanWilde↑ comment by OrphanWilde · 2013-01-10T22:50:25.810Z · LW(p) · GW(p)
No, what he's saying is that Louie's -comments- about imperative programming amount to ideology.
Being a standard ideology doesn't make it less of an ideology.
Replies from: siodine↑ comment by siodine · 2013-01-10T22:56:31.594Z · LW(p) · GW(p)
There are two major branches of programming: Functional and Imperative. Unfortunately, most programmers only learn imperative programming languages (like C++ or python). I say unfortunately, because these languages achieve all their power through what programmers call "side effects". The major downside for us is that this means they can't be efficiently machine checked for safety or correctness. The first self-modifying AIs will hopefully be written in functional programming languages, so learn something useful like Haskell or Scheme.
Comes from the post not the comments (maybe you mean it's louie's comment about the functional programming recommendation in the main post).
Being a standard ideology doesn't make it less of an ideology.
He's just saying it's an ideology and importing the negative connotation (of it being bad), rather than saying why or how it's an ideology and why that's bad. Now I think you're being really stupid. I don't like repeating myself.
Replies from: OrphanWilde↑ comment by OrphanWilde · 2013-01-10T23:11:27.608Z · LW(p) · GW(p)
Yes, it's his comment about imperative languages, in the main post.
He's stating that it will invoke arguments and distract from the thrust of the point - and guess what, he's right. Look at what you're doing, right here. You're not merely involved in the holy war, you're effectively arguing, here, that the holy war is more important than the point Louie was -actually- trying to make in his post, which he distracted some users from with an entirely unnecessary-to-his-post attack on imperative programming languages.
Replies from: siodine↑ comment by siodine · 2013-01-10T23:22:17.553Z · LW(p) · GW(p)
He's stating that it will invoke arguments and distract from the thrust of the point - and guess what, he's right. Look at what you're doing, right here.
No. "It" didn't invoke this thread, jimrandomh's fatuous comment combined with it being at the top of the comment section did (I don't care that it was a criticism of functional programming). You keep failing to understand the situation and what I'm saying, and because of this I've concluded that you're a waste of my time and so I won't be responding to you further.
Replies from: Nonecomment by earthwormchuck163 · 2013-01-10T00:19:51.043Z · LW(p) · GW(p)
I am well versed in most of this math, and a fair portion of the CS (mostly the more theoretical parts, not so much the applied bits). Should I contact you now, or should I study the rest of that stuff first?
In any case, this post has caused me to update significantly in the direction of "I should go into FAI research". Thanks.
Replies from: lukeprogcomment by TsviBT · 2013-01-10T00:19:55.994Z · LW(p) · GW(p)
Main is that-a-way ->
Replies from: None↑ comment by [deleted] · 2013-01-10T00:33:24.828Z · LW(p) · GW(p)
<- that way actually.
Replies from: Richard_Kennaway↑ comment by Richard_Kennaway · 2013-08-04T19:35:46.412Z · LW(p) · GW(p)
<- but some people go both ways ->
comment by Wei Dai (Wei_Dai) · 2013-01-18T23:54:25.357Z · LW(p) · GW(p)
I was a CS major but I haven't taken most of the CS courses listed here, including Numerical Analysis, Parallel Computing, Quantum Computing, Machine Learning, Artificial Intelligence, Functional Programming, and Automated Program Verification. I think it's probably not necessary to have more than a cursory understanding of most these topics at the current stage of Friendliness research.
I would suggest swapping one or more of these courses out for a course in cryptography. Cryptography, besides possibly having direct applications, is good for giving a sense of the limits of human intelligence and mathematical reasoning. You can see how far "provable security" (which seems like the closest analogue we have to "provable Friendliness") as well as "heuristical security" got after thousands of mathematician-years worth of effort.
comment by gwern · 2013-01-09T17:42:33.397Z · LW(p) · GW(p)
Why are you suggesting Discrete Math and Its Applications when its Amazon reviews are uniformly negative?
Replies from: Louie↑ comment by Louie · 2013-01-09T18:00:05.658Z · LW(p) · GW(p)
Good question. If I remember correctly, Berkeley teaches from it and one person I respect agreed it was good. I think the impenetrability was consider more of a feature than a bug by the person doing the recommending. IOW, he was assuming that people taking my recommendations would be geniuses by-and-large and that the harder book would be better in the long-run for the brightest people who studied from it.
Part of my motivation for posting this here was to improve my recommendations. So I'm happy to change the rec to something more accessible if we can crowd-source something like a consensus best choice here on LW that's still good for the smartest readers.
Replies from: Vladimir_Nesov, pragmatist, John_Maxwell_IV, VincentYu↑ comment by Vladimir_Nesov · 2013-01-09T20:14:54.592Z · LW(p) · GW(p)
[he was assuming that] people taking my recommendations would be geniuses by-and-large and that the harder book would be better in the long-run for the brightest people who studied from it.
Is this actually true? My current guess is that even though for a given level of training, smarter people can get through harder texts, they will learn more if they go through easier texts first.
Replies from: wedrifid, BrandonReinhart↑ comment by wedrifid · 2013-01-10T07:50:13.530Z · LW(p) · GW(p)
Is this actually true? My current guess is that even though for a given level of training, smarter people can get through harder texts, they will learn more if they go through easier texts first.
I think you're right here (enough so that you beat me to my reply nearly verbatim.)
↑ comment by BrandonReinhart · 2013-01-11T03:49:49.642Z · LW(p) · GW(p)
Furthermore, a hard to use text may be significantly less hard to use in the classroom where you have peers, teachers, and other forms of guidance to help digest the material. Recommendations for specialists working at home or outside a classroom might not be the same as the recommendations you would give to someone taking a particular class at Berkeley or some other environment where those resources are available.
A flat out bad textbook might seem really good when it is something else such as the teacher, the method, or the support that makes the book work.
↑ comment by pragmatist · 2013-01-09T18:14:14.600Z · LW(p) · GW(p)
I haven't read multiple discrete math textbooks, so I can't make a comparative judgment, but I can confirm that Concrete Mathematics is a delightful and useful text.
Also, while Sipser's book is great for theoretical computer science, I think Moore's The Nature of Computation is much better, at least in terms of being fun to read.
↑ comment by John_Maxwell (John_Maxwell_IV) · 2013-01-10T09:57:11.046Z · LW(p) · GW(p)
My recollection of Berkeley's discrete math course (compsci-oriented version that also emphasizes probability heavily) was that it was taught mostly from some pretty nice lecture notes. Looks like the lecture notes from the Fall 2012 compsci version of the course are available for download from the course website.
It occurred to me the other year that lecture notes could be a good way to learn things in general, or at least pick up the basics of a subject:
There's no incentive for instructors to pad them to appease publishers.
Unlike textbooks or Wikipedia, they're being updated constantly based on what seems to work for explaining concepts most effectively.
They're often available freely for download. (site:youruniversity.edu yoursubject on Google, OCWs, etc.)
They're probably written to communicate understanding rather than achieve perfect rigor.
If you know how long the corresponding lecture took, you can set a timer for that length of time (or 0.8 times as long, or whatever) and aim to get through the lecture notes before the timer rings (taking notes if you want; keep glancing at the timer and your progress to calibrate your speed). This is a pretty good motivational hack, in my experience. I like it better than attending an actual lecture because I can dive deeper or skim over stuff as I like, so it's kind of like a personalized version of the lecture. I don't worry if I end up skimming over some stuff and not understanding it perfectly--I don't understand everything in a "real" lecture either (much less, really).
They break the course material in to nice manageable chunks: if the class covered one set of notes per day, for instance, you could do the same in your self-study. (Don't Break the Chain and BeeMinder come to mind as macro-level motivational hacks that may be useful.)
↑ comment by VincentYu · 2013-01-10T03:56:22.049Z · LW(p) · GW(p)
I think the impenetrability was consider more of a feature than a bug by the person doing the recommending. IOW, he was assuming that people taking my recommendations would be geniuses by-and-large and that the harder book would be better in the long-run for the brightest people who studied from it.
You must have misremembered; Rosen's text is very verbose and clear. It will certainly be an elementary text for any second-year or higher math undergraduate.
The book is partly designed to be a crash course in math for CS students, so there are introductory chapters on proofs, logic, sets, etc., with plenty of examples. Exercises are mainly drills and computations, with a smaller proportion of exercises on proofs.
As for the bad Amazon reviews—many students using this textbook will be encountering mathematical proofs for the first time, so frustration from some of the students is to be expected. I don't think the reviews are representative of the book.
All in all, the text serves its purpose well as an introductory math book for CS undergraduates. I think its greatest downfall is its excessive verbosity—there is so much redundancy that examples take up around half the book.
comment by Mitchell_Porter · 2013-01-09T15:48:32.179Z · LW(p) · GW(p)
Friendliness researchers also need to study what human values actually are and how they are implemented in the brain.
There is apparently a pervasive assumption (not quite spelled out) that a general theory of reflective ethical idealization will be found, and also a general method of inferring the state-machine structure of human cognition, and then Friendliness will be obtained by applying the latter method to human cognitive and neuroscientific data, and then using the general theory to extrapolate a human-relative ideal decision theory from the relevant aspects of the inferred state machine.
I think this is somewhat utopian, and the efficient path forward will involve close engagement with the details of moral cognition (and other forms of decision-making cognition) as ascertained by human psychologists and neuroscientists. The fallible, evolving "first draft" of human state-machine architecture that they produce should offer profound guidance for anyone trying to devise rigorous computational-epistemic methods for going from raw neuro-cognitive data, to state-machine model of the generic human brain, to idealized value system suitable for implementing friendly AI. (For that matter, the whole process also needs to consider the environmental, social, and cultural embedding of human cognition - the brains whose volitions we want to extrapolate are human brains that grow up in humane supportive societies, not feral wolf-child Robinson-Crusoe brains that never learn language or socialization.)
So the ideal curriculum needs to contain an element, not just of formal decision theory, but of the empirical study of human decision-making. But I'm not sure where that is best covered.
Replies from: Louie, ESRogs↑ comment by Louie · 2013-01-09T16:21:22.629Z · LW(p) · GW(p)
But I'm not sure where that is best covered.
Yeah, universities don't reliably teach a lot of things that I'd want people to learn to be Friendliness researchers. Heuristics and Biases is about the closest most universities get to the kind of course you recommend... and most barely have a course on even that.
I'd obviously be recommending lots of Philosophy and Psychology courses as well if most of those courses weren't so horribly wrong. I looked through the course handbooks and scoured them for courses I could recommend in this area that wouldn't steer people too wrong. As Luke has mentioned (partially from being part of this search with me), you can still profitably take a minority of philosophy courses at CMU without destroying your mind, a few at MIT, and maybe two or three at Oxford. And there are no respectable, mainstream textbooks to recommend yet.
Believe me, Luke and I are sad beyond words every day of our lives that we have to continue recommending people read a blog to learn philosophy and a ton of other things that colleges don't know how to teach yet. We don't particularly enjoy looking crazy to everyone outside of the LW bubble.
Replies from: Qiaochu_Yuan, Halfwitz↑ comment by Qiaochu_Yuan · 2013-01-10T01:13:16.313Z · LW(p) · GW(p)
Believe me, Luke and I are sad beyond words every day of our lives that we have to continue recommending people read a blog to learn philosophy and a ton of other things that colleges don't know how to teach yet. We don't particularly enjoy looking crazy to everyone outside of the LW bubble.
This doesn't look as bad as it looks like it looks. Among younger mathematicians, I think it's reasonably well-known that the mathematical blogosphere is of surprisingly high quality and contains many insights that are not easily found in books (see, for example, Fields medalist Terence Tao's blog). So I would expect that younger mathematicians would not care so much about the difference between a good blog recommendation and a good book recommendation. I, for one, have been learning math from blog posts for years, but I might be an outlier in this regard.
↑ comment by Halfwitz · 2013-07-28T18:41:19.195Z · LW(p) · GW(p)
Believe me, Luke and I are sad beyond words every day of our lives that we have to continue recommending people read a blog to learn philosophy and a ton of other things that colleges don't know how to teach yet. We don't particularly enjoy looking crazy to everyone outside of the LW bubble.
One hack for this would be to roll the blogposts into an ebook. A small change in title and presentation can make a big difference in terms of perception.
Replies from: Vladimir_Nesov↑ comment by Vladimir_Nesov · 2013-07-28T18:51:47.323Z · LW(p) · GW(p)
Good idea, and people at MIRI already thought of it.
comment by ChrisHallquist · 2013-01-10T07:33:06.483Z · LW(p) · GW(p)
Here's my reaction to this post.
Replies from: DaFranker
↑ comment by DaFranker · 2013-01-10T17:28:37.077Z · LW(p) · GW(p)
+1
I shall beat this course dead with the power of Learning!
Replies from: IlyaShpitser↑ comment by IlyaShpitser · 2013-01-10T18:54:43.813Z · LW(p) · GW(p)
By the way, going through all this would be roughly equivalent to a double major in cs/math at a major university (minus breadth requirements).
Replies from: DaFranker↑ comment by DaFranker · 2013-01-10T19:10:06.897Z · LW(p) · GW(p)
Heh. Unfortunately, doing all of this exclusively through the online courses would be far less socially recognized than said double major in a major university. I have no easy access to major universities, at all, let alone in ones that hold these courses, and that's only the start of my troubles. (hint: Colleges and Universities sometimes make people sign certain "contracts", and these are usually automatically disclosed to all affiliated educational or academic institutions.)
On that note, the major downside to the Coursera online classes is that they only take place at certain predetermined times, and many of the courses I want are TBA / don't start until a very long time in the future, or have already run with no planned reruns (the Functional Programming course linked in the recommendations is an example of the latter, so essentially we can't take the recommended course at all, period).
Replies from: TsviBT↑ comment by TsviBT · 2013-01-10T21:52:20.389Z · LW(p) · GW(p)
Some courses on Coursera make the video lectures available before and after the official run of the class. E.g., Probabilistic Graphical Models.
comment by roystgnr · 2013-01-09T20:55:23.390Z · LW(p) · GW(p)
"Getting the right answer" doesn't really describe numerical analysis. I'd have said "Recognizing when you're going to get the wrong answer, and getting a controllable upper bound on how wrong". The book you list seems typical: only one chapter even begins by discussing an exact rather than an approximate method, and the meat of that chapter is about how badly the error can blow up when you try to use that method with inexact floating-point arithmetic.
comment by Alex_Altair · 2013-01-09T15:30:45.204Z · LW(p) · GW(p)
Thanks for posting this! I love how you put course numbers on the left, to make it extra-actionable!
comment by incariol · 2013-01-23T00:10:28.806Z · LW(p) · GW(p)
Apart from Numerical Analysis and Parallel Computing which seem a bit out of place here (*), and swapping Bishop's Pattern Recognition for Murphy's ML: A Probabilistic Perspective or perhaps Barber's freely available Bayesian Reasoning and ML, this is actually quite a nice list - if complemented with Vladimir Nesov's. ;)
(*) We're still in a phase that's not quite philosophy in a standard sense of the word, but nonetheless light years away from even starting to program the damn thing, and although learning functional programming from SICP is all good and well due to its mind-expanding effects, going into specifics of designing programs for parallel architectures or learning about various techniques for numerical integration is ... well, I'd rather invest my time in going through the Princeton Companion to get a nice bird's eye view of math, or grab Pearl's Probabilistic Reasoning in Intelligent Systems and Causality to get a feel for what a formal treatment/reduction of an intuitive concept looks like, and leave numerics and other software design issues for a time when they become relevant.
comment by John_Maxwell (John_Maxwell_IV) · 2013-01-10T09:34:07.773Z · LW(p) · GW(p)
There's threading, deadlocks, message passing, etc.
These typically aren't used with a purely functional progamming style, are they?
On another note, if you're thinking of doing FAI research, it might be a good idea to also study relevant-seeming stuff that's not on this list in order to give yourself a different set of tools than other FAI thinkers.
Replies from: IlyaShpitser↑ comment by IlyaShpitser · 2013-01-10T18:56:51.441Z · LW(p) · GW(p)
Not only is threading used in purely functional languages, but one of the main points of pure functional languages is they make threading very easy.*
- not really, but easier because no state to share.
comment by ESRogs · 2013-01-09T21:58:14.398Z · LW(p) · GW(p)
Anything on ethics / meta-ethics, or are those considered to be covered by the sequences?
Replies from: lukeprog↑ comment by lukeprog · 2013-01-10T22:26:41.655Z · LW(p) · GW(p)
It's really hard to find good writing on metaethics. My recommendation would be to read the chapter on ethical reductionism from Miller's introduction to contemporary metaethics, my own unfinished sequence on metaethics, and Eliezer's new sequence (most of it's not metaethics, but it's required reading for understanding the explanation of his 2nd attempt to explain metaethics, which is more precise than his first attempt in the earlier Sequences).
comment by Klao · 2013-01-09T17:19:15.306Z · LW(p) · GW(p)
Very interesting list, thanks Louie!
I just randomly clicked on a few links for online courses, and it seems there's at least one issue: The "Probability and Computing" part points to "Analytic Combinatorics, Part I" coursera course, which is not about probability at all. The MIT and CMU links for this part seem wrong too. Someone should carefully go through all the links and fix them.
Replies from: Louie↑ comment by Louie · 2013-01-14T20:47:54.073Z · LW(p) · GW(p)
Just to clarify, I recommend the book "Probability and Computing" but the course I'm recommending is normally called something along the lines of "Combinatorics and Discrete Probability" at most universities. So the online course isn't as far off base as it may have looked. However, I agree there are better choices that cover more exactly what I want. So I've updated it with a more on-point Harvard Extension course.
The MIT and CMU courses both cover combinatorics and discrete probability. They are probably the right thing to take or very close to it if you're at those particular schools.
Thanks again for the feedback Klao.
comment by Ben Pace (Benito) · 2013-01-11T15:10:21.114Z · LW(p) · GW(p)
The set theory book only links to an image of the book, not the amazon page,
Replies from: Louiecomment by ChrisHallquist · 2013-01-10T08:26:32.625Z · LW(p) · GW(p)
Now a more serious comment: as someone who already has their undergraduate degree, how could I best go about going back to school to take these courses in a classroom setting? Particularly, how could I do so on a sensible budget?
Replies from: Kaj_Sotala, John_Maxwell_IV↑ comment by Kaj_Sotala · 2013-01-10T15:53:33.824Z · LW(p) · GW(p)
There are some countries which offer free higher education, to foreign students as well as natives (if you get accepted into the program). E.g. although there has been pressure to introduce tuition fees to foreign students, and some universities have experimented with this, Finnish universities generally remain free. Undergraduate courses are mostly taught in Finnish, but the Master's level courses of many degree programs (such as my university's computer science program) are taught predominantly in English.
Of course, this requires a willingness to move to a foreign country.
Replies from: ChrisHallquist↑ comment by ChrisHallquist · 2013-01-11T09:49:53.345Z · LW(p) · GW(p)
This listing looks like largely undergraduate courses, and unfortunately I don't speak Finnish! Though in the US, some Master's programs are set up to accept students who want to do something different than what they did in undergrad, I have no idea if this applies to Finnish universities at all. And honestly Finland sounds like a wonderful country, would not mind moving there except maybe because of the cold.
Replies from: Kaj_Sotala↑ comment by Kaj_Sotala · 2013-01-11T10:14:30.103Z · LW(p) · GW(p)
Yeah, it is true that Finnish universities do generally require your Master's to be pretty similar than your undergrad - for example, looking at the admission criteria document from my faculty's international admission pages (it's my understanding that other universities have similar policies), it says that
Following receipt of a reasoned application, the admissions board may grant the right to pursue the Master of Science degree without an entrance examination to an applicant who has completed an applicable Bachelor’s degree with good grades. In such cases, the applicant must meet the Finnish, Swedish or English language requirements, and the content and scope of the completed degree must be correspond sufficiently to the Bachelor of Science degree in the same discipline at the Faculty of Science. Admission will be based on the amount, quality and grades of the completed studies as well as on the letter of motivation.
On the other hand, it does also say that
If the content of a successful applicant’s Bachelor’s degree does not correspond sufficiently to the Faculty requirements for the Bachelor of Science degree in the relevant major subject, the applicant may have to complete up to 60 credits of supplementary major subject, minor subject and language studies before beginning to pursue advanced studies for the Master’s degree.
But I don't know how similar the previous degree has to be in order for them to say "okay, but you have to complete extra courses" instead of rejecting the application outright.
↑ comment by John_Maxwell (John_Maxwell_IV) · 2013-01-10T10:16:02.702Z · LW(p) · GW(p)
How do you like the look of my lecture note freerider self-study plan? (Note: largely untested.)
Replies from: ChrisHallquist↑ comment by ChrisHallquist · 2013-01-10T12:35:49.769Z · LW(p) · GW(p)
It's an interesting idea, and I may test it. Not really what I was looking for, though. The benefits of actual classroom courses I'm thinking about here include instructor feedback and the ability to put something on your resume. So I'm hoping for advice on certificate programs, etc.
comment by backslash · 2013-06-30T17:14:28.823Z · LW(p) · GW(p)
MIT has an HTML of Structure and Interpretation of Computer Programs
comment by KnaveOfAllTrades · 2013-01-13T08:45:01.988Z · LW(p) · GW(p)
Is Julian Barbour's book suitable for this list, or does it, say, require too big a detour through detailed physics?
comment by Kyre · 2013-01-10T05:15:22.949Z · LW(p) · GW(p)
Nice list.
Do any of the programming or mathematics course cover Domain Theory ?
Given the importance of self-modification, fixed points and so forth in FAI, that might be a useful subject to know about. That's just a guess, I don't know enough about either domain theory or FAI to know if it is really relevent.
comment by musen · 2015-10-17T11:38:05.329Z · LW(p) · GW(p)
Hi. Thanks for the recommendations above!
I'm currently a philosophy student that has primarily thought about and done work in foundational value theory, where I've looked at how one might build moral and political theories from the ground up (considering almost everyone in the field doesn't even seem to care about this).
I've spend the last couple of months getting into the topic of specifying a "value" or "governing principle(s)" for an AGI, reading up Bostrom's new book as well as several of his papers and other papers by MIRI. The angle I'm trying to approach it from is to look at this problem from more of a value-theoretic perspective - it's surprising to me that almost no moral/political philosophers are interested in this issue considering the parallels (especially with methodological work being done in the value theory fields).
I'm interested to know, and I'd like to ask, (1) Whether people here think that there is any severe (possibly unsurmountable) limitation of coming from this kind of angle to the problem of FAI and if so what that limitation is, and (2) if I were to approach the problem of FAI from this angle, what kind of mathematical/decision-theoretic stuff would I ABSOLUTELY need to know to even have hopes of doing anything relevant?
Generally, I'd like to also hear any other thoughts on approaching the problem of FAI from my kind of background. Thanks a lot!
Edit: It seems to me that there are two problems in FAI - finding out the content of any value or principle-set that would allow for friendliness (such as CEV), and then ensuring that the exact sense/intention of this set is represented in the AI's goal content. Would I be right in assuming that to draw a very rough distinction, the latter problem is what requires the brunt of the mathematical/computational training? To whatever extent a line can be drawn in between the two, how would you guys recommend one go about exploring the first problem? Thanks!
comment by NancyLebovitz · 2013-01-09T17:03:10.389Z · LW(p) · GW(p)
Should there be some biology/medicine/ecology in there, or is the xrisks book enough?
Replies from: Louie↑ comment by Louie · 2013-01-09T18:04:35.523Z · LW(p) · GW(p)
I don't think those courses would impoverish anyones' minds. I expect people to take courses that aren't on this list without me having to tell people to do that. But I wouldn't expect courses drawn from these subjects to be mainstream recommendations for Friendliness researchers who were doing things like formalizing and solving problems relating to self-referencing mathematical structures and things along those lines.
Replies from: NancyLebovitz↑ comment by NancyLebovitz · 2013-01-09T18:07:41.860Z · LW(p) · GW(p)
What I was thinking was "would you expect a FAI to do its own research about what it needs to for people to be physically safe enough, or should something on the subject be built in?
Replies from: Louie, earthwormchuck163↑ comment by Louie · 2013-01-09T18:40:25.758Z · LW(p) · GW(p)
Ahh. Yeah, I'd expect that kind of content is way too specific to be built into initial FAI designs. There are multiple reasons for this, but off the top of my head,
I expect design considerations for Seed AI to favor smaller designs that only emphasize essential components for both superior ability to show desirable provability criteria, as well as improving design timelines.
All else equal, I expect that the less arbitrary decisions or content the human programmers provide to influence the initial dynamic of FAI, the better.
And my broadest answer is it's not a core-Friendliness problem, so it's not on the critical path to solving FAI. Even if an initial FAI design did need medical content or other things along those lines, this would be something that we could hire an expert to create towards the end of solving the more fundamental Friendliness and AI portions of FAI.
↑ comment by earthwormchuck163 · 2013-01-10T08:15:48.989Z · LW(p) · GW(p)
Note that this actually has very little to do with most of the seemingly hard parts of FAI theory. Much of it would be just as important if we wanted to create a recursively self modifying paper-clip maximizer, and be sure that it wouldn't accidentally end up with the goal of "do the right thing".
The actual implementation is probably far enough away that these issues aren't even on the radar screen yet.
comment by Dr_Manhattan · 2013-01-09T16:54:03.174Z · LW(p) · GW(p)
some quick notes -
- this has got to fit in somwhere https://www.coursera.org/course/compneuro
- Udacity's AI course is actually taught by the book's author, though Berkley is very good and gives programming assignments. Also, Berkley course is more in-depth and will be running a part 2 next year.
comment by Louie · 2013-01-09T14:39:20.458Z · LW(p) · GW(p)
PS - I had some initial trouble formatting my table's appearance. It seems to be mostly fixed now. But if an admin wants to tweak it somehow so the text isn't justified or it's otherwise more readable, I won't complain! :)
Replies from: bgaesopcomment by [deleted] · 2013-02-01T21:38:27.241Z · LW(p) · GW(p)
Though narrower in scope, Hutter's AI Recommendation Page is also very informative; the Undergraduate Books and Courses section provides a useful list of textbooks, as well as general advice and recommended ANU courses.
Additionally, I'm available to compile an extensive list of Friendliness theory course recommendations for the ANU, if this is of interest to anyone. (from there I could also expand it to cover other major Australian universities)
comment by [deleted] · 2015-07-30T00:37:21.913Z · LW(p) · GW(p)
Thanks for your recommendations Louie.
This is a tremendous amount of reading. I've read parts of Cognitive Science and skimmed through bits of Heuristics and Biases. But, reading books bores me in the age of the internet!
I'm currently interested in conceptualising the open problems in FAI research. So, here is my expedient strategy in case someone wants to critique or copy it.
I'm just planning to Wikipedia and google interesting questions around BQP (bounded error quantum polynomial time) to understand Pascal's mugging better, and perhaps Universal Artificial Intelligence though I'm somewhat averse to sources that is not readily available to people I might want to converse about it (and I don't want to carry around a textbook).
comment by [deleted] · 2013-01-09T17:52:37.609Z · LW(p) · GW(p)
It requires only a little understanding of complex systems, and the nature of cognition, to realize that the recommendation "Go study math and theoretical computer science" is useless for understanding the Friendliness Problem.
For more details you can read one of the papers I have published on this topic. (See richardloosemore.com/papers). The short version of a long argument is that there are no known AI control (motivation) algorithms that are stable in the limit as the supergoal statements become abstract enough to drive a general intelligence. So, instead, we would expect their motivation systems to be structured as massive, weak constraint satisfaction engines. Such a weak constraint engine, although potentially stable (and friendly), is a complex system, so however friendly and stable it might be, those features will likely remain forever unprovable. Hence, mathematics is of no use.
Replies from: Kaj_Sotala↑ comment by Kaj_Sotala · 2013-01-09T20:00:19.890Z · LW(p) · GW(p)
The short version of a long argument is that there are no known AI control (motivation) algorithms that are stable in the limit as the supergoal statements become abstract enough to drive a general intelligence.
At least, not yet?