Posts
Comments
like, my views aren’t that internally coherent. My suspicion is others’ views are even less internally coherent.
I would appreciate hearing more concretely what it means to be internally coherent/incoherent (e.g. are there examples of contradictory statements a single person is making?).
It’s lots of saving throws, you know? And you multiply the saving throws together and things look better. And they interact better than that because– well, in one way worse because it’s correlated: If you’re incompetent, you’re more likely to fail to solve the problem and more likely to fail to coordinate not to destroy the world. In some other sense, it’s better than interacting multiplicatively because weakness in one area compensates for strength in the other. I think there are a bunch of saving throws that could independently make things good, but then in reality you have to have a little bit here and a little bit here and a little bit here, if that makes sense.
I don't understand this part. Translating to math, I think it's saying something like, if is the probability that saving throw works, then the probability that at least one of them works is (assuming the saving throws are independent), which is higher the more saving throws there are; but due to correlation, the saving throws are not independent, so we effectively have fewer saving throws. I don't understand what "weakness in one area compensates for strength in the other" or "a little bit here and a little bit here and a little bit here" mean.
Christiano cares more about making aligned AIs that are competitive with unaligned AIs, whereas MIRI is more willing to settle for an AI with very narrow capabilities.
Looking at the transcript, it seems like "AI with very narrow capabilities" is referring to the "copy-paste a strawberry" example. It seems to me that the point of the strawberry example (see Eliezer's posts 1, 2, and Dario Amodei's comment here) is that by creating an AGI that can copy and paste a strawberry, we necessarily solve most of the alignment problem. So it isn't the case that MIRI is aiming for an AI with very narrow capabilities (even task AGI is supposed to perform pivotal acts).
Thanks! I think I understand this now.
I will say some things that occurred to me while thinking more about this, and hope that someone will correct me if I get something wrong.
- "Human imitation" is sometimes used to refer to the outward behavior of the system (e.g. "imitation learning", and in posts like "Just Imitate Humans?"), and sometimes to refer to the model of the human inside the system (e.g. here when you say "the human imitation is telling the system what to do").
- A system that is more capable than a human can still be a "human imitation", because "human imitation" is being used in the sense of "modeling humans inside the system" instead of "has the outward behavior of a human".
- There is a distinction between the counterfactual training procedure vs the resulting system. "Counterfactual oracle" (singular) seems to be used to refer to the resulting system, and Paul calls this "the system" in his "Human-in-the-counterfactual-loop" post. "Counterfactual oracles" (plural) is used both as a plural version of the resulting system and also as a label for the general training procedure. "Human-in-the-counterfactual-loop", "counterfactual human oversight", and "counterfactual oversight" all refer to the training procedure (but only when the procedure uses a model of the human).
Paul Christiano does have a blog post titled Counterfactual oversight vs. training data, which talks about the same thing as this post except that he uses the term "counterfactual oversight", which is just Counterfactual Oracles applied to human imitation (which he proposes to use to "oversee" some larger AI system).
I am having trouble parsing/understanding this part.
- The linked post by Paul doesn't seem to talk about human imitation. Is there a separate post/comment somewhere that connects counterfactual oversight to human imitation, or is the connection to human imitation somehow implicit in the linked post?
- The linked post by Paul seems to be talking about counterfactual oversight as a way to train the counterfactual oracle, but I'm parsing your sentence as saying that there is then a further step where the counterfactual oracle is used to oversee a larger AI system (i.e. the "oversee" in your sentence is different from the "oversight" in "counterfactual oversight"). Is this right?
I noticed that I have two distinct "mental pictures" for what the overseer is, depending on how the Distill procedure works (i.e. depending on the narrow technique used in the Distill procedure).
- For imitation learning and narrow inverse reinforcement learning: a "passive" overseer that just gets used as a template/target for imitation.
- For narrow reinforcement learning and in discussions about approval-directed agents: an "active" overseer that rates actions or provides rewards.
I wonder if this way of thinking about the overseer is okay/correct, or if I'm missing something (e.g. maybe even in case (1), the overseer has a more active role than I can make out). Assuming this way of thinking about the overseer is okay, it seems like for case (1), the term "overseer" has connotations that extend beyond the role played by the overseer (i.e. it doesn't really provide any oversight since it is passive).
Based on discussion between Vladimir Slepnev and Paul in this thread, it seems like statements in this post ("we assume that A[0] can acquire nearly human-level capabilities through this process", "Given an aligned agent H we can use narrow safe learning techniques to train a much faster agent A which behaves as H would have behaved") that the first stage of IDA will produce nearly-human-level assistants are misleading. In the same thread, Paul says that he "will probably correct it", but as far as I can tell, neither the Medium post nor the version of the post in this sequence (which was published after the discussion) has been corrected.
I had this same thought, but my understanding (which is not solid) is that in the first iteration, since A is random, H can just ignore A and go with its own output (if my assistants are unhelpful, I can just try to perform the task all on my own). So Amplify(H, A) becomes H, which means A <- Distill(Amplify(H, A)) is basically A <- Distill(H), exactly as you suggested.
I'm still confused about the difference between HCH and the amplification step of IDA. Initially I thought that the difference is that with HCH, the assistants are other copies of the human, whereas in IDA the assistants are the distilled agents from the previous step (whose capabilities will be sub-human in early stages of IDA and super-human in later stages). However, this FAQ says "HCHs should not be visualized as having humans in the box."
My next guess is that while HCH allows the recursion for spawning new assistants to be arbitrarily deep, the amplification step of IDA only allows a single level of spawning (i.e. the human can spawn new assistants, but the assistants themselves cannot make new assistants). Ajeya Cotra's post on IDA talks about the human making calls to the assistant, but not about the assistants making further calls to other assistants, so it seems plausible to me that the recursive nature of HCH is the difference. Can someone who understands HCH/IDA confirm that this is a difference and/or name other differences?
The page https://arbital.greaterwrong.com/p/AI_safety_mindset/ is blank in the GreaterWrong version, but has content in the obormot.net version.
I was surprised to see, both on your website and the white paper, that you are part of Mercatoria/ICTP (although your level of involvement isn't clear based on public information). My surprise is mainly because you have a couple of comments on LessWrong that discuss why you have declined to join MIRI as a research associate. You have also (to my knowledge) never joined any other rationality-community or effective altruism-related organization in any capacity.
My questions are:
- What are the reasons you decided to join or sign on as a co-author for Mercatoria/ICTP?
- More generally, how do you decide which organizations to associate with? Have you considered joining other organizations, starting your own organization, or recruiting contract workers/volunteers to work on things you consider important?
"Sam Harris and the Is–Ought Gap" might be one example.
Some of the math seems to be garbled, e.g. this page (compare with the version on obormot.net) and this page (compare with the version on obormot.net).
The scrape seems to be missing the Solomonoff induction dialogue, which was available at https://arbital.com/p/solomonoff_induction/?l=1hh (at the moment I just get an error).
ETA (2019-06-28): The new version of the scrape has this page, and can be viewed on the GreaterWrong version.
Going by public documents, it seems like the intention was to have a separate organization pretty early on (although I can't say about the very beginning since I wasn't involved).
The Articles of Incorporation for CFAR (called Feynman Foundation at the time) are dated July 18, 2011, and 501(c)(3) status was approved on July 26, 2011.
MIRI's 2011 strategic plan (updated August 2011; unclear when it was first written) says "Encourage a new organization to begin rationality instruction similar to what Singularity Institute did in 2011 with Rationality Minicamp and Rationality Boot Camp."
The Minicamp took place in May–June 2011, and it seems like the Boot Camp took place in June–August 2011, so it looks like by the time the first workshop finished and the second was in progress, the plan was already to start the new organization. However, it's still possible that there was no plan for a separate organization before or during the first workshop.
MIRI's December 2011 progress report also talks about plans to create the "Rationality Org".
I also wrote a timeline of CFAR a while back, which has more links.
The examples in the buckets error post have "modus delens" as the correct response. To take the diet example from the post, A = "diet worth being on", B = "zero studies suggesting health risks". Adam has stored in his brain, and Betty presents , so Adam's brain computes . The "protecting epistemology" move is to instead adamantly believe A ("I need to stay motivated!") which ends up rejecting what Betty said. But the desired response is to instead deny B but also accept A, and hence to deny the implication .
So in these buckets error examples, modus ponens corresponds to the "automatic" reasoning, modus tollens corresponds to the "flinching away from the truth" move, and modus delens corresponds to the "rational" move that avoids the buckets error.
I explained this more in a comment on the post.
I can't comment as to the relative frequency of this response and how often it's correct (this sort of question seems difficult to answer).
Was this post ever published? (Looking at Zvi's posts since January, I don't see anything that looks like it.)
I've made around 250 Anki cards about AI safety. I haven't prioritized sharing my cards because I think finding a specific card useful requires someone to have read the source material generating the card (e.g. if I made the card based on a blog post, one would need to read that exact blog post to get value out of reviewing the card; see learn before you memorize). Since there are many AI safety blog posts and I don't have the sense that lots of Anki users read any particular blog post, it seems to me that the value generated from sharing a set of cards about a blog post isn't high enough to overcome the annoyance cost of polishing, packaging, and uploading the cards.
More generally, from a consumer perspective, I think people tend to be pretty bad at making good Anki cards (I'm often embarrassed at the cards I've created several months ago!), which makes it unexciting for me to spend a lot of effort trying to collaborate with others on making cards (because I expect to receive poorly-made cards in return for the cards I provide). I think collaborative card-making can be done though, e.g. Michael Nielsen and Andy Matuschak's quantum computing guide comes with pre-made cards that I think are pretty good.
Different people also have different goals/interests so even given a single source material, the specifics one wants to Ankify can be different. For example, someone who wants to understand the technical details of logical induction will want to Ankify the common objects used (market, pricing, trader, valuation feature, etc.), the theorems and proof techniques, and so forth, whereas someone who just wants a high-level overview and the "so what" of logical induction can get away with Ankifying much less detail.
Something I've noticed is that many AI safety posts aren't very good at explaining things (not enough concrete examples, not enough emphasis on common misconceptions and edge cases, not enough effort to answer what I think of as "obvious" questions); this fact is often revealed by the comments people make in response to a post. This makes it hard to make Anki cards because one doesn't really understand the content of the post, at least not well enough to confidently generate Anki cards (one of the benefits of being an Anki user is having a greater sensitivity to when one does not understand something; see "illusion of explanatory depth" and related terms). There are other problems like conflicting usage of terminology (e.g. multiple definitions of "benign", "aligned", "corrigible") and the fact that some of the debates are ongoing/some of the knowledge is still being worked out.
For "What would be a good strategy for generating useful flashcards?": I try to read a post or a series of posts and once I feel that I understand the basic idea, I will usually reread it to add cards about the basic terms and ask myself simple questions. Some example cards for iterated amplification:
- what kind of training does the Distill step use?
- in the pseudocode, what step gets repeated/iterated?
- how do we get A[0]?
- write A[1] in terms of H and A[0]
- when Paul says IDA is going to be competitive with traditional RL agents in terms of time and resource costs, what exactly does he mean?
- advantages of A[0] over H
- symbolic expression for the overseer
- why should the amplified system (of human + multiple copies of the AI) be expected to perform better than the human alone?
You might have in mind a Facebook post that Vipul Naik wrote in 2017.
We can model success as a combination of doing useful things and avoiding making mistakes. As a particular example, we can model intellectual success as a combination of coming up with good ideas and avoiding bad ideas. I claim that rationality helps us avoid mistakes and bad ideas, but doesn’t help much in generating good ideas and useful work.
Eliezer Yudkowsky has made similar points in e.g. "Unteachable Excellence" ("much of the most important information we can learn from history is about how to not lose, rather than how to win", "It's easier to avoid duplicating spectacular failures than to duplicate spectacular successes. And it's often easier to generalize failure between domains.") and "Teaching the Unteachable".
Thoughts on #10:
I am confused about this exercise. The standard/modern proof of Gödel's second incompleteness theorem uses the Hilbert–Bernays–Löb derivability conditions, which are stated as (a), (b), (c) in exercise #11. If the exercises are meant to be solved in sequence, this seems to imply that #10 is solvable without using the derivability conditions. I tried doing this for a while without getting anywhere.
Maybe another way to state my confusion is that I'm pretty sure that up to exercise #10, nothing that distinguishes Peano arithmetic from Robinson arithmetic has been introduced (it is only with the introduction of the derivability conditions in #11 that this difference becomes apparent). It looks like there is a version of the second incompleteness theorem for Robinson arithmetic, but the paper says "The proof is by the construction of a nonstandard model in which this formula [i.e. formula expressing consistency] is false", so I'm guessing this proof won't work for Peano arithmetic.
My solution for #12:
Suppose for the sake of contradiction that such a formula exists. By the diagonal lemma applied to , there is some sentence such that, provably, . By the soundness of our theory, in fact . But by the property for we also have , which means , a contradiction.
This seems to be the "semantic" version of the theorem, where the property for is stated outside the system. There is also a "syntactic" version where the property for is stated within the system.
Attempted solution and some thoughts on #9:
Define a formula taking one free variable to be .
Now define to be . By the definition of we have .
We have
The first step follows by the definition of , the second by the definition of , the third by the definition of , and the fourth by the property of mentioned above. Since by the type signature of , this completes the proof.
Things I'm not sure about:
It's a little unclear to me what the notation means. In particular, I've assumed that takes as inputs Gödel numbers of formulas rather than the formulas themselves. If takes as inputs the formulas themselves, then I don't think we can assume that the formula exists without doing more arithmetization work (i.e. the equivalent of would need to know how to convert from the Gödel number of a formula to the formula itself).
If the biconditional "" is a connective in the logic itself, then I think the same proof works but we would need to assume more about than is given in the problem statement, namely that the theory we have can prove the substitution property of .
The assumption about the quantifier complexity of and was barely used. It was just given to us in the type signature for , and the same proof would have worked without this assumption, so I am confused about why the problem includes this assumption.
That link works, thanks!
The link no longer works (I get "This project has not yet been moved into the new version of Overleaf. You will need to log in and move it in order to continue working on it.") Would you be willing to re-post it or move it so that it is visible?
Some other sources of exercises you might want to check out (that have solutions and that I have used at least partly):
- Multiple choice quizzes (the ones related to linear algebra are determinants, elementary matrices, inner product spaces, linear algebra, linear systems, linear transformations, matrices, and vector spaces)
- Vipul Naik's quizzes (disclosure: I am friends with Vipul and also do contract work for him)
Regarding Axler's book (since it has been mentioned in this thread): there are several "levels" of linear algebra, and Axler's book is at a higher level (emphasis on abstract vector spaces and coordinate-free ways of doing things) than the 3Blue1Brown videos (more concrete, working in ). Axler's book also assumes that the reader has had exposure to the lower level material (e.g. he does not talk about row reduction and elementary matrices). So I'm not sure I would recommend it to someone starting out trying to learn the basics of linear algebra.
Gratuitous remarks:
- I think different resources covering material in a different order and using different terminology is in some sense a feature, not a bug, because it allows one to look at the subject from different perspectives. For instance, the "done right" in Axler's book comes from one such change in perspective.
- I find that learning mathematics well takes an unintuitively long time; it might be unrealistic to expect to learn the material well unless one puts in a lot of effort.
- I think there is a case to be made for the importance of struggling in learning (disclosure: I am the author of the page).
Is there (or will there be) a way to see a list of the latest posts, restricted to posts that are questions? (I am wondering about this both in the GraphQL API and in the site UI.)
I think we are working off different editions. According to the errata, the condition for strict contraction was changed to for all distinct .
Can you say more about why exercise 17.6.3 is wrong?
If we define by then for distinct we have
We also have since
In general, the derivative is , which is continuous on .
There are some more data (post count, comment count, vote count, etc., but not pageviews) at "History of LessWrong: Some Data Graphics".
My solution for #3:
Define by . We know that is continuous because and the identity map both are, and by the limit laws. Applying the intermediate value theorem (problem #2) we see that there exists such that . But this means , so we are done.
Counterexample for the open interval: consider defined by . First, we can verify that if then , so indeed maps to . To see that there is no fixed point, note that the only solution to in is , which is not in . (We can also view this graphically by plotting both and and checking that they do not intersect in .)
Here is my attempt, based on Hoagy's proof.
Let be an integer. We are given that and . Now consider the points in the interval . By 1-D Sperner's lemma, there are an odd number of such that and (i.e. an odd number of "segments" that begin below zero and end up above zero). In particular, is an even number, so there must be at least one such number . Choose the smallest and call this number .
Now consider the sequence . Since this sequence takes values in , it is bounded, and by the Bolzano–Weierstrass theorem there must be some subsequence that converges to some number .
Consider the sequences and . We have for each . By the limit laws, as . Since is continuous, we have and as . Thus and , showing that , as desired.
I'm having trouble understanding why we can't just fix in your proof. Then at each iteration we bisect the interval, so we wouldn't be using the "full power" of the 1-D Sperner's lemma (we would just be using something close to the base case).
Also if we are only given that is continuous, does it make sense to talk about the gradient?
I had a similar thought while reading this post, but I'm not sure invoking causality is necessary (having a direction still seems necessary). Just in terms of propositional logic, I would explain this post as follows:
1. Initially, one has the implication stored in one's mind.
2. Someone asserts .
3. Now one's mind (perhaps subconsciously) does a modus ponens, and obtains .
4. However, is an undesirable belief, so one wants to deny it.
5. Instead of rejecting the implication , one adamantly denies .
The "buckets error" is the implication , and "flinching away" is the denial of . Flinching away is about protecting one's epistemology because denying is still better than accepting . Of course, it would be best to reject the implication , but since one can't do this (by assumption, one makes the buckets error), it is preferable to "flinch away" from .
ETA (2019-02-01): It occurred to me that this is basically the same thing as "one man's modus ponens is another man's modus tollens" (see e.g. this post) but with some extra emotional connotations.
I was confused about this too, but now I think I have some idea of what's going on.
Normally probability is defined for events, but expected value is defined for random variables, not events. What is happening in this post is that we are taking the expected value of events, by way of the conditional expected value of the random variable (conditioning on the event). In symbols, if is some event in our sample space, we are saying , where is some random variable (this random variable is supposed to be clear from the context, so it doesn't appear on the left hand side of the equation).
Going back to cousin_it's lottery example, we can formalize this as follows. The sample space can be and the probability measure is defined as and . The random variable represents the lottery, and it is defined by and .
Now we can calculate. The expected value of the lottery is:
The expected value of winning is:
The "probutility" of winning is:
So in this case, the "probutility" of winning is the same as the expected value of the lottery. However, this is only the case because the situation is so simple. In particular, if was not equal to zero (while winning and losing remained exclusive events), then the two would have been different (the expected value of the lottery would have changed while the "probutility" would have remained the same).
I don't see reference number 17 ("Personal correspondence with Carl Shulman") used in the body of the post. What information from that reference is used in the post?
do we have any statistics about it?
For sessions and pageviews from Google Analytics, I wrote a post about it in April 2017. Since you mention scraping, perhaps you mean something like post and comment counts; if so, I'm not aware of any statistics about that.
Wei Dai has a web service to retrieve all posts and comments of particular users that I find useful (not sure if you will find it useful for gathering statistics, but I thought I would mention it just in case).
Based on descriptions on the FHI website, it looks like Kyle Scott filled this role, from July 2015 to September 2017.
From the earliest snapshot of his FHI bio page:
Kyle brings over 5 years of operations experience to the Future of Humanity Instutute. He keeps daily operations running smoothly, and manages incoming and outgoing requests for Prof. Nick Bostrom.
Strategically, he works to improve the processes and capacity of the office and free up the attention and time of Prof. Nick Bostrom.
Kyle came to the Future of Humanity Institute from the Effective Altruism movement, determining that this job position would be his most effective contribution to society. Learn more about Effective Altruism here.
The page is still up but it doesn't look like he holds the position anymore.
He seems to be a project manager at BERI now:
Kyle manages various projects supporting BERI's partner institutions. He graduated Whitman College with a B.A. in Philosophy. He spent two years working in career services and subsequently moved to Oxford where he worked for 80,000 Hours, the Centre for Effective Altruism and most recently at the Future of Humanity Institute as Nick Bostrom's Executive Assistant.
On November 13, 2017 FHI opened the position for applications.
ETA: Louis Francini comes to the same conclusion on Quora. (Context: I asked the question on Quora, figured out the answer, posted this comment, then Louis answered my question.)
In some recent comments over at the Effective Altruism Forum you talk about anti-realism about consciousness, saying in particular "the case for accepting anti-realism as the answer to the problem of consciousness seems pretty weak, at least as explained by Brian". I am wondering if you could elaborate more on this. Does the case for anti-realism about consciousness seem weak because of your general uncertainty on questions like this? Or is it more that you find the case for anti-realism specifically weak, and you hold some contrary position?
I am especially curious since I was under the impression that many people on LessWrong hold essentially similar views.
Thanks for the feedback. I could add wordcount. Not sure what you mean by quality rating; LW, OB, and EA Forum have their own voting/rating mechanisms but are not compatible (so putting them in a column might be confusing, although grouping by venue and looking at ratings within each venue might be interesting). Summary would be the most time-consuming to produce, and many of Carl's posts have summaries at the top.
I recently wrote an updated timeline. It includes not just formal publications, but also blog posts and conversations. To see just the formal publications, it is possible to sort by the "Format" column in the full timeline and look at the rows with "Paper".
I confirm that I also experience this problem, but I don't have additional insight on the cause.
Hi Evan, did you ever write this post?
Okay I've created a Facebook group here: https://www.facebook.com/groups/LessWrongTokyo/
(To be sure, I don't currently live in Tokyo, but I visit there every summer and would be very interested in attending during that time.)
I took the survey.
Gwern still links to some of muflax's writings, using his own backups. Googling something like "site:gwern.net muflax" turns up some results (though not many).
I usually ask these as questions on Quora. Quora is incredibly tolerant of even inane questions, and has the benefit of allowing others to provide feedback (in the form of answers and comments to the question). If a question has already been asked, then you will also be able to read what others have written in response and/or follow that question for future answers. Quora also has the option of anonymizing questions. I've found that always converting my thoughts into questions has made me very conscious of what sort of questions are interesting to ask (not that there's anything right with that).
Another idea is to practice this with writing down dreams. After waking up, I often think "It's not really worth writing that dream down anyway", whereas in reality I would find it quite interesting if I came back to it later. Forcing oneself to write thoughts down even when one is not inclined to may lead to more sedulous record-keeping. (But this is just speculation.)
It's worth noting that there is also DuckDuckGo (a search engine), which has bang expressions for outsourcing results. Just to give some of the equivalents for those listed above: "!gi" for Google Images, "!yt" for YouTube, "!w" for Wikipedia, etc. To be sure, one has to rely on DuckDuckGo for adding the expressions (although I've had success suggesting a new expression before).
I am also interested in doing Japanese translations.