Yeah, sorry, we are currently importing your post directly as HTML. We don't do code-injection, we figure out what the right HTML for displaying the LaTeX is server-side, and then store that directly in the HTML for the post. 

The reason why it isn't working out of the box is that we don't support single-dollar-sign delimiters for LaTeX in HTML, because they have too many false-positives with people just trying to use dollar signs in normal contexts. Everything would actually work out by default if you used the MathJax \( and \) delimiters instead, which are much less ambiguous. 

I will convert this one manually for now, not sure what the best way moving forward is. Maybe there is a way you can configure your blog to use the \( and \) delimiters instead, or maybe we can adjust our script to get better at detecting when people want to use the single-dollar-delimiter for MathJax purposes, versus other purposes. 

If I were trying to state my claim more clearly, it would be something like "generically, for the large majority of problems of the sort you would come across in ML, once you can distinguish good answers you can find good answers (modulo some amount of engineering work), because non-convex optimization generally works and there are a large number of techniques for solving the sparse rewards problem, which are also getting better over time".

I am a bit confused by what we mean by "of the sort you would come across in ML". Like, this situation, where we are trying to derive an algorithm that solves problems without optimizers, from an algorithm that solves problems with optimizers, is that "the sort of problem you would come across in ML"?. It feels pretty different to me from most usual ML problems. 

I also feel like in ML it's quite hard to actually do this in practice. Like, it's very easy to tell whether a self-driving car AI has an accident, but not very easy to actually get it to not have any accidents. It's very easy to tell whether an AI can produce a Harry Potter-level quality novel, but not very easy to get it to produce one. It's very easy to tell if an AI has successfully hacked some computer system, but very hard to get it to actually do so. I feel like the vast majority of real-world problems we want to solve do not currently follow the rule of "if you can distinguish good answers you can find good answers". Of course, success in ML has been for the few subproblems where this turned out to be easy, but clearly our prior should be on this not working out, given the vast majority of problems where this turned out to be hard.

(Also, to be clear, I think you are making a good point here, and I am pretty genuinely confused for which kind of problems the thing you are saying does turn out to be true, and appreciate your thoughts here)

This is true if all the other datapoints are entirely indistinguishable, and the only signal is "good" vs. "bad". But in practice you would compare / rank the datapoints, and move towards the ones that are better.

Take the backflip example from the human preferences paper: if your only signal was "is this a successful backflip?", then your argument would apply and it would be pretty hard to learn. But the signal is "is this more like a successful backflip than this other thing?" and this makes learning feasible.

More generally, I feel that the thing I'm arguing against would imply that ML in general is impossible (and esp. the human preferences work), so I think it would help to say explicitly where the disanalogy occurs.

I should note that comparisons is only one reason why the situation isn't as bad as you say. Another is that even with only non-approved data points to label, you could do things like label "which part" of the plan is non-approved. And with very sophisticated AI systems, you could ask them to predict which plans would be approved/non-approved, even if they don't have explicit examples, simply by modeling the human approvers very accurately in general.

Well, sure, but that is changing the problem formulation quite a bit. It's also not particularly obvious that it helps very much, though I do agree it helps. My guess is even with a rank-ordering, you won't get the 33 bits out of the system in any reasonable amount of time at 10 hours evaluation cost. I do think if you can somehow give more mechanistic and detailed feedback, I feel more optimistic in situations like this, but also feel more pessimistic that we will actually figure out how to do that in situations like this. 

More generally, I feel that the thing I'm arguing against would imply that ML in general is impossible (and esp. the human preferences work), so I think it would help to say explicitly where the disanalogy occurs.

I feel like you are arguing for a very strong claim here, which is that "as soon as you have an efficient way of determining whether a problem is solved, and any way of generating a correct solution some very small fraction of the time, you can just build an efficient solution that solves it all of the time". 

This sentence can of course be false without implying that the human preferences work is impossible, so there must be some confusion happening. I am not arguing that this is impossible for all problems, indeed ML has shown that this is indeed quite feasible for a lot of problems, but making the claim that it works for all of them is quite strong, but I also feel like it's obvious enough that this is very hard or impossible for a large other class of problems (like, e.g. reversing hash functions), and so we shouldn't assume that we can just do this for an arbitrary problem. 

When you say "costly to replace", this is with respect to what cost function? Do you have in mind the system's original training objective, or something else?

If you have an original cost function F(x) and an approval cost A(x), you can minimize F(x) + c * A(x), increasing the weight on c until it pays enough attention to A(x). For an appropriate choice of c, this is (approximately) equivalent to asking "Find the most approved policy such that F(x) is below some threshold"--more generally, varying c will trace out the Pareto boundary between F and A.

I was talking about "costly" in terms of computational resources. Like, of course if I have a system that gets the right answer in 1/100,000,000 cases, and I have a way to efficiently tell when it gets the right answer, then I can get it to always give me approximately always the right answer by just running it a billion times. But that will also take a billion times longer. 

In-practice, I expect most situations where you have the combination of "In one in a billion cases I get the right answer and it costs me $1 to compute an answer" and "I can tell when it gets the right answer", you won't get to a point where you can compute a right answer for anything close to $1.

Hmm, I don't currently find myself very compelled by this argument. Here are some reasons: 

In order to even get a single expected datapoint of approval, I need to sample 10^8 examples, which in our current sampling method would take 10^8 * 10 hours, e.g. approximately 100,000 years. I don't understand how you could do "Learning from Human Preferences" on something this sparse

I feel even beyond that, this still assumes that the reason it is proposing a "good" plan is pure noise, and not the result of any underlying bias that is actually costly to replace. I am not fully sure how to convey my intuitions here, but here is a bad analogy: It seems to me that you can have go-playing-algorithms that lose 99.999% of games against an expert AI, but that doesn't mean you can distill a competitive AI that wins 50% of games, even though it's "only 33 bits of information". 

Like, the reason why your AI is losing has a structural reason, and the reason why the AI is proposing consequentialist plans also has a structural reason, so even if we get within 33 bits (which I do think seems unlikely), it's not clear that you can get substantially beyond that, without drastically worsening the performance of the AI. In this case, it feels like maybe an AI maybe gets lucky and stumbles upon a plan that solves the problem without creating a consequentialist reasoner, but it's doing that out of mostly luck, not because it actually has a good generator for non-consequentialist-reasoner-generating-plans, and there is no reliable way to always output those plans without actually sampling at least something like 10^4 plans. 

The intuition of "as soon as I have an oracle for good vs. bad plans I can chain an optimizer to find good plans" feels far too strong to me in generality, and I feel like I can come up with dozen of counterexamples where this isn't the case. Like, I feel like... this is literally a substantial part of the P vs. NP problem, and I can't just assume my algorithm just like finds efficient solution to arbitrary NP-hard problems.

Can you be more concrete about how you would do this? If my method for evaluation is "sit down and think about the consequences of doing this for 10 hours", I have no idea how I would chain it with an optimizer to find good plans even if they are rare.

Typo: 

No real power-seeking tendencies if we only plausibly will specify a negative vector.

Seems like two sentences got merged together.

Should be part of the contents field: 

{
  post(input: {selector: {slug: "the-new-editor"}}) {
		result {
      contents {
        markdown
      }
    }
  }
}

I share the sense that something pretty substantial changed with MIRI in ~2017 and that something important got lost when that happened. I share some of the sense that people's thinking about timelines is confused, though I do think overall pretty short timelines are justified (though mine are on the longer end of what MIRI people tend to think, though much shorter than yours, IIRC). I think you are saying some important things about the funding landscape, and have been pretty sad about the dynamics here as well, though I think the actual situation is pretty messy and some funders are really quite pro-critique, and some others seem to me to be much more optimizing for something like the brand of the EA-coalition.

I found this a very helpful and useful comment, and resonate with various bits of it (I also think I disagree with a good chunk of it, but a lot of it seems right overall).

We did ask Jessica first whether she would want to participate in the experiment. Also, the reason why we want to experiment with additional voting systems is because specifically threads like this seem like they kind of tend to go badly using our current voting system, and switching it out is an attempt at making them go better, in a way that I think Jessica also wants.

https://www.lesswrong.com/graphiql 

The reason why we ordered so many books there is because we didn't really think about how small Australia is, and had to make the decision on a somewhat short deadline with a bunch of other stuff happening at the same time. I think if we had thought about it for longer, we wouldn't have made that mistake.

Yeah, I also thought it was pretty interesting. I only thought about it for a few minutes, but it seems interesting enough to give it a shot, IMO.

The post feels like it's trying pretty hard to point towards an alternative forecasting method, though I also agree it's not fully succeeding at getting there. 

I feel like de-facto the forecasting methodology of people who are actually good at forecasting don't usually strike me as low-inferential distance, such that it is obvious how to communicate the full methodology. My sense from talking to a number of superforecasters over the years is that they do pretty complicated things, and I don't feel like the critique of "A critique is only really valid if it provides a whole countermethodology" is a very productive way of engaging with their takes. I feel like I've had lots of conversations of the type "X methodology doesn't work" without someone being able to explain all the details of what they do instead, that were still valuable and helped me model the world, and said meaningful things. Usually the best they can do is something like "well, instead pay attention and build a model of these different factors", which feels like a bar Eliezer is definitely clearing in this post.

FWIW, "plausible" sounds to me basically the same as "possibly". So my guess is this is indeed a linguistic thing.

Yeah, let's also make it a link post then. Some people prefer more prominence, some prefer less, for their cross-posts.

I am also interested in this.

Huh, that's surprising, if by that he means "no contracts between anyone currently at Leverage and anyone at CEA". I currently still think it's the case, though I also don't see any reason for Geoff to lie here. Maybe there is some technical sense in which there is no contract between Leverage and CEA, but there are contracts between current Leverage employees, who used to work at CEA, and current CEA employees? 

One of my ideas for this (when thinking about voting systems in general) is to have a rating that is trivially inconvenient to access. Like, you have a ranking system from F to A, but then you can also hold the A button for 10 seconds, and then award an S rank, and then you can hold the S button for 30 seconds, and award a double S rank, and then hold it for a full minute, and then award a triple S rank. 

The only instance I've seen of something like this implemented is Medium's clap system, which allows you to give up to 50 claps, but you do have to click 50 times to actually give those claps. 

I think some of these are pretty reasonable points, but I am kind of confused by the following: 

This Leverage drama is not important to anyone except a small group of people and does not belong on LW. Perhaps the relatively small group of Bay Area rationalists who are always at the center of these things need to create a separate forum for their own drama. Nobody outside of Berkeley needs to hear about this. This sort of thing gets upvoted because tribal instincts are being activated, not because this is good and ought to be here.

It seems to me that Leverage had a large and broad effect on the Effective Altruism and Rationality communities worldwide, with having organized the 2013-2014 EA Summits, and having provided a substantial fraction of the strategic direction for EAG 2015 and EAG 2016, and then shared multiple staff with the Centre For Effective Altruism until 2019. 

This suggests to me that what happened at Leverage clearly had effects that are much broader reaching than "some relatively small group of Bay Area rationalists". Indeed, I think the Bay Area rationality community wasn't that affected by all the stuff happening at Leverage, and the effects seemed much more distributed. 

Maybe you also think all the EA Summit and EA Global conferences didn't matter? Which seems like a fine take. Or maybe you think how CEA leadership worked also didn't matter, which also seems fine. But I do think these both aren't obvious takes, and I think I disagree with both of them. 

Given that there is lots of "let's comment on what things about a comment are good and which things are bad" going on in this thread, I will make more explicit a thing that I would have usually left implicit: 

My current sense is that this comment maybe was better to write than no comment, given the dynamics of the situation, but I think the outcome would have been better if you had waited to write your long comment. This comment felt like it kicked up the heat a bunch, and while I think that was better than just leaving things unresponded, my sense is the discussion overall would have gone better if you had just written your longer comment.

Seems great! It's a bit on ice this week, but we've been thinking very actively about changes to the voting system, and so right now is the right time to strike the iron if you want to change the teams opinion on how we should change things, and what we should experiment with.

I liked the effort put into this comment, and found it worth reading, but disagree with it very substantially. I also think I expect it to overall have bad consequences on the discussion, mostly via something like "illusion of transparency" and "trying to force the discussion to happen that you want to happen, and making it hard for people to come in with a different frame", but am not confident. 

I think the first one is sad, and something I expect would be resolved after some more rounds of comments or conversations. I don't actually really know what to do about the second one, like, on a deeper level. I feel like "people wanting to have a different type of discussion than the OP wants to have" is a common problem on LW that causes people to have bad experiences, and I would like to fix it. I have some guesses for fixes, but none that seem super promising. I am also not totally confident it's a huge problem and worth focussing on at the margin.

When counting down we are all savages dancing to the sun gods in a feeble attempt to change the course of history.

More seriously though, yeah, definitely when I count down, I see a ton of stuff that could be a lot better. A lot of important comments missing, not enough courage, not enough honesty, not enough vulnerability, not enough taking responsibility for the big picture.

I am also surprised by this! I think this sentence is kind of true, and am dissatisfied with the threads, but I don't feel like my take is particularly well-summarized with the above language, at least in the context of this post (like, I feel like this sentence implies a particular type of agreement with the OP that I don't think summarizes my current position very well, though I am also not totally confident I disagree with the OP). 

I am in favor of experimenting more with some karma stuff, and have been encouraging people to work on that within the Lightcone team. I think there is lots of stuff we could do better, and definitely comparing us to some ideal that I have in my head, I think things definitely aren't going remotely as well as I would like them to, but I do feel like the word "dissatisfied" seems kind of wrong. I think there are individual comments that seem bad, but overall I think the conversations have been quite good, and I am mildly positively surprised by how well they have been going. 

I like this idea. It has a lot of nice attributes. 

I wrote some in the past about what all the different things are that a voting/karma system on LW is trying to produce, with some thoughts on some proposals that feel a bit similar to this: https://www.lesswrong.com/posts/EQJfdqSaMcJyR5k73/habryka-s-shortform-feed?commentId=8meuqgifXhksp42sg 

We do offer an RSS with all posts, but no emails for all posts, sorry.

We do! Just subscribe to Zvi's post via the "Subscribe" button on their profile, and then set the "Posts by users I am subscribed to" setting in the notification settings to "email" and "immediately": 

That thread (the subset of it that was happening on LW 1.0) was one of the things that convinced me to build LW 2.0 (I was already working on it, but wasn't sure how much I would commit to it). Because that thread was really quite bad, and a lot of it had to do with deep site architecture things that were hard to change.

I did indeed misunderstand that! I didn't downvote, but my misunderstanding did cause me to not upvote. 

Yes, sorry! We realized the opportunity to run this over the winter only a few weeks ago, and have been moving as fast as we can since then. There is a decent chance there will be future cohorts, so if anyone is in this situation, they could still apply for future cohorts.

Thank you so much! This is great!

That is vastly more readership than I had thought. A naive look at these numbers suggests that a small city’s worth of people read Elizabeth’s latest post. But I assume that these numbers can’t be taken at face value.

I think this number can be relatively straightforwardly taken at face value. Elizabeth's post was at the top of HN for a few hours, so a lot of people saw it. A small city's worth seems about right for the number of people who clicked through and at least skimmed it.

I don't think LW gets at all fewer comments than ACX. I think indeed LW has more comments than ACX, it's just that LW comments are spread out over 60+ posts in a given week, whereas ACX has like 2-3 posts a week. LessWrong gets about 150-300 comments a day, which is roughly the same as what ACX gets per day. 

https://www.similarweb.com/website/astralcodexten.substack.com/?competitors=lesswrong.com Currently shows ACX at something like 1.7x of LessWrong. At some points in the past LessWrong was slightly ahead.

LessWrong is a pretty big website. Here is a random snapshot of top-viewed pages from the last month from Google Analytics: 

As you can see from the distribution, it's a long tail of many pages getting a few hundred pageviews each month, which adds up a lot.

I believe that LW must be have a readership two orders of magnitude lower than SSC/ACX (in the thousands, or even just the hundreds, for LW, in the hundreds of thousands for SSC/ACX)

LW's readership is about the same order of magnitude as SSC. Depending on the mood of the HN and SEO gods.

This is a very good point. IIRC Paul is working on some new blog posts that summarize his more up-to-date approach, though I don't know when they'll be done. I will ask Paul when I next run into him about what he thinks might be the best way to update the sequence.

The Babble and Prune sequence might also be relevant here. 

After discussing the matter with some other (non-Leverage) EAs, we've decided to wire $15,000 to Zoe Curzi (within 35 days).

A number of ex-Leveragers seem to be worried about suffering (financial, reputational, etc.) harm if they come forward with information that makes Leverage look bad (and some also seem worried about suffering harm if they come forward with information that makes Leverage look good). This gift to Zoe is an attempt to signal support for people who come forward with accounts like hers, so that people in Zoe's reference class are more inclined to come forward.

We've temporarily set aside $85,000 in case others write up similar accounts -- in particular, accounts where it would be similarly useful to offset the incentives against speaking up. We plan to use our judgment to assess reports on a case-by-case basis, rather than having an official set of criteria. (It's hard to design formal criteria that aren't gameable, and we were a bit wary of potentially setting up an incentive for people to try to make up false bad narratives about organizations, etc.)

Note that my goal isn't to evaluate harms caused by Leverage and try to offset such harms. Instead, it's trying to offset any incentives against sharing risky honest accounts like Zoe's.

Full disclosure: I worked with a number of people from Leverage between 2015 and 2018. I have a pretty complicated, but overall relatively negative view of Leverage (as shown in my comments), though my goal here is to make it less costly for people around Leverage to share important evidence, not to otherwise weigh in on the object-level inquiry into what happened. Also, this comment was co-authored with some EAs who helped get the ball rolling on this, so it probably isn't phrased the way I would have fully phrased it myself.

I think the epistemic state of CEA was some mixture of something pretty close to what you list here, and something that I would put closer to something more like "Leverage maybe is bad, or maybe isn't, but in any case it looks bad, and I don't think I want people to think EA or CEA is bad, so we are going to try to avoid any associations between these entities, which will sometimes require stretching the truth".

Pretty sure you should just be able to copy the structure of the query from the Chrome network tab, and reverse engineer it this way. IIRC the structure was pretty straightforward, and the response pretty well structured.

Geoff was interested in publishing a transcript and a video, so I think Geoff would be happy with you publishing the audio from the recording you have.

Yep, I think CEA has in the past straightforwardly misrepresented (there is a talk on the history of EA by Will and Toby that says some really dubious things here, IIRC) and sometimes even lied in order to not mention Leverage's history with Effective Altruism. I think this was bad, and continues to be bad.

Yep, I think the situation is closer to what Jeff describes here, though, I honestly don't actually know, since people tend to get cagey when the topic comes up.

I would be happy to contribute my part of this, with the memory I have. I think I could cover a decent amount of the questions above, though would also likely get some things wrong, so wouldn't be a totally dependent observer.

drug addicts have or develop very strong preferences for drugs. The assertion that they can't make their own decisions is a declaration of intent to coerce them, or an arrogation of the right to do so.

I really don't think this is an accurate description of what is going on in people's mind when they are experiencing drug dependencies. I've spent a good chunk of my childhood with an alcoholic father, and he would have paid most of his wealth to stop being addicted to drinking, went through great lengths trying to tie himself to various masts to stop, and generally expressed a strong preference for somehow being able to self-modify the addiction away, but ultimately failed to do so. 

Of course, things might be different for different people, but at least in the one case where I have a very large amount of specific data, this seems like it's a pretty bad model of people's preferences. Based on the private notebooks of his that I found after his death, this also seemed to be his position in purely introspective contexts without obvious social desirability biases. My sense is that he would have strongly preferred someone to somehow take control away from him, in this specific domain of his life.

No, Eliezer's comment seems like a straightforward "I am making a non-anonymous upvote" which is indeed a functionality I also sometimes want, since sometimes the identity of the upvoter definitely matters. The comment above seems like it's doing something different, especially in combination with the other comment I linked to.

Outside of "these people probably talked to each other like once every few months" I think there is no major connection between Leverage and the Vassarites that I am aware of.

I noticed I had downvoted this comment, and kind of felt bad about it. I think this is a reasonable suggestion to make, but also think it is a bad suggestion for a variety of reasons. Generally I prefer voting systems to reward comments I think are good to have been made, and punish comments that seem to detract from the conversation, and despite my disagreement with the proposed policy, I do think this comment overall made things better. So I changed my downvote to an upvote, and am now leaving this comment to definitely disambiguate.