LessWrong 2.0 Reader

This proposal increases the influence of the states, in the sense of "how much does it matter that any given person bothered to vote?", but does it increase their preference satisfaction? If the 4 states each conceive of themselves as red or blue states, then each of them will be thinking "under the current system I estimate an X% chance that we'll elect my party's president while under the new system I estimate a Y% chance we'll elect my party's president". If both sides are perfect predictors then one will conclude that Y<X so they should not do the deal. If both sides are imperfect predictors such that they both think Y>X, then the outside view still tells them it's equally likely that they're the sucker here and shouldn't participate.

Were whichever markets you're looking at open at this time? Most stuff doesn't trade that much out of hours.

I mostly think it's too loose a heuristic and that you should dig into more details

What's the most important technical question in AI safety right now?

Finding two bugs in a large codebase doesn't seem especially suspicious to me.

While I agree that people are in general overconfident, including LessWrongers, I don't particularly think this is because Bayesianism is philosophically incorrect, but rather due to both practical limits on computation combined with sometimes not realizing how data-poor their efforts truly are.

(There are philosophical problems with Bayesianism, but not ones that predict very well the current issues of overconfidence in real human reasoning, so I don't see why Bayesianism is so central here. Separately, while I'm not sure there can ever be a complete theory of epistemology, I do think that Bayesianism is actually quite general, and a lot of the principles of Bayesianism is probably implemented in human brains, allowing for practicality concerns like cost of compute.)

We have discussed this dynamic before but just for the record:

I think that if it became industry-standard practice for AGI corporations to write, publish, and regularly update (actual instead of just hypothetical) safety cases at at this level of rigor and detail, my p(doom) would cut in half.

This is IMO not the type of change that should be able to cut someone's P(doom) in half. There are so many different factors that are of this size and importance or bigger (including many that people simply have not thought of yet) such that, if this change could halve your P(doom), then your P(doom) should be oscillating wildly all the time.

I flag this as an example of prioritizing inside-view considerations too strongly in forecasts. I think this is the sort of problem that arises when you "take bayesianism too seriously", which is one of the reasons why I wrote my recent post on why I'm not a bayesian [LW · GW] (and also my earlier post on Knightian uncertainty [LW · GW]).

For context: our previous discussions about this related to Daniel's claim that appointing one specific person to one specific important job could change his P(doom) by double digit percentage points. I similarly think this is not the type of consideration that should be able to swing people's P(doom) that much (except maybe changing the US or Chinese leaders, but we weren't talking about those).

Lastly, since this is a somewhat critical comment, I should flag that I really appreciate and admire Daniel's forecasting, have learned a lot from him, and think he's generally a great guy. The epistemology disagreements just disproportionately bug me.

In my experience, the first step in reconciling conflict is to understand one's own values, before listening to those of others.  There are multiple reasons for this step, but the one relevant to your point is that by reflecting on the tradeoffs that I accept or reject and why, I can feel secure in listening to someone else's point of view.  If their approach addresses my own concerns, then I can recognize it and that dissolves the disagreement.  If it doesn't, then I know enough about what I really want to suggest modifications to their approach that would address my concerns.  Either way, it keeps me safe from value-drift, especially on important principles like ethics.  

Just because someone else has valid concerns doesn't mean I have to give up any of my own, but it doesn't mean we're at an impasse either.  Humans have a habit of turning disagreements into false dichotomies.  When they listen to each other, the conversation becomes, "alright, I understand your concerns, but you understand why mine are more important, right?"  They are so quick to ask other people to sacrifice their values that they don't think of exploring alternative approaches, ones that can change the situation to fulfill the values of all the stakeholders.  That's what I'm working on changing.  

Does that all make sense?  

I also found this hard to parse. I suggest the following edit:

Omega will send you the following message whenever it is true: "Exactly one of the following statements is true: (1) you will not pull the lever (2) the stranger will not pull the lever " You receive the message. Do you pull the lever?

I’ve reread the comment thread and I think I’ve figured out what went wrong here. Starting from a couple posts ago, it looks like you were assuming that the reason I thought you were wrong was that I disagreed with your reasons for believing that people sometimes feel that way, and were trying to offer arguments for that point. I, on the other hand, found it obvious that the issue was that you were privileging the hypothesis, and was confused about why you were arguing the object-level premises of the post, which I hadn’t mentioned; this led me to assume it was a non-sequiter and respond with attempted clarifications of the presumed misunderstanding.
To clarify, I agree that some people view old things negatively. I don’t take issue with the claim that they do; I take issue with the claim that this is the likeliest or only possible explanation. (I do, however, think disagree-voting Anders' comment is a somewhat implausible way for someone to express that feeling, which for me is a reason to downweight the hypothesis.) I think you’re failing to consider sufficient breadth in the hypothesis-space, and in particular the mental move of assuming my disagreement was with the claim that your hypothesis is possible (rather than several steps upstream of that) is one which can make it difficult to model things accurately.