LessWrong 2.0 Reader

Except there's more at play than just winning the election. If you're a voter in a swing state, the candidates are paying more attention to you, and making more promises catering to you. The parties are picking candidates they think will appeal to you. Even if your odds of winning stay the same, the prize for winning gets bigger.

It was exiting a few elections ago when Colorado was in play by both parties. We even got to host the Democratic convention in Denver. Now, they just ignore us.

I really feel like we're talking past each other here, because I have no idea how any of what you said relates to what I said, except the first paragraph.

As for that, what you describe sounds worse than a median researcher problem, instead sounding like a situation ripe for group think instead!

This proposal increases the influence of the states, in the sense of "how much does it matter that any given person bothered to vote?", but does it increase their preference satisfaction? If the 4 states each conceive of themselves as red or blue states, then each of them will be thinking "under the current system I estimate an X% chance that we'll elect my party's president while under the new system I estimate a Y% chance we'll elect my party's president". If both sides are perfect predictors then one will conclude that Y<X so they should not do the deal. If both sides are imperfect predictors such that they both think Y>X, then the outside view still tells them it's equally likely that they're the sucker here and shouldn't participate.

Were whichever markets you're looking at open at this time? Most stuff doesn't trade that much out of hours.

I mostly think it's too loose a heuristic and that you should dig into more details

What's the most important technical question in AI safety right now?

Finding two bugs in a large codebase doesn't seem especially suspicious to me.

While I agree that people are in general overconfident, including LessWrongers, I don't particularly think this is because Bayesianism is philosophically incorrect, but rather due to both practical limits on computation combined with sometimes not realizing how data-poor their efforts truly are.

(There are philosophical problems with Bayesianism, but not ones that predict very well the current issues of overconfidence in real human reasoning, so I don't see why Bayesianism is so central here. Separately, while I'm not sure there can ever be a complete theory of epistemology, I do think that Bayesianism is actually quite general, and a lot of the principles of Bayesianism is probably implemented in human brains, allowing for practicality concerns like cost of compute.)

We have discussed this dynamic before but just for the record:

I think that if it became industry-standard practice for AGI corporations to write, publish, and regularly update (actual instead of just hypothetical) safety cases at at this level of rigor and detail, my p(doom) would cut in half.

This is IMO not the type of change that should be able to cut someone's P(doom) in half. There are so many different factors that are of this size and importance or bigger (including many that people simply have not thought of yet) such that, if this change could halve your P(doom), then your P(doom) should be oscillating wildly all the time.

I flag this as an example of prioritizing inside-view considerations too strongly in forecasts. I think this is the sort of problem that arises when you "take bayesianism too seriously", which is one of the reasons why I wrote my recent post on why I'm not a bayesian [LW · GW] (and also my earlier post on Knightian uncertainty [LW · GW]).

For context: our previous discussions about this related to Daniel's claim that appointing one specific person to one specific important job could change his P(doom) by double digit percentage points. I similarly think this is not the type of consideration that should be able to swing people's P(doom) that much (except maybe changing the US or Chinese leaders, but we weren't talking about those).

Lastly, since this is a somewhat critical comment, I should flag that I really appreciate and admire Daniel's forecasting, have learned a lot from him, and think he's generally a great guy. The epistemology disagreements just disproportionately bug me.

In my experience, the first step in reconciling conflict is to understand one's own values, before listening to those of others.  There are multiple reasons for this step, but the one relevant to your point is that by reflecting on the tradeoffs that I accept or reject and why, I can feel secure in listening to someone else's point of view.  If their approach addresses my own concerns, then I can recognize it and that dissolves the disagreement.  If it doesn't, then I know enough about what I really want to suggest modifications to their approach that would address my concerns.  Either way, it keeps me safe from value-drift, especially on important principles like ethics.  

Just because someone else has valid concerns doesn't mean I have to give up any of my own, but it doesn't mean we're at an impasse either.  Humans have a habit of turning disagreements into false dichotomies.  When they listen to each other, the conversation becomes, "alright, I understand your concerns, but you understand why mine are more important, right?"  They are so quick to ask other people to sacrifice their values that they don't think of exploring alternative approaches, ones that can change the situation to fulfill the values of all the stakeholders.  That's what I'm working on changing.  

Does that all make sense?