Why so much variance in human intelligence? 2019-08-22T22:36:55.499Z · score: 41 (11 votes)
Announcement: Writing Day Today (Thursday) 2019-08-22T04:48:38.086Z · score: 29 (10 votes)
"Can We Survive Technology" by von Neumann 2019-08-18T18:58:54.929Z · score: 32 (9 votes)
A Key Power of the President is to Coordinate the Execution of Existing Concrete Plans 2019-07-16T05:06:50.397Z · score: 105 (31 votes)
Bystander effect false? 2019-07-12T06:30:02.277Z · score: 19 (10 votes)
The Hacker Learns to Trust 2019-06-22T00:27:55.298Z · score: 77 (22 votes)
Welcome to LessWrong! 2019-06-14T19:42:26.128Z · score: 80 (32 votes)
Von Neumann’s critique of automata theory and logic in computer science 2019-05-26T04:14:24.509Z · score: 30 (11 votes)
Ed Boyden on the State of Science 2019-05-13T01:54:37.835Z · score: 64 (16 votes)
Why does category theory exist? 2019-04-25T04:54:46.475Z · score: 35 (7 votes)
Formalising continuous info cascades? [Info-cascade series] 2019-03-13T10:55:46.133Z · score: 17 (4 votes)
How large is the harm from info-cascades? [Info-cascade series] 2019-03-13T10:55:38.872Z · score: 23 (4 votes)
How can we respond to info-cascades? [Info-cascade series] 2019-03-13T10:55:25.685Z · score: 15 (3 votes)
Distribution of info-cascades across fields? [Info-cascade series] 2019-03-13T10:55:17.194Z · score: 15 (3 votes)
Understanding information cascades 2019-03-13T10:55:05.932Z · score: 55 (19 votes)
(notes on) Policy Desiderata for Superintelligent AI: A Vector Field Approach 2019-02-04T22:08:34.337Z · score: 46 (16 votes)
How did academia ensure papers were correct in the early 20th Century? 2018-12-29T23:37:35.789Z · score: 79 (20 votes)
Open and Welcome Thread December 2018 2018-12-04T22:20:53.076Z · score: 28 (10 votes)
The Vulnerable World Hypothesis (by Bostrom) 2018-11-06T20:05:27.496Z · score: 47 (17 votes)
Open Thread November 2018 2018-10-31T03:39:41.480Z · score: 17 (6 votes)
Introducing the AI Alignment Forum (FAQ) 2018-10-29T21:07:54.494Z · score: 88 (31 votes)
Quick Thoughts on Generation and Evaluation of Hypotheses in a Community 2018-09-06T01:01:49.108Z · score: 56 (21 votes)
Psychology Replication Quiz 2018-08-31T18:54:54.411Z · score: 49 (14 votes)
Goodhart Taxonomy: Agreement 2018-07-01T03:50:44.562Z · score: 44 (11 votes)
Benito's Shortform Feed 2018-06-27T00:55:58.219Z · score: 21 (4 votes)
Bounded Rationality: Two Cultures 2018-05-29T03:41:49.527Z · score: 23 (4 votes)
Temporarily Out of Office 2018-05-08T21:59:55.021Z · score: 6 (1 votes)
Brief comment on frontpage/personal distinction 2018-05-01T18:53:19.250Z · score: 63 (14 votes)
Form Your Own Opinions 2018-04-28T19:50:18.321Z · score: 62 (16 votes)
Community Page Mini-Guide 2018-04-24T15:04:53.641Z · score: 16 (3 votes)
Hold On To The Curiosity 2018-04-23T07:32:01.960Z · score: 97 (30 votes)
LW Update 04/06/18 – QM Sequence Updated 2018-04-06T08:53:45.560Z · score: 37 (9 votes)
Why Karma 2.0? (A Kabbalistic Explanation) 2018-04-02T20:43:02.032Z · score: 28 (9 votes)
A Sketch of Good Communication 2018-03-31T22:48:59.652Z · score: 151 (51 votes)
LessWrong Launch Party 2018-03-22T02:19:14.468Z · score: 18 (3 votes)
The Costly Coordination Mechanism of Common Knowledge 2018-03-15T20:20:41.566Z · score: 183 (58 votes)
The Building Blocks of Interpretability 2018-03-14T20:42:30.674Z · score: 23 (4 votes)
Editor Mini-Guide 2018-03-11T20:58:59.828Z · score: 45 (12 votes)
Welcome to Oxford University Rationality Community 2018-03-11T20:17:46.086Z · score: 13 (4 votes)
Moderation List (warnings and bans) 2018-03-06T19:18:44.226Z · score: 34 (7 votes)
Extended Quote on the Institution of Academia 2018-03-01T02:58:11.159Z · score: 128 (42 votes)
A model I use when making plans to reduce AI x-risk 2018-01-19T00:21:45.460Z · score: 136 (48 votes)
Field-Building and Deep Models 2018-01-13T21:16:14.523Z · score: 55 (18 votes)
12/31/17 Update: Frontpage Redesign 2018-01-01T03:21:11.408Z · score: 16 (4 votes)
Against Love Languages 2017-12-29T09:51:29.609Z · score: 27 (12 votes)
Comments on Power Law Distribution of Individual Impact 2017-12-29T01:49:26.791Z · score: 54 (15 votes)
Comment on SSC's Review of Inadequate Equilibria 2017-12-01T11:46:13.688Z · score: 30 (9 votes)
Bet Payoff 1: OpenPhil/MIRI Grant Increase 2017-11-09T18:31:06.034Z · score: 40 (12 votes)
Brief comment on featured 2017-10-29T17:12:51.320Z · score: 33 (12 votes)
Maths requires less magical ability than advertised. Also, messy details are messy (and normal). 2017-10-27T03:11:50.773Z · score: 16 (5 votes)


Comment by benito on Announcement: Writing Day Today (Thursday) · 2019-08-22T23:46:38.472Z · score: 8 (4 votes) · LW · GW

This is true in most cases, like when we did the Alignment Forum sequences we released them over a longer schedule.

The goal of the day at the event I'm at is for the participants (who mostly are not used to writing blogposts on the internet) to get comfortable with / practise the whole experience of writing, publishing, and getting feedback (in person too!). There isn't enough time left in MSFP to have them write on multiple days. And more importantly, I think for practising getting comfortable publishing (for which many folks aren't) inserting a delay until after the event for it to be published can make it much less good, like delaying for a week on sending an email you're worried about sending.

I think the roundups will mostly mitigate the costs of having lots of posts on one day, but agree it would be better if somehow they were distributed on multiple days.

Comment by benito on Chris_Leong's Shortform · 2019-08-21T23:59:16.146Z · score: 4 (2 votes) · LW · GW

“I’m sorry, I didn’t have the time to write you a short email, so I wrote you a long one instead.”

Comment by benito on Davis_Kingsley's Shortform · 2019-08-21T18:35:57.168Z · score: 7 (3 votes) · LW · GW

My model of what causes maybe most of the drama in the Berkeley rationality community is that it is a professional network superimposed upon a social graph, without much/any explicit optimisation for separating the conflicts that such things would arise. Friendships and romantic relationships optimise for certain things, professional relationships optimise for others, and status is often a fairly simple collapsing of those metrics into one dimension, which means that professional and personal status interact a lot. It's like if people at your company are only able to date other people who worked at your own company.

Poly just massively increases the number of internal relationships that are subject to this effect (assume # of internal relationships is directly proportional to drama). I don't expect poly to be nearly as problematic in communities that don't have this property. There are still arguments against being poly, but I expect your "I empirically observe massive drama" to not be a strong reason elsewhere.

Comment by Benito on [deleted post] 2019-08-21T01:11:20.758Z

I feel like you think this is a transgressive boundary or otherwise surprising, whereas I think it mostly seems fine. I'll write some thoughts regardless.

You write fairly broadly here. I know (broadly) what you're talking about, and I'm sad about it too, like when I'm regularly in spaces (e.g. CFAR) where I would've expected to have common knowledge that everyone's read the sequences, understands the concept of a technical explanation of a technical explanation, how many bits of information it takes to justify a belief, etc, but I don't. I don't feel like "the epistemics are failing" is the coarse-grained description I'd use, I think there's more details about which bits are going on and why (and which bits actually seem to be going quite excellently!), but I wanted to agree with feeling sad about this particular bit.

many of these ideas have essentially never been justified using the paradigm that the community already operates in

I am not sure whether you feel this way when reading LessWrong though. If you scroll through the curated posts of the last few months, I don't expect it seems mostly like a lot of obviously terrible ideas are being treated unsceptically (thought you're welcome to surprise me and say it seems just as bad!).

(A few counterexamples on LessWrong: Oliver wrote an attempted translation for chakras the other day. Kaj's most popular post (277 karma!) was an attempt to explain a bunch of enlightenment/meditation stuff in non-mysterious term and has a whole interesting sequence offering explicit models behind things like Internal Family Systems. After Scott began a discussion of cultural evolution, Vaniver wrote a post I found fascinating Steelmanning Divination. I wrote in pretty explicit language about my experience circling here. Zvi has written in a 'postmodernist beat-poem' style about things that are out to get you and why choices are bad, but also tries to give simple, microeconomic explanations for how systems (like blackmail and facebook) can optimise to destroy all value. Back on the cultural evolution frame, Zvi and Ben have both elucidated explicit models for why the Sabbath is an important institution one should respect.)

(Not to mention the great obvious straightforward rationality writing, like Abram's recent Mistakes With Conservation of Expected Evidence and loads more.)

So when you talk about weird/mysterious ideas not being explained in an explicit and clear epistemology, I do want to say I think people on LessWrong are often making that effort, and I think we've tried to signal that we have higher standards here. It's okay to write poetry and so on when you've not yet learned how to make your idea explicit, but the goal is a technical understanding, that comes with an explicit, communicable model.

Comment by benito on Chapter 21: Rationalization · 2019-08-20T23:57:38.405Z · score: 2 (1 votes) · LW · GW

You're totally right! Thanks, fixed.

Comment by benito on A misconception about immigration · 2019-08-20T18:15:09.182Z · score: 4 (2 votes) · LW · GW
In one scenario, the political leaders of the two towns decide to unite the two towns. Nothing except the name will change. So, there is now a town C that has 1010 inhabitants. Will town C have a functional economy? Of course! After all, all that was done was subsuming two already functional economies.

I feel a bit wary of using this as an axiom for further thinking about immigration between countries. I've not really thought about this issue before, but my guess is if two countries with very different levels of economic development open their borders to each other this could be fairly damaging. It might substantially increase prices and demand of land in the less economically developed country, causing upheaval (e.g. poorer people in the more developed country buying out middle class people in the less developed country); it may damage a lot of their industries where the more economically developed country has more efficient abilities. I also think, especially if there's big cultural and language barriers that a lot of basic norms for interacting with strangers and such can be very confused and cause damage in the years/decades it takes to integrate.

I don't have much of an opinion here, just thought I'd mention that I got off the bus a bit at this part of the argument.

Comment by benito on Swimmer963's Shortform · 2019-08-18T18:46:15.496Z · score: 2 (1 votes) · LW · GW

(Or, let's be honest, my biggest self-indulgence in writing: gratuitous medical emergencies)

There’s something really funny about the phrase “gratuitous medical emergencies” :)

Comment by benito on Buck's Shortform · 2019-08-18T18:08:10.340Z · score: 2 (1 votes) · LW · GW

This sounds like a really fun thing I can do at weekends / in the mornings. I’ll try it out and report back sometime.

Comment by benito on Benito's Shortform Feed · 2019-08-18T17:50:28.248Z · score: 4 (2 votes) · LW · GW

I think you’re missing me with this. I’m not very distractable and I don’t need to learn to be okay with leisure time. I’m trying to actually have hobbies, and realising that is going to take work.

I could take up meditation as a hobby, but at the minute I want things that are more social and physical.

Comment by benito on Benito's Shortform Feed · 2019-08-18T17:45:45.000Z · score: 2 (1 votes) · LW · GW

Either way I’ll still only check it in a 2 hour window on Saturdays, so I feel safe trying it out.

Comment by benito on Benito's Shortform Feed · 2019-08-18T03:01:24.567Z · score: 2 (1 votes) · LW · GW

Huh, 95% is quite extreme. But I realise this probably also solves the problem whereby if the people I'm interested in comment on *someone else's* wall, I still get to see it. I'll try this out next week, thx.

(I don't get to be confident I've seen 100% of all the interesting people's good content though, the news feed is fickle and not exhaustive.)

Comment by benito on Benito's Shortform Feed · 2019-08-18T00:13:21.253Z · score: 9 (4 votes) · LW · GW

I've finally moved into a period of my life where I can set guardrails around my slack without sacrificing the things I care about most. I currently am pushing it to the limit, doing work during work hours, and not doing work outside work hours. I'm eating very regularly, 9am, 2pm, 7pm. I'm going to sleep around 9-10, and getting up early. I have time to pick up my hobby of classical music.

At the same time, I'm also restricting the ability of my phone to steal my attention. All social media is blocked except for 2 hours on Saturday, which is going quite well. I've found Tristan Harris's advice immensely useful - my phone is increasingly not something that I give all of my free attention to, but instead something I give deliberate attention and then stop using. Tasks, not scrolling.

Now I have weekends and mornings though, and I'm not sure what to do with myself. I am looking to get excited about something, instead of sitting, passively listening to a comedy podcast while playing a game on my phone. But I realise I don't have easy alternative options - Netflix is really accessible. I suppose one of the things that a Sabbath is supposed to be is an alarm, showing that something is up, and at the minute I've not got enough things I want to do for leisure that don't also feel a bit like work.

So I'm making lists of things I might like (cooking, reading, improv, etc) and I'll try those.

Comment by benito on Benito's Shortform Feed · 2019-08-17T03:32:16.193Z · score: 2 (1 votes) · LW · GW

Yeah, it’s what I do with Twitter, and I’ll probably start this with FB. Won’t show me all their interesting convo on other people’s walls though. On a Twitter I can see all their replies, not on FB.

Comment by benito on Benito's Shortform Feed · 2019-08-17T03:05:12.702Z · score: 12 (7 votes) · LW · GW

I block all the big social networks from my phone and laptop, except for 2 hours on Saturday, and I noticed that when I check Facebook on Saturday, the notifications are always boring and not something I care about. Then I scroll through the newsfeed for a bit and it quickly becomes all boring too.

And I was surprised. Could it be that, all the hype and narrative aside, I actually just wasn’t interested in what was happening on Facebook? That I could remove it from my life and just not really be missing anything?

On my walk home from work today I realised that this wasn’t the case. Facebook has interesting posts I want to follow, but they’re not in my notifications. They’re sparsely distributed in my newsfeed, such that they appear a few times per week, randomly. I can get a lot of value from Facebook, but not by checking once per week - only by checking it all the time. That’s how the game is played.

Anyway, I am not trading all of my attention away for such small amounts of value. So it remains blocked.

Comment by benito on Power Buys You Distance From The Crime · 2019-08-16T19:03:09.456Z · score: 26 (9 votes) · LW · GW

Thank you, this comment helped me understand your position quite a bit. You're right, discussing conflict theories are not inherently costly, it's that they're often costly because powerful optimization pressures are punishing discussion of them.

I strongly agree with you here:

I am advocating a conflict theory, rather than a mistake theory, for why discussions of conflict can be bad. I think, if you consider conflict vs mistake theories, you will find that a conflict theory makes better predictions for what sorts of errors people make in the course of discussing conflict, than a mistake theory does.

This is also a large part of my model of why discussions of conflict often go bad - power struggles are being enacted out through (and systematically distorting the use of) language and reasoning.

(I am quite tempted to add that even in a room with mostly scribes, given the incentive on actors to pretend to be scribes, can make it very hard for a scribe to figure out whether someone is a scribe or an actor, and this information asymmetry can lead to scribes distrusting all attempts to discuss conflict theories and reading such discussions as political coordination.

Yet I notice that I pretty reflexively looked for a mistake theory there, and my model of you suggested to me the hypothesis that I am much less comfortable with conflict theories than mistake theories. I guess I'll look out for this further in my thinking, and consider whether it's false. Perhaps, in this case, it is way easier than I'm suggesting for scribes to recognise each other, and the truth is we just have very few scribes.)

The next question is under what norms, incentives and cultures can one have discussions of conflict theories where people are playing the role of Scribe, and where that is common knowledge. I'm not sure we agree on the answer to that question, or what the current norms in this area should be. I'm working on a longer answer, maybe post-length, to Zach's comment below, so I'll see if I can present my thoughts on that.

Comment by benito on Slack for your belief system · 2019-08-14T21:47:17.620Z · score: 2 (1 votes) · LW · GW

The opening Feyeraband quote is sounds very similar to (Scott's review of) Kuhn's Structure of Scientific Revolutions. Related: Jacob's post on the copernican revolution from the inside.

Comment by benito on avturchin's Shortform · 2019-08-13T20:38:42.003Z · score: 3 (2 votes) · LW · GW

Here's his wikipedia page.

Comment by benito on Power Buys You Distance From The Crime · 2019-08-12T16:47:49.739Z · score: 2 (1 votes) · LW · GW

Nice :-)

Comment by benito on Power Buys You Distance From The Crime · 2019-08-11T18:09:25.136Z · score: 15 (7 votes) · LW · GW

I’m hearing you say “Politics is not the mind-killer, talking inaccurately and carelessly about politics is the mind-killer! If we all just say true things and don’t try to grab attention with misleading headlines then we’ll definitely just have a great and net positive conversation and nobody will feel needlessly threatened or attacked”. I feel like you are aware of how toxic things like bravery debates are, and I expect you agree they’d be toxic even if everyone tried very hard to only say true things. I’m confused.

I’m saying it always bears a cost, and a high one, but not a cost that cannot be overcome. I think that the cost is different in different communities, and this depends on the incentives, norms and culture in those communities, and you can build spaces where a lot of good discussion can happen with low cost.

You’re right that Hanson feels to me pretty different than my other examples, in that I don’t feel like marginal overcoming bias blogposts are paying a cost. I suspect this might have to do with the fact that Hanson has sent a lot of very costly signals that he is not fighting a side but is just trying to be an interested scientist. But I’m not sure why I feel differently in this case.

Comment by benito on Power Buys You Distance From The Crime · 2019-08-11T01:15:48.291Z · score: 28 (10 votes) · LW · GW

Sorry for the delay, a lot has happened in the last week.

Let me point to where I disagree with you.

Holding a conflict theory about any particular situation is still a mind-killer, at least to the extent that we're talking about conflict theory in the form of "bad thing happens because of this bad person" as opposed to "this person's incentives are misaligned".

My sense is you are underestimating the cost of not being able to use conflict theories. Here are some examples, where I feel like prohibiting me from even considering that a bad thing happened because a person was bad will severely limit my ability to think and talk freely about what is actually happening.

There's something very valuable that you're pointing at, and I agree with a lot of it. There shouldn't be conflict theories in a math journal. It's plausible to me there shouldn't be conflict theories in an economics journal. And it's plausible to me that the goal should be for the frontpage of LessWrong to be safe from them too, because they do bring major costs in terms of mindkilling nature, and furthermore because several of the above are bullet points are simply off-topic for LessWrong. We're not here to discuss current-day tribal politics in various institutions, industries and communities.

And if I were writing publicly about any of the above topics, I would heavily avoid bringing conflict theories - and have in the past re-written whole essays to be making only object-level points about a topic rather than attacking a particular person's position, because I felt the way I had written it would come across as a bias-argument / conflict theory and destroy my ability to really dialogue with people who disagreed with me. Rather than calling them biased or self-interested, I prefer to use the most powerful of rebuttals in the pursuit of truth, which is showing that they're wrong.

But ruling it out wholly in one's discourse and life seems way too much. I think there are cases where wholly censoring conflict theories will be far more cost than it's worth, and that removing them entirely from your discourse will cripple you and allow you to be taken over by outside forces that want your resources.

For example, I can imagine a relatively straightforward implementation of "no conflict theories" in a nearby world meaning that I am not able to say that study after study is suspect, or that a position is being pushed by political actors, unless I first reinvent mechanism theory and a bunch of microeconomics and a large amount of technical language to discuss bias. If I assume the worst about all of the above bullet points, not being able to talk about bad people causing bad things could mean we are forced to believe lots of false study results and ignore a new theory of fundamental physics, plus silence economists, bloggers, and public intellectuals.

The Hanson examples above feel the strongest to me because it’s the one that's a central example of something that's able to lead to a universal, deep insight about reality and be a central part of LessWrong's mission in understanding human rationality, whereas the others are mostly about current tribal politics. But I think they all substantially affect how much to trust our info sources.

My current sense is that I should think of posing conflict theories as a highly constrained, limited communal resource, and that while spending it will often cause conflict and people to be mind-killed, a rule that says one can never use that resource will mean that when that resource is truly necessary, it won’t be available.



I re-read the OP, and realise I actually identify a lot with your initial comment, and that I gave Elizabeth similar feedback when I read an earlier draft of hers a month ago. The wording of the OP crosses a few of my personal lines such that I would not publish it. And it's actually surprisingly accurate to say that the key thing I'd be doing if I were editing the OP would be turning it from things that had a hint of being like a conflict theory (aren't people with power bad!) to things that felt like a mistake theory (here's an interesting mechanism where you might mistakenly allocate responsibility). Conflict theories tends to explode and eat up communal resources in communities and on the internet generally, and are a limited (though necessary) resource that I want to use with great caution.

And if I were writing publicly about any topics where I had conflict theories, I would heavily avoid bringing conflict theories - and have in the past re-written whole essays to be making only object-level points about a topic rather than attacking a particular person's position, because I felt the way I had written it would come across as a bias-argument / conflict theory and destroy my ability to really dialogue with people who disagreed with me. When I get really irritated with someone's position and have a conflict theory about the source of the disagreement, I still write mistake-theory posts like this, a post with no mention of the original source of motivation.

I think that one of the things that's most prominent to me on the current margin is that I feel like there are massive blockers on public discourse, stopping people from saying or writing anything, and I have a model whereby telling people who write things like the OP to do more work to make it all definitely mistake theory (which is indeed a standard I hold myself to) will not improve the current public discourse, but on the current margin simply stop public discourse. I feel similarly about Jessicata's post on AI timelines, where it is likely to me that the main outcome has been quite positive - even though I think I disagree with each of the three arguments in the post and its conclusion - because the current alternative is almost literally zero public conversation about plans for long AI timelines. I already am noticing personal benefits from the discourse on the subject.

In the first half of this comment I kept arguing against the position "We should ban all conflict theories" rather than "Conflict theories are the mind-killer" which are two very different claims and only one of which you've been making. Right now I want to defend people's ability to write down their thoughts in public, and I think the OP is strongly worth publishing in the situation we're in. I could imagine a world where there was loads of great discussion of topics like what the OP is about, where the OP stands out as not having met a higher standard of effort to avoid mind-killing anyone that the other posts have, where I'd go "this is unnecessarily likely to make people feel defensive and like there's subtle tribal politics underpinning its conclusions, consider these changes?" but right now I'm very pro "Cool idea, let me share my thoughts on the subject too."

(Some background: The OP was discussed about 2 weeks ago on Elizabeth's FB wall, and in it someone else was proposing a different reason why this post needed re-writing for PR reasons, and there I argued already that they shouldn't put such high bars to writing things on people. I think that person’s specific suggestion, if taken seriously, would be incredibly harmful to public discourse regardless of its current health, whereas in this case I think your literal claims are just right. Regardless, I am strongly pro the post and others like it being published.)

Comment by benito on Verification and Transparency · 2019-08-11T00:08:44.848Z · score: 2 (1 votes) · LW · GW

The idea I took away here is that verification and transparency are complementary, in that increasing transparency makes verification easier, and that stronger verification power removes the need for more transparency.

Comment by benito on Which of these five AI alignment research projects ideas are no good? · 2019-08-10T22:52:05.347Z · score: 3 (2 votes) · LW · GW

Note that if people do only give upvotes, then you can hover over a comment’s score to see the total number of votes on it, which is what you’re looking for here.

Comment by benito on Is there a source/market for LW-related t-shirts? · 2019-08-08T17:42:39.989Z · score: 6 (4 votes) · LW · GW

I have a t-shirt that says “World domination is such an ugly phrase, I prefer world Optimization” with a really cool design, and a separate t-shirt that says “Optimise everything” with a picture of a galaxy, and use to have a third t-shirt that says “Probably a member of the Bayesian Conspiracy”. I think Ray also has a t-shirt that just says “Belief As Attire” which I think is excellent.

I got my t-shirts from the people who made, but I think it’s shut down now.

We’re planning to make LW t-shirts at LW (have tried some already that I don’t like).

Comment by benito on Benito's Shortform Feed · 2019-08-08T17:38:08.084Z · score: 4 (2 votes) · LW · GW

Strong agree with Jacob.

Comment by benito on Benito's Shortform Feed · 2019-08-07T20:42:47.057Z · score: 10 (4 votes) · LW · GW

Related hypothesis: people feel like they've wasted some period of time e.g. months, years, 'their youth', when they feel they cannot see an exciting path forward for the future. Often this is caused by people they respect (/who have more status than them) telling them they're only allowed a small few types of futures.

Comment by benito on Benito's Shortform Feed · 2019-08-07T19:26:32.414Z · score: 4 (2 votes) · LW · GW


"Being a good person standing next to the development of dangerous tech makes the tech less dangerous."
Comment by benito on Benito's Shortform Feed · 2019-08-07T14:43:33.926Z · score: 12 (4 votes) · LW · GW

To complement that: Requiring my interlocutor to make everything explicit is also a defence against having my mind changed in ways I don't endorse but that I can't quite pick apart right now. Which kinda overlaps with your example, I think.

I sometimes will feel like my low-level associations are changing in a way I'm not sure I endorse, halt, and ask for something that the more explicit part of me reflectively endorses. If they're able to provide that, then I will willingly continue making the low-level updates, but if they can't then there's a bit of an impasse, at which point I will just start trying to communicate emotionally what feels off about it (e.g. in your example I could imagine saying "I feel some panic in my shoulders and a sense that you're trying to control my decisions"). Actually, sometimes I will just give the emotional info first. There's a lot of contextual details that lead me to figure out which one I do.

Comment by benito on Power Buys You Distance From The Crime · 2019-08-04T10:38:33.975Z · score: 32 (8 votes) · LW · GW

I think it's a genuinely difficult problem to draw the boundary between a conflict and a mistake theory, in no small part due to the difficulties in drawing the boundary between lies and unconscious biases (which I rambled a bit about here). You can also see the discussion on No, it's not The Incentives - it's you as a disagreement over where this boundary should be.

That said, one thing I'll point out is that explaining Calhoun and Buchanan's use of public choice theory as entirely a rationalisation for their political goals, is a conflict theory. It's saying that them bringing public choice theory into the conversation was not a good faith attempt to convey how they see the world, but obfuscation in favour of their political side winning. And more broadly saying that public choice theory is racist is a theory that says the reason it is brought up in general is not due to people having differing understandings of economics, but due to people having different political goals and trying to win.

I find for myself that thinking 'conflict theorists' is a single coherent group is confusing me, and that I should instead replace the symbol with the substance when I'm tempted to use it, because there are many types of people who sometimes use conflict theories, and it is confusing to lump them in with people who always use them, because they often have different reasons for using them when they do.

To give one example of people who always use it: there are certain people who have for most of their lives found that the main determinant of outcomes for them is political conflict by people above them, who are only really able to understand the world using theories of conflict. They've also never gained a real understanding of any of the fascinating and useful different explanations for how social reality works (example, example), or a sense that you often can expand massively rather than fight over existing resources. And when they're looking at someone bringing in public choice theory to argue one side of a social fight, they get an impression that the person is finding clever arguments for their position, rather than being honest.

(This is a mistake theory of why some people primarily reason using conflict theories. There are conflict theories that explain it as well.)

I think it's good to be able to describe what such people are doing, and what experiences have lead them to that outlook on life. But I also think that there are many reasons for holding a conflict theory about a situation, and these people are not at all the only examples of people who use such theories regularly.

Added: clone of saturn’s 3 point explanation seems right to me.

Comment by benito on Open & Welcome Thread August 2019 · 2019-08-03T20:11:26.222Z · score: 4 (2 votes) · LW · GW

Strong upvote for the quality of the explanation of this idea.

Comment by benito on Power Buys You Distance From The Crime · 2019-08-03T20:00:03.067Z · score: 70 (19 votes) · LW · GW

I changed my mind about conflict/mistake theory recently, after thinking about Scott's comments on Zvi's post. I previously thought that people were either conflict theorists or mistake theorists. But I now do not use it to label people, but instead to label individual theories.

To point to a very public example, I don't think Sam Harris is a conflict theorist or a mistake theorist, but instead uses different theories to explain different disagreements. I think Sam Harris views any disagreements with people like Stephen Pinker or Daniel Dennett as primarily them making reasoning mistakes, or otherwise failing to notice strong arguments against their position. And I think that Sam Harris views his disagreements with people like <quickly googles Sam Harris controversies> Glenn Greenwald and Ezra Klein as primarily them attacking him for pushing different goals to their tribes.

I previously felt some not-insubstantial pull to pick sides in the conflict vs mistake theorist tribes, but I don't actually think this is a helpful way of talking, not least because I think that sometimes I will build a mistake theory for why a project failed, and sometimes I will build a conflict theory.

To push back on this part:

Models like "arguments are soldiers" or "our enemies are evil" are the core of Yudkowsky's original argument for viewing politics as a mind-killer. But these sort of models are essentially synonymous with conflict theory; if we could somehow have a tribalistic or political discussion without those conflict-theoretic elements, I'd expect it wouldn't be so mindkiller-ish.

"Arguments are soldiers" and "our enemies are evil" are not imaginary phenomena, they exist and people use such ideas regularly, and it's important that I don't prevent myself from describing reality accurately when this happens. I should be able to use a conflict theory.

I have a model of a common type of disagreement where people get angry at someone walking in with a mistake theory that goes like this: Alice has some power over Bob, and kinda self-deceives themselves into a situation where it's right for them to take resources from Bob, and as Bob is getting angry at Alice and tries to form a small political force to punish Alice, then Charlie comes along and is like "No you don't understand, Alice just made an error of reasoning and if I explain this to them they won't make that mistake again!" and Bob gets really angry at Charlie and thinks they're maybe trying to secretly help Alice or else are strikingly oblivious / conflict averse to an unhealthy degree. (Note this is a mistake theory about the disagreement between Bob and Charlie, and a conflict theory about the disagreement between Bob and Alice. And Charlie is wrong to use a mistake theory.)

I think the reason I'm tempted to split mistake and conflict into tribes, is because I do know people that largely fit into one or the other. I knew people at school who always viewed interpersonal conflict as emanating from tribal self-interest, and would view my attempt to show a solution that didn't require someone being at fault as me trying to make them submit to some kinda weird technicality, and got justifiably irritated. I also know people who are very conflict averse but also have an understanding of the complexity of reality, and so always assume it is merely a principal-agent problem or information flow problem, as opposed to going "Yeah, Alice is just acting out of self-interest here, we need to let her know that's not okay, and let's not obfuscate this unnecessarily." But I think the goal is to have one's beliefs correspond to reality - to use a conflict theory when that's true, a mistake theory when that's true, and not pre-commit to one side or the other regardless of how reality actually is.

I do think that conflict theories are often pretty derailing to bring up when trying to have a meaningful 1-1 public debate, and that it's good to think carefully about specific norms for how to do such a thing. I do think that straight-up banning them is likely the wrong move though. Well, I think that there are many places where they have no place, such as a math journal. However the mathematical community will need a place to be able to discuss internal politics + norm-violations where these can be raised.

Comment by benito on What woo to read? · 2019-08-01T17:50:52.120Z · score: 9 (4 votes) · LW · GW

I failed to really get into the dialogue first time around, thanks a lot for the summary.

Comment by benito on Forum participation as a research strategy · 2019-07-31T10:34:02.603Z · score: 6 (3 votes) · LW · GW

Which is 100% how I initially read your comment, but if you look at your comment it actually says you were both con :-)

Comment by benito on Does it become easier, or harder, for the world to coordinate around not building AGI as time goes on? · 2019-07-30T17:50:26.315Z · score: 6 (4 votes) · LW · GW

Thanks. I'll further add Paul's post What Failure Looks Like, and say that the Alignment Forum sequences raise a lot more specific technical concerns.

Comment by benito on Does it become easier, or harder, for the world to coordinate around not building AGI as time goes on? · 2019-07-30T17:27:56.820Z · score: 8 (5 votes) · LW · GW

So where can I find a concise, exhaustive list of all sound arguments pro and con AGI being likely an existential risk?

Nick Bostrom’s book ‘Superintelligence’ is the standard reference here. I also find the AI FOOM Debate especially enlightening, which hits a lot of the same points. Both you can find easily using google.

But I suppose that most people talking about AGI risk have not enough knowledge about what technically constitute an AGI.

I agree most people who talk about it are not experts in mathematics, computer science, or the field of ML, but the smaller set of people that I trust often are, such as researchers at UC Berkeley (Stuart Russell, Andrew Critch, many more), OpenAI (Paul Christiano, Chris Olah, many more), DeepMind (Jan Leike, Vika Krakovna, many more), MIRI, FHI, and so on. And of course just being an expert in a related technical domain does not make you an expert in long-term forecasting or even AGI, of which there are plausibly zero people with deep understanding.

And in this community Eliezer has talked often about actually solving the hard problem of AGI, not bouncing off and solving something easier and nearby, in part here but also in other places I’m having a hard time linking right now.

Comment by benito on Benito's Shortform Feed · 2019-07-23T10:04:16.077Z · score: 3 (2 votes) · LW · GW

Yeah good point I may have reinvented the wheel. I have a sense that’s not true but need to think more.

Comment by benito on When does adding more people reliably make a system better? · 2019-07-19T14:15:46.046Z · score: 2 (1 votes) · LW · GW

Thanks, this was really interesting.

Comment by benito on When does adding more people reliably make a system better? · 2019-07-19T13:29:10.827Z · score: 4 (3 votes) · LW · GW

Ah, so they're very grateful that he made billions of dollars with their money, but that it was through a process that had a massive amount of risk, and they just weren't interested in risking that much again in the future.

I still think that they'd be interested in risking a smaller fraction of their money (e.g. give him 10-20% of what they gave last time and then invest the rest elsewhere). I don't get the 'frozen' out part.

Comment by benito on Benito's Shortform Feed · 2019-07-19T12:30:59.544Z · score: 7 (4 votes) · LW · GW

I think of myself as pretty skilled and nuanced at introspection, and being able to make my implicit cognition explicit.

However, there is one fact about me that makes me doubt this severely, which is that I have never ever ever noticed any effect from taking caffeine.

I've never drunk coffee, though in the past two years my housemates have kept a lot of caffeine around in the form of energy drinks, and I drink them for the taste. I'll drink them any time of the day (9pm is fine). At some point someone seemed shocked that I was about to drink one after 4pm, and I felt like I should feel bad or something, so I stopped. I've not been aware of any effects.

But two days ago, I finally noticed. I had to do some incredibly important drudge work, and I had two red bulls around 12-2pm. I finished work at 10pm. I realised that while I had not felt weird in any way, I had also not had any of the normal effects of hanging around for hours, which is getting tired, distracted, needing to walk around, wanting to do something different. I had a normal day for 10 hours solely doing crappy things I normally hate.

So I guess now I see the effect of caffeine: it's not a positive effect, it just removes the normal negative effects of the day. (Which is awesome.)

Comment by benito on When does adding more people reliably make a system better? · 2019-07-19T08:30:02.517Z · score: 2 (1 votes) · LW · GW

Do you know why? I’d expect that investors who were angry for a few years would come round after he made them billions of dollars.

Comment by benito on What's up with self-esteem? · 2019-07-19T08:18:59.686Z · score: 12 (3 votes) · LW · GW

This makes a lot of sense to me. It fits in with my sense that:

  • people with low self-esteem don’t expect others to help/support them long-term, as they feel they’ll be ‘found out’ as being worthless/terrible
  • whenever anyone is helpful to them they respond disproportionately effusively and say how kind the other person is.

In future I will model surprising low self-esteem as failing to accurately read signals about their level of respect/power. And that people with appropriately low levels of low self-esteem should focus on being useful to the people and communities around them.

Comment by benito on Benito's Shortform Feed · 2019-07-18T20:16:44.498Z · score: 20 (5 votes) · LW · GW

I will actually clean this up and into a post sometime soon [edit: I retract that, I am not able to make commitments like this right now]. For now let me add another quick hypothesis on this topic whilst crashing from jet lag.

A friend of mine proposed that instead of saying 'lies' I could say 'falsehoods'. Not "that claim is a lie" but "that claim is false".

I responded that 'falsehood' doesn't capture the fact that you should expect systematic deviations from the truth. I'm not saying this particular parapsychology claim is false. I'm saying it is false in a way where you should no longer trust the other claims, and expect they've been optimised to be persuasive.

They gave another proposal, which is to say instead of "they're lying" to say "they're not truth-tracking". Suggest that their reasoning process (perhaps in one particular domain) does not track truth.

I responded that while this was better, it still seems to me that people won't have an informal understanding of how to use this information. (Are you saying that the ideas aren't especially well-evidenced? But they sound pretty plausible to me, so let's keep discussing them and look for more evidence.) There's a thing where if you say someone is a liar, not only do you not trust them, but you recognise that you shouldn't even privilege the hypotheses that they produce. If there's no strong evidence either way, if it turns out the person who told it you is a rotten liar, then if you wouldn't have considered it before they raised it, don't consider it now.

Then I realised Jacob had written about this topic a few months back. People talk as though 'responding to economic incentives' requires conscious motivation, but actually there are lots of ways that incentives cause things to happen that don't require humans consciously noticing the incentives and deliberately changing their behaviour. Selection effects, reinforcement learning, and memetic evolution.

Similarly, what I'm looking for is basic terminology for pointing to processes that systematically produce persuasive things that aren't true, that doesn't move through "this person is consciously deceiving me". The scientists pushing adult neurogenesis aren't lying. There's a different force happening here that we need to learn to give epistemic weight to the same way we treat a liar, but without expecting conscious motivation to be the root of the force and thus trying to treat it that way (e.g. by social punishment).

More broadly, it seems like there are persuasive systems in the environment that weren't in the evolutionary environment for adaptation, that we have not collectively learned to model clearly. Perhaps we should invest in some basic terminology that points to these systems so we can learn to not-trust them without bringing in social punishment norms.

Comment by benito on Integrity and accountability are core parts of rationality · 2019-07-17T09:12:14.867Z · score: 2 (1 votes) · LW · GW

If I think about preserving system integrity over time, or an agent preserving goal integrity over time, I think of the ability for the system/agent to move through a wide variety of environments without being broken / fundamentally changed in ways it doesn't want by outside forces. This conceptualisation of integrity - being able to preserve the core parts of you and your goals over time - seems good to me. (Reminds me of Ray / Critch talking about being a robust agent.) Someone with integrity is wholly them and will stay whole over the long run, even if crazy things are thrown at them. It's not a claim about their competences/goals/beliefs now, it's a claim about the long-term integrity of their competences/goals/beliefs.

Comment by benito on What are we predicting for Neuralink event? · 2019-07-17T08:12:01.361Z · score: 6 (3 votes) · LW · GW

For those of us who block most news and social media, can someone write a summary of updates from the announcement? What was announced, and how significant do you think it is?

Comment by benito on The AI Timelines Scam · 2019-07-17T08:03:27.127Z · score: 20 (7 votes) · LW · GW

Scott's post explaining his opinion is here, and is called 'Against Lie Inflation'.

Comment by benito on Benito's Shortform Feed · 2019-07-17T00:34:28.009Z · score: 2 (1 votes) · LW · GW

Note: I just wrote this in one pass when severely jet lagged, and did not have the effort to edit it much. If I end up turning this into a blogpost I will probably do that. Anyway, I am interested in hearing via PM from anyone who feels that it was sufficiently unclearly written that they had a hard time understanding/reading it.

Comment by benito on Benito's Shortform Feed · 2019-07-17T00:31:47.536Z · score: 31 (7 votes) · LW · GW

Responding to Scott's response to Jessica.

The post makes the important argument that if we have a word whose boundary is around a pretty important set of phenomena that are useful to have a quick handle to refer to, then

  • It's really unhelpful for people to start using the word to also refer to a phenomena with 10x or 100x more occurrences in the world because then I'm no longer able to point to the specific important parts of the phenomena that I was previously talking about
    • e.g. Currently the word 'abuser' describes a small number of people during some of their lives. Someone might want to say that technically it should refer to all people all of the time. The argument is understandable, but it wholly destroys the usefulness of the concept handle.
  • People often have political incentives to push the concept boundary to include a specific case in a way that, if it were principled, indeed makes most of the phenomena in the category no use to talk about. This allows for selective policing being the people with the political incentive.
  • It's often fine for people to bend words a little bit (e.g. when people verb nouns), but when it's in the class of terms we use for norm violation, it's often correct to impose quite high standards of evidence for doing so, as we can have strong incentives (and unconscious biases!) to do it for political reasons.

These are key points that argue against changing the concept boundary to include all conscious reporting of unconscious bias, and more generally push back against unprincipled additions to the concept boundary.

This is not an argument against expanding the concept to include a specific set of phenomena that share the key similarities with the original set, as long as the expansion does not explode the set. I think there may be some things like that within the category of 'unconscious bias'.

While it is the case that it's very helpful to have a word for when a human consciously deceives another human, my sense is that there are some important edge cases that we would still call lying, or at least a severe breach of integrity that should be treated similarly to deliberate conscious lies.

Humans are incentivised to self-deceive in the social domain in order to be able to tell convincing lies. It's sometimes important that if it's found out that someone strategically self-deceived, that they be punished in some way.

A central example here might be a guy who says he wants to be in a long and loving committed relationship, only to break up after he is bored of the sex after 6-12 months, and really this was predictable from the start if he hadn't felt it was fine to make big commitments things without introspecting carefully on their truth. I can imagine the woman in this scenario feeling genuinely shocked and lied to. "Hold on, what are you talking about that you feel you want to move out? I am organising my whole life around this relationship, what you are doing right now is calling into question the basic assumptions that you have promised to me." I can imagine this guy getting a reputation for being untrustworthy and lying to women. I think it is an open question about whether it is accurate for the woman cheated by this man to tell other people that he "lied to her", though I think it is plausible that I want to punish this behaviour in a similar way that I want to punish much more conscious deception, in a way that motivates me to want to refer to it with the same handle - because it gives you basically very similar operational beliefs about the situation (the person systematically deceived me in a way that was clearly for their personal gain and this hurt me and I think they should be actively punished).

I think I can probably come up with an example where a politician wants power and does whatever is required to take it, such that they end up not being in alignment with the values they stated they held earlier in their career, and allow the meaning of words to fluctuate around them in accordance with what the people giving the politician votes and power want that they end up saying something that is effectively a lie, but that they don't care about or really notice. This one is a bit slippery for me to point to.

Another context that is relevant: I can imagine going to a scientific conference in a field that has been hit so hard by the replication crisis, that basically all the claims in the conference were false, and I could know this. Not only are the claims at this conference false, but the whole subfield has never been about anything real (example, example, and of course, example). I can imagine a friend of mine attending such a conference and talking to me afterwards, and them thinking that some of the claims seemed true. And I can imagine saying to them "No. You need to understand that all the claims in there are lies. There is no truth-tracking process occurring. It is a sham field, and those people should not be getting funding for their research." Now, do I think the individuals in the field are immoral? Not exactly, but sorta. They didn't care about truth and yet paraded themselves as scientists. But I guess that's a big enough thing in society that they weren't unusually bad or anything. While it's not a central case of lying, it currently feels to me like it's actively helpful for me to use the phrase 'lie' and 'sham'. There is a systematic distortion of truth that gives people resources they want instead of those resources going to projects not systematically distorting reality.

(ADDED: OTOH I do think that I have myself in the past been prompted to want to punish people for these kinds of 'lies' in ways that isn't effective. I have felt that people who have committed severe breaches of integrity in the communities I'm part of are bad people and felt very angry at them, but I think that this has often been an inappropriate response. It does share other important similarities with lies though. Probably want to be a bit careful with the usage here and signal that the part of wanting to immediately socially punish them for a thing that they obviously did wrong is not helpful, because they will feel helpless and not that it's obvious they did something wrong. But it's important for me internally to model them as something close to lying, for the sanity of my epistemic state, especially when many people in my environment will not know/think the person has breached integrity and will socially encourage me to positively weight their opinions/statements.)

My current guess at the truth: there are classes of human motivations, such as those for sex, and for prestigious employment positions in the modern world, that have sufficiently systematic biases in favour of self-deception that it is not damaging to add them to the category of 'lie' - adding them is not the same as a rule that admits all unconscious bias consciously reported, just a subset that reliably turns up again and again. I think Jessica Taylor / Ben Hoffman / Michael Arc want to use the word 'fraud' to refer to it, I'm not sure.

Comment by benito on A Key Power of the President is to Coordinate the Execution of Existing Concrete Plans · 2019-07-16T22:48:31.146Z · score: 6 (3 votes) · LW · GW

Yeah this is a really great point, and I should've incorporated it into the OP. Thx :)

Comment by benito on A Key Power of the President is to Coordinate the Execution of Existing Concrete Plans · 2019-07-16T22:38:56.323Z · score: 2 (1 votes) · LW · GW

nods FWIW I think it’s plausible I exaggerated the levels of competence in government and could be persuaded to edit wording; my main intention with the post was to discuss what type signature the top of government in fact has.

Comment by benito on A Key Power of the President is to Coordinate the Execution of Existing Concrete Plans · 2019-07-16T18:56:38.330Z · score: 4 (2 votes) · LW · GW

Thx! And what’s your impression - that the White House mostly focuses on innovation and execution of plans rather than coordinating other actors?

Comment by benito on Integrity and accountability are core parts of rationality · 2019-07-16T07:52:26.631Z · score: 12 (6 votes) · LW · GW

I tend to think of integrity as the ability to have true beliefs and take good action in spite of incentives. I'm thinking of the person who chooses not to break a principle, even when nobody is looking and it's not an obviously important case of the principle and there's a lot of moral/personal value to be gained from it.

But there is also the second part. A person with high integrity can make many principled decisions in spite of incentives, but a person with high integrity also notices when they're entering an environment where they're going to be faced with too many decisions to be able to make good ones.

For example, I have a personal rule against googling a large class of gossipy/political things that feel yummy except when I have explicitly time-boxed time to think through it, because at the minute I don't trust my background reasoning processes to incorporate the evidence in an unbiased way. I used to google gossipy/political topics a fair bit. Sometimes I still do it at midnight when I am bored and not tired. But I increasingly have been able to catch myself and say "this is not behaviour I endorse", even though it's quite difficult because there's not an alternative yummy thing to do. Increasingly, I'm becoming someone who can say "well, I made a policy decision to not take this action, so I won't". However the more useful thing is noticing that my phone is generally giving me a lot of decisions to make that are difficult and often many of the choices I don't endorse, and then systematically remove those options. I've done things like blocked social media on my phone except for 2 hours on Saturday, and blocked it permanently on my laptop, and followed some of Tristan Harris's advice on organising my apps. Both of these things preserve my ability to think clearly and take good action.

There's this idea that people with integrity can be handed power and be expected to continue doing the sort of things they did when they had less power - be the same sort of person, hold the same principles, etc. Or alternatively they will turn down the power if they think that they won't be able to be the same person. Following that, there's the old idea that the people who should be given power are those who don't want it. I'm not sure this really holds up - those who don't want it often have actual models predicting that they will experience failures of integrity. Though at least they have a model of where the mistakes will come and can try to prepare for them. Most people don't even know where their errors will come.

I'm trying to figure out whether "acting in accordance with your stated beliefs" feels like the right description. I guess that there's this thing relating to noticing when you will stop being the same kind of person, and avoiding taking that action unless you endorse it. I expect a person with a lot of integrity to change things about themselves and their actions, but in ways that they reflectively endorse, rather than being pulled around by the winds of the local incentives.

If I am to propose an alternative definition, it's that someone with integrity is someone I can trust to follow their long-term goals and not be thrown off-course by following short-term incentives. Someone who is able to turn down power when it will throw them off their long-term course, even if they haven't figured out how to get power a different way yet. Someone who will learn to say no to the short-term offers they are getting.