Comment by thrasymachus on What are the advantages and disadvantages of knowing your own IQ? · 2019-04-07T15:19:12.949Z · score: 13 (4 votes) · LW · GW

It looks generally redundant in most cases to me: Given how pervasive IQ-correlations are, I think most people can get a reasonable estimate of their IQ by observing their life history so far. E.g.

  • Educational achievement
  • Performance on other standardised tests
  • Job type and professional success
  • Peer esteem/reputation

Obviously, none of these are perfect signals, but I think taking them together usually gives a reasonable steer to a credible range not dramatically larger than test-restest correlations of an IQ test. An IQ test would still provide additional information, but I'm not sure there are many instances where (say) knowing the answer in a 5 point band versus a 10 point band is that important.

The case where I think it could be worthwhile is for those whose life history hasn't generated the usual signals to review: maybe one was initially homeschooled and became seriously ill before starting employment/university, etc.

Comment by thrasymachus on How good is a human's gut judgement at guessing someone's IQ? · 2019-02-26T01:55:18.592Z · score: 28 (11 votes) · LW · GW

Googling around phrases like 'perception of intelligence' seems to be a keyword for a relevant literature. On a very cursory skim (i.e. no more than what you see here) it seems to suggest "people can estimate intelligence of strangers better than chance (but with plenty of room for error and bias), even with limited exposure". E.g.:

Perceived Intelligence Is Associated with Measured Intelligence in Men but Not Women (Note in this study the assessment was done purely on looking at a photograph of someone's face)

Accurate Intelligence Assessments in Social Interactions: Mediators and Gender Effects (Abstract starts with: "Research indicates that people can assess a stranger's measured intelligence more accurately than expected by chance, based on minimal information involving appearance and behavior.")

Thin Slices of Behavior as Cues of Personality and Intelligence. (Short 1-2min slices of behaviour in a variety of contexts leads to assessments by strangers that positively correlate with administered test scores for IQ and big 5)

Comment by thrasymachus on Epistemic Tenure · 2019-02-19T10:39:18.465Z · score: 7 (7 votes) · LW · GW

As you say, Bob's good epistemic reputation should count when he says something that appears wild, especially if he has a track record that endorses him in these cases ("We've thought he was crazy before, but he proved us wrong"). Maybe one should think of Bob as an epistemic 'venture capitalist', making (seemingly) wild epistemic bets which are right more often than chance (and often illuminating even if wrong), even if they aren't right more often than not, and this might be enough to warrant further attention ("well, he's probably wrong about this, but maybe he's onto something").

I'm not sure your suggestion pushes in the right direction in the case where - pricing all of that in - we still think Bob's belief is unreasonable and he is unreasonable for holding it. The right responses in this case by my lights are two-fold.

First, you should dismiss (rather than engage with) Bob's wild belief - as (ex hypothesi) all things considered it should be dismissed.

Second, it should (usually) count against Bob's overall epistemic reputation. After all, whatever it was that meant despite Bob's merits you think he's saying something stupid is likely an indicator of epistemic vice.

This doesn't mean it should be a global black mark to taking Bob seriously ever again. Even the best can err badly, so one should weigh up the whole record. Furthermore, epistemic virtue has a few dimensions, and Bob's weaknesses in something need not mean his strengths in others be sufficient for attention esteem going forward: An archetype I have in mind with 'epistemic venture capitalist' is someone clever, creative, yet cocky and epistemically immodest - has lots of novel ideas, some true, more interesting, but many 'duds' arising from not doing their homework, being hedgehogs with their preferred 'big idea', etc.

I accept, notwithstanding those caveats, this still disincentivizes epistemic venture capitalists like Bob to some degree. Although I only have anecdata, this leans in favour of some sort of trade-off: brilliant thinkers often appear poorly calibrated and indulge in all sorts of foolish beliefs; interviews with superforecasters (e.g.) tend to emphasise things like "don't trust your intuition, be very self sceptical, canvass lots of views, do lots of careful research on a topic before staking out a view". Yet good epistemic progress relies on both - and if they lie on a convex frontier, one wants to have a division of labour.

Although the right balance to strike re. second order norms depends on tricky questions on which sort of work is currently under-supplied, which has higher value on the margin, and the current norms of communal practice (all of which may differ by community), my hunch is 'epistemic tenure' (going beyond what I sketch above) tends disadvantageous.

One is noting the are plausible costs in both directions. 'Tenure'-esque practice could spur on crack pots, have too lax a filter for noise-esque ideas, discourage broadly praiseworthy epistemic norms (cf. virtue of scholarship), and maybe not give Bob-like figures enough guidance so they range too far and unproductively (e.g. I recall one Nobel Laureate mentioning the idea of, "Once you win your Nobel Prize, you should go and try and figure out the hard problem of consciousness" - which seems a terrible idea).

The other is even if there is a trade-off, one still wants to reach the one's frontier on 'calibration/accuracy/whatever'. Scott Sumner seems to be able to combine researching on the inside view alongside judging on the outside view (see). This seems better for Sumner, and the wider intellectual community, than Sumner* who could not do the latter.

Comment by thrasymachus on What are the open problems in Human Rationality? · 2019-01-14T17:53:39.533Z · score: 18 (6 votes) · LW · GW

FWIW: I'm not sure I've spent >100 hours on a 'serious study of rationality'. Although I have been around a while, I am at best sporadically active. If I understand the karma mechanics, the great majority of my ~1400 karma comes from a single highly upvoted top level post I wrote a few years ago. I have pretty sceptical reflexes re. rationality, the rationality community, etc., and this is reflected in that (I think) the modal post/comment I make is critical.

On the topic 'under the hood' here:

I sympathise with the desire to ask conditional questions which don't inevitably widen into broader foundational issues. "Is moral nihilism true?" doesn't seem the right sort of 'open question' for "What are the open questions in Utilitarianism?". It seems better for these topics to be segregated, no matter the plausibility or not for the foundational 'presumption' ("Is homeopathy/climate change even real?" also seems inapposite for 'open questions in homeopathy/anthropogenic climate change'). (cf. 'This isn't a 101-space').

That being said, I think superforecasting/GJP and RQ/CART etc. are at least highly relevant to the 'Project' (even if this seems to be taken very broadly to normative issues in general - if Wei_Dai's list of topics are considered elements of the wider Project, then I definitely have spent more than 100 hours in the area). For a question cluster around "How can one best make decisions on unknown domains with scant data", the superforecasting literature seems some of the lowest hanging fruit to pluck.

Yet community competence in these areas has apparently declined. If you google 'lesswrong GJP' (or similar terms) you find posts on them but these posts are many years old. There has been interesting work done in the interim: here's something on the whether the skills generalise, and something else of a training technique that not only demonstrably improves forecasting performance, but also has a handy mnemonic one could 'try at home'. (The same applies to RQ: Sotala wrote a cool sequence on Stanovich's 'What intelligence tests miss', but this is 9 years old. Stanovich has written three books since expressly on rationality, none of which have been discussed here as best as I can tell.)

I don't understand, if there are multiple people who have spent >100 hours on the Project (broadly construed), why I don't see there being a 'lessons from the superforecasting literature' write-up here (I am slowly working on one myself).

Maybe I just missed the memo and many people have kept abreast of this work (ditto other 'relevant-looking work in academia'), and it is essentially tacit knowledge for people working on the Project, but they are focusing their efforts to develop other areas. If so, a shame this is not being put into common knowledge, and I remain mystified as to why the apparent neglect of these topics versus others: it is a lot easier to be sceptical of 'is there anything there?' for (say) circling, introspection/meditation/enlightenment, Kegan levels, or Focusing than for the GJP, and doubt in the foundation should substantially discount the value of further elaborations on a potentially unedifying edifice.

[Minor] I think the first para is meant to be block-quoted?

Comment by thrasymachus on What are the open problems in Human Rationality? · 2019-01-13T15:44:36.207Z · score: 19 (10 votes) · LW · GW

There seem some foundational questions to the 'Rationality project', and (reprising my role as querulous critic) are oddly neglected in the 5-10 year history of the rationalist community: conspicuously, I find the best insight into these questions comes from psychology academia.

Is rationality best thought of as a single construct?

It roughly makes sense to talk of 'intelligence' or 'physical fitness' because performance in sub-components positively correlate: although it is hard to say which of an elite ultramarathoner, Judoka, or shotputter is fittest, I can confidently say all of them are fitter than I, and I am fitter than someone who is bedbound.

Is the same true of rationality? If it were the case that performance on tests of (say) callibration, sunk cost fallacy, and anchoring were all independent, then this would suggest 'rationality' is a circle our natural language draws around a grab-bag of skills or practices. The term could therefore mislead us into thinking it is a unified skill which we can 'generally' improve, and our efforts are better addressed at a finer level of granularity.

I think this is plausibly the case (or at least closer to the truth). The main evidence I have in mind is Stanovich's CART, whereby tests on individual sub-components we'd mark as fairly 'pure rationality' (e.g. base-rate neglect, framing, overconfidence - other parts of the CART look very IQ-testy like syllogistic reasoning, on which more later) have only weak correlations with one another (e.g. 0.2 ish).

Is rationality a skill, or a trait?

Perhaps key is that rationality (general sense) is something you can get stronger at or 'level up' in. Yet there is a facially plausible story that rationality (especially so-called 'epistemic' rationality) is something more like IQ: essentially a trait where training can at best enhance performance on sub-components yet not transfer back to the broader construct. Briefly:

  • Overall measures of rationality (principally Stanovich's CART) correlate about 0.7 with IQ - not much worse than IQ test subtests correlate with one another or g.
  • Infamous challenges in transfer. People whose job relies on a particular 'rationality skill' (e.g. gamblers and calibration) show greater performance in this area but not, as I recall, transfer improvements to others. This improved performance is often not only isolated but also context dependent: people may learn to avoid a particular cognitive bias in their professional lives, but remain generally susceptible to it otherwise.
  • The general dearth of well-evidenced successes from training. (cf. the old TAM panel on this topic, where most were autumnal).
  • For superforecasters, the GJP sees it can get some boost from training, but (as I understand it) the majority of their performance is attributed to selection, grouping, and aggregation.

It wouldn't necessarily be 'game over' for the 'Rationality project' even if this turns out to be the true story. Even if it is the case that 'drilling vocab' doesn't really improve my g, I might value a larger vocabulary for its own sake. In a similar way, even if there's no transfer, some rationality skills might prove generally useful (and 'improvable') such that drilling them to be useful on their own terms.

The superforecasting point can be argued the other way: that training can still get modest increases in performance in a composite test of epistemic rationality from people already exhibiting elite performance. But it does seem crucial to get a general sense of how well (and how broadly) can training be expected to work: else embarking on a program to 'improve rationality' may end up as ill-starred as the 'brain-training' games/apps fad a few years ago.

Comment by thrasymachus on Open Thread January 2019 · 2019-01-12T19:39:41.171Z · score: 32 (17 votes) · LW · GW

On Functional Decision Theory (Wolfgang Schwarz)

I recently refereed Eliezer Yudkowsky and Nate Soares's "Functional Decision Theory" for a philosophy journal. My recommendation was to accept resubmission with major revisions, but since the article had already undergone a previous round of revisions and still had serious problems, the editors (understandably) decided to reject it. I normally don't publish my referee reports, but this time I'll make an exception because the authors are well-known figures from outside academia, and I want to explain why their account has a hard time gaining traction in academic philosophy. I also want to explain why I think their account is wrong, which is a separate point.
Comment by thrasymachus on Why is so much discussion happening in private Google Docs? · 2019-01-12T19:26:20.318Z · score: 17 (8 votes) · LW · GW

I'm someone who both prefers and practises the 'status quo'.

My impression is the key feature of this is limited (and author controlled) sharing. (There are other nifty features for things like gdocs - e.g. commenting 'on a line' - but this practice predates gdocs). The key benefits for 'me as author' are these:

1. I can target the best critics: I usually have a good idea of who is likely to help make my work better. If I broadcast, the mean quality of feedback almost certainly goes down.

2. I can leverage existing relationships: The implicit promise if I send out a draft to someone for feedback is I will engage with their criticism seriously (in contrast, there's no obligation that I 'should' respond to every critical comment on a post I write). This both encourages them to do so, and may help further foster a collegial relationship going forward.

3. I can mess up privately: If what I write makes a critical (or embarrassing) mistake, or could be construed to say something objectionable, I'd prefer this be caught in private rather than my failing being on the public record for as long as there's an internet archive (or someone inclined to take screen shots). (This community is no stranger to people - insiders or outsiders - publishing mordant criticisms of remarks made 'off the cuff' to infer serious faults in the speaker).

I also think the current status quo is a pretty good one from an ecosystem wide perspective too: I think there's a useful division of labour between 'early stage' writings to be refined by a smaller network with lower stakes, and 'final publications' which the author implicitly offers an assurance (backed by their reputation) that the work is a valuable contribution to the epistemic commons.

For most work there is a 'refining' stage, which is better done by smaller pre-selected networks rather than of authors and critics mutually 'shouting into the void' (from the author's side, there will likely be a fair amount of annoying/irrelevant/rubbish criticism; from a critic's side, a fair risk your careful remarks will be ignored or brushed off).

Publication seems to be better for polished or refined work, as at this stage a) it hopefully it has fewer mistakes and so generally more valuable to the non-critical reader, b) if there is a key mistake/objection neglected (e.g. because the pre-selected network resulted in an echo chamber) disagreement between ('steel-manned') positions registered publicly and hashed seems a useful exercise. (I'm generally a fan of more 'adversarial' - or at least 'adversarial-tolerant' norms for public discussion for this reason.)

This isn't perfect, although I don't see the 'comments going to waste' issue as the greatest challenge (one can adapt one's private comments to a public one to post, although I appreciate this is a costlier route than initially writing the public comment - ultimately, if one finds ones private feedback is repeatedly neglected, one can decline to provide it in the first place).

The biggest one I see is the risk of people who can benefit from a 'selective high-quality feedback network' (either contributing useful early stage criticism, having good early stage posts, or both) not being able to enter one. Yet so long as members of existing ones still 'keep an eye out' for posts and comments from 'outsiders', this does provide a means for such people to build up a reputation to be included in future (i.e. if Alice sees Bob make good remarks etc., she's more interested in 'running a draft by him' next time, or to respond positively if Bob asks her to look something over).

Comment by thrasymachus on Genetically Modified Humans Born (Allegedly) · 2018-12-02T05:10:44.786Z · score: 14 (5 votes) · LW · GW

Once again I plead that when you see that an expert community looks like they don't know what their doing, it is usually more accurate to 'reduce confidence' in your understanding rather than their competence. The questions were patently not 'about forms', and covered pretty well the things I would have in mind (I'm a doctor, and I have fairly extensive knowledge of medical ethics).

To explain:

  • Although 'institutional oversight' in medicine is often derided (IRB creep, regulatory burden, and so on and so forth), one of its main purposes is to act as a check on researchers (whatever their intent) causing harm to their patients, and the idea it is good to have other people besides the researcher (who might be biased) and the patient (who might be less well informed) be the only ones making these decisions. That typical oversight was bypassed here is telling, but perhaps unsurprising as no one would green-light violating a moratorium to subject healthy embryos to poorly tested medical procedures for at best marginal clinical benefit.
  • A lot of questions targeted how informed the consent was, because this was often relied upon in the presentation (e.g. "Well, we didn't get the right mutation, but it was pretty close, and the parents were happy for us to go ahead, so we did").
  • The 'read and understand' question (I'm using the transcript, so maybe there were dumber questions which were edited out) wasn't a question about whether the patients were literate, but whether they had adequate understanding of (e.g.) the technical caveats which they were giving consent to proceed with (e.g. one mutation was a 15 del rather than a 32 del, which rather than the natural mutation which induces a frame shift and the non-functional protein gives a novel protein with a five aa removal, which may still generate an HIV susceptible protein and some remote chance of other biological effects).
  • The 'training' question is because establishing whether consent is 'informed', or providing the necessary information to make it so, isn't always straightforward (have you ever had a conversation where you thought someone understood you, but later you found out they didn't?) I did a fair amount of this in medschool, and I don't think many people think this should be an amateur sport.
  • (As hopefully goes without saying, having two rounds of consent where in each the consent taker is a researcher with a vested interest in the work going ahead has obvious problems, and hence why we're so keen on third party oversight).
  • I also see in the transcript fairly extensive discussion about risks (off-target worries would have been tacit knowledge to the audience, so some of this was pre-empted in the presentation then later picked at), and plans for followup etc.
Comment by thrasymachus on No Really, Why Aren't Rationalists Winning? · 2018-11-05T20:35:32.921Z · score: 24 (9 votes) · LW · GW

I don't see the 'why aren't you winning?' critique as that powerful, and I'm someone who tends critical of rationality writ-large.

High-IQ societies and superforecasters select for demonstrable performance at being smart/epistemically rational. Yet on surveying these groups you see things like, "People generally do better-than-average by commonsense metrics, some are doing great, but it isn't like everyone is a millionaire". Given the barrier to entry to the rationalist community is more, "sincere interest" than "top X-percentile of the population", it would remarkable if they exhibited even better outcomes as a cohort.

There's also going to be messy causal inference worries that cut either way. If there is in some sense 'adverse selection' (perhaps alike IQ societies) for rationalists tending to have less aptitude at social communication, greater prevalence of mental illness (or whatever else), then these people enjoying modest to good success in their lives reflects extremely well on the rationalist community. Contrariwise, there's plausible confounding where smart creative people will naturally gravitate to rationality-esque discussion, even if this discussion doesn't improve their effectiveness (I think a lot of non-rationalists were around OB/LW in the early days): the cohort of people who 'teach themselves general relativity for fun' may also enjoy much better than average success, but it probably wasn't the relativity which did it.

A deeper worry wrt to rationality is there may not be anything to be taught. The elements of (say) RQ don't show much of a common factor (unlike IQ), correlate more strongly with IQ than one another, and improvements in rational thinking have limited domain transfer. So there might not be much of a general sense of (epistemic) rationality, and limited hope for someone to substantially improve themselves in this area.

Comment by thrasymachus on No standard metric for CFAR workshops? · 2018-09-08T12:51:24.971Z · score: 21 (5 votes) · LW · GW

Another thing I'd be particularly interested in is longer term follow-up. It would be impressive if the changes to conscientiousness etc. observed in the 2015 study persist now.

Comment by thrasymachus on Great Founder Theory · 2018-09-06T15:08:16.242Z · score: 12 (4 votes) · LW · GW

I'd be hesitant to defend Great Man theory (and so would apply similar caution) but I think it can go some way, especially for defending a fragility of history hypothesis.

In precis (more here):

1. Conception of any given person seems very fragile. If parents decide to conceive an hour earlier or later (or have done different things earlier in the day, etc. etc.), it seems likely another one of the 100 million available sperm fuses than the one which did. The counterpart seems naturally modelled by a sibling, and siblings are considerably different from one another.

2. Although sometimes (/often) supposed Great Men are mere errands of providence, its hard to say this is always the case. It seems the 20th century would have been pretty different if Hitler was not around to rise to power, the character of world religions would be different with siblings of Jesus, Muhammad etc. and Tolstoy's brother probably wouldn't have written War and Peace anyway. (Although maybe in some areas ramifications are less pronounced - Great Scientists may alter the timing of discoveries a bit, but it looks plausible that we'd have Relativity by now even without Einstein).

3. 1 and 2 suggests you could get a lot of scrambling of who is around. Even if it was inevitable there was a Mongol expansion, the precise nature of this seems sensitive to who is in charge, and so whether Ghengis Khan or his sibling was born. The precise details of this expansion (where gets encroached on first, which battles are fought, etc. etc.) does horizontally perturb whether, when (and with who) other people conceive children. These different children go on to alter vertically and horizontally who else is conceived, and so the conceptive chaos propagates. I'd semi-seriously defend the thesis that none of us would be here if Ghengis Khan's parents decided to wait an hour before having sex.

4. This wouldn't mean the world is merely putty to be sculpted by great men. But even if the stage (and dramatis personae) of history is set by broader factors, which actors take on the role might still have considerable effects on the performance.

Comment by thrasymachus on Historical mathematicians exhibit a birth order effect too · 2018-08-21T11:22:49.112Z · score: 8 (4 votes) · LW · GW

I not sure t-tests are the best approach to take compared to something non-parametric, given smallish sample, considerable skew, etc. (this paper's statistical methods section is pretty handy). Nonetheless I'm confident the considerable effect size (in relative terms, almost a doubling) is not an artefact of statistical technique: when I plugged the numbers into a chi-squared calculator I got P < 0.001, and I'm confident a permutation technique or similar would find much the same.

Comment by thrasymachus on Informational hazards and the cost-effectiveness of open discussion of catastrophic risks · 2018-07-03T22:49:02.542Z · score: 5 (3 votes) · LW · GW

0: We agree potentially hazardous information should only be disclosed (or potentially discovered) when the benefits of disclosure (or discovery) outweigh the downsides. Heuristics can make principles concrete, and a rule of thumb I try to follow is to have a clear objective in mind for gathering or disclosing such information (and being wary of vague justifications like ‘improving background knowledge’ or ‘better epistemic commons’) and incur the least possible information hazard in achieving this.

A further heuristic which seems right to me is one should disclose information in the way that maximally disadvantages bad actors versus good ones. There are a wide spectrum of approaches that could be taken that lie between ‘try to forget about it’, and ‘broadcast publicly’, and I think one of the intermediate options is often best.

1: I disagree with many of the considerations which push towards more open disclosure and discussion.

1.1: I don’t think we should be confident there is little downside in disclosing dangers a sophisticated bad actor would likely rediscover themselves. Not all plausible bad actors are sophisticated: a typical criminal or terrorist is no mastermind, and so may not make (to us) relatively straightforward insights, but could still ‘pick them up’ from elsewhere.

1.2: Although a big fan of epistemic modesty (and generally a detractor of ‘EA exceptionalism’), EAs do have an impressive track record in coming up with novel and important ideas. So there is some chance of coming up with something novel and dangerous even without exceptional effort.

1.3: I emphatically disagree we are at ‘infohazard saturation’ where the situation re. Infohazards ‘can’t get any worse’. I also find it unfathomable ever being confident enough in this claim to base strategy upon its assumption (cf. eukaryote’s comment).

1.4: There are some benefits to getting out ‘in front’ of more reckless disclosure by someone else. Yet in cases where one wouldn’t want to disclose it oneself, delaying the downsides of wide disclosure as long as possible seems usually more important, and so rules against bringing this to an end by disclosing yourself save in (rare) cases one knows disclosure is imminent rather than merely possible.

2: I don’t think there’s a neat distinction between ‘technical dangerous information’ and ‘broader ideas about possible risks’, with the latter being generally safe to publicise and discuss.

2.1: It seems easy to imagine cases where the general idea comprises most of the danger. The conceptual step to a ‘key insight’ of how something could be dangerously misused ‘in principle’ might be much harder to make than subsequent steps from this insight to realising this danger ‘in practice’. In such cases the insight is the key bottleneck for bad actors traversing the risk pipeline, and so comprises a major information hazard.

2.2: For similar reasons, highlighting a neglected-by-public-discussion part of the risk landscape where one suspects information hazards lie has a considerable downside, as increased attention could prompt investigation which brings these currently dormant hazards to light.

3: Even if I take the downside risks as weightier than you, one still needs to weigh these against the benefits. I take the benefit of ‘general (or public) disclosure’ to have little marginal benefit above more limited disclosure targeted to key stakeholders. As the latter approach greatly reduces the downside risks, this is usually the better strategy by the lights of cost/benefit. At least trying targeted disclosure first seems a robustly better strategy than skipping straight to public discussion (cf.).

3.1: In bio (and I think elsewhere) the set of people who are relevant setting strategy and otherwise contributing to reducing a given risk is usually small and known (e.g. particular academics, parts of the government, civil society, and so on). A particular scientist unwittingly performing research with misuse potential might need to know the risks of their work (likewise some relevant policy and security stakeholders), but the added upside to illustrating these risks in the scientific literature is limited (and the added downsides much greater). The upside of discussing them in the popular/generalist literature (including EA literature not narrowly targeted at those working on biorisk) is limited still further.

3.2: Information also informs decisions around how to weigh causes relative to one another. Yet less-hazardous information (e.g. the basic motivation given here or here, and you could throw in social epistemic steers from the prevailing views of EA ‘cognoscenti’) is sufficient for most decisions and decision-makers. The cases where this nonetheless might be ‘worth it’ (e.g. you are a decision maker allocating a large pool of human or monetary capital between cause areas) are few and so targeted disclosure (similar to 3.1 above) looks better.

3.3: Beyond the direct cost of potentially giving bad actors good ideas, the benefits of more public discussion may not be very high. There are many ways public discussion could be counter-productive (e.g. alarmism, ill-advised remarks poisoning our relationship with scientific groups, etc.). I’d suggest the examples of cryonics, AI safety, GMOs and other lowlights of public communication of policy and science are relevant cautionary examples.

4: I also want to supply other more general considerations which point towards a very high degree caution:

4.1: In addition to the considerations around the unilateralist’s curse offered by Brian Wang (I have written a bit about this in the context of biotechnology here) there is also an asymmetry in the sense that it is much easier to disclose previously-secret information than make previously-disclosed information secret. The irreversibility of disclosure warrants further caution in cases of uncertainty like this.

4.2: I take the examples of analogous fields to also support great caution. As you note, there is a norm in computer security of ‘don’t publicise a vulnerability until there’s a fix in place’, and initially informing a responsible party to give them the opportunity to to do this pre-publication. Applied to bio, this suggests targeted disclosure to those best placed to mitigate the information hazard, rather than public discussion in the hopes of prompting a fix to be produced. (Not to mention a ‘fix’ in this area might prove much more challenging than pushing a software update.)

4.3: More distantly, adversarial work (e.g. red-teaming exercises) is usually done by professionals, with a concrete decision-relevant objective in mind, with exceptional care paid to operational security, and their results are seldom made publicly available. This is for exercises which generate information hazards for a particular group or organisation - similar or greater caution should apply to exercises that one anticipates could generate information hazardous for everyone.

4.4: Even more distantly, norms of intellectual openness are used more in some areas, and much less in others (compare the research performed in academia to security services). In areas like bio, the fact that a significant proportion of the risk arises from deliberate misuse by malicious actors means security services seem to provide the closer analogy, and ‘public/open discussion’ is seldom found desirable in these contexts.

5: In my work, I try to approach potentially hazardous areas as obliquely as possible, more along the lines of general considerations of the risk landscape or from the perspective of safety-enhancing technologies and countermeasures. I do basically no ‘red-teamy’ types of research (e.g. brainstorm the nastiest things I can think of, figure out the ‘best’ ways of defeating existing protections, etc.)

(Concretely, this would comprise asking questions like, “How are disease surveillance systems forecast to improve over the medium term, and are there any robustly beneficial characteristics for preventing high-consequence events that can be pushed for?” or “Are there relevant limits which give insight to whether surveillance will be a key plank of the ‘next-gen biosecurity’ portfolio?”, and not things like, “What are the most effective approaches to make pathogen X maximally damaging yet minimally detectable?”)

I expect a non-professional doing more red-teamy work would generate less upside (e.g. less well networked to people who may be in a position to mitigate vulnerabilities they discover, less likely to unwittingly duplicate work) and more downside (e.g. less experience with trying to manage info-hazards well) than I. Given I think this work is usually a bad idea for me to do, I think it’s definitely a bad idea for non-professionals to try.

I therefore hope people working independently on this topic approach ‘object level’ work here with similar aversion to more ‘red-teamy’ stuff, or instead focus on improving their capital by gaining credentials/experience/etc. (this has other benefits: a lot of the best levers in biorisk are working with/alongside existing stakeholders rather than striking out on one’s own, and it’s hard to get a role without (e.g.) graduate training in a relevant field). I hope to produce a list of self-contained projects to help direct laudable ‘EA energy’ to the best ends.

Comment by thrasymachus on Informational hazards and the cost-effectiveness of open discussion of catastrophic risks · 2018-07-03T22:48:43.530Z · score: 17 (4 votes) · LW · GW

Thanks for writing this. How best to manage hazardous information is fraught, and although I have some work in draft and under review, much remains unclear - as you say, almost anything could have some some downside risk, and never discussing anything seems a poor approach.

Yet I strongly disagree with the conclusion that the default should be to discuss potentially hazardous (but non-technical) information publicly, and I think your proposals of how to manage these dangers (e.g. talk to one scientist first) generally err too lax. I provide the substance of this disagreement in a child comment.

I’d strongly endorse a heuristic along the lines of, “Try to avoid coming up with (and don’t publish) things which are novel and potentially dangerous”, with the standard of novelty being a relatively uninformed bad actor rather than an expert (e.g. highlighting/elaborating something dangerous which can be found buried in the scientific literature should be avoided).

This expressly includes more general information as well as particular technical points (e.g. “No one seems to be talking about technology X, but here’s why it has really dangerous misuse potential” would ‘count’, even if a particular ‘worked example’ wasn’t included).

I agree it would be good to have direct channels of communication for people considering things like this to get advice on whether projects they have in mind are wise to pursue, and to communicate concerns they have without feeling they need to resort to internet broadcast (cf. Jan Kulveit’s remark).

To these ends, people with concerns/questions of this nature are warmly welcomed and encouraged to contact me to arrange further discussion.

Comment by thrasymachus on Societal Growth Requires Rehabilitation · 2018-05-26T13:54:57.548Z · score: 27 (6 votes) · LW · GW

This seems right to me, and at least the 'motte' version of growth mindset accepts that innate ability may set pretty hard envelopes on what you can accomplish regardless of how energetic/agently you pursue self improvement (and this can apply across a range of ability - although it seems cruel and ludicrous to suggest someone with severe cognitive impairment can master calculus, it also seems misguided to suggest someone in middle age can become a sports star if they really go for it). As you say, taking growth mindset 'too far' has a dark side in that we might start thinking that people fail or struggle because they aren't trying hard enough (generally a fault which we morally criticise) rather than lacking the ability (generally 'blameless').

But I'd venture a broader criticism about growth mindset which apply both to the 'motte' form sketched above, but also its sincere use in the rationalist community - that we shouldn't only 'not take it too far', but not take it anywhere at all:

1) Growth mindset as expounded by Dweck and colleagues has not weathered replication well. The most recent systematic reviews give extremely minor effects on achievement (r=0.1) and even smaller intervention effects (d=0.08). The authors of the meta-analysis are about as sceptical as I am about whether these residual effects are real, but even if real they are extremely minor compared to more traity things (you can get much better prediction of academic achievement by genotyping than assessing growth mindest).

2) There's a natural story of reverse causation, which also applies to the closely related 'internal versus external locus of control'. If you're smart and living in propitious circumstances, you may be right in thinking "I can get good at this if I really try" for many different things. If you lack this good fortune, your belief "Even if I try really hard at something, for most somethings I probably won't develop mastery (or even competence)" might be a case of accurate and laudable insight.

3a) I can think of more than a few occasions in my life where the latter was better for me. One was when I was contemplating what subjects to keep before I went to university, and I had discussions with various teachers along the lines of, "You're good but not exceptional at this, maybe think about something else?" Or (in medical school) a conversation along the lines of, "You have dysgraphia, which probably makes you somewhat weaker at fine manual dexterity. Opthalmology requires really good fine manual dexterity, so maybe this isn't the specialty for you."

3b) It also seemed to serve me better when I couldn't circumvent my limitations by picking a different line of work. I focused especially hard on training myself to perform practical procedures because I realised I was working from a disadvantage, and so had to try harder to be satisfactory (I maintained no illusions of becoming great at it).

3c) My impression is that conversations (or thinking) like this tend to be more emotionally difficult than more aspirational, "Don't worry, you can do it!" exhortations. So I'd guess they are probably under-supplied from their optimum.

In essence, there's an underlying empirical topic which 'growth mindset' relies upon: that a lot of whether one accomplishes something or not depends on mindset or attitude. The answer to that, as best as I can tell, is this isn't really true: we live in a world which has the uncomfortable features where which tickets one draws from the genetic lottery, birthplace lottery, and early environment lottery (etc) determine the broad strokes of one's life far more than particular efforts of will and mindset (and growth mindset in particular, which seems to have slim-to-no effect). Many things which are possible for someone are not possible for us, no matter what we do, and no matter how hard we try.

Then there's a prudential question of (even if it isn't true) whether it would be better to act and believe as-a growth mindset would suggest. Again, it doesn't seem so: the evidence for mindset interventions working is slim to none, and insofar as one can survey anecdata my impression is 'anti-growth mindset' advice is undersupplied relative to its importance.

It is inarguable one should often persevere in work to improve oneself, to not give up 'too soon', and to encourage others when trying to do the same. Yet there are times when it is better to recognise the limits of one's abilities, that one should cut their losses, and to shoulder the burden of (if one believes it to be the case) telling someone they should quit something because they 'don't have what it takes'. The right judgement in these cases is a matter for practical wisdom. Insofar as growth mindset as it is preached (but also as it is practised) biases us more to the former sort of behaviour, it should be resisted.

Comment by thrasymachus on The Berkeley Community & The Rest Of Us: A Response to Zvi & Benquo · 2018-05-22T09:23:19.781Z · score: 38 (9 votes) · LW · GW
A healthy topology of the field should have approximately power-law distribution of hub sizes. This should be true also for related research fields we are trying to advance, like AI alignment or x-risk. If the structure is very far from that (e.g. one or two very big hubs, than nothing, than a lot of two orders of magnitude smaller groups fighting for mere existence), the movement should try to re-balance, supporting growth of medium-tier hubs.

Although my understanding of network science is abecedarian, I'm unsure of both whether this feature is diagnostic (i.e. divergence from power-law distributions should be a warning sign) or whether we in fact observe overdispersion even relative to a power law. The latter first.

1) 'One or two big hubs, then lots of very small groups' is close to what a power law distribution should look like. If anything, it's plausible the current topology doesn't look power-lawy enough. The EA community overlaps with the rationalist community, and it has somewhat better data on topology: If anything the hub sizes of the EA community are pretty even. This also agrees with my impression: although the bay area can be identified as the biggest EA hub, there are similar or at least middle sized hubs elsewhere (Oxford, Cambridge (UK), London, Seattle, Berlin, Geneva, etc. etc.) If we really thought a power law topology was desirable, there's a plausible case to push for centralisation.

The closest I could find to a 'rationalist survey' was the SSC survey, which again has a pretty 'full middle', and not one or two groups ascendant. That said, I'd probably defer to others impressions here as I'm not really a rationalist and most of the rationalist online activity I see does originate from the bay. But even if so, this impression wouldn't worry us if we wanted to see a power law here.

2) My understanding is there are a few generators of power law distributions. One is increasing returns to scale (e.g. cities being more attractive to live in the larger they are, ceteris paribus), another is imperfect substitution (why listen to an okay pianist when I can have a recording of the world's best?), a third could be positive feedback loops or Matthew effects (maybe 'getting lucky' with a breakout single increases my chance of getting noticed again, even when controlling for musical ability versus the hitless).

There are others, but many of these generators are neutral, and some should be welcomed. If there's increasing marginal returns to rationalist density, inward migration to central hubs seems desirable. Certain 'jobs' seem to have this property: a technical AI researcher in (say) Japan probably can have greater EV working in an existing group (most of which are in the bay) rather than trying to seed a new AI safety group in Japan. Ditto if the best people in a smaller hub migrate to contribute to a larger one (although emotions run high, I don't think calling this 'raiding' is helpful - the people who migrate have agency).

[3) My hunch is what might be going on is that the 'returns' are sigmoid, and so are diminishing with new entrants to the Bay Area. 'Jobs'-wise, it is not clear the Bay Area is the best place to go if you aren't going to work on AI research, and even if so this is a skill set that is rare in absolute terms amongst rationalists). Social-wise, there's limited interaction bandwidth, especially among higher status folks, and so the typical rationalist who goes to the bay won't get the upside the most desirable bits of bay area social interactions - when weighed across from the transaction costs, staying put and fostering another hub might look better.]

(I echo Chris's exhortation)

Comment by Thrasymachus on [deleted post] 2018-03-21T21:55:31.062Z


It also risks a backfire effect. If one is in essence a troll happy to sneer at what rationalists do regardless of merit (e.g. "LOL, look at those losers trying to LARP enders game!"), seeing things like Duncan's snarky parenthetical remarks would just spur me on, as it implies I'm successfully 'getting a rise' out of the target of my abuse.

It seems responses to criticism that is unpleasant or uncharitable are best addressed specifically to the offending remarks (if they're on LW2, this seems like pointing out the fallacies/downvoting as appropriate), or just ignored. More broadcasted admonishment ("I know this doesn't apply to everyone, but there's this minority who said stupid things about this") seems unlikely to marshall a corps of people who will act together to defend conversational norms, but bickering and uncertainty about whether or not one is included in this 'bad fraction'.

(For similar reasons, I think amplifying rebuttals along the lines of, "You're misinterpreting me, and that people who don't interpret others correctly is one of the key problems with the LW community" seems apt to go poorly - few want to be painted as barbarians at the gates, and prompts those otherwise inclined to admit their mistake to instead double down or argue the case further.)

Comment by Thrasymachus on [deleted post] 2018-03-20T21:34:15.429Z
I also think I got things about right, but I think anyone else taking an outside view would've expected roughly the same thing.

I think you might be doing yourself a disservice. I took the majority of contemporary critcism was more directed towards (in caricature) 'this is going to turn into a nasty cult!' than (what I took your key insight to be) 'it will peter out because the commander won't actually have the required authority'.

So perhaps the typical 'anyone else' would have alighted on the wrong outside view, or at least the wrong question to apply it to ('How likely would a group of rationalists end up sustaining a 'dictator' structure?', rather than 'Conditional on having this structure, what would happen next?')

Comment by Thrasymachus on [deleted post] 2018-03-20T21:16:01.417Z

Bravo - I didn't look at the initial discussion, or I would have linked your pretty accurate looking analysis (on re-skimming, Deluks also had points along similar lines). My ex ante scepticism was more a general sense than a precise pre-mortem I had in mind.

Comment by Thrasymachus on [deleted post] 2018-03-19T14:59:56.430Z

Although I was sufficiently sceptical of this idea to doubt it was 'worth a shot' ex ante,(1) I was looking forward to being pleasantly surprised ex post. I'm sorry to hear it didn't turn out as well as hoped. This careful and candid write-up should definitely be included on the 'plus' side of the ledger for this project.

With the twin benefits of no skin in the game and hindsight. I'd like to float another account which may synthesize a large part of 'why it didn't work'.

Although I understand DAB wasn't meant to simply emulate 'military style' living, it did borrow quite a lot of that framing (hence the 'army' and 'barracks' bit of DAB). Yet these arrangements require a considerable difference in power and authority between the commander and their subordinates. I don't think DAB had this, and I suggest this proved its downfall.

[R]ight, as the world goes, is only in question between equals in power, while the strong do what they can and the weak suffer what they must
Thucydides - The Melian Dialogue

First, power. Having a compelling or else matters for maintaining discipline - even if people come in planning to obey, they may vacillate. In military contexts, accomodation at a barracks (or a boot camp) is a feature that can be unilaterally rescinded - if I argue back against the drill instructor, even if I'm in the right, they can make the credible threat "Either do as I say, or I kick you out". This threat has great 'downside-asymmetry': it's little skin off the instructor's nose if he boots me, but it means the end of my military career if they follow through. In consequence, the instructor has a lot more de facto bargaining power to make me do things I don't want to do. In negotiation theory, the drill sergeant's BATNA is way better than mine, and so good the only 'negotiation' they use are threats rapidly escalating to expulsion for any disobedience to their dictation.

I assume the legal 'fact on the ground' is that the participants of DAB were co-signatories on a lease, making significant financial contributions, with no mechanism for the designated 'commander' to kick people out unilaterally. Given this realpolitic, whatever 'command' the commander has in this situation is essentially roleplay. If a 'Red Knight' character decides not to play ball with the features of DAB that go beyond typical legal arrangements of house-sharing (e.g. "Nah, I'm not going to do this regular communal activity, I'm not going to do press-ups because you're telling me to, and I'm going to do my own thing instead."), the commander can't 'expel' them, but at most ask them to leave.

In fact the commander can only credibly threaten to expel subordinates from the social game of Dragon Army. If I'm a participant this might be a shame, but it might not be a great cost if I'm not feeling it anymore: I can probably (modulo some pretty dodgy social ostracising which other participants may not abide by) still interact with the remaining dragons in a normal housemate-y way.

Such a situation has pretty symmetrical downsides for both parties: if I'm the Red Knight, I probably would end up leaving (it wouldn't be fun being the 'odd man out', although the difficulty is shared by the dragons who have people in the house who aren't 'buying in', and the commander dealing with the ever-present example of the pretty mild consequence of what happens if others likewise disobey), but not until I'd lined up better accomodation, and I certainly wouldn't feel any obligation - given you're kicking me out rather than I'm choosing to leave - to pay rent after I go or help find others to 'fill my spot' to defray the proportionally increased cost of the lease (itself an incentive for other subordinates to pressure the commander to relent).

Instead of the boot camp case, where the much inferior BATNA of recruits gives them no negotiation position with their drill sergeant; in the DA case, the similarly equidistant BATNAs mean the commander and a dragon are at near-party (whatever affectations both maintain to the contrary). The stage is set for negotiation over discipline (even if coded or implicit), rather than dictation from the commander to the subordinate.

Authority (in the informal or moral sense, c.f. Weberian 'charismatic authority') can partly - though imperfectly - substitute for power (e.g. doctors and clergy tend to have more 'influence' than 'ordering people around'). What is notable that authority like this tends to be linked to some particular concrete achievement or track record, and is mainly constrained to a particular domain, with only mild 'bleed through' into general social status.

Given the nature of DAB, there is no convincing concrete thing a putative-commander could point to which legitimizes broad-raging authority over others beyond assent by the putative-subordinates this is an experiment they want to try. I suspect this results in pseudo-authority and obedience highly conditional on results: "I'm going to act as-if I am much lower status than this 'commander', and so kowtow towards them. But in terms of real status they're a peer to me, and I spot them this loan of social game status because I may benefit. If this doesn't happen, I'm less inclined to play along." The limited capital this supplies doesn't give the commander much to spend on pushing things through despite dissent: as the matters are between equals, 'right' becomes the question again.

I suggest this may explain the need for a better off-ramp (I note military or religious organisations which emphasize discipline and obedience generally provide accomodation etc. gratis, perhaps in part to provide the power initiates are there by invitation - I wonder whether reciprocal arrangements with another house substitute adequately); the reluctance to 'give enough orders' (better to maintain a smaller stake of authority than gamble a wider bid that could lose lots of face if flouted); the lack of a 'big enough stick' ("I'm happy to undergo token punishment - to lose pretend status - as a ritual for the sake of the game, but if the price of you 'disciplining' me costs real status, I'm out"); the need to avoid transgressing what less bought-in subordinates wanted (perhaps the result of a coded and implicit negotiation suggested earlier); and the shortfall between the ostensible standards asserted (set by aspiration), and those observed (set by the balance of power). One could imagine recognition of these features may have been key in transforming 'White Knights' into 'Black Knights'.

I'd guess, if this is the (forgive me) crux of the problem, whether it can really be fixed. Perhaps even more assiduous selection could do the trick, yet the base rate in the Bay Area (which isn't renowned for selecting people easygoing with authority) is not propitious. An alternative - but an exorbitantly expensive one - which would supply the commander with unquestioned power/authority would be if rent is free for subordinates: "You're here for free, so I can kick you if you don't play ball". I suspect a more modest 'Phase II' (maybe just social norms to do communal activity, with no pretended 'boss') might be the best realistic iteration.


1: It also seemed a bit risky from an 'abuse' (very broadly construed) perspective to me, although I stress this worry arose solely from the set-up rather than any adverse judgement of Duncan personally. Although understandable to be snarky to critics (e.g. one despicable pseudonymous commenter on the original 'pitch' recommended Duncan kill himself) I think the triumphalism of, "Although it didn't work, this 'not working' was not achieving all the good targets, rather than something horrible happening to someone - guess this wasn't a hotbed of abuse after all, and I'm not some abusive monster like you said! Eat humble pie! (but you won't, because you're irrational!)" is a bit misplaced.

The steelman of this is something like, "Although I don't think it blowing up really badly is more likely than not, the disjunction of bad outcomes along these lines is substantially higher than the base rate for a typical group-house, and this isn't outweighed across the scales by the expected benefit". On this view the observation the proposed risk wasn't realised is only mild discomfirmation for the hazard, although it rules strongly against outlandish 'disaster by default' views.

[Edit: Slight rewording, concision, and I remembered the theory I was gesturing at imprecisely]

Comment by thrasymachus on Comments on Power Law Distribution of Individual Impact · 2018-03-07T15:44:28.499Z · score: 8 (2 votes) · LW · GW

This new paper may be of relevance (H/T Steve Hsu). The abstract:

The largely dominant meritocratic paradigm of highly competitive Western cultures is rooted on the belief that success is due mainly, if not exclusively, to personal qualities such as talent, intelligence, skills, efforts or risk taking. Sometimes, we are willing to admit that a certain degree of luck could also play a role in achieving significant material success. But, as a matter of fact, it is rather common to underestimate the importance of external forces in individual successful stories. It is very well known that intelligence or talent exhibit a Gaussian distribution among the population, whereas the distribution of wealth - considered a proxy of success - follows typically a power law (Pareto law). Such a discrepancy between a Normal distribution of inputs, with a typical scale, and the scale invariant distribution of outputs, suggests that some hidden ingredient is at work behind the scenes. In this paper, with the help of a very simple agent-based model, we suggest that such an ingredient is just randomness. In particular, we show that, if it is true that some degree of talent is necessary to be successful in life, almost never the most talented people reach the highest peaks of success, being overtaken by mediocre but sensibly luckier individuals. As to our knowledge, this counterintuitive result - although implicitly suggested between the lines in a vast literature - is quantified here for the first time. It sheds new light on the effectiveness of assessing merit on the basis of the reached level of success and underlines the risks of distributing excessive honors or resources to people who, at the end of the day, could have been simply luckier than others. With the help of this model, several policy hypotheses are also addressed and compared to show the most efficient strategies for public funding of research in order to improve meritocracy, diversity and innovation.

Comment by thrasymachus on Meta-tations on Moderation: Towards Public Archipelago · 2018-02-25T19:57:22.909Z · score: 14 (5 votes) · LW · GW

I endorse Said's view, and I've written a couple of frontpage posts.

I also add that I think Said is a particularly able and shrewd critic, and I think LW2 would be much poorer if there was a chilling effect on his contributions.

Comment by thrasymachus on [Meta] New moderation tools and moderation guidelines · 2018-02-18T23:45:03.991Z · score: 7 (8 votes) · LW · GW

Let's focus on the substance, please.

Comment by thrasymachus on [Meta] New moderation tools and moderation guidelines · 2018-02-18T16:25:36.952Z · score: 39 (21 votes) · LW · GW

I'm also mystified at why traceless deletition/banning are desirable properties to have on a forum like this. But (with apologies to the moderators) I think consulting the realpolitik will spare us the futile task of litigating these issues on the merits. Consider it instead a fait accompli with the objective to attract a particular writer LW2 wants by catering to his whims.

For whatever reason, Eliezer Yudkowsky wants to have the ability to block commenters and have the ability to do traceless deletion on his own work, and he's been quite clear this is a condition for his participation. Lo and behold precisely these features have been introduced, with suspiciously convenient karma thresholds which allow EY (at his current karma level) to traceless delete/ban on his own promoted posts, yet exclude (as far as I can tell) the great majority of other writers with curated/front page posts from being able to do the same.

Given the popularity of EY's writing (and LW2 wants to include future work of his), the LW2 team are obliged to weigh up the (likely detrimental) addition of these features versus the likely positives of his future posts. Going for the latter is probably the right judgement call to make, but let's not pretend it is a principled one: we are, as the old saw goes, just haggling over the price.

Comment by thrasymachus on How the LW2.0 front page could be better at incentivizing good content · 2018-01-21T23:26:48.735Z · score: 12 (3 votes) · LW · GW

FWIW, I struggle to navigate the front page to look at good posts (I struggle to explain why - I think I found 'frontpage etc.' easier for earlier versions). What I do instead is look at the comments feed and click through to articles that way, which seems suboptimal, as lots of comments may not be a very precise indicator of quality.

Comment by thrasymachus on Kenshō · 2018-01-21T02:37:46.828Z · score: 27 (8 votes) · LW · GW

FWIW, this aptly describes my own adverse reaction to the OP. "I have this great insight, but I not only can't explain it to you, but I'm going to spend the balance of my time explaining why you couldn't understand it if I tried to explain it" sounds awfully close to bulveristic stories like, "If only you weren't blinded by sin, you too would see the glory of the coming of the lord".

That the object level benefits offered seem to be idiographic self-exhaltations augur still poorer (i.e. I cut through confusion so much more easily now (no examples provided); I have much greater reserves to do stuff; I can form much deeper pacts with others who, like I, can See the Truth.) I recall the 'case' for Ander's Connection Theory was of a similar type. But at least connection theory at least sketched something like a theory to consider on its merits.

There needs to be either some object-level description (i.e. "This is what Looking is"), or - if that really isn't possible - demonstration of good results (i.e. "Here's a great post on a CFAR-adjecent topic, and this was thanks to Looking.") Otherwise, the recondite and the obscurantist look very much alike.

Comment by thrasymachus on Comments on Power Law Distribution of Individual Impact · 2017-12-29T08:43:17.238Z · score: 4 (1 votes) · LW · GW

I was unaware of the range restriction, which could well compress SD. That said, if you take the '9' scorers as '9 or more', then you get something like this (using 20-25)

Mean value is around 7 (6.8), 7% get 9 or more, suggesting 9 is at or around +1.5SD assuming normality, so when you get a sample size in the thousands, you should start seeing scores at 11 or so (+3SD) - I wouldn't be startled to find Ben has this level of ability. But scores at (say) 15 or higher (+6SD) should only be seen with extraordinarily rarely.

If you use log-normal assumptions, you should expect something like if +1.5SD is 2, 3SD is around 6 (i.e. ~13), and 4.5SD would give scores at 21 or so.

An unfortunate challenge at picking at the tails here is one can train digit span - memory athletes drill this and I understand the record lies in the three figures.

Perhaps a natural test would be getting very smart but training naive people (IMOers?) to try this. If they're consistently scoring 15+, this is hard to reconcile with normalish assumptions (digit span wouldn't correlate perfectly with mathematical ability, so lots of 6 sigma+ results look weird), and vice versa.

Comment by thrasymachus on Comments on Power Law Distribution of Individual Impact · 2017-12-29T07:36:25.022Z · score: 11 (3 votes) · LW · GW

I'm aware of normalisation, hence I chose things which have some sort of 'natural cardinal scale' (i.e. 'how many Raven's do you get right' doesn't really work, but 'how many things can you keep in mind at once' is better, albeit imperfect).

Not all skew entails a log-normal (or some similar - assumedly heavy tailed) distribution. This applies to your graph for digit span you cite here. The mean of the data is around 5, and the SD is around 2. Having ~11% at +1SD (7) and about 3% at +2SD (9) is a lot closer to normal distribution land (or, given this is count data, a pretty well-behaved poisson/slightly overdispersed binomial) than a hypothetical log normal. Given log normality, one should expect a dramatically higher maximum score when you increase the sample size from 78 in the cited study to 2400 or so. Yet in the standardization sample of the WAIS III of this size no individual had greater than 9 in forward digit span (and no one higher than 8 in reverse). (This is, I assume, the foundation for the famous '7 plus or minus 2' claim.)

A lot turns on 'vary dramatically', but I think on most commonsense uses of this would not be it. I'd take reaction time data to be similar - although there is a 'long tail', this is a long tail of worse performance - and the tail isn't that long. So I don't buy claims I occasionally see made along the lines of 'Einstein was just miles smarter than a merely average physicist'.

Comment by thrasymachus on In defence of epistemic modesty · 2017-11-03T00:44:29.582Z · score: 14 (4 votes) · LW · GW

Sorry you disliked the post so much. But you might have liked it more had you looked at the bit titled 'community benefits to immodesty', where I talk about the benefits of people striking out outside expert consensus (but even if they should act 'as if' their contra-expert take was correct, they should nonetheless defer to it for 'all things considered' views).

Comment by thrasymachus on In defence of epistemic modesty · 2017-10-30T19:17:42.002Z · score: 31 (9 votes) · LW · GW

No. I chose him as a mark of self-effacement. When I was younger I went around discussion forums about philosophy, and people commonly named themselves after ancient greats like Socrates, Hume, etc. Given Thrasymachus's claim to fame is being rude, making some not great arguments, and getting spanked by Socrates before the real discussion started (although I think most experts think Socrates's techne based reply was pretty weak), I thought he would be a more accurate pseudonym,

In defence of epistemic modesty

2017-10-29T20:00:31.366Z · score: 42 (24 votes)
Comment by thrasymachus on Contra double crux · 2017-10-15T15:07:31.744Z · score: 7 (1 votes) · LW · GW

Sorry for misreading your original remark. Happy to offer the bet in conditional, i.e.:

Conditional on CFAR producing results of sufficient quality for academic publication (as judged by someone like Christiano or Karnofsky) these will fail to demonstrate benefit on a pre-specified objective outcome measure

Comment by thrasymachus on Contra double crux · 2017-10-14T23:20:10.371Z · score: 11 (2 votes) · LW · GW

Thanks for your reply. Given my own time constraints I'll decline your kind offer to discuss this further (I would be interested in reading some future synthesis). As consolation, I'd happily take you up on the modified bet. Something like:

Within the next 24 months CFAR will not produce results of sufficient quality for academic publication (as judged by someone like Christiano or Karnofsky) that demonstrate benefit on a pre-specified objective outcome measure

I guess 'demonstrate benefit' could be stipulated as 'p<0.05 on some appropriate statistical test' (the pre-specification should get rid of the p-hacking worries). 'Objective' may remain a bit fuzzy: the rider is meant to rule out self-report stuff like "Participants really enjoyed the session/thought it helped them". I'd be happy to take things like "Participants got richer than controls", "CFAR alums did better on these previously used metrics of decision making", or whatever else.

Happy to discuss further to arrive at agreeable stipulations - or, if you prefer, we can just leave them to the judges discretion.

Comment by thrasymachus on Contra double crux · 2017-10-14T21:45:08.879Z · score: 24 (7 votes) · LW · GW

I hope readers will forgive a 'top level' reply from me, it's length, and that I plan to 'tap out' after making it (save for betting). As pleasant as this discussion is, other demands pull me elsewhere. I offer a summary of my thoughts below - a mix of dredging up points I made better 3-4 replies deep than I managed in the OP, and to reply to various folks at CFAR. I'd also like to bet (I regret to decline Eli's offer for reasons that will become apparent, but I hope to make some agreeable counter-offers).

I persist in three main worries: 1) That double crux (or 'cruxes' simpliciter) are confused concepts; 2) It doesn't offer anything above 'strong consideration', and insofar as it is not redundant, framing in 'cruxes' harms epistemic practice; 3) The evidence CFAR tends to fall back upon to nonetheless justify the practice of double crux is so undermined that it is not only inadequate public evidence, but it is inadequate private evidence for CFAR itself.

The colloid, not crystal, of double crux

A common theme in replies (and subsequent discussions) between folks at CFAR and I is one of a gap in understanding. I suspect 'from their end' (with perhaps the exception of Eli) the impression is I don't quite 'get it' (or, as Duncan graciously offers, maybe it's just the sort of thing that's hard to 'get' from the written up forms): I produce sort-of-but-not-quite-there simulacra of double crux, object to them, but fail to appreciate the real core of double crux to which these objections don't apply. From mine, I keep trying to uncover what double crux is, yet can't find any 'hard edges': it seems amorphous, retreating back into other concepts when I push on what I think it is distinct, yet flopping out again when I turn to something else. So I wonder if there's anything there at all.

Of course this seeming 'from my end' doesn't distinguish between the two cases. Perhaps I am right double crux is no more than some colloid of conflated and confused concepts; but perhaps instead there is a a crystallized sense of what double crux is 'out there' that I haven't grasped. Yet what does distinguish these cases in my favour is that CFAR personnel disagree with one another about double crux.

For a typical belief which one might use double crux (or just 'single cruxing') should one expect to find one crux, or find multiple cruxes?

Duncan writes (among other things on this point):

The claim that I derive from "there's surprisingly often one crux" is something like the following: that, for most people, most of the time, there is not in fact a careful, conscious, reasoned weighing and synthesis of a variety of pieces of evidence. [My emphasis]

By contrast, Dan asserts in his explanation:

A typical belief has many cruxes. For example, if Ron is in favor of a proposal to increase the top marginal tax rate in the UK by 5 percentage points, his cruxes might include "There is too much inequality in the UK", "Increasing the top marginal rate by a few percentage points would not have much negative effect on the economy", and "Spending by the UK government, at the margin, produces value". [my emphasis]

This doesn't seem like a minor disagreement, as it flows through to important practical considerations. If there's often one crux (but seldom more), once I find it I should likely stop looking; if there's often many cruxes, I should keep looking after I find the first.

What would this matter, beyond some 'gotcha' or cheap point-scoring? This: I used to work in public health, and one key area is evaluation of complex interventions. Key to this in turn is to try and understand both that the intervention works but also how it works. The former without the latter raises introduces a troublesome black box: maybe elaborate high-overhead model for your intervention works through some much simpler causal path (c.f. that many schools of therapy with mutually incompatible models are in clinical equipoise, but appear also in equipoise with 'someone sympathetic listening to you'); maybe you mistake the key ingredient as intrinsic to the intervention where it is instead contingent on the setting so it doesn't work when this is changed (c.f. the external validity concerns that plague global health interventions).

In CFAR's case there doesn't seem a shared understanding of the epistemic landscape (or, at least, where cruxes lie within it) between 'practicioners'. It also looks to me there's not a shared understanding on the 'how it works' question - different accounts point in different directions: Vanvier seems to talk more about 'getting out of trying to win the argument mode to getting to the truth mode', Duncan emphasizes more potential rationalisations one may have for a belief, Eli suggests it may help locate cases where we differ in framing/fundamental reasons that are in common with more proximal reasons (i.e. the 'earth versus moon' hypothetical). Of course, it could do all of these, but I don't think CFAR has a way to tell. Finding the mediators would also help buttress claims of causal impact.

Cruxes contra considerations

I take it as a 'bad news' for an idea, whatever its role, if one can show it is a) a proposed elaboration of another idea, and b) yet this elaboration makes the idea worse. I offer an in theory reason to think 'cruxes' are inapt elaborations for 'considerations', a couple of considerations as to why 'double crux' might degrade epistemic practice, and a bet that, in fact, people who are 'double cruxing' (or just 'finding cruxes') are often not in fact using cruxes.

Call a 'consideration' something like this:

A consideration for some belief B is another belief X such that believing X leads one to assign a higher credence to B.

This is (unsurprisingly) broad, including stuff like 'reasons', 'data' and the usual fodder for bayesian updating we know and love. Although definitions of a 'crux' slightly vary, it seems to be something like this:

A crux for some belief B is another belief C such that if one did not believe C, one would not believe B.


A crux for some belief B is another belief C such that if one did not believe C, one would change one's mind about B.

'Changing one's mind' about B is not ultra-exact, but nothing subsequent turns on this point (one could just encode B in the first formulation as 'I do not change my mind about another belief (A)', etc.

The crux rule

I said in a reply to Dan given this idea of a crux, a belief should held no more strongly than its (weakest) crux (call this the 'crux rule'). He expressed uncertainty about whether this was true. I hope this derivation is persuasive:

¬C -> ¬B (i.e. if I don't believe C, I don't believe B - or, if you prefer, if I don't believe the crux, I should not 'not change my mind about' B)


B -> C (i.e. if I believe B, I must therefore believe C).

If B -> C, P(C) >= P(B): there is no possibility C is false yet B is true, yet there is a possibility where C is true and B is false (compare modus tollens to affirming the consequent).

So if C is a crux for B, one has inconsistent credences if one offers a higher credence for B than for C. An example: suppose I take "Increasing tax would cause a recession" as a crux for "Increasing taxes is bad" - if I thought increasing taxes would not cause a recession, I would not think increasing taxes is bad. Suppose my credence for raising taxes being bad is 0.9, and my credence for raising taxes causing a recession is 0.6. I'm inconsistent: if I assign a 40% chance raising taxes would not cause a recession, I should think there's at least a 40% chance raising taxes would not be bad, not 10%.

(In a multi-crux case with C1-n cruxes for B, the above argument applies to C1-n, so B must not be higher than any of them, and thus equal to or lower than the lowest. Although this is a bound, one may anticipate B's credence to be substantially lower, as the probability of a union of (mostly) independent cruxes approximates P(C1)*P(C2) etc.)

Note this does not apply to considerations, as there's no neat conditional parsing of 'consideration' in the same way as 'crux'. This also agrees with common sense: imagine some consideration one is uncertain of which nonetheless favours B over ¬B: one can be less confident of X than B.

Why belabour this logic and probability? Because it offers a test of intervention fidelity: whether people who are 'cruxing' are really using cruxes. Gather a set of people one takes as epistemically virtuous who 'know how to crux' to find cruxes for some of their beliefs. Then ask them to offer their credences for both the belief and the crux(s) for the belief. If they're always finding cruxes, there will be no cases where they offer higher credence for the belief than its associated crux(s).

I aver the actual proportion of violations of this 'crux rule' will be at least 25%. What (epistemically virtuous) people are really doing when finding 'cruxes' are strong considerations which they think gave them large updates toward B over ¬B. However, despite this they will often find their credence in the belief is higher than the supposed crux. I might think the argument from evil is the best consideration for atheism, but I may also hold a large number of considerations point in favour in atheism, such they work together to make me more confident of atheism than the soundness of the argument from evil. Readers (CFAR alums or not) can 'try this at home'. For a few beliefs 'find your cruxes'. Now offer credences for these - how often do you need to adjust these credences to obey the 'crux rule'? Do you feel closer to reflective equilibrium when you do so?

Even if CFAR can't bus in some superforecasters or superstar philosophers to try this on, they can presumably do this with their participants. I offer the following bet (and happy to haggle over the precise numbers):

(5-1 odds [i.e. favouring you].) From any n cases of beliefs and associated cruxes for CFAR alums/participants/any other epistemically virtuous group who you deem 'know cruxing', greater than n/4 cases will violate the crux rule.

But so what? Aren't CFAR folks already willing to accept often 'crux' (in Vanvier's words) 'degrades gracefully' into something like what I call a 'strong consideration'? Rather than castle-and-keep, isn't this more like constructing a shoddier castle somewhat nearby and knocking its walls down? To-may-to/To-mar-to?

Yet we already have words for 'things which push us towards a belief'. I used consideration, but we can also use 'reasons', or 'evidence' or whatever. 'Strong consideration' has 16 more characters than crux, but it has the benefits of its meaning being common knowledge, being naturally consonant with bayesianism, and accurately captures how epistemically virtuous people think and how they should be thinking. To introduce another term which is not common knowledge and forms either a degenerate or redundant version of this common knowledge term looks, respectfully, like bloated jargon by my lights.

If you think there's a crux, don't double crux, think again

Things may be worse than 'we've already got a better concept'. It looks plausible to me that teaching cruxes (or double crux) teaches bad epistemic practice. A contention I made in the OP is that as crux incidence is anti-correlated with epistemic virtue: epistemically virtuous people usually find in topics of controversy that the support for their beliefs is distributed over a number of considerations, without a clear 'crux', rather than they would change their mind in some matter based on a single not-that-resilient consideration. Folks at CFAR seem to (mostly) agree, e.g. Duncan's remarks:

I note that, if correct, this theory would indicate that e.g. your average LessWronger would find less value in double crux than your average CFAR participant (who shares a lot in common with a LessWronger but in expectation is less rigorous and careful about their epistemics). This being because LessWrongers try very deliberately to form belief webs like the first image [many-one, small edge weighs - T], and when they have a belief web like the third image [not-so-many-one, one much bigger edge -T] they try to make that belief feel to themselves as unbalanced and vulnerable as it actually is.

This suggests one's reaction on finding you have a crux should be alarm: "My web of beliefs doesn't look like what I'd expect to see from a person with good epistemics", and one's attitude towards 'this should be the crux for my belief' should be scepticism: "It's not usually the case some controversial matter depends upon a single issue like this". It seems the best next step in such a situation is something like this, "I'm surprised there is a crux here. I should check with experts/the field/peers to whether they agree with me that this is the crux of the matter. If they don't, I should investigate the other considerations suggested to bear upon this matter/reasons they may offer to assign lower weight to what I take to the be crux".

The meta-cognitive point is that it is important to not only get the right credences on the considerations, but also to the weigh these considerations rightly to form a good 'all things considered' credence on the topic. Webs of belief that greatly overweigh a particular consideration track truth poorly even they are accurate on what it (mis)takes as the key issue. In my experience among elite cognisers, there's seldom disagreement that a consideration bears upon a given issue. Disagreement seldom occurs about the direction of that consideration either: parties tend to agree a given consideration favours one view or another. Most of the action occurs at the aggregation: "I agree with you this piece of evidence favours your view, but I weigh it less than this other piece of evidence that favours mine."

Cruxing/double crux seems to give entirely wrong recommendations. It pushes one to try to find single considerations that would change their mind, despite this usually being pathological; it focuses subsequent thinking on those considerations identified as cruxes, instead of the more important issue of whether one is weighing these considerations too heavily; it celebrates when you and your interlocutor agree on the crux of your disagreement, instead of cautioning such cases often indicate you've both gotten things wrong.

The plural of plausibly biased anecdote is effectively no evidence

Ultimately, the crux (forgive me) is whether double crux actually works. Suppose 'meditation' is to 'relaxing' as I allege 'crux/double crux' is to 'consideration'. Pretend all the stuff you hear about 'meditation' is mumbo-jumbo that confuses the issue that the only good 'meditation' does is that it prompts people to relax. This would be regrettable, but meditation would still be a good thing even if its only parasitic on the good of relaxing. One might wonder if you could do something better than 'meditation' by focusing on actually valuable relaxing bit, but maybe this is one of those cases where the stuff around 'meditation' is a better route to get people to relax than targeting 'relaxing' directly. C.f. Duncan:

I think there's a third path here, which is something like "double crux may be an instrumentally useful tool in causing these admirable epistemic norms to take root, or to move from nominally-good to actually-practiced.

The evidence base for double crux (and I guess CFAR generally) seems to be something like this:

  • Lots of intelligent and reasonable people report cruxing/double crux was helpful for them. (I can somewhat allay Duncan's worry that the cases he observes might be explained by social pressure he generates - people have reported the same in conversations in which he is an ocean away).

  • Folks at CFAR observe many cases where double crux works, and although it might work particularly well between folks at CFAR (see Eli's comment) but in any case they still observe it to be handy with non-CFAR staff.

  • Duncan notes in a sham control test (i.e. double crux versus 'discussing the benefits of epistemic virtues').

  • Dan provides some participant data: about half 'find a double crux', and it looks like finding a disagreement, finding a double crux (or both) was associated with a more valuable conversation.

Despite general equanimity, Duncan noted distress at the 'lack of epistemic hygiene' around looking at double crux, principally (as I read him) that of excessive scepticism from some outside CFAR. With apologies to him (and the writer of Matthew 7), I think the concern is more plausible in reverse: whatever motes blemish outsider eyes do not stop them seeing the beams blocking CFAR's insight. It's not only the case that outsiders aren't being overly sceptical in doubting this evidence, CFAR is being overly credulous taking it as seriously as they do. Consider this:

  1. In cases where those who are evaluating the program are those involved in delivering the intervention, and they expectedly benefit the better the results, there's a high risk of bias. (c.f. blinding, conflict of interest)

  2. In cases where individuals enjoy some intervention (and often spent a quite a lot of money to participate) there's a high risk of bias for their self-report. (c.f. choice-supportive bias, halo effect, among others).

  3. Neither good faith nor knowledge of a potential bias risk do not, by themselves, help one much to avoid this bias.

  4. Prefer hard metrics with tight feedback loops when trying to perform well at something.

  5. Try and perform some reference class forecasting to avoid getting tricked by erroneous insider views (but I repeat myself).

What measure of credulity should a rationalist mete out to an outsider group with a CFAR-like corpus of evidence? I suggest it would be meagre indeed. One can recite almost without end interventions with promising evidence fatally undercut by minor oversights or bias (e.g. inadequate allocation concealment in an RCT). In the class of interventions where the available evidence has multiple, large, obvious bias risks, the central and modal member of this class is an intervention with no impact.

We should mete out this meagre measure of credulity to ourselves: we should not on the one hand remain unmoved by the asseveration of a chiropractor that they 'really see it works', yet take evidence of similar quality and quantity to vindicate rationality training. CFAR's case is unpersuasive public evidence. I go further: it's unpersuasive private evidence too. In the same way we take the chiropractor be irrational if they don't almost-entirely discount their first-person experience of chiropractic successes when we inform them of the various cognitive biases that undercut the evidentiary value of this experience, we should expect a CFAR instructor or alum, given what they already know about rationality, to almost entirely discount these sources of testimonial evidence to judge whether double crux works.

Yet this doesn't happen. Folks at CFAR tend to lead with this anecdata when arguing that double crux works. This also mirrors 'in person' conversations I have, where otherwise epistemically laudable people cite their personal experience as what convinces them of the veracity of a particular CFAR technique. What has a better chance of putting one in touch with reality about whether double crux (or CFAR generally) works is the usual scientific suspects: focusing on 'hard outcomes', attempting formal trials, randomisation, making results public, and so forth. That this generally hasn't happened across the time of CFARs operation I take to be a red flag.

For this reason I respectfully decline Eli's suggestion to make bets on whether CFAR will 'stick with' double crux (or something close to it) in the future. I don't believe CFAR's perception of what is working will track the truth, and so whether or not it remains 'behind double crux' is uninformative for the question of whether double crux works. I'm willing to offer bets against whether CFAR will gain 'objective' evidence of efficacy, and bet in favour of the null hypothesis for these:

(More an error bounty than a bet - first person to claim gets £100) CFAR's upcoming "EA impact metrics report" will contain no 'objective measures' (defined somewhat loosely - objective measure is something like "My income went up/BMI went down/independent third party assessor rated the conversation as better", not things along the lines of, "Participants rate the workshop as highly valuable/instructor rates conversations as more rational/ etc."

(3-to-1): CFAR will not generate in the next 24 months any peer reviewed literature in psychology or related fields (stipulated along the lines of, "either published in the 'original reports' section of a journal with impact factor >1 or presenting at an academic conference."

(4 to 1): Conditional on a CFAR study getting past peer review, it will not show significantly positive effects on any objective, pre-specified outcome measure.

I'm also happy to offer bets on objective measures of any internal evaluations re. double crux or CFAR activity more broadly.

Comment by thrasymachus on Contra double crux · 2017-10-10T00:04:29.637Z · score: 4 (2 votes) · LW · GW

I also notice that I can't predict whether you'll look at the "prioritize discussion based on the slope of your possible update combined with the other party's belief" version that I give here and say "okay, but that's not double crux" or "okay, but the motion of double crux doesn't point there as efficiently as something else" or "that doesn't seem like the right step in the dance, tho."

I regret it is unclear what I would say given what I have written, but it is the former ("okay, but that's not double crux"). I say this for the following reasons.

  1. The consideration with the greatest slope need not be a crux. (Your colleague Dan seems to agree with my interpretation that a crux should be some C necessary for ones attitude over B, so that if you changed your mind about C you'd change your mind about B).

  2. There doesn't seem to be a 'double' either: identifying the slopiest consideration regarding ones own credence doesn't seem to demand comparing this to the beliefs of any particular interlocutor to look for shared elements.

I guess (forgive me if I'm wrong) what you might say is that although what you describe may not satisfy what was exactly specified in the original introduction to double crux, this was a simplification and these are essentially the same thing. Yet I take what distinguishes double crux over related and anodyne epistemic virtues (e.g. 'focus on important less-resilient considerations', 'don't act like a lawyer') is the 'some C for which if ¬C then ¬B' characteristic. As I fear may be abundantly obvious, I find eliding this distinction confusing rather than enlightening: if (as I suggest) the distinguishing characteristic of double crux neither works as good epistemic tool nor good epistemic training, that there may be some nearby epistemic norm that does one or both of these is little consolation.

Comment by thrasymachus on Contra double crux · 2017-10-09T23:37:35.462Z · score: 4 (1 votes) · LW · GW

Hello Dan,

I'm not sure whether these remarks are addressed 'as a reply' to me in particular. That you use the 'marginal tax rate in the UK' example I do suggests this might be meant as a response. On the other hand, I struggle to locate the particular loci of disagreement - or rather, I see in your remarks an explanation of double crux which includes various elements I believe I both understand and object to, but not reasons that argue against this belief (e.g. "you think double crux involves X, but actually it is X*, and thus your objection vanishes when this misunderstanding is resolved", "your objection to X is mistaken as Y", etc.) If this is a reply, I apologise for not getting it; if it is not, I apologise for my mistake.

In any case, I take the opportunity to suggest to concretely identify one aspect of my disagreement:

A typical belief has many cruxes. For example, if Ron is in favor of a proposal to increase the top marginal tax rate in the UK by 5 percentage points, his cruxes might include "There is too much inequality in the UK", "Increasing the top marginal rate by a few percentage points would not have much negative effect on the economy", and "Spending by the UK government, at the margin, produces value". If he thought that more inequality would be good for society then he would no longer favor increasing the top marginal rate. If he thought that increasing the top marginal rate would be disastrous for the UK economy then he would no longer favor increasing it (even if he didn't change his mind about there being too much equality). If he thought that marginal government spending was worthless or harmful then he would no longer favor increasing taxes.

This seems to imply agreement with my take that cruxes (per how CFAR sees them) have the 'if you change your mind about this, you should change your mind about that', and so this example has the sequence think-esque characteristic that these cruxes are jointly necessary for ron's belief (i.e. if Ron thinks ¬A, ¬B, or ¬C, he should change his mind about the marginal tax rate). Yet by my lights it seems more typical considerations like these exert weight upon the balance of reason, but not of such strength that their negation provides a decisive consideration against increasing taxes (e.g. it doesn't seem crazy for Ron to think "Well, I don't think inequality is a big deal, but other reasons nonetheless favour raising taxes", or "Even though I think marginal spending by the UK government is harmful, this negative externality could be outweighed by other considerations").

I think some harder data can provide better information than litigating hypothetical cases. If the claim that a typical belief has many cruxes, one should see that if one asks elite cognisers to state their credence for a belief, and then state their credences for the most crucial few considerations regarding it, the credence for the belief should only be very rarely higher than the lowest credence among the considerations. This is because if most beliefs have many (jointly necessary) cruxes which should usually comprise at least the top few considerations, and thus this conjunction is necessary (but not sufficient) for believing B, and P(one crux) >= P(conjunction of cruxes). In essence ones credence in a belief should be no greater than ones weakest crux (I guess usually the credence in the belief of a sequence-thinking argument should generally approximate a lower credence set by P(crux1)*P(crux2) etc, as these are usually fairly independent.)

In contrast, if I am closer to the mark, one should fairly commonly see the credence for the belief be higher than the lowest credence of the set of important considerations. If each consideration offers a bayesian update favouring B, a set of important considerations that support B may act together (along with other less important considerations) to increase its credence such that one is more confident of B than of some (or all) of the important considerations that support it.

I aver relevant elite cognisers (e.g. superforecasters, the philosophers I point to) will exhibit the property I suggest. I would also venture that when reasonable cognisers attempt to double crux, their credences will also behave in the way I predict.

Comment by thrasymachus on Contra double crux · 2017-10-09T22:47:32.672Z · score: 5 (2 votes) · LW · GW

Thanks for presenting this helpful data. If you'll forgive the (somewhat off topic) question, I understand both that you are responsible for evaluation of CFAR, and that you are working on a new evaluation. I'd be eager to know what this is likely to comprise, especially (see various comments) what evidence (if any) is expected to be released 'for public consumption'?

Comment by thrasymachus on Contra double crux · 2017-10-09T22:21:17.934Z · score: 9 (3 votes) · LW · GW

Thank you for your gracious reply. I interpret a couple of overarching themes in which I would like to frame my own: the first is the 'performance issue' (i.e. 'How good is double crux at resolving disagreement/getting closer to the truth'); the second the 'pedagogical issue' (i.e. 'how good is double crux at the second order task of getting people better at resolving disagreement/getting closer to the truth'). I now better understand you take the main support from double crux to draw upon the latter issue, but I'd also like to press on some topics about the former on which I believe we disagree.

How well does double crux perform?

Your first two diagrams precisely capture the distinction I have in mind (I regret not having thought to draw my own earlier). If I read the surrounding text right (I'm afraid not to know what 'fompmott' means, and google didn't help me), you suggest that even if better cognisers find their considerations form a denser web like the second diagram, double-crux amenable 'sparser' webs are still common in practice, perhaps due to various non-rational considerations. You also add:

I note that, if correct, this theory would indicate that e.g. your average LessWronger would find less value in double crux than your average CFAR participant (who shares a lot in common with a LessWronger but in expectation is less rigorous and careful about their epistemics). This being because LessWrongers try very deliberately to form belief webs like the first [I think second? - T] image, and when they have a belief web like the third image they try to make that belief feel to themselves as unbalanced and vulnerable as it actually is. Ergo, LessWrongers would find the "Surprise! You had an unjustified belief!" thing happening less often and less unexpectedly.

This note mirrors a further thought I had (c.f. Ozymandias's helpful remark in a child about sequence versus cluster thinking). Yet I fear this poses a further worry for the 'performance issue' of double crux, as it implies that the existence of cruxes (or double cruxes) may be indicative of pathological epistemic practices. A crux implies something like the following:

  1. You hold some belief B you find important (at least, important enough you think it is worth your time to discuss).

  2. Your credence in B depends closely on some consideration C.

  3. Your credence in C is non-resilient (at least sufficiently non-resilient you would not be surprised to change your mind on it after some not-unduly-long discussion with a reasonable interlocutor).*

* What about cases where one has a resilient credence in C? Then the subsequent worries do not apply. However, I suspect these cases often correspond to "we tried to double crux and we found we couldn't make progress on resolving our disagreement about theories of truth/normative ethics/some other foundational issue".

It roughly follows from this you should have low resilience in your credence of B. As you note, this is vulnerable, and knowing one had non-resilient credences in important Bs is to be avoided.

As a tool of diagnosis, double crux might be handy (i.e. "This seems to be a crux for me, yet cruxes aren't common among elite cognisers - I should probably go check whether they agree this is the crux of this particular matter, and if not maybe see what else they think bears upon B besides C"). Yet (at least per the original exposition) it seems to be more a tool for subsequent 'treatment'. Doing so could make things worse, not better.

If X and Y find they differ on some crux, but also understand that superior cognisers tend not to have this crux, and distribute support across a variety of considerations, it seems a better idea for them to explore other candidate considerations rather than trying to resolve their disagreement re. C. If they instead do the double-cruxy thing and try and converge on C, they may be led up the epistemic garden path. They may agree with one another on C (thus B), and thus increase their resilience of C (thus B), yet they also confirm a mistaken web of belief around B which wrongly accords too much weight to C. If (as I suggest) at least half the battle on having good 'all things considered' attitudes to recondite matters comprises getting the right weights for relevant considerations on the matter, double crux may celebrate them converging further away from the truth. (I take this idea to be expressed in kernel in Ozymandias's worry of double crux displacing more-expectedly-accurate cluster thinking with less-expectedly-accurate sequence thinking).

How good is double crux at 'levelling people up at rationality'

The substantial independence of the 'performance issue' from the pedagogical issue'

In the same way practising scales may not be the best music, but make one better at playing music, double crux may not be the best discussion technique, but make one better at discussions. This seems fairly independent of its 'object level performance' (although I guess if the worry above is on the right track, we would be very surprised if a technique that on the object level leads beliefs to track truth more poorly nonetheless has a salutatory second-order effect).

Thus comparisons to practices of elite philosophers (even if they differ) are inapposite - especially, as I understand from one of them, the sort of superior pattern I observe occurs only at a far right tail even among philosophers (i.e. 'world-class' as you write, rather then 'good', as I write in the OP). It is obviously a great boon if I could get some fraction more like someone like Askell or Shulman without either their profound ability or the time they have invested in these practices.

On demurring the 'double crux challenge'

I regret I don't think it would be hugely valuable to 'try double crux' with an instructor in terms of resolving this disagreement. One consideration (on which more later) is that conditional on me not being persuaded by a large group of people who self-report double crux is great, I shouldn't change my mind (for symmetry reasons) if this number increases by one other person, or it increases by including me. Another is that the expected yield may not be great, at least in one direction: although I hope I am not 'hostile' to double crux, it seems one wouldn't be surprised if it didn't work with me, even if its generally laudable.

Yet I hope I am not quite as recalcitrant as 'I would not believe until I felt the stigmata with my own hands'. Apart from a more publicly legible case (infra), I'm a bit surprised at the lack of 'public successes' of double cruxing (although this may confuse performance versus pedagogy). In addition to Constantin, Raemon points to their own example with gjm. Maybe I'm only seeing what I want to, but I get a similar impression. They exhibit a variety of laudable epistemic practices, but I don't see a crux or double crux (what they call 'cruxes' seem to be more considerations they take to be important).

The methods of rational self-evaluation

You note a head-to-head comparison between double crux and an approximate sham-control seemed to favour double crux. This looks like interesting data, and it seems a pity it emerges in the depths of a comment thread (ditto the 'large n of successes') rather than being written up and presented - it seems unfortunate that the last 'public evaluation report' is about 2 years old. I would generally urge trying to produce more 'public evidence' rather than the more private "we've generally seen this work great (and a large fraction of our alums agree!)"

I recognise that "Provide more evidence to satisfy outside sceptics" should not be high on CFAR's priority list. Yet I think it is instrumental to other important goals instead. Chiefly: "Does what we are doing actually work?"

You noted in your initial reply undercutting considerations to the 'we have a large n of successes', yet you framed this in way that these would often need to amount to a claim of epistemic malice (i.e. 'either CFAR is lying or participants are being socially pressured into reporting a falsehood'). I don't work at a rationality institute or specialise in rationality, but on reflection I find this somewhat astonishing. My impression of cognitive biases were that they were much more insidious, that falling prey to them was the rule rather than the exception, and that sincere good faith was not adequate protection (is this not, in some sense, what CFAR casus belli is predicated upon?)

Although covered en passant, let me explicitly (although non-exhaustively) list things which might bias more private evidence of the type CFAR often cites:

  1. CFAR staff (collectively) are often responsible for developing the interventions they hope will improve rationality. One may expect them to be invested in them, and more eager to see that they work than see they don't (c.f. why we prefer double-blinding over single-blinding).

  2. Other goods CFAR enjoys (i.e. revenue/funding, social capital) seem to go up the better the results of their training. Thus CFAR staff have a variety of incentives pushing them to over-report how good their 'product' is (c.f. why conflicts of interest are bad, the general worries about pharma-funded drug trials).

  3. Many CFAR participants have to spend quite a lot of money (i.e. fees and travel) to attend a workshop. They may fear looking silly if it turns out after all this it didn't do anything, and so incentivised to assert it was much more helpful than it actually was (c.f. choice supportive bias).

  4. There are other aspects of CFAR workshops that participants may enjoy independent of the hoped-for improvement of their rationality (e.g. hanging around interesting people like them, personable and entertaining instructors, romantic entanglements). This extraneous benefits may nonetheless bias upwards their estimate of how effective CFAR workshops are at improving their rationality (c.f. halo effect).

I am sure there are quite a few more. One need not look that hard to find lots of promising studies supporting a given intervention undermined by any one of these.

The reference class of interventions with "a large corpus of (mainly self-reported) evidence of benefit, but susceptible to these limitations" is dismal. It includes many branches of complementary medicine. It includes social programs (e.g. 'scared straight') that we now know to be extremely harmful. It includes a large number of ineffective global poverty interventions. Beyond cautionary tales, I aver these approximate the modal member of the class: when the data is so subjective, and the limitations this severe, one should expect the thing in question doesn't actually work after all.

I don't think this expectation changes when we condition on the further rider "And the practicioners really only care about the truth re. whether the intervention works or not." What I worry about going on under the hood is a stronger (and by my lights poorly substantiated) claim of rationalist exceptionalism: "Sure, although cognitive biases plague entire fields of science and can upend decades of results, and we're appropriately quick to point out risk of bias of work done by outsiders, we can be confident that as we call ourselves rationalists/we teach rationality/we read the sequences/etc. we are akin to Penelope refusing her army of suitors - essentially incorruptible. So when we do similarly bias-susceptible sorts of things, we should give one another a pass."

I accept 'gold standard RCTs' are infeasible (very pricey, and how well can one really do 'sham CFAR'?) yet I aver there is quite a large gap between this ideal of evidence and the actuality (i.e. evidence kept in house, and which emerges via reference in response to criticism) which could be bridged by doing more write-ups, looking for harder metrics that put one more reliably in touch with reality, and so on. I find it surprisingly incongruent that the sort of common cautions about cognitive biases - indeed, common cautions that seem predicates for CFAR's value proposition (e.g. "Good faith is not enough", "Knowing about the existence of biases does not make one immune to them", Feynmann's dictum about 'you are the easiest person to fool'), are not reflected in its approach to self-evaluation.

If nothing else, opening up more of CFAR's rationale, evidence, etc. to outside review may allow more benefits of outside critique. Insofar as it is the case you found this exchange valuable, one may anticipate greater benefit from further interaction with higher-quality sceptics.

Comment by thrasymachus on Contra double crux · 2017-10-09T18:16:08.720Z · score: 3 (1 votes) · LW · GW

I guess my overall impression is that including the cases you specify in a double cruxy style look more like epicycles by my lights rather than helpful augmentations to the concept of double crux.

Non-common knowledge cruxes

I had a sentence in the OP on crux asymmetry along the lines of 'another case may be is X believes they have a crux for B which Y is unaware of'. One may frame this along the lines of an implicit crux of 'there's no decisive consideration that changes my mind about B' for which a proposed 'silver bullet' argument would constitute disagreement.

One of the past-times of my mis-spent youth was arguing about god on the internet. A common occurrence was Theist and Atheist would meet, and both would offer their 'pet argument' which they took to be decisive for A/Theism. I'm not sure they were high-quality discussions, so I'd not count it as a huge merit if they satisfy double crux.

I guess this ties back to my claim that on topics on which reasonable people differ, decisive considerations of this type should be very rare. One motivation for this would veer along social epistemiological lines: a claim about a decisive consideration seems to require some explanation as others nonetheless hold the belief the decisive consideration speaks against. Explaining why your interlocutor is not persuaded is easy - they may simply have not come across it. Yet on many of the sort of recondite topics of disagreement one finds that experts are similarly divided to the laity, and usually the experts who disagree with you are aware of the proposed decisive consideration you have in mind (e.g. the non-trivial proportion of economists who are aware of the Laffer curve yet nonetheless support higher marginal tax rates, etc.). Although it is possible there's some systemic cause/bias/whatever which could account for why this section of experts are getting this wrong, it seems the more common explanation (also favoured by an outside view) is that you overrate the importance of the consideration due to inadequate knowledge of rebutting/undercutting defeaters.

I probably have a non-central sample of discussions I observe, so it may be the case there are a large family of cases of 'putative decisive considerations unknown by one party'. Yet in those cases I don't think double cruxing is the right 'next step'. In cases where the consideration has in mind is widely deemed to settle the matter by the relevant body of experts, it seems to approximate a 'lets agree to check it on Wikipedia case' (from a semi recent conversation, "I'm generally quite sympathetic to this particular formulation of the equal weight view on disagreement" "Oh the field has generally turned away from that due to work by Bloggs showing this view can be dutch-booked?" "If that is so that's a pretty decisive show-stopper, can you point me to it?" /end conversation). In cases where the decisive consideration X has in mind is not held as decisive by the expert body, that should be a red flag to X, and X and Y's time, instead of being spent inexpertly hashing out this consideration, is better spent looking the wider field of knowledge to see likely more sophisticated treatment of the same.

Conjunctive cruxes

I agree one could summarise the case in where many small pieces of evidence provide support for the belief could be summarized into some wider conjunction (e.g. "I'd believe god exists if my credence in the argument from evil goes down by this much, and credence in the argument from design goes up by this much") which could be a crux for discussion.

Yet in such cases there's a large disjunction of conjunctions that would lead to a similar shift in credence, as each consideration likely weights at least someone independently on the scales of reason (e.g. I'd believe god exists if my credence in AfE goes down by X, and credence in the argument from design goes up by Y, or credence in AfE goes down by X-e, and credence in AfD goes up by Y+f, or credence in AfE goes down by X-e, credence in AfD goes up by Y, and credence in argument from religious disagreement goes down by Z, etc. etc.) Although there are not continua in practice due to granularity in how we store credences (I don't back myself to be more precise than the first significant digit in most cases), the size of this disjunctive set grows very rapidly with number of relevant considerations. In consequence, there isn't a single neat crux to focus subsequent discussion upon.

I don't think discussion is hopeless in these cases. If X and Y find (as I think they should in most relevant cases) their disagreement arises from varying weights they place on a number of considerations that bear upon B and ¬B, they can prioritize considerations to discuss which they differ the most on, and for which it appears their credences are the least resilient (I guess in essence to optimise expected d(credence)/dt). This is what I observe elite cognisers doing, but this doesn't seem to be double crux to me.

Comment by thrasymachus on Contra double crux · 2017-10-09T17:12:32.863Z · score: 3 (1 votes) · LW · GW

I guess there might be a selection effect: 'mature philosophers' might have spent a lot of time hashing our their views at earlier stages (e.g. undergrad, graduate school). So it may not be that surprising to find in the subject of their expertise their credences on the issues are highly resilient such that they change their mind rarely, and only after considerable amounts of evidence gathered over a long time.

Good data would be whether outside of this whether these people are good at hashing out cases where they have less resilient credences, but these cases will seldom come up publicly (Putnam and Russell are famed for changing their view often, but it is unclear how much 'effort' that took or whether it depended on interlocutors). I can offer my private experience of some of these exceptional philosophers that they are exceptional at this, but I anticipate reasonable hesitance of this type of private evidence.

Comment by thrasymachus on Contra double crux · 2017-10-08T22:58:28.495Z · score: 13 (4 votes) · LW · GW

I'm not sure I can provide exactly what you're after: although philosophers do change their minds, they usually explain it in terms of 'I was persuaded by these considerations' rather than 'I talked with X for a while about it' - and often they hold many-many discussions in the literature. I could offer my own anecdata about how I changed my mind on various philosophical topics after discussing the matter with an elite philosopher, but I'm hardly one myself.

Perhaps the closest example that springs to mind is Wittgenstein. His initial masterwork was the Tractatus, one of the touchstones of logical positivism, a leading approach to analytic philosophy in the early twentieth century. His second masterwork, composed in later life (Philosophical investigations) is generally held to repudiate many of the claims (as well as the overall direction) of the Tractatus. He credits Piero Sraffa (admittedly an economist) for 'most of the consequential ideas' in his foreword.

Comment by thrasymachus on Contra double crux · 2017-10-08T22:36:27.371Z · score: 29 (11 votes) · LW · GW

Hello Duncan,

My thanks for your reply. I apologise if my wording in the OP was inflammatory or unnecessarily 'triggering' (another commenter noted an 'undertone of aggression', which I am sorry for, although I promise it wasn't intended - you are quoted repeatedly as you wrote the canonical exposition for what I target in the OP, rather than some misguided desire to pick a fight with you on the internet). I hope I capture the relevant issues below, but apologies in advance if I neglect or mistake any along the way.

CFAR's several hundred and the challenge of insider evidence

I was not aware of the of the several hundred successes CFAR reports of double crux being used 'in the wild'. I'm not entirely sure whether the successes are a) those who find double crux helpful or b) particular instances of double crux resolving disagreement, but I think you would endorse plenty of examples of both. My pretty sceptical take on double crux had 'priced in' the expectation of CFAR instructors at least some/many alums thought it was pretty nifty.

You correctly anticipate the sort of worries I would have about this sort of evidence. Self-reported approbation from self-selected participants is far from robust. Given branches of complementary medicine can probably tout thousands to millions of 'positive results' and happy customers, yet we know it is in principle intellectually bankrupt, and in practice performs no better than placebo in properly conducted trials. (I regret to add replies along the lines of, "if you had received the proper education in the technique you'd - probably - see it works well", or "I'm a practicioner with much more experience than any doubter in terms of using this, and it works in my experience" also have analogies here).

I don't think one need presume mendacity on the part of CFAR, nor gullibility on the part of workshop attendees, to nonetheless believe this testimonial evidence isn't strongly truth-tracking: one may anticipate similarly positive reports in worlds where (perhaps) double crux doesn't really work, but other stuff CFAR practices does work, and participants enjoy mingling with similarly rationally minded participants, may have had to invest 4 figure sums to get on the workshop, and so on and so forth. (I recall CFAR's previous evaluation had stupendous scores on self-reported measures, but more modest performance on objective metrics).

Of course, unlike complementary medicine, double crux does not have such powerful disconfirmation as 'violates known physics' or 'always fails RCTs'. Around the time double crux was proposed I challenged double crux on theoretical grounds (i.e. double cruxes should be very rare), this post was prompted by some of the dissonance on previous threads, but also the lack of public examples of double crux working. Your experience of the success of double crux in workshops is essentially private evidence (at least for now): in the same way it is hard to persuade me of its validity, it is next to impossible for me to rebut it. I nonetheless hope other lines of inquiry are fruitful.

How sparse is a typical web of belief?

One is the theoretical point. I read in your reply disagreement on the 'double cruxes' should be rare point ("... the number of people who have been surprised to discover that double cruxes do in fact exist and are findable an order of magnitude more often than one would have naively guessed"). Although you don't include it as a crux in your reply, it looks pretty crucial to me. If cruxes are as rare as I claim, double cruxing shouldn't work, and so the participant reports are more likely to have been innocently mistaken.

In essence, I take the issue to be around what web of belief surrounding a typical subject of disagreement looks like. It seems double crux is predicated on this being pretty sparse (at least in terms of important considerations): although lots of beliefs might have some trivial impact on your credence B, B is mainly set by small-n cruxes (C), which are generally sufficient to change ones mind if ones attitude towards them changes.

By contrast, I propose the relevant web tends to be much denser (or, alternatively, the 'power' of the population of reasons that may alter ones credence in B is fairly evenly distributed). Credence in B arises from a large number of considerations that weigh upon it, each of middling magnitude. Thus even if I am persuaded one is mistaken, my credence in B does not change dramatically. It follows that 'cruxes' are rare, and so two people happening to discover their belief on some recondite topic B is principally determined by some other issue (C), and it is the same for both of them is rare.

This is hard to make very crisp, as (among others) the 'space of all topics reasonable people disagree' are hard to pin down. Beyond appeals to my own experience and introspection ("do you really find your belief in (let's say, some political view like gay marriage or abortion) depends on a single consideration to such a degree that, if it was refuted, you would change your view?) I'd want to marshal a couple of other considerations.

  1. When one looks at a topic in philosophy, or science, or many other fields of enquiry, one usually sees a very one-to-many relationship of the topic to germane considerations. A large number of independent lines of evidence support the theory of evolution; a large number of arguments regarding god's existence in philosophy receive scrutiny (and in return they spawn a one-to-many relationship of argument to objections, objection to counter-objections). I suggest this offers analogical evidence in support of my thesis.

  2. Constantin's report of double cruxing (which has been used a couple of times as an exemplar in other threads) seems to follow the pattern I expect. I struggle to identify a double-crux in the discussion Constatin summarizes: most of the discussion seems to involve whether Salvatier's intellectual project is making much progress, with then a host of subsidiary considerations (e.g. how much to weigh 'formal accomplishments', the relative value of more speculative efforts on far future considerations, etc.), but it is unclear to me if Constantin was persuaded Salvatier's project was making good progress this would change her mind about the value of the rationalist intellectual community (after all, one good project may not be adequate 'output') or vice versa (even if Salvatier recognises his own project was not making good progress, the rationality community might still be a fertile ground to cultivate his next attempt, etc.)

What comprises double-crux?

I took the numbered list of my counter-proposal to have 25% overlap with double crux (i.e. realising your credences vary considerably), not 85%. Allow me to be explicit on how I see 2-4 in my list are in contradistinction to the 'double crux algorithm':

  • There's no assumption of an underlying single 'crux of the matter' between participants, or for either individually.

  • There's no necessity for a given consideration (even the strongest identified) to be individually sufficient to change ones mind about B

  • There's also no necessity for the strongest considerations proposed by X and Y to have common elements.

  • There's explicit consideration of credence resilience. Foundational issues may by 'double cruxes' in that (e.g.) my views on most applied ethics questions would change dramatically if I were persuaded of the virtue ethics my interlocutor holds, but one often makes more progress discussing a less resilient non-foundational claim even if 'payoff' in terms of the subsequent credence change in the belief of interest is lower.

This may partly be explained by a broader versus narrower conception of double crux. I take the core idea of double crux to be the 'find some C for which your disagreement over B relies upon, then discuss C' (this did, in my defense, comprise the whole of the 'how to play' section in the initial write-up). I take you to holding a broader view, where double crux incorporates other related epistemic practices, and it has value in toto.

My objection is expressly this. Double crux is not essential for these incorporated practices. So one can compare discussion with the set of these other practices to this set with the addition of double crux. I aver the set sans double crux will lead to better discussions.

Pedagogy versus performance

I took double crux was mainly being proposed as a leading strategy to resolve disagreement. Hence the comparison to elite philosophers was to suggest it wasn't a leading strategy by pointing to something better. I see from this comment (and the one you split off into its own thread) you see it more as a more a pedagogical role - even if elite performers do something different, it does valuable work in improving skills. Although I included a paragraph about its possible pedagogical value (admittedly one you may have missed as I started it with a self-indulgent swipe at the rationalist community), I would have focused more on this area had I realised it was CFAR's main contention.

I regret not to surprise you with doubts about the pedagogical value as well. This mostly arises from the above concerns: if double cruxes are as rare as I propose, it is unclear how searching for them is that helpful an exercise. A related worry (related to the top of the program) is this seems to entail increasing reliance on private evidence regarding whether the technique works: in principle objections to the 'face value' of the technique apply less (as it is there to improve skills rather than a proposal for what the 'finished article' should look like); adverse reports from non-CFAR alums don't really matter (you didn't teach them, so it is no surprise they don't get it right). What one is left with is the collective impressions of instructors, and the reports of the students.

I guess I have higher hopes for transparency and communicability of 'good techniques'. I understand CFAR is currently working on further efforts to evaluate itself. I hope to be refuted by the forthcoming data.

Contra double crux

2017-10-08T16:29:35.193Z · score: 10 (2 votes)
Comment by thrasymachus on Reframing Average Utilitarianism · 2016-12-10T06:56:27.940Z · score: 3 (3 votes) · LW · GW

The line of reply I tend to make here goes something along these lines:

If you're looking at a veil of ignorance, and choosing between world A or world B, it seems you should be also veiled to whether or not you exist. Whatever probability p you have to exist in world B, you have 10000p the probability of existing in world A, because it has 10000x the number of existing people. So opting for world B seems a much better bet.

As an aside, average views tend not to be considered 'goers' by those who specialise in population ethics, but for different reasons:

1) It looks weird with negative numbers. You can make a world where everyone has lives not worth living (-10, say) by adding people whose lives are also not worth living, but not quite as bad (e.g. -9).

2) It also looks pretty weird with positive numbers. Mere addition seems like it should be okay, even if it drags down the average.

3) Inseparability also looks costly. If the averaging is over all beings, then the decision as to whether humanity should euthanise itself becomes intimately sensitive to whether there's an alien species in the next supercluster who are blissfully happy.

Comment by thrasymachus on Sleepwalk bias, self-defeating predictions and existential risk · 2016-04-22T22:21:07.722Z · score: 1 (1 votes) · LW · GW

I'm not sure. It seems important to see whether there is sleepwalk bias is to try and gather a representative sample of predictions/warnings and see how they go. Yet this is pretty hard to do: I can think of examples (like those mentioned in the post) where the disaster was averted, but I can think of others where the disaster did happen despite warnings (I'd argue climate change fits into this category, for example).

Comment by thrasymachus on Open Thread, Dec. 28 - Jan. 3, 2016 · 2016-01-26T19:46:17.062Z · score: 1 (1 votes) · LW · GW

(Sorry for delay, and thanks for the formatting note.)

My knowledge is not very up to date re. machine medicine, but I did get to play with some of the commercially available systems, and I wasn't hugely impressed. There may be a lot more impressive results yet to be released commercially but (appealing back to my priors) I think I would have heard of it as it would be a gamechanger for global health. Also, if fairly advanced knowledge work of primary care can be done by computer, I'd expect a lot of jobs without the protective features of medicine to be automated.

I agree that machine medicine along the lines you suggest will be superior to human performance, and I anticipate this to be achieved (even if I am right and it hasn't already happened) fairly soon. I think medicine will survive less by the cognitive skill required, but rather though technical facility and social interactions, where machines comparably lag (of course, I anticipate they will steadily get better at this too).

I grant a hansonian account can accomodate this sort of 'guided by efficacy' data I suggest by 'pretending to actually try' considerations, but I would suggest this almost becomes an epicycle: any data which supports medicine being about healing can be explained away by the claim that they're only pretending to be about healing as a circuitous route to signalling. I would say the general ethos of medicine (EBM, profileration of trials) looks like pro tanto reasons in favour about being about healing, and divergence from this (e.g. what happened to semmelweis, other lags) is better explained by doctors being imperfect and selfish, and patients irrational, rather than both parties adeptly following a signalling account.

But I struggle to see what evidence could neatly distinguish between these cases. If you have an idea, I'd be keen to hear it. :)

I agree with the selection worry re. Metamed's customers: they also are assumedly selected from people who modern medicine didn't help, which may also have some effects (not to mention making Metameds task harder, as their pool will be harder to treat than unselected-for-failure cases who see the doctor 'first line'). I'd also (with all respect meant to the staff of Metamed) suggest staff of Metamed may not be the most objective sources of why it failed: I'd guess people would prefer to say their startups failed because of the market or product market fit, rather than 'actually, our product was straight worse than our competitors'.

Comment by thrasymachus on Beware surprising and suspicious convergence · 2016-01-26T19:25:11.632Z · score: 1 (1 votes) · LW · GW

I put an inline link in the post. Have I missed a norm about puttting related posts I have written in the post more prominently?

Beware surprising and suspicious convergence

2016-01-24T19:13:17.813Z · score: 14 (16 votes)
Comment by thrasymachus on Open Thread, Dec. 28 - Jan. 3, 2016 · 2016-01-09T15:23:18.411Z · score: 6 (6 votes) · LW · GW

I broadly differ with the hansonian take on medicine. I think metamed failed not because it offered more effective healing but went bust because medicine doesn't really demand healing; but rather that medicine is about healing, generally does this pretty well, and Metamed was unable to provide a significant edge in performance over standard medicine. (I should note I am a doctor, albeit a somewhat contrarian one. I wrote the 80k careers guide on medicine).

I think medicine is generally less fertile ground for hansonian signalling accounts, principally because health is so important for our life and happiness we're less willing to sacrifice it to preserve face (I'd wager it is an even better tax on bs than money). If the efficacy of marginal health spending is near zero in rich countries, that seems evidence in support of, 'medicine is really about healing' - we want to live healthily so much we chase the returns curve all the way to zero!

There are all manner of ways in which western world medicine does badly, but I think sometimes the faults are overblown, and the remainder are best explained by human failings rather than medicine being a sham practice:

1) My understanding of the algorithms for diagnosis is that although linear regressions and simple methods can beat humans at very precise diagnostic questions (e.g. 'Given these factors of a patient who is mentally ill, what is their likelihood of committing suicide?), humans still have better performance in messier (and more realistic) situations. It'd be surprising for IBM to unleash Watson on a very particular aspect of medicine (therapeutic choice in oncology) if simple methods could beat doctors across most of the board.

(I'd be very interested to see primary sources if my conviction is mistaken)

2) Medicine has become steadily more and more protocolized, and clinical decision rules, standard operating procedures and standards of care are proliferating rapidly. I agree this should have happened sooner: that Atul Gwande's surgical checklist happened within living memory is amazing, but it is catching on, and (mildly against hansonian explanations) has been propelled by better outcomes.

I can't speak for the US, but there are clear protocols in the UK about initial emergency management of heart attacks. Indeed, take a gander at the UK's 'NICE Pathways' which gives a flow chart on how to act in all circumstances where a heart attack is suspected.

3) I agree that the lack of efficacy information about individual doctors isn't great. Reliable data on this is far from trivial to acquire however, and that with doctors understandable self-interest not to be too closely monitored seems to explain this lacuna as well as the hansonian story. (Patients tend to want to know this information if it is available, which doesn't fit well with them colluding with their doctors and family in a medical ritual unconnected to their survival).

4) Over-treatment is rife, but the US is generally held up as an anti-examplar of this fault, and (at least judging by the anecdotes) medics in the UK are better (albeit still far from perfect) at flogging the patient to death with medical torture. Outside of this zero or negative margin, performance is better: it is unclear how much is attributable to medicine, but life expectancy, disease free life expectancy, and age-standardized mortality rates for most conditions are declining.

Now, why Metamed failed (I appreciate one should get basically no credit for predicting a start up will fail given this is the usual outcome, but I called it a long time ago):

Metamed's business model relied on there being a lot of low hanging fruit to pluck. That in many cases, a diagnosis or treatment would elude the clinician because they weren't appraised of the most recent evidence, were only able to deal in generalities rather than personalized recommendations, or that they just were less adept at synthesizing the evidence available.

If it were Metamed versus the average doctor - the one who spends next-to-no time reading academic papers, who is incredibly busy, stressed out, and so on, you'd be forgiven for thinking that metamed has an edge. However, medics (especially generalists) have long realized they have no hope of keeping abreast of a large medical literature on their own. Enter division of labour: they instead commission the relevant experts to survey, aggregate and summarize the current state of the evidence base, leaving them the simpler task of applying in their practice. To make sure it was up to date, they'd commission the experts to repeat this fairly often.

I mentioned NICE (National Institute of Clinical Excellence) earlier. They're a body in the UK who are responsible (inter alia) for deciding when drugs and treatments get funded on the NHS. They spend a vast amount of time on evidence synthesis and meta-analysis. To see what sort of work this produces google 'NICE {condition}'. An example for depression is here. Although I think the UK is world leading in this aspect, there are similar bodies in similar countries in other countries, as well as commercial organizations (e.g. Uptodate.)

Against this, Metamed never had any edge: they didn't have groups of subject matter experts to call upon for each condition or treatment in question, nor (despite a lot of mathsy inclination amongst them) did they by and large have parity in terms of meta-analysis, evidence synthesis and related skills. They were also outmatched in terms of quantity of man hours that could be deployed, and the great headstart NICE et al. already had. When their website was still up I looked at some of their example reports, and my view was they were significantly inferior to what you could get via NICE (for free!) or Uptodate or similar services for their lower fees.

MEtamed might have had a hope if in the course of producing these general evidence summaries, a lot of fine-grained data was being aggregated out to produce something 'one size fits all' - their edge would be going back to the original data to find out that although generally drug X is good for a condition, in ones particular case in virtue of age, genotype, or whatever else, drug Y is superior.

However, this data by and large does not exist: much of medicine is still at the stage of working out whether something works generally, rather than delving into differential response and efficacy. It is not clear it ever will - humans might be sufficiently similar to one another that for almost all of them one treatment will be the best. The general success of increasing protocolization in medicine is some further weak evidence of this point.

I generally adduce meta-med as an example of rationalist overconfidence. That insurgent Bayesians can just trounce relevant professionals in terms of what they purport to do thanks to signalling etc. But again, given the expectation was for it to fail (as most start ups do), this doesn't provide evidence. If it had succeeded, I'd have updated much more strongly in the magic of rationalism meaning you can win and the world being generally dysfunctional.

Comment by thrasymachus on Results of a One-Year Longitudinal Study of CFAR Alumni · 2015-12-26T00:05:22.502Z · score: 4 (4 votes) · LW · GW

Congratulations on doing this sort of careful self-analysis. I'd like to recommend a further improvement.

Pre-register/Publish your intentions for these trials and analysis in advance

The file-drawer problem is well-known, as are the risks of post-hoc changes in analysis. Publishing what data you are gathering and the analyses you will perform on it reassure more skeptical people against both of these worries, and seems pretty easy to do.

Log-normal Lamentations

2015-05-19T21:12:52.019Z · score: 12 (15 votes)
Comment by thrasymachus on Against the internal locus of control · 2015-04-05T00:58:59.776Z · score: 2 (2 votes) · LW · GW

[I've seen your follow-up post on discussion. I thought it would be best to reply to both here.]

It may be that everything is determined by prior events all the way to the big bang. So there's no 'internal willer' isolated from previous events that can steer us one way or another. But we can keep talking about 'internal' and 'external' loci of control on a compatibilist view of free will (which I'd guess is the common view, including amongst those affirming an internal locus of control).

On this sort of view, internal factors are just those our choices can change - external factors, those which our choices cannot. If I want to run faster, how much time I spend training is an internal factor: it influences how fast I can run, and I can choose (in the compatibilist sense) how much time I spend training. If I have a dense hemiparesis secondary to a birth injury, that's an external factor - it also influences how fast I can run (indeed, whether I can run at all), and can't choose whether or not to have a hemiparesis.

So I take those with an internal locus of control to think that - in the main - the outcomes that matter are mainly sensitive to factors that in turn are sensitive to our choices (how hard I work, how long I practice, etc.), whilst those with an external locus of control say that these things are primarily determined by factors outside of that person's control.

It seems clear to me that IQ should be in the 'external factors' camp: IQ seems to be set early in life, has a large heritable component, and the non heritable bit is likely due to environmental things that I also can't change for myself, either at the time or retroactively. The failure of brain training programs suggests that you can't improve your IQ by any feat of effort. And we know it has all sorts of influences on how our lives turn out. If I have (due to factors outside my control) an IQ more than one standard deviation below the mean, I won't be able to become a doctor, or a physicist (or, indeed, joining the US armed services) - no matter what else I do. Mutatis mutandis cases where it might not serve as a strict bar but a variable handicap (c.f. evidence that the beneficial effects of IQ have no clear ceiling).

The alternative account you propose for demarcating 'external' versus 'internal' factors - internal factors are those causally distal to your brain's neural output - looks too broad: all internal factors need to be downstream of our neural output, but that isn't sufficient. The hemiparesis case I allude to above would be one example - that I can't move one side of my body is due to my neural output, but that is because of this insult which wasn't due to my neural output. I think the same applies for other cases of brain damage and particular types of mental illness: indeed, this is implicitly recognised by the criminal justice system.

(I've added remarks to this effect in the body of the post - thanks for this comment!)

Against the internal locus of control

2015-04-03T17:48:56.400Z · score: 6 (11 votes)
Comment by thrasymachus on Immortality: A Practical Guide · 2015-01-27T01:26:17.246Z · score: 13 (13 votes) · LW · GW

A general point: It is generally helpful to express risks as absolute risks (or number needed to treat) whenever possible, as odds ratios or percentages changes are hard to interpret, and often mislead if the 'base rate risk' is minute: often people's brains mistake '25% increased risk of Y if you do X', as risk of '25% of Y if you do X'.

E.g. pretend mobile phone use really caused an exotic brain cancer, and the OR is 100. But the base-rate of this brain cancer is something like 1/ten million. So although mobile phone users are 100 times more likely to get exotic brain cancer, their absolute risk goes up from 1/ten million to 1/million, so the absolute risk reduction of avoiding mobile phones would be a 0.0009% mortality risk benefit - just over a million people would have to stop using mobiles to avoid a single case of exotic brain cancer.

Comment by thrasymachus on Why the tails come apart · 2014-11-19T01:07:04.864Z · score: 0 (0 votes) · LW · GW

Belatedly updated. Thanks for your helpful comments!

Funding cannibalism motivates concern for overheads

2014-08-30T00:42:13.492Z · score: 26 (27 votes)

Why the tails come apart

2014-08-01T22:41:00.044Z · score: 134 (135 votes)

UFAI cannot be the Great Filter

2012-12-22T11:26:41.010Z · score: 37 (37 votes)