Posts
Comments
These are good points. Do you think that if the researchers did find the sort of discretization that they are hypothesizing, that this would represent at least some weak evidence in favor of the simulation hypothesis, or do you think it's completely uninformative with respect to the simulation hypothesis?
Damn. I quickly checked to see if this link had been posted, but I guess I didn't look far back enough--I assumed that if it had been, it would have been very recently, but apparently it was actually posted 10 days ago... my bad.
Have to disagree with you on, well, several points here.
Heuristics in Heuristics and Biases are only descriptive. [...] Heuristics in Heuristics and biases are defined as having negative side effects.
If your claim is that heuristics are defined by H&B theorists as being explicitly not prescriptive, in the sense of never being "good" or "useful," this is simply not the case. For instance, in the opening paragraph of their seminal 1974 Science article, Kahneman & Tversky clearly state that "...people rely on a limited number of heuristic principles which reduce the complex tasks of assessing probabilities and predicting values to simpler judgmental operations. In general, these heuristics are quite useful, but sometimes they lead to severe and systematic errors." Gigerenzer et al. would not necessarily disagree with this definition (they tend to define heuristics in terms of "ignoring information" rather than "reducing complexity," although the end result is much the same), although they would almost certainly phrase it in a more optimistic way.
...nor probably could you even if you tried. You could not intentionally reproduce the pattern of cognitive biases that their heuristics allegedly cause, many appear to be irretrievably outside of conscious awareness or control.
Representativeness, one of the earliest examples of a heuristic given by the H&B program, is certainly used in a conscious and deliberate way. When asked, subjects routinely report relying on representativeness to make frequency or probability judgments, and they generally see nothing wrong or even really remarkable about this fact. Nick Epley's work also strongly suggests that people very deliberately rely on anchoring-and-adjustment strategies when making some common judgments (e.g., "When was George Washington elected president?" "Hmm, well it was obviously some time shortly after the Declaration of Independence, which was in 1776... so maybe 1786?").
Fast and Frugal heuristics, however, you can learn and use intentionally.
One can certainly learn to use any heuristic strategy, but for some heuristics proposed by the F&F camp, such as the so-called fluency heuristic (Hertwig et al., 2008), it is not at all obvious that in practice they are utilized in any intentional way, or even that subjects are aware of using them. The fluency heuristic in particular is extremely similar to the availability heuristic proposed decades earlier by Kahneman & Tversky.
Descriptive F&F heuristics aren't evolutionary quirks.
I'm not sure what you mean here. If an "evolutionary quirk" is a locally optimal solution that falls short of a global maximum, then the heuristics described by both H&B and F&F theorists are most certainly "evolutionary quirks." The claim being advanced by F&F theorists is not that the heuristics we tend to use are optimal in any sense of having maximal evolutionary adaptedness, but simply that they work just fine thanks. Note, however, that they are outperformed in simple inference tasks even by relatively simple strategies like multiple regression, and outperformed in more challenging prediction tasks by, e.g., Bayes Nets. They are decidedly not globally optimal.
...besides the obvious that Fast and Frugal heuristics are "good" while heuristics as in Heuristics and biases are "bad".
This impression is entirely due to differences in the framing and emphasis employed by the two camps. It does not represent anything like a fundamental distinction between how they each view the nature or role of heuristics in judgment and decision making.
What I'm saying is that is how many people tend to wrongly interpret such statistics to define their own null hypothesis in the way I outlined in the post.
But that's not right. The problem that your burden of proof example describes is a problem of priors. The theist and the atheist are starting with priors that favor different hypotheses. But priors (notoriously!) don't enter into the NHST calculus. Given two statistical models, one of which is a nested subset of the other (this is required in order to directly compare them), there is not a choice of which is the null: the null model is the one with fewer parameters (i.e., it is the nested subset). It isn't up for debate.
There are other problems with NHST--as you point out later in the post, some people have a hard time keeping straight just what the numbers are telling them--but the issue I highlighted above isn't one of them for me.
Also, isn't model complexity quite hard to determine with the statements "God exists" and "God does not exist". Isn't the complexity in this sense subject to easy bias?
Yes. As you noted in your OP, forcing this pair of hypotheses into a strictly statistical framework is awkward no matter how you slice it. Statistical hypotheses ought to be simple empirical statements.
As an aspiring scientist, I hold the Truth above all.
That will change!
More seriously though...
As one can see, the biggest problem is determining burden of proof. Statistically speaking, this is much like the problem of defining the null hypothesis.
Well, not really. The null and alternative hypotheses in frequentist statistics are defined in terms of their model complexity, not our prior beliefs (that would be Bayesian!). Specifically, the null hypothesis represents the model with fewer free parameters.
You might still face some sort of statistical disagreement with the theist, but it would have to be a disagreement over which hypothesis is more/less parsimonious--which is really a rather different argument than what you've outlined (and IMO, one that the theist would have a hard time defending).
It doesn't sound unreasonable to me given the severity of your symptoms. But I'm not a sleep doctor.
Consider also that there are other ways to procure drugs like this, i.e., shady online vendors from overseas. Just make sure you do your research on the vendors first. There are people who have ordered various drugs from these vendors, chemically verified that the drugs were in fact what they were advertised to be, and then posted their results in various places online for the benefit of others. Bottom line: some companies are more trustworthy than others--do your homework. And obviously you should exercise due caution when taking a new drug without a doctor's consent.
How about Modafinil or a similar drug? It is prescribed for narcolepsy. More generally, can I safely assume that "everything" includes having talked to your doctor about how serious these symptoms are?
I think you're taking the fundamentally wrong approach. Rather than trying to simply predict when you'll be sleepy in the near-term, you should try to actively get your sleeping patterns under control.
Robin Hanson's posts from the AI Foom debate are not included in the list of all articles. Covering only Yudkowsky's side of the debate would be a little strange for readers I think. Should we feature Hanson's posts (and others who participated in the debate) during that time as well?
Yes, that's exactly right.
And although I'm having a hard time finding a news article to verify this, someone informed me that the official breast cancer screening recommendations in the US (or was it a particular state, perhaps California?) were recently modified such that it is now not recommended that women younger than 40 (50?) receive regular screening. The young woman who informed me of this change in policy was quite upset about it. It didn't make any sense to her. I tried to explain to her how it actually made good sense when you think about it in terms of base rates and expected values, but of course, it was no use.
But to return to the issue clinical implications, yes: if a woman belongs to a population where the result of a mammogram would not change our decision about whether a biopsy is necessary, then probably she shouldn't have the mammogram. I suspect that this line of reasoning would sound quite foreign to most practicing doctors.
I agree with previous comments about publishing in journals being an important status issue, but I think there is other value as well which is being ignored. For all of its annoyances and flaws, one good thing about peer review is that it really makes your paper better. When you submit a pretty good paper to a journal and get back the "revise and resubmit" along with the detailed list of criticisms and suggestions, then by the time the paper actually makes it into the journal, chances are that it will have become a really good paper.
But to return to the issue of papers being taken more seriously when published in a journal, I think that this view is actually quite justified. For researchers who are not already very knowledgeable in the precise area that is the topic for a given paper, whether or not the paper has withstood peer review is a very useful heuristic cue toward how much weight you should place on it. Basically, peer review keeps the author honest. An author posting a paper on his website can say pretty much whatever he wants. One of the purposes of peer reviewers is to make sure that the author isn't making unreasonable claims, mischaracterizing theoretical positions, "reviewing" the relevant previous literature in a grossly selective way, etc. Like I said, if someone is already very familiar with the area, then they can evaluate these aspects of the paper for themselves. But if you'd like to communicate your position to a wider academic audience, peer review can help carry your paper a longer way.
As far as the take-home practical message goes, on my reading it was never about how well doctors could "diagnose cancer" per se based on mammogram results--rather, the reason we ask about P(cancer | positive) is because it ought to inform our decision about whether a biopsy is really warranted. If a healthy young woman from a population with an exceedingly low base rate for breast cancer has a positive mammogram, the prior probability of her having cancer may still be low enough that there might actually be negative expected value in following up with a biopsy; after all, let's not forgot that a biopsy is not a trivial procedure and things do sometimes go wrong.
So I think this actually does have some implication for real-world clinical care: we ought to question whether it is wise to automatically follow up all positive mammograms with biopsies. Maybe it is, and maybe it isn't, but I don't think we should take the question for granted as appears to be the case.
Should we go somewhere else to discuss, rather than heavily nested comments? Would a new discussion topic page be the right place?
I agree that the nested comment format is a little cumbersome (in fact, this is a bit of a complaint of mine about the LW format in general), but it's not clear that this discussion warrants an entirely new topic.
Terminology isn't terribly important . . . If you want to take the Popperian conception of a good theory and label it "justified" it doesn't matter so much.
Okay. So what is really at issue here is whether or not the Popperian conception of a good theory, whatever we call that, leads to regress problems similar to those experienced by "justificationist" systems.
It seems to me that it does! You claim that the particular feature of justificationist systems that leads to a regress is their reliance on positive arguments. Popper's system is said to avoid this issue because it denies positive arguments and instead only recognizes negative arguments, which circumvents the regress issue so long as we accept modus tollens. But I claim that Popper's system does in fact rely on positive arguments at least implicitly, and that this opens the system to regress problems. Let me illustrate.
According to Popper, we ought to act on whatever theory we have that has not been falsified. But that itself represents a positive argument in favor of any non-falsified theory! We might ask: okay, but why ought we to act only on theories which have not been falsified? We could probably come up with a pretty reasonable answer to this question--but as you can see, the regress has begun.
If i convinced you of this one single issue (that there is a method for making the decision), would you follow up with a thousand other objections to Popperian epistemology, or would we have gotten somewhere?
Yes, we will have gotten somewhere. This issue is my primary criticism of Popperian epistemology. That is, given what I understand about the set of ideas, it is not clear to me how we would go about making practical scientific decisions. With that said, I can't reasonably guarantee that I will not have later objections as well before we've even had the discussion!
So let me see if I'm understanding this correctly. What we are looking for is the one conjecture which appears to be completely impervious to any criticism that we can muster against it, given our current knowledge. Once we have found such a conjecture, we -- I don't want to say "assume that it's true," because that's probably not correct -- we behave as if it were true until it finally is criticized and, hopefully, replaced by a new conjecture. Is that basically right?
I'm not really seeing how this is fundamentally anti-justificationist. It seems to me that the Popperian epistemology still depends on a form of justification, but that it relies on a sort of boolean all-or-nothing justification rather than allowing graded degrees of justification. For example, when we say something like, "in order to make a decision, we need to have a guiding theory which is currently impervious to criticism" (my current understanding of Popper's idea, roughly illustrated), isn't this just another way of saying: "the fact that this theory is currently impervious to criticism is what justifies our reliance on it in making this decision?"
In short, isn't imperviousness to criticism a type of justification in itself?
So, how do Popperians decide? They conjecture an answer, e.g. "yes". Actually, they make many conjectures, e.g. also "no". Then they criticize the conjectures, and make more conjectures. So for example I would criticize "yes" for not providing enough explanatory detail about why it's a good idea. Thus "yes" would be rejected, but a variant of it like "yes, because nuclear power plants are safe, clean, and efficient, and all the criticisms of them are from silly luddites" would be better. If I didn't understand all the references to longer arguments being made there, I would criticize it and ask for the details. Meanwhile the "no" answer and its variants will get refuted by criticism. Sometimes entire infinite categories of conjectures will be refuted by a criticism, e.g. the anti-nuclear people might start arguing with conspiracy theories. By providing a general purpose argument against all conspiracy theories, I could deal with all their arguments of that type. Does this illustrate the general idea for you?
Almost, but you seem to have left out the rather important detail of how actually make the decision. Based on the process of criticizing conjectures you've described so far, it seems that there are two basic routes you can take to finish the decision process once the critical smoke has cleared.
First, you can declare that, since there is no such thing as confirmation, it turns out that no conjecture is better or worse than any other. In this way you don't actually make a decision and the problem remains unsolved.
Second, you can choose to go with the conjecture that best weathered the criticisms you were able to muster. That's fine, but then it's not clear that you've done anything different from what a Bayesian would have done--you've simply avoided explicitly talking about things like probabilities and priors.
Which of these is a more accurate characterization of the Popperian decision process? Or is it something radically different from these two altogether?
Might it be a good idea to feature the IRC channel more centrally on the website? Eliezer's concern notwithstanding, if I'm going to kill time anyway (and believe me, I'm going to anyway), it might be nice to do so in a busy LW IRC room. I could think of less productive things to do for an hour.
Sure there are associated values. By implying that a particular out-group is "ugly, smelly, no friends, socially unacceptable, negative, aggressive," etc. etc., you simultaneously imply that your in-group is none of those things. You elevate the in-group by derogating the out-group. Presumably you and your in-group value not having all of those negative traits.
If situationism is true, why do the folk have such a robust theory of character traits? Can we provide an error theory for why people have such a theory?
Jones and Nisbett attempted to answer this question in their classic paper on actor-observer bias. It's an interesting read.
However, beware of falling into an overly strict interpretation of situationism (as I think Jones and Nisbett did) which amounts to little more than behaviorism in new clothes. People do tend to underestimate the extent to which their behavior and the behavior of others is driven by the environment, but there is nevertheless still good evidence that stable dispositions predict a respectable chunk of the variance in a person's behavior (where "respectable" means "similar in size to that for the situational data"). One of the errors of the strict situationist movement that arose in the late 1960s and 1970s is that it relied on an implicit endorsement of Kahneman and Tversky's "law of small numbers": situationist researchers erroneously expected a very small sample of a person's behavior (such as a single encounter with a person in need of help) to be representative of the general population of behaviors from which it was drawn. Not surprisingly, they found that stable dispositions are a modest predictor of these small samples of behavior. However, we have since learned that when behavior is properly aggregated across time, a robust effect of stable dispositions reliably emerges.
So in short, part of the reason that people have robust theories of character traits is almost surely that they actually map pretty well onto the territory. Situationism remains a useful paradigm, but it can easily be taken too far.
Wikipedia has a page on Just-world phenomenon which lists the following references:
Lerner, M, & Simmons, CH. (1966). Observer reaction to the 'innocent victim': Compassion or rejection? Journal of Personality and Social Psychology 4 (2): 203–210.
Carli, L.L. (1999). Cognitive reconstruction, hindsight, and reactions to victims and perpetrators. Personality and Social Psychology Bulletin, 25, 966-979.
Lerner, M.J. & Miller, D.T. (1978). Just world research and the attribution process: Looking back and ahead. Psychological Bulletin, 85, 1030-1051.
I bet that a "related articles" search on ISI Web of Science for the Carli 1999 article will turn up some additional relevant, and hopefully more recent, papers.
Edit: Some additional references from Wiki page on Victim blaming
Janoffbulman, R. (1985). Cognitive biases in blaming the victim. Journal of Experimental Social Psychology 21: 161–178.
Maes, Jürgen (1994). Blaming the victim: Belief in control or belief in justice? Social Justice Research 7: 69–90.
McCaul, K. D.; Veltum, L. G.; Boyechko, V.; Crawford, J. J. (1990). Understanding Attributions of Victim Blame for Rape: Sex, Violence, and Foreseeability. Journal of Applied Social Psychology 20: 1–26.
I can report with some degree of confidence that the Blanton paper represents a skeptical view which is very much a minority in the field. This doesn't necessarily mean that it's biased or "wrong," but I think a LessWronger such as yourself will understand what this suggests regarding the intellectual status of their claims.
A couple papers to balance out the view from above:
Rebuttal to above by authors of "reanalyzed" study http://www.bsos.umd.edu/psyc/hanges/Ziegert%20and%20Hanges%202009.pdf
Reply to a different but similar Tetlock-and-friends critique: http://www.columbia.edu/~dc2534/RIOB_jost.et.al.pdf
I'm not sure. You may not be able to in any feasible or satisfactory way, which was sort of my point.
This sort of conditioning works best when the reward is administered within about 500 ms of the response (sorry, don't have a citation). Something to keep in mind.
It's also possible many people are simply not terribly good at using Internet, or that many disciplines don't yet have information available on the Internet - in the long term the normal case will far more information than you ever need available online, but this might not always be the case yet.
It's not the first possibility, it's the second. I'm quite comfortable in saying that I am very capable at finding specific online content if it's out there to be found. The problem is that most of the disciplines I'm interested in reading about don't have the good, hard content available for free on the web. (Scientific journals can be accessed online, but these are just books.) It is not the "normal case" that far more information than one could want is available online for most domains, it is absolutely the abnormal case. To be frank, the idea that one could ever get a thorough grounding in any serious, empirical scientific discipline by scouring the Internet is, at least at this time, laughable.
That's a pretty good list they have going, but in my opinion the Gigerenzer et al. volume should be replaced by one published 3 years earlier by the same research group: Simple Heuristics That Make Us Smart. It's the same basic thing, but a bit more comprehensive and more directly relevant to cognitive psych (no chapters on animal rationality and etc).
Also, while the 1982 H&B volume is obviously very good and certainly belongs on the list, the picture is pretty incomplete without the updated 2002 H&B volume as well as Choices, Values, and Frames (1999).
Hi.
I'm a grad student studying social psychology, more or less in the heuristics & biases tradition. I've been loosely following the blog for maybe six months or so. The discussions are always thought provoking and frequently amusing. I hope to participate more in the near future.
Okay, but is it a part of the typical Bayesian routine to wield formal decision theory, or do we just calculate P(H|E) and call it a day?
We could just as easily imagine the selection bias having worked the other way (LessWrongers are hardly a representative sample and some have motivated reasons for choosing one way or another, especially having read through the thread), but you're of course right that, in any case, this sample isn't telling us much.
I thought the baby was cuter... but why bother voting in a meaningless poll like this? (No offense :P)
I mean, not only is the "p-value" threshold arbitrary, not only are we depriving ourselves of valuable information by "accepting" or "not accepting" a hypothesis rather than quantifying our certainty level, but...what about P(E|H)?? (Not to mention P(H).)
Well, P(E|H) is actually pretty easy to calculate under a frequentist framework. That's the basis of power analysis, a topic covered in any good intro stat course. The real missing ingredient, as you point out, is P(H).
I'm not fully fluent in Bayesian statistics, so while I'm on the topic I have a question: do Bayesian methods involve any decision making? In other words, once we've calculated P(H|E), do we just leave it at that? No criteria to decide on, just revising of probabilities?
This is my current understanding, but it just seems so contrary to everyday human reasoning. What we would really like to say at the end of the day (or, rather, research program) is something like "Aha! Given the accumulated evidence, we can now cease replication. Hypothesis X must be true." Being humans, we want to make a decision. But decision making necessarily involves the ultimately arbitrary choice of where to set the criterion. Is this anti-Bayesian?
Something is falsifiable if if it is false, it can be proven false.
Isn't this true of anything and everything in mathematics, at least in principle? If there is "certainly an objective truth or falsehood to P = NP," doesn't that make it falsifiable by your definition?
Yeah, that was a pretty clever turn of phrase.
My secret is out!