Personal Psychiatric Analysis

A small investigational drug trial won't be powered to detect outliers, and you won't be able to reliably solve that by invoking Bayesian statistics.

In large drug trials I think this is to some degree already done, but it's limited by the extreme sketchiness of suddenly inventing new endpoints for your study after you have the data. It would probably take the form of increasing the threshold for an endpoint (for example, "No significant difference between drug and placebo was found with the planned endpoint of decreasing HAM-D ratings by 3 or more, but there were significantly more patients in the drug group who had their HAM-D ratings decrease by 10 or more". Everyone is rightly suspicious of people who do this, because, again, changing endpoints. But if it happened enough someone would take notice. Trust me, "not coming up with clever ways to make their drug look effective for at least some people" is not one of pharmaceutical companies' failure modes.

But keep in mind that you sort of loaded the original example by choosing something that almost never happens (someone living to 110 without any signs of aging). In a psychiatry study, what's the most extreme example you're going to get? Someone's depression remits completely? Big deal. Most people's depressive episodes remit completely after a couple of months anyway, and in 25% of people they never return (in even more people, they take many years to return, and almost no studies continue for the many years it would take to notice). In a drug trial of 10000 people (the number you gave above) hundreds or thousands of people in each group are going to have their depression remit completely; if the drug has a superpowerful effect on one person and cures her depression forever, that will get lost in noise in the way that someone living to 110 with the body of a 30 year old might not.

(it's instructive to compare this to the way studies investigate side effects. If one person in a 10000 person study has their arms fall off, the investigators will notice, because that's sufficiently rare as to raise suspicion it was caused by the drug. The drug will then end up with a black box warning saying "may make arms fall off.")

Another way these sorts of outlier effects might be detected is by subgroup analyses (which are also extremely sketchy). If there is no effect in general, researchers may check whether there is an effect among men, among women, among blacks, among whites, among Latinos, among postmenopausal Burmese women who wear hats and own at least two pets and have a history of disease in their left kidney, anything that turns up a positive result. But again, this is hardly something we want to encourage.

But all these things are for investigational drugs. if we're talking about a drug that's already been approved and has a strong prescription history, then your worries about individualized response would get subsumed into the good responder / bad responder distinction, which is a very very big area of research which we know a lot about and when we don't know it it's not for lack of trying.

For example, among bipolar patients, response to lithium can be (very inconsistently) predicted by selecting for patients who have stronger family history of disease, have fewer depressive symptoms, have slower cycles, have more euthymic periods, have less of a history of drug use, start with a manic episode, demonstrate psychomotor retardation, demonstrate premorbid mood lability, lack premorbid personality disturbance, possibly have deranged serotonin metabolism in platelets, possibly have increased calcium binding to red blood cells, possibly lack the HLA-A3 antigen, possibly have a particular variant of the gene GADL1, etc etc etc.

(in practice we don't expend much effort to check most of these things, because their predictive power is so weak that it's almost always a worse idea than just making a best guess based on the data you have, putting someone on lithium or on an alternative, then switching if it doesn't work)

As far as I know, psychiatrists cannot reliably predict that a given drug will improve a patient's long-term diagnosis, and psychiatrists/psychologists cannot even reliably agree on what condition a patient is manifesting. Mental disorders appear to resist diagnosis and solution, unlike, say, a broken leg or a sucking chest wound.

The whole "medical doctors can always consistently treat medical diseases, but psychiatrists are throwing darts blindfolded" story is something of a myth - see for example Putting the efficacy of psychiatric and general medicine medication into perspective: review of meta-analyses

The basic question to ask is: did he live to 110 because of the drug, or was he going to be unusually long-lived anyway and happened to be enrolled in the trial?

If the drug was developed to combat some mechanism of senescence, one might reasonably entertain the possibility of a causal effect, but if it was for an unrelated matter, I don't see a reason to expect it. Either way, one would want the scientists to do a lot more tests on that individual to discover the mechanisms of his longevity.

("Personally, young man, I attribute my years to a diet of whisky, cigars, and strictly fried food.")

often in clinical trials there are odd outliers in the results

Sure, but outliers usually go both ways. There are people who magically got cured, but there are also people who just keeled over and died and no one is sure why.

Living to the age of 110 is abnormal enough that within 6,666 people selected as a statistical representation of the population, it is extremely unlikely that anyone would live that long, much less continue performing at the apparent health of an 80 year old.

(Did you mean 10,000, not 6,666?)

I, too, thought that living to 110 was really rare, but I looked for some statistics and it actually is about 1/10,000, so the age alone is not surprising. Wikipedia says that getting from 100 to 110 is 1 in 1000, but getting from 80 to 100 is not that bad. In England, men have a 6.2% chance and women a 9.2% chance.

I think that the tricky thing is that a psychiatrist has to put statistics into perspective on two sides of the equation - both in regards to the medication, and then in regards to the diagnosis/presentation of the patient. In a nutshell, we diagnose patients based on their behaviour, not their brain chemistry and function, while we are using medication to treat their brain chemistry and function. We are not treating something that, in most cases, a doctor can see or quantify absolutely (in most instances doctors are not doing brain scans in order to medicate - I don't even think at this point in our understanding they do).

Looking at the side effects for many psychiatric medications, it is fairly clear that the side-effects are vast and for a huge number also include non-temporary worsening of symptoms, including risk of suicide. To me that indicates that we do not understand nearly enough about brain chemistry or the cause of mental illness to medicate with reason - as much of it is intuition and experience and guesswork as formal training.

In this link it notes that "Gjedde explains we don't really know what is wrong with people suffering from depression but looking at the symptoms we get a good idea of where the fault could lie within the brain". Furthermore, major theories such as that the serotonin causes depression are suggested to be simplistic and innacurate, so while we have an idea that using medication to affect serotonin levels can help depression, it doesn't mean that low serotonin levels are the cause. Those two articles are generally depression-specific, but considering that depression is probably the most studied mental illness (if not one of), if our understanding of that illness is vague, it seems likely that our understanding of other psychiatric illnesses are too (which is supported very much by the "we'll try this medication and see if it works" approach, as opposed to "you have a hole in your leg, we'll dose you up with morphine and you won't feel it as much").

How small would the sample size have to be before you would consider trying the drug yourself, just to see if you, too, lived forever as long as you took it?

Reducing sample size also blinds you to any ill effects the drug might have. If you're looking to generalize your idea about outliers, immortality seems to be an especially poor example since it's more unusual than any real outlier you might come across.

As far as I know, psychiatrists cannot reliably predict that a given drug will improve a patient's long-term diagnosis, and psychiatrists/psychologists cannot even reliably agree on what condition a patient is manifesting.

Taboo reliable. How would scrutinizing outliers make them more reliable? Acknowledging people are unique snowflakes doesn't help if you have no tools to know when and how they're unique. Other specialties have the same problem, psychiatrists' tools are just especially crude in comparison.

Mental disorders appear to resist diagnosis and solution, unlike, say, a broken leg or a sucking chest wound

Substitute those surgical conditions with some endocrine condition for example and the contrast may not be so stark.

If you're treating yourself, you're especially prone to bias. Doctors acknowledge this and many think they shouldn't treat themselves. If you want to utilize outliers, at least have someone else do or confirm the research. Doesn't have to be a doctor if you don't trust them.

Well, here are the thoughts that you provoked from me about this.

Here is the topic for discussion: should we trust psychiatric analysis using frequentist statistics and ignore the outliers, or should we individually analyze psychiatric studies to see if they contain outliers who show symptoms which we personally desire? Should we act differently when seeking nootropics to improve performance than we do when seeking medication for crippling OCD? Should we trust our psychiatrists, who are probably not very statistically savvy and probably don't read the cases of the outliers?

I think we may want to split this up into two questions: What you should do personally, if you feel you have condition X, and what you should do as a government if you want to help treatment for condition X.

For instance, as a government, I would go for repeatability, and size. Just forget the idea of trusting or not trusting suggestive outliers and go for more verification, with a larger sample: If for no other reason then determining the frequency of the outlying effect, which would be important for making large scale medical recommendations.

However, a single person can't generally commission large medical studies, so they might want to just read through the literature and read multiple papers about the effects of such things, perhaps also cross referencing their own medical history. I think Metamed, which was mentioned on Less Wrong a while back, does something like this, if you don't have the time to analyze your health that carefully personally.

If you don't want to do that, (either independently or through commissioning experts), then chances are you will be relying on:

I think that if even the right placebo could cause changes which improve my effectiveness, it would be worth a shot.

Now, psychiatric placebos can be shown to have a variety of effects in papers, and it can get hammed up a bit in headline rereporting.

http://nymag.com/thecut/2014/01/study-placebo-sleep-just-as-good-as-real-sleep.html

But it isn't all hammed up either: it gets rather complicated.

http://en.wikipedia.org/wiki/Placebo

(As a side note, I upvoted you, since that was a good way of provoking thoughts, and it seemed like a placebo for increasing your mental wellbeing.)