Stupid Researcher Tricks

When David Letterman was a late night TV talk show host, he had a recurring segment called "stupid pet tricks." In this vein, this post is titled "stupid researcher tricks." The authors of certain studies in psychiatry employ logical fallacies - either knowingly or in some cases unknowingly - in their discussions of their results. This often results in their drawing totally misleading conclusions from their data.

I will discuss two telling examples of such fallacies. The first involves employing a hidden assumption that is not true. I discussed the general issue of hidden assumptions in a previous post. The particular example in this post can actually be thought of as Part II of my post on the heritability fraud. The second example involves a subtle logical fallacy known as the ecological fallacy: making inferences about individuals based on data obtained that characterizes an entire group, using averages on various measures, to which that individual belongs.

In the previous "heritablility" post I wrote about how that term is falsely used as a synonym with "genetic" in studies that purport to sort out genetic versus environmental factors leading to the development of personality traits or behavioral disorders. The statistic is developed from twin studies: identical versus fraternal twins, and/or those identical twins raised together and those raised apart. 

As previously described, these studies - despite frequent claims to the contrary - do almost nothing to sort out genetic versus environmental influences. The reason is because the statistic is based on phenotype (the final interactional product of genes + environment) and not genotype. As such, it includes a mix of purely genetic factors and factors that result from gene-environmental interactions, and there is no way to know how much of each is contained therein.

Another aspect of these studies I mentioned in the previous post: they also divide environmental influences into "shared" (family and home) and "unshared" (peers, media, teachers, and other outside factors). I mentioned that a determination of which parts of an environment are shared by siblings and which are unshared has a lot in common with finding water with a divining rod, because parents do not treat all of their children alike. Nor does each twin have exactly the same interactions with each and every other family member from the moment they are born to the moment they die.

Interestingly, the "shared" environments in these studies usually come out as less important in leading to behavioral issues than the "unshared!" This would mean that family and parental behavior is less of a factor in personality development than outside influences - something that runs counter to logic for a variety of reasons (for example: which peer group someone chooses to hang out with - when there are several different ones to choose from -  is not an accident).  

The researchers almost have to find that family is less important that peers and media because they just ASSUME that each twin is subject to identical influences inside the home. If you make this assumption, and then if the twins turn out differently on some characteristic, of course the home will appear to have no influence!

Maybe these ignoramuses should read the family systems literature on how siblings are treated differently. The more dysfunctional the family, the bigger the differences! 

Example number two, studies that employs the ecological fallacy, involves those that compare two different psychotherapy treatments for the same disorder. These studies are relatively uncommon, as most psychotherapy outcome studies compare a treatment with a "control" condition like a wait list or "treatment as usual" rather than with a second type of treatment. (Those control groups are also invalid, but that is a matter not relevant to this discussion).

In the few studies that compare one school of therapy with another, an interesting statistic is that 85% of the time, the treatment favored by the person designing the experiment "wins" and outperforms the other treatment [Luborsky, L., Diguer, L., Seligman, D. A., et. al. 1999.  "The Researcher’s Own Therapy Allegiances:  A “Wild card” in Comparisons of Treatment Efficacy." Clinical Psychology: Science and Practice, 6, 95-106]. This is due to something called the allegiance effect - the more enthusiastic a therapist in a study is about their own school, the better the patient tends to do.

But even ignoring this clear-cut sign that research conclusions in comparative outcome studies are inherently misleading, let us suppose that with one therapy treatment in the study, 45% of the patients improve significantly, while in the other, only 30% do. The conclusion of the researchers: the first treatment is superior.

Wrong.

This conclusion presupposes that all patients react to treatments somewhat identically, despite the fact that the majority (or at least a significant percentage) of patients in both arms of the study did not improve. It is quite likely that some patients are more comfortable and do much better with one of the therapy treatments than with the other. Of course there is no way to know for certain, but it is quite possible that the 45 percent of people who respond to the allegedly superior treatment are very different in many respects than are the 30% who respond to the allegedly inferior therapy.

A well-designed study, on the other hand, would have to recognize these differences and would look at the characteristics of the four different groups that comprised the study: those who got better with treatment A, those who got better with treatment B, those who responded poorly to treatment A, and those who responded poorly to treatment B. The researchers could then match the patient with the type of therapy they seemed to do best with, and then and only then compare outcomes.

In other words, as I have pointed out elsewhere, no matter what therapy intervention you use, some people will improve with it, while others either will not improve or may even get worse! Different strokes for different folks, people.