When David Letterman was a late
night TV talk show host, he had a recurring segment called "stupid pet
tricks." In this vein, this post is titled "stupid researcher tricks." The authors of certain studies
in psychiatry employ logical fallacies - either knowingly or in some cases unknowingly - in their discussions of their results. This often results in their drawing totally misleading
conclusions from their data.
I will discuss two telling examples of such fallacies. The first involves employing a hidden assumption that is not true. I discussed the general issue
of hidden
assumptions in a previous post. The particular example in this post can actually be thought of as Part II
of my post on the heritability
fraud. The second example involves a subtle logical fallacy known as the
ecological fallacy: making inferences about individuals based on data obtained
that characterizes an entire group, using averages on various measures, to which that individual belongs.
In the previous "heritablility"
post I wrote about how that term is falsely used as a synonym with "genetic" in
studies that purport to sort out genetic versus environmental factors leading to the development of personality traits or behavioral disorders. The statistic
is developed from twin studies: identical versus fraternal twins, and/or those identical
twins raised together and those raised apart.
As previously described, these studies - despite frequent claims
to the contrary - do almost nothing to sort out genetic versus environmental
influences. The reason is because the statistic is based on phenotype (the final interactional
product of genes + environment) and not genotype. As such, it includes a mix of
purely genetic factors and factors that result from gene-environmental
interactions, and there is no way to know how much of each is contained
therein.
Another
aspect of these studies I mentioned in the previous post: they also divide environmental influences into "shared"
(family and home) and "unshared" (peers,
media, teachers, and other outside factors). I mentioned that a determination
of which parts of an environment are shared by siblings and which are unshared
has a lot in common with finding water with a divining rod, because parents do
not treat all of their children alike. Nor does each twin have exactly
the same interactions with each and every other family member from the moment they
are born to the moment they die.
Interestingly,
the "shared" environments in these studies usually come out as less important in leading to behavioral issues than the "unshared!" This would mean that
family and parental behavior is less of a factor in personality development than outside
influences - something that runs counter to logic for a variety of reasons (for
example: which peer group someone chooses to hang out with - when there are
several different ones to choose from - is not an accident).
The researchers almost have to find that family is less important that peers and media because they just ASSUME that each twin is subject to identical influences inside the home. If you make this assumption, and then if the twins turn out differently on some characteristic, of course the home will appear to have no influence!
Maybe these ignoramuses should
read the family systems literature on how siblings are treated differently. The
more dysfunctional the family, the bigger the differences!
Example number two, studies that
employs the ecological fallacy, involves those that compare two different psychotherapy
treatments for the same disorder. These studies are relatively uncommon, as
most psychotherapy outcome studies compare a treatment with a "control" condition like a
wait list or "treatment as usual" rather than with a second type of treatment. (Those control groups are also invalid, but that is a matter not relevant to this discussion).
In the few studies that compare
one school of therapy with another, an interesting statistic is that 85% of the
time, the treatment favored by the person designing the experiment
"wins" and outperforms the other treatment [Luborsky, L., Diguer, L., Seligman, D. A., et.
al. 1999. "The Researcher’s Own Therapy Allegiances: A “Wild card” in Comparisons of Treatment Efficacy."
Clinical Psychology: Science and Practice, 6, 95-106]. This is due to
something called the allegiance effect - the
more enthusiastic a therapist in a study is about their own school, the better the patient
tends to do.
But
even ignoring this clear-cut sign that research conclusions in comparative outcome studies are inherently misleading, let us suppose
that with one therapy treatment in the study, 45% of the patients improve
significantly, while in the other, only 30% do. The conclusion of the
researchers: the first treatment is superior.
Wrong.
This
conclusion presupposes that all patients react to treatments somewhat
identically, despite the fact that the majority (or at least a significant percentage) of patients in both arms of the study did not
improve. It is quite likely that some patients are more comfortable and do much better with one of
the therapy treatments than with the other. Of course there is no way
to know for certain, but it is quite possible that the 45 percent of people who respond to
the allegedly superior treatment are very different in many respects than
are the 30% who respond to the allegedly inferior therapy.
A
well-designed study, on the other hand, would have to recognize these differences and would look at the
characteristics of the four different groups that comprised the study: those
who got better with treatment A, those who got better with treatment B, those
who responded poorly to treatment A, and those who responded poorly to
treatment B. The researchers could then match the patient with the type of
therapy they seemed to do best with, and then and only then compare
outcomes.
In
other words, as I have pointed out elsewhere, no matter what therapy
intervention you use, some people will improve with it, while others either
will not improve or may even get worse! Different strokes for different folks, people.
