|
Cherry picking: pointing to individual cases or data that
seem to confirm a particular position, while ignoring a significant portion of
related cases or data that may contradict that position
|
In
part I of this post, I discussed the issue of inductive reasoning, and how it
can be a source of quite valid scientific evidence. This brings up the question of what is a
valid conclusion based on inductive reason versus an invalid conclusion based
on one person’s personal experience.
Anecdotal
evidence
in medicine is often misleadingly defined as evidence based on only one
clinician’s personal experience with a treatment or diagnosis in question. If
that is the standard that is to be used, clearly many reasons exist to question
the validity of inferences drawn from these experiences. Individuals are well
known to have various biases that color their observations and the conclusions
they draw from them.
They may have blind spots because of their own emotional
conflicts. They may ignore evidence that is contradictory to their point of
view. Their observations may be limited by their pet theories about the
phenomena in question.
Perhaps even more importantly, they may be seeing unusual cases that are not
representative of more “typical” cases in one way or another – a so-called selection bias.
An
obvious case of selection bias was illustrated by a statement I heard made at a
conference by a family therapy pioneer, the late Jay Haley. I had always
admired Mr. Haley for many of his fascinating and utilitarian ideas and
observations. However, in this case he betrayed some ignorance. He stated that
he did not believe antidepressant drugs were ever effective because none of the
patients referred to him had ever responded to them.
Of
course, his being a well-known family therapist who did not believe in
medication had a tremendous effect on exactly who would be referred to him. Not
everyone does respond to drug treatment. Anyone who had responded to an antidepressant would, in all probability,
rarely if ever darken his door. Hence, with his sample, he would be misled into
thinking that the medicines were not effective for anybody. This form of bias
is very common and can be quite subtle. For example, it can affect one’s
beliefs about such matters as racial stereotypes or a determination of how
trustworthy members of a city’s police department are.
Do
these types of biases invalidate all clinical experience? Hardly. First of all, we have to distinguish
between the descriptions of the actual events contained within specific
anecdotes, and the conclusions or
inferences which are drawn from these events.
Let us first examine the descriptions
of what actually happened. A specific anecdote may be accurately observed and
described, or not so accurately. If important details are altered or left out
entirely, the anecdote may indeed be worthless. However, the exact same thing
can be said about empirical studies.
Important
details may not even be known to an observer. Particularly with observations of family
behavior within a practitioner’s or researcher’s office, important information
is almost always hidden, either by design or unwittingly. In addition to the fact that one does not see the
whole picture in any single context, there is also a basic problem inherent in
the nature of interactions between intimates.
With
verbal behavior, for instance, linguists refer to a quality called ellipsis. What this means is that in
conversations among people who have known each other for a while, certain
information is not spelled out verbally because the other person already knows
it. Strangers such as therapists who are listening in and who have not been
privy to these prior experiences may think they know what the family is talking
about, but they may in reality be completely clueless.
Let
us now consider the separate issue of conclusions that are drawn from
anecdotes, as opposed to their description. The questions raised by an
accurately-described clinical observation can be quite valid, but the answers
inferred from it can be completely wrong. Conclusions based on clinical
“anecdotes” exist on a continuum from relatively accurate ones to those that
are extremely biased to those that are based on spectacular inferential or
logical leaps of faith.
Relatively
unbiased clinical conclusions based on anecdotes by mental health professionals
have many things in common:
· They
are based on a sample that one has a reasonable expectation is at least
somewhat representative of a larger population.
· They make use, not just the practitioner’s
observations, but of the observations of other professionals whom one knows to
be reliable and open minded. These clinicians should also be ones known to take
the time with their patients necessary to take a complete history. Widespread
clinical experience by competent clinicians is something upon which someone can
make a very valid inductive conclusion, and is not just “anecdotal evidence.” In fact, conclusions drawn from this source
tend to be more accurate that those drawn from so called “empirical” studies. Many dangerous side effects from drugs that did not show up in the initial drug studies have been discovered in this manner.
· They make use of other informants besides the
patient when possible.
· They take into consideration that people and
their family members behave quite differently behind closed doors than they do
in public, and therefore if at all possible include observations of patient
behavior when patients are unaware that they
are being observed.
· They are based on longitudinal observations. That
is, the patients on whom conclusions are based have been seen on multiple
occasions over an extended period of time.
· They are not contradicted by commonly
observed examples of behavior in everyday life related to the behavior in
question.
· The person proposing the conclusion
acknowledges potential biases, such as a financial stake in a certain drug or
allegiance to a specific school of therapy, and acknowledges his or her
limitations. What former president of the Society of Clinical Psychology,
Gerald C. Davison, calls “ex cathedra statements based upon flimsy and
subjective evidence,” a hallmark of some psychotherapy gurus, are
always highly suspicious. In fact, charlatans are relatively easy to spot. They have a tendency to sidestep challenges. I will give an example of this shortly.
· The conclusions reached should lead to
predictions of patient behavior under certain circumstances that prove to be
accurate in a significant number of cases. This is called predictive validity. Of course, human behavior being as
unpredictable as it is, at times the predictions will not be completely
accurate even if the conclusions are valid, and so this fact must also be taken
into account.
· Conclusions based on anecdotes about
treatment efficacy or the reasons for certain observed behavior should consider
several alternate possible explanations for the observations. If several
explanations are possible, one must make a judgment about which ones are more
likely and which are less likely based not on the anecdote alone, but on all sources of data available. These
sources include empirical studies, but also include observations from everyday
life, as well as material seen in some relatively reliable media such as
reputable newspapers.
Now
of course stories in the media also do not tell the whole story or may be biased,
so one needs to realize again that one can be fooled, and take this into
account as well. I used to believe the common myth, for example, that in nature
under certain conditions the animals called lemmings would follow each other
off a cliff and commit mass suicide.
I
was surprised when I learned that this was untrue because I had as a child in
1958 seen a film clip of said mass suicide that was part of a Disney “True Life
Adventure” nature movie called White
Wilderness. I later learned that, because the Disney crew could not find a
real example, they had from behind the scenes driven the group of lemmings off
the cliff for the cameras.
On
the other hand, many people believe that men have never been to the moon and
that films of the moon landings were made in a movie studio using special
effects. I must say, I tend to believe that those film clips are real, but few
know for certain.
· If
other anecdotes about similar patients and treatments seem to contradict the
conclusions based on a given anecdote, an attempt should be made to account for
this difference.
As
an illustration of the latter point and an example of a the “quick step side
step” in scientific presentations. I once heard an expert present new evidence from neuroscience that certain
capabilities of which human brains are capable seem to develop only at certain
times during early childhood development. This brain development could be
adversely affected by a baby’s early social environment. Of course, that is
somewhat true.
Like
psychoanalysts will, however, the expert went on to conclude that if the
adverse early experiences had taken place, the child had no chance of growing
up to be normal. I raised my hand and asked about those children who come from
horribly adverse backgrounds, are adopted away at an age past the alleged
crucial developmental time, and yet still turn out wonderfully. The expert then
changed the subject without ever addressing my question.
If the data doesn't fit your pre-conceived conclusions, just change the subject!
Remember, there are NO empirical, placebo-controlled, double blind controlled studies on whether parachutes reduce the incidence of deaths or injuries after falling out of airplanes. Or that appendectomies are effective in preventing complications and deaths from appendicitis. And yet we all take those things for granted.