At the end of Part
I of this post of September 12, 2012, I mentioned that doing
psychotherapy outcome studies is diabolically difficult, and promised to
discuss the reasons why in this post. In fact, because of the literally
infinite numbers of uncontrolled variables in studies of human beings both
individually and in relationship to other human beings, psychotherapy outcome research
can never be the only standard by which the
"science" of human behavior-change technology should be measured.
In fact, it's not even the gold standard.
In order to better understand ourselves and what leads
us to change our behavior, we must use ALL available sources of information. We
have to look at the widespread clinical experience of psychotherapists who use a
variety of techniques and theories with a variety of clinical populations. We have to look for potential biases in both clinical studies and within an
individual therapist's anecdotes and the conclusions that we draw from them. In forming conclusions about both anecdotal and controlled-trials data, we have to look at a wide variety of possible explanations, as well as for any information and experiences that would seem to contradict those explanations.
We also have to look at the experience of marketers
who have been able to induce a large numbers of people to buy a product or vote for a
politician (many of whose very powerful techniques have been discussed in
previous posts on this blog). We have to look at the social psychology
literature, which shows that people behave very differently when interacting with
different groups of people and with different individuals than they might do in a
therapist's office.
We have to look at historical and sociological trends. We have to look at new knowledge from the neurosciences that might
account for findings that are difficult to explain or reconcile with other
beliefs. We must look at evolutionary biology. We must examine our own
beliefs for logical inconsistencies. We have to be honest about thinking about
ourselves what makes us tick personally.
What follows is a list of some of the difficulties psychotherapy outcome researchers encounter, as well as the reasons they may overstate
their results. A fuller discussion of each of them can be found in my
book, How Dysfunctional
Families Spur Mental Disorders.
1. The Problem of the “False Self”
People
do not act the same way in all social contexts.
They do not act or speak the same way around a boss that they do when
they are alone with a lover. A man’s
behavior in a strip club is very different than his behavior when he is playing
with his children. We have different
“faces” or masks which we apply to ourselves in different environments. Not infrequently, these masks are meant to
manipulate others to get them to do what we want them to do.
I
never cease to be amazed at how mental health professionals and researchers
seem to believe that they really know what is going on in a patient’s or a
research subject’s life based solely on the self report of the patient, or
solely on the reports of the patient’s intimates, or even on the reports of
people like teachers who observe the behavior of children in only one context
that includes thirty other distracting students. If these professionals were asked if they
believe that people often act differently in public than they do behind closed
doors, they would of course say yes, but they seem to develop amnesia for this
fact in discussions and in studies.
A
patient’s family members may be just as motivated to give a distorted view of a
patient as is a patient. Parents, for
example, may prefer to believe that their child has some sort of mental defect,
so as not to experience as much of their own covert guilt about their
parenting skills. Conversely, some may
actually prefer to blame the child’s behavior completely on themselves, in
order to let their “perfect” child off the hook. Most mental health practitioners do not make
home visits to watch patients and family members interact in their natural
environment. Even if they did, unless
they had a camera operating twenty four hours a day as in the movie The Truman Show, they could still be easily deceived.
When
it comes to psychotherapy treatment outcome studies, we cannot do double blind
placebo controlled comparisons of two different types of psychotherapy
treatments. This is true because, in a sense, the therapist – or more correctly
the relationship between the patient and the therapist – is the treatment. If the
study were to meet the criteria for being double blind, that would mean that
the therapists who administer the treatment would have to not know what they
were doing.
Of
course, they cannot administer psychotherapy without being aware of what
techniques they are using. If they
could, that would mean that they were incompetent. Not a fair test of a treatment! The fact that the therapy relationship is one
of the basic aspects of the treatment also makes placebo or “sham” treatments
difficult, because any relationship
has some effect on an individual. Does this mean we should give up on
evaluating psychotherapy treatment scientifically and rely exclusively on
clinical anecdotes? Of course not. Studies are still important. We just have to understand their limitations.
One
strategy used in therapy studies is to compare the outcomes of two different,
presumably active types of psychotherapy treatment techniques with each other,
rather than with placebo, on similar groups of patients, without the benefit of
double blinding. These studies are
referred to as randomized clinical trials,
or RCT’s.
A
large number of different schools of
psychotherapy have different approaches to the understanding of and methodology
for changing a patient’s repetitive dysfunctional habits. Most of these therapy schools were designed
by charismatic and creative individuals who based their ideas on clinical
anecdotes. These innovators are highly invested emotionally in their own personal
theories, and want them to look good in comparative psychotherapy outcome
studies.
This
leads to a so-called allegiance effect
in RCT’s. The preferred psychotherapy
school of the researcher is likely to be delivered more enthusiastically and
with more rigor to subjects in the study than is the competing therapy
treatment. One survey study examined 29
RCT outcome studies that compared one type of therapy to another and found a
correlation of .85 between researchers’ therapy allegiance and outcome. That is, the researcher’s preferred treatment
came out ahead 85% of the time. Just as in sponsored drug studies, this number
is too way high to discount the presence of a significant bias in the studies.
When
differences are found between two therapies, they are often statistically but
not clinically significant. They show that one therapy is slightly more advantageous than the other, but that the actual improvement of the subjects was so minimal as to be inconsequential.
When both
groups of therapists who are providing the treatment in the study are equally
committed to the paradigms they are delivering, comparative psychotherapy
outcome studies almost always result in a tie. In psychotherapy research circles, it is known as the “dodo bird verdict.” This refers
to a character from Alice in Wonderland,
the dodo bird, who in one passage said “Everybody has won, and all must have prizes.”
Lately,
many researchers engaged in psychotherapy RCT’s have employed a control group
called treatment as usual (TAU), which
really stacks the deck in favor of their pet psychotherapy school. Lining up practitioners of a different school from the researcher’s to
act as therapists for a comparison group is often difficult. Additionally, if the researcher were able to
do so, the study might not show his or her therapy to be superior to another
type. For these reasons, the use of TAU
control groups has become almost epidemic in psychotherapy RCT’s.
Subjects
randomly assigned to the TAU condition, which serves as a comparison group to
the group of subjects receiving the researcher’s therapy model, are simply
released back into the community to get whatever other treatments are already
out there. Some may see practitioners
from other therapy schools, some may get medications, some may get both, and
many others may get neither. Both the
TAU group and the experimental group are followed up at equal time intervals
and given all the same outcome measures.
The psychotherapy methodology that serves as the investigated treatment
always seems to beat TAU.
The
reader should understand that within any widely-practiced therapy model, both
good therapists and bad therapists can be found, just as practicing physicians
can be either good or bad psychopharmacologists. For the subjects in the TAU condition who
receive treatment, the results of the good clinicians in the community are
probably cancelled out by those of the bad ones. TAU subjects may also be seen less
frequently. Some, as I mentioned, are getting no treatment
at all. Meanwhile, the experimenter’s
group is usually getting a lot more individualized attention by therapists who are highly committed to the particular treatment model. These factors may be the
real reasons their patients do better than those receiving TAU.
The
experimenter’s therapy is provided by uniformly well-trained and enthusiastic
therapists under very well controlled conditions. The psychotherapy that is provided is applied
with rigor and consistency, and is scrutinized by other observers through the
use of videotapes of the sessions. Therapists who make errors are supervised almost immediately. On top of this, research therapists often
have caseloads that are very much smaller than those of folks out in practice,
allowing them to spend more time deciding how to approach the clinical issues
they face. I cannot recall a single
instance of a study in which TAU beat another therapy delivered in such a manner. If it did, I would have to wonder how the
experimenter could have possibly accomplished such an unlikely feat.
Another
huge controversy in psychotherapy RCT’s is the question of which outcome
measures should be used, because the different psychotherapy schools target
different aspects of the patient's problems. What should we measure, exactly? Symptom relief? Whether or not the patient continues to meet
criteria for a specific DSM disorder?
Whether they stop engaging in chronic repetitive dysfunctional
behavior? Whether they experience fewer
conflicted emotions? Whether they have
improved and more satisfying relationships? Whether they can hold a good job?
In
their November 1995 issue, Consumer
Reports published a large survey on the public’s experience with
psychotherapy. In response to the
survey, satisfaction with their experience was reported widely by those who had
been in therapy. Furthermore, longer
term therapy was rated higher than therapy limited in duration by managed care,
and satisfaction with psychotherapy was rated about equally by those who had
therapy alone and those who were given a combination of therapy and medication.
In
surveys of this sort, the question of selection bias arises - whether or not
the sample of people polled is representative of the general therapy population. The respondents in this case were almost all
middle class and educated, so we have to limit our conclusions to that
population rather than the population of the United States as a whole. Nonetheless, the study was the most extensive
study of its type on record. Should
opinions expressed by patients evaluating psychotherapy outcomes for themselves
count as science? Many in the field
would say no.
But one must consider,
however, that psychotherapy target's a patient’s subjective sense of personal well being. That is, we want to know whether patients
feel better mentally about their
lives. By this measure, assuming that
the survey respondents were being somewhat honest with their answers, we would
have to say it should count. A paradox
about the “science” of psychotherapy is that we are attempting to be objective
about subjectivity, two concepts which are usually defined as antonyms.
Another big issue for the field concerns which
psychotherapy treatment outcome research studies get funded. If a scientist cannot get money for a
project, it will rarely get off the ground, because these types of studies are
very expensive to mount. Most successful
psychotherapy RCT’s have employed CBT because of the predominance of this model
in professional psychology training programs, and because most CBT treatments
aim primarily for symptom reduction, which is relatively easy to measure,
rather than personality change, which is not.
Another
big problem with all psychotherapy RCT’s is that most studies require that the
subject population be homogeneous,
meaning that the subjects in the study must be very similar in the nature of
the disorder they exhibit, and in how severe it is. This requirement means that patients who have
more than one DSM disorder or psychological problem are often excluded from
studies. In contrast, most patients seen in practice,
at least by psychiatrists, have more than one disorder (co-morbidity). This fact
alone limits what is termed the generalizability
of the study’s finding. We do not know
from a study using patients who have only one disorder if the treatment
employed in the study would work as well with patients who have multiple
problems.
Because
studies need subjects who will stick with the treatment until the end of the
research project, some subjects who have certain characteristics that are
common in clinical practice tend to be excluded. This problem further limits the
generalizability of the study. For
instance, most studies of treatments for depression exclude patients who are
suicidal! Another problem with RCT’s is
that many subjects drop out of a study or are removed from a study as it
proceeds because they do not completely cooperate with the treatment in some
way. CBT studies, as do many others,
tend to have a fairly high dropout rate. The subjects who end up completing the study are usually the most
motivated to change, and would therefore be expected to do better than those
who lack this motivation. This all makes
the treatment method look much better than it would be if it were employed in a
typical clinical practice setting.
Another big issue in psychotherapy outcome
studies is that, in comparing different treatments for different behavioral
problems, researchers need to have some way of knowing that all the therapists
employed in the study are doing mostly the same things. In early psychotherapy studies, the methods
section of a journal report might say that such and such number of patients was
“given psychoanalytic therapy,” with no description of what the therapists
actually did. This was clearly
unacceptable.
The
solution to this problem that has been employed most frequently is the use of treatment
manuals, as well as manuals that measure the study therapists’ competence in
and their adherence to the treatment paradigm.
Treatment and adherence manuals specify exactly what the therapist should
be doing under a variety of circumstances. Tapes of psychotherapy sessions of patients in the studies are reviewed
to make sure that the therapists are employing the same interventions. A “rater” will count the times the therapist
did the right thing and the times the therapist strayed from the procedures
specified in the manual. This process
assures that all patients are getting approximately the same treatment.
Unfortunately, a good therapist has to be flexible and employ a variety of different strategies in
ways that are tailored to the proclivities and sensitivities of the patient in
front of them. Generic interventions may
not only fail to work, they may backfire and make matters worse. Verbal interventions must often be phrased
differently to different patients in order to reduce the patient’s
defensiveness. Treatment and adherence
manuals take away a lot of this flexibility, so that the therapy as performed in
an RCT is not always similar to the way that clinicians out in the field
practice it. Creating treatment manuals
can itself be a daunting task. I heard a
respected researcher tell a group of other researchers that his team was having
trouble designing such a manual because the founder of the treatment they
wanted to study was observed to perform psychotherapy completely differently
than his own wife, who supposedly was a practitioner of the same model of
therapy.
No comments:
Post a Comment