The Pervasive Weaknesses of Psychotherapy Studies

A psychiatrist with Intense, bulging eyes by C. Josef, CC Attributions 4.0

 

In my last post, I mentioned that the research into both psychotherapy outcomes and personality disorders is extremely weak, and even that characterization may even be giving it too much credit. Extensive clinical experience has been dismissed as “anecdotal,” even when therapists see the same things over and over again and their observations are confirmed by many other therapists who actually look at the same phenomena.

The irony here is that almost ALL of the “research” data in these two areas is a collection of anecdotes, since they are entirely based on patient self-report or the experimenters’ personal observations – all of which are subject to significant bias. We cannot read minds and people act and lie a lot, and a lot of other influences on the “data” are extant and unknown to the researchers.

Most psychotherapy outcome studies are characterized by frequent patient drop-outs and by the fact that a significant portion of the study subjects do not respond to the treatment being offered. And outcome measures in these studies are typically the relief of symptoms, not changes in the patient’s abilities to love, work, and play successfully. And the subjects are rarely followed up for a significant period of time to see if any results that are attained last. A significant portion of the study “gains” are often lost after a year or so.

There are over 200 different models for understanding psychopathology and doing psychotherapy, although most are variations of the five major models: psychodynamic, cognitive, behavioral, affect focused, and family systems. Most therapists borrow techniques from schools other than the one they were trained  in.

When results from several different studies using different schools are compared, most tend to come out with about the same success rates. In the beginning of a movement to try to integrate the different schools, this was known jokingly as the Dodo Bird verdict (after a character in Alice in Wonderland) – all have won and all must have prizes. And when two schools are compared in a single study, the school of person who is the lead author of the study comes out the winner in 85% of them (an allegiance effect). Bias, anyone?

Even then, when a certain percentage of the study subjects did respond to the “inferior” treatment, we don’t know whether or not they would have done well in the “better” treatment. Or if those who did not respond to the “better” one would have responded to the other treatment.

Over the years I have posted critiques of the “research” and in this post will summarize a bunch more of the points I made. If there is a whole post about them, I’ll include a link to the original.

A big one I mentioned in the last post: when a school of therapy is evaluated, the individual interventions which comprise them (of which there are quite a few) usually are not, so we don’t know which of them worked and which of them did not or were even counterproductive. Responses to the individual interventions are important to know about because, despite the use of treatment manuals supposedly insuring that all therapists in a study using a specific school are doing the same things, this is not possible. Subjects all respond differently to a given intervention. Therapists have to pick and choose which intervention will be used next. Also differing - with significant impact - is the way the intervention is presented: phrasing, body language, tone of voice etc.

Another major study weakness: Those that try to apportion causation of psychological behavioral syndromes to genetic vs. environmental influences use studies of twins raised apart. This type of study routinely over-estimates genetic contributions by assuming parents treat all their children alike, which is way off. Furthermore, they are looking at the end result of gene and environmental interaction (phenotype, not genotype) without any way to know how much of a given finding to apportion to each of them.

Most psychotherapy outcome studies exclude patients with more than one disorder, although a high percentage of patients have co-morbid affective and anxiety disorders as well as more than one personality disorder. The therapy will of course look more effective if you include only the easiest patients.

In studies of psychiatric symptoms which may occur in response to stress, reactions are evaluated without any reference to what the actual stresses were to which the subjects were responding. 

Confusion between correlation and causation is illustrated in such studies as those that attempt to determine the causes or the results of drug abuse. For example: Does marijuana cause poor school performance or the other way around - or is there actually a third factor which leads to both of them?

Differences in brain area size and functioning between different groups on fMRI scans are automatically interpreted as abnormalities. In fact, most differences are due to normal neural plasticity in response to changes in the environment.

In studying  the nature of the relationship between parents and children, No one can  precisely measure the nature of the relationship. These relationships are not constants but vary across time and situational contexts. Parents might be good disciplinarians when it comes to providing children with adequate curfews, for example, but terrible at allowing them to stay up all hours of the night. Furthermore, the disciplinary practices certainly change over time as the children get older. Second, how does a study even attempt to measure the tone of parenting practices? Third, oftentimes studies are based on parent self report. If a mother were abusive or inconsistent, how likely do these authors think she would admit to it, even if she were very self-aware, which obviously many people are not.

In some Cognitive Behavioral Therapy outcome sudies, therapy  is at times compared with "treatment as usual" —letting subjects get whatever other treatments outside of the study treatment that they chose to have, allowing good therapists and bad therapists, and good therapies and bad therapies, to essentially cancel each other out. Even so, the sizes of treatment effects are only small to moderate.  “Response” just meant there was some significant improvement in symptoms, not that the symptoms of the disorders actually went away. Rates for actual remission from the disorders were even smaller. A considerable proportion of study patients do not sufficiently benefit from CBT.

In epidemiological research into environmental risk factors for various psychiatric disorders, most studies try to measure the effect of a single environmental exposure on a single outcome—something that rarely exists in the real world. Individuals are exposed to environmental elements as they accumulate over time, so that one single exposure usually means very little. Exposure also is “dynamic, interactive, and intertwined" with various other domains including those internal to individuals, what individuals do within various contexts, and the external environment itself—which is constantly changing. Last but not least, each individual attributes a different, and sometimes changing, psychological meaning to everything that happens to them.

The difference between “cannot” and “do not:” Study are often characterized by lack of attention to subject motivation, and ignorance of the concept of “false self.” In one study, high-psychopathy participants showed atypical, significantly reduced neural responses in the brain on an fMRI to negatively-toned pictures under passive viewing conditions. However, this effect seemed to disappear when the subjects were instructed to try to maximize their naturally occurring emotional reactions to these same pictures!

Researchers mistake a high index of suspicion for an “inability” to correctly read the mental states of others.

Studies show that changing a parent’s behavior towards BPD children can make those with BPD better—but seem to ignore the possibility that their behavior apparently helped cause the disorder in the first place.

Different Schools of Thought in Psychotherapy

At last count, there were over 200 different "schools" of psychotherapy, each with its own ideas about why people act in self-defeating ways or in ways which bring them emotional or even physical pain, and how to help them to stop. Of course psychotherapy is, despite having been around for a hundred years, a young science, but our field is more difficult to study “empirically” than any other. 

The problems we have are enormous because we cannot read minds, and people can choose to some extent how they react to any therapy intervention. Patients withhold information about their situations from therapists all the time due to protecting their families from negative judgments, guilt, shame, or a concern the therapist might not be interested in it.

In psychotherapy outcome studies, seemingly minor variations in therapist techniques that are in fact vitally important (such as body language and tone of voice) aren’t even measured. There are no good active control treatments, and, when two therapies are compared,  the therapy method favored by the first author of the study comes out ahead 85% of the time due to the authors’ biases (allegiance effects). 

We cannot do double blinding because that would mean the therapists wouldn’t know what they were doing, which would not be a good test of the treatment. And of course once again there can be a major lack of complete candor by subjects. Much of the study results are based on patient self report, a notoriously unreliable method of data collection. And there is no way to distinguish an act patients may be playing for their family of origin (a false self or persona) from their real beliefs and feelings, or performance from ability.

The ecological fallacy – thinking all patients with a particular disorder react exactly like an average patient - is rampant in the literature. If 20 % of clients with a particular problem respond to one intervention and 40% respond to a second one, this does not mean that the second one is better for everyone than the first. The 20% who responded to the first one could actually get worse with the second one.

There is also a huge and highly problematic groupthink problem in the psychotherapy field, with purveyors of various schools claiming a monopoly on truth. Often the need for ideological purity, the admiration for an academic leader within a hierarchy, or the profit motive causes science to take a back seat in favor of a group's other interests. 

Fallacious arguments ensue. One of the most common is that entire complex groups of theoretical constructs that characterize a given school are rejected in total by another school, as if, if one theoretical part of a school is wrong, the whole thing must be wrong. Psychoanalysis may have been wrong about penis envy, for example, but dismissing intrapsychic conflict entirely as a construct because of that is - in a word - stupid.

Another is that a phenomena that two schools are looking at but explain differently are just being called different names and are given different explanations, which are then accepted by a given school as gospel without even a thought to investigating other possible explanations. I recently wrote in a post about how both the cognitive-behaviorists' "irrational thoughts" and the psychoanalyst's "defense mechanisms" probably serve the same purpose, but that neither school explains that purpose with reference to group dynamics - IMO the key factor.

There is still hope. IMO we have to look for recurring patterns in our therapy patients (not in research subjects, because contact is minimal) as well as within their social milieu. At times, we have to meet with clients along with their significant others in order to get a more well-rounded picture. We have to do so in long-term psychotherapy, because it takes quite a while for the whole story to unfold. 

We should do this in order to figure out commonalities and in order to figure out what questions to ask. In particular, we should look for evidence of motivated reasoning in what our clients report – logical fallacies, inconsistencies and contradictions (sometimes voiced months apart – the importance of extensive therapy notes cannot be overestimated), and defensive reactions. If handled well, this will help us unearth what clients may be trying to hide from us. 

Doing so also suggests questions we may have not thought to ask, or pay attention to environmental variables we were not even aware of that turn out to be major contributing factors to psychopathology that demand attention.

The Amazing Complexity of Environmental Research in Psychiatry

In my Psychology Today post of 12/24/12, Why Psychotherapy Outcome Studies are Nearly Impossible, I discussed the large number of variables that are not taken into consideration in those studies which bring any conclusions drawn from them into question. These include variations in therapist techniques that aren’t measured, sampling problems with people that can have wide variations in their proclivities and sensitivities, problems with finding an active control treatment, the lack of double blinding, and lack of complete candor by subjects.

The same types of issues apply to epidemiological research into environmental risk factors for various psychiatric disorders. Most studies try to measure the effect of a single environmental exposure on a single outcome—something that rarely exists in the real world.

In a “viewpoint” article from JAMA Psychiatry published online on June 6, 2018, by Guloksuz, van Os, and Rutten ("The Exposome Paradigm and the Complexities of Environmental Research in Psychiatry"),the authors discuss characteristics of the environment as they do function in the real world. They speak of multiple “networks of many interacting elements…”

Individuals are exposed to these elements as they accumulate over time, so that one single exposure usually means very little. Exposure also is “dynamic, interactive, and intertwined" with various other domains including those internal to individuals, what individuals do within various contexts, and the external environment itself—which is constantly changing. Last but not least, each individual attributes a different, and sometimes changing, psychological meaning to everything that happens to them. This meaning attribution can alter the effect of each environmental exposure dramatically.

Each environmental factor confers risk for a "diverse set of mental disorders." These factors are far from universal so that some people remain completely unexposed to them. They interact with each other so they are not independent. They are time sensitive. They are dose dependent even within similar environments, meaning individuals are not exposed to them at the same level. They can be subject to being confounded by each individual’s differing genetic propensities.

With all that to consider, drawing final conclusions from a few studies just does not cut it as real science. But the field tends to believe in those conclusions as if they were gospel.

Cognitve Behavioral Therapy "Evidence-Base" Grossly Exaggerated

In my post on my Psychology Today blog on November 21, 2011, I discussed how the purveyors of today’s most predominant psychotherapy methodology, cognitive behavioral therapy, grossly exaggerate the strength of their research evidence base in the psychotherapy outcome literature.

My opinion was recently confirmed in a review of meta-analyses of the CBT literature in the Journal of the American Medical Association, published online September 21, 2017 (“Cognitive Behavioral Therapy the Gold Standard for Psychotherapy:  The Need for Plurality in Treatment and Research” by Falk Leichsenring and Christiane Steinert).
 
They reported that a recent meta-analysis using criteria of the Cochrane risk of bias tool reported that only 17% (24 of 144) of randomized clinical trials of CBT for anxiety and depressive disorders were of high quality. The “allegiance factor”—study authors were CBT therapists themselves and often designed the studies to make their treatment look better than it was, and opposing treatments look worse that they were—was rarely controlled for.

Compared with "treatment as usual" —letting subjects get whatever other treatments outside of the study treatment that they chose to have, allowing good therapists and bad therapists, and good therapies and bad therapies, to essentially cancel each other out—the sizes of treatment effects were only small to moderate and might eventually even be found to be due to the allegiance effects.

In panic disorder, CBT was not more effective than treatment as usual but only to being on a waiting list.

Even with these amazing biases, for depressive disorders, response rates of about 50% were reported. This was true for anxiety disorders as well. “Response” just meant there was some significant improvement in symptoms, not that the symptoms of the disorders actually went away. Rates for actual remission from the disorders were even smaller. Conclusion: a considerable proportion of patients do not sufficiently benefit from CBT.

Last but certainly not least, there was no clear evidence that CBT was more effective than other psychotherapies, either for depressive disorders, anxiety disorders, personality disorders or specific eating disorders.

Personally, my biggest beef with CBT and other psychotherapy outcome studies has less to do with symptom relief than with actually changing maladaptive interpersonal behavior. The latter is almost never even looked at, let alone measured in these studies.

CBT’ers seem to think anxiety, depression, and self-destructive behavior are all due to screwed up thinking by individuals rather than being normal reactions to stress-inducing environments. In experimental psychology circles, this is known as the fundamental attribution error. Telling people with these particular symptoms that their problems are basically “all in their heads” in this manner is very invalidating for them.  Ironically, an ‘invalidating environment” is one of the two primary factors these very same therapists cite as the main causes for borderline personality disorder.

Stupid Researcher Tricks

When David Letterman was a late night TV talk show host, he had a recurring segment called "stupid pet tricks." In this vein, this post is titled "stupid researcher tricks." The authors of certain studies in psychiatry employ logical fallacies - either knowingly or in some cases unknowingly - in their discussions of their results. This often results in their drawing totally misleading conclusions from their data.

I will discuss two telling examples of such fallacies. The first involves employing a hidden assumption that is not true. I discussed the general issue of hidden assumptions in a previous post. The particular example in this post can actually be thought of as Part II of my post on the heritability fraud. The second example involves a subtle logical fallacy known as the ecological fallacy: making inferences about individuals based on data obtained that characterizes an entire group, using averages on various measures, to which that individual belongs.

In the previous "heritablility" post I wrote about how that term is falsely used as a synonym with "genetic" in studies that purport to sort out genetic versus environmental factors leading to the development of personality traits or behavioral disorders. The statistic is developed from twin studies: identical versus fraternal twins, and/or those identical twins raised together and those raised apart. 

As previously described, these studies - despite frequent claims to the contrary - do almost nothing to sort out genetic versus environmental influences. The reason is because the statistic is based on phenotype (the final interactional product of genes + environment) and not genotype. As such, it includes a mix of purely genetic factors and factors that result from gene-environmental interactions, and there is no way to know how much of each is contained therein.

Another aspect of these studies I mentioned in the previous post: they also divide environmental influences into "shared" (family and home) and "unshared" (peers, media, teachers, and other outside factors). I mentioned that a determination of which parts of an environment are shared by siblings and which are unshared has a lot in common with finding water with a divining rod, because parents do not treat all of their children alike. Nor does each twin have exactly the same interactions with each and every other family member from the moment they are born to the moment they die.

Interestingly, the "shared" environments in these studies usually come out as less important in leading to behavioral issues than the "unshared!" This would mean that family and parental behavior is less of a factor in personality development than outside influences - something that runs counter to logic for a variety of reasons (for example: which peer group someone chooses to hang out with - when there are several different ones to choose from -  is not an accident).  

The researchers almost have to find that family is less important that peers and media because they just ASSUME that each twin is subject to identical influences inside the home. If you make this assumption, and then if the twins turn out differently on some characteristic, of course the home will appear to have no influence!

Maybe these ignoramuses should read the family systems literature on how siblings are treated differently. The more dysfunctional the family, the bigger the differences! 

Example number two, studies that employs the ecological fallacy, involves those that compare two different psychotherapy treatments for the same disorder. These studies are relatively uncommon, as most psychotherapy outcome studies compare a treatment with a "control" condition like a wait list or "treatment as usual" rather than with a second type of treatment. (Those control groups are also invalid, but that is a matter not relevant to this discussion).

In the few studies that compare one school of therapy with another, an interesting statistic is that 85% of the time, the treatment favored by the person designing the experiment "wins" and outperforms the other treatment [Luborsky, L., Diguer, L., Seligman, D. A., et. al. 1999.  "The Researcher’s Own Therapy Allegiances:  A “Wild card” in Comparisons of Treatment Efficacy." Clinical Psychology: Science and Practice, 6, 95-106]. This is due to something called the allegiance effect - the more enthusiastic a therapist in a study is about their own school, the better the patient tends to do.

But even ignoring this clear-cut sign that research conclusions in comparative outcome studies are inherently misleading, let us suppose that with one therapy treatment in the study, 45% of the patients improve significantly, while in the other, only 30% do. The conclusion of the researchers: the first treatment is superior.

Wrong.

This conclusion presupposes that all patients react to treatments somewhat identically, despite the fact that the majority (or at least a significant percentage) of patients in both arms of the study did not improve. It is quite likely that some patients are more comfortable and do much better with one of the therapy treatments than with the other. Of course there is no way to know for certain, but it is quite possible that the 45 percent of people who respond to the allegedly superior treatment are very different in many respects than are the 30% who respond to the allegedly inferior therapy.

A well-designed study, on the other hand, would have to recognize these differences and would look at the characteristics of the four different groups that comprised the study: those who got better with treatment A, those who got better with treatment B, those who responded poorly to treatment A, and those who responded poorly to treatment B. The researchers could then match the patient with the type of therapy they seemed to do best with, and then and only then compare outcomes.

In other words, as I have pointed out elsewhere, no matter what therapy intervention you use, some people will improve with it, while others either will not improve or may even get worse! Different strokes for different folks, people.

Where the Analysts Went Wrong: Introduction

Cognitive Behavioral Therapy (CBT) is currently the predominant psychotherapy treatment paradigm taught to clinicians-in-training in psychology graduate schools. However, when I first received psychotherapy training in the mid 1970’s, by far the predominant school of psychotherapy was psychoanalysis. We did receive a smattering of behavior therapy training, and we were even assigned one book about family systems ideas by Virginia Satir. Our training program was a bit unusual in that regard. 

Just like the CBT industry does now, analysts exaggerated the validity of the scientific evidence for psychoanalytic theory, and made grossly inflated claims about the effectiveness of psychoanalytic treatment. Its theory was applied to everything, even to schizophrenia, although by then it was pretty clear to most of us that they were completely wrong about that condition. I’m surprised analysts did not try to treat ingrown toenails with psychoanalysis.

Just as now, economics played a huge part. (That’s why I just called it the CBT industry).  When I made some rather mild criticisms of one aspect of CBT theory and practice on my Psychology Today blog, a CBT therapist wrote a rebuttal, and several people wrote in to say how unethical I was because I was not practicing a “scientifically validated” form of psychotherapy, and was therefore by implication a snake oil salesman. 

Readers of my book, How Family Dysfunction Spurs Mental Disorders, know that the “evidence base” in all of psychotherapy is actually quite weak, and that CBT controls the funding of research and denies it to practitioners from other schools – although nonetheless there are still many studies from other schools that are every bit as strong (or should I say as weak) as the CBT studies. That includes analytically-oriented therapies, as a meta-analysis by Jonathan Shedler in the February-March 2010 issue of the American Psychologist clearly showed.

Not to mention that most psychotherapy outcome studies rely on so-called treatment manuals that spell out what therapists are supposed to do, so that all the therapists in the study are doing the same thing – regardless of how the patient is reacting to what they are doing.  NO competent practicing clinician does that. 

Psychotherapy involves science of course, but in many ways it is an art form as well. People are complicated, and each patient is unique in many ways, and can always choose to respond positively, negatively, or not at all to any intervention made by a therapist.

Additionally, each treatment manual is comprised of multiple ingredients – some of which may help or be the primary active ingredient, some of which may do nothing, and others of which may actually be counterproductive. And yet many CBT proponents argue as if everything they do has been scientifically proven.

It is clear to me that CBT practitioners say this stuff for the same reason that Big Pharma controls and distorts the practice of doing, and the dissemination of results from, studies of medication effectiveness: They want to “sell” their goods and stamp out the competition. (BTW, that does not mean that therefore a pharma-sponsered study has NO validity - only that bias must be taken into account).

Analysts also protected their turf back in the day, very arrogantly, although ultimately they failed. As a trainee, if you criticized any aspect of analytic theory, you were told in no uncertain terms that you needed to go into psychoanalysis yourself, so you could find out why you were “resistant” to analytic theory. In other words, the only reason you were questioning the theory was because you were neurotic!  

This recommendation involved the use of not one, but three logical fallacies, all wrapped up in a single statement. It was a non sequitur, since someone might be questioning the theory for any number of other reasons besides their own psychological issues. It was an ad hominem attack, since it was going after the questioner and not the question. And of course it was begging the question.  The accusation of being neurotic might be true if analytic theory is true, but that is the very issue in doubt and under debate.

Another trick that different economic interests in mental health use to denigrate the competition also involves the issue that all of the so called schools of therapy do not consist of single ideas, but consist of a large number and a wide variety of different ideas and techniques that are tied together by some common threads. Anyone who bothers to think about it has to know that some of these ideas could be wrong while others could still be right, or that an idea may have validity for some situations and for some phenomena while being inappropriate for explaining or addressing others.

For instance, people in the field may try to argue against all the ideas of a particular school, even though many are obviously quite valid, by throwing up a few incidences of when the school had something clearly wrong. They will of course pick the most egregious examples they can find. To criticize psychoanalysis, they might bring up such discredited - and now fairly much discarded - analytic ideas such as "penis envy," or the wild overemphasis on the Oedipus Complex.

Hyper-biological psychiatrists love to bring up the awful effects of the fallacious psychoanalytic theory of the “schizophrenogenic mother” to argue against all of psychotherapy, not even just psychoanalysis! I tell biological reductionists that if they don’t hold the theory of schizophrenogenic mothers against psychotherapists, then I will not hold the theory of eugenics against them.

I even hear Pharma-sponsored speakers indirectly and implicitly attack therapy by posing really, really stupid questions such as “What is better for depression –medication or psychotherapy?”  Well, first of all, that’s like asking, “What’s better for patients with coronary artery disease, taking nitroglycerin when they feel chest pain, or losing weight?” Those two interventions target two completely different aspects of the problem. 

Second, the question lumps together all types of psychotherapy. What type of therapy are we talking about? Which interventions? It would be like me arguing against the use of medication by pointing out how ineffective penicillin is for treating clinical depression.

Just as with many aspects of CBT (CBT'ers please take note that I am saying that), many aspects of psychoanalytic theory retain much explanatory power. They are so widely accepted that they have even become part of the cultural conventional wisdom in industrialized countries.

Who doesn’t believe that people sometimes take their anger out about something on someone or something else?  Mad at your boss, come home and kick the dog? That’s the defense mechanism of displacement. Yeah, like that never happens.

Intrapsychic conflict creating emotional and interpersonal problems because people want something really bad but feel guilty about it? Check. Conversations have unspoken subtexts?  Check. Acting towards authority figures in a certain way because they remind you of your father? That’s transference. Check. CBT folks may prefer to call the phenomenon schemas instead, but it's still transference. 

People wanting to avoid unpleasant subjects and in response changing the subject or explaining away inconvenient facts? That's resistance. Check.

Forgetting about unpleasant memories? Well, whether that’s unconscious or subconscious may be debatable, but the fact that repression exists? The whole Catholic Church child molestation scandal started with a case of "recovered" memory. Check.

The analysts are even right about personality problems stemming from childhood experiences within the family. The attachment literature is extremely powerful, and we all know that one of the biggest risks for just about every psychiatric condition in the DSM is a history of childhood abuse and/or neglect.

Ah, but there is where the psychoanalysts started to go wrong. They seemed to assume that childhood experiences completely determined what psychological problems a patient has and that subsequent experiences were somehow inconsequential. Orthodox analysts believe that your personality is fixed by the time you are five years old. Some go back even further than that.

Of course, if subsequent experiences could not affect personality, it would do a person no good at all to go into psychoanalysis, because the experience of psychotherapy would have absolutely no effect - according to the orthodox analysts’ own assumptions about personality formation.

The thing is, family experiences that start to create problems for children do not magically disappear when a child reaches a certain age.  In fact, they often go on and on and on in sometimes somewhat different forms until the parents die. And the human brain is structured to be highly responsive to what parents do, even in adults.  That will be the subject of Part II of this post.