Anecdotal Evidence and Inductive Conclusions in Psychiatry: Part II

Cherry picking: pointing to individual cases or data that seem to confirm a particular position, while ignoring a significant portion of related cases or data that may contradict that position

In part I of this post, I discussed the issue of inductive reasoning, and how it can be a source of quite valid scientific evidence. This brings up the question of what is a valid conclusion based on inductive reason versus an invalid conclusion based on one person’s personal experience.

The following comes from my book, How Dysfunctional Families Spur Mental Disorders.

Anecdotal evidence in medicine is often misleadingly defined as evidence based on only one clinician’s personal experience with a treatment or diagnosis in question. If that is the standard that is to be used, clearly many reasons exist to question the validity of inferences drawn from these experiences. Individuals are well known to have various biases that color their observations and the conclusions they draw from them. 

They may have blind spots because of their own emotional conflicts. They may ignore evidence that is contradictory to their point of view. Their observations may be limited by their pet theories about the phenomena in question. 

Perhaps even more importantly, they may be seeing unusual cases that are not representative of more “typical” cases in one way or another – a so-called selection bias.

An obvious case of selection bias was illustrated by a statement I heard made at a conference by a family therapy pioneer, the late Jay Haley. I had always admired Mr. Haley for many of his fascinating and utilitarian ideas and observations. However, in this case he betrayed some ignorance. He stated that he did not believe antidepressant drugs were ever effective because none of the patients referred to him had ever responded to them.

Of course, his being a well-known family therapist who did not believe in medication had a tremendous effect on exactly who would be referred to him. Not everyone does respond to drug treatment. Anyone who had responded to an antidepressant would, in all probability, rarely if ever darken his door. Hence, with his sample, he would be misled into thinking that the medicines were not effective for anybody. This form of bias is very common and can be quite subtle. For example, it can affect one’s beliefs about such matters as racial stereotypes or a determination of how trustworthy members of a city’s police department are.

Do these types of biases invalidate all clinical experience? Hardly. First of all, we have to distinguish between the descriptions of the actual events contained within specific anecdotes, and the conclusions or inferences which are drawn from these events. 

Let us first examine the descriptions of what actually happened. A specific anecdote may be accurately observed and described, or not so accurately. If important details are altered or left out entirely, the anecdote may indeed be worthless. However, the exact same thing can be said about empirical studies.

Important details may not even be known to an observer. Particularly with observations of family behavior within a practitioner’s or researcher’s office, important information is almost always hidden, either by design or unwittingly. In addition to the fact that one does not see the whole picture in any single context, there is also a basic problem inherent in the nature of interactions between intimates.

With verbal behavior, for instance, linguists refer to a quality called ellipsis. What this means is that in conversations among people who have known each other for a while, certain information is not spelled out verbally because the other person already knows it. Strangers such as therapists who are listening in and who have not been privy to these prior experiences may think they know what the family is talking about, but they may in reality be completely clueless.

Let us now consider the separate issue of conclusions that are drawn from anecdotes, as opposed to their description. The questions raised by an accurately-described clinical observation can be quite valid, but the answers inferred from it can be completely wrong. Conclusions based on clinical “anecdotes” exist on a continuum from relatively accurate ones to those that are extremely biased to those that are based on spectacular inferential or logical leaps of faith.

Relatively unbiased clinical conclusions based on anecdotes by mental health professionals have many things in common: 

·   They are based on a sample that one has a reasonable expectation is at least somewhat representative of a larger population.

·    They make use, not just the practitioner’s observations, but of the observations of other professionals whom one knows to be reliable and open minded. These clinicians should also be ones known to take the time with their patients necessary to take a complete history. Widespread clinical experience by competent clinicians is something upon which someone can make a very valid inductive conclusion, and is not just “anecdotal evidence.” In fact, conclusions drawn from this source tend to be more accurate that those drawn from so called “empirical” studies. Many dangerous side effects from drugs that did not show up in the initial drug studies have been discovered in this manner.

  

·     They make use of other informants besides the patient when possible.

·     They take into consideration that people and their family members behave quite differently behind closed doors than they do in public, and therefore if at all possible include observations of patient behavior when patients are unaware that they are being observed.

·    They are based on longitudinal observations. That is, the patients on whom conclusions are based have been seen on multiple occasions over an extended period of time.

·     They are not contradicted by commonly observed examples of behavior in everyday life related to the behavior in question.

·      The person proposing the conclusion acknowledges potential biases, such as a financial stake in a certain drug or allegiance to a specific school of therapy, and acknowledges his or her limitations. What former president of the Society of Clinical Psychology, Gerald C. Davison, calls “ex cathedra statements based upon flimsy and subjective evidence,” a hallmark of some psychotherapy gurus, are always highly suspicious. In fact, charlatans are relatively easy to spot. They have a tendency to sidestep challenges. I will give an example of this shortly.

  

·     The conclusions reached should lead to predictions of patient behavior under certain circumstances that prove to be accurate in a significant number of cases. This is called predictive validity. Of course, human behavior being as unpredictable as it is, at times the predictions will not be completely accurate even if the conclusions are valid, and so this fact must also be taken into account.

·     Conclusions based on anecdotes about treatment efficacy or the reasons for certain observed behavior should consider several alternate possible explanations for the observations. If several explanations are possible, one must make a judgment about which ones are more likely and which are less likely based not on the anecdote alone, but on all sources of data available. These sources include empirical studies, but also include observations from everyday life, as well as material seen in some relatively reliable media such as reputable newspapers.

Now of course stories in the media also do not tell the whole story or may be biased, so one needs to realize again that one can be fooled, and take this into account as well. I used to believe the common myth, for example, that in nature under certain conditions the animals called lemmings would follow each other off a cliff and commit mass suicide.

I was surprised when I learned that this was untrue because I had as a child in 1958 seen a film clip of said mass suicide that was part of a Disney “True Life Adventure” nature movie called White Wilderness. I later learned that, because the Disney crew could not find a real example, they had from behind the scenes driven the group of lemmings off the cliff for the cameras.

On the other hand, many people believe that men have never been to the moon and that films of the moon landings were made in a movie studio using special effects. I must say, I tend to believe that those film clips are real, but few know for certain.

·   If other anecdotes about similar patients and treatments seem to contradict the conclusions based on a given anecdote, an attempt should be made to account for this difference.

As an illustration of the latter point and an example of a the “quick step side step” in scientific presentations. I once heard an expert present new evidence from neuroscience that certain capabilities of which human brains are capable seem to develop only at certain times during early childhood development. This brain development could be adversely affected by a baby’s early social environment. Of course, that is somewhat true.

Like psychoanalysts will, however, the expert went on to conclude that if the adverse early experiences had taken place, the child had no chance of growing up to be normal. I raised my hand and asked about those children who come from horribly adverse backgrounds, are adopted away at an age past the alleged crucial developmental time, and yet still turn out wonderfully. The expert then changed the subject without ever addressing my question.  

If the data doesn't fit your pre-conceived conclusions, just change the subject!

Remember, there are NO empirical, placebo-controlled, double blind controlled studies on whether parachutes reduce the incidence of deaths or injuries after falling out of airplanes. Or that appendectomies are effective in preventing complications and deaths from appendicitis. And yet we all take those things for granted.

Anecdotal Evidence and Inductive Conclusions in Psychiatry: Part I

“Can a valid diagnosis be arrived at if laboratory tests are not available? Of course it can. Total reliance on on lab tests to the exclusion of other factors when seeking validity is not science but scientism” ~ Hannah Decker, Ph.D

In my post of April 1, 2010, Is “Data” the Plural of “Anecdote?” I pointed out that in most hard sciences, stories about a scientist or practitioner’s personal experience can be contrasted with “empirical” data from studies, and that the latter usually has more validity that the former.  

As I also pointed out, however, “…many of the randomized, placebo-controlled, double-blind drug studies can be every bit as biased as the worst anecdotal evidence. We are now living in an age of 'marketing-based medicine,' as opposed to evidence–based medicine" (randomized = the use of chance alone to assign the participants in an experiment or trial to different groups; placebo = inert sugar pill; double-blind = neither the person evaluating the research subject nor the subject knows during the experiment if the subject received the placebo or the active medication).

In psychiatry, the situation is even more complex, since there are almost no objective lab tests and we can't read minds, so we must rely on the experimenter's personal observations, or answers by subjects to psychological tests in which we do not know how honest subjects are being or even if they really understand the questions, or patient self reports. All of these sources of data, while important, are highly subjective and can hardly be considered "empirical" in the same sense as, say, a chemical reaction.

Another big point about anecdotes is that they are not worthless in science. Some of the most important discoveries in all of medicine were based initially on anecdotal evidence. In Part II of this post, I will describe how to determine the validity and generalizability of conclusions generated from a group of “anecdotal” clinical observations. That is, how can we tell if the anecdotes are the typical situation seen with a particular combination of clinical variables and which therefore can lead to a conclusion that will probably apply to most patients with a given disorder? Are the conclusions probably true, or probably false? 

In this post, however, I want to contrast conclusions based on a “clinical anecdote” with inductive reasoning. Remember, the first step in the scientific method is observation.

According to Wikipedia, inductive reasoning (as opposed to deductive reasoning) is reasoning in which the premises seek to supply strong evidence for, but not absolute proof of, the truth of the conclusion. It is usually based on seeing the same phenomena over and over again without any exceptions.

The truth of an inductive argument is supposed to be probable, based upon the evidence given. The same phenomena involving two variables are observed repeatedly, and other variables which correlate with the phenomena are shown to be not necessarily connected it, and a tentative conclusion or assumption is drawn regarding the relationship between the two variables.

Deductive reasoning, whose conclusions are more certain, is the process of reasoning from one or more general statements (premises) to reach a logically certain conclusion. Deductive reasoning links premises with conclusions. If all premises are true, if the terms are clear, and if the rules of deductive logic are followed, then the conclusion reached is necessarily true.

To anyone familiar with formal logic, deductive reasoning takes the form of a syllogism:

All men are mortal

Socrates is a man

Therefore, Socrates in mortal

If the first two statements are true, then the third statement must be true. Deductive reasoning is considered the entire basis of science by some scientifically illiterate individuals. 

The problem is that both premises in the above examples are inductive conclusions! Just because all observed men have turned out to be mortal does not in any way prove that the next man you come across cannot be immortal!

The first thing you should have learned from your Geometry 1 class in high school is that there is absolutely no way to prove that the shortest distance between two points on a flat two-dimensional surface is a straight line. It is just that every time you measure it, it is. You just have to assume it.

I have a pen in my hand which I am holding about three feet off the ground. Every time I have dropped it under controlled conditions, it fell to the ground. I can never prove that the next time I let it go, it will fall. So let’s see. Well, I’ll be darned. It did it again!

In other words, ALL scientific deductions are based on premises which are unprovable inductive conclusions. Therefore, presto change-o, if no inductive conclusions are scientific, then no deductive conclusions are either. Or in other words, science would not really exist.

Syllogistically:

              All scientific conclusions are based on inductive reasoning

              Inductive reasoning never proves anything

      Therefore, no scientific conclusions are proven.

Bullcrap.