Thursday, April 1, 2010

Is “Data” the Plural of “Anecdote?”

The title of this post is an old joke in scientific circles. Most scientists would argue that there is a big difference between scientific data and a collection of anecdotes (stories about a personal experience). For most hard science, this is often somewhat true. However, I am here to argue that, in the behavioral sciences, data is indeed the plural of anecdote.

One charge that biological psychiatrists level at psychotherapists is that there are not enough “empirical” data that “prove” its effectiveness. This is particularly true of the more humanistic and relationship-oriented therapies. Hence, they say, therapy is “unscientific” and should be thrown out because it is not “evidenced-based medicine.”

Of course, many of the randomized, placebo-controlled, double-blind drug studies (randomized = the use of chance alone to assign the participants in an experiment or trial to different groups; placebo = inert sugar pill; double-blind = the person evaluating the research subject does not know if the subject received the placebo or the active medication)can be every bit as biased as the worst anecdotal evidence. We are now living in an age of “marketing-based medicine,” as opposed to evidence –based medicine.

As an aside, it is actually impossible to do a double blind, placebo-controlled study in psychotherapy. That would mean that the therapists in the study would have to not know what treatment they were administering – that is, they would have to be incompetent. Not really a fair test of the treatment! Furthermore, one cannot have a relationship between two people that is completely inert like a sugar pill, so finding a good control treatment is difficult.

And then there is the matter of choice. Unless one completely dismisses the concept of free will, one has to know that no matter what intervention a therapist uses, a patient can choose not to respond to it. Maybe the patient is just feeling contrary that day. Contrast this with some drugs. Can one choose not to go to sleep if highly-potent anesthesia is administered? I think not.

We cannot read minds, and there are precious few laboratory tests by which we can monitor changes in feelings, cognitions, or even behavior. Mental health professionals who do “empirical” experiments regarding the effect on the psyche of drugs or psychotherapy therefore rely on their own observations and on psychological tests.

However, how a given individual chooses to respond to a psychological test on a given day is in all instances an anecdote, as is any observation that the experimenter makes. The result always involves a highly subjective assessment of the feelings, cognitions, or behaviors in question, either by the experimenter or by the subject. It is in no way empirical.

The designers of some psychological tests seem to know this, because they design their tests to include measures of what is termed “test validity.” By this they mean they incorporate into the test ways to screen out the subjects who have exaggerated some aspect of their mental state and others who have minimized it. Subjects can do so either consciously or subconsciously. If done consciously, we speak of those who are “faking good” or “faking bad.”

Validity is obtained by having a high number of test items. Some of the test items are so extreme that even a psychotic patient would not endorse them unless they were trying to look bad. Other items are repeated several times using different wording and sentence structure to see if the subject answers consistently. If subjects are lying, they often will not. The prime example of a test which incorporates this strategy is the MMPI (Minnesota Multiphasic Personality Inventory). It has 567 test items.

The vast majority of “empirical” psychological measures used in clinical trials consist of a much smaller number of items and have no validity scales. If the experimenter instead assesses the mental state of someone else through their own subjective judgment, he or she has all the possible bias inherent in any other clinical anecdote.

Now of course, when psychological tests are employed in clinical trials, if the study is properly randomized and the number of subjects is sufficiently large, one hopes that the number of subjects faking good or bad will be equal in the group of patients receiving active treatment and the group of people receiving a sham treatment. These folks would then cancel one another out, so to speak, leading to a valid result. Unfortunately, there is no way to know for sure whether any two groups are really comparable in this regard, and the number of subjects in clinical trials is often somewhat smaller than necessary for the testers to be even somewhat confident that they are.

Because free choice exists and because judgments about mental states are by necessity subjective, a psych test is just not comparable to a blood calcium level or a bacterial count. Hence, a psych test is an anecdote. As I said in the beginning, in psychiatry and psychology, data is indeed the plural of anecdote.

Subjects who take psychological tests may be faking, but they can also be somewhat uninterested. They may answer test items without really thinking about them very much, or even answer completely randomly. If they are being paid to be a research subject, they know that they will receive remuneration regardless of whether or not they are diligent. Test subjects may also have completely different ideas about what individual test items even mean.

Other motives can come into play. A prime example is performance on IQ tests. Egalitarians are always dismayed when anyone points out that African American students score, on average, more poorly on IQ tests than white students (of course, some black students do far better than the average white student).

The IQ test difference can be easily explained, not by racial differences, but by the fact that African American folks have a very good reason to be unmotivated to do well on IQ tests, and therefore do not take them as seriously as do whites. No means currently exist to accurately assess the strength of motivation test takers have to do well, but such motivation certainly influences scores on tests. Some researchers have found that people will generally do better on tests if they are paid to do so than if they are not.

As many African American commentators have pointed out, not all that long ago blacks who looked smart or who strived for upward social mobility were called “uppity” and were in significant danger of being lynched. The comic Chris Rock tells a joke about a black motorist in the old South who stopped at a stop sign, and was therefore shot by a police officer because he had the ability to read the sign. (This fear has been turned on its head as it has been transmitted from one generation to the next. Doing poorly on intellectual tests is actually lauded in some black communities, where doing well is equated with “acting white”).


  1. Personally, I don't give care about double blind tests and empirical data when it comes to mental health, which is a no man's land of scientifically dubious information. The best information I got to help my son through schizophrenia came from anecdotes. Two come to mind. 1) People who recover from schizophrenia are not usually good patients (doctors have noticed this). 2) People who recover from schizophrenia had family members who didn't overreact to the situation, often behaving as if they didn't realize their relative had a problem.
    These are anecdotes, not science, but then psychiatry isn't really science, so I figure my understanding of what will work is as good, if not better, than the scientific community's.

  2. Rossa Forbes - There is quite a bit of good science in psychiatry, but I agree that a fair percentage of it is utter rubbish.

    I'm not sure what you're defining as a "good patient," but your second observation has in fact been validated by studies. There's a construct called "expressed emotion" in families that includes what you're talking about. If family expressed emotion is high, it can trigger a relapse in patients with schizophrenia. (This does not mean that families "cause" schizophrenia in their children - but they can make it worse).

  3. A good patient is someone who buys into the disease model, accepts their diagnosis and their meds (they have a "chronic disease" after all), and probably dies 25 years sooner because of it, having not managed to live and earn independently. The Expressed Emotion thing - very important, but try to pin someone down on what that really means. It took me a few years to figure out for myself that worrying about my son wasn't helpful. But that's exactly what the diagnosis delivers. Huge worry and stress. However, let's keep in mind that Expressed Emotion is also practiced by the medical profession. EE can be telling someone they have a disease, that they are chronic, and giving someone medications is also EE. The "patient" thinks, I must be sick, I must be helpless, everybody else is so sad when they look at me. Mine is a layperson's view, obviously. I am simply the mother, but I think I get it. By the way, I appreciate the premise of your blog. It's about time that people started linking mental illness with the family environment. It's a heck of lot more hopeful than telling someone their brain is damaged. People can work on improving the environment and forgiving the trauma, but where can you go with a damaged brain?

  4. There's an excellent example of what being a good patient means, posted at The Trouble with Spikol.

    "You have been in the hospital multiple times, in a couple of partial hospitalization programs, and have spent years in sheltered workshops and day programs. You’ve received the Prophecy of Doom, “Too sick for too long to get any better.” You’ve heard plenty of statements beginning with “You can’t, You won’t, and You will never.” You’ve been told endlessly that something is intrinsically (genetically) wrong with you and the only thing that will truly save you is a medication yet to be discovered. You’ve also been told that the most important thing you can do is get on SSI or SSDI in light of the prolonged and persistent nature of your illness."

  5. Ms. Forbes - my heart goes out to the parents of people with schizophrenia. I agree with you that constantly giving the message that "You can't" and "You're impaired" can have devastating effects on anyone.

    I do believe, however, that schizophrenia is a true disease of the brain, unlike much of what is in the DSM.

    Severity levels can vary all over the map; some people with the disorder do not even come to medical attention because they are quite functional, and are just considered odd or eccentric.

    In traditional cultures in which extended families live together, people with the disorder tend to be much less impaired because the family treats them as one of their own and looks out for them.

    On the other hand, many severe schizophrenics have demonstrated atrophy of the brain as they get older, which is of a highly significant magnitude, irrespective of whether or not they have received anti-psychotic medication. In cases like this, sorry to say, the individual really is severely impaired.

  6. Dr. Allen: You refer to brain atrophy shown in people both on and off neuroleptics, and conclude that it must be caused by a disease called "schizophrenia". Recent research has shown childhood trauma to change genes as well as brain structure (McGill University). Also we know by now that our thinking influences neuronal pathways in the brain. All in all, we know today that the brain is a lot more plastic than earlier assumed. On the other hand there's strong evidence that a remarkable number of people labelled with "schizophrenia" are victims of physical/sexual child abuse (Hammersley, Read, 2007). While we, at least as far as I know, don't have newer studies that look at psychological/emotional abuse alone, which may account for the remaining cases (in as far as we're talking "real" "schizophrenia", and not "schizophrenia"-like symptoms caused by somatic conditions only). All we have in regard to the latter is Laing and Bateson, whose theories usually are dismissed as hopelessly outdated and worthless.

    I know quite a few people labelled with "schizophrenia" who are fully recovered. Some of them it took decades to get there, so you would imagine that these people have suffered some sort of brain damage from their recurring crises. Nevertheless, these are some of the brightest people I know *). No sign of brain atrophy whatsoever. But yes, I've also seen people who went in and out of hospital all of their lives, and never "made it". Some of them were never "compliant", so, if the drugs did any damage to their brains it's certainly insignificant. Still, their thinking is clearly impaired. The question is, whether it is the disease that has caused the impairment, whether it is recurring "psychosis" that has "worn out" their brains, so to speak, or whether it is that their thinking got stuck in a certain pattern which in its turn is reflected by their brain structure.

    Personally, I have great difficulty imagining anyoneto be able to reason their way out of a biological brain disease, like, let's say Alzheimer's. Nevertheless, this is exactly what happens, time and again, in the case of "schizophrenia". So I wonder whether the brain "abnormalities" that can be observed in people labelled with "schizophrenia" truly are a symptom of a biological disease, or whether they maybe are reflecting certain thought patterns developed as a defence mechanism for the individual to survive severe trauma/stress, i.e. whether they are symptoms of severe psychological/emotional trauma/stress.

    *) among these Joanne Greenberg, Catherine Penney and Dorothea Buck, who at the age of 93 shows no sign of neither brain atrophy nor dementia nor anything along those lines.

  7. Marian - My blog posting of March 14 dealt with the issue of neural plasticity. I decried the fact that, for a lot of psychiatric diagnoses, differences on fMRI scans are immediately label as "abnormalities" when they could be in fact be adaptations or conditioned responses. We also know, as you pointed out, that trauma can lead to major changes in brain functioning.

    With schizophrenia, however, I do not think that the evidence that the condition is due to to trauma or abuse is particularly strong. There is a lot of evidence now being developed that there are marked changes in the neural networks of the frontal lobes in schizophenia that are well outside the bounds of normal neural plasticity.

    Admittedly, we still have not pinpointed the exact nature of the brain pathology in schizophrenia, nor have we found out what causes it. In the past, patients with true bipolar were often misdiagnosed as schizophrenic, so some of the people who "got better" may have been misdiagnosed in the first place.

    After dealing for over 30 years with what I consider to be the extreme case of normal reactions to abnormal situations - borderline personality disorder - and with schizophrenia, I remain convinced that schizophrenia is a true brain disease. Sorry.

  8. Dr Allen: Thanks for your answer. I have one last question: Let's say, the evidence of "schizophrenia" to be a biological brain disease is about as strong as the evidence that points to trauma as the cause (and, in the light of the scientific facts we have, I don't think it realistic to say the evidence in favor of the bio-model is stronger than that in favor of the trauma model). Now, you have the choice: either you tell someone you assess as suffering from "schizophrenia" that they have a brain disease, and the inevitable result will be that the person feels rather helpless in regard to overcoming this disease (since the drugs don't cure anything, but only, and at best, address symptoms), and consequently the person is unlikely to make an effort at anything else but symptom management and, well, learning how to live with this disease. Or you tell them that, although nothing is proven, at least there is a chance that what they experience is a kind of PTSD, rather than a brain disease, and that there is a possibility for them to achieve full recovery if they manage to decode the meaning of their symptoms in context with their life story, which may have the person engage in what I from an admittedly somewhat "biased" point of view will call a search for their own truth. Which choice is in your opinion the more ethical one?

  9. Marian: Sorry again, but I do not agree with the premise of your question. I have not seen any replicated, convincing evidence that schizophrenia is a form of PTSD, either in the literature or clinically. Many patients with schizophrenia have a mild form of the illness and function quite well, while others will never function well. I always try to get a patient to function as best as he or she can. In my own experience, however, they get worse if you try to decode the meaning of their symptoms in therapy.

  10. So, this then means that, although there's no evidence to prove the hypothesis of "schizophrenia" to be a biological brain disease true, although, according to your previous comment, psychiatrists tend to misdiagnoze this disease in about 75 - 85% (the recovery rate of non-medical alternatives like Soteria, the Finnish Open Dialog, etc.. Have you ever heard of these alternatives? Or the Hearing Voices Network, for that sake? Apropos of people getting worse if they try to decode... Check it out, for your clients sake!) of their clientele, it is perfectly all right for psychiatrists to tell people rather hopeless messages about "bad" genes and an imbalanced brain chemistry there's no cure for, to concentrate almost exclusively on drug "treatment", with quite a lot of side effects, some of them fatal, and not to seek to eliminate the possibility of either the hypothesis' incorrectness or simply a misdiagnosis first by at least giving a non-medical approach a chance. Hm. Sounds to me rather backwards, I must confess (and, in fact, in conflict with Hippocrates).

    Thank you for your time and consideration - from someone who achieved full recovery from "schizophrenia" thanks to non-medical help.

  11. Just for other readers, claims of 75-85% recovery rates for schizophrenia from non-traditional medicine are pure snake oil. I hate to see people given false help through outrageous claims. True psychosis is never a normal variant of anything. And I never tell people that they have a chemical imbalance, because that is truly a meaningless phrase.

  12. "I remain convinced that schizophrenia is a true brain disease"
    It is your belief, a doctors belief that makes reality exist when written down in a medical report.
    If schizophrenia is a "true brain disease" there should be an empirical standard lab test to diagnose, not just an opinion.
    If this physical test existed, schizophrenia could be claimed to be a neurological disease to be treated by a neurologist not a psychiatrist.
    But then a schizophrenic can't complain about his/her treatment, or he/she only gets more "helpful" treatment.
    Regarding recovery,
    Harding's study in The American Journal of Psychiatry (Vol. 144, No. 6, p. 718-735)

    "People say 'Oh, you were misdiagnosed," says Bassman. "Otherwise, you couldn't be where you are now.' I mean, that's an impossible circular argument."

  13. Mark:

    You say: "If schizophrenia is a "true brain disease" there should be an empirical standard lab test to diagnose, not just an opinion." Unfortunately, our knowledge of the brain, especially of neural networks which are the basis of behavior, is still in its infancy. We're getting there, however. For now, we only have descriptive criteria in the DSM for diagnosing most disorders.

    However, we do have indirect evidence, epidemiology, and long-term follow up of representative samples. Some of this research is good and some of it sucks. And I personally think schizophrenia should be treated by neurologists, but they won't do it!

    Regarding the "recovery" model, I think mild to moderately ill patients with schizophrenia would benefit greatly from this. It's a crime they are not getting it, but most of them can't afford it, and politicians have been decimating the public mental health system for decades.

    Re circular arguments: how right you are! Sort of like what the Angelina Jolie character in the movie "Changling" was put through.

  14. Well, if neurologists won't do it, doesn't that say something about the brain disease argument? But neurologists aren't the only ones who won't do it. How many psychiatrists want to really treat patients with schizophrenia? It's much easier to give them a drug and tell them that they have a brain disease. The psychiatrist is then off the hook. That, too, is circular.

  15. Ms. Forbes,

    I think the neurologists' lack of interest is due to discomfort in dealing with anyone with mental problems. You're right, though: a lot of psychiatrists do not want to treat patients with schizophrenia either.

    We make a lot more money just writing scripts rather than doing psychotherapy, let alone rehabilitation, and the latter is much more difficult.

    I myself no longer see a lot of patients with schizophrenia, though I used to, because I subspecialize in something called borderline personality disorder. Most psychiatrists hate dealing with them even more than they hate dealing with patients with schizophrenia.

    Many psychiatrists discount the painfully obvious and severe family problems seen in patients carrying the BPD diagnosis, and call them "Bipolar II" or some other b.s. term to justify just sedating them with drugs. Again, though, psychosis is a very different beast.