Is “Data” the Plural of “Anecdote?”

The title of this post is an old joke in scientific circles. Most scientists would argue that there is a big difference between scientific data and a collection of anecdotes (stories about a personal experience). For most hard science, this is often somewhat true. However, I am here to argue that, in the behavioral sciences, data is indeed the plural of anecdote.

One charge that biological psychiatrists level at psychotherapists is that there are not enough “empirical” data that “prove” its effectiveness. This is particularly true of the more humanistic and relationship-oriented therapies. Hence, they say, therapy is “unscientific” and should be thrown out because it is not “evidenced-based medicine.”

Of course, many of the randomized, placebo-controlled, double-blind drug studies (randomized = the use of chance alone to assign the participants in an experiment or trial to different groups; placebo = inert sugar pill; double-blind = the person evaluating the research subject does not know if the subject received the placebo or the active medication)can be every bit as biased as the worst anecdotal evidence. We are now living in an age of “marketing-based medicine,” as opposed to evidence –based medicine.

As an aside, it is actually impossible to do a double blind, placebo-controlled study in psychotherapy. That would mean that the therapists in the study would have to not know what treatment they were administering – that is, they would have to be incompetent. Not really a fair test of the treatment! Furthermore, one cannot have a relationship between two people that is completely inert like a sugar pill, so finding a good control treatment is difficult.

And then there is the matter of choice. Unless one completely dismisses the concept of free will, one has to know that no matter what intervention a therapist uses, a patient can choose not to respond to it. Maybe the patient is just feeling contrary that day. Contrast this with some drugs. Can one choose not to go to sleep if highly-potent anesthesia is administered? I think not.

We cannot read minds, and there are precious few laboratory tests by which we can monitor changes in feelings, cognitions, or even behavior. Mental health professionals who do “empirical” experiments regarding the effect on the psyche of drugs or psychotherapy therefore rely on their own observations and on psychological tests.

However, how a given individual chooses to respond to a psychological test on a given day is in all instances an anecdote, as is any observation that the experimenter makes. The result always involves a highly subjective assessment of the feelings, cognitions, or behaviors in question, either by the experimenter or by the subject. It is in no way empirical.

The designers of some psychological tests seem to know this, because they design their tests to include measures of what is termed “test validity.” By this they mean they incorporate into the test ways to screen out the subjects who have exaggerated some aspect of their mental state and others who have minimized it. Subjects can do so either consciously or subconsciously. If done consciously, we speak of those who are “faking good” or “faking bad.”

Validity is obtained by having a high number of test items. Some of the test items are so extreme that even a psychotic patient would not endorse them unless they were trying to look bad. Other items are repeated several times using different wording and sentence structure to see if the subject answers consistently. If subjects are lying, they often will not. The prime example of a test which incorporates this strategy is the MMPI (Minnesota Multiphasic Personality Inventory). It has 567 test items.

The vast majority of “empirical” psychological measures used in clinical trials consist of a much smaller number of items and have no validity scales. If the experimenter instead assesses the mental state of someone else through their own subjective judgment, he or she has all the possible bias inherent in any other clinical anecdote.

Now of course, when psychological tests are employed in clinical trials, if the study is properly randomized and the number of subjects is sufficiently large, one hopes that the number of subjects faking good or bad will be equal in the group of patients receiving active treatment and the group of people receiving a sham treatment. These folks would then cancel one another out, so to speak, leading to a valid result. Unfortunately, there is no way to know for sure whether any two groups are really comparable in this regard, and the number of subjects in clinical trials is often somewhat smaller than necessary for the testers to be even somewhat confident that they are.

Because free choice exists and because judgments about mental states are by necessity subjective, a psych test is just not comparable to a blood calcium level or a bacterial count. Hence, a psych test is an anecdote. As I said in the beginning, in psychiatry and psychology, data is indeed the plural of anecdote.

Subjects who take psychological tests may be faking, but they can also be somewhat uninterested. They may answer test items without really thinking about them very much, or even answer completely randomly. If they are being paid to be a research subject, they know that they will receive remuneration regardless of whether or not they are diligent. Test subjects may also have completely different ideas about what individual test items even mean.

Other motives can come into play. A prime example is performance on IQ tests. Egalitarians are always dismayed when anyone points out that African American students score, on average, more poorly on IQ tests than white students (of course, some black students do far better than the average white student).

The IQ test difference can be easily explained, not by racial differences, but by the fact that African American folks have a very good reason to be unmotivated to do well on IQ tests, and therefore do not take them as seriously as do whites. No means currently exist to accurately assess the strength of motivation test takers have to do well, but such motivation certainly influences scores on tests. Some researchers have found that people will generally do better on tests if they are paid to do so than if they are not.

As many African American commentators have pointed out, not all that long ago blacks who looked smart or who strived for upward social mobility were called “uppity” and were in significant danger of being lynched. The comic Chris Rock tells a joke about a black motorist in the old South who stopped at a stop sign, and was therefore shot by a police officer because he had the ability to read the sign. (This fear has been turned on its head as it has been transmitted from one generation to the next. Doing poorly on intellectual tests is actually lauded in some black communities, where doing well is equated with “acting white”).