Hidden Assumptions in Conclusions about Research Data in Psychology

In evaluating the conclusions of the authors from the results of any “empirical” study, two important questions one should ask oneself are: What assumptions are the authors making, and are those assumptions justified?

In today’s world, particularly in studies of the psychology of human beings, study authors often make assumptions which they do not bother to spell out in their reports, so their conclusions may seem logical. However, if they were to spell out those assumptions, everyone would immediately recognize them as completely and obviously preposterous.

In his book How Not to Be Wrong, Jordan Ellenberg mentions an illustrative anecdote about the importance of hidden assumptions that involved a group of government scientists from World War II. Their task was to determine where on warplanes to best place armor, since too much armor weighed the planes down and decreased their maneuverability. The scientists closely examined the airplanes that were returning home safely. 

At first, they inspected the planes in order to determine where the bullet holes mostly were. They figured that the parts of the plane that were hit the most often should be where the most armor should be placed, since (as the thinking went) those places must be where being hit was the most likely. Strangely, the engine seemed to be the part of the planes most frequently spared from bullet holes.

Wrong strategy. They should have been looking at where the bullet holes mostly were not. The planes hit in those places were the ones that were not making it home safely! If the engine got hit, the plane crashed. If a plane had been hit in the places they were looking at, it was apparently much less likely to crash, since it made it home. The armor should therefore be put around the engine. But only one scientist in the group made this seemingly obvious point before everyone else saw how it obvious it was! And these were some of the best minds in the field.

So let me take a study that I recently found during my weekly literature search on borderline personality disorder (BPD) on the medical database Ovid.  I’m just going to discuss the abstract, since that is all most doctors are ever going to read, if they read anything at all. (The authors did not spell out their assumptions any better in the body of the paper, but the odds are no one is going to actually read that anyway).  

Here is the abstract:

Authors:  Nicol K.  Pope M.  Sprengelmeyer R.  Young AW.  Hall J.
Title: Social judgment in borderline personality disorder.
Source: PLoS ONE [Electronic Resource].  8(11):e73440, 2013.
Abstract:
  BACKGROUND:  Those with a diagnosis of BPD often display difficulties with  social  interaction and struggle to form and maintain interpersonal relationships.  Here we investigated the ability of participants with BPD to make social inferences from faces.
  METHOD: 20 participants with BPD and 21 healthy controls were shown a series of faces and asked to judge these according to one of six characteristics (age, distinctiveness, attractiveness, intelligence, approachability, trustworthiness). The number and direction of errors made (compared to population norms) were recorded for analysis.
  RESULTS: Participants with a diagnosis of BPD displayed significant impairments in making judgments from faces. In particular, the BPD Group judged faces as less approachable and less trustworthy than controls. Furthermore, within the BPD Group there was a correlation between scores on the Childhood Trauma Questionnaire (CTQ) and bias towards judging faces as unapproachable.
  CONCLUSION: Individuals with a diagnosis of BPD have difficulty making
  appropriate social judgments about others from their faces. Judging more faces as unapproachable and untrustworthy indicates that this group may have a heightened sensitivity to perceiving potential threat, and this should be considered in clinical management and treatment.

Now, many other studies have shown that patients with BPD are actually better at reading faces than controls, so in trying to draw any conclusions of course we have to figure out why different studies get different results. But ignoring that for the time being, let us just look at this one study abstract in isolation.

The conclusions was that the subjects with BPD had “significant impairments” and "difficulties" in making judgment. To be fair, the authors also used the words "heightened sensitivity to perceiving potential threat," which is actually a far more accurate description of their findings. But it is the words "impairments'" and "difficulties" that will be the ones that will jump out at most readers. And in the body of the paper, those terms are if fact more in line with the conclusions discussed by the authors than the phrase "heightened sensitivity."

In using these nouns, the authors are making some rather strange assumptions. A clue that they are doing that is also in the abstract: It mentions that the patients with BPD were far more traumatized as children than the controls.

That being the case, it is highly likely that the people in the social environment of the BPD subjects were far more likely to have hostile intentions than those of the controls. In such an environment, you’d have to be an idiot not to generally have a high index of suspicion when evaluating the faces of people. 

The assumption the authors seem to be making is that somehow the BPD subjects were just naturally worse at reading faces, rather than they were justifiably more suspicious of other people - the latter conclusion being one that would be predicted by error management theory.

So the assumptions they seem to making that need to be questioned are:

1        1. We can just ignore the social context of research subjects in making these sorts of judgments about people’s abilities.

          2. It is true that people rarely if ever use their brains to develop strategies for dealing with other people that have little to do with their innate abilities.

Clearly, those are really stupid assumptions.