I often write about the logical errors in the design of
medical studies of personality disorders, psychotherapy outcomes, and labeling
behavioral problems as some sort of brain disease because of some average
difference on an MRI scan. I discussed how epidemiological studies that try to
correlate two things are inherently misleading because of the inability to
control for other factors which may effect the outcome.
In the book Wrong: Why
Experts Keep Failing Us and How to Know When Not to Trust Them, David H.
Freedman does an amazing job of showing how experts in a lot of other fields,
especially when they interact with public media, draw conclusions which are
completely wrong. It’s not that we have made no progress in such things as
medical treatments obviously, but the process of drawing valid conclusions is
more problematic than people think. Because of contradictory information about,
say coffee being good or bad for certain conditions, this may have led to our
current craziness regarding obviously effective treatments like vaccines.
Freedman goes into detail about the mis-informational follies
of government officials, industry insiders, financial rating agencies, business
gurus, stockbrokers, and real estate agents. Experts often pander to certain
segments of the public, and may therefore be biased to gain prestige in professional groups,
may also be corrupt, and may even fake data to gain publication in a prestigious
journal. Scientists may measure what doesn’t matter or toss out inconvenient
data. Positive studies need to be replicated but often aren’t, and if they are
and the new study doesn’t do so, the second study is unlikely to be published
because journals are prejudiced against negative studies.
Business advice is often based on the latest fad. Because of
initial enthusiasm, a new philosophy of management may seem to bring added
success, but it often does not seem to last and is then replaced by some new fad. I’ve noticed something similar with teachers –
new fads on techniques for teaching kids how to read will be popular but then
eventually be replaced by some other new technique.
When it comes to separating the wheat from the chafe, so to
speak, I think the author is at times more pessimistic than I think he needs to
be. After all, despite all this, scientists have made real progress with
demonstrable effects. He does offer some strategies in the last chapter: with
new info, make sure it doesn’t trip any alarms; appreciate the importance of
negative findings; see if it includes sensible qualifying statements; see if it’s
candid about refutational evidence; see that it includes candid discussions providing context and perspective.
My own favorite question when it comes to whether I think
the conclusions of a study is valid or not is, “If I believed this, what else would I have to believe?" For example,
most therapists think that people with borderline traits “split” ( ‘They see
people as all good or all bad) and have deficiencies in mentalizing’ (reading
other people’s thinking or intensions). Yet they aIso think they are master
manipulators. How on earth can those things all be true?
I am automatically suspicious of anything that in an
epidemiological study purports to show cause and effect relationships between two
things. Which is the cause and which is the effect? And how valid is a
connection when results could be due to other variables that are impossible to
control? And how about logical
fallacies? We used to think marijuana use was a major risk factor for later
heroin abuse because many addicts started with pot. But almost all of them had
also used coca-cola (a “post coke ergo propter coke fallacy” :)
