Tuesday, 14 April 2015

A rant: "puckish", "provocative" science journalist fails to understand basic facts about scientific expertise

Time for a rant.

Last week I read a Scientific American blog post by John Horgan, and it left me fuming. It was simplistic, sloppy, ultimately misleading, and he should know better.

His post was on scientific expertise. Some people argue that we should defer to scientific experts on complex scientific questions. Sounds reasonable. Horgan instead exhorts us to be sceptical of the scientists, with the seductive cliche, "Question everything!" Which also sounds reasonable.

Doesn't that leave us with a contradiction? Yes, it does. A vanishingly small number of people have sufficient background and expertise to make a reasonable judgement on a large number of important issues, yet they may themselves be mistaken, or self-deluding, or downright lying. With our daily lives dominated by science, technology and medicine, and our planet apparently at serious risk from climate changes that only a handful of scientists are able to reliably identify, this is a severe problem indeed.


Yet over the course of 1100 words, Horgan shows no sign that he is even aware of that contradiction.

Instead he argues that scientific bloopers can be identified by everyone -- you, me, John Horgan and even, just to ramp up the provocative click-bait appeal of his post, Jenny McCarthy. For those blissfully unaware, Jenny McCarthy is a celebrity who has energetically promoted the view that child vaccinations cause autism, on the irrefutable evidence that her son (a) has autism, and (b) was vaccinated. Legions of scientists have responded that there is not a shred of credible evidence for this claim but, well, you know, we should be sceptical of scientists.

The inspiration for Horgan's post is in part an article by Chris Mooney, about a recent book by Harry Collins, "Are we all scientific experts now?" Mooney's article was written almost a year ago, and in the intervening months I assume Horgan has taken the time to read Collins's book; it is only 132 pages long in a fun orange paperback. If so, apart from a feeble attempt to appear contrarian and "start a debate", there is no excuse for the slipshod piece he wrote.

The fist-thumping conclusion of Horgan's post is the following rousing call-to-arms to the citizen sceptics of science: "You have to do the hard work of digging into it and weighing its pros and cons. And anybody can do that, including me, Mooney and even Jenny McCarthy."

Horgan does not appear to have "dug" very far into Collins's book. The most important point that Collins made, and which he made extremely clearly and convincingly, was that Horgan's view is a myth. It is absolutely, utterly untrue. You can pore over the scientific literature all you like, but unless you have the experience of practitioners in the field -- how to spot the weak evidence, the discredited references, the sloppy statistics -- you cannot draw reliable conclusions.

You may think you can. The human mind is exceptional at making confident decisions based on scant evidence. But you cannot.

This is a trap that Horgan falls into headlong. He makes a laughable reference to last year's announcement of results from the BICEP2 experiment. He gleefully reports that none other than Stephen Hawking hailed them as "confirmation of inflation", while he, little old John Horgan, who has "long-standing doubts about inflation", wasn't convinced. And it was Horgan who turned out to be right! The measurements were ultimately flawed.

That's right, folks. John Horgan knows a bit about science, and he's "dug into it" and "weighed the pros and cons", and he's concluded that the notion of cosmological inflation is wrong. And guess what? You can do it, too -- and so can Jenny McCarthy! It's easy!

What's easy is unlimited self-delusion. A number of expert cosmologists are sceptical of inflation, and Horgan has acquired his scepticism from them. One is Paul Steinhardt, whose views you can read in an interview with none other than John Horgan.

Horgan is doing his job as a science journalist by reporting the scepticism of other scientists, and I applaud that he publicises those doubts instead of just reprinting a press release, but he is failing utterly if he wants to imply that the "outsider" can "see through" a scientist's exaggerated claims.

It is not that Horgan has no expertise at all. He very likely has (in some fields) what Collins calls "interactional expertise". He's spent enough time around scientists and discussed their ideas with them enough, that he can in some cases make surprisingly accurate judgements about technical issues. But not always, and not reliably. Any claim he makes should be checked back with the experts. If he is a good journalist, then he knows how to do that -- but he still has to make that check. He's mistaken if he's deluded himself into thinking that he can make the judgement call all by himself, and he's dangerously mistaken if he wants to argue that anyone else, complete non-experts with no experience of science, can do so as well.

Horgan argues that science journalists play an important role in checking up on scientists. I agree, but it is a low-level check. They can check whether there is sufficient debate within the community of experts. They can make sure that that debate is known to the public. They can make sure that it's the real experts that we're hearing from. They can also help to uncover cases where the scientists aren't the only experts; Collins provides several examples of this in his book. They can also do what I assume good journalists are best suited for: checking facts, confirming sources, uncovering lies and deception, following the money.

That is what they should do, but often they do not. In 1998 Andrew Wakefield published a study claiming a link between the MMR vaccine and autism, and the media leaped on it. Vaccinations were brought into serious disrepute. This had consequences. Fifteen years later there was a measles outbreak in Swansea, where vaccination levels had dropped from 94% in 1995 to 67.5% in 2003. It turned out that Wakefield's study was fraudulent. He wanted to promote an alternative vaccine that he had a financial stake in. As Collins pointed out in his book, this is precisely the sort of story that the journalist are most skilled at uncovering. Yet they didn't. They instead promoted angry parents to the role of experts, and broadcast their indignation. The regular journalists should have deferred to science journalists, who should in turn have known which scientific experts to turn to for reliable information. In fact, Collins notes that the statistical sample in Wakefield's study was so small that any scientist with minimal statistical training should have been able to identify a problem.

Horgan stated that the purpose of his post was to start a debate. Perhaps. I'm not sure that making ignorant statements for others to correct is a constructive way to "start a debate".

To make it worse, he posted a follow-up piece later last week. This time he referred to Philip Tetlock's book on "Expert Political Judgement", which demonstrates that experts are only correct marginally more often than non-experts.

Talk about digging yourself into a hole. Tetlock's book is about political judgement [1]. In particular, it's about making predictions. Making predictions about political or economic events is in an entirely different category to scientific predictions, let alone an expert understanding of established scientific results. A good scientific journalist should know that. In fact, anyone with a half-decent education should know that.

Last year I wrote a series of blog posts on the complex issues of scepticism and expertise in science, using the example of climate change (summarized at the end of this post). Perhaps John Horgan should read them. I'm afraid that as an example of a lay citizen sceptic I failed to think of such a well-known lightning rod of a figure as Jenny McCarthy, and instead used the critic Clive James, who has often publicly bashed climate change. My reason was not that I thought him ignorant or foolish, but quite the reverse. As a sharp intellect who knows far more than me about many topics, almost certainly including climate change, I thought it interesting to ask why he felt he could second-guess thousands of expert scientists.

I took pains to make it clear that I have immense respect for James, and (like Horgan) I respect a sceptic's right to air their views, and I consider his views honest and the result of serious consideration. (I also accept that the same cannot always be said for those who agree with the climate scientists.) I tried to explain that the best sceptics of science are other scientists. I pointed out the distinction between scientific and economic/political/other experts. And I concluded by dispensing with Horgan's mythical belief that anyone sufficiently sharp of mind and stout of heart can uncover scientific truth.

Those posts arose largely from my reflections on reading Collins's book. As I said before, for me the biggest lesson from that book was that a non-expert cannot judge the validity of scientific research, even if they read and understand the primary literature. It takes more than that.

I was left with one serious issue with Collins's view of expertise: how do we reliably identify the experts in a given field? When I put that question to Collins, I was not especially encouraged by the answer. So we are still left with questions, and we are left with complicated issues to resolve.

It would be nice if science journalists like John Horgan reported on those issues. But I guess it is easier to trumpet hollow cliches and make provocative statements to spark a "debate".


Postscript:
For completeness, here is a list of my posts on climate change and scientific expertise:
1. A content-free introduction.
2. A gushing declaration of my admiration for Clive James.
3. A summary of James's criticism of climate change.
4. A defence of scepticism, even if it appears irresponsible.
5. Bias on both sides of the climate-change arguments (and "The Hannam Effect").
6. Why scientists are the best sceptics of science (with reference to Climategate... and BICEP2).
7. The distinction between the usual (useless) experts, and scientific experts.
8. Summary and conclusion: listening to the experts is, unfortunately, the best we can do.

And finally:
- A discussion of Collins's book, and the question of how we identify the experts.
- My enlightening (but inconclusive) encounter with Harry Collins.



Note.
1. I have not read Tetlock's book. My understanding of it, such as it is, comes from Tim Harford's "Adapt".

7 comments:

  1. You are overstating what Horgan is saying - he is not saying "scientific bloopers can be identified by everyone" nor is he saying "anyone sufficiently sharp of mind and stout of heart can uncover scientific truth".

    He is saying that everyone has the right to challenge "experts", and that journalists should question authority and "experts". That "experts" often get things wrong (Iraq, BICEP, financial crisis), and that the most credible "experts" are those who warn against reliance on "experts" (Tetlock, Feynman).

    ReplyDelete
    Replies
    1. You are right that the lines you've quoted from my post do not appear in his. But the quotation that I have taken from him ("You have to do the hard work...") explicitly states that we are all capable of understanding scientific issues sufficiently well to draw valid conclusions, if we work hard at it. This is much more than saying that we have a right to challenge experts, which I completely agree with. I even agree that Jenny McCarthy had the right to endanger lives by spreading lies, although I wish she hadn't. (See post No. 2 above.)

      Experts get things wrong, but the point is that often we are *not* able to tell. That's a big problem, and pretending it doesn't exist is not helpful.

      He does not make the case at all that "the most credible experts are those who warn against reliance...", and that view doesn't hold up anyway. In science, the most credible experts are among those who best know the topic, and *all* scientists I know are sceptical and prefer not to rely on anyone. (See post No. 6.)

      BICEP2 sticks out like a sore thumb in your three examples. Inflation is still very much alive, and we are still waiting for evidence for or against. Predicting whether it will turn out to be right is the province of the "bad" kind of expert. (See post No. 7.) A scientific question (whether MMR causes autism) is entirely different, and the best answer you'll get will be from an expert.

      I'm not sure why bring up Feynman, but he also perpetuated the myth that "A clever, non-nonsense, honest fellow can always uncover the truth". He might deserve a blog post of his own...

      Delete
  2. Keep writing Mark! We love you!

    ReplyDelete
  3. Seriously Mark,
    Are you OK? Don't keep your fans on edge like that!

    ReplyDelete
    Replies
    1. Sorry, I got distracted, and lazy, and various other things. More blog posts are coming soon...

      Delete
    2. That's a relief!

      Delete
    3. It's very touching that there are people out there who I've never met, who are checking up to see if I'm Ok. Thanks!

      It reminds me of when I used to check in, at least daily, on the Radiohead fan site, followmearound.com. I think it was maintained by someone in Toronto. Once there was no news for over a week, and it made me think that it would be cool to write a story about a guy who obsessively reads a band's fan website, and then when it's down for a while goes on an epic road trip to track down the person who runs the site, and check whether they're Ok.

      The site is no longer maintained, and I don't really care. So you can see what a feat of imagination it would have required.

      I vaguely remember that the band played the song "Follow me around" in Toronto in, I guess, the early 2000s, purely because of that website. And now the site is defunct. I take that as a strong indictment on those recent albums. But maybe they just moved on with their life.

      PS: I've just checked. You can still go to the site, but from a quick click around, it doesn't look to have been updated since around 2006.

      PPS: That Toronto performance of Follow Me Around was from 2000. It's here.)

      Delete

[Note: comments do not seem to work from Facebook.]