Thursday, 28 June 2018

The Danish Paper: One Year Later

Pre-requisite reading: The Irresistable Allure of Controversy.

Here is where I left the story one year ago.

The “Danish Paper” had claimed to find previously unnoticed correlations in the background noise in the first LIGO gravitational-wave observation, GW150914, and many readers’ interpretation of this interpretation of the data was that maybe the first detection was not as slam-dunk as it first appeared. Sabine Hossenfelder wrote an online post in Forbes that propelled the paper out of its just-another-arXiv-entry obscurity, with the innocent title, “Was it all just noise?” A subset of the LIGO collaboration bravely sacrificed time that would otherwise have been wasted doing actual scientific research, and leaped to PR firefighting duty, and their response, a careful dismantling of the Danish Paper’s claims, was released to the world through that true apotheosis of science communication, Sean Carroll’s blog. That was the point at which I decided to join the fun.

Why did I bother? Was it a cheap attempt at some publicity of my own? Certainly it was. But there were also points that I felt were not being made in this faux-civilised tit-for-tat. Several people called this public airing of doubts “science at it should be” — Sean Carroll on Twitter, Peter Coles in his blog, and many others — and this sloppy thinking needed cleaning up. I was also astounded that Hossenfelder had provided the paper such visibility, given that it was not only wrong, but so obviously wrong. In the rant inspired by all that, I argued against speculating on results like this, or even bothering to report them, until the dust has settled and the real answer is clear (or a real crisis or controversy remains).

I then decided to follow my own advice, and wait a year and then return with the (hopefully) clear final story.

So here we are. One year has passed, and the paper is published. It is time to revisit the story and my reaction or, as we sensitive twenty-first-century fellows like to say, to reflect.

But first: what happened next?

A few people expressed surprise that the spectacle of science being conducted in public had not transported me directly to intellectual Nirvana. (Notably Peter Coles again, and Jason Wright here and again here, and later Gizmodo here. Yeah, Gizmodo. Cool.) I tried to argue back that this was all spectacle and no science — “This was not science communication. This was just fly swatting”, was in one tweet I wrote. (Ok, I did not keep entirely quiet…) Even if this had been a respectful exchange between experts on an authentic controversy, I found the idea of it being conducted in public simply grotesque. Scientists are not trained to do their work on stage, and it is hard to think of any reason why they should. Science is already as public as it needs to be: results are published, and, if science journalists and “communicators” do their job well, significant results are discussed more widely. Perhaps that is a big “if”.

There was also some response from the open-data disciples, who noted that not all of the LIGO data used to analyse the first detection were publicly available: taxpayers are paying for this science, and so the data must be there for all to use to check the results, and we should be profoundly grateful that someone is checking them. Indeed, we have a duty to show them how to use the data, so that it is as easy as possible for them to make those crucial checks.

This was just too many layers of manure to shovel through, and I retired from the field.

If such open-data blasphemy horrifies you, let me try to explain. This strident cantering around on a moral high horse belies complexities. Before we even get started, remember that the data can be open, but the detector cannot, so any verification of the results is partial at best. It’s like the approach of Christmas: no amount of wishing and hoping will save you from the reality of waiting, in this case for more, and more sensitive, detectors. Restricting ourselves just to the data, repeating any analysis requires specialist expertise. Demanding that scientists bring anyone who is interested up to speed is as ridiculous as demanding that concert pianists train their audiences to also be concert pianists. It is, of course, reasonable that the data come with some documentation and support — but that requires resources that funding agencies may have forgotten to provide along with their ethical exhortations. And the huffing and puffing was unnecessary anyway, because the LIGO-Virgo collaborations had every intention of releasing the data in their own sweet time, as they now have. Again: more patience, less whining.

As for the “controversy” itself, its much-celebrated public journey stalled in a bog of confusion. The Danish group responded to the rebuttal, but neither accepted nor criticised most of its statements. So much for “science out in the open”. The Danish group’s response instead focussed only on whether one figure in their paper could be reproduced. They also tracked down a bug in the code used to produce the LIGO response, but it looks like fixing it made no difference to the conclusions, although, again, that point was neither conceded nor challenged. This all looked less like a scientific discussion and more like a politician being evasive in a TV interview. (Gosh, who would have thought that the public spotlight would encourage such behaviour!) If you stepped back and asked, “Did the LIGO detection fail to meet the standard five-sigma detection criteria?” the answer would of course be, “That was never explicitly questioned in the Danish Paper anyway.” If you asked, “Was any error at all uncovered in the official LIGO-Virgo analysis?” the answer would have to be, “None has been clearly identified.” Although you could be forgiven for declining to draw any conclusions from the public record of the so-called debate.

Given this murky state of affairs, I was happy to sit back and wait for a year. After all, maybe in the end the Danish group’s work would contain useful and novel ideas that would improve LIGO-Virgo analyses? Was this in fact an interesting case of two communities failing to communicate with each other? (The Danish group’s background is mostly in cosmology.) Would my ranting prove premature? Might I even have to concede that Hossenfelder’s article, which I got in such a tizz about, actually forced these two groups to work out how to speak to each other?

Short answer: No to all of the above.

My assessment of the Danish Paper one year ago was, “Ignorable”. With the passage of twelve months, and not a single constructive development, I would qualify that assessment: “Definitely ignorable”. I wish my LIGO colleagues had not wasted so much time with it.

I suspect that many people would disagree, and this is where we get to the bit where I do some reflection.

There is one point I entirely failed to appreciate at the time.

I reacted strongly because I thought that the Danish Paper was obviously flawed. By “obviously” I mean that, if you talked to a gravitational-wave data-analysis expert, you would be left in no doubt. I could not understand why others (like Hossenfelder) could not ask those experts, see immediately that this was a non-story, and move on. With every scientist who was quoted saying, “If this holds up, it is a big deal”, I became more amazed. What was wrong with these people!?

The point is that, as a member of the LIGO collaboration, I was surrounded by data-analysis experts. I had not looked at the paper any more closely than all those people who claimed to be very worried, but I had complete confidence in the expertise of the people who were telling me what was wrong with it.

People outside the collaboration could not do the same. There was no external community of gravitational-wave experts to turn to. If you doubted a result from LIGO, there was no-one to ask for a “second opinion”. What was obvious from the inside was not at all obvious from the outside.

There are a lot of reasons to trust the insiders’ opinions — the strongest being that this community has lived in the shadow of Joe Weber’s incorrect detection claims for over forty years, and are petrified (even after six confirmed detections) of making a mistake — but, still, nothing beats a solid crowd of independent experts.

I did not appreciate that, and that is why I was so confused by the reaction. That is why my rant had such a strong tone, and possibly also why "outsiders" found it so perplexing. I now see that it was quite reasonable that people on the outside saw the episode positively, as an opportunity to explain the science more thoroughly, while from my inside vantage point, people had wasted their time and energy cleaning up a mess that should not have been made. At the time I diagnosed the problem as part of the craziness that results from high-publicity science. I still think that was part of it, but I now think that the embryonic state of the field is an equal part of the explanation: if there had been a broad well-recognised community of independent experts, the Danish Paper would probably not have been written, or, if it was, that the Forbes post would not have been published about it.

I would still vote to ignore such a paper. A media kerfuffle is more difficult to ignore; terror of the capriciousness of jittery funding agencies is hard to wave away. One can argue over how LIGO did (or did not) respond, or the communications and mis-communications that lead to the Forbes article, but it is likely that any lessons will ultimately prove irrelevant. After all, the external (non-detector-collaborations) community is already growing. They can provide context in potential controversies, and they are also, of course, the groups of people who really will be able to double check results. Open data is only useful if there are independent groups of experts to look at it — the notion that a twelve-year-old whizkid from Saskatchewan will refute a major scientific result is a romantic myth — and in time those groups will be plentiful.

A final point: my dismissal of the Danish paper’s results does not translate into dismissal of the authors. I know nothing about them, and have no reason to doubt their intelligence, technical skill, or their attempts to do a thorough, correct job. The issue is simply that they were unfamiliar with the subtleties of these data and how they were analysed. Sometimes an outside perspective reveals something new, and sometimes not. This time not. Next time it might be different. That would be great — my smugness could do with a corrective — and I will be very pleased when it happens. But I will also wait to be confident they are correct before I trumpet it to the world.

More Gravitational-Wave Stories

February, 2016:
The Discovery
How it Felt
How We Squeezed Out the Juicy Science

March, 2016:
Trying to Explain Gravitational Waves (Part I) (Part II)

June, 2016:
Book Review: Black Hole Blues
Detection Number 2 -- Black Holes Rule!
Rumours, Secrets and Other Sounds of Gravitational Waves

February, 2017:
One Year Anniversary (of being world famous)

June, 2017:
Detection Number 3 -- Nothing to see here: they are black holes
A hint of controversy

September 2017:
Detection Number 4 -- Virgo nails it

October 2017:
Did I just win the Nobel prize?

No comments:

Post a Comment

[Note: comments do not seem to work from Facebook.]