Monday, 7 July 2014

Confusing results


My friend Richard had a big problem.

A prestigious East Coast university, home to the world's leading research group in his field, was about to interview him for a faculty job -- on the basis of incorrect results that he had foolishly announced in a seminar -- in the vain hope that they would later transform into correct results.

Three weeks later, they remained obstinately incorrect.

On inspection, the computer code he and his collaborator John had written was such a mess that the original results could be correct -- they just had to fix up all the bugs to find out. That was fine. All their previous projects were like that, too. Each day John hacked three new ideas into the code, and the next day hacked them back out; Richard went through it methodically, line by line, routine by routine, cross-checking and verifying and debugging; and they were both spurred on by the sight of their supervisor's look of profound disappointment.

But not this time. The impending job interview ruined all that.

John said, "You have to turn them down!"

"Would you turn them down?"

"Ask them to offer me the job instead, and we'll find out."

Yes, in science it's all "pure intellectual curiosity", "the search for truth" and "the democracy of ideas". Until someone gets offered a job.

They barely spoke after that. Richard tried to work on the code by himself, but he was too distressed to concentrate. He was sure he wouldn't be able to reproduce their results, he wouldn't get the job, and now that his great long-standing collaboration with John was in tatters, his career would be over, too.

It's a lucky thing that he called me for help. He knew he could count on me for sensible, practical advice, and moral clarity.

"Just go to the interview and lie," I said. "The preliminary results can turn out to be wrong after you get the job."

Richard could see the serious ethical complications of this approach. "But what if John tells them?"

"John would be crazy to ruin a collaboration with a professor in the top research group in the world."



"Oh yeah."

"He'll get over it -- whether you get the job, or not."

Richard was still unconvinced. "I can't lie to these people."

"You certainly can't tell them the truth! And it's not lying. The results really are preliminary. You really are still checking them. You haven't published them. And what do you mean, 'these people'? If they're stupid enough to offer you a job based on preliminary results from one seminar, then they're morons. Come on! How unprofessional are they? They deserve what they get!" Before he could object, or even absorb this deep analysis, I added, "There's only one serious thing I would advise."

"What's that?"

"Negotiate immediate tenure. Because when they find out the truth, you're never going to get it any other way."

He thought that was rotten advice, but he decided to accept the part about John. The guy was still in their office working hard, and Richard had to believe that as soon as he fixed the code, he'd be so excited that their rift would be instantly healed. In the mean time Richard agonised every day over a new research talk to present as part of the interview that he was sure he was going to cancel tomorrow.

"I really was going to cancel it," he told me later. "I was sure I would. Even after I flew there. Even after I went in the building. And maybe I really would have, if I didn't meet those grad students."

Just after he entered the physics building of that illustrious East Coast university, a group of graduate students rushed up to him.

"Thank God you're here!" they said. "After your last visit, we've been trying to reproduce your results, and we just can't."

The game was up! Richard just nodded dumbly.

"There's a different cool result that shows up, but nothing like what you see."

"What different cool result?"

Richard told me, "When they explained, I really thought I was going to faint. We had spent so much time trying to get our amazing result back, that we'd missed something else." Result B paled in comparison to Result A, but if Result A didn't exist, then Result B suddenly looked like a very big deal indeed.

Richard kept his composure, and said, "That sounds very odd. I think we should compare results."

"Certainly! Definitely! That would be great!" gushed the enthusiastic gullible students.

After that Richard could go to his interview and honestly tell the panel what an exciting and vibrant field he worked in. So many results were emerging! There were so many possibilities! "This whole research area is exploding!"

They gave him the job.

Richard got the code from the local students, and compared it with the one he and John had produced. The students' code also had bugs, but the new result wasn't one of them.

"The different results will take some time to reconcile," Richard told them. "But in the end, I suggest we write a joint paper."

The students readily agreed. While the students kept trying to reproduce his spectacular result with their code, he corrected his code to verify their result. And it really was his code, now. John had quit his postdoc and accepted a five-figure-salary with a software company. He hadn't been working all that time on the code, after all: he was writing job applications.

Richard's original result was the focus of the first draft of the paper. As the weeks progressed, its prominence decreased. It was finally relegated to an appendix, but wouldn't go away completely, because the students, touchingly loyal to their new superstar professor, were determined that it must be true, and eventually they would prove it.

"I told them to get rid of it, but they thought I was just being modest."

They were so convinced that they foolishly submitted the paper, with the appendix, to the electronic preprint server, arXiv. Richard was horrified, but he let it sit there for a couple of weeks, and when the students still couldn't reproduce the big result, they agreed, "Go ahead, doctor. Take out the appendix."

The paper was a big success nonetheless. There were follow-up results. Other groups reproduced them, and then expanded on them. For a few years Richard's group -- it was his group now -- lead the field, and did some great work. And it was great work: Richard knew his stuff, and was even more methodical and rigorous than he had been in the past. He wasn't going to make the same mistake again. He relentlessly demanded care and precision from his students and postdocs and collaborators. He became famous for it.

The original, incorrect, result is still there in the first electronic preprint of the paper. It baits researchers on the margins of the field, who regularly set their students the task of reproducing it. Sometimes they even manage to, although the results never hold up. Richard cringes every time.

Whenever we talk, he always goes back to it. "I'm still ashamed. Just think how many hundreds of student hours have been wasted!"

But he only says that so I can reassure him. "It's fine," I say. "It's a good problem to play with, and they learn a lot. If they're good, it won't do them any harm. And who knows -- maybe it will turn out to be right after all?"

And maybe it will. Such is science.

Repercussions -- "Is this the end?"

No comments:

Post a comment

[Note: comments do not seem to work from Facebook.]