Positives matter - negatives matter too

Author: Diamanto Mamuneas

 

Carl Sagan once commented that at the heart of science lies the readiness to let go of one’s dearest illusions and accept hard truths.

The problem of publication bias has been identified and discussed extensively already but the extent to which this, ultimately a peculiarity of human nature, may have repercussions does not seem to be something many are willing to think about. As Hugo Mercier and Dan Sperber presented recently, people have a tendency to look for arguments that support their own pet theory and ignore those that seem to stand against their ideas – reason has evolved as a weapon for use in debating where the truth itself does not matter. We each have an in-built argumentative device which cherry-picks evidence to support our ideas.

It is no wonder then that we look for positive results in our experiments and set aside those that are not deemed a success. And it is equally unsurprising to find that it is positive results that disproportionately make it into peer-reviewed journals.

There is no harm in publishing the positives, one might think. However, any worldly statistician should be able to foresee the potential failures of such an approach. Every time an experiment is run, there is a chance that the results that come out of it will represent reality (i.e. be ‘true’) and a chance that they will not (i.e. be false). And this is the case whether the results are positive or negative with respect to the hypothesis being tested.

Now consider – exaggerated though this scenario may be - that the only results ever published are those deemed supportive of the alternative hypothesis being posited. A proportion of those results will not represent reality, by virtue of chance alone. The corollary, of course, is that a portion of unpublished results would have turned out positive if it were not for chance.

The consequence of such a situation is that, when someone attempts to repeat and thus verify a published study, there is a disproportionately high chance that they will not be able to duplicate it. This is not because they have performed the experiment inaccurately or because of some mysterious force driving effect sizes down; it is simply that each successive replication increases the probability of finding the ‘truth’ and the population of studies from which replications are drawn is already biased in favour of so-called Type I errors.

As long as replications are made, this seems relatively harmless. However, this method - the one currently employed by scientists globally – raises some disconcerting questions. Firstly, what happens to all those unpublished false negatives? Possibly these ideas are doomed to gather proverbial dust until being independently stumbled upon once more for a second chance at positivity. Secondly, how efficient is all this? It seems more sensible to publish all results judged to be reasonably methodologically reliable, irrespective of whether they fall in favour of a hypothesis or against it. Turning to Carl Sagan’s words once more, a scientist should welcome defeat as readily as assurance.

So we have a problem – progress is being hampered by a statistical inevitability inherent in the way science is currently done. The solution is change. We need the motivation to submit not only our treasured supportive results but those negative results we resent also. A possible approach is a global database into which all hypotheses and subsequent results can be entered and stored. Not only would this mean a coordinated effort that solves these basic problems, but a chance to have convenient access to results that can be searched for relevancy and used in meta-analyses much more effectively than ever before.

Peter Norvig, Director of Research at Google, recently featured in an interview published on Nature News. On being asked to account for the 2008 retreat from Palimpsest, an ambitious project that would host open science data, he explained that scientists “weren't quite ready to share” or “maybe we were just a little bit too early”. Science is simultaneously cooperative and competitive. The people established within it have flourished under the current system and are understandably protective of their data, which are directly linked to their income. At the same time, a more efficient system favours both career advancement and valuable progress – and this ‘value’ needs not just be fuzzy, moral warmth but a real, tangible monetary one where medical breakthroughs and technological innovations lower social costs. Scientists have to pay taxes too and they do not have a divinely-prescribed exemption from disease either. Norvig did suggest that maybe a time would come when a similar endeavour could again be made and that perhaps scientists just need to better understand what’s in it for them. On the 18th of July, 2011, a UK governmental select committee conducting an inquiry into peer review recommended that "researchers should aim for the gold standard of making their data fully disclosed and made publicly available" so perhaps that time is finally approaching.

We need to feel that we are acting for science and not simply within it; searching for truth at whatever cost to our individual pride in an effort to accept those hard truths Carl Sagan thought of as precious, beautiful, indispensable. Many of us seem to start our careers with this ideal but lose the way somewhere along the road, perhaps ironically driven by our personal argumentative device. Resistance is to be expected, of course but there is also the hope that a fresh desire for truth or acceptance of a moral calling will make such a vision plausible. There are people for whom such a change would represent more than just a step up to a higher tax break: for a newcomer in the field, this is potentially life-changing; for a cancer patient and their family, potentially life-saving.