In the weekly ‘follow-up’ section we present a sequel to last week’s best-read article here on Innovation Origins. This week: how come reputable medical journals such as The Lancet and the New England Journal of Medicine flagrantly accepted the results of a study based on an unreliable dataset?
It was a monumental scientific feat. Contributors to The Lancet stated that COVID-19 patients who are given the antimalarial drug hydroxychloroquine are almost 50% more likely to die than patients who do not take it. They did this using data from 96,000 patients in more than 600 hospitals. On the basis of this study, the WHO discontinued an international study into the effects of hydroxychloroquine, as did the French, English, and Australian governments.
Except that the data turned out to be too good to be true. The article was withdrawn by the majority of the writers. Just like another article – based on the same data in The New England Journal of Medicine. The reason why: it turned out that the researchers themselves were not familiar with the data that had been analyzed by the tiny Surgisphere company. Therefore, they were also unable to verify the raw data themselves.
Usually, data is peer-reviewed before a journal publishes anything about it. Nevertheless, the fact that the lead author, distinguished scientist Mandeep Mehra, heart surgeon and professor at Harvard University, did write about the results, is something that Frank van Kolfschooten, investigative and science journalist, finds ‘interesting.’
Trust
Van Kolfschooten: “Normally there are long-standing collaborations with people, and scientists are acquainted with each other. That didn’t seem to be the case here. There tends to be more trust in long-standing collaborations at any rate. So it seems that Mehra had a degree of trust in the company, but unjustifiably so.” According to Van Kolfschooten, this singular act ”undoubtedly has to do with these times of corona.”
“What you now see happening all the time is that procedures which should normally eliminate these kinds of errors and safeguard the quality of science are either accelerated or even abandoned. As there are huge public health concerns at stake.”
Nobel Prize-worthy
Besides, ”it is, of course, a Nobel Prize-worthy subject”, says science historian Klaas van Berkel, emeritus professor of history. “Whoever comes up with the right vaccine first that will conquer this virus, has a good chance of winning a few awards.” Scientists can then use the argument that they are helping others with their pre-publication, Van Berkel goes on to say. “So that they can then switch from doing the same kind of research to doing something else. Often it says in small print that it has not been peer-reviewed as yet. A full range of incentives is in place for bringing out results sooner rather than later.”
Publication pressure
Both Van Berkel and Van Kolfschooten also mention the pressure that is always present when it comes to academic publications. Van Kolfschooten: “The funding of entire research groups and departments is linked to the number of scientific publications. People can, therefore, feel pressured to publish as quickly as possible. Even if it does not actually merit being published at that point. That may not necessarily involve particularly serious matters, like completely fabricating or plagiarizing data. But that you actually are aware that you should carry out more experiments to test the data. Or you should take a closer look at the database for even better statistical analyses that demonstrate the robustness of a correlation; for example. Sometimes there just isn’t enough time for that.”
In his book Ontspoorde Wetenschap (Science Dreailed, 2012), Van Kolfschooten describes and analyzes numerous examples of scientific fraud and plagiarism. He details the Stapel affair, among other cases. Eminent social psychologist Diederik Stapel was found to have falsified 55 of his 130 publications during his career, after being investigated by the Levelt Committee. One of the reasons for an extensive investigation was a press release from 2011 that stated that carnivores are lousier people than vegetarians. This turned out to have been taken from an unpublished body of work. Subsequently, regulations for publishing results were sharpened. “A university must never go public with any results before they have been thoroughly peer-reviewed.”
Ever since the start of the last century
Science fraud or plagiarism is not something new to us, Van Berkel stresses. Ever since the start of the last century, we have known what research needs to conform to. There is just greater scrutiny on it these days. Not only from the science world itself but also from the media. What with a specialist field such as science journalism.”
One of the first accusations of science fraud that attracted media attention and which required an investigative committee to assess what had gone wrong had to do with the case of Henk Buck. Twenty years later emotions still flare up when discussing the details,” van Kolfschooten writes in his book. The Eindhoven chemist Buck proclaimed on April 12, 1990, that he had found a cure for the AIDS virus. A statement that proved to be far too hasty in the eyes of the Lemstra committee.
Flawless science
In any event, Van Berkel maintains that a whole world exists between science fraud and error. “There is no such thing as flawless science. There are so many flawed theories that were a stepping stone to a better theory. The history of science is chock-full of them. It might remind you of Descartes from those earlier days. His laws on motion were almost all wrong as far as collisions are concerned. Yet they put Christiaan Huygens on the right track towards his own, improved, and essentially correct, laws of collision. Also from the standpoint of more modern history: consider the electron theory from Lorentz, which Einstein utilized to develop his own theories. Incorrect presentations often provoke a lot of research that would otherwise not have been happened or might have been carried out to a lesser degree.”
Scientists are given the freedom to corroborate their hypotheses with research. It is work that is carried out by people, and well, to err is all too human. Van Berkel: “There are plenty of examples of accusations of plagiarism and fraud. If you take a closer look at them, you will discover that they were actually wrong. In fact, these were nothing more than simple and trivial mistakes, or minor errors due to carelessness. This does not mean that the entire result of that research should be wiped out. The outside world usually demands flawless science, which just does not exist, unfortunately.”
‘Caught with their trousers down’
For Van Berkel, Surgisphere’s research results are a textbook example of how gross errors ought to be detected at an early stage. Van Kolfschooten and Van Berkel are aware that both The Lancet and the New England Journal of Medicine have now ‘been caught with their trousers down’ and must admit their error of judgment.
Although they both assert that there seems to be a political edge to it all.
Van Kolfschooten: “An elitist, liberal magazine like the New England, really should have nothing to do with Trump. So, it was also a prestigious matter to be able to refute Trump’s statements on hydroxychloroquine so quickly.”
Van Berkel: “Politics soon comes into play. Whether you publish or not. If you do not publish anything, then you are accused of being politically correct.”