I skimmed through Daniel Engber’s piece on reproducibility in Biomedicine this morning (titled “Cancer Research Is Broken“). There’s several problems in my opinion with the way the piece is written.
The first problem is that the piece starts to claim that there is widespread “waste of time and money”. The first mention of the word “waste” is in this sentence: “Many science funders share Parker’s antsiness over all the waste of time and money.” Note that there is no source listed. I think if you are going to call out waste of time and money, you could be specific who told you they thought so, and provide data about the extent of the waste, and how exactly it is characterized (i.e., what are you comparing to?). In my book, waste happens when one performs an activity in one way, when another way exists that can save time or money. In the example of cancer research, I think many researchers would love to know what way would be better than the one followed, which would save time and money, if indeed “all the waste of time and money” exists.
The next sentence does not bring anything to address this question. We have to read another 11 sentence that do not use the word waste. The word occurs again in the following sentence:
“Given current U.S. spending habits, the resulting waste amounts to more than $28 billion”. We are now given a huge number to clarify the scope of the implied waste, but note that we are no closer to receiving an explanation of the term waste (i.e., what other activity would result in the same outcome but save time and money).
At this point, I skimmed through the rest.
One point this piece tries to make is that all the money spent is “wasted” because of lack of reproducibility. Reproducibility is defined as the ability to redo the exact same experience and get the same results. There is a broad discussion about reproducibility in biomedicine, and this piece seems to attempt to infer that lack of reproducibility is the result of corruption: “data are corrupt” (data don’t care about money, I guarantee you: they are not human) or inability: “The science doesn’t work.” (another example of misplaced anthropomorphism, science does not do anything, it’s a method. Scientists do use the scientific method, and have more success that people who do not use the scientific method).
One phrase that makes sense to me in this piece is “The findings are unstable.” I would have liked this point to be developed, because I believe it is the main reason why some studies conclusions/interpretations can end up being shown wrong (often later after the study has been published). A finding is unstable if you change the material of the plates and you get a different result. A finding is unstable if you try to do the same experiment in a different cell type where you expect the same result, and cannot observe the same effect. Reproducibility has little to do with it. If you change a factor that should be irrelevant to the outcome, but the outcome changes, the outcome is brittle, not robust, not worth writing about, and certainly should not end up being published in the scientific literature.
So I think if Biomedicine is in a crisis, it one of generalization more than reproducibility. I believe that if we could devise fully reproducible approaches, publishing results that are true when obtained with only one specific type of plates would not be particularly useful. Instead, I think that scientists continue to focus on understanding what factors should affect the result of an experiment and which should not, and design their experiments to include a at least one factor that they don’t expect to change the interpretation. I believe good scientists aim to report robust findings and are already taking such steps to filter results before they send them for publication. Others should take note.