We're living in an epidemic of bad research

The reproducibility crisis in biology

A story was published recently that should send shivers down the spines of biologists worldwide:

Serious errors plague DNA tool that’s a workhorse of biology

The article, which appeared in the journal Nature, reported how a group of researchers found that nearly half of lab-made plasmids had defects. Plasmids are circular pieces of DNA that are routinely used in modern biology experiments.

Electron micrograph image showing plasmids with various genes highlighted. Credit: Dr Gopal Murti/SPL

Apparently, many scientific papers have been published “for which the results are not reproducible owing to errors in plasmid design.”

The team found that nearly half of the plasmids had design flaws, including errors in sequences crucial to expressing a therapeutic gene.

Nearly half of the plasmids had design flaws.

The researchers who found this were “surprised” by the frequency of errors uncovered in the study.

But perhaps they would not have been surprised if they had looked at the problems in other routinely-used molecular biology techniques.

It’s not just the plasmids

In 2015, Nature also reported on a major problem with antibodies, which are proteins that are among the most commonly used tools in the biological sciences: Reproducibility crisis: Blame it on the antibodies

They reported that different batches of antibodies that were sold by the same company, that were supposed to be identical, did not give identical results.

The batch-to-batch variability… can produce dramatically differing results.

This is bad; the majority of molecular biology studies probably use antibodies for at least part of the study. You may have heard of western blots; antibodies are key to that technique. They’re also used in many diagnostic tools, including many tests for COVID.

The article also mentioned that an evaluation of 246 antibodies used in epigenetic studies found that one-quarter failed tests for specificity, meaning that they often bound to more than one target.

Image from here.

Now, it’s not news that sometimes antibodies aren’t specific enough, but we’ve likely grossly underestimated how problematic they are. In this study, they tested thousands of commercial antibodies, and found that only about half were broadly reliable. Eek.

Many such cases

Problems abound with many experimental techniques. I previously reported on major problems with staining. For more on that, see the “Artifacts in immunofluorescence” section here.

And I’ve previously written on how many cell cultures are contaminated:

Bad science: a case study in pollution of the scientific literature

This is a problem, because this can radically affect the results of experiments.

What’s most alarming, is that even when a problem is discovered, often nothing gets done about it. The problem of contaminated cell cultures has persisted for decades.

Errors and artifacts are commonplace in experimental biology. Here are some more random examples:

• Autophagy: assays and artifacts

• Addressing artifacts of colorimetric anticancer assays for plant-based drug development

• Sequencing artifacts derived from a library preparation method using enzymatic fragmentation

• Artifacts and biases of the reverse transcription reaction in RNA sequencing

And of course, there are cases of fraud.

More examples of fraud here.

It’s not just experimental error

The problem is worse than just experimental error or fraud.

In this article, I discussed how different biologists were given the same dataset and asked to analyze it:

Biologists get completely different results from the same data sets

The researchers came up with vastly different conclusions, even though they were working with the same data. This was because they differed in their analytical decisions.

You can imagine how much worse it gets when we’re talking about models. Not only might a model be based on bad data, either due to experimental error or outright fraud, but different decisions on how to make the model can significantly affect its predictions.

It’s hard not to conclude that we are living through an epidemic of bad research.