Antibody Validation and Reproducibility: A Path Forward

“I didn’t start out with the intention of stepping into this mess.” 

Glenn Begley is sitting in an unfurnished office in Thousand Oaks, California, the new home of a biotechnology company that he is consulting for. With his clean-shaven pate and large framed glasses he speaks in a careful, measured tone that screams precision.

Watch Documentary: Reproducibility Issues in Life Science
Glenn Begley and others discuss
validation challenges and solutions.Watch Now

The former Vice President and Global Head of Hematology/Oncology Research at Amgen is reflecting on the work that his group did in an attempt to reproduce 53 seminal studies in the field of cancer research. The fact that they were only able to reproduce the findings in 6 of those 53 studies, despite, in many cases, working directly with the scientists that performed the initial work, led Begley to write a commentary in the journal Nature in 2012. That paper, entitled “Drug development: Raise standards for preclinical cancer research”, outlined changes that the authors felt should be made to the way research is performed and published along with the way researchers are awarded grants and tenure in order to address a growing reproducibility problem.

Perverse Incentives

“The fundamental problem is the perverse incentives that apply within science today,” says Begley. “I need to get my next research grant, I need to get promoted, I need to get my post-doc position. For that to happen I need to get published in a so-called top-tier journal. That’s a very powerful incentive.”

These perverse incentives, according to Begley, lead to behaviors across the entire life science community that result in the reproducibility problem. Researchers cherry-pick, or p-hack, their data in order to tell an exciting story, journals only want to publish impressive data, granting agencies put more of an emphasis on publishing in top-tier journals. And underlying all of it is the question of whether or not all of the reagents used in the experiments have been properly validated. 

Documenting the Solutions

The solution to these issues turns out, not surprisingly, to be complex and vary according to who is asked. Each stakeholder group, the researchers, the publishers, the product manufacturers/suppliers, the professional societies, and the granting agencies, views the reproducibility problem through a different lens and hence all have a slightly different take on the best way to address it.  

For the past several months Biocompare has been interviewing representatives from each of these groups to find out what they think needs to be done to improve the way science is performed. These interviews were filmed and have been put together as a documentary currently on the Biocompare website. The focus of the film is specifically on the role that antibody validation plays in scientific reproducibility. While this is a small piece of the overall reproducibility puzzle it is one that may play an outsized role.

“Informal conversations I’ve had with people like Glenn Begley…he will tell you that a big source of the inability to replicate the papers that they tried to replicate was, in one way or another, the source of the antibodies,” says Leonard Freedman, President of the Global Biological Standards Institute (GBSI), a non-profit organization formed to promote standards in basic biological research. 

Organizing a Response

In the fall of 2016 GBSI organized a conference that brought together members of the life science community to focus on the issue of antibody validation. The goal of this summit was to put forth and debate the problems associated with validating antibodies and develop a consensus around the best way to address those problems.  And because so many groups within the life science community were represented—researchers, antibody suppliers, publishers, societies and granting agencies—the debate promised to be robust.

What came out of the conference was not so much a consensus on what to do but a promise from each of the stakeholder groups to continue working on the issue. 

One concrete way this is happening is through a set of working groups that have been organized around specific antibody-based applications. Each working group has representatives from the research community as well as antibody producers, and the mandate is to come up with a set of recommendations for how antibodies should be validated for that use.

Need for Training

While the need for better antibody validation is clear, there are a number of other themes that are brought up no matter which stakeholder group is being interviewed. The one that is mentioned the most is training.

“There is a little bit of a lag in how young scientists are trained,” says Roberto Polakiewicz, Chief Scientific Officer at Cell Signaling Technology. “Training is the most important thing.”

That sentiment is echoed by many; there is a clear need for more training when it comes to using antibodies in the lab. This training would include everything from choosing which antibody to use, how to perform an experiment, and, probably most importantly, how to interpret the results from that experiment. 

“Even if manufacturers were producing perfect, recombinant antibodies, people would still find a way to screw it up,” says Andrew Bradbury, Chief Scientific Officer at Specifica. “I think there’s a need for more training.”

Transparency from Manufacturers

If “more training” is the most common refrain heard from the life science community, then “transparency” is a close second. To understand what “transparency” means in this context requires a little background on how antibodies are made, sold and distributed in the research market.

There are essentially three routes that an antibody can take to a lab: it can be sold by the primary manufacturer that made it, it can be sold to a distributor who purchases antibodies from many different manufacturers before selling it to the researcher or it can be purchased by a distributor who re-labels it and then sells it to the researcher under a new brand name. This last route is known as the original equipment manufacturer or OEM model and it is more common in the life science antibody market than one might think.

The reason these different delivery modes can affect researchers is convoluted but real. If a scientist purchases an antibody for an experiment that doesn’t work and then purchases three more antibodies to the same antigen, there is a chance that they may be purchasing the same antibody three different times if the companies selling those antibodies were acting as OEMs. The problem is that many antibody companies make, distribute and OEM at least some part of their portfolio so it’s hard for researchers to know the actual source of a given antibody. And thus the call for transparency.

The issue of transparency is not as black and white as some would make it seem. The research antibody market is extremely competitive, a fact that has kept prices low, and it is almost impossible for companies to determine which antibodies will sell and which will sit on the shelves. So in order to ensure they are meeting all of their customers needs, antibody companies need to have as broad a portfolio as possible which leads to most using all three delivery modes outlined above. 

“It may not be realistic for companies to disclose the origin of the antibody,” says Alejandra Solache, Head of Reagents Product Development and Manufacturing at Abcam “It could be there are contractual agreements in place with the supplier that actually prevents that.”

For some manufacturers identifying the source of a given antibody is almost beside the point. According to Carl Ascoli, Chief Scientific Officer at Rockland Immunochemicals “If they (researchers) have all the information that tells you how an antibody was made and tested, they’re going to be able to reproduce their studies regardless of the path the reagent took to get to them.”

Next Steps

The need to solve the issue of antibody validation is clear. It is a problem that has been percolating for many years and there now appears to be a critical mass of effort from the entire life science community to finally address it. 

The solution is likely going to require changes in the way that everyone involved in biological research goes about their job. This includes, but is certainly not limited to, getting more training for researchers, more data transparency from suppliers, more data in the Materials and Methods sections from publishers, more detailed guidelines from granting agencies and possibly more oversight from research institutions. 

“No one stakeholder group alone can solve this issue,” says Larry Tabak, Principal Deputy Director at NIH. “We’re all in this together.”

The good news is that the path forward is beginning to take shape.