Q&A Part 4: Opinions on the Efforts Underway to Tackle the Reproducibility Problem

One effort that is realistic and implementable is adding a gold-standard method in validation (knockout, knock-down or mass spectrometry). Knowing how important it is to the scientist that their antibody truly recognizes their researched target, we were the first to implement a gold standard in validation using knockdown specifically. Another effort that should be implemented but is not yet is the identification of the manufacturer by antibody resellers. Scientists need to know this information to avoid purchasing the same antibody from multiple companies who have rebranded the same product which, if faulty, wastes their time, money and efforts and slows down science drastically. This disclosure will allow the manufacturer to take responsibility for an irreproducible antibody as well. Once this is implemented, the positive measures GBSI and Nature are taking will progress significantly. It will be straightforward to identify and trace antibodies to their lots and to the manufacturer since they are revealed. You won’t see multiple identification numbers for the same antibody’s lot due to rebranding from multiple antibody companies. There will be one identification number for that antibody’s lot and it can be traced to the manufacturer.

The meeting organized by the Global Biological Standards Institute (GBSI) in September of 2016 and the preceding meeting of the International Working Group on Antibody Validation where the “pillars of validation” were proposed were both steps in the right direction. These meetings brought together a diverse group of stakeholders: academic researchers, industry, funding agencies, teaching universities and journals. Global change will take a community effort by all of these stakeholders. If these efforts remain focused on a broad-based approach to improvement, then they will likely succeed, be implemented and result in significant improvement regarding data reproducibility. If, however, the focus of change becomes limited to one or a few stakeholders, or if the recommendations become skewed to favor particular users or commercial interests or manufacturers, then the outcome will likely not be attained.

The current efforts underway for the antibody reproducibility problem are a major step forward in solving the antibody reproducibility crisis, but it will take the entire community to embrace these new standards for proper implementation. Researchers will need to carefully consider which antibody companies are providing enhanced validation data to ensure antibodies are binding to the right target and are validated for proper application use. Research journals and funding organizations will need to require researchers to provide assurance that the antibodies used in their research and research proposals have gone through this additional rigor in validation. Antibody companies will need to expand their validation testing on the antibodies they provide and seamlessly provide researchers with the data and information required by journals and funding agencies to show the antibodies have been thoroughly tested for specificity and application validation. By working together, these groups will be able to ensure the antibodies used in their experiments are of the highest quality and can be comparable between research studies.

We must separate the discussions about premarket validation and post-market validation, even though both issues are equally important. Reproducibility is a post-market validation question, where the suppliers must have quality control processes in place to ensure that each newly produced lot of a product on the market will perform the same way as the previous lot in each approved application.

The current efforts underway are mainly focusing on setting formal criteria for premarket validation to ensure target specificity and identify potential off-target binding. Both antibody validation and reproducibility have always been of the highest priority at Atlas Antibodies, and we welcome all initiatives to create general guidelines for antibody validation.

General guidelines for antibody validation, agreed upon by all stakeholders, will make it easier for the antibody users to select the best antibody on the market for their specific protein analysis. In addition, increased transparency of validation results, protocols used and immunogen sequence will facilitate comparison of available antibodies for a specific target.

The guidelines suggested by the International Working Group for Antibody Validation (IWGAV), published in Nature Methods, September 2016, are realistic and implementable.

At Bio-Techne, we are fully behind current efforts to raise antibody validation standards in the industry, and collectively tackle the reproducibility problem. One of the key programs we are engaged with are the antibody validation workshops organized by the Global Biological Standards Institute (GBSI). The five conceptual validation "pillars" published in Nature Methods by the International Working Group on Antibody Validation (IWGAV) provide a helpful framework for discussion of application-specific standards for validating antibody specificity. These include genetic strategies, independent antibody strategies, orthogonal strategies, expression of tagged proteins and immunocapture followed by mass spec (IP-MS). We agree with IWGAV recommendations, including the need for application-specific and context-specific validation of antibodies. We are working with the stakeholders on developing these guidelines further to ensure recommendations are practical, workable and usable.

Some of the guidelines may be unrealistic for smaller vendors, due to technological or economic limitations. Let’s take the genetic validation pillar; to make a CRISPR-based knockout of all genes in several standard cell lines would be a huge undertaking for any single manufacturer. Right now, there are only a few leading vendors such as ourselves, who have committed to applying CRISPR knockout studies for specificity validation, but the issue of cost, speed and scalability remains. It should also be borne in mind that there are technical challenges with gene-editing technology, appropriate cell lines, and its inapplicability for certain targets such as epigenetic targets or PTMs.

Besides development of validation method standards by producers, we believe that there is a huge role to play by all the different stakeholders: from journals, funding agencies, antibody databases, research tool identification initiatives (e.g. RRIDs) and institutions, to help promote antibody training for researchers or more detailed materials and methods such as providing all clone and vendor information or full-size Western blot results.

We believe the current discussions around the reproducibility problem, for example the Asilomar conference sponsored by GBSI, have the potential to be very useful for the community. We are actively engaged in the discussions and in efforts to design achievable validation standards. There is currently a significant push to design a scoring system to qualify antibodies in the marketplace. It is an interesting idea, but I have serious doubts about its feasibility, impartiality, cost and potential unintended consequences. There hasn’t been a serious discussion about this, in my opinion. Other ideas practiced by other industries such as some type of quality certification have not been discussed much, but I feel this is one potential alternative to scoring individual commercial and academic antibodies.

No validation guidelines will be complete without engaging all stakeholders to tackle the different factors contributing to the reproducibility problem, not just the antibody providers. Encouraging the training of young scientists in the use of antibodies is crucial—the responsibility of this task can fall on both academic institutions and vendors by ensuring that students have access to resources and information. Enforcing journal publication standards that require authors to clearly identify the source, catalog and lot number of any antibodies used and thoroughly describe methods is just as important and potentially easier to accomplish in the short-term.

From the supplier’s point of view, I think the answers are going to be somewhat unique to each company. I think everyone agrees it is a big challenge, especially if we try to think of just one or two ways to tackle such a diverse market of products and their uses.

Thus, what we like are the ideas of standardizing test methodology, education and increasing the transparency of how the product was used/tested. And in that context we still prefer the rigor of peer-reviewed publication data over just peer review. Our goal is to bring more of that data to the surface to help customers find what they are looking for with confidence.

GBSI has set a very ambitious goal, as it is a tough task to define universal guidelines that are valid for different cell types, different experimental conditions and even different types of researchers. However, given the fact that there is an absence of any standards at all, the GBSI initiative is definitely a step toward improving reproducibility. It will provide companies and researchers a common framework to test and validate different antibodies. Also, it is just the beginning, and I am sure that once grant agencies and funding bodies start enforcing antibody validation guidelines, the current scattered efforts will become a global mission.

The reproducibility issue is a significant one that we must tackle immediately. It is a daunting task to execute uniform standards of quality to ensure accountability and reproducibility. I must commend current efforts from all sections of academia, pharma, and reagent vendors aimed at dealing with it. If we ignore this issue too long, it won’t go away on its own and is likely to get worse, leaving us with a growing number of irreproducible experiments and incorrect conclusions. Even though it is a huge challenge to come up with an effective system to monitor antibody quality that is realistic to implement, this is not something that we should avoid. I don’t think anyone believes that the reproducibility issue will be solved overnight. However, as we bring awareness to this issue and start to seek solutions, researchers will start to understand the value of highly validated antibodies and demand high quality reagents, which in turn will drive the market to meet this demand.

Developing a minimum set of criteria that an antibody should meet to be considered validated could benefit the industry. There are a large number of players who vary in their level of commitment to quality. If researchers could rely on a certified product meeting a known set of criteria, it would allow buyers an assurance of at least a minimal degree of quality. For those producers and resellers who do not currently meet that standard, it would provide a great incentive to elevate their practices. Having a set of minimum standards is both realistic and implementable.

There has been significant discussion of using recombinant antibodies to create more defined and reproducible products. BioLegend manufactures our mAb products using both traditional hybridoma clones and recombinant methods. A mAb produced off an engineered DNA source is not inherently more defined than one produced from a traditional hybridoma. We often obtain the relevant sequences of our in-house developed hybridoma clones. Whether we choose to express these directly from the hybridoma, from a transfected mammalian cell line, or from a permanent cell line selected to express our recombinant construct is largely decided based on performance and efficiency. In all cases, the peptide sequences of the heavy and light chains we express can be known, but the growth conditions of each production run must still be carefully controlled and monitored for consistency. Recombinant antibody constructs can be useful, but they are not a magic elixir that will guarantee reproducibility.

The notion that there are “bad” antibodies on the market has been around since the dawn of the commercial antibody market, though it has only been very recently that there has been a concerted call for action on an industry-wide level. Publications in major journals, surveys and studies by nonprofit organizations, and scientific conferences have all contributed to expanding the general awareness of this issue and possible solutions.^1-4 These have included, for example, better training and education of junior scientists with regard to research antibody selection and usage, standardization of antibody reagents through recombinant means, and proposals listing specific “conceptual pillars” for antibody evaluation, among others.^1,2,4 Other more technical papers describe the use of mass spec in conjunction with immunoprecipitation or other techniques to evaluate antibody quality.

Perhaps the major concern about addressing the antibody reproducibility issue is the sheer magnitude of the problem. Companies have thousands of antibody reagents in their catalogs, and new targets are identified each year. Attempts to evaluate all of these products through knockdown/knockout or other definitive approaches will require significant financial and manpower outlays. In addition, more specific testing is needed to assess the fidelity of antibodies directed to particular isoforms or post-translational modifications. Nevertheless, the benefits of being able to pare down one’s catalog and offer reagents with complete confidence, not to mention the fact that this is the future of the industry, will encourage companies to move forward. It will be a long process, but perhaps the IP/mass spec approaches being developed now will expedite this.

References

1. Bradbury, A. & Plückthun, A. Nature 518, 27-29 (2015).
2. Uhlen, M. et al. Nature Methods 13, 823-827 (2016).
3. Marcon, E. et al. Nature Methods 12, 725-731 (2015).
4. Freedman, L.P. et al. BioTechniques 61, 16-18 (2016).

Reproducibility in scientific research is a complex problem and as scientists, we have a shared responsibility to identify, understand, and address the factors that contribute to this problem. The increased visibility and awareness of reproducibility issues generated by the publishers of Nature and the Journal of Biological Chemistry (JBC), the National Institutes of Health’s rigor and reproducibility initiative, and the 2016 GBSI workshop on antibody validation are huge steps in the right direction.

These efforts are realistic and badly needed, but transforming guidelines and recommendations into standards is not easy or straightforward. Standards must be flexible enough to adapt to the experimental context, without being overly complex or burdensome. Awareness and training are important aspects of implementation. Open communication of antibody performance and validation data, along with consistent reporting of detailed validation and experimental methodologies by authors, will also be critical for success.

We are aware that procedures and standards for validation differ among suppliers. Sometimes, a biochemical condition of the storage buffer can influence the performance of an antibody. Therefore, establishing universal criteria or an authorized third-party validation might be the best approach. At Sino Biological, we compare our new products with the most popular antibodies utilized in publications for similar or enhanced performance.

However, because of the complicated nature of biological samples, the reproducibility issue might not merely be a problem of ensuring the consistent quality of antibodies by producers. Enhancing communication between users and vendors would be helpful for both sides. Scientists at antibody companies should participate more deeply in customers’ research efforts to help improve “precise design” for each individual study.

We look forward to new techniques that are cost-effective, high-throughput, and acceptable to the public. All of which is essential to improve the efficiency and precision of antibody validation. It will eventually benefit all antibody users.