Scientific Publishing. Image generated by Google's Gemini language model on June 20, 2025.

Holden Thorp, the Editor-in-Chief of Science journals, recently cautioned in an Editorial (America is ceding the lead in creating the future) that continued funding cuts are putting our nation's global scientific leadership position at risk. Especially China’s ever-increasing numbers of research publications are threatening our leadership position. However, a more important, yet less emphasized issue is poor-quality research.

The success rate of clinical trials has remained largely stagnant for decades, with more than 90% of clinical trials usually missing their endpoints or being terminated for unexpected signals. While some attribute this to “moving goalposts” as new therapies are benchmarked against evolving standards of care, up to roughly 90% of peer-reviewed preclinical research studies are not reproducible either, suggesting a more systemic problem in the life sciences (cf. Prinz et al. 2011, Begley & Ellis 2012, Baker 2016).

As scientists, we take pride in our independence and self-governance by incorporating peer review at two critical stages of the research endeavor: the allocation of resources by granting agencies and the evaluation of outcomes during the editorial publishing process. Considering the low reproducibility rates, one must ask whether the peer-review process truly fulfills its promise of efficient self-governance.

Because humans are inherently susceptible to groupthink and confirmation biases, I recently proposed in a recent op-ed (Can AI-enabled Red Teams Tackle the Reproducibility Crisis in the Life Sciences?) to establish an AI Red Team as the “third peer reviewer”. I believe an AI as the third peer reviewer could streamline the scientific peer-review process and set minimum quality standards across journals and funding agencies. This would increase the quality of our research and ensure continued self-governance of our research in the future.

The AI Red Team would be implemented by an independent, user-funded agency, analogous to the FDA. If mandated by legislation, federal agencies like the NIH could access it for free, while requiring that research outcomes are published in journals that incorporate the AI Red Team into the editorial process. An AI-empowered peer-review process would benefit not only our society but also publishers and research sponsors.

Scientific journals may find value in incorporating the AI Red Team into the publication process as it would create an entry-barrier for predatory journals. At the same time, the AI Red Team would evaluate material and methods sections for completeness and clarity, validate references, and standardize data formats and interpretations across journals without interfering with a journal’s unique layout and style. This would facilitate automated data access and support data sharing with machine-learning applications while further training the AI Red Team.

Functioning as a third peer reviewer alongside two human peers, the AI Red Team would excel at uncovering contradictions, missing controls, temporal trends, and logical inconsistencies. Its ability to analyze a body of literature vastly larger than any human expert's capacity would flag experiments demanding further cross-examination, and identify contradictions not only with previous studies but also across diverse scientific domains (e.g., discrepancies between animal models and human diseases).

One could even envision an AI Red Team simulating experiments based on published methods and data, then comparing simulated outcomes to reported ones. Crucially, this AI Red Team would be designed to identify subtle and systemic anomalies that human bias or sheer data volume might obscure, while human scientists would retain essential oversight and decision-making authority.

Beyond enhancing research quality, the insights generated by an AI Red Team would also guide and inform grant funding decisions. By rigorously evaluating the robustness and novelty of the proposed research, or by identifying areas of saturated or irreproducible findings, the AI Red Team would help funding agencies allocate resources more effectively and in alignment with their mission. This could promote the deepening of research in truly promising avenues, or conversely, encourage diversification into under-explored or emerging fields where robust initial data signals are more likely to lead to genuine breakthroughs.

Incorporating AI into the peer-review process of scientific journals and granting agencies will raise overall research quality and promote high-impact studies in the life sciences. The proposed independent, user-sponsored AI Red Team offers a cost-effective path to more rigorous research evaluation, benefiting society without imposing additional financial burdens. This approach would empower scientists to focus on truly meaningful topics and areas. Above all, it would solidify our nation's global leadership in the sciences, innovation and technology.

I invite you to join this conversation. Share your reactions and ideas on this very critical topic, as we all collectively seek solutions for a more reproducible future.

#artificialintelligence #science #scicomm