Artificial intelligence is reshaping how scientific manuscripts get evaluated before publication, though not by replacing human reviewers. The shift is subtler: AI tools now assist with finding the right reviewers, flagging statistical errors, screening for fraud, and even drafting preliminary feedback on papers. A cross-disciplinary analysis of publisher policies found that most institutions restrict AI to auxiliary tasks like grammar checks and reviewer matching, insisting on human oversight for actual editorial decisions.1Learned Publishing. A Cross‐Disciplinary Analysis of AI Policies in Academic Peer Review The result is a publishing landscape where AI is increasingly present in every stage of peer review but rarely in charge of any of them.
Where AI Already Fits Into the Peer Review Workflow
The most widespread uses of AI in peer review are the unglamorous ones. Journal editors spend enormous amounts of time on logistics: deciding whether a submission fits their journal’s scope, finding qualified reviewers who are available and free of conflicts of interest, and checking manuscripts for basic formatting and statistical consistency. AI tools handle several of these steps with reasonable accuracy. One framework designed for open journal systems uses natural language processing to compare a manuscript’s abstract and keywords against reviewer expertise profiles, then recommends the best-matched reviewers automatically.2Computational and Applied Science. AI-Assisted Editorial Decision Making in Open Journal Systems (OJS): A Framework for Manuscript Scope Analysis and Reviewer Recommendation This kind of matching saves editors hours of manual searching through databases of potential reviewers.
Reviewers themselves report that AI can automate preliminary screening, plagiarism detection, and language verification, reducing workload and making the application of review standards more consistent.3Journal of Academic Ethics. Exploring the Impact of Generative AI on Peer Review: Insights from Journal Reviewers In practice, this means a reviewer might receive a manuscript that has already been checked for duplicated text, flagged for missing ethics statements, and matched to their specific expertise. The review still requires human judgment, but the groundwork is done faster.
Statistical error detection is another area where automated tools have gained traction. Programs like Statcheck and the GRIM-Test scan papers for internal inconsistencies in reported statistics, catching errors that human reviewers often miss.4PubMed Central. Artificial Intelligence in Detecting Statistical Errors: Implications for Authors, Reviewers, and Editors A paper might report a p-value that does not match the test statistic and degrees of freedom it claims. These tools flag such mismatches automatically, though they still need a human to determine whether the error is a typo or something more concerning.
Can AI Actually Write a Decent Peer Review?
This is the question that gets the most attention, and the early evidence is more interesting than a simple yes or no. A study comparing reviews of BMJ submissions found that reviews generated by large language models scored higher than human reviews on several quality measures, including identifying strengths and weaknesses, providing useful comments on writing and organization, and overall constructiveness.5Peer Review Congress. Quality and Comprehensiveness of Peer Reviews of Journal Submissions Produced by Large Language Models vs Humans The LLM-generated reviews were rated roughly 4 out of 5 for identifying strengths and weaknesses, compared to about 2.7 for human reviewers. That is a meaningful gap, and it held across multiple quality dimensions.
But scores on a structured rubric do not tell the whole story. A larger analysis of how well LLM reviews align with actual peer review decisions found that the picture depends heavily on which model you use and what kind of paper it is evaluating. For papers that were ultimately rejected, two major LLMs produced ratings similar to human reviewers. For the strongest accepted papers, though, those same models were harsher than humans, while a third model was consistently more generous across all categories.6arXiv. How Closely Do LLM Reviews Align with Human Peer Review? The finding undercuts any simple narrative about AI being uniformly tougher or softer than people. The pattern depends on the model, the paper quality, and probably the field.
The practical implication is that AI-generated reviews can be genuinely useful as a first pass. They tend to be thorough in covering structural issues, methodological basics, and writing quality. Where they fall short is in the kind of deep domain insight that distinguishes a competent review from a great one. An AI can notice that a statistical test seems inappropriate for the data described, but it cannot assess whether a novel experimental approach represents a real advance over existing methods in the way an experienced researcher can. The most productive use, at least for now, seems to be as a supplement: the AI drafts feedback, and a human reviewer edits, refines, and adds the expert judgment that the model lacks.
Catching Fraud and Paper Mills
One of the most consequential applications of AI in peer review has nothing to do with improving feedback quality. It is about catching fraudulent science before it gets published. A machine learning model trained to identify potential paper mill publications in cancer research flagged about one in ten papers across a corpus of over 2.6 million publications, and the rate of flagged papers increased sharply between 1999 and 2024.7PubMed Central. Machine learning based screening of potential paper mill publications in cancer research: methodological and cross sectional study The model achieved an accuracy of 0.91, meaning it got the classification right about nine times out of ten. That figure held even when the analysis was restricted to the highest-impact journals.
Paper mills are operations that produce fabricated or deeply flawed manuscripts for sale to researchers who need publications for career advancement. They represent one of the largest threats to the integrity of the scientific literature, and their output has grown dramatically. Traditional peer review was never designed to catch this kind of systematic fraud. An individual reviewer evaluates a single paper in isolation, with no way to notice that the same slightly varied figures or boilerplate methods sections appear across dozens of unrelated submissions. Machine learning models can analyze patterns across entire databases, spotting the statistical and textual fingerprints that paper mills leave behind.
Image manipulation is a related but distinct challenge. Generative AI can now produce realistic-looking microscopy images, Western blots, and other scientific figures that are difficult for the human eye to distinguish from authentic data. Detection tools exist, but their accuracy remains limited. One assessment found that automated image forensics methods performed only about as well as human visual inspection, and neither approach was reliable enough to confidently mitigate the threat of AI-generated image forgery.8Patterns. AI-enabled image fraud in scientific publications This is an area where the technology that creates the problem is outpacing the technology meant to solve it.
AI also shows promise in detecting more subtle integrity issues. Emerging algorithms can identify abnormal citation behaviors, where authors or groups of authors cite each other excessively to inflate metrics, and can flag missing or vague ethics statements in submitted manuscripts.9PubMed Central. Perspectives of Artificial Intelligence Use for In-House Ethics Checks of Journal Submissions These checks used to be entirely manual, prone to being overlooked when editors are processing hundreds of submissions.
The Bias Problem Cuts Both Ways
AI tools in peer review introduce their own biases, and some of these disproportionately affect researchers who are already disadvantaged. A well-known study demonstrated that AI-generated text detectors frequently misclassify writing by non-native English speakers as AI-generated.10PubMed Central. GPT detectors are biased against non-native English writers If journals start using such detectors as screening tools, researchers whose first language is not English could face higher rejection rates or be falsely accused of using AI to write their papers. The irony is thick: a technology meant to catch AI misuse could end up punishing the people who had nothing to do with it.
Language bias in peer review is not a new problem, but AI may amplify it in new ways. An analysis of over 15,000 peer reviews found that papers studying non-English-language topics faced substantially higher rates of negative bias compared to English-only papers, with negative bias consistently outweighing any positive bias.11arXiv. Are Non-English Papers Reviewed Fairly? Language-of-Study Bias in NLP Peer Reviews When AI systems are trained on review data that already contains these biases, they can reproduce and entrench them. A model learning what a “good” paper looks like from historically biased reviews will inherit those prejudices.
A scoping review of AI applications in scholarly peer review identified several additional risk categories: algorithmic bias favoring elite institutions or male authors, and the homogenization of scholarly voice as AI-assisted writing smooths out the distinctive styles that different researchers bring to their work.12PubMed. Artificial intelligence in scholarly peer review: a scoping review of applications, risks, and governance challenges That last point rarely gets discussed, but it matters. If every manuscript is polished by the same AI before submission, and every review is partly drafted by the same models, the literature starts to sound like it was written by one entity. The diversity of perspective that makes science productive could erode quietly.
Hallucinations and Fabricated References
Large language models generate text by predicting what word comes next, not by understanding whether what they are saying is true. This fundamental limitation produces hallucinations: outputs that are factually wrong, internally contradictory, or simply made up. In academic contexts, the most common forms include citation fabrication, where the model invents a plausible-sounding reference that does not exist, factual distortion, and logical inconsistencies.13PubMed. AI hallucinations in academic writing: implications for research integrity An AI-generated review that confidently cites a nonexistent study to support its critique could mislead both the editor and the author.
The problem extends beyond individual errors. Hallucinated findings can propagate through the literature if they are not caught. An AI might fabricate a result that an author then accepts and references in a revision, or an editor might take a hallucinated criticism at face value and base a decision on it.14PubMed Central. Hallucinations in Scholarly LLMs: A Conceptual Overview and Practical Implications This is why every major publisher policy insists on human oversight: the AI’s output needs to be checked by someone who can verify claims against real sources. Treating AI review feedback as authoritative without verification invites errors that look credible precisely because the language is fluent and confident.
Confidentiality Is Harder Than It Sounds
When a reviewer uploads an unpublished manuscript to an AI tool for assistance, that manuscript’s content enters a system controlled by a third party. Handling sensitive, unpublished research data through AI systems brings real risks to data privacy and intellectual property.15Journal of Korean Medical Science. Artificial Intelligence in Peer Review: Enhancing Efficiency While Preserving Integrity Without clear protocols, confidential findings or novel methods described in a manuscript could leak or be incorporated into training data.
The problem is more stubborn than it appears. Even when AI tools claim that uploaded data will be deleted on request, independent verification of that deletion is impossible. One researcher’s experiment with ChatGPT indicated that materials may be retained for training purposes even after a deletion request, making complete removal theoretically impossible.16PubMed Central. Zero Risk Is Impossible: Confidentiality and Artificial Intelligence Use in Peer Review Reviewers also sometimes include highly confidential comments or keywords in their review sheets. Uploading those to an external AI system could compromise the confidentiality that the entire peer review process depends on. A reviewer’s blunt private assessment of a manuscript, or their knowledge of the authors’ identities in a supposedly blinded process, could end up in a training dataset with no way to retrieve or remove it.
What Publisher Policies Actually Say
The publishing industry’s response to AI in peer review has been cautious but uneven. A survey of 163 academic publishers found that only about a third had publicly available policies addressing generative AI use by authors. None permitted AI tools to be listed as authors, citing accountability concerns. Nearly all publishers with policies in place required disclosure of AI use, but how that disclosure should happen varied widely: some provided standardized templates while others left requirements vague.17Peer Review Congress. Policies on Artificial Intelligence Among Academic Publishers
Most publishers with policies allowed AI for drafting non-methodological sections like introductions, and about a third permitted it for research methods such as data analysis. Very few addressed AI use in image generation or proofreading. Only one publisher allowed citations of AI as primary sources, while about 12% explicitly prohibited such citations. Four publishers banned AI from manuscript preparation entirely. The overall picture is one of cautious permission with mandatory transparency: you can use AI tools, but you must say so, and the human authors remain responsible for everything.
The gap in these policies is the reviewer side. Most guidance targets authors, not the people evaluating their work. The increasing adoption of AI tools across academic publishing has prompted calls for standardized guidelines to ensure fairness, transparency, and accountability on both sides of the review process.18PubMed. Using AI Tools in Writing Peer Review Reports: Should Academic Journals Embrace the Use of ChatGPT? A reviewer who uses ChatGPT to help draft feedback faces few formal restrictions at most journals, even as the authors they evaluate are subject to increasingly detailed disclosure requirements.
Detecting AI-Written Reviews
If reviewers are quietly using AI to write their reviews, can journals detect it? Researchers have framed this as a binary classification problem: given a review, determine whether it was written by a human or generated by an AI system. The challenge is that many reviews fall in between, since a reviewer might draft their comments manually and then polish the text with an AI tool. Detection models trained on reviews from one year also tend to degrade when applied to reviews from later years, as both AI capabilities and writing norms shift over time.19arXiv. Detecting AI-Generated Content in Academic Peer Reviews
This temporal drift is a serious practical obstacle. A detector that works well on 2023 reviews might perform poorly on 2025 reviews because the language models have changed and reviewers have adapted their usage patterns. The cat-and-mouse dynamic between generation and detection technology means journals cannot simply deploy a detector and consider the problem solved. Continuous retraining and updating is required, and the hybrid human-AI reviews that are probably most common in practice are the hardest to classify correctly.
Performance Varies Across Disciplines
AI tools for peer review do not work equally well in every field. An analysis of AI-assisted reviewer selection found higher accuracy in STEM fields compared to humanities and social sciences. Qualitative feedback from editors revealed appreciation for the system’s ability to identify lesser-known experts who might otherwise be overlooked, but also concerns about its grasp of interdisciplinary work.20Learned Publishing. Enhancing peer review efficiency: A mixed‐methods analysis of artificial intelligence‐assisted reviewer selection across academic disciplines
The discrepancy makes intuitive sense. STEM papers tend to have more standardized structures, more consistent terminology, and more quantitative content that AI can parse. A paper in molecular biology follows predictable sections with methods that can be matched against a reviewer’s publication record. A paper in philosophy or critical theory might draw on idiosyncratic frameworks that resist the kind of keyword-based matching AI excels at. The risk is that journals in fields where AI works less well either adopt tools ill-suited to their needs or get left behind as STEM journals accelerate their editorial processes.
Adoption Is Real but Uneven
How widely are academics actually using these tools? A study of Nigerian university academics found a moderate adoption rate of about 62%, though generative AI tools were used less frequently than simpler grammar and writing assistance tools.21International Journal of Research and Scientific Innovation. Measuring the Adoption and Efficacy of AI-Powered Tools in Academic Writing and Peer Review among Academics in Nigerian Universities That pattern likely generalizes beyond Nigeria: most researchers are comfortable using AI for spell-checking and sentence-level editing, but fewer are using it for substantive intellectual tasks like analyzing data or drafting review arguments.
The adoption gap also tracks with institutional resources. Researchers at well-funded institutions in wealthy countries have easier access to premium AI tools, more training in how to use them effectively, and a professional culture that is quicker to embrace new technology. Researchers in lower-resource settings may rely on free or older tools with weaker capabilities. If AI-assisted reviews become the norm, this disparity could create a two-tier system where some reviewers produce polished, AI-enhanced feedback while others work without the same support, potentially affecting the perceived quality of reviews and the journals that depend on them.
When AI Helps the Reviewer Rather Than Replacing Them
The most sustainable model that is emerging across the literature is not AI as reviewer but AI as reviewer’s assistant. AI-assisted review processes can reduce the workload of human reviewers and provide a preliminary evaluation, while leaving the judgment calls to people.22Green Analytical Chemistry. Reflections on the impact of artificial intelligence on peer-review practices and its implications for greener scientific evaluation In this model, the AI handles the checks that are tedious but well-defined: Does the paper cite relevant prior work? Are the statistics internally consistent? Does the methodology section include all the details a replication would require? Does the manuscript fall within the journal’s scope?
The human reviewer then focuses on the questions that require expertise: Is the research question worthwhile? Is the experimental design adequate for the claims being made? Does the interpretation follow from the data? Are there alternative explanations the authors have not considered? These are the questions that matter most for scientific progress, and they are the ones AI handles least reliably. A hybrid approach also addresses the hallucination problem, since a human reviewer checking AI-generated feedback against their own reading of the paper is likely to catch fabricated references or distorted claims that the AI presents with unwarranted confidence.
The reviewer shortage in academic publishing is real and getting worse. Journals struggle to find qualified reviewers willing to donate their time, and turnaround times have lengthened across many fields. If AI can cut the time a reviewer spends on a single paper from several hours to one or two, more researchers might be willing to review. The technology does not need to be perfect to be useful. It just needs to handle enough of the routine work that the human part of the job becomes manageable again.