Beyond fact-checking: Harnessing AI for news quality assessment

Reliable scientific information is crucial for shaping public opinion and informing economic, social, and health-related decisions. While significant research has focused on detecting misinformation, the automated assessment of the overall quality of journalistic content remains largely unexplored. Addressing this gap, our study investigates whether different artificial intelligence approaches ranging from classical statistical methods (“bag-of-words”) to advanced large language models (LLMs) can assist in evaluating the quality of medical news articles based on ten established journalistic quality criteria. Drawing on a dataset of 240 articles and corresponding expert reviews, we show that machine learning models can indeed assess article quality with considerable accuracy. While several models fall short of human-level performance, fine-tuned LLMs perform well and, in most cases, match the level of agreement with a consolidated expert benchmark observed among individual human reviewers. The best-performing model, a fine-tuned GPT-4o model, detects poor-quality articles reliably when evaluated against expert assessments (macro-averaged F1: 0.69 vs. 0.57), suggesting its potential for use in editorial workflows, such as filtering incoming material or double-checking an already written text. Although our study focuses on medical journalism’s well-defined criteria, the approach likely holds promise for enhancing quality assurance in other areas of journalism and science communication where similar quality criteria exist.