bei Google ScholarTitelangaben
Wenderoth, Johanna ; von Philipsborn, Peter ; Leibinger, Anna ; Cimiano, Philipp:
A systematic workflow for utilizing and evaluating large language models in public health policy framing analysis.
In: Archives of Public Health.
Bd. 84
(2026)
.
- 225.
ISSN 2049-3258
DOI: https://doi.org/10.1186/s13690-026-02089-6
Abstract
Background
For public health policy research, it is crucial to monitor policy debates to inform evidence-based policymaking. One popular method is to analyze the news for salient frames used to argue for or against a controversial topic. However, this method is time and resource-intensive since domain experts need to manually analyze relevant articles to identify frames. This study presents an approach for using large language models (LLMs) to extract frames and to systematically evaluate results.
Methods
Building on the results of a framing analysis of the German media debate about a tax on animal-based foods, we developed a systematic workflow to utilize LLMs to extract all instances of the 10 most frequent manually coded negative frames from 46 news articles. With one prompt per article in a few-shot setting we generated a list of quote/frame pairs. We then evaluated results against human-coded ones and calculated precision, recall, and F1-score.
Results
GPT-5 achieved 79.8% precision and 79.8% recall, identifying 75.2% of manually coded frames. Extracted quotes generally aligned with human-extracted text sections. Gemini 3.1 Pro Preview yielded higher precision but lower recall, resulting in a comparable F1-score (79.0% vs. 79.8% with GPT-5). These findings suggest the approach is generalizable across different frontier models.
Conclusions
This study introduces a workflow for LLM-assisted frame extraction and transparent evaluation against an expert reference standard. Integrating this approach into public health policy coding can save resources and accelerate analysis. Ultimately, this could increase the representativeness and timeliness of data, enhancing our understanding of ongoing public health policy debates.
Weitere Angaben
| Publikationsform: | Artikel in einer Zeitschrift |
|---|---|
| Begutachteter Beitrag: | Ja |
| Institutionen der Universität: | Fakultäten > Fakultät für Lebenswissenschaften: Lebensmittel, Ernährung und Gesundheit > Lehrstuhl Public-Health Nutrition > Lehrstuhl Public-Health Nutrition - Univ.-Prof. Dr. Peter von Philipsborn |
| Titel an der UBT entstanden: | Ja |
| Themengebiete aus DDC: | 600 Technik, Medizin, angewandte Wissenschaften 600 Technik, Medizin, angewandte Wissenschaften > 610 Medizin und Gesundheit |
| Eingestellt am: | 23 Okt 2026 06:25 |
| Letzte Änderung: | 23 Okt 2026 06:25 |
| URI: | https://eref.uni-bayreuth.de/id/eprint/100027 |