1. The biomedical literature challenge
What if the key to groundbreaking biomedical discoveries lies in how we summarize existing research?
Biomedical literature is one cornerstone of scientific research. Comparing what information is out there and how it is connected immensely helps drive computational analysis forward. Furthermore, it enables the creation of biological hypotheses that ultimately lead to new clinical approaches.
The most important aspect for in-silico companies like Delta4 is not only the identification and condensation, but also the clean integration of such literature information. In the past decade, the exponential growth of scientific literature has created a demand for text summarization methods. These methods ease the burden of manual review and support scientists in literature analysis, knowledge extraction, and the interpretation of research findings for downstream computational applications.
Automated text summarization methods have evolved from statistical approaches to deep learning-based models, thus becoming increasingly sophisticated and reliable in capturing the essential parts of complex research articles. Although these methods have been previously evaluated and described, only a few analyses have focused on scientific literature.
We saw a clear need for and high value in creating a guide through the jungle of approaches in the context of biomedical literature—traditional extractive methods, specialized biomedical models, general-purpose large language models (LLMs), and reasoning models.
Delta4’s daily reliance on literature analysis, coupled with the need for a rigorous comparison of summarization methods and the absence of standardized evaluations, motivated us to systematically compare a broad range of summarization approaches under the same conditions.
Our ultimate goal was to answer a critical question: Which approach produces summaries that are both accurate and complete?
2. The study: Benchmarking 62 summarization methods
Our dataset consisted of 1,000 biomedical abstracts from 20 journals. The respective author-written highlights were used as reference summaries, providing a useful human-generated benchmark.
Methods compared
The benchmark spans a wide range of approaches, from relatively simple statistical methods to modern language models —including general-purpose, domain-specific, and reasoning-oriented models. This enables a direct comparison of fundamentally different summarization strategies.
Large language models (LLM); n=21
Traditional/statistical extractive methods (including word-frequency and TextRank); n=2
Encoder-decoder models; n=11
Small language models (SLM); n=28
3. How do you measure a “good” scientific summary?
A strong biomedical summary should not merely reuse the source’s wording; it should preserve meaning and maintain factual accuracy. Summarization quality is multidimensional, therefore a multi-metric evaluation is essential.
To assess overall performance, we combined 12 evaluation methods across 3 categories, resulting in a rank that reflects the summary’s quality.
Lexical similarity (n=5)
Assesses how closely the generated summary corresponds to the reference at the word and phrase level.
- ROUGE-1/-2/-L
- BLEU
- METEOR
Semantic similarity (n=3)
Evaluates if the generated summary communicates the same meaning as the reference, even when different wording is used.
Models/metrics:
- RoBERTa
- DeBERTa
- all-mpnet-base-v2
Factual consistency (n=4)
Assesses if every claim in the summary is supported by the information in the source, penalizing hallucinations.
Metrics:
- AlignScore
- SummaC
- MiniCheck (FT5 and 7B variants)
The key message of our metrics is that using multiple complementary metrics prevents models from being judged by a single, potentially misleading definition of summary quality.
For example, our factual-consistency scores barely correlated with overall quality because these methods tend to reward summaries that closely reuse the source text (as extractive methods do). Thus, a high factual-consistency score may indicate a faithful summary—or it may simply reflect excessive copying.
4. Key finding #1: General-purpose language models perform best
In our study, general-purpose large language models (LLMs) achieved the strongest overall performance. They excelled across both lexical and semantic metrics and, notably, outperformed both reasoning-oriented and domain-specific models. The latter result was particularly surprising, as it suggests that domain-specific models may lack the broad linguistic and contextual understanding that general-purpose LLMs acquire through large-scale pretraining.
The superiority of general-purpose LMs overall was also observed in qualitative evaluation. In addition to the automated metrics, we evaluated the summaries using two complementary approaches: blinded domain-expert ratings and LLM-based judging, with results averaged across three providers. Summaries were assessed for coherence, fluency, relevance, and consistency. Across both evaluation approaches, general-purpose LMs ranked highest, ahead of both reasoning-oriented and domain-specific models.
In summary, broad pretraining appears to confer a significant advantage for scientific text summarization, even when compared to models specifically fine-tuned for biomedical content.
5. Key finding #2: Bigger is not necessarily better
Our analysis led to a clear conclusion: for biomedical text summarization, model selection should not default to the largest available model—especially when models are applied at scale across large literature collections, where cost and time become critical factors.
We found that frontier-scale models were not automatically the best-performing approaches. In fact, medium-sized general-purpose models frequently achieved better results. Many of these models can be deployed on a single compute node. This reveals a compelling trade-off between performance, computational efficiency, cost, and deployability.
6. Key finding #3: Traditional extractive approaches have fallen behind
This analysis provides valuable perspective on the evolution of biomedical text summarization. It demonstrates that statistical and extractive methods underperform compared to neural approaches. Their core limitation lies in their reliance on selecting existing sentences or phrases, rather than generating a condensed representation of the underlying meaning.
By 2026, methods once considered standard baselines may no longer meet the requirements for state-of-the-art text summarization. However, they are not obsolete: unlike generative models, they cannot introduce factual errors or invent content, though they may omit, misprioritize, or decontextualize information.
7. A surprising result: Domain-specific does not automatically mean better
Our key takeaway is that domain specialization alone does not guarantee superior performance. While one might expect biomedical or domain-specific models to have an inherent advantage in processing biomedical literature, our benchmark results did not support this assumption. General-purpose models outperformed their domain-specific counterparts, despite not being narrowly optimized for the biomedical domain.
Possible explanations for this include:
- The breadth and diversity of their pretraining data
- Stronger general language capabilities
- Model architecture and capacity
- A superior ability to capture context and relationships across scientific text
Additionally, fine-tuning a model on biomedical data does not only add domain knowledge—it can also lead to a decline in other general capabilities, such as instruction-following and text summarization, a phenomenon known as catastrophic forgetting.
8. What does this mean for biomedical research?
Reliable automated text summarization offers significant benefits in scenarios where human reviewers assess the output, such as screening large literature sets, generating hypotheses, systematic reviews, or competition scanning. However, it carries risks in cases where summaries are treated as final results without verification—such as feeding summary-derived information directly into automated pipelines, knowledge graphs or downstream models.
These methods enhance literature-intensive biomedical research by accelerating the process of reading and comprehension. They enable faster indexing of literature compared to manual review and allow researchers to search for papers based on their content and results, rather than just individual terms.
Yet, researchers must carefully consider their choice of model. One might assume that a more specialized, larger, or reasoning-capable model would perform better for summarization tasks. However, our results challenge this assumption: a biomedical model is not inherently superior for biomedical tasks, larger models do not guarantee better performance, and reasoning capabilities do not confer an advantage in text summarization.
Factual consistency and semantic fidelity are particularly critical in biomedical applications. In this field, even subtle changes in wording can entirely alter the meaning. For example, if a generated summary replaces “inhibits” with “reduces” or omits the distinction between results derived from cells versus patients, the meaning shifts dramatically. This is why factual consistency and semantic fidelity are indispensable: researchers must trust the accuracy of what they read. A fluent but subtly incorrect summary is far more hazardous than an obviously flawed one.
9. What are the implications for large-scale literature mining or knowledge extraction?
Before the LLM era, large-scale literature mining primarily relied on metadata extraction, named entity recognition, and relation extraction. However, much of the text—often containing critical contextual information—was discarded. Affordable and accurate text summarization now enables us to leverage this untapped content: instead of extracting only entities and relations, we can condense and aggregate the key findings of each publication across an entire body of evidence.
The real breakthrough emerges when analyzing the sum of these summaries rather than individual papers. Patterns appear that no single publication explicitly states and that traditional data structures cannot capture: recurring mechanisms, repeated caveats, and the broader context surrounding a finding rather than the finding in isolation. This shifts the focus from hard evidence and its ambiguities to revealing latent information.
This transition moves the bottleneck from extraction to curation—and this is where LLM-as-a-judge can play a pivotal role. In our benchmark, it tracked expert consensus as closely as an individual expert, which suggests it could serve as a first-pass filter for large volumes of generated summaries. While it does not replace human oversight, it efficiently directs curators to areas where their expertise is most needed.
Local or single-node models are particularly attractive due to their cost-effectiveness, scalability, reproducibility, and confidentiality. Reproducibility is especially critical for LLM-generated content, as closed API models can change or become deprecated without warning.
10. Relevance to Delta4
Literature processing and information extraction are closely connected to Delta4’s Hyper-KG knowledge graph. Scientific literature provides a vast source of biomedical information, but extracting information is only one part of the process: it must also be placed in the appropriate biological context and assessed for relevance and reliability.
This systematic evaluation of AI methods reflects Delta4’s broader approach of selecting computational tools based on empirical performance rather than preconceived assumptions about model size or specialization.
- From literature to structured information: Reliable extraction and summarization help transform unstructured publications into structured data that feeds Delta4’s computational approaches.
- Reliability at scale: Processing large volumes of biomedical literature reliably and efficiently is essential when extracted information contributes to the generation and evaluation of biological hypotheses. Catching errors early in the process helps prevent misinterpretations from propagating into downstream analyses.
11. Conclusion: Choosing the right model, not simply the biggest one
From our observation, modern language models clearly outperform traditional text summarization approaches in our benchmark. General-purpose models performed surprisingly strongly compared to specialized alternatives, whilst medium-sized models offer an attractive combination of quality and computational efficiency.
Systematic benchmarking, not model size or domain labels, should therefore guide the choice of AI tools for biomedical research.
About the Authors
Enrico Bono is an MSCA PhD student in the EU-funded PICKED (Personalized Medicine in Chronic Kidney Disease) project with a strong passion for machine learning (ML), artificial intelligence (AI), and multi-omics approaches for advancing precision medicine.
Fabio Baumgärtel earned his Bachelor’s degree in Bioinformatics from the University of Applied Sciences Upper Austria in Hagenberg (BSc, 2024) and joined Delta4 to contribute to the development of a graph visualization tool. For his Bachelor’s thesis, Fabio focused on the analysis and processing of biological data.
