Uncovering the Influence of Large Language Models in Scientific Writing

Large language models (LLMs) have quickly become a staple in the field of artificial intelligence, revolutionizing the way we generate and analyze written content. However, detecting the use of LLMs in writing has posed a significant challenge for researchers and AI companies alike. Recent studies have shown that the introduction of LLMs has led to a drastic shift in vocabulary and writing style, particularly in scientific writing.

A group of researchers from Germany’s University of Tübingen and Northwestern University set out to examine the impact of LLMs on scientific writing by analyzing 14 million paper abstracts published on PubMed between 2010 and 2024. By tracking the frequency of specific words before and after the widespread adoption of LLMs in 2023, the researchers were able to identify significant changes in vocabulary usage.

One of the key findings of the study was the emergence of “excess words” that became more prevalent in scientific writing following the introduction of LLMs. Words like “delves,” “showcasing,” and “underscores” saw a sharp increase in frequency in abstracts from 2024, indicating a shift in writing style that was unprecedented in both quality and quantity.

The researchers identified hundreds of “marker words” that exhibited a significant increase in usage after the LLM era began. These marker words, which were predominantly verbs, adjectives, and adverbs, served as telltale signs of LLM usage in scientific writing. By comparing the frequency of these marker words in pre-2023 abstracts to post-2023 abstracts, the researchers were able to estimate that at least 10 percent of papers in the PubMed corpus were written with LLM assistance.

The findings of this study raise important questions about the evolving landscape of scientific writing in the age of LLMs. As LLMs continue to be integrated into the writing process, researchers and writers must be mindful of the impact these tools have on vocabulary choice and writing style. By understanding how LLMs influence writing, we can ensure that scientific communication remains clear, accurate, and effective in the years to come.