Libraries & Librarians

Research Paper (preprint): Let’s Use ChatGPT to Write Our Paper! Benchmarking LLMs to Write the Introduction of a Research Paper

DrWeb

August 22, 2025

The research paper (preprint) linked below was recently posted on arXiv.

Title

Let’s Use ChatGPT to Write Our Paper! Benchmarking LLMs to Write the Introduction of a Research Paper

Authors

Krishna Garg
University of Illinois Chicago

Firoz Shaikh
University of Illinois Chicago

Sambaran Bandyopadhyay
Adobe Research

Cornelia Caragea
University of Illinois Chicago

Source

via arXiv

DOI: 10.48550/arXiv.2508.14273

Abstract

As researchers increasingly adopt LLMs as writing assistants, generating high-quality research paper introductions remains both challenging and essential. We introduce Scientific Introduction Generation (SciIG), a task that evaluates LLMs’ ability to produce coherent introductions from titles, abstracts, and related works. Curating new datasets from NAACL 2025 and ICLR 2025 papers, we assess five state-of-the-art models, including both open-source (DeepSeek-v3, Gemma-3-12B, LLaMA 4-Maverick, MistralAI Small 3.1) and closed-source GPT-4o systems, across multiple dimensions: lexical overlap, semantic similarity, content coverage, faithfulness, consistency, citation correctness, and narrative quality. Our comprehensive framework combines automated metrics with LLM-as-a-judge evaluations. Results demonstrate LLaMA-4 Maverick’s superior performance on most metrics, particularly in semantic similarity and faithfulness. Moreover, three-shot prompting consistently outperforms fewer-shot approaches. These findings provide practical insights into developing effective research writing assistants and set realistic expectations for LLM-assisted academic writing. To foster reproducibility and future research, we will publicly release all code and datasets.