TY - GEN
T1 - How Do So ware Developers Use ChatGPT? An Exploratory Study on GitHub Pull Requests
AU - Chouchen, Moataz
AU - Bessghaier, Narjes
AU - Begoug, Mahi
AU - Ouni, Ali
AU - Alomar, Eman Abdullah
AU - Wiem Mkaouer, Mohamed
N1 - Publisher Copyright:
© 2024 ACM.
PY - 2024
Y1 - 2024
N2 - Nowadays, Large Language Models (LLMs) play a pivotal role in software engineering. Developers can use LLMs to address software development-related tasks such as documentation, code refactoring, debugging, and testing. ChatGPT, released by OpenAI, has become the most prominent LLM. In particular, ChatGPT is a cutting-edge tool for providing recommendations and solutions for developers in their pull requests (PRs). However, little is known about the characteristics of PRs that incorporate ChatGPT compared to those without it and what developers usually use it for. To this end, we quantitatively analyzed 243 PRs that listed at least one ChatGPT prompt against a representative sample of 384 PRs without any ChatGPT prompts. Our findings show that developers use ChatGPT in larger, time-consuming pull requests that are five times slower to be closed than PRs that do not use ChatGPT. Furthermore, we perform a qualitative analysis to build a taxonomy of the topics developers primarily address in their prompts. Our analysis results in a taxonomy comprising 8 topics and 32 sub-topics. Our findings highlight that ChatGPT is often used in review-intensive pull requests. Moreover, our taxonomy enriches our understanding of the developer's current applications of ChatGPT.CCS CONCEPTS• Software and its engineering → Collaboration in software development.
AB - Nowadays, Large Language Models (LLMs) play a pivotal role in software engineering. Developers can use LLMs to address software development-related tasks such as documentation, code refactoring, debugging, and testing. ChatGPT, released by OpenAI, has become the most prominent LLM. In particular, ChatGPT is a cutting-edge tool for providing recommendations and solutions for developers in their pull requests (PRs). However, little is known about the characteristics of PRs that incorporate ChatGPT compared to those without it and what developers usually use it for. To this end, we quantitatively analyzed 243 PRs that listed at least one ChatGPT prompt against a representative sample of 384 PRs without any ChatGPT prompts. Our findings show that developers use ChatGPT in larger, time-consuming pull requests that are five times slower to be closed than PRs that do not use ChatGPT. Furthermore, we perform a qualitative analysis to build a taxonomy of the topics developers primarily address in their prompts. Our analysis results in a taxonomy comprising 8 topics and 32 sub-topics. Our findings highlight that ChatGPT is often used in review-intensive pull requests. Moreover, our taxonomy enriches our understanding of the developer's current applications of ChatGPT.CCS CONCEPTS• Software and its engineering → Collaboration in software development.
KW - ChatGPT
KW - Large Language Models
KW - Manual analysis
KW - Mining Software Repositories
KW - Pull Requests
UR - http://www.scopus.com/inward/record.url?scp=85197444292&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85197444292&partnerID=8YFLogxK
U2 - 10.1145/3643991.3645084
DO - 10.1145/3643991.3645084
M3 - Conference contribution
AN - SCOPUS:85197444292
T3 - Proceedings - 2024 IEEE/ACM 21st International Conference on Mining Software Repositories, MSR 2024
SP - 212
EP - 216
BT - Proceedings - 2024 IEEE/ACM 21st International Conference on Mining Software Repositories, MSR 2024
T2 - 21st IEEE/ACM International Conference on Mining Software Repositories, MSR 2024
Y2 - 15 April 2024 through 16 April 2024
ER -