Assessing the Quality of GitHub Copilot’s Code Generation
The introduction of GitHub’s new code generation tool, GitHub Copilot, seems to be the first well-established instance of an AI pair-programmer. GitHub Copilot has access to a large amount of open-source projects, enabling it to utilize more extensive code in various programming languages than other code generation tools. Although the initial and informal assessments are promising, a systematic evaluation is needed to explore the limits and benefits of GitHub Copilot. The main objective of this study is to assess the quality of generated code provided by GitHub Copilot. We also aim to evaluate the impact of the quality and variety of input parameters fed to GitHub Copilot. To achieve this aim, we created an experimental setup for evaluating the generated code in terms of validity, correctness, and efficiency. The results suggest that GitHub Copilot was able to generate valid code with a 91.5% success rate. In terms of code correctness, out of 164 problems, 28.7% were correct, while 51.2% were partially correct, and 20.1% were incorrectly generated. Our empirical analysis shows that GitHub Copilot is a promising tool based on the results we obtained, however further and more comprehensive assessment is needed in the future.
Fri 18 NovDisplayed time zone: Beijing, Chongqing, Hong Kong, Urumqi change
11:00 - 12:30 | |||
11:00 20mResearch paper | On the Effectiveness of Data Balancing Techniques in the Context of ML-based Test Case Prioritization PROMISE Jediael Mendoza , Jason Mycroft , Lyam Milbury , Nafiseh Kahani University of Carlton, Jason Jaskolka | ||
11:20 20mResearch paper | Profiling Developers to Predict Vulnerable Code Changes PROMISE Tugce Coskun Istanbul Technical University, Rusen Halepmollasi Istanbul Technical University, Khadija Hanifi Ericsson, Ramin Fadaei Fouladi Ericsson, Pinar Comak De Cnudde Ericsson, Ayse Tosun Istanbul Technical University | ||
11:40 20mResearch paper | Assessing the Quality of GitHub Copilot’s Code Generation PROMISE |