Write a Blog >>
ESEC/FSE 2022
Mon 14 - Fri 18 November 2022 Singapore
Fri 18 Nov 2022 11:40 - 12:00 at Town Plaza GLR - Software Quality

The introduction of GitHub’s new code generation tool, GitHub Copilot, seems to be the first well-established instance of an AI pair-programmer. GitHub Copilot has access to a large amount of open-source projects, enabling it to utilize more extensive code in various programming languages than other code generation tools. Although the initial and informal assessments are promising, a systematic evaluation is needed to explore the limits and benefits of GitHub Copilot. The main objective of this study is to assess the quality of generated code provided by GitHub Copilot. We also aim to evaluate the impact of the quality and variety of input parameters fed to GitHub Copilot. To achieve this aim, we created an experimental setup for evaluating the generated code in terms of validity, correctness, and efficiency. The results suggest that GitHub Copilot was able to generate valid code with a 91.5% success rate. In terms of code correctness, out of 164 problems, 28.7% were correct, while 51.2% were partially correct, and 20.1% were incorrectly generated. Our empirical analysis shows that GitHub Copilot is a promising tool based on the results we obtained, however further and more comprehensive assessment is needed in the future.

Fri 18 Nov

Displayed time zone: Beijing, Chongqing, Hong Kong, Urumqi change

11:00 - 12:30
Software QualityPROMISE at Town Plaza GLR
11:00
20m
Research paper
On the Effectiveness of Data Balancing Techniques in the Context of ML-based Test Case Prioritization
PROMISE
11:20
20m
Research paper
Profiling Developers to Predict Vulnerable Code Changes
PROMISE
Tugce Coskun Istanbul Technical University, Rusen Halepmollasi Istanbul Technical University, Khadija Hanifi Ericsson, Ramin Fadaei Fouladi Ericsson, Pinar Comak De Cnudde Ericsson, Ayse Tosun Istanbul Technical University
11:40
20m
Research paper
Assessing the Quality of GitHub Copilot’s Code Generation
PROMISE
Burak Yetistiren Bilkent University, Isik Ozsoy Bilkent University, Eray Tüzün Bilkent University