Write a Blog >>
ESEC/FSE 2022
Mon 14 - Fri 18 November 2022 Singapore
Tue 15 Nov 2022 10:45 - 11:00 at SRC LT 51 - Empirical II Chair(s): Lwin Khin Shar

We describe a large-scale study to reduce the amount of time code review takes. Each quarter at Meta we survey developers. Combining sentiment data from a developer experience survey and telemetry data from our diff review tool, we address, ``When does a diff review feel too slow?'' From the sentiment data alone, we learn that 84.7% of developers are satisfied with the time their diffs spend in review. By enriching the survey results with telemetry for each respondent, we determined that sentiment is closely associated with the 75th percentile time in review for that respondent's diffs, ie those that take more than 24 hours.

To encourage developers to act on stale diffs that have had no action for 24 or more hours, we designed a NudgeBot to notify, ie nudge, reviewers. To determine who to nudge when a diff is stale, we created a model to rank the reviewers based on the probability that they will make a comment or perform some other action on a diff. This model outperformed models that looked at files the reviewer had modified in the past. Combining this information with prior author-review relationships, we achieved an MRR and AUC of .81 and .88, respectively.

To evaluate NudgeBot in production, we conducted an A/B cluster-randomized experiment on over 30k engineers. We observed substantial statistically significant decrease in both time in review (-6.8%, p=0.049) and time to first reviewer action (-9.9%, p=0.010). We also used guard metrics to ensure that most reviews were still done in fewer than 24 hours and that reviewers still spend the same amount of time looking at diffs, and saw {\it no} statistically significant change in these metrics.
NudgeBot is now rolled out company wide and is used daily by thousands of engineers at Meta.

Tue 15 Nov

Displayed time zone: Beijing, Chongqing, Hong Kong, Urumqi change

10:45 - 12:30
Empirical IIIndustry Paper / Research Papers / Demonstrations at SRC LT 51
Chair(s): Lwin Khin Shar Singapore Management University
10:45
15m
Talk
Using Nudges to Accelerate Code Reviews at ScaleDistinguished Paper Award
Research Papers
Qianhua Shan Meta, David Sukhdeo Meta, Qianying Huang Meta, Seth Rogers Meta, Lawrence Chen Meta, Elise Paradis Meta, Peter Rigby Concordia University; Meta, Nachiappan Nagappan Meta
DOI
11:00
15m
Talk
First Come First Served: The Impact of File Position on Code ReviewDistinguished Paper Award
Research Papers
Enrico Fregnan University of Zurich, Larissa Braz University of Zurich, Marco D'Ambros USI Lugano, Gül Calikli University of Glasgow, Alberto Bacchelli University of Zurich
DOI Pre-print Media Attached
11:15
15m
Talk
Code, Quality, and Process Metrics in Graduated and Retired ASFI Projects
Research Papers
Ştefan Stănciulescu University of California at Davis, Likang Yin University of California at Davis, Vladimir Filkov University of California at Davis
DOI Pre-print Media Attached
11:30
15m
Talk
Are Elevator Software Robust against Uncertainties? Results and Experiences from an Industrial Case Study
Industry Paper
Liping Han Nanjing University of Aeronautics and Astronautics; Simula Research Laboratory, Tao Yue Simula Research Laboratory, Shaukat Ali Simula Research Laboratory, Aitor Arrieta Mondragon University, Maite Arratibel Orona
DOI
11:45
15m
Talk
CommentFinder: A Simpler, Faster, More Accurate Code Review Comments Recommendation
Research Papers
Yang Hong Monash University, Chakkrit Tantithamthavorn Monash University, Patanamon Thongtanunam University of Melbourne, Aldeida Aleti Monash University
DOI
12:00
7m
Talk
TAPHSIR: Towards AnaPHoric Ambiguity Detection and ReSolution In Requirements
Demonstrations
Saad Ezzini University of Luxembourg, Sallam Abualhaija University of Luxembourg, Chetan Arora Deakin University, Mehrdad Sabetzadeh University of Ottawa
12:08
7m
Talk
COREQQA - A COmpliance REQuirements Understanding using Question Answering Tool
Demonstrations
Sallam Abualhaija University of Luxembourg, Chetan Arora Deakin University, Lionel Briand University of Luxembourg; University of Ottawa