Software often produces biased outputs. In particular, machine learning (ML) based software is known to produce erroneous predictions when processing discriminatory inputs. Such unfair program behaviour can be caused by societal bias. In the last few years, Amazon, Microsoft and Google have provided software services that produce unfair outputs, mostly due to societal bias (e.g. gender or race). In such events, developers are saddled with the task of conducting fairness testing. Fairness testing is challenging; developers are tasked with generating discriminatory inputs that reveal and explain biases. We propose a grammar-based fairness testing approach (called ASTRAEA) which leverages context-free grammar to generate discriminatory inputs that reveal fairness violations in software systems. Using probabilistic grammar, ASTRAEA also provides fault diagnosis by isolating the cause of observed software bias. ASTRAEAs diagnoses facilitate the improvement of ML fairness. ASTRAEA was evaluated on 18 software systems that provide three major natural language processing (NLP) services. In our evaluation, ASTRAEA generated fairness violations at a rate of about 18%. ASTRAEA generated over 573K discriminatory test cases and found over 102K fairness violations. Furthermore, ASTRAEA improves software fairness by about 76% via model retraining, on average.
Tue 15 NovDisplayed time zone: Beijing, Chongqing, Hong Kong, Urumqi change
10:45 - 12:15 | |||
10:45 15mTalk | Context-Aware Code Change Embedding for Better Patch Correctness Assessment Journal First Bo Lin National University of Defense Technology, Shangwen Wang National University of Defense Technology, Ming Wen Huazhong University of Science and Technology, Xiaoguang Mao National University of Defense Technology Link to publication DOI Pre-print | ||
11:00 15mTalk | BiRD: Race Detection in Software Binaries under Relaxed Memory Models Journal First Ridhi Jain Indraprastha Institute of Information Technology Delhi, Rahul Purandare IIIT-Delhi, Subodh Sharma IIT Delhi Link to publication DOI | ||
11:15 15mTalk | ASTRAEA: Grammar-based Fairness Testing Journal First Ezekiel Soremekun SnT, University of Luxembourg, Sakshi Udeshi Singapore University of Technology and Design, Sudipta Chattopadhyay Singapore University of Technology and Design Link to publication DOI Pre-print | ||
11:30 15mTalk | Exploring Performance Assurance Practices and Challenges in Agile Software Development: An Ethnographic Study Journal First Luca Traini University of L'Aquila Link to publication DOI | ||
11:45 15mTalk | Studying logging practice in test code Journal First Haonan Zhang Concordia University, Yiming Tang Concordia University, Maxime Lamothe Polytechnique Montréal, Heng Li Polytechnique Montréal, Weiyi Shang Concordia University | ||
12:00 15mTalk | Locating Faults with Program Slicing: An Empirical Analysis Journal First Ezekiel Soremekun SnT, University of Luxembourg, Lukas Kirschner Saarland University, Marcel Böhme MPI-SP, Germany and Monash University, Australia, Andreas Zeller CISPA Helmholtz Center for Information Security Link to publication DOI Pre-print |