Fuzzing Deep-Learning Libraries via Automated Relational API Inference
Deep Learning (DL) has gained wide attention in recent years. Meanwhile, bugs in DL systems can lead to serious consequences, and may even threaten human lives. As a result, a growing body of research has been dedicated to DL model testing. However, there is still limited work on testing DL libraries, e.g., PyTorch and TensorFlow, which serve as the foundations for building, training, and running DL models. Prior work on fuzzing DL libraries can only generate tests for APIs which have been invoked by documentation examples, developer tests, or DL models, leaving a large number of APIs untested. In this paper, we propose DeepREL, the first approach to automatically inferring relational APIs for more effective DL library fuzzing. Our basic hypothesis is that for a DL library under test, there may exist a number of APIs sharing similar input parameters and outputs; in this way, we can easily ``borrow'' test inputs from invoked APIs to test other relational APIs. Furthermore, we formalize the notion of value equivalence and status equivalence for relational APIs to serve as the oracle for effective bug finding. We have implemented DeepREL as a fully automated end-to-end relational API inference and fuzzing technique for DL libraries, which 1) automatically infers potential API relations based on API syntactic/semantic information, 2) synthesizes concrete test programs for invoking relational APIs, 3) validates the inferred relational APIs via representative test inputs, and finally 4) performs fuzzing on the verified relational APIs to find potential inconsistencies. Our evaluation on two of the most popular DL libraries, PyTorch and TensorFlow, demonstrates that DeepREL can cover 157% more APIs than state-of-the-art FreeFuzz. To date, DeepREL has detected 162 bugs in total, with 106 already confirmed by the developers as previously unknown bugs. Surprisingly, DeepREL has detected 13.5% of the high-priority bugs for the entire PyTorch issue-tracking system in a three-month period. Also, besides the 162 code bugs, we have also detected 14 documentation bugs (all confirmed).
Mon 14 NovDisplayed time zone: Beijing, Chongqing, Hong Kong, Urumqi change
11:00 - 12:30 | |||
11:00 15mTalk | Testing of Autonomous Driving Systems: Where Are We and Where Should We Go? Research Papers Guannan Lou Macquarie University, Yao Deng Macquarie University, Xi Zheng Macquarie University, Mengshi Zhang Meta, Tianyi Zhang Purdue University DOI | ||
11:15 15mTalk | Fuzzing Deep-Learning Libraries via Automated Relational API Inference Research Papers Yinlin Deng University of Illinois at Urbana-Champaign, Chenyuan Yang University of Illinois at Urbana-Champaign, Anjiang Wei Stanford University, Lingming Zhang University of Illinois at Urbana-Champaign DOI | ||
11:30 15mTalk | Perfect Is the Enemy of Test Oracle Research Papers Ali Reza Ibrahimzada University of Illinois Urbana-Champaign, Yigit Varli Middle East Technical University, Dilara Tekinoglu University of Massachusetts at Amherst, Reyhaneh Jabbarvand University of Illinois at Urbana-Champaign DOI Pre-print Media Attached | ||
11:45 15mTalk | Scenario-Based Test Reduction and Prioritization for Multi-Module Autonomous Driving Systems Research Papers Yao Deng Macquarie University, Xi Zheng Macquarie University, Mengshi Zhang Meta, Guannan Lou Macquarie University, Tianyi Zhang Purdue University DOI | ||