Understanding Performance Problems in Deep Learning Systems
Deep learning (DL) has been widely applied to many domains. Unique challenges in engineering DL systems are posed by the programming paradigm shift from traditional systems to DL systems, and performance is one of the challenges. Performance problems (PPs) in DL systems can cause severe consequences such as excessive resource consumption and financial loss. While bugs in DL systems have been extensively investigated, PPs in DL systems have hardly been explored. To bridge this gap, we present the first comprehensive study to i) characterize symptoms, root causes, and introducing and exposing stages of PPs in DL systems developed in \textsc{TensorFLow} and \textsc{Keras}, with 224 PPs collected from 210 StackOverflow posts, and to ii) assess the capability of existing performance analysis approaches in tackling PPs, with a constructed benchmark of 58 PPs in DL systems. Our findings shed light on the implications on developing high-performance DL systems, and detecting and localizing PPs in DL systems. To demonstrate the usefulness of our findings, we develop a static checker DeepPerf to detect three types of PPs. It has detected 488 new PPs in 130 GitHub projects. 105 and 27 PPs have been confirmed and fixed.
Tue 15 NovDisplayed time zone: Beijing, Chongqing, Hong Kong, Urumqi change
10:45 - 12:15 | Machine Learning IIResearch Papers / Ideas, Visions and Reflections / Industry Paper at SRC Auditorium 2 Chair(s): Atif Memon Apple | ||
10:45 15mTalk | Understanding Performance Problems in Deep Learning Systems Research Papers Junming Cao Fudan University, Bihuan Chen Fudan University, Chao Sun Fudan University, Longjie Hu Fudan University, Shuaihong Wu Fudan University, Xin Peng Fudan University DOI | ||
11:00 15mTalk | API Recommendation for Machine Learning Libraries: How Far Are We? Research Papers Moshi Wei York University, Yuchao Huang Institute of Software at Chinese Academy of Sciences, Junjie Wang Institute of Software at Chinese Academy of Sciences; University of Chinese Academy of Sciences, Jiho Shin York University, Nima Shiri Harzevili York University, Song Wang York University DOI Pre-print | ||
11:15 15mTalk | No More Fine-Tuning? An Experimental Evaluation of Prompt Tuning in Code Intelligence Research Papers Chaozheng Wang Harbin Institute of Technology, Yuanhang Yang Harbin Institute of Technology, Cuiyun Gao Harbin Institute of Technology, Yun Peng Chinese University of Hong Kong, Hongyu Zhang University of Newcastle, Michael Lyu Chinese University of Hong Kong DOI | ||
11:30 15mTalk | Improving ML-Based Information Retrieval Software with User-Driven Functional Testing and Defect Class Analysis Industry Paper DOI | ||
11:45 15mTalk | Discrepancies among Pre-trained Deep Neural Networks: A New Threat to Model Zoo Reliability Ideas, Visions and Reflections Diego Montes Purdue University, Pongpatapee Peerapatanapokin Purdue University, Jeff Schultz Purdue University, Chengjun Guo Purdue University, Wenxin Jiang Purdue University, James C. Davis Purdue University DOI | ||