Watch and track your favorite playlist.
Curated by: USENIX (112 videos)
NSDI '24 - Characterization of Large Language Model Development in the Datacenter Qinghao Hu, Shanghai AI Laboratory and S-Lab, Nanyang Technological University; Zhisheng Ye, Shanghai AI Laboratory and Peking University; Zerui Wang, Shanghai AI Laboratory and Shanghai Jiao Tong University; Guoteng Wang, Shanghai AI Laboratory; Meng Zhang and Qiaoling Chen, Shanghai AI Laboratory and S-Lab, Nanyang Technological University; Peng Sun, Shanghai AI Laboratory and SenseTime Research; Dahua Lin, Shanghai AI Laboratory and CUHK; Xiaolin Wang and Yingwei Luo, Peking University; Yonggang Wen and Tianwei Zhang, Nanyang Technological University Large Language Models (LLMs) have presented impressive performance across several transformative tasks. However, it is non-trivial to efficiently utilize large-scale cluster resources to develop LLMs, often riddled with numerous challenges such as frequent hardware failures, intricate parallelization strategies, and imbalanced resource utilization. In this paper, we present an in-depth characterization study of a six-month LLM development workload trace collected from our GPU datacenter Acme. Specifically, we investigate discrepancies between LLMs and prior task-specific Deep Learning (DL) workloads, explore resource utilization patterns, and identify the impact of various job failures. Our analysis summarizes hurdles we encountered and uncovers potential opportunities to optimize systems tailored for LLMs. Furthermore, we introduce our system efforts: (1) fault-tolerant pretraining, which enhances fault tolerance through LLM-involved failure diagnosis and automatic recovery. (2) decoupled scheduling for evaluation, which achieves timely performance feedback via trial decomposition and scheduling optimization. View the full NSDI '24 program at https://www.usenix.org/conference/nsdi24/technical-sessionsa