Asplos 23 Session 7a Optimus Cc Efficient Large Nlp Model Training

Asplos 20 Session 6b Learning Based Memory Allocation For C In training of modern large natural language processing (nlp) models, it has become a common practice to split models using 3d parallelism to multiple gpus. such technique, however, suffers from a high overhead of inter node communication. compressing the communication is one way to mitigate the overhead by reducing the inter node traffic volume; however, the existing compression techniques. Asplos'23: the 28th international conference on architectural support for programming languages and operating systemssession 7a: deep learning systemssessio.

Asplos 23 Session 7a Optimus Cc Efficient Large Nlp Model Training In this paper, we present optimus cc, a fast and scalable distributed training framework for large nlp models with aggressive communication compression. optimus cc differs from existing communication compression frameworks in the following ways: first, we compress pipeline parallel (inter stage) traffic. Optimus cc: efficient large nlp model training with 3d parallelism aware communication compression jaeyong song [email protected] yonsei university seoul, south korea jinkyu yim∗ [email protected] seoul national university seoul, south korea jaewon jung [email protected] yonsei university seoul, south korea hongsun jang hongsun. [asplos'23] optimus cc: efficient large nlp model training with 3d parallelism aware communication compression machinelearningsystem optimus cc. In this work, we proposed optimus cc, which compresses the communications of large, distributed nlp models that utilize 3d parallelism. because the conventional communication compression algorithms fail to exploit pipeline related opportunities and result in a model quality drop, we proposed multiple techniques that reduce the amount of communications while maintaining the model quality.