• Posts
AI
• Infrastructure
  • Guides On Choosing Deep Learning Server
• LLM
  • Asksage
• PyTorch
  • Learning PyTorch Part I
  • Pytorch Distributed Data Parallel With Model Parallel in an HPC Environment
Tools
• NeoVim
• An Intro to a CLI Password Management: Pass
• Exercism Cli Shortcut
• Random Docker/Podman tips
HPC
• ALCF
  • Distributed Training
  • QWen2.5-VL
Linux
• Manage Users in Linux
• Setup Ubuntu 22.04
Embedded Systems
Programming
• C++
  • C++ Enum Pattern
• Competitive Programming
  • How to Learn Programming
  • Mistakes I Have Made
  • TopCoder
    • HoleCakeCuts topcoder SRM411 div2 level3
    • InfiniteSequence topcoder SRM413 div2 level3
    • StringsAndTabs topcoder SRM412 div2 level3
    • TeleportsNetwork topcoder SRM409 div2 level3
• Design Patterns
  • Object-Oriented Analysis
  • Object-Oriented Design Principles
• Python
  • Python Conditional Timeit Decorator

Pytorch Distributed Data Parallel With Model Parallel in an HPC Environment

Distributed Data Parallel with Model Parallel in an HPC environment Objective This tutorial is on : how to separate a model and put it on multiple GPUs. how to train such model in a distributed data parallel fashion. how to use torch.distributed.launch and create a slurm job script for HPC environment. Model Parallel (Pipelining) When a model is too large to fit in one GPU device, we can cut it in half and put each part on different GPU device. To do this, we need to partition the model into “head” and “tail” and specify which device to put them on. In the following toy example, we simply put the first part in to current GPU device and the second part to the next device.

• Documentation

Thursday, December 12, 2019 | 5 minutes Read

Learning PyTorch Part I

Introduction Currently, I am participating the deep learning part1v2 course as an “international fellow”. This course is taught by Jeremy Howard from fast.ai. The course is not available to the public yet, but it will be in future. During the course, Jeremy introduced PyTorch and the fastai package built on top of PyTorch. Before this, I only used Tensorflow and Keras. PyTorch is quite different (in a good way). I am very impressed by the elegant and flexible design of PyTorch. I would like to introduce some features I think interesting.

Monday, November 13, 2017 | 3 minutes Read