|
所在平台: Udemy |
课程主页: https://www.udemy.com/course/distributed-deep-learning-with-jax-large-scale-training/
课程评论：没有评论
课程名称：AI 分布式训练基础 - 在多 GPU 上训练 LLMs 课程概述：本课程全面介绍了分布式训练 GPT 风格语言模型的艺术与科学，使用 JAX 和 XLA。学习如何从零开始扩展自回归变换器的训练，实施顶级 AI 实验室用于高效训练大语言模型的相同技术。 学习内容： - 分布式训练基础知识 - 了解语言模型的分布式计算 - JAX 的转换和并行化原理 - XLA（加速线性代数）编译优化 - 掌握集体通信原理，如： - All-reduce 用于梯度聚合 - All-gather 用于分布式数据收集 - Broadcast 用于参数同步 - Scatter/Gather 用于高效的标记分配 高级并行化策略： - 张量并行性：拆分注意力头和 MLP 层 - 管道并行性：优化变换器层分布 - 序列并行性：高效处理长序列 - 数据并行性：扩展批处理 - GPT 模型特定的混合并行方法 - GSPMD 和自动分片 - 为 GPT 模型组件实施 GSPMD - 注意力层的自动分片策略 - 优化自回归模型的通信模式 - 自定义变换器块的分片注解 实践实施： - 从零开始构建分布式 GPT 架构 - 高效的参数分片和同步管理 - 处理分布式注意力计算 - 管理分布式环境中的因果掩码 - 实施高效的参数服务器 性能优化与调试： - 分析分布式 GPT 训练的性能 - 优化注意力计算 - 实现梯度检查点 - 内存优化技术 - 处理分布式设置中的数值稳定性 动手项目： - 从零开始实现一个迷你 GPT，并进行分布式训练 - 为 GPT 训练构建混合并行策略 - 使用 GSPMD 创建自定义分片策略 - 优化变换器块的集体通信模式 - 在多个节点上部署和管理分布式 GPT 训练 适合人群： - 从事语言模型的机器学习工程师 - 扩展 GPT 风格模型的深度学习研究人员 - 转型为大规模机器学习的软件工程师 - 任何对训练大型语言模型感兴趣的人 先决条件： - 扎实的 Python 编程能力 - 对变换器架构的基本了解 - 熟悉语言建模概念 - 基本的线性代数和微积分知识 - 具备任意深度学习框架的经验 完成本课程后，您将了解如何在分布式系统中实施和扩展 GPT 模型训练，掌握大语言模型特有的并行训练策略、通信原理和优化技术的细微之处。
Master the art and science of distributed training for GPT-style language models in this comprehensive course using JAX and XLA. Learn how to scale up autoregressive transformer training from scratch, implementing the same techniques used by leading AI labs to train large language models efficiently across multiple devices.What You'll Learn:Distributed Training FundamentalsUnderstanding distributed computing for language modelsJAX's transformation and parallelization primitivesXLA (Accelerated Linear Algebra) compilation optimizationMastering collective communication primitives: • All-reduce for gradient aggregation • All-gather for distributed data collection • Broadcast for parameter synchronization • Scatter/Gather for efficient token distributionAdvanced Parallelization Strategies for GPTTensor Parallelism: Splitting attention heads and MLP layersPipeline Parallelism: Optimal transformer layer distributionSequence Parallelism: Handling long sequences efficientlyData Parallelism: Scaling batch processingHybrid parallelism approaches specific to GPT modelsGSPMD and Automated ShardingImplementing GSPMD for GPT model componentsAutomatic sharding strategies for attention layersOptimizing communication patterns for autoregressive modelsCustom sharding annotations for transformer blocksPractical ImplementationBuilding distributed GPT architectures from scratchEfficient parameter sharding and synchronizationHandling distributed attention computationManaging causal masks in distributed settingsImplementing efficient parameter serversPerformance Optimization and DebuggingProfiling distributed GPT trainingOptimizing attention computationImplementation of gradient checkpointingMemory optimization techniquesHandling numerical stability in distributed settingsHands-on Projects:Implement a mini-GPT from scratch with distributed trainingBuild hybrid parallelism strategies for GPT trainingCreate custom sharding strategies using GSPMDOptimize collective communication patterns for transformer blocksDeploy and manage distributed GPT training across multiple nodesWho This Course is For:Machine Learning Engineers working on language modelsDeep Learning Researchers scaling up GPT-style modelsSoftware Engineers transitioning to large-scale MLAnyone interested in training large language modelsPrerequisites:Strong Python programming skillsBasic understanding of transformer architecturesFamiliarity with language modeling conceptsBasic linear algebra and calculusExperience with any deep learning frameworkBy the end of this course, you'll understand how to implement and scale GPT model training across distributed systems. You'll master the intricacies of parallel training strategies, communication primitives, and optimization techniques specific to large language models.
