|
所在平台: Udemy |
课程主页: https://www.udemy.com/course/ahol-dl4nlp5/
课程评论：没有评论
课程名称：自然语言处理的深度学习 - 第五部分 概述：本课程是“自然语言处理的深度学习”系列的一部分，主要讨论高效Transformer模型的设计方案。这些技术对于学术界和工业界的参与者都非常实用。在工业应用中，Transformer模型在许多自然语言处理任务中展现出极高的准确性，但其内存和计算复杂度是平方级的，使得实际应用困难重重。因此，本课程专注于提升Transformer模型效率的方法，对于希望将Transformer模型应用于产品中的任何人来说都至关重要。 在Transformer中，时间和激活内存随着序列长度的增长而呈平方增加。这是因为在每一层中，每个注意力头都试图通过“关注”其他位置的token来为每个位置生成转化表示。平方复杂度意味着最大输入大小在实践中是有限的，因此我们无法通过将长文档作为输入来提取语义表示。在本模块中，我们将探讨解决这一挑战的方法。 本课程分为两个主要部分。在第一部分中，我们将讨论高效的Transformer模型，包括Star Transformers、Sparse Transformers、Reformer、Longformer、Linformer和Synthesizer等方法。在第二部分中，我们将讨论ETC（扩展Transformer构建）、Big bird、线性注意力Transformer、Performer、Sparse Sinkhorn Transformer和Routing transformers等技术。我们将介绍Long Range Arena，这是一项用于评估模型在长序列任务上的准确性、内存使用和推理时间的最新基准。最后，我们将进行高效Transformer方法的哲学分类。 对于每种方法，我们将讨论特定的优化方案、架构以及在预训练和下游任务中获得的结果。
This course is a part of "Deep Learning for NLP" Series. In this course, I will talk about various design schemes for efficient Transformer models. These techniques will come in very handy for academic as well as industry participants. For industry use cases, Transformer models have been shown to lead to very high accuracy values across many NLP tasks. But they have quadratic memory as well as computational complexity making it very difficult to ship them. Thus, this course which focuses on methods to make Transformers efficient is very critical for anyone who wants to ship Transformer models as part of their products.Time and activation memory in Transformers grows quadratically with the sequence length. This is because in every layer, every attention head attempts to come up with a transformed representation for every position by "paying attention" to tokens at every other position. Quadratic complexity implies that practically the maximum input size is rather limited. Thus, we cannot extract semantic representation for long documents by passing them as input to Transformers. Hence, in this module we will talk about methods to address this challenge.The course consists of two main sections as follows. In the two sections, I will talk about Efficient Transformer Models, Efficient Transformer benchmark and a Comparison of various efficient Transformer methods. In the first section, I will talk about methods like Star Transformers, Sparse Transformers, Reformer, Longformer, Linformer, Synthesizer.In the second section, I will talk about methods like ETC (Extended Transformer Construction), Big bird, Linear attention Transformer, Performer, Sparse Sinkhorn Transformer, Routing transformers. Long Range Arena is a recent benchmark for evaluating models on long sequence tasks with respect to accuracy, memory usage and inference time. We will discuss details about long range arena and finally wrap up with a philosophical categorization of various efficient Transformer methods.For each method, we will discuss specific scheme for optimization, architecture and results obtained for pretraining as well as downstream tasks.
