|
所在平台: Udemy |
课程主页: https://www.udemy.com/course/ahol-dl4nlp6/
课程评论：没有评论
课程名称：深度学习在自然语言处理中的应用 - 第六部分 概述：本课程是“深度学习在自然语言处理（NLP）中的应用”系列的一部分。在本课程中，我将探讨在之前的课程中未覆盖的多种流行的变换器模型。这些变换器模型包括基于编码器和解码器的模型，从输入形式、预训练目标、预训练数据、架构变化等多个方面存在差异。这些模型均是在2019年后提出的，其中一些模型甚至来自2021年初。因此，截止到2021年8月，这些模型在多个NLP任务中都是最新的、最先进的。 课程主要包括三个部分： 第一部分：讨论几种扩展原始变换器框架的变换器编码器和解码器模型。具体包括SpanBERT、Electra、DeBERTa和DialoGPT。SpanBERT、Electra和DeBERTa为变换器编码器，而DialoGPT是变换器解码器模型。对每个模型，我们将讨论其架构或预训练相较于标准变换器的不同之处，以及在各类NLP任务中的重要结果。 第二部分：讨论多模态变换器模型。近年来，多模态学习得到了广泛关注，因此需要出现能够同时处理文本和图像数据的变换器模型。在这一部分，我们将介绍VisualBERT和vilBERT，这两个模型能够有效处理多模态输入，我们将详细讨论它们的相似性和差异。 第三部分：讨论大规模变换器模型。首先介绍混合专家（MoE）架构。接着讲解GShard如何适应MoE架构，并在大规模多语言机器翻译中取得优异效果。最后，将讨论Switch Transformers，这种模型简化了MoE路由算法，并进行多项工程优化，以降低网络通信和计算成本，缓解不稳定性。 总体而言，这些论文往往较长，因此理解起来较为困难并且耗时。在这些课程中，我试图用简明扼要的方式进行总结，提炼出直观的理解，并将不同论文中重要概念以连贯的故事进行关联。希望这对你的工作和理解有所帮助。
This course is a part of "Deep Learning for NLP" Series. In this course, I will talk about various popular Transformer models beyond the ones I have already covered in the previous sessions in this series. Such Transformer models including encoder as well as decoder based models and differ in terms of various aspects like form of input, pretraining objectives, pretraining data, architecture variations, etc.These Transformer models have been all proposed after 2019 and some of them are also from early 2021. Thus, as of Aug 2021, these models are very recent and state of the art across multiple NLP tasks.The course consists of three main sections as follows. In the first section, I will talk about a few Transformer encoder and decoder models which extend the original Transformer framework. Specifically I will cover SpanBERT, Electra, DeBERTa and DialoGPT. SpanBERT, Electra and DeBERTa are Transformer encoders while DialoGPT is a Transformer decoder model. For each model, we will also talk about their architecture or pretraining differs from standard Transformer. We will also talk important results on various NLP tasks.In the second section, I will talk about multi-modal Transformer models. Multimodal learning has gained a lot of momentum in recent years. Thus, there was a need to come up with Transformer models which could handle text and image data together. In this part, I will cover VisualBERT and vilBERT which both process the multi-modal input very effectively. Both the models have many similarities. We will discuss about theri similarities and differences in detail.Lastly, in the third section, I will talk about lareg scale Transformer models. I will introduce the mixture of experts (MoE) architecture. Then I will talk about how GShard adapts the MoE architecture, and shows great results on massive multilingual machine translation. Lastly, I will discuss Switch Transformers which simplify the MoE routing algorithm and also do several engineering optimizations to reduce network communciation and computation costs and mitigate instabilities.In general, each of these papers is pretty long and thus it becomes very difficult and time consuming to understand them. In these sessions, I have tried to summarize them nicely bringing out the intuitions and tying the important concepts across such papers in a coherent story. Hope you will find it useful for your work and understanding.
