|
所在平台: Udemy |
课程主页: https://www.udemy.com/course/mastering-aws-elastic-map-reduce-for-data-engineers/
课程评论：没有评论
**课程名称：** Mastering AWS Elastic Map Reduce (EMR) for Data Engineers **课程概述：** 本课程专注于AWS Elastic Map Reduce (EMR) 服务，这是构建大规模数据处理解决方案的关键AWS服务之一，广泛应用于Hadoop、Spark、Hive等大数据技术。通过本课程的学习，您将掌握使用Apache Spark和AWS Step Functions构建端到端数据处理流水线。 **课程内容详情：** 1. **AWS EMR 入门：** * 使用AWS Web Console 创建和管理EMR集群。 * 了解Web Console 的各项核心功能。 * 连接集群主节点，验证spark-shell, pyspark, hive等CLI接口，以及hdfs和aws CLI命令。 2. **设置开发集群：** * 学习使用AWS EMR 设置开发集群的优势及实践。 3. **Spark 应用开发生命周期：** * 在AWS EMR 开发集群上进行Spark应用程序的开发。 * 利用Visual Studio Code 进行远程开发。 4. **部署Spark 应用：** * 将Spark应用程序打包成zip文件。 * 学习在客户端和集群部署模式下使用CLI运行Spark应用程序。 * 了解如何将Spark应用程序作为步骤部署到EMR集群。 * 学习如何通过分析相关日志来故障排除Spark应用程序问题。 5. **使用Python Boto3 管理EMR 集群：** * 学习使用Python Boto3 以编程方式创建EMR集群。 * 学习使用Python Boto3 以编程方式将Spark应用程序部署为步骤。 6. **构建EMR 数据处理流水线：** * 使用AWS Step Functions 构建基于EMR的工作流或流水线。 * 学习如何创建一个包含创建集群、部署Spark应用程序作为步骤以及终止集群的基本流水线（状态机）。 * **Enhancing AWS EMR-based State Machine/Pipeline：** 在状态机中添加验证步骤，例如检查指定文件是否存在。 7. **使用Spark SQL on EMR 进行数据处理：** * 学习如何使用Spark SQL 脚本设计和开发解决方案。 * 学习通过传递运行时参数来验证Spark SQL 脚本。 8. **部署Spark SQL 数据处理流水线：** * 使用AWS Step Functions 部署Spark SQL 脚本到EMR集群。 * 了解Boto3 Waiters 的概念，确保步骤按顺序执行。
AWS Elastic Map Reduce (EMR) is one of the key AWS Services used in building large-scale data processing leveraging Big Data Technologies such as Apache Hadoop, Apache Spark, Hive, etc. As part of this course, you will end up learning AWS Elastic Map Reduce (EMR) by building end-to-end data pipelines leveraging Apache Spark and AWS Step Functions.Here is the detailed outline of the course.First, you will learn how to Get Started with AWS Elastic Map Reduce (EMR) by understanding how to use AWS Web Console to create and manage EMR Clusters. You will also learn about all the key features of Web Console and also how to connect to the master node of the cluster and validate all the important CLI interfaces such as spark-shell, pyspark, hive, etc as well as hdfs and aws CLI commands.Once you understand how to get started with AWS EMR, you will go through the details related to Setting up Development Cluster using AWS EMR. There are quite a few advantages to using AWS EMR Clusters for development purposes and most enterprises do so.After setting up a development cluster using AWS EMR, you will go through the Development Life Cycle of Spark Applications using AWS EMR Development Cluster. You will be using Visual Studio Code Remote Development on top of the AWS EMR Development Cluster to go through the details.Once the development is done, you will go through the details related to Deploying Spark Application on AWS EMR Cluster. You will build the zip file and understand how to run using CLI in both clients as well as cluster deployment modes. You will also understand how you can deploy the spark application as a step on AWS EMR Clusters. You will also understand the details related to troubleshooting the issues related to Spark Applications by going through relevant logs.Typically we run Spark Applications programmatically. After going through the details related to deploying spark applications on AWS EMR Clusters, you will be learning how to Manage AWS EMR Clusters using Python Boto3. You will not only learn how to create clusters programmatically but also how to deploy Spark Applications as Steps programmatically using Python Boto3.End to End Data Pipelines using AWS EMR is built using AWS Step Functions. Once you understand how to manage EMR Clusters using Python Boto3 and also deploy Spark Applications on EMR Clusters using the same, it is important to learn how to Build EMR-based Workflows or Pipelines using AWS Step Functions. You will be learning how to create the cluster, deploy Spark Application as Step on to the cluster, and then terminate the cluster as part of a basic pipeline or State Machine using AWS Step Functions.You will also learn how to perform validations as part of State Machines by Enhancing AWS EMR-based State Machine or Pipeline. You will check if the files specified already exist as part of the validations.We can also build Data Processing Applications or Pipelines using Spark SQL on AWS EMR. First, you will learn how to design and develop solutions using Spark SQL Script, how to validate by using appropriate commands by passing relevant runtime arguments, etc.Once you understand the development process of implementing solutions using Spark SQL on AWS EMR, you will learn how to deploy Data Pipeline using AWS Step Function to deploy Spark SQL Script on EMR Cluster. You will also learn the concept of Boto3 Waiters to make sure the steps are executed in a linear fashion.
