|
所在平台: Coursera |
课程主页: https://www.coursera.org/learn/etl-and-data-pipelines-shell-airflow-kafka
课程评论：没有评论
课程名称：使用Shell、Airflow和Kafka进行ETL和数据管道 课程概述：在完成本课程后，您将能够描述将原始数据转换为可分析数据的两种不同方法。一种方法是提取、转换和加载（ETL）过程，另一种是提取、加载和转换（ELT）过程。ETL过程适用于数据仓库和数据集市，而ELT过程则适用于数据湖，数据在被请求的应用程序中按需转换。无论是ETL还是ELT，都涉及从源系统提取数据、通过数据管道移动数据并将数据存储到目标系统中。本课程将让您体验ETL和ELT处理的不同，并识别这两者的使用场景。 您将识别提取数据的方法和工具，了解逻辑或物理合并提取数据的方式，以及如何将数据导入到数据仓库中。您还将定义对源数据应用的转换，以使数据对用户可信、相关和可访问。您将能够概述多种将数据加载到目标系统中的方法，验证数据质量，监控加载失败，并在失败情况下使用恢复机制。 最后，您将完成一个可分享的最终项目，以展示您在每个模块中获得的技能。 课程大纲： 1. 数据处理技术：探索ETL与ELT的关键区别，包括转换的位置、灵活性、大数据支持和洞察时间。您将学习到，随着对原始数据访问需求的增加，ETL逐渐演变为ELT。数据提取涉及先进的技术，包括数据库查询、网络抓取和API。您还将学习到数据转换是关于为应用格式化数据，数据可以批量加载或持续流式加载。 2. ETL与数据管道：工具和技术：了解如何使用Bash脚本创建提取、转换和加载（ETL）管道，并通过cron定时运行。数据管道负责从一个地方或形式移动数据到另一个地方。学习调度、监控、维护和优化数据管道的过程，并辨别批处理管道与流式数据管道的不同。 3. 使用Airflow构建数据管道：Apache Airflow将数据管道表示为DAG的方式的主要优势在于可以用代码表达，增强数据管道的可维护性、可测试性和协作性。学习如何使用Airflow的丰富UI简化数据管道的工作，并探索DAG的可视化方式。 4. 使用Kafka构建流媒体管道：了解Apache Kafka作为流事件处理管道的核心组件。学习Kafka Streams API以及如何使用Kafka构建事件流管道。 5. 最终作业：在本模块中，您将应用新获得的知识进行两个动手实验：“使用Apache Airflow创建ETL数据管道”和“使用Kafka创建流媒体数据管道”。您将处理真实场景下的数据，提取、转换并加载到CSV文件中，同时在Apache Kafka中创建主题“toll”，并验证流媒体数据已被收集到数据库表中。
Name:Data Processing Techniques
Description:ETL or Extract, Transform, and Load processes are used for cases where flexibility, speed, and scalability of data are important. You will explore some key differences between similar processes, ETL and ELT, which include the place of transformation, flexibility, Big Data support, and time-to-insight. You will learn that there is an increasing demand for access to raw data that drives the evolution from ETL to ELT. Data extraction involves advanced technologies including database querying, web scraping, and APIs. You will also learn that data transformation is about formatting data to suit the application and that data is loaded in batches or streamed continuously.
Name:ETL & Data Pipelines: Tools and Techniques
Description:Extract, transform and load (ETL) pipelines are created with Bash scripts that can be run on a schedule using cron. Data pipelines move data from one place, or form, to another. Data pipeline processes include scheduling or triggering, monitoring, maintenance, and optimization. Furthermore, Batch pipelines extract and operate on batches of data. Whereas streaming data pipelines ingest data packets one-by-one in rapid succession. In this module, you will learn that streaming pipelines apply when the most current data is needed. You will explore that parallelization and I/O buffers help mitigate bottlenecks. You will also learn how to describe data pipeline performance in terms of latency and throughput.
Name:Building Data Pipelines using Airflow
Description:The key advantage of Apache Airflow's approach to representing data pipelines as DAGs is that they are expressed as code, which makes your data pipelines more maintainable, testable, and collaborative. Tasks, the nodes in a DAG, are created by implementing Airflow's built-in operators. In this module, you will learn about Apache Airflow having a rich UI that simplifies working with data pipelines. You will explore how to visualize your DAG in graph or tree mode. You will also learn about the key components of a DAG definition file, and you will learn that Airflow logs are saved into local file systems and then sent to cloud storage, search engines, and log analyzers.
Name:Building Streaming Pipelines using Kafka
Description:Apache Kafka is a very popular open source event streaming pipeline. An event is a type of data that describes the entity’s observable state updates over time. Popular Kafka service providers include Confluent Cloud, IBM Event Stream, and Amazon MSK. Additionally, Kafka Streams API is a client library supporting you with data processing in event streaming pipelines. In this module, you will learn that the core components of Kafka are brokers, topics, partitions, replications, producers, and consumers. You will explore two special types of processors in the Kafka Stream API stream-processing topology: The source processor and the sink processor. You will also learn about building event streaming pipelines using Kafka.
Name:Final Assignment
Description:In this final assignment module, you will apply your newly gained knowledge to explore two very exciting hands-on labs. “Creating ETL Data Pipelines using Apache Airflow” and “Creating Streaming Data Pipelines using Kafka”. You will explore building these ETL pipelines using real-world scenarios. You will extract, transform, and load data into a CSV file. You will also create a topic named “toll” in Apache Kafka, download and customize a streaming data consumer, as well as verifying that streaming data has been collected in the database table.
After taking this course, you will be able to describe two different approaches to converting raw data into analytics-ready data. One approach is the Extract, Transform, Load (ETL) process. The other contrasting approach is the Extract, Load, and Transform (ELT) process. ETL processes apply to data warehouses and data marts. ELT processes apply to data lakes, where the data is transformed on demand by the requesting/calling application. Both ETL and ELT extract data from source systems, move the data through the data pipeline, and store the data in destination systems. During this course, you will experience how ELT and ETL processing differ and identify use cases for both. You will identify methods and tools used for extracting the data, merging extracted data either logically or physically, and for importing data into data repositories. You will also define transformations to apply to source data to make the data credible, contextual, and accessible to data users. You will be able to outline some of the multiple methods for loading data into the destination system, verifying data quality, monitoring load failures, and the use of recovery mechanisms in case of failure. Finally, you will complete a shareable final project that enables you to demonstrate the skills you acquired in each module.
