|
所在平台: Coursera |
课程主页: https://www.coursera.org/learn/developing-pipelines-on-dataflow
课程评论：没有评论
课程名称：无服务器数据处理与Dataflow：开发管道 概述：在Dataflow课程系列的第二部分中，我们将深入探讨如何使用Beam SDK开发数据处理管道。课程开始时将回顾Apache Beam的概念，接着讨论如何利用窗口、时间戳和触发器处理流数据。随后，我们将介绍管道中的数据源和接收器选项、用于表达结构化数据的模式，以及如何使用状态和定时器API进行有状态转换。接下来，课程将回顾最大化管道性能的最佳实践。课程结束时，我们会介绍如何使用SQL和数据框架在Beam中表示业务逻辑，并通过Beam笔记本迭代地开发管道。 课程大纲： 1. 介绍 - 本模块介绍课程及其大纲。 2. Beam概念回顾 - 回顾Apache Beam的主要概念，以及如何将其应用于编写自己的数据处理管道。 3. 窗口、时间戳与触发器 - 学习如何在Dataflow中处理流数据，重点掌握窗口分组、时间戳的重要性以及如何控制窗口输出的时间与次数。 4. 数据源与接收器 - 学习Dataflow中的数据源和接收器，包括TextIO、FileIO、BigQueryIO、PubsubIO、KafKaIO、BigtableIO、AvroIO和Splittable DoFn等示例，以及各I/O的重要特性。 5. 模式 - 介绍模式，使开发者能够在Beam管道中表达结构化数据。 6. 状态与定时器 - 探讨状态和定时器这两个强大的功能，如何在DoFn中实现有状态的转换。 7. 最佳实践 - 讨论最佳实践，回顾能够最大化Dataflow管道性能的常见模式。 8. Dataflow SQL与数据框架 - 介绍两个新的API，SQL和数据框架，用于在Beam中表示业务逻辑。 9. Beam笔记本 - 介绍Beam笔记本，为Python开发者提供一个在Jupyter笔记本环境中迭代开发管道的接口。 10. 总结 - 对课程内容进行回顾与总结。
Name:Introduction
Description:This module introduces the course and course outline
Name:Beam Concepts Review
Description:Review main concepts of Apache Beam, and how to apply them to write your own data processing pipelines.
Name:Windows, Watermarks, and Triggers
Description:In this module, you will learn about how to process data in streaming with Dataflow. For that, there are three main concepts that you need to learn: how to group data in windows, the importance of watermark to know when the window is ready to produce results, and how you can control when and how many times the window will emit output.
Name:Sources & Sinks
Description:In this module, you will learn about what makes sources and sinks in Dataflow. The module will go over some examples of TextIO, FileIO, BigQueryIO, PubsubIO, KafKaIO, BigtableIO, AvroIO, and Splittable DoFn. The module will also point out some useful features associated with each I/O.
Name:Schemas
Description:This module will introduce schemas, which give developers a way to express structured data in their Beam pipelines.
Name:State and Timers
Description:This module covers State and Timers, two powerful features that you can use in your DoFn to implement stateful transformations.
Name: Best Practices
Description:This module will discuss best practices and review common patterns that maximize performance for your Dataflow pipelines.
Name:Dataflow SQL & DataFrames
Description:This modules introduces two new APIs to represent your business logic in Beam: SQL and Dataframes.
Name:Beam Notebooks
Description:This module will cover Beam notebooks, an interface for Python developers to onboard onto the Beam SDK and develop their pipelines iteratively in a Jupyter notebook environment.
Name:Summary
Description:This module provides a recap of the course
In this second installment of the Dataflow course series, we are going to be diving deeper on developing pipelines using the Beam SDK. We start with a review of Apache Beam concepts. Next, we discuss processing streaming data using windows, watermarks and triggers. We then cover options for sources and sinks in your pipelines, schemas to express your structured data, and how to do stateful transformations using State and Timer APIs. We move onto reviewing best practices that help maximize your pipeline performance. Towards the end of the course, we introduce SQL and Dataframes to represent your business logic in Beam and how to iteratively develop pipelines using Beam notebooks.
