|
所在平台: Udemy |
课程主页: https://www.udemy.com/course/apache-beam-a-hands-on-course-to-build-big-data-pipelines/
课程评论：没有评论
## Coursera 课程总结：Apache Beam 大数据管道实战 本课程《Apache Beam 大数据管道实战》旨在教授学员如何使用 Apache Beam 这一统一且可移植的编程模型来构建批量和流式数据处理管道。 **核心内容：** * **Apache Beam 概念详解：** 从零开始，全面讲解 Apache Beam 的核心概念，包括其架构、各种 PTransforms（如 Map, FlatMap, Filter, ParDo 等）、Combiner、Side inputs/outputs。 * **高级主题：** 深入探讨类型提示（Type Hints）、编码与解码、水印（Watermarks）、触发器（Triggers）等高级特性。 * **实战应用：** 通过大量的动手实践和实际案例，帮助学员深入理解和掌握 Beam 的应用。 * **窗口函数实现：** 学习如何实现多种窗口函数，如滚动窗口（Tumbling Windows）、滑动窗口（Sliding Windows）、全局窗口（Global Windows）和会话窗口（Session Windows）。 * **端到端案例研究：** 构建两个真实的、端到端的大数据处理案例，将所学知识应用于实际场景。 * **与 Google Cloud 集成：** 演示如何通过 Dataflow 将处理后的数据加载到 Google Cloud BigQuery 表中。 * **可移植性优势：** 强调 Apache Beam 的可移植性，使其能够运行在多种大数据引擎上，如 Apache Spark, Flink, Google Cloud Dataflow 等，无需为特定平台编写代码，是未来大数据处理管道构建的重要方向。 **课程特色：** * **从零开始：** 适合初学者，逐步深入。 * **实践导向：** 每个概念都配有实际操作示例，强调动手能力。 * **理论结合实践：** 涵盖核心概念和高级特性，并通过实战案例巩固。 * **提供资源：** 课程中提供的所有代码和数据集方便学员练习。 本课程使学员能够构建语言无关、高度可移植的大数据处理管道，并具备将处理结果高效地集成到云平台的能力。
Apache Beam is a unified and portable programming model for both Batch and Streaming data use cases.Earlier we could run Spark, Flink & Cloud Dataflow Jobs only on their respective clusters. But now Apache Beam has come up with a portable programming model where we can build language agnostic Big data pipelines and run it on any Big data engine (Apache Spark, Flink or in Google Cloud Platform using Cloud Dataflow service and many more Big data engines).Apache Beam is the future of building Big data processing pipelines and is going to be accepted by mass companies due to its portability. Many big companies have even started deploying Beam pipelines in their production servers.What's included in the course ?Complete Apache Beam concepts explained from Scratch to Real-Time implementation. Every Apache Beam concept is taught through Hands-on, practical examples for better understandingCore Apache Beam topics including Architecture, Various PTransforms (Map, FlatMap, Filter, ParDo etc.), Combiner, Side inputs/outputs. ADVANCE topics - Type Hints, Encoding & Decoding, Watermarks, Triggers and many more.Build 2 Real-time Big data case studies using Apache Beam programming model.Learn to implement Windows functions - Tumbling, Sliding, Global and Session Windows.Load processed data to Google Cloud BigQuery Tables from Apache Beam pipeline via Dataflow.All codes and datasets used in lessons are attached in the course for your convenience.
