|
所在平台: Coursera |
课程主页: https://www.coursera.org/learn/serverless-data-processing-with-dataflow-operations
课程评论：没有评论
课程名称：无服务器数据处理与 Dataflow：运维 概述：在 Dataflow 课程系列的最后一部分，我们将介绍 Dataflow 运维模型的组成部分。我们将探讨故障排除和优化管道性能的工具和技术。随后，我们将回顾 Dataflow 管道的测试、部署和可靠性最佳实践。最后，我们将介绍模板的使用，帮助轻松扩展 Dataflow 管道以满足成百上千用户的组织需求。这些课程将帮助确保您的数据平台稳定且具备抵御意外情况的能力。 课程大纲： 1. **介绍**：本模块涵盖课程大纲。 2. **监控**：我们将学习如何使用作业列表页面筛选我们要监控或调查的作业，了解作业图、作业信息和作业指标选项卡如何共同提供 Dataflow 作业的全面概述，并学习如何利用 Dataflow 和指标浏览器的集成创建数据流指标的警报策略。 3. **日志记录和错误报告**：我们将学习如何使用作业图和作业指标页面底部的日志面板，以及了解集中式错误报告页面。 4. **故障排除和调试**：本模块将教授如何故障排除和调试 Dataflow 管道，回顾 Dataflow 中常见的四种故障模式：管道构建失败、管道未能在 Dataflow 上启动、管道执行过程中的失败以及性能问题。 5. **性能**：我们将讨论在开发 Dataflow 的批处理和流处理管道时应考虑的性能因素。 6. **测试与CI/CD**：本模块将探讨对 Dataflow 管道进行单元测试，并介绍可优化 Dataflow 管道 CI/CD 工作流程的框架和特性。 7. **可靠性**：我们将讨论构建能够抵御数据损坏和数据中心停机的系统的方法。 8. **Flex 模板**：本模块涵盖 Flex 模板，这一功能帮助数据工程团队标准化和重用 Dataflow 管道代码。许多运营挑战可以通过 Flex 模板解决。 9. **总结**：本模块回顾课程中涉及的主题。 通过本课程，学员将掌握 Dataflow 管道的操作、监控、优化和扩展等全面技能。
Name:Introduction
Description:This module covers the course outline
Name:Monitoring
Description:In this module, we learn how to use the Jobs List page to filter for jobs that we want to monitor or investigate. We look at how the Job Graph, Job Info, and Job Metrics tabs collectively provide a comprehensive summary of your Dataflow job. Lastly, we learn how we can use Dataflow’s integration with Metrics Explorer to create alerting policies for Dataflow metrics.
Name:Logging and Error Reporting
Description:In this module, we learn how to use the Log panel at the bottom of both the Job Graph and Job Metrics pages, and learn about the centralized Error Reporting page.
Name:Troubleshooting and Debug
Description:In this module, we learn how to troubleshoot and debug Dataflow pipelines. We will also review the four common modes of failure for Dataflow: failure to build the pipeline, failure to start the pipeline on Dataflow, failure during pipeline execution, and performance issues.
Name:Performance
Description:In this module, we will discuss performance considerations we should be aware of while developing batch and streaming pipelines in Dataflow.
Name:Testing and CI/CD
Description:This module will discuss unit testing your Dataflow pipelines. We also introduce frameworks and features available to streamline your CI/CD workflow for Dataflow pipelines.
Name:Reliability
Description:In this module we will discuss methods for building systems that are resilient to corrupted data and data center outages.
Name:Flex Templates
Description:This module covers Flex Templates, a feature that helps data engineering teams standardize and reuse Dataflow pipeline code. Many operational challenges can be solved with Flex Templates.
Name:Summary
Description:This module reviews the topics covered in the course
In the last installment of the Dataflow course series, we will introduce the components of the Dataflow operational model. We will examine tools and techniques for troubleshooting and optimizing pipeline performance. We will then review testing, deployment, and reliability best practices for Dataflow pipelines. We will conclude with a review of Templates, which makes it easy to scale Dataflow pipelines to organizations with hundreds of users. These lessons will help ensure that your data platform is stable and resilient to unanticipated circumstances.
