|
所在平台: Coursera |
课程主页: https://www.coursera.org/learn/source-systems-data-ingestion-and-pipelines
课程评论：没有评论
课程名称：源系统、数据摄取与管道 课程概述：本课程将探讨各类源系统，了解它们如何生成和更新数据，以及在实际连接这些系统时可能遇到的常见问题。您将深入了解常见的摄取模式，并构建批处理和流处理管道。课程还将介绍如何使用基础设施即代码和管道即代码工具来自动化和协调您的数据管道。此外，还将探讨监控数据的AWS与开源工具。 课程大纲： 1. **源系统的工作** 描述：在第一课中，您将探索数据工程师常接触的源系统。在第二课中，您将学习如何连接到各种源系统，并解决常见的连接问题。 2. **数据摄取** 描述：本周，您将深入研究批处理和流处理的摄取模式。您将识别各自的用例和注意事项，并构建一个批处理和一个流处理摄取管道。在讨论批处理摄取时，您将比较ETL与ELT的范式，同时探索AWS在批处理和流处理摄取方面的各种服务。 3. **数据运营（DataOps）** 描述：在第一课中，您将探索DataOps的自动化实践，包括将CI/CD应用于数据和代码，以及使用像Terraform这样的基础设施即代码工具来自动化资源的配置和管理。在第二课中，您将探讨DataOps的可观察性和监控实践，包括使用Great Expectation监控数据质量，以及使用Amazon CloudWatch监控基础设施。 4. **管道的协调、监控与自动化** 描述：本周，您将学习如何协调数据管道任务。您将识别各种协调工具，重点关注Airflow——当前最流行和广泛使用的工具之一。您将探索Airflow的核心组件、Airflow UI，以及如何使用各种Airflow功能创建和管理DAG。
Name:Working with Source Systems
Description:In lesson 1, you will explore source systems data engineers commonly interact with. Then in lesson 2, you will learn how to connect to various source systems and troubleshoot common connectivity issues.
Name:Data Ingestion
Description:This week you will dive deep into the batch and streaming ingestion patterns. You will identify use cases and considerations for each, and then build a batch and a streaming ingestion pipeline. When looking at batch ingestion, you will compare and contrast the ETL and ELT paradigms. You will also explore various AWS services for batch and streaming ingestion.
Name:DataOps
Description:In the first lesson, you will explore DataOps automation practices, including applying CI/CD to both data and code, and using infrastructure as code tools like Terraform to automate the provisioning and management of your resources. Then in lesson 2, you will explore DataOps observability and monitoring practices, including using tools like Great Expectation to monitor data quality, and using Amazon CloudWatch to monitor your infrastructure.
Name:Orchestration, Monitoring, and Automating Your Data Pipelines
Description:This week, you will learn all about orchestrating your data pipeline tasks. You'll identify the various orchestration tools, but will focus on Airflow -- one of the most popular and widely used tools in the field today. You'll explore the core components of Airflow, the Airflow UI, and how to create and manage DAGs using various Airflow features.
In this course, you will explore various types of source systems, learn how they generate and update data, and troubleshoot common issues you might encounter when trying to connect to these systems in the real world. You’ll dive into the details of common ingestion patterns and implement batch and streaming pipelines. You’ll automate and orchestrate your data pipelines using infrastructure as code and pipelines as code tools. You’ll also explore AWS and open source tools for monitoring your data
