|
所在平台: Udemy |
课程主页: https://www.udemy.com/course/codeless-data-engineering-in-gcp-beginner-to-advanced/
课程评论：没有评论
**课程名称：** GCP 实践数据工程：从入门到高级 **课程概述：** 本课程将指导您在 Google Cloud Platform (GCP) 上构建一个现代化的数据湖和数据仓库架构，即“湖仓一体”（Lakehouse）。您将深入学习如何利用 Google Cloud Storage (GCS) 搭建数据湖，并借助 Google BigQuery 实现数据仓库功能。 课程将带领您完成一个实际项目，通过动手实践，一步步搭建数据湖，创建用于数据摄取和转换的**无代码数据管道**。我们将重点介绍并使用包括 DataStream, Dataflow, Dataprep, Pub/Sub, Data Fusion, Cloud Storage, BigQuery 在内的多种 GCP 服务。 **课程内容概要：** * **第一章：Google Cloud 项目设置与基础服务介绍** * 在 Google Cloud 中设置项目 * Google Cloud Storage (GCS) 简介 * Google BigQuery 简介 * **第二章：数据管道 1 - 实时数据摄取与流处理** * 创建 Cloud SQL 数据库并填充数据 * 使用 DataStream 进行变更数据捕获 (CDC)，将数据从 Cloud SQL 流式传输到 GCS 数据湖 * 为 GCS 存储桶添加 Pub/Sub 通知 * 创建 Dataflow 管道，将流式数据导入 BigQuery * **第三章：数据管道 2 - 数据转换与治理** * Google Data Fusion 简介 * 创作和监控 ETL 作业，在数据湖的不同区域之间转换数据 * 使用 Data Fusion 的 Wrangler 功能进行数据剖析 (profiling) 和理解 * 数据清洗和规范化 * 通过 Data Fusion 的元数据进行数据发现和治理 * **第四章：数据管道 3 - 实时消息流处理** * Google Pub/Sub 简介 * 构建 .NET 应用程序，将数据发布到 Pub/Sub 主题 * 构建实时数据管道，将消息流式传输到 BigQuery * **第五章：数据管道 4 - 数据准备与转换** * Cloud DataPrep 简介 * 使用 DataPrep 进行数据剖析、创作和监控 ETL 作业，实现数据转换
In this course, we will be creating a data lake using Google Cloud Storage and bring data warehouse capabilites to the data lake to form the lakehouse architecture using Google BigQuery. We will be building four no code data pipelines using services such as DataStream, Dataflow, DataPrep, Pub/Sub, Data Fusion, Cloud Storage, BigQuery etc.The course will follow a logical progression of a real world project implementation with hands on experience of setting up a data lake, creating data pipelines for ingestion and transforming your data in preparation for analytics and reporting.Chapter 1 We will setup a project in Google CloudIntroduction to Google Cloud StorageIntroduction to Google BigQueryChapter 2 - Data Pipeline 1We will create a cloud SQL database and populate with data before we start performing complex ETL jobs.Use DataStream Change Data Capture for streaming data from our Cloud SQL Database into our Data lake built with Cloud StorageAdd a pub/sub notification to our bucketCreate a Dataflow Pipeline for streaming jobs into BigQueryChapter 3 - Data Pipeline 2Introduce Google Data FusionAuthor and monitor ETL jobs for tranforming our data and moving them between different zone of our data lakeWe will explore the use of Wrangler in Data Fusion for profiling and understanding our data before we starting performing complex ETL jobs.Clean and normalise dataDiscover and govern data using metadata in Data FusionChapter 4 - Data Pipeline 3Introduction to Google Pub/SubBuilding a.Net application for publishing data to a Pub/Sub topicBuilding a realtime data pipeline for streaming messages to BigQueryChapter 5 - Data Pipeline 4Introduction to Cloud DataPrepProfile, Author and monitor ETL jobs for tranforming our data using DataPrep
