|
所在平台: Udemy |
课程主页: https://www.udemy.com/course/master-data-engineering-using-gcp-data-analytics/
课程评论：没有评论
**课程名称：** 掌握使用 GCP 数据分析进行数据工程 **课程概述：** 本课程专注于数据工程的核心——构建数据管道，将数据从多个源头整合到数据湖或数据仓库，再输送到下游系统。课程将深入讲解如何利用 Google Cloud Platform (GCP) 的数据分析服务栈来构建这些管道。**核心服务包括：Google Cloud Storage、Google BigQuery、GCP Dataproc、Databricks on GCP 等。** **主要学习内容：** 1. **环境搭建与 GCP 账户设置：** * 在 Windows 和 Mac 上使用 VS Code 作为开发环境。 * 注册 Google Cloud 账户，包括了解账单和获取 $300 免费额度。 2. **Google Cloud Storage (GCS) 的数据湖应用：** * 学习将 GCS 用作数据湖。 * 掌握使用命令行和 Python 管理 GCS 文件。 * 了解 Pandas 与 GCS 文件的集成。 3. **Cloud SQL 的关系型数据库应用：** * 设置 PostgreSQL 数据库服务器。 * 创建应用程序数据库和用户。 * 学习开发 Python 应用，并通过 GCP Secret Manager 安全地检索凭据。 4. **BigQuery 作为数据仓库：** * 深入了解 BigQuery 作为数据仓库的功能。 * 学习通过 Python 和 Pandas 进行集成。 5. **GCP Dataproc 的大数据处理：** * 了解 GCP Dataproc 的完全托管服务 (Hadoop, Spark, Kafka)。 * 学习设置 GCP Dataproc 集群，包括单节点集群用于开发。 * 在 VS Code 中设置远程连接到 Dataproc 集群进行开发。 * 使用 Dataproc Workflow Templates 构建端到端的 ELT 数据管道。 * 学习提交 Dataproc 作业和工作流的关键命令。 * 通过 Spark SQL 构建 ELT 管道。 6. **Databricks on GCP 的大数据处理：** * 了解 Databricks on GCP 的基本使用。 * 使用 Databricks 作业和工作流构建端到端的 ELT 数据管道。 7. **集成与高级管道构建：** * 将 BigQuery 和 GCP Dataproc 集成，构建端到端的 ELT 数据管道。 * 在管道中整合使用 Spark BigQuery Connector 的 PySpark 应用程序。 8. **Spark 应用开发与故障排除：** * 回顾 Spark 应用开发生命周期。 * 学习如何使用 YARN Timeline Server, Spark UI 等界面进行 Spark 相关问题的故障排除。
Data Engineering is all about building Data Pipelines to get data from multiple sources into Data Lakes or Data Warehouses and then from Data Lakes or Data Warehouses to downstream systems. As part of this course, I will walk you through how to build Data Engineering Pipelines using GCP Data Analytics Stack. It includes services such as Google Cloud Storage, Google BigQuery, GCP Dataproc, Databricks on GCP, and many more.As part of this course, first you will go ahead and setup environment to learn using VS Code on Windows and Mac.Once the environment is ready, you need to sign up for Google Cloud Account. We will provide all the instructions to sign up for Google Cloud Account including reviewing billing as well as getting USD 300 Credit.We typically use Cloud Object Storage as Data Lake. As part of this course, you will learn how to use Google Cloud Storage as Data Lake along with how to manage the files in Google Cloud Storage both by using commands as well as Python. It also covers, integration of Pandas with files in Google Cloud Storage.GCP provides RDBMS as service via Cloud SQL. You will learn how to setup Postgresql Database Server using Cloud SQL. Once the Database Server is setup, you will also take care of setting up required application database and user. You will also understand how to develop Python based applications by integrating with GCP Secretmanager to retrieve the credentials.One of the key usage of Data is nothing but building reports and dashboards. Typically reports and dashboards are built using reporting tools pointing to Data Warehouse. As part of Google Data Analytics Services, BigQuery can be used as Data Warehouse. You will learn the features of BigQuery as a Data Warehouse along with key integrations using Python and Pandas.At times, we need to process heavy volumes of data which also known as Big Data Processing. GCP Dataproc is a fully manage Big Data Service with Hadoop, Spark, Kafka, etc. You will not only learn how to setup the GCP Dataproc cluster, but also you will learn how to use single node Dataproc cluster for the development. You will setup development environment using VS Code with remote connection to the Dataproc Cluster.Once you understand how to get started with Big Data Processing using Dataproc, you will take care of building end to end ELT Data Pipelines using Dataproc Workflow Templates. You will learn all key commands to submit Dataproc Jobs as well as Workflows. You will end up building ELT Pipelines using Spark SQL.While Dataproc is GCP Native Big Data Service, Databricks is another prominent Big Data Service available in GCP. You will also understand how to get started with Databricks on GCP.Once you go through the details about how to get started with Databricks on GCP, you will take care of building end to end ELT Datapipelins using Databricks Jobs and Workflows.Towards the end of the course you should be fairly comfortable with BigQuery for Data Warehouse and GCP Dataproc for Data Processing, you will learn how to integrate these two key services by building end to end ELT Data Pipeline using Dataproc Workflow. You will also understand how to include Pyspark based application with Spark BigQuery connector as part of the Pipeline.In the process of building Data Pipelines, you will also revise application development life cycle of Spark, troubleshooting issues related to the spark using relevant web interfaces such as YARN Timeline Server, Spark UI, etc.
