|
所在平台: Udemy |
课程主页: https://www.udemy.com/course/gcp-professional-data-engineer-exam-mini-practice/
课程评论：没有评论
课程名称：GCP专业数据工程师考试迷你练习 课程概述：本课程专注于设计数据处理系统，涵盖了批处理和流处理的关键概念与应用。学习如何使用Cloud Dataflow（Apache Beam）进行大规模批处理转化，利用Cloud Dataproc处理Hadoop/Spark工作负载，并通过BigQuery实现ELT模式。掌握Cloud Pub/Sub用于消息摄取，Cloud Dataflow进行实时转化与窗口处理。此外，课程介绍了Cloud Composer（Apache Airflow）在调度、管理和监控复杂数据管道中的重要性。 课程内容包括数据湖和数据仓库的特征及案例，如何选择相应服务（如Cloud Storage用于数据湖，BigQuery用于数据仓库），以及虚拟私有云（VPC）在数据管道连接性和安全性方面的作用。 在构建和运营数据处理系统方面，课程将教授数据摄取的方式，通过云存储传输服务、BigQuery数据传输服务和数据库迁移服务等工具将数据导入Google Cloud。参与者还将获得在Dataflow（Beam SDK）、BigQuery中的SQL以及Dataprep进行数据转化的实践经验。课程涉及使用Cloud Monitoring和Cloud Logging观察管道健康及性能，设置警报以排查问题，以及如何识别和优化Dataflow、BigQuery或Dataproc作业的瓶颈。 数据存储方面，BigQuery作为核心，课程将探讨其架构、分区、聚类、优化表设计及成本管理等。不仅如此，Cloud Storage的不同存储类别、访问控制和生命周期管理，以及何时使用Cloud SQL/Cloud Spanner和Cloud Bigtable也是重点内容。此外，还介绍了Memorystore的缓存解决方案。 在为分析和机器学习准备及使用数据方面，课程讲解Dataplex在数据发现、元数据管理、质量和治理中的作用，确保数据流中的数据质量策略，以及如何使用BigQuery ML在BigQuery中直接构建机器学习模型。课程还将介绍Vertex AI的基本知识及Looker Studio（前身为Data Studio）在自助分析中的数据可视化能力。 最后，课程将讨论确保解决方案质量的重要性，包括身份和访问管理（IAM）在安全性中的角色，设计高可用性、灾难恢复与容错的可靠性，以及合规性的基本理解，以帮助满足相关标准。 通过本课程，学员将能够加深对Google Cloud下数据工程师职责和技能的理解，为GCP专业数据工程师考试做好准备。
Designing Data Processing Systems:Batch Processing: Understanding when to use services like Cloud Dataflow (Apache Beam) for large-scale batch transformations, Cloud Dataproc for Hadoop/Spark workloads, and BigQuery for ELT patterns.Streaming Processing: Mastering Cloud Pub/Sub for messaging ingestion, and Cloud Dataflow for real-time transformations, windowing, and handling late data.Orchestration: Cloud Composer (Apache Airflow) is crucial for scheduling, managing, and monitoring complex data pipelines with dependencies.Data Lake vs. Data Warehouse: Knowing the characteristics, use cases, and appropriate services for each (Cloud Storage for data lakes, BigQuery for data warehouses).Networking: Understanding VPCs, Shared VPC, Private Service Connect, and how they impact data pipeline connectivity and security.Building and Operationalizing Data Processing Systems:Data Ingestion: How to get data into Google Cloud from various sources (on-premises databases, other clouds, streaming sources). Tools include Cloud Storage Transfer Service, BigQuery Data Transfer Service, Database Migration Service, Datastream, Pub/Sub.Data Transformation: Hands-on experience with Dataflow (Beam SDK), SQL in BigQuery, and potentially Dataprep for visual data preparation.Monitoring and Logging: Using Cloud Monitoring and Cloud Logging to observe pipeline health, performance, and troubleshoot issues. Setting up alerts.Debugging and Optimization: Understanding how to identify bottlenecks in Dataflow, BigQuery, or Dataproc jobs, and how to optimize them for cost and performance.Storing Data:BigQuery: This is central. Understand its architecture, partitioning, clustering, optimal table design, cost optimization, streaming inserts, external tables, materialized views, and BigQuery ML.Cloud Storage: Different storage classes, access controls, lifecycle management, and its role as a data lake.Cloud SQL/Cloud Spanner: When to use relational databases (transactional workloads) versus analytical data warehouses.Cloud Bigtable: When to use NoSQL wide-column stores for high-throughput, low-latency access.Memorystore: Caching solutions.Preparing and Using Data for Analysis and Machine Learning:Data Governance: Understanding Dataplex for data discovery, metadata management, quality, and governance across data lakes and warehouses.Data Quality: Strategies for ensuring data quality within pipelines.BigQuery ML: Building machine learning models directly within BigQuery using SQL.Vertex AI: Basic understanding of Vertex AI's capabilities for custom ML model development and deployment, especially how data engineers provide data to ML engineers.Looker Studio (formerly Data Studio): Basic understanding of data visualization to enable self-service analytics.Ensuring Solution Quality (Security, Reliability, Compliance):Security: IAM (Identity and Access Management) is paramount. Understand roles (primitive, predefined, custom), service accounts, least privilege principle. Data encryption (at rest and in transit). VPC Service Controls for data exfiltration prevention.Reliability: Designing for high availability, disaster recovery, fault tolerance, and data integrity. Understanding data consistency models.Compliance: Basic understanding of compliance standards and how Google Cloud services help meet them.
