|
所在平台: Udemy |
课程主页: https://www.udemy.com/course/google-cloud-certified-professional-data-engineer-exam/
课程评论：没有评论
课程名称：Google Cloud认证专业数据工程师 课程概述： 本课程专注于设计数据处理系统和选择合适的存储技术，包括以下几个方面： 1. **数据存储系统**： - 按照业务需求映射存储系统 - 数据建模与延迟、吞吐量、事务的权衡 - 分布式系统与模式设计 2. **数据管道设计**： - 数据发布与可视化（如BigQuery） - 批量和流式数据处理（使用Dataflow、Dataproc、Apache Beam等） - 线上交互预测与批处理预测 3. **数据处理解决方案设计**： - 基础设施选择与系统可用性 - 分布式系统与容量规划 - 迁移数据仓库和处理系统的考虑 4. **构建与运营数据处理系统**： - 有效利用托管服务（如Cloud Bigtable、Cloud Spanner等） - 数据清洗与管道监控 5. **运用机器学习模型**： - 利用预构建的机器学习模型作为服务 - 部署机器学习管道与数据的连续评估 6. **确保解决方案质量**： - 设计安全性与合规性 - 确保可扩展性与效率 - 可靠性与稳定性保障 7. **灵活性与可移植性**： - 设计满足当前与未来商业需求的数据与应用的可移植性 本课程旨在帮助学员掌握Google Cloud平台上的数据工程实践，提升他们在数据处理和分析领域的专业技能。
Designing data processing systemsSelecting the appropriate storage technologies. Considerations include:● Mapping storage systems to business requirements● Data modeling● Trade-offs involving latency, throughput, transactions● Distributed systems● Schema designDesigning data pipelines. Considerations include:● Data publishing and visualization (e.g., BigQuery)● Batch and streaming data (e.g., Dataflow, Dataproc, Apache Beam, Apache Spark and Hadoop ecosystem, Pub/Sub, Apache Kafka)● Online (interactive) vs. batch predictions● Job automation and orchestration (e.g., Cloud Composer)Designing a data processing solution. Considerations include:● Choice of infrastructure● System availability and fault tolerance● Use of distributed systems● Capacity planning● Hybrid cloud and edge computing● Architecture options (e.g., message brokers, message queues, middleware, service-oriented architecture, serverless functions)● At least once, in-order, and exactly once, etc., event processingMigrating data warehousing and data processing. Considerations include:● Awareness of current state and how to migrate a design to a future state● Migrating from on-premises to cloud (Data Transfer Service, Transfer Appliance, Cloud Networking)● Validating a migrationBuilding and operationalizing data processing systemsBuilding and operationalizing storage systems. Considerations include:● Effective use of managed services (Cloud Bigtable, Cloud Spanner, Cloud SQL, BigQuery, Cloud Storage, Datastore, Memorystore)● Storage costs and performance● Life cycle management of dataBuilding and operationalizing pipelines. Considerations include:● Data cleansing● Batch and streaming● Transformation● Data acquisition and import● Integrating with new data sourcesBuilding and operationalizing processing infrastructure. Considerations include:● Provisioning resources● Monitoring pipelines● Adjusting pipelines● Testing and quality controlOperationalizing machine learning modelsLeveraging pre-built ML models as a service. Considerations include:● ML APIs (e.g., Vision API, Speech API)● Customizing ML APIs (e.g., AutoML Vision, Auto ML text)● Conversational experiences (e.g., Dialogflow)Deploying an ML pipeline. Considerations include:● Ingesting appropriate data● Retraining of machine learning models (AI Platform Prediction and Training, BigQuery ML, Kubeflow, Spark ML)● Continuous evaluationChoosing the appropriate training and serving infrastructure. Considerations include:● Distributed vs. single machine● Use of edge compute● Hardware accelerators (e.g., GPU, TPU)Measuring, monitoring, and troubleshooting machine learning models. Considerations include:● Machine learning terminology (e.g., features, labels, models, regression, classification, recommendation, supervised and unsupervised learning, evaluation metrics)● Impact of dependencies of machine learning models● Common sources of error (e.g., assumptions about data)Ensuring solution qualityDesigning for security and compliance. Considerations include:● Identity and access management (e.g., Cloud IAM)● Data security (encryption, key management)● Ensuring privacy (e.g., Data Loss Prevention API)● Legal compliance (e.g., Health Insurance Portability and Accountability Act (HIPAA), Children's Online Privacy Protection Act (COPPA), FedRAMP, General Data Protection Regulation (GDPR))Ensuring scalability and efficiency. Considerations include:● Building and running test suites● Pipeline monitoring (e.g., Cloud Monitoring)● Assessing, troubleshooting, and improving data representations and data processing infrastructure● Resizing and autoscaling resourcesEnsuring reliability and fidelity. Considerations include:● Performing data preparation and quality control (e.g., Dataprep)● Verification and monitoring● Planning, executing, and stress testing data recovery (fault tolerance, rerunning failed jobs, performing retrospective re-analysis)● Choosing between ACID, idempotent, eventually consistent requirementsEnsuring flexibility and portability. Considerations include:● Mapping to current and future business requirements● Designing for data and application portability (e.g., multicloud, data residency requirements)● Data staging, cataloging, and discovery
