|
所在平台: Udemy |
课程主页: https://www.udemy.com/course/google-professional-data-engineer-gcp-exams/
课程评论：没有评论
课程名称：Google专业数据工程师 - GCP考试 课程概述：谷歌云认证是2022年薪资最高的IT认证之一。该专业数据工程师考试评估您在以下方面的能力： 1. **设计数据处理系统** - 选择合适的存储技术，考虑业务需求、数据建模、延迟与吞吐量的权衡、分布式系统及模式设计等。 - 设计数据管道，包括数据发布与可视化（如BigQuery）、批处理与流处理（如Dataflow、Dataproc、Apache Beam、Apache Spark和Hadoop生态系统、Pub/Sub、Apache Kafka）、在线预测与批量预测的选择，作业自动化与编排（如Cloud Composer）。 - 设计数据处理解决方案，关注基础架构选择、系统可用性与容错能力、使用分布式系统、容量规划、混合云与边缘计算等。 - 数据迁移，包括从本地迁移到云（数据传输服务、传输设备、云网络）的战略及验证。 2. **构建和运营数据处理系统** - 构建和运营存储系统，管理服务的有效使用（如Cloud Bigtable、Cloud Spanner、Cloud SQL等）以及存储成本与性能的考虑。 - 构建和运营数据管道，关注数据清洗、批量与流处理、转换及数据获取整合。 - 构建和运营处理基础设施，包括资源配置、管道监控、管道调整和测试与质量控制。 3. **运营机器学习模型** - 利用预构建的机器学习模型作为服务（如ML API），并定制这些API（如AutoML）。 - 部署机器学习管道，包括合适数据的摄取及模型重训练。 - 选择合适的训练和服务基础设施，考虑分布式与单机的使用。 - 测量、监控和排除机器学习模型的故障，包括统计和评估模型的常见错误。 4. **确保解决方案质量** - 设计安全与合规性，关注身份与访问管理、数据安全和隐私等。 - 确保可扩展性与效率，建立测试套件、监控管道，并评估和改进数据处理基础设施。 - 确保可靠性与准确性，进行数据准备、质量控制和恢复测试。 - 确保灵活性与可迁移性，设计适合当前与未来商业需求的数据与应用。 本课程将为您在数据工程领域打下坚实的基础，增强您在谷歌云平台上的实践能力与认证竞争力。
Google Cloud certifications are among the highest paying IT certifications of 2022.The Professional Data Engineer exam assesses your ability to:1: Designing data processing systems1.1 Selecting the appropriate storage technologies. Considerations include:● Mapping storage systems to business requirements● Data modeling● Trade-offs involving latency, throughput, transactions● Distributed systems● Schema design1.2 Designing data pipelines. Considerations include:● Data publishing and visualization (e.g., BigQuery)● Batch and streaming data (e.g., Dataflow, Dataproc, Apache Beam, Apache Spark and Hadoop ecosystem, Pub/Sub, Apache Kafka)● Online (interactive) vs. batch predictions● Job automation and orchestration (e.g., Cloud Composer)1.3 Designing a data processing solution. Considerations include:● Choice of infrastructure● System availability and fault tolerance● Use of distributed systems● Capacity planning● Hybrid cloud and edge computing● Architecture options (e.g., message brokers, message queues, middleware, service-oriented architecture, serverless functions)● At least once, in-order, and exactly once, etc., event processing1.4 Migrating data warehousing and data processing. Considerations include:● Awareness of current state and how to migrate a design to a future state● Migrating from on-premises to cloud (Data Transfer Service, Transfer Appliance, Cloud Networking)● Validating a migration 2: Building and operationalizing data processing systems2.1 Building and operationalizing storage systems. Considerations include:● Effective use of managed services (Cloud Bigtable, Cloud Spanner, Cloud SQL, BigQuery, Cloud Storage, Datastore, Memorystore)● Storage costs and performance● Life cycle management of data2.2 Building and operationalizing pipelines. Considerations include:● Data cleansing● Batch and streaming● Transformation● Data acquisition and import● Integrating with new data sources2.3 Building and operationalizing processing infrastructure. Considerations include:● Provisioning resources● Monitoring pipelines● Adjusting pipelines● Testing and quality control3: Operationalizing machine learning models3.1 Leveraging pre-built ML models as a service. Considerations include:● ML APIs (e.g., Vision API, Speech API)● Customizing ML APIs (e.g., AutoML Vision, Auto ML text)● Conversational experiences (e.g., Dialogflow)3.2 Deploying an ML pipeline. Considerations include:● Ingesting appropriate data● Retraining of machine learning models (AI Platform Prediction and Training, BigQuery ML, Kubeflow, Spark ML)● Continuous evaluation3.3 Choosing the appropriate training and serving infrastructure. Considerations include:● Distributed vs. single machine● Use of edge compute● Hardware accelerators (e.g., GPU, TPU)3.4 Measuring, monitoring, and troubleshooting machine learning models. Considerations include:● Machine learning terminology (e.g., features, labels, models, regression, classification, recommendation, supervised and unsupervised learning, evaluation metrics)● Impact of dependencies of machine learning models● Common sources of error (e.g., assumptions about data) 4: Ensuring solution quality4.1 Designing for security and compliance. Considerations include:● Identity and access management (e.g., Cloud IAM)● Data security (encryption, key management)● Ensuring privacy (e.g., Data Loss Prevention API)● Legal compliance (e.g., Health Insurance Portability and Accountability Act (HIPAA), Children's Online Privacy Protection Act (COPPA), FedRAMP, General Data Protection Regulation (GDPR))4.2 Ensuring scalability and efficiency. Considerations include:● Building and running test suites● Pipeline monitoring (e.g., Cloud Monitoring)● Assessing, troubleshooting, and improving data representations and data processing infrastructure● Resizing and autoscaling resources4.3 Ensuring reliability and fidelity. Considerations include:● Performing data preparation and quality control (e.g., Dataprep)● Verification and monitoring● Planning, executing, and stress testing data recovery (fault tolerance, rerunning failed jobs, performing retrospective re-analysis)● Choosing between ACID, idempotent, eventually consistent requirements4.4 Ensuring flexibility and portability. Considerations include:● Mapping to current and future business requirements● Designing for data and application portability (e.g., multicloud, data residency requirements)● Data staging, cataloging, and discovery
