|
所在平台: Udemy |
课程主页: https://www.udemy.com/course/microsoft-azure-dp-203-certification-practice-exam/
课程评论：没有评论
本课程是为有志于通过微软Azure DP-203认证考试的数据工程师量身打造的实践练习课程。 **课程概述：** 本课程将深入探讨Azure数据工程师的核心职责，包括： * **数据源识别与摄取：** 学习如何从各种数据源高效地摄取数据。 * **数据处理与存储：** 掌握使用Azure数据服务和框架来存储、清洗和增强数据集的技巧，以满足不同的业务需求，支持现代数据仓库（MDW）、大数据或Lakehouse等多种架构模式。 * **解决方案构建与维护：** 学习如何使用Azure Data Factory, Azure Synapse Analytics, Azure Stream Analytics, Azure Event Hubs, Azure Data Lake Storage, and Azure Databricks等工具和技术，构建和维护安全合规的数据处理管道。 * **数据探索与洞察：** 助力利益相关者通过数据探索理解数据。 * **性能优化与可靠性：** 确保数据管道和数据存储的高性能、高效率、有组织和可靠性，并识别和解决操作及数据质量问题。 **课程重点内容（按比例分配）：** * **设计和实现数据存储 (15-20%)** * 文件、分析工作负载、流工作负载以及Azure Synapse Analytics的分区策略。 * 确定Azure Data Lake Storage Gen2何时需要分区。 * 设计和实现数据探索层，创建和执行查询，推荐和实现Azure Synapse Analytics数据库模板。 * 将新的或更新的数据血缘推送到Microsoft Purview，并在Microsoft Purview Data Catalog中浏览和搜索元数据。 * **开发数据处理 (40-45%)** * **数据摄取与转换：** 使用Azure Synapse Pipelines/Data Factory、Azure Databricks（Spark）和Azure Stream Analytics进行增量加载、数据转换、数据清洗（处理重复、缺失、延迟到达数据、拆分、解析JSON、编码解码）、错误处理、数据规范化与反规范化，以及探索性数据分析。 * **批处理解决方案：** 使用Azure Data Lake Storage, Azure Databricks, Azure Synapse Analytics, Azure Data Factory开发批处理解决方案，利用PolyBase加载数据到SQL池，实现Azure Synapse Link并查询复制数据，创建和管理数据管道（包括资源扩展、批量大小配置、管道测试、Jupyter/Python notebooks集成、Upsert/Revert数据、异常处理、批次保留、读写Delta Lake）。 * **流处理解决方案：** 使用Stream Analytics和Azure Event Hubs创建流处理解决方案，处理窗口聚合、Schema Drift、时序数据、跨分区数据以及分区内数据，配置检查点/水印，优化管道，处理中断和异常，Upsert数据，重播存档流数据。 * **批次与管道管理：** 管理和触发批次，处理失败的批次加载，验证批次加载，管理Azure Data Factory/Synapse Pipelines中的数据管道，调度管道，实现管道构件的版本控制，管理Spark作业。 * **安全、监控和优化数据存储及数据处理 (30-35%)** * **数据安全：** 实现数据屏蔽、数据加密（静态和传输中）、行级和列级安全、Azure RBAC、Data Lake Storage Gen2的POSIX-like ACLs、数据保留策略、安全端点（私有和公开）、Azure Databricks中的资源令牌。 * **数据存储和处理监控：** 使用Azure Monitor实现日志记录，配置监控服务，测量数据移动性能，监控系统数据统计信息，测量数据管道性能，测量查询性能，调度和监控管道测试，解读Azure Monitor指标和日志，实现管道警报策略。 * **优化和故障排除：** 优化存储（压缩小文件、处理数据倾斜、处理数据溢出），优化资源管理，调优查询（使用索引器、缓存），故障排除Spark作业失败、管道运行失败（包括外部服务活动）。 **先决条件：** 候选人应具备扎实的SQL、Python和Scala等数据处理语言知识，理解并行处理和数据架构模式。 **总结：** 本课程将全方位提升您在Azure数据领域的专业技能，帮助您为DP-203认证考试做好充分准备，并在数据工程领域取得成功。
As a data engineer working on Azure, you will be responsible for managing various data-related tasks such as identifying data sources, ingesting data from various sources, processing data, and storing data in different formats. You will also be responsible for building and maintaining secure and compliant data processing pipelines using various tools and techniques.Azure data engineers use a variety of Azure data services and frameworks to store and produce cleansed and enhanced datasets for analysis. Depending on the business requirements, data stores can be designed with different architecture patterns, including modern data warehouse (MDW), big data, or Lakehouse architecture.Azure data engineers help stakeholders understand the data through exploration, and they build and maintain secure and compliant data processing pipelines by using different tools and techniques. These professionals use various Azure data services and frameworks to store and produce cleansed and enhanced datasets for analysis. This data store can be designed with different architecture patterns based on business requirements, including modern data warehouse (MDW), big data, or bakehouse architecture.Azure data engineers also help to ensure that the operationalization of data pipelines and data stores are high-performing, efficient, organized, and reliable, given a set of business requirements and constraints. These professionals help to identify and troubleshoot operational and data quality issues. They also design, implement, monitor, and optimize data platforms to meet the data pipelines.Candidates for this exam must have solid knowledge of data processing languages, including SQL, Python, and Scala, and they need to understand parallel processing and data architecture patterns. They should be proficient in using Azure Data Factory, Azure Synapse Analytics, Azure Stream Analytics, Azure Event Hubs, Azure Data Lake Storage, and Azure Data bricks to create data processing solutions.Design and implement data storage (15-20%)Develop data processing (40-45%)Secure, monitor, and optimize data storage and data processing (30-35%)Design and implement data storage (15-20%)Implement a partition strategyImplement a partition strategy for filesImplement a partition strategy for analytical workloadsImplement a partition strategy for streaming workloadsImplement a partition strategy for Azure Synapse AnalyticsIdentify when partitioning is needed in Azure Data Lake Storage Gen2Design and implement the data exploration layerCreate and execute queries by using a compute solution that leverages SQL serverless and Spark clusterRecommend and implement Azure Synapse Analytics database templatesPush new or updated data lineage to Microsoft PurviewBrowse and search metadata in Microsoft Purview Data CatalogDevelop data processing (40-45%)Ingest and transform dataDesign and implement incremental loadsTransform data by using Apache SparkTransform data by using Transact-SQL (T-SQL) in Azure Synapse AnalyticsIngest and transform data by using Azure Synapse Pipelines or Azure Data FactoryTransform data by using Azure Stream AnalyticsCleanse dataHandle duplicate dataHandle missing dataHandle late-arriving dataSplit dataShred JSONEncode and decode dataConfigure error handling for a transformationNormalize and denormalize dataPerform data exploratory analysisDevelop a batch processing solutionDevelop batch processing solutions by using Azure Data Lake Storage, Azure Databricks, Azure Synapse Analytics, and Azure Data FactoryUse PolyBase to load data to a SQL poolImplement Azure Synapse Link and query the replicated dataCreate data pipelinesScale resourcesConfigure the batch sizeCreate tests for data pipelinesIntegrate Jupyter or Python notebooks into a data pipelineUpsert dataRevert data to a previous stateConfigure exception handlingConfigure batch retentionRead from and write to a delta lakeDevelop a stream processing solutionCreate a stream processing solution by using Stream Analytics and Azure Event HubsProcess data by using Spark structured streamingCreate windowed aggregatesHandle schema driftProcess time series dataProcess data across partitionsProcess within one partitionConfigure checkpoints and watermarking during processingScale resourcesCreate tests for data pipelinesOptimize pipelines for analytical or transactional purposesHandle interruptionsConfigure exception handlingUpsert dataReplay archived stream dataManage batches and pipelinesTrigger batchesHandle failed batch loadsValidate batch loadsManage data pipelines in Azure Data Factory or Azure Synapse PipelinesSchedule data pipelines in Data Factory or Azure Synapse PipelinesImplement version control for pipeline artifactsManage Spark jobs in a pipelineSecure, monitor, and optimize data storage and data processing (30-35%)Implement data securityImplement data maskingEncrypt data at rest and in motionImplement row-level and column-level securityImplement Azure role-based access control (RBAC)Implement POSIX-like access control lists (ACLs) for Data Lake Storage Gen2Implement a data retention policyImplement secure endpoints (private and public)Implement resource tokens in Azure DatabricksLoad a DataFrame with sensitive informationWrite encrypted data to tables or Parquet filesManage sensitive informationMonitor data storage and data processingImplement logging used by Azure MonitorConfigure monitoring servicesMonitor stream processingMeasure performance of data movementMonitor and update statistics about data across a systemMonitor data pipeline performanceMeasure query performanceSchedule and monitor pipeline testsInterpret Azure Monitor metrics and logsImplement a pipeline alert strategyOptimize and troubleshoot data storage and data processingCompact small filesHandle skew in dataHandle data spillOptimize resource managementTune queries by using indexersTune queries by using cacheTroubleshoot a failed Spark jobTroubleshoot a failed pipeline run, including activities executed in external servicesJoin us on this transformative journey into Azure Data Engineering, empowering yourself with the knowledge and skills to conquer the DP-203 Exam and excel in your data engineering career.
