|
所在平台: Udemy |
课程主页: https://www.udemy.com/course/azure-databricks-data-engineering-with-real-time-project/
课程评论：没有评论
课程名称: Azure DataBricks - 数据工程与实时项目 课程概述: 完成此课程后，您将掌握数据工程师在实际项目中的角色与责任，包括以下内容： - 设计和配置 Unity Catalogue，以实现更好的访问控制和连接外部数据存储。 - 设计和开发 Databricks (PySpark) 笔记本，以从 Web (HTTP) 服务中摄取数据。 - 设计和开发 Databricks (PySpark) 笔记本，以从 SQL 数据库中摄取数据。 - 设计和开发 Databricks (PySpark) 笔记本，以从 API 源系统中摄取数据。 - 设计和开发 Spark SQL 外部和管理表。 - 开发可重用的 Databricks Spark SQL 笔记本，以创建和填充 Delta Lake 表。 - 开发 Databricks SQL 代码以填充报告维度表。 - 开发 Databricks SQL 代码以填充报告 SCD 类型 2 维度表。 - 开发 Databricks SQL 代码以填充报告事实表。 - 设计和开发 Databricks (PySpark) 笔记本，以使用 EXPLODE 函数处理和扁平化半结构化 JSON 数据。 - 设计和开发 Databricks (PySpark) 笔记本，以集成数据并加载到 Datalake 金层。 - 设计和开发 Databricks (PySpark) 笔记本，以处理 Datalake 银层中的半结构化 JSON 数据。 - 设计和开发 Databricks (SQL) 笔记本，以集成数据并加载到 Datalake 金层。 - 开发 Databricks 作业用于调度数据摄取和转换笔记本。 - 设计和配置 Delta Live Tables，以实现无缝数据集成。 - 设置 Azure Monitor 和日志分析，以自动监控作业运行和存储扩展日志详细信息。 - 设置 Azure Key Vault 并在 Databricks 工作区中配置 Key Vault 支持的秘密范围。 - 配置 GitHub 存储库并在 Databricks 工作区中创建 Git Repo 文件夹。 - 设计和配置 CI/CD 流水线，以将代码发布到多个环境。 - 识别性能瓶颈并使用 ZORDER BY、BROADCAST JOIN、ADAPTIVE QUERY EXECUTION、DATA SALTING 和 LIQUID CLUSTERING 进行性能优化。 该课程将帮助您深入理解 Azure DataBricks 的数据工程技术，并在实际项目中应用所学的技能。
By Completing this course you will be equipped with below Data Engineer Roles & Responsibilities in the real time project• Designing and Configuring Unity Catalogue for Better Access Control & Connecting to External Data Stores• Designing and Developing Databricks(PySpark) Notebooks to Ingest the data from Web(HTTP) Services• Designing and Developing Databricks(PySpark) Notebooks to Ingest the data from SQL Databases• Designing and Developing Databricks(PySpark) Notebooks to Ingest the data from API source Systems• Designing and Developing Spark SQL External and Managed Tables• Developed Databricks Spark SQL Reusable Notebooks To Create and populate Delta Lake Tables• Developed Databricks SQL Code to populate Reporting Dimension tables• Developed Databricks SQL Code to populate Reporting SCD Type 2 Dimension tables• Developed Databricks SQL Code to populate Reporting Fact Table• Designing and Developing Databricks(PySpark ) Notebooks to Process and Flatten Semi Structured JSON Data using EXPLODE function • Designing and Developing Databricks(PySpark ) Notebooks to Integrate(JOIN) Data and load into Datalake Gold Layer• Designing and Developing Databricks(PySpark) Notebooks to Process Semi Structured JSON Data in DataLake Silver Layer• Designing and Developing Databricks(SQL) Notebooks to Integrate Data and load into Datalake Gold Layer• Developed Databricks Jobs for Scheduling the Data Ingestion and Transformation Notebooks• Designing and Configuring Delta Live Tables in all layers for seamless Data Integration• Setup Azure Monitor and Log Analytics for Automated Monitoring of Job Runs and Stored Extended Log Details• Setup Azure Key Vault and Configure Key Vault Backed Secret Scopes in Databricks Workspace• Configuring GitHub Repository and creating Git Repo Folders in Databricks Workspace• Designing and Configuring CI/CD Pipelines to release the code into multiple environment• Identifying performance bottle necks and perform the performance tuning using ZORDER BY , BROADCAST JOIN , ADAPTIVE QUERY EXECUTION , DATA SALTING and LIQUID CLUSTERING
