课程详情

The main objective of this course is to help you to learn Data Engineering techniques of building Metadata-Driven frameworks with Azure Data Engineering tools such as Data Factory, Azure SQL, and others.Building Frameworks are now an industry norm and it has become an important skill to know how to visualize, design, plan and implement data frameworks.The framework that we are going to build together is referred to as the Metadata-Driven Ingestion Framework. Data ingestion into the data lake from the disparate source systems is a key requirement for a company that aspires to be data-driven, and finding a common way to ingest the data is a desirable and necessary requirement. Metadata-Driven Frameworks allow a company to develop the system just once and it can be adopted and reused by various business clusters without the need for additional development, thus saving the business time and costs. Think of it as a plug-and-play system.The first objective of the course is to onboard you onto the Azure Data Factory platform to help you assemble your first Azure Data Factory Pipeline. Once you get a good grip of the Azure Data Factory development pattern, then it becomes easier to adopt the same pattern to onboard other sources and data sinks.Once you are comfortable with building a basic azure data factory pipeline, as a second objective we then move on to building a fully-fledged and working metadata-driven framework to make the ingestion more dynamic, and furthermore, we will build the framework in such a way that you can audit every batch orchestration and individual pipeline runs for business intelligence and operational monitoring.Creating your first PipelineWhat will be covered is as follows;1. Introduction to Azure Data Factory2. Unpack the requirements and technical architecture3. Create an Azure Data Factory Resource4. Create an Azure Blob Storage account5. Create an Azure Data Lake Gen 2 Storage account6. Learn how to use the Storage Explorer7. Create Your First Azure Pipeline.Metadata Driven Ingestion1. Unpack the theory on Metadata Driven Ingestion2. Describing the High-Level Plan for building the User3. Creation of a dedicated Active Directory User and assigning appropriate permissions4. Using Azure Data Studio5. Creation of the Metadata Driven Database (Tables and T-SQL Stored Procedure)6. Applying business naming conventions7. Creating an email notifications strategy8. Creation of Reusable utility pipelines9. Develop a mechanism to log data for every data ingestion pipeline run and also the batch itself10. Creation of a dynamic data ingestion pipeline11. Apply the orchestration pipeline12. Explanation of T-SQL Stored Procedures for the Ingestion Engine 13. Creating an Azure DevOps Repository for the Data Factory PipelinesEvent-Driven Ingestion1. Enabling the Event Grid Provider2. Use the Getmetadata Activity3. Use the Filter Activity4. Create Event-Based Triggers5. Create and Merge new DevOps BranchesBonus Course: Provision Infra with Azure BICEPThe goal of this course is to help students learn how to professionally write and develop Azure DevOps Infrastructure as Code with BICEP, YAML, Git, and PowerShell.Azure DevOps is a leading automation and DevOps platform and the students will be taken through the following;An in-depth introduction to Infrastructure as Code with the Azure DevOps platformA definition of DevOps and how Azure as a SaaS (Software as a Service) platform facilitates the practice of the DevOps methodologyAn Introduction to YAML pipelines on the Azure DevOps platformAn Introduction to BICEP and ARM templates for developing Infrastructure as Code (IaC) on the Azure DevOps PlatformAn overview of Industry leading DevOps toolsGit is an industry-leading distributed version control system and is a very critical component of Azure DevOps therefore students will be taken through a Git Crash Course that covers the following basic aspects;The creation of a local Git RepositoryLearn how to stage and commit single and multiple filesBranching management with Git including MergingGit with Bash and Visual Studio CodeLearn how to time travel and undo changesStudents may find it necessary to learn about how to set up Azure DevOps Pipeline Agents as Self-Hosted Azure DevOps agents for running CI / CD pipelines, perhaps the situation could be cost saving in a work environment or a cost-effective personal environment, and therefore the students will learn the following;Set up Billing for Microsoft and Self Hosted pipeline agentsInstallation and Set Up for a Self Hosted pipeline agentsSetting up of a Personal Access TokenConfiguration of a Self-Hosted AgentYAML is a leading configuration management technology for developing CI / CD pipelines, perhaps the best way to learn how to write YAML pipelines is for the student to be taken through how to provision infrastructure with YAML, Powershell, and BICEP. The initial focus will be the provisioning of the resource group and there and therefore the students will learn the following;How to Create an Azure Service ConnectionCloning an Azure DevOps RepositoryWriting PowerShell Script to Provision a Resource GroupHow to Add Stages, Jobs, and Steps in a YAML pipeline templateRunning the YAML pipeline on Azure DevOpsHow to develop Azure Variables Group and pass them into YAML templatesHow to override BICEP parameters using YAMLOne aspect of professionalism in coding is how projects are structured for coding efficiency and ease of management, the other aspect in the naming convention of resources. The course will take through students on the following.Creating Project Structures for a DevOps and BICEP project using Bash and GitEstablish a standard naming convention for resources using BICEP and PowerShellThe heart of provisioning and deploying infrastructure in Azure is the adoption of BICEP, and students will learn the following in terms of developing BICEP in a professional manner;Development of a BICEP template to provision Log Analytics and Data FactoryHow to add Input Parameters to a BICEP templateHow to create BICEP Modules for Log Analytics and Data FactoryHow to add Tagging Information to BICEP modulesHow to structure a naming convention with BICEPHow to use run time and compile time variables and parametersHow to write a PowerShell Script to Transpile BICEP to an ARM templateHow to Manage Dependencies between Resources with BICEPHow to manage BICEP template errors

课程标签

0人关注该课程

主题相关的课程