|
所在平台: Udemy |
课程主页: https://www.udemy.com/course/data-engineering-etl-web-scraping-big-datasqlpower-bi/
课程评论：没有评论
课程名称：数据工程 - ETL、网络爬虫、大数据、SQL、Power BI 课程概述：许多组织面临的一个常见问题是如何从多个来源、不同格式收集数据，并将其移动到一个或多个数据存储中。提取、转换和加载（ETL）是一种数据管道，用于从各种来源收集数据，根据业务规则进行转换，然后将其加载到目标数据存储中。SQL Server集成服务（SSIS）是一种强大的商业智能工具，专为与SQL Server数据库配合使用而设计。该课程将介绍如何使用SSIS进行数据集成、数据转换、复杂业务问题解决、数据仓库更新、数据清洗、数据挖掘等操作。 网络爬虫是自动下载网页数据并提取特定信息的过程。提取的信息可以存储在数据库中或以各种文件格式保存。网络爬虫涉及获取网页内容并从中提取信息。此外，大数据可以描述为具有高容量、高多样性和高速度的数据，包含数字、文本、图像、音频、视频等信息类型。 SQL是一种标准的数据库访问和操作语言，可以执行数据库查询、检索、插入、更新、删除记录，创建数据库和表，设置权限等。本课程将提供SQL的基本操作知识。 Power BI是一种商业分析解决方案，可以帮助用户可视化数据并分享见解，支持从数百个数据源连接。课程中将介绍如何快速获取数据洞察、连接和转换基于网络的数据源，并在Power BI服务上发布和共享报告和可视化内容。 通过本课程，您将学习数据工程的基础，掌握ETL、网络爬虫、大数据处理、SQL操作及Power BI的使用，为数据分析和商业智能的工作打下坚实基础。
A common problem that organizations face is how to gathering data from multiple sources, in multiple formats, and move it to one or more data stores. The destination may not be the same type of data store as the source, and often the format is different, or the data needs to be shaped or cleaned before loading it into its final destination.Extract, transform, and load (ETL) is a data pipeline used to collect data from various sources, transform the data according to business rules, and load it into a destination data store. SQL Server Integration Services (SSIS) is a useful and powerful Business Intelligence Tool. It is best suited to work with SQL Server Database. It is added to SQL Server Database when you install SQL Server Data Tools (SSDT)which adds the Business Intelligence Templates to Visual studio that is used to create Integration projects.SSIS can be used for: Data Integration Data Transformation Providing solutions to complex Business problems Updating data warehouses Cleaning data Mining data Managing SQL Server objects and data Extracting data from a variety of sources Loading data into one or several destinationsWeb scraping is the process of automatically downloading a web page's data and extracting specific information from it. The extracted information can be stored in a database or as various file types.Web scraping software tools may access the World Wide Web directly using the Hypertext Transfer Protocol, or through a web browser. While web scraping can be done manually by a software user, the term typically refers to automated processes implemented using a bot or web crawler. It is a form of copying, in which specific data is gathered and copied from the web, typically into a central local database or spreadsheet, for later retrieval or analysis.Scraping a web page involves fetching it and extracting from it. Fetching is the downloading of a page (which a browser does when you view the page). to fetch pages for later processing. Once fetched, then extraction can take place. The content of a page may be parsed, searched, reformatted, its data copied into a spreadsheet, and so on. Web scrapers typically take something out of a page, to make use of it for another purpose somewhere else. An example would be to find and copy names and phone numbers, or companies and their URLs, to a list (contact scraping).Big data can be characterised as data that has high volume, high variety and high velocity. Data includes numbers, text, images, audio, video, or any other kind of information you might store on your computer. Volume, velocity, and variety are sometimes called "the 3 V's of big data." What kind of datasets are considered big data? Examples includes social media network analysing their members' data to learn more about them and connect them with content and advertising relevant to their interests, or search engines looking at the relationship between queries and results to give better answers to users' questions.SQL is a standard language for accessing and manipulating databases.SQL stands for Structured Query LanguageWhat Can SQL do?SQL can execute queries against a databaseSQL can retrieve data from a databaseSQL can insert records in a databaseSQL can update records in a databaseSQL can delete records from a databaseSQL can create new databasesSQL can create new tables in a databaseSQL can create stored procedures in a databaseSQL can create views in a databaseSQL can set permissions on tables, procedures, and viewsPower BI is a business analytics solution that lets you visualize your data and share insights across your organization, or embed them in your app or website. Connect to hundreds of data sources and bring your data to life with live dashboards and reports.Discover how to quickly glean insights from your data using Power BI. This formidable set of business analytics tools-which includes the Power BI service, Power BI Desktop, and Power BI Mobile-can help you more effectively create and share impactful visualizations with others in your organization.In this beginners course you will learn how to get started with this powerful toolset. We will cover topics like connecting to and transforming web based data sources. You will learn how to publish and share your reports and visuals on the Power BI service.
