|
所在平台: Udemy |
课程主页: https://www.udemy.com/course/master-web-scraping/
课程评论：没有评论
课程名称：Scrapy大师班：使用Scrapy框架学习网络爬虫 概述：网络爬虫是从网站中提取所需数据的过程。在本课程中，您将通过逐步深入的指导，学习并掌握使用Python和Scrapy框架进行网络爬虫。课程从基础开始，假设您对网络爬虫、网络爬取或Scrapy框架一无所知。在第一部分，您将逐步了解网络爬虫的流程（以信息图的形式展示，无需编码），学习如何从网站抓取数据以及如何使用Scrapy进行抓取。 在掌握基础知识并了解网络爬虫的工作原理后，我们将开始使用Python和Scrapy框架进行实际的网络爬虫。课程内容将分成小节，逐步进行，确保您能轻松理解抓取和提取数据的每个步骤。 接下来，在掌握基本的网络爬虫工具后，您将了解Scrapy的重要概念，包括： - CSS选择器选择网页元素 - XPath选择器选择网页元素 - Scrapy Shell测试和验证选择器 - Items用于组织提取的数据 - 使用ItemLoaders进行数据处理 - 导出数据到JSON、CSV、XLSX（Excel）和XML文件格式 - 使用ItemPipelines将提取的数据保存到在线数据库，如MongoDB 在深入学习网络爬虫后，您将掌握更高级的爬虫技术，成为正式的网络爬虫专家，包括： - 遍历网页中的链接 - 爬取多个页面和提取数据（分页） - 使用正则表达式（RegEx）抓取数据 - 登录网站并通过Scrapy FormRequest发送请求 - 绕过CSRF保护的登录表单 - 使用Scrapy Playwright抓取动态或JavaScript渲染的网站 - 与网页元素互动，如填写表单、点击按钮等 - 处理无限滚动的网站 - 等待加载内容的数据 - 截取网站的屏幕截图 - 将网站保存为PDF - 识别网站的API调用并抓取API数据 - 在Scrapy项目中使用中间件 - 配置Scrapy项目中的设置 - 使用和轮换用户代理及代理 课程还将介绍网络爬虫的最佳实践。为了巩固所学的知识，您将完成三个实际项目： - 联赛冠军表（ESPN） - 产品跟踪器（Amazon） - 爬虫应用（GUI） 加入我们，在这门深入的课程中，您将从零开始学习网络爬虫，逐步掌握从网站提取数据的过程。欢迎观看预览课程，了解网络爬虫的工作原理！期待在课程中见到您！
Web scraping is the process of scraping websites and extracting desired data from the same, and in this course, you'll learn and master web scraping using python and scrapy framework with a step-by-step and in-depth guide.A Step-By-Step GuideAssuming that you know nothing about web scraping, web crawling, scrapy framework, web scraping, or even web scrapping, we will start from the complete basics. In the first section, you'll learn about the web scraping process step-by-step (with infographics - no code), how to scrape data from websites and how to use scrapy for the same (i.e. scrapy meaning).After getting the basics clear and having an idea of how web scraping works, we will start web scraping using python & scrapy framework! Again, we'll move step-by-step and perform each step learned in the basics with bite-sized lessons. We'll take it slow so that it's easier for you to understand every step involved in scraping and extracting data from websites.Web Scraping & Scrapy EssentialsHaving built an actual web scraper, you'll get an idea of how web scraping works first-hand. Now it's crucial to cover the essential concepts of web scraping and scrapy, which we will do next.CSS Selectors to select web elementsXPath to select web elements.Scrapy Shell to test & verify selectors.Items to organise extracted dataLoad Items with ItemLoaders with input & output ProcessorsExport data to JSON, CSV, XLSX (Excel) & XML file formatsSave extracted data to online databases like MongoDB using ItemPipelines.Master Web Scraping In-DepthLearning how to scrape websites and the essentials already makes you a complete web scraper, but we'll take this even further and learn the advanced web scraping techniques to become an expert!Follow links in a webpage to another page.Crawl multiple pages and extract data, i.e. Pagination.Scrape data using Regular Expressions (RegEx)Extract Data From HTML TablesLogin Into Websites Using Scrapy FormRequestBypass CSRF-protected Login forms.Scrape Dynamic or JavaScript Rendered Websites using Scrapy Playwright.Interact with web elements like fill forms, click buttons, etc.Handle Infinite Scroll websites.Wait For Elements when contents/data take time to loadTake Screenshots of websites.Save websites as PDFs.Identify API calls from websites and scrape data from APIsUse middleware in a scrapy project.Configure settings in a scrapy projectUse and Rotate User-Agents & ProxiesWeb scraping Best PracticesReal-World ProjectsAfter mastering web scraping and web crawling, we need projects to start! That's why you'll perform three projects as well:Champions League Table [ ESPN ]Product Tracker [ Amazon ]Scraper Application [ GUI ]Join us in this in-depth course, where you'll learn about web scraping from scratch and master the process of extracting data from websites step-by-step. Check out the preview lessons to learn how web scraping works! See you there~
