|
所在平台: Udemy |
课程主页: https://www.udemy.com/course/automate-web-scraping-using-python-scripts-and-spiders/
课程评论：没有评论
课程名称：利用Python脚本和爬虫自动化网络爬取 课程概述：网络爬取是自动下载网页数据并提取特定信息的过程。提取的信息可以存储在数据库中或以各种文件类型保存。在进行网络爬取时，遵循基本规则很重要：首先，务必检查网站的条款与条件，以避免法律问题；其次，合理使用程序请求数据，避免过于激进的请求（如垃圾邮件行为）。网站的布局可能会随着时间变化，因此需要确保代码能够适应这些改变。 本课程中将介绍一些流行的网络爬取工具，包括BeautifulSoup和Scrapy。BeautifulSoup是一个用于从HTML和XML文件中提取数据的Python库，而Scrapy是一个开源框架，适用于爬取网站和提取结构化数据，可用于数据挖掘、研究、信息处理或历史归档。 网络爬取工具可以直接通过超文本传输协议访问万维网，也可以通过网络浏览器进行。虽然网络爬取可以由软件用户手动进行，但通常指的是采用自动化的方式，通过机器人或网络爬虫实现的过程。网络爬取是一种复制形式，特定数据从网络上收集并复制，通常存入中心本地数据库或电子表格，以供后续检索或分析。 爬取网页的过程包括获取页面和从中提取信息。获取是指下载网页（类似于用户用浏览器查看网页时的过程），下载后可以进行信息的提取与处理，比如解析内容、搜索、重新格式化，将数据复制到电子表格等。常见的爬虫应用包括联系人信息爬取、网页索引、在线价格变动监测、产品评论爬取、实时房地产信息获取、天气数据监控、网站变更检测、研究和在线声誉追踪等。 网页使用基于文本的标记语言（如HTML和XHTML）构建，通常蕴含大量有用的文本数据。网络爬虫作为应用程序接口（API），用于从网站提取数据。亚马逊AWS和谷歌等公司提供多种网络爬取工具、服务及公共数据，终端用户可以免费使用。
Web scraping is the process of automatically downloading a web page's data and extracting specific information from it. The extracted information can be stored in a database or as various file types. Basic Scraping Rules: Always check a website's Terms and Conditions before you scrape it to avoid legal issues. Do not request data from a website too aggressively (spamming) with your program as this may break the website. The layout of a website may change from time to time ,so make sure your code adapts to it when it does.Popular web scraping tools include BeautifulSoup and Scrapy.BeautifulSoup is a python library for pulling data (parsing) out of HTML and XML files.Scrapy is a free open source application framework used for crawling web sites and extracting structured datawhich can be used for a variety of things like data mining,research ,information process or historical archival. Web scraping software tools may access the World Wide Web directly using the Hypertext Transfer Protocol, or through a web browser. While web scraping can be done manually by a software user, the term typically refers to automated processes implemented using a bot or web crawler. It is a form of copying, in which specific data is gathered and copied from the web, typically into a central local database or spreadsheet, for later retrieval or analysis.Scraping a web page involves fetching it and extracting from it. Fetching is the downloading of a page (which a browser does when you view the page). to fetch pages for later processing. Once fetched, then extraction can take place. The content of a page may be parsed, searched, reformatted, its data copied into a spreadsheet, and so on. Web scrapers typically take something out of a page, to make use of it for another purpose somewhere else. An example would be to find and copy names and phone numbers, or companies and their URLs, to a list (contact scraping).Web scraping is used for contact scraping, and as a component of applications used for web indexing, web mining and data mining, online price change monitoring and price comparison, product review scraping (to watch the competition), gathering real estate listings, weather data monitoring, website change detection, research, tracking online presence and reputation, web mashup and, web data integration.Web pages are built using text-based mark-up languages (HTML and XHTML), and frequently contain a wealth of useful data in text form.. A web scraper is an Application Programming Interface (API) to extract data from a web site. Companies like Amazon AWS and Google provide web scraping tools, services and public data available free of cost to end users.
