|
所在平台: Udemy |
课程主页: https://www.udemy.com/course/the-mahalanobis-distance-test-for-outliers/
课程评论：没有评论
课程名称：马哈拉诺比斯距离异常值检测 课程概述： A) 课程目的 在大多数机器学习方法和算法中，分析数据集时需要研究数据项之间的“近”或“远”关系。这有助于分析人员寻找异常值、分类数据项、确定数据项之间的关联等。机器学习方法依赖于一个数学概念：数据项之间的距离。我们通常将距离视为两个点之间的长度，但数学家对距离有更广泛的理解。一个由数千客户组成的客户数据集将包含关于每个客户的一组属性。如果这些属性的数量超过三，数据点之间的距离将变成一个数学表达式，涉及每个数据项的属性值的向量计算。而且，距离的计算方法有多种，如欧几里得距离、曼哈顿距离、明科夫斯基距离和切比雪夫距离。尽管欧几里得距离是最常见的，但随着时间推移，它在多变量空间中的应用引发了一些结果上的异常。马哈拉诺比斯通过其统计学的天才，提出了在应用欧几里得距离前先对数据集进行转换的想法，从而解决了许多问题。课程的目标是介绍一种7步法，用于计算马哈拉诺比斯距离，并通过指定显著性水平（如0.1%、1%和5%）来识别异常值。 B) 为何使用Excel进行教学？ 本课程将用Excel进行教学，目的是教育而非作为机器学习工具。课程并不涵盖如何在现实中应用7步法，这需要更高级的编程环境。本课程主要旨在阐明通过马哈拉诺比斯距离测试识别异常值的过程，Excel因其简单易懂而被选为教学工具。 C) 课程内容 课程分为三个部分： 1. 课程介绍：包括课程目标、结构和资源等内容。 2. 课程核心部分：包括7节讲座，内容涵盖马哈拉诺比斯距离的介绍、欧几里得距离的矩阵表示及其缺陷、马哈拉诺比斯距离的计算步骤、协方差矩阵的条件以及如何识别异常值等。 3. 支持讲座：包括4节讲座，讲解进行7步法计算所需的先决知识，涵盖矩阵及变换、Cholesky分解、多变量数据及其参数、协方差和相关性等。 所有讲座将提供多种资源支持，包括每节课的PowerPoint演示文稿、已解决的Excel练习、专用工作簿和相关书籍和文章链接。
A) The Purpose of the CourseIn most of the Machine Learning methods and algorithms that analyze datasets, there is a need to investigate how "close" or "far" the items of the dataset are to each other. This can allow analysts to look for outliers, anomalies, classify data items into clusters, establish if there are associations between the items or not and such issues.To do that, Machine Learning methods rely on the use of a mathematical concept: the distance between items in a dataset. We are used to consider distance as the length between two points. Mathematicians have a wider use of the term. A customer dataset consisting of 1000s of customers will have a set of M attributes about each customer. M can be in the 2 digit range. If M = 1, 2 or 3, we can visualize the distance between points in terms of charts. This stops being possible for M > 3. The distance becomes a mathematical expression consisting of a vector for each item in the dataset where the vector is a set of the instances or values of the attributes for each item in the dataset.What makes this more interesting is that there are various ways distances can be calculated: Euclidian, Manhattan, Minkovsky and Chebyshev distances. The Euclidian is the most common. However, with time, Machine Learning methods using the Euclidian Distance resulted in anomalies in the results giving invalid answers to the use of the distances.Since the Euclidian Distance is calculated in multivariate space by multiplying the Transpose of the dataset, PT with the dataset P. This is where Mr. Prasad Mahalanobis with his genius in statistics, came up with the idea: why not transform that dataset P before multiplying it by its transpose. This resolve a large number of issues with the Euclidian Distance.The objective of the course is to present a 7-Step procedure used to calculate the Mahalanobis Distances and from the resulting matrix, identify the outliers. Identification will be based on specifying a significance level (such as 0.1%, 1% and 5%).The course will also provide support lectures that are required as pre-requisites or knowledge and practices needed to apply the 7 steps.B) So, why do we Present a Course based on Excel?The course will then use Excel specifically for educational purposes and not as a machine learning tool. The course is not setup to show you how to use the 7-Step in real life. That would require more advanced programming environments. The course mainly aims to clarify the procedure for identifying outliers using the Mahalanobis Distance test. For that, Excel is used as an educational tool as it is easy to understand and is well known by mots business analysts.B) What Does the Course Cover?The course is made up of 3 sectionsSection 1: Introducing the CourseThis section consists of one lecture that presents the objectives of the course, its structure and resources as well as what to expect and what not to expect.Section 2: This is the heart of the course and consists of 7 Lectures:2) Introducing Distances, Specifically the Mahalanobis Distance A. Introducing Prasanta Chandra Mahalanobis B. Introducing Mahalanobis Distance C. The Importance of Measuring Distance between Items of Data3) Practices in our Data and the Matrix Representation of the Euclidian Distance D. Introducing Some Terms and Practices in our Data E. Starting with the Matrix Representation of the Euclidian Distance4) Shortcoming of Euclidian Distances F. Shortcoming of Euclidian Distances5) The 7-Step Procedure for Calculating Mahalanobis Distances G. The 7-Step Procedure for Calculating Mahalanobis Distances6) Conditions for the Covariance Matrix to be Positive Definite H. Interlude: Conditions for the Covariance Matrix to be Positive Definite7) How to Identify Outliers and More Examples I. Calculating the Mahalanobis Distance for the Two Equidistance Points J. How to Identify Outliers by using the Chi-Square DistributionSection 3: Support PresentationsThis section consists of 4 lectures covering material that should be known, as a pre-requisite, to appreciate and use the calculations in the 7-Step Procedure that results in identification of outliers through the Mahalanobis Distance test:8) Support - Matrices and Transformation9) Support - The Cholesky Decomposition10) Support - Multivariate Data and their Parameters K. Introducing Univariate and Bivariate Data and their Parameters L. Calculating the Variance, Standard Deviation and Covariance11) Support - Covariance and Correlation M. The Covariance Matrix N. 6 Methods for Calculating the Covariance Matrix O. Correlation P. How to Calculate the Correlation Coefficient RResourcesAll lectures will be supported by a variety of resources:· Each lecture will have its PowerPoint presentation uploaded in PDF format for your later use· Solved and documented workouts in Excel· Dedicated workbooks that animate and describe various probability distributions· Links to Interesting articles and books
