课程详情

The key objectives of this course are as follows;Learn the Spark ArchitectureLearn Spark Execution ConceptsLearn Spark Transformations and Actions using the Structured APILearn Spark Transformations and Actions using the RDD (Resilient Distributed Datasets) APILearn how to set up your own local PySpark EnvironmentLearn how to interpret the Spark Web UILearn how to interpret DAG (Directed Acyclic Graph) for Spark ExecutionLearn the RDD (Resilient Distributed Datasets) API (Crash Course)RDD TransformationsRDD ActionsLearn the Spark DataFrame API (Structured APIs)Create Schemas and Assign DataTypesRead and Write Data using the DataFrame Reader and WriterRead Semi-Structured Data such as JSONCreate and New Data Columns to the DataFrame using ExpressionsFilter the DataFrame using the "Filter" and "Where" TransformationsEnsure that the DataFrame has unique rowsDetect and Drop DuplicatesAugment the DataFrame by Adding New Rows Combine 2 or More DataFramesOrder the DataFrame by Specific ColumnsRenaming and Drop Columns from the DataFrameClean the DataFrame by detecting and Removing Missing or Bad DataCreate User-Defined Spark FunctionsRead and Write to/from Parquet FilePartition the DataFrame and Write to Parquet FileAggregate the DataFrame using Spark SQL functions (count, countDistinct, Max, Min, Sum, SumDistinct, AVG)Perform Aggregations with GroupingLearn Spark SQL and DatabricksCreate a Databricks AccountCreate a Databricks ClusterCreate Databricks SQL and Python NotebooksLearn Databricks shortcutsCreate Databases and Tables using Spark SQLUse DML, DQL, and DDL with Spark SQLUse Spark SQL FunctionsLearn the differences between Managed and Unmanaged TablesRead CSV Files from the Databricks File SystemLearn to write Complex SQLUse Spark SQL FunctionsCreate Visualisations with DatabricksCreate a Databricks DashboardThe Python Spark project that we are going to do together;Sales DataCreate a Spark SessionRead a CSV file into a Spark DataframeLearn to Infer a SchemaSelect data from the Spark DataframeProduce analytics that shows the topmost sales orders per Region and CountryConvert Fahrenheit to Degrees CentigradeCreate a Spark SessionRead and Parallelize data using the Spark Context into an RDDCreate a Function to Convert Fahrenheit to Degrees CentigradeUse the Map Function to convert data contained within an RDDFilter temperatures greater than or equal to 13 degrees celsiusXYZ ResearchCreate a set of RDDs that hold Research DataUse the union transformation to combine RDDsLearn to use the subtract transformation to minus values from an RDDUse the RDD API to answer the following questionsHow many research projects were initiated in the first three years?How many projects were completed in the first year?How many projects were completed in the first two years?Sales AnalyticsCreate the Sales Analytics DataFrame to a set of CSV FilesPrepare the DataFrame by applying a StructureRemove bad records from the DataFrame (Cleaning)Generate New Columns from the DataFrameWrite a Partitioned DataFrame to a Parquet DirectoryAnswer the following questions and create visualizations using Seaborn and MatplotlibWhat was the best month in sales?What city sold the most products?What time should the business display advertisements to maximize the likelihood of customers buying products?What products are often sold together in the state "NY"? Technology SpecPythonJupyter NotebookJupyter LabPySpark (Spark with Python)PandasMatplotlibSeaborneDatabricksSQL

课程标签

0人关注该课程

主题相关的课程