Apache Spark : Master Big Data with PySpark and DataBricks

Learn Pyspark, streaming using Kafka, Delta lake, crazy optimization techniques, NLP, time series, distributed computing

This course is designed to help you develop the skill necessary to perform ETL operations in Databricks using pyspark, build production ready ML models, learn spark optimization techniques and master distributed computing.

What you’ll learn

  • Learn the Spark Architecture.
  • What is distributed computing.
  • Learn Spark Transformations and Actions using the Structured API.
  • Learn Spark on Databricks.
  • Spark optimization techniques.
  • Data Lake House architecture.
  • Spark structured streaming using Kafka.
  • Information retriever system using word2vec.
  • Sentiment analysis using pyspark.
  • Training hundreds of time series forecasting models in parallel with Prophet and Spark.

Course Content

  • Introduction –> 4 lectures • 5min.
  • Spark architecture –> 4 lectures • 18min.
  • Spark Transformations – Demo –> 6 lectures • 39min.
  • Spark Actions – Demo –> 1 lecture • 6min.
  • Spark User Defined Functions –> 2 lectures • 26min.
  • Building Blocks of Apache Spark –> 3 lectures • 27min.
  • Spark Optimizations techniques –> 7 lectures • 42min.
  • Adaptive query execution –> 2 lectures • 12min.
  • Data Lake house Architecture –> 4 lectures • 23min.
  • Spark Structured Streaming –> 1 lecture • 14min.
  • USE CASE : Spark Structured Streaming with Kafka –> 2 lectures • 27min.
  • USE CASE : Natural Language Processing –> 6 lectures • 45min.
  • Training hundreds of time series forecasting models in parallel with spark –> 2 lectures • 11min.

Apache Spark : Master Big Data with PySpark and DataBricks

Requirements

  • Python.

This course is designed to help you develop the skill necessary to perform ETL operations in Databricks using pyspark, build production ready ML models, learn spark optimization techniques and master distributed computing.

 

Big Data engineering:

 

Big data engineers interact with massive data processing systems and databases in large-scale computing environments. Big data engineers provide organizations with analyses that help them assess their performance, identify market demographics, and predict upcoming changes and market trends.

 

Azure Databricks:

 

Azure Databricks is a data analytics platform optimized for the Microsoft Azure cloud services platform. Azure Databricks offers three environments for developing data intensive applications: Databricks SQL, Databricks Data Science & Engineering, and Databricks Machine Learning.

 

Data Lake House:

 

A data lakehouse is a data solution concept that combines elements of the data warehouse with those of the data lake. Data lakehouses implement data warehouses’ data structures and management features for data lakes, which are typically more cost-effective for data storage .

 

Spark structured streaming:

 

Structured Streaming is a scalable and fault-tolerant stream processing engine built on the Spark SQL engine. .In short, Structured Streaming provides fast, scalable, fault-tolerant, end-to-end exactly-once stream processing without the user having to reason about streaming.

 

Natural language processing:

 

Natural Language Processing, or NLP for short, is broadly defined as the automatic manipulation of natural language, like speech and text, by software.

 

The study of natural language processing has been around for more than 50 years and grew out of the field of linguistics with the rise of computers.

 

 

Get Tutorial