Skill Sigma || Azure Data Bricks

Why Azure - Data Bricks is so Important

Industry growth

$16 billion by 2025

The Indian big data analytics market is expected to reach $16 billion by 2025, growing at a CAGR of 26%, as per a NASSCOM report. This growth is driven by increased adoption of data-driven decision-making in businesses.

Cloud services market

CAGR of 24.1%

According to IDC, the public cloud services market in India is projected to reach $10.8 billion by 2025, growing at a CAGR of 24.1%. Azure is one of the leading cloud service providers in India, driving demand for Azure-related skills.

The AI market in India

Grow at 20.2%

The AI market in India is expected to grow at a CAGR of 20.2% to reach $7.8 billion by 2025, according to NASSCOM. Skills in AI and machine learning, integrated with platforms like Azure Databricks, are highly sought after in the Indian job market.

Course Curriculum

MODULE 1 (PySpark and Python)

Hadoop Vs Spark
MapReduce limitations
Spark History
Spark Architecture
Spark and Hadoop Advantages
Benefits of Spark + Hadoop
Introduction to Spark Eco-system
Persistence in Spark
HDFS data from Spark

Spark Cluster Architecture
Spark Master & Worker node Service Spark DAG
Spark Executor

Understanding RDD
Loading data into RDD
Scala RDD, Paired RDD, Double RDD & General RDD Functions Implementing HadoopRDD,
Filtered RDD, Joined RDD Transformations, Actions and Shared Variables
Spark Operations on YARN
Sequence File Processing
Lazy Evaluation

What are Transformations?
Different transformations with example e.g. map,filter,flatmap What are Actions?
Code example on transformations and actions

What are PairRDD?
Code example on transformations and actions on Pair RDD For each
Types of join with example

Seting up PySpark on EC2
Starting PySpark with Yarn
Staring PySpark with Standalone resource Manager Running Python Script using spark submit
Spark Shell
Basic operations on Shell
Spark Context and Spark Properties

Understanding various spark Resource managers Spark Dynamic and Static Allocation
Spark on YARN
Spark Standalone
Spark Executor Memory Spark Memory Storage levels

Introduction to Spark SQL
Introduction to Dataframes
Different Operations on Dataframe like groupBy, join, filter, select, distinct, min, max, etc
Various mathematical operations of Dataframe
Querying Files as Tables
Text file Format
Schemas
Overview of Structured Spark Types Columns/Rows
Logical / Physical Planning
Columns and Expressions
Records and Rows
Dataframe Transformations
Creating Dataframes
select and selectExpr
Adding Columns
Renaming Columns
Changing Column Types
Filtering Rows
Getting Unique Rows
Limit
Repartition and Coalesce
Joins

Introduction to Spark Streaming
Implementing Spark Streaming using Python Script

Structured Streaming Basics Core Concepts Transformations and Actions Input Sources
Sinks
Output Modes
Triggers
Structured Streaming in Action Transformations on Streams Input and Output