Azure Data Engineering

Live Interactive Classes

Enrol Now

New Batch starts from 29th February 2024

Azure Data Engineering course

Launch your cloud data engineering career with this immersive program. Gain the expertise to design, develop, and manage efficient data pipelines using Microsoft Azure's data services. Through hands-on learning, you'll master essential tools like Azure SQL, Data Factory, Databricks, Delta Lake, and Synapse Analytics. This comprehensive program equips you with the necessary skills to tackle real-world data engineering challenges and prepares you for the sought-after Microsoft Azure Data Engineering Associate (DP-203) certification, empowering you to thrive in the ever-evolving field of cloud data management.
TIMELINE
10 weeks
PREREQUISITES
None
SKILL LEVEL
Intermediate

₹16000

₹19000 16%

Career prospects of Azure Data Engineering

Thriving Career Prospects in Cloud Data Engineering: The cloud data engineering field offers a promising career outlook due to the ever-growing demand for organizations to store, manage, and analyze vast amounts of data in the cloud. This demand is driven by factors like:

  • Exponential Cloud Adoption: Businesses are rapidly migrating to cloud platforms for their scalability, cost-effectiveness, and flexibility.
  • Big Data Surge: The volume of data generated continues to explode, requiring robust data pipelines for efficient processing and analysis.
  • Data-Driven Decision-Making: Organizations are increasingly relying on data insights to inform strategic decisions across various departments.
These factors create a growing need for skilled cloud data engineers who can design, build, and maintain data pipelines in the cloud. As a cloud data engineer, you can expect competitive salaries, ample job opportunities, and career advancement possibilities in various industries.

What you will learn

Chapter 1
Data Engineering Introduction
  • What is data engineering
  • Introduction to Azure
  • Azure vs AWS Data Engineering services
  • Big data introduction
Chapter 2
Azure Basics
  • What is Azure and Cloud
  • Resource Group Creation
  • Services Offered in Azure
  • Azure Portal Walk Through
  • SDKs or Tools for Azure Resources
  • Create Free Azure Subscription
Chapter 3
Python Overview
  • History & features of Python
  • Python vs Other Programming Languages
  • First Python Program
  • Python basic syntax
  • Python Development Tools & Packages
Chapter 4
Python Environment Setup
  • Installing Python
  • Verify & Setup Python environment
Chapter 5
Python Data Types
  • Variables
  • Data Types
  • Strings
  • Type Casting
Chapter 6
Python Operators
  • Arithmetic, Relational, Logical, Bitwise Operators
  • Assignment Operator
Chapter 7
Python Flow Control & Loops
  • if, if else, if else if
  • while, do while loops
  • for loops
Chapter 8
Python Functional Programming
  • Function Declarations
  • Call-by-Name
  • Functions with Named Arguments, Variable Arguments
  • Default Parameter Values, Lambda Functions
Chapter 9
Python Collections
  • Lists
  • Tuples
  • Sets
  • Dictionaries
Chapter 10
Python Files
  • Reading input from console
  • Reading data from Files
  • Writing data to Files
Chapter 11
Python Object Oriented Programming Python Classes
  • Simple class
  • Class objects
  • Inheritance
Chapter 12
Python Exception Handling
  • Throwing Exceptions
  • try, catch, finally
  • Catching Exceptions
  • The finally Clause
Chapter 13
Python Miscellaneous
  • Modules
  • Dates
  • RegEx
Chapter 14
Spark introduction
  • Spark Overview
  • Spark features
  • Spark vs Hadoop MapReduce
  • Programming Language choices in Spark
  • Spark History
  • Spark use cases
Chapter 15
Spark Components or modules
  • Spark Core
  • Spark SQL
  • Spark Streaming
Chapter 16
Spark Architecture
  • Spark Application flow
  • Spark Driver, Executors
  • Spark Context, Spark Session
  • Spark dependency on Cluster Managers
  • Spark execution modes
  • Standalone cluster mode, Spark on YARN mode
Chapter 17
Spark Core
  • RDD Introduction
  • Creating RDDs, Saving Files
  • Data Manipulation using RDDs
  • Transformations & Actions
  • RDD Partitions & Coalesce
  • Memory Management: cache & persist
  • Data Loading and Saving through RDDs
  • Aggregations, Joins through RDDs
  • RDD Advanced concepts – Accumulators, Broadcast variables
Chapter 18
Spark Structured APIs
  • DataFrames
  • Columns, Rows, Spark Types
  • Performance optimization
  • Logical planning, Physical planning, Execution
Chapter 19
Spark DataFrames
  • Basics, Creating DataFrames
  • Schemas, DataFrame Operations
  • Column-wise, Row-wise operations
  • Aggregations, Joins using DataFrames
Chapter 20
Data Sources
  • Reading & writing file formats
  • CSV, JSON, Parquet, ORC Files
  • SQL Databases, TextFiles
Chapter 21
Spark SQL
  • Apache Hive vs Spark SQL
  • Catalog, Tables, Views, Databases
  • Data selection, manipulation
  • User Defined Functions
  • Integration with Hive
Chapter 22
PySpark Application Developer Tools
  • PySpark interactive shell
  • PyCharm, Spyder, Jupyter Notebooks, Zeppelin
  • Other developer tools
Chapter 23
Executing PySpark Application
  • Local mode, Client mode, Cluster mode
  • Spark UI, Monitoring Spark Applications
  • Spark logs, Resource selection
Chapter 24
Spark Streaming
  • Batch vs Streaming processing
  • DStreams, Structured streaming
  • Transformation of Streams data
  • Streaming sources & sinks
  • Event Time, Stateful processing
Chapter 25
Azure Storage and Data Lake
  • Blob Storage details, Container creation
  • Blob upload, deletion
  • Data Lake Gen 2 details, Account creation
  • Folder creation, Data upload, Security
Chapter 26
Azure Key Vault
  • Introduction to Azure Key Vault
  • Store Secrets in Azure Key Vault using Azure Portal
Chapter 27
Azure Data Factory
  • Introduction to Azure Data Factory
  • Top-level Concepts, First Data Factory
  • Working with Azure Data Factory elements
  • Pipelines, Activities, Linked Services, Datasets, Triggers
Chapter 28
Azure Synapse Analytics
  • Introduction, Workspace creation
  • Basic Concepts, SQL Architecture
  • Serverless SQL Pool, Dedicated SQL Pool
  • External Data source, File Format
  • CETAS, CTAS, External Tables
  • Administrative accounts, Temporary Tables
  • IDENTITY, OPENROWSET, Spark in Synapse Analytics
Chapter 29
Azure Data Bricks
  • Introduction, Workspace creation
  • Community Edition Account, Workspace assets
  • Workspace Objects, Running Spark Job
  • Architecture overview, DBFS, Utilities
  • Widgets, Mount Point creation, Secrets Overview
Chapter 30
Delta Lake usage in Databricks
  • Architecture, Storage Understanding
  • Table creation, DML Operations
  • Partitions, Schema Enforcement, Evolution
  • Versions, Time Travel, Vaccum
  • Delta Lake Merge (SCD Type 1 and SCD Type2)
Chapter 31
Azure Application Insights
  • Azure Application Insights Tutorial, Overview
  • Codeless Monitoring, Code-based Monitoring
  • Features, Creation Prerequisites
  • Enabling for Applications

FAQs

Quick Enquiry
Successfully Submitted