Azure Data Engineer

ADF | Databricks | PySpark | SQL

I build scalable data pipelines and transform raw data into business insights.

Download Resume

About Me

Started as a PHP backend developer and transitioned into Azure Data Engineering to focus on scalable data pipelines and analytics systems.

Experienced in building end-to-end ETL pipelines using Azure Data Factory, Databricks, and PySpark, following Bronze, Silver, Gold architecture.

Skilled in transforming raw data into reliable, analytics-ready datasets and designing pipelines that support reporting and business insights.

Skills

Data Engineering
ETL Pipelines, Data Modeling

Programming
Python, PySpark, SQL

Cloud
Azure, ADLS Gen2, ADF

Big Data
Databricks, Delta Lake

Databases
MySQL, Azure SQL

Visualization
Power BI

Tools
Git, GitHub

Concepts
SCD Type 1 & 2, Data Warehousing

Projects

Customer Account Data Pipeline

End-to-end Azure pipeline for processing customer financial datasets.

  • ADF ingestion to ADLS (Bronze)
  • Data cleaning & transformation using Databricks
  • SCD Type 1 & 2 implementation
  • Loaded into SQL + Gold layer

Tech: ADF, ADLS, Databricks, SQL, Power BI

View Code
Transaction & Loan Pipeline

Scalable ETL pipeline for customer transaction and loan analytics.

  • ADF pipeline Bronze → Silver → Gold
  • PySpark data cleaning & deduplication
  • Delta Lake tables for analytics
  • Automated pipelines with parameters

Tech: Databricks, PySpark, Delta Lake, ADF

View Code
NYC Taxi Data Pipeline 🚀

Dynamic Azure pipeline for automated ingestion and large-scale data processing.

  • Automated monthly ingestion (ForEach loop)
  • Processed ~1M+ records using PySpark
  • Bronze → Silver → Gold architecture
  • Delta Lake (time travel + versioning)

Tech: ADF, Databricks, PySpark, ADLS, Delta Lake

View Code

Architecture

This architecture represents an end-to-end Azure Data Engineering pipeline. Data is ingested using Azure Data Factory, stored in ADLS Gen2 using Bronze, Silver, Gold layers, processed using Databricks with PySpark, and served for analytics using Power BI and Azure SQL.

Looking for a Data Engineer?

I am actively seeking opportunities to build scalable data solutions.

Contact Me

Contact

Email: jayavarapusrimani@gmail.com

Phone: +91-8309779064