I will build and automate azure etl pipelines using azure data factory and databricks

Pakistan

I speak English

Building End to End Azure Data Pipelines with Databricks and Power BI

I’m Arslan Ahmed, a Senior Data Engineer specializing in end-to-end Azure data solutions. I build and automate ETL pipelines using Azure Data Factory, Databricks (PySpark, Spark SQL), and Power BI. My...

About this Gig

Professional Azure Data Engineer with 3+ years of experience and hands-on developing highly complex ETL pipelines.

Experienced in implementing scalable end-to-end optimized workflows on Azure Data Factory / Databricks to extract, process, and store data from diverse sources, which resulted in an average 40% reduction in data processing time for my clients.
Experienced in migration of data from legacy systems (On-premise) and APIs to Azure Data Lake, performing complex transformations on data using DataBricks (Python and PySpark), moving the processed data to SQL Server / PostgreSQL / Data or Delta Lake, and writing SQL Stored Procedures / Functions that can transform the data and perform some calculations on it based on some business logic.
I am also skilled in developing and maintaining DeltaLake architectures using Databricks and leveraging DataLakes to store the underlying data and enforce security standards.
Scheduling workloads using Azure Data Factory triggers or Databricks workflows.