I will create scalable data pipelines using azure data factory and databricks
Big Data Engineer
About this Gig
I specialize in designing and developing end-to-end ETL/ELT pipelines using Azure Data Factory (ADF) and Databricks (PySpark). I integrate data from multiple sourcesAPIs, SQL databases, cloud storage, files, and moreinto clean, structured, and production-ready pipelines.
With strong expertise in ADF orchestration, Databricks Delta pipelines, and PySpark transformations, I can help you build robust workflows that are easy to maintain and optimized for performance. Whether you need a simple ingestion pipeline or a complete multi-source data solution, I deliver scalable, secure, and well-documented results.
What I offer:
Data ingestion from multiple sources
ADF pipelines, triggers, orchestration
Databricks Delta ETL/ELT pipelines
PySpark transformations
Data quality checks and validation
Streaming or batch pipelines
Documentation and best practices
Lets build a pipeline that delivers accurate, reliable, and production-grade data for your business!
Tools & Platforms:
AWS Glue DataBrew
•
Azure Data Factory
