Browse categories
Explore
Fiverr Pro
English
$
USD
I will develop azure databricks etl data pipelines and pyspark jobs
Azure Data Engineer for Databricks ETL Pipelines
I am a motivated Software Engineering undergraduate with hands-on experience in data engineering pipelines and full-stack development. I am proficient in building scalable ETL processes using PySpark,...
About this Gig
Welcome to my Azure Data Engineering Consulting Gig!
Are you looking to transform your raw data into clean, actionable insights? I specialize in designing and building robust, scalable data pipelines using the modern Azure Data Stack, with a primary focus on Azure Databricks and PySpark.
Whether you need an architectural roadmap, a simple script optimized, or a complete end-to-end ETL/ELT pipeline built from the ground up, I am here to help your data move seamlessly and securely.
What I offer in this gig:
- Data Lake Setup: Organizing your storage using best practices like the Bronze/Silver/Gold (Medallion) architecture.
- ETL Pipeline Development: Writing clean, efficient PySpark or SQL code to extract, transform, and load your data.
- Orchestration: Designing the automated flow of your data using tools like Azure Data Factory.
- Performance Optimization: Debugging and speeding up slow Spark notebooks.
- Architecture Diagrams & Audits: Providing clear visual documentation and professional recommendations for your current setup.
FAQ
Do I need to provide my own Azure environment?
Yes, you will need to provide me with secure access to your Azure subscription (such as a Guest account with specific resource group permissions) so I can build, test, and deploy the pipelines directly in your workspace.
What is the Medallion Architecture you mentioned?
It is a highly efficient data design pattern used in Databricks. It logically organizes data in a lakehouse into three distinct layers: Bronze (raw, ingested data), Silver (cleaned and filtered data), and Gold (business-level aggregates ready for analytics and reporting).
Can you fix a bug in my existing PySpark code?
Absolutely! The Basic package is perfect for this. I can audit your current notebook, identify the bottleneck or error, and provide the optimized, corrected code.
Do we have to do a video call for the consulting minutes?
Not at all! While Fiverr provides a Zoom link, we can keep our cameras off for an audio-only call, or use that dedicated time to chat extensively via text while I review your architecture and data goals.
