I will build pyspark etl pipelines for batch and streaming data on AWS
Data Engineer for PySpark, AWS and QuickSight
About this Gig
Need a reliable PySpark job or a full ETL pipeline? I'm a full-time Data Engineer, and building batch and streaming pipelines with PySpark on AWS is my day job.
What I can do for you:
- Write PySpark ETL jobs: ingest, clean, transform, load
- Process large datasets in batch on Amazon EMR
- Build streaming jobs for real-time data
- Integrate your jobs with Apache Hive and Apache Hudi
- Set up or organise a data lake on Amazon S3
- Optimise or debug existing PySpark code
What you get:
- Clean, commented code
- Short notes on how to run and schedule it
- Replies usually within an hour
How it works:
1. Send me your data source, your target, and the transformation you need
2. I confirm scope, price, and timeline
3. I deliver the code and revise based on your feedback
Please message me before ordering so we can agree on scope first.
Expertise:
Big data
•
Data manipulation
•
ETL
•
Transformation
•
SQL
•
NoSQL
Technology:
Apache Hadoop
•
Apache Spark
•
Excel
•
Python
•
SQL
•
NoSQL
FAQ
Which platforms do you work on?
Mainly Amazon EMR and Amazon S3, with PySpark, Apache Hive and Apache Hudi. If you use a different platform, message me before ordering so I can confirm it's a good fit.
Can you fix a PySpark job that is slow or failing?
Yes. Send me the error message or the Spark UI details along with the code, and I'll tell you what's wrong and what it will take to fix before you order.
What do you need from me to get started?
Your data source and target, a sample of the data or its schema, and the transformation you need. For work in your AWS account, a limited IAM role with access to the relevant EMR cluster or S3 bucket is enough.
