I will build etl data pipelines with python, airflow, kafka and postgresql
Data Scrapping, Python Developer, Web Developer
About this Gig
Is your data spread across APIs, files and databases? I build automated ETL pipelines that collect, clean and load your data so it's always ready to use.
My experience:
- Kafka pipeline processing 8 GB of media per minute with 35% lower latency
- Celery + Redis cluster handling 1,000+ tasks per minute
- Daily crawling pipeline storing 500,000+ job listings
- Software Engineering degree with a Data Science specialization (ETL, data mining)
What I can build:
- ETL / ELT pipelines from APIs, CSV, Excel, websites or databases
- Scheduled jobs with Apache Airflow, Celery or cron
- Real-time streaming with Kafka or RabbitMQ
- Loading into PostgreSQL, MySQL, Elasticsearch or AWS (S3, RDS, DynamoDB)
- Data cleaning, transformation and validation
Tools: Python, Pandas, Airflow, Kafka, Celery, Redis, PostgreSQL, AWS
You get clean, documented code that you own. Message me with your data sources and goals, and I'll suggest the simplest pipeline that works.
Destination Platform:
PostgreSQL
•
MySQL
•
Amazon S3
Tools & Platforms:
Kafka Connect
FAQ
Can you deploy the pipeline?
Yes. I can deploy it on your server or on AWS (EC2, Lambda, S3) and set up scheduling.
Do you work with my existing database?
Yes. I work with PostgreSQL, MySQL, SQLite and Elasticsearch, and AWS services like RDS and DynamoDB.
Will I get the source code?
Yes. You get the full source code with setup instructions and documentation.
Can you build real-time streaming pipelines?
Yes. I have built Kafka pipelines handling 8 GB of data per minute, plus RabbitMQ and Celery task queues.
What data sources can you connect?
REST APIs, CSV and Excel files, websites (via scraping), databases and cloud storage like S3.
What do you need from me to start?
Your data sources, the destination, how often the data should update, and any access details for test environments.

