I will build etl data pipelines with python, airflow, kafka and postgresql

Pakistan

I speak English

2 orders completed

Data Scrapping, Python Developer, Web Developer

I am a professional and dedicated freelancer, with a passion for providing high-quality services to my clients. I have a strong track record of delivering projects on time and within budget, and I am ...
About this Gig

Is your data spread across APIs, files and databases? I build automated ETL pipelines that collect, clean and load your data so it's always ready to use.

My experience:

  • Kafka pipeline processing 8 GB of media per minute with 35% lower latency
  • Celery + Redis cluster handling 1,000+ tasks per minute
  • Daily crawling pipeline storing 500,000+ job listings
  • Software Engineering degree with a Data Science specialization (ETL, data mining)

What I can build:

  • ETL / ELT pipelines from APIs, CSV, Excel, websites or databases
  • Scheduled jobs with Apache Airflow, Celery or cron
  • Real-time streaming with Kafka or RabbitMQ
  • Loading into PostgreSQL, MySQL, Elasticsearch or AWS (S3, RDS, DynamoDB)
  • Data cleaning, transformation and validation

Tools: Python, Pandas, Airflow, Kafka, Celery, Redis, PostgreSQL, AWS

You get clean, documented code that you own. Message me with your data sources and goals, and I'll suggest the simplest pipeline that works.

Destination Platform:

PostgreSQL

•

MySQL

•

Amazon S3

Tools & Platforms:

Kafka Connect