Browse categories
Explore
Fiverr Pro
English
$
USD
I will build automated python etl pipelines from rest apis into bigquery
Python Data Engineer for BigQuery ETL Pipelines and API Integrations
I build automated data pipelines that replace manual exports and spreadsheet reporting. I'm a Python data engineer with 3+ years of production experience moving data from APIs, databases and files int...
About this Gig
Is your data stuck in APIs, databases and spreadsheets that someone exports by hand every week? I replace that with an automated Python pipeline that loads it into BigQuery reliably.
I'm a Python data engineer with 3+ years of experience. In my current role I build and run production pipelines that pull data from multiple platform APIs into a BigQuery warehouse.
What I build
- Python ETL from REST APIs, PostgreSQL/MySQL, CSV/Excel or Google Sheets into BigQuery
- Incremental loads with keyed merges to prevent duplicate rows on reruns
- Retries, rate-limit handling, logging and data validation
- Scheduled with Cloud Scheduler and Cloud Run, or cron on your server, packaged in Docker
- Clean SQL reporting tables for Looker Studio or any BI tool
How it works
- You share API docs or sample data and grant access through your own service account
- I confirm fields, schedule and package before starting
- You receive the code, a setup guide and a pipeline running in your project
Not included: paid connector licences, API access you don't have, or ongoing monitoring (see extras).
Message me with your source and destination and I'll recommend the right package.
Destination Platform:
Google BigQuery
•
PostgreSQL
•
MySQL
Tools & Platforms:
Other
FAQ
What do you need from me to start?
API docs or sample files, the fields you need and your BigQuery project. Please grant access through a service account or OAuth in your own account with the minimum permissions needed. Never send me your main account password.
Can you work with databases other than BigQuery?
Yes. I also load data into PostgreSQL and MySQL, and I've automated Oracle PL/SQL to PostgreSQL migrations with Python.
Will the pipeline keep running after delivery?
Standard and Premium pipelines are scheduled, safe to rerun and log every run; Premium also sends failure alerts. If the source API changes later, updates are available through the 30-day support extra or a custom offer.
Which data sources can you connect?
Any REST or GraphQL API with documentation and credentials, PostgreSQL, MySQL, CSV/Excel, Google Sheets and Cloud Storage. If your source has no official API, message me first and I'll confirm whether it's feasible.
Do I need my own Google Cloud account?
Yes. The pipeline runs in your GCP project, so you own the code, data and billing. I'll guide you through any setup needed.
How much will BigQuery cost me?
It depends on data volume, storage, queries and refresh frequency. I partition and cluster tables and design queries to keep costs under control. Billing stays in your own Google Cloud account.
How often can the data refresh?
Hourly, daily or on a custom schedule, limited by your source API's rate limits. Real-time streaming is a custom quote.
How much historical data can you load?
As much as the source API allows. Premium includes a 12-month backfill, and the Historical backfill extra adds up to 24 more months.
