I will build an solidgate api to bigquery pipeline with historical backfill
Cloud Data Engineer, BigQuery, Snowflake, dbt, Python, ETL
About this Gig
Your payment, billing or e-commerce data lives in an API, and every report starts with a CSV export?
I build production pipelines that load data from REST APIs (Soligate, Stripe, Shopify, payment gateways, CRMs, custom APIs) into Google BigQuery, including full historical backfill.
Recent project: a serverless ingestion platform for a payment provider's API. 6 endpoints, 12+ months of history backfilled, raw JSON kept in cloud storage, deployed with CI/CD.
What you get:
- Incremental loads: no duplicates, safe re-runs
- Historical backfill with pagination and rate-limit handling
- Untouched raw layer, so tables can be rebuilt any time
- Clean, typed tables ready for Looker Studio or Power BI
- Monitoring and failure alerts
- Code in your repo + setup docs
Stack: Python, BigQuery, Cloud Run/Functions or AWS Lambda, Airflow, dbt.
15+ years in data engineering and ERP integrations.
Message me with your source API and what you want to report on. I'll reply with a fixed quote.
My Portfolio
Other Data Engineering Services I Offer
FAQ
Where does the pipeline run?
In your own GCP or AWS account. You own the code and the data.
What will BigQuery cost me?
For small and mid volumes it is usually low. I estimate it before we start.
What if the API changes?
Raw JSON is stored as-is, so new fields don't break loads and history can be re-processed.
Can you load into Snowflake or Redshift instead?
Yes, through a custom offer.
