Browse categories
Explore
Fiverr Pro
English
$
USD
I will develop and fix pyspark etl pipelines and data transformations
About this Gig
Need help with a PySpark job that fails, returns unexpected results, or needs a scoped ETL transformation? I am a computer engineer with professional experience in software and data engineering. I have built PySpark transformations and batch ETL workflows with Azure Databricks and Azure Synapse Analytics, including a medallion-style flow and troubleshooting a pipeline processing about 1.7 million records. I can fix a specific bug, implement one bounded transformation or batch ETL flow, read and write agreed data formats, add focused checks, and explain the code. Before ordering, share Spark/PySpark versions, schemas, a small safe sample, expected output, and runtime context. I will confirm scope and assumptions first. Never share passwords, tokens, or production secrets. Packages exclude cloud deployment, infrastructure setup, streaming, unlimited volumes, and guaranteed performance gains. Optimization covers only the agreed code-level review and changes; outcomes depend on data and runtime.
Tools & Platforms:
Other
FAQ
What do you need from me to get started?
Please share your goal, versions, schemas, safe input and expected output examples, runtime environment, and existing code or error details. Never send passwords, tokens, or production credentials. Use synthetic, anonymized, or safely shareable data.
Which Spark and PySpark versions do you support?
I can work with a specified version when its APIs and required environment are available. Please share the exact Spark, PySpark, and Python versions plus library constraints so I can confirm compatibility and scope before work begins.
Which data formats can you work with?
Common formats include CSV, JSON, and Parquet. Delta tables are possible in a compatible Spark or Databricks setup. Please share your schema and a small safe sample first so I can confirm the format and scope.
Can you debug or fix a pipeline I already have?
Yes. Share the relevant code, error details, schemas, and a minimal reproducible case using safe data. I will first confirm that the issue fits the selected package; broader investigations or multiple jobs need a separately agreed scope.
Do you include testing or documentation?
Basic includes focused verification. Standard includes agreed data checks, and Premium also includes brief documentation. Unit or integration test suites are not included unless specifically agreed and technically supported by the supplied project.
Can you work locally or in a cloud environment?
I can develop and check code in a compatible local or client-provided runtime and review cloud-specific code when the required details are available. Deployment and cloud infrastructure setup are excluded. Please do not share production credentials; use client-run steps where needed.
Can you guarantee faster performance or any data volume?
No. Results depend on data shape, runtime, cluster settings, and available test data. I can review a defined code path and implement agreed changes, but I cannot guarantee speedups, process unlimited volumes, or tune infrastructure without access and a separately agreed scope.
How should I share data securely?
Please use synthetic, anonymized, or otherwise safely shareable samples whenever possible. Share only material you have permission to provide, and never send passwords, tokens, or production credentials. We can agree on the minimum data needed before starting.
What counts as a revision?
A revision is a change to work already delivered within the agreed inputs, output, and behavior. New requirements, additional sources or transformations, another job, or broader debugging are outside the package and may need a custom offer.
