I will develop advanced web scraping data pipeline engineering

I
ido_goldblatt
I
ido_goldblatt
Ido Goldblatt
develop advanced web scraping data pipeline engineering
Full Screen
develop advanced web scraping data pipeline engineering
Full Screen
develop advanced web scraping data pipeline engineeringdevelop advanced web scraping data pipeline engineering

About this gig

This gig combines backend automation with sophisticated data processing.

The Tech Stack:

  • Extraction Engine: Python is the primary language, utilizing Selenium, Playwright, or Puppeteer for browser automation. These tools can render JavaScript, click buttons, and handle infinite scrollingtasks that BeautifulSoup cannot handle alone.
  • Anti-Detection Layer: Integration of proxy rotation services (Bright Data, Smartproxy) and the use of undetected-chromedriver to bypass Cloudflare/Akamai WAFs (Web Application Firewalls).
  • Data Processing: Once raw data is extracted, Pandas is used to clean itremoving duplicates, normalizing currency formats, filling missing values, and validating data types.   
  • Storage/Delivery: Data is delivered via CSV, JSON, or injected directly into the client's PostgreSQL or Firebase database.

  • Programming language

    • JavaScript

  • Expertise

    • Algorithms & Data structures
    • Security
    • Databases

  • Frontend framework

    • React.js
    • Tailwind CSS

Get to know Ido Goldblatt

Ido Goldblatt
4.9(4)
  • FromIsrael
  • Member sinceSep 2016
  • Avg. response time2 hours
  • Last delivery8 months

  • Languages

    English, Hebrew
With over four years of professional experience in software development, I specialize in crafting dynamic, efficient, and scalable applications. My expertise lies in Python, JavaScript, React, and Nodejs, enabling me to build robust full-stack solutions that cater to diverse business needs.

Other Software Development Services I Offer

Related tags