Browse categories
Explore
Fiverr Pro
English
$
USD
I will block ai bots and scrapers from your website using robots file
Pavel V
About this gig
I will update your robots.txt file to block the top 50+ most aggressive AI web scrapers and crawlers (including OpenAI, Anthropic, and Common Crawl). Perfect for small blogs or portfolios looking for immediate privacy.
Respect third-party rights
Please be aware that it is against Fiverr's policies for sellers to include themes, templates, or any other elements that infringe third-party rights or applicable laws in the delivered work. Read more about in our Guide to Responsible Digital Creation.
Get to know Pavel V
Pavel V
Founder WebflowForge Driving Growth with Web Development and AI Automations
- FromIsrael
- Member sinceJul 2024
Languages
English, Russian, Hebrew
Founder of Webflowforge with 9+ years of experience building scalable web platforms and digital products. I specialize in Webflow, WordPress, automations, AI solutions, and RevOps—combining UX, development, and business logic to create high-performing, conversion-focused systems.
I help with UI/UX, advanced integrations, CMS/database architecture, and full platform builds.
From idea to execution, I turn concepts into production-ready, lead-generating machines built for growth, performance, and scale.
My Portfolio
FAQ
Will blocking AI bots hurt my Google Search ranking?
No. Standard SEO (Google, Bing) and AI Bot blocking are two different things. We use specific "User-Agent" commands that tell AI crawlers (like GPTBot or CCBot) to stay away while allowing Google’s main search index to crawl your site normally. Your human traffic will not be affected.
What is the difference between robots.txt and llms.txt?
Think of robots.txt as the "No Trespassing" sign—it tells AI bots they are not allowed to scrape your data for training. Think of llms.txt as a "Table of Contents"—it provides a clean, markdown-friendly version of your site info for AI search engines (like Perplexity or ChatGPT Search) so they can s
Do I need to give you my website login credentials?
It depends on your comfort level. I can either log in to your WordPress/Shopify/Hosting panel and do it for you (fastest), OR I can provide you with the custom files and a step-by-step video guide on how to upload them yourself.
Can you block specific AI bots but let others through?
Absolutely. If you want to block OpenAI from training on your content but you want to allow Perplexity to show your site in its search results, I can customize the code specifically for those preferences.
Is this a one-time fix or do I need to update it?
This is a permanent setup. However, new AI bots are released every month. The file I provide includes the most comprehensive list of 2026 scrapers. I recommend a quick "health check" every 6 months to ensure new, aggressive bots haven't entered the scene.
