Welcoming LLM AI to My Site

I hope it's for the better

   ,   1 min read

As you may know, I’ve written a tutorial on how to block LLM AI like ChatGPT from crawling my site without attribution via robots.txt file.

I implemented that on my own websites, and so far I haven’t seen any traffic coming from the LLM bots.

Enough time has passed since then, and I’ve noticed that some visitors to my YouTube channel are coming from LLMs like ChatGPT.

They’re a small number for now, but I guess the future of web traffic will come from LLMs.

I hope allowing AI to crawl my site will have a positive impact for both the author and the AI itself.

That’s all I can write today.

Thanks for reading, and see you next time.

More from Junian Network

How to allow LLMs like ChatGPT to crawl your website for search purposes while preventing AI model training by using Cloudflare’s Content Signals

Robots.txt to Allow LLM AI Like ChatGPT for Search Only Using Content Signals

How to allow LLMs like ChatGPT to crawl your website for search purposes while preventing AI model training by using Cloudflare’s Content Signals

3 min read
A collection of Visual Studio Code (VS Code) forks that offer free AI features for vibe coding or AI-assisted programming.

Visual Studio Code (VS Code) and Its Forks with Free AI Assistants

A collection of Visual Studio Code (VS Code) forks that offer free AI features for vibe coding or AI-assisted programming.

2 min read
How to block Large Language Model AI from crawling your website with robots.txt User-agent block list

Robots.txt Rules to Block LLM AI like ChatGPT from Crawling Your Web Content

How to block Large Language Model AI from crawling your website with robots.txt User-agent block list

2 min read