@kkarhan@infosec.space@fuchsiii @lynn @LunaDragofelis if you look up the robits.txt specs [www.robotstxt.org] you'll see it's literally just an ask...
If you want to prevent ChatGPT from crawling shit you need to literally block it [github.com] on #firewall level!
- And since "OpenAI" are assholes, they use ClownFlare [infosec.space] to prevent you from just using
curl/wgetto pull their bots' IP ranges [openai.com]...
Personally, I'd recommend to forward every request they do to this little test file [hil-speed.hetzner.com] and let them get hetzner'd [hil-speed.hetzner.com]!
www.robotstxt.orgThe Web Robots Pages