How to protect content from AI crawlers and bad bots with robots.txt file.
TL;DR: Protect content from AI crawlers by combining robots.txt rules, server-level controls, HTTP headers, and realistic expectations about what can and cannot be blocked. As of 2025, no single method fully prevents AI scraping, but layered controls significantly reduce unauthorised reuse and resource drain. This guide explains what actually works in the UK context, where […]