User-agent: magpie-crawler Disallow: / user-agent: AhrefsBot Disallow: / User-agent: SemrushBot Disallow: /