AI crawlers

Keep crawlers that collect training data out, and let AI search and people using an assistant in. You choose per purpose, and your own server applies the choice.

Decision trace

How it works

Training crawlers are blocked by default. Search crawlers and fetches made for a person are allowed. Search crawlers include Google and Bing, so blocking them also removes the site from search results. A request that claims a crawler name from an address outside its published ranges counts as fake and is blocked.

Nolgard is not a proxy. The Node SDK downloads the crawler list and your policy once a day and decides on your server, so no page view waits on Nolgard. It sends counts back once a minute, and the portal shows the last 7 days by crawler. The SDK also builds a robots.txt from the same policy.

Limits

• Only crawlers that identify themselves are covered. A scraper with a browser User-Agent needs bot detection.
• Google-Extended and Applebot-Extended are robots.txt tokens, so they can be opted out of there and nowhere else.
• Your server enforces the policy, and a portal change reaches it at the next daily refresh.