AI crawlers

What it does

Nolgard is not a proxy, so your own server enforces the policy. The SDK reports counts back once a minute, and the portal shows what was allowed and blocked over the last 7 days.

The policy has four settings in site settings under AI crawlers. Each is allow or block.

Search crawlers include Google and Bing, so blocking them also removes the site from search results. The verified-agent policy used at signup is a separate setting and does not change.

Express

import express from 'express';
import { createNolgard } from '@nolgard/node';

const nolgard = createNolgard({ secret: process.env.NOLGARD_SECRET });
const crawlers = nolgard.crawlers();
const app = express();

app.use(crawlers.express());

Blocked requests get a 403 with the text Forbidden. Behind a proxy or load balancer, make sure Express reports the visitor's address rather than the proxy's.

Next.js

// middleware.ts
import { NextResponse, type NextRequest } from 'next/server';
import { createNolgard } from '@nolgard/node';

const crawlers = createNolgard({ secret: process.env.NOLGARD_SECRET! }).crawlers();

export async function middleware(request: NextRequest) {
  const blocked = await crawlers.handle(request);
  return blocked ?? NextResponse.next();
}

handle takes a fetch API Request. It returns a 403 Response when the request is blocked and null otherwise. The address comes from cf-connecting-ip, then x-real-ip, then the first x-forwarded-for value. Pass an address as the second argument if your host provides it another way.

Cloudflare Workers

import { createNolgard } from '@nolgard/node';

let crawlers;

export default {
  async fetch(request, env) {
    crawlers ??= createNolgard({ secret: env.NOLGARD_SECRET }).crawlers();
    const blocked = await crawlers.handle(request);
    if (blocked) return blocked;
    return fetch(request);
  }
};

Create the checker once per isolate so the list is shared between requests. The SDK uses no node: imports on this path.

robots.txt

Well-behaved crawlers read robots.txt before they fetch pages. robotsTxt() returns a group for every crawler whose purpose your policy blocks, built from the last loaded list.

app.get('/robots.txt', async (req, res) => {
  res.type('text/plain').send(await crawlers.robotsTxt());
});

The text is empty when nothing is blocked. Add your own rules before it if you keep other groups. Google-Extended and Applebot-Extended are robots.txt tokens only. They are not sent as a user agent, so the request check never matches them and robots.txt is the only way to opt out.

Methods

const crawlers = nolgard.crawlers({ refreshMs: 86_400_000, reportMs: 60_000 });

await crawlers.check({ userAgent, ip });
// { action: 'allow' | 'block', agent, operator, purpose, verified }

await crawlers.handle(request, ip?);  // Response (403) or null
crawlers.express();                   // (req, res, next) middleware
await crawlers.robotsTxt();           // string
await crawlers.flush();               // send pending counts now

Before the first successful download, or when a download fails, check allows the request. A failed refresh keeps the last good list. Set reportMs to 0 to turn reporting off. Timers do not keep a Node process running.

Other languages

Download the list with your site secret. Cache it for the ttl in seconds and decide locally.

GET https://nolgard.com/v1/crawlers
Authorization: Bearer SITE_SECRET

{
  "version": "5f0c2a9e1b47d3c8a0e6f29b4d175ac3",
  "ttl": 86400,
  "policy": { "training": "block", "search": "allow", "user": "allow", "spoofed": "block" },
  "agents": [
    { "name": "GPTBot", "operator": "openai.com", "purpose": "training",
      "ua": "GPTBot", "ranges": ["20.15.240.64/28"] },
    { "name": "Google-Extended", "operator": "google.com", "purpose": "training",
      "ua": null, "ranges": [] }
  ],
  "robots": "User-agent: GPTBot\nDisallow: /\n\nUser-agent: Google-Extended\nDisallow: /\n"
}

A wrong secret returns 401 unauthorized. ua is a JavaScript regular expression source, or null for robots.txt-only tokens. Classify each request like this:

1. Take the first agent whose ua matches the User-Agent header, ignoring case. If none matches, allow the request. It is not a known crawler.
2. If that agent has a non-empty ranges list and the client address is in none of them, the request is spoofed. Apply policy.spoofed.
3. Otherwise apply policy[agent.purpose]. The crawler is verified when ranges is non-empty.

To show counts in the portal, send them at most once every 10 seconds. Extra reports get 429 rate-limited.

POST https://nolgard.com/v1/crawlers/report
Authorization: Bearer SITE_SECRET

{ "counts": [ { "agent": "GPTBot", "action": "block", "verified": true, "n": 12 } ] }

Send up to 200 entries. n is a whole number from 1 to 1,000,000. Unknown agent names are ignored. The response is {"ok": true}.

Limits

This covers crawlers that identify themselves. A scraper that sends a browser User-Agent is not recognised, so use bot detection for it. A crawler that claims a user agent with published IP ranges and comes from elsewhere is treated as fake. Crawlers without published ranges cannot be verified, so a fake one using their name passes as the real one.

The policy is enforced by your server, so a request that never reaches it is not counted. Changes in the portal reach a running server at its next daily refresh. Restart it, or lower refreshMs, to apply a change sooner.