About our crawler

We visit public business websites to work out what kind of creative work a business does. Here's exactly how it behaves.

How to identify it

It sends this user agent string:

KilnBot/0.1 (+https://makersmap.co.uk/about/bot; contact [email protected])

What it does

• Reads and honours robots.txt before fetching anything.
• Waits at least two seconds between requests to the same site.
• Fetches your home page, and sometimes your contact or about page.
• Keeps a short text extract, your council area, and whether the site displays a company registration number. Not the whole page.
• Deletes stored text after 90 days.

What it doesn't do

• Submit forms, or collect email addresses for marketing.
• Ignore robots.txt, or crawl behind a login.
• Hammer your server.

To block it

Add this to your robots.txt and we'll stop:

User-agent: KilnBot
Disallow: /

Or ask us to remove your listing and we won't come back.