AI Crawlers

AI assistants send automated visitors, called crawlers, to read websites. Some are collecting pages to train the next model. Others arrive because someone asked a question a minute ago and the assistant went to find the answer. AI Crawlers shows which ones visit your site, how often, and which pages they read.

AI Crawlers is included on Plus, Pro and Agency, and during a Pro trial.

Why it needs connecting

Everything else in brandflare is measured from the outside, by asking the engines questions. Visits to your website can't be seen from the outside. They're recorded in your website's visitor log, which notes every visit, AI assistants included. So this section needs a copy of that log, and your web team or hosting company can switch that on.

Until the site is connected, the page explains what it needs in three steps:

1. Create a key. It's what lets your web team send the log to us, and it only works for this brand.
2. Your web team connects it. They send your visitor log to brandflare with the key. It usually takes under an hour. A folded section on the page has the technical details to hand over.
3. See AI visits here. Visits from AI assistants appear within a day.

You can forward the log unfiltered. brandflare picks out the AI visitors, and everything else in the log is ignored.

Stuck? The page has a button that opens a support request, already filled in, asking us to help with the connection.

What you see once visits arrive

A summary, split into the two kinds of visit that matter:

• Answering now. A crawler opened your page because someone was asking a question at that moment, or to find pages for an assistant's search results. This is AI reading your site live, and it's the traffic that changes answers this week.
• Learning for later. A crawler collecting pages to train future models. Slower to pay back, but it's how your site gets into what the next model knows by heart.

If none of the visits were to answer a question, the page says so: AI isn't reading your pages live yet.

Visits per day, one line per company, in the same colours the engines have everywhere else in the product.

Each crawler, named by its assistant, with what its visits mean and when it last came by. Among them:

Crawlers from Apple, Meta, Amazon and others are recognised too.

The pages AI reads most. Make sure these are accurate and current. They're what AI learns from.

Reading it well

• No visits from a crawler you'd expect? Check your robots.txt and any bot protection on your CDN first. Blocking training crawlers is a legitimate licensing choice, with a slow cost to visibility. Blocking the answering crawlers means the assistant can't read your page when a customer asks about you, which is rarely what anyone intended. More in do AI chatbots crawl my website?
• Plenty of learning, no answering? Your site is known but isn't being chosen when engines search. That's a content and sources problem: see what Sources says the engines are citing instead.
• Compare with Your pages. AI Crawlers shows the pages AI read. Your pages, in Sources, shows the pages AI cited in answers to your questions. A page that's read often and never cited isn't answering the question clearly enough.
• Visits stopped? If the last AI visit was a while ago, the page tells you the date and suggests checking with your web team that the log is still being sent.

What it isn't

AI Crawlers isn't web analytics: it doesn't count people, and it doesn't tell you how many customers arrived from an assistant. It answers a narrower question that nothing else does: is AI reading my site, which AI, and what is it reading?