A few days ago I looked at the traffic in my Cloudflare dashboard and noticed that AI crawlers already make up a noticeable share. Tools like ChatGPT and Perplexity send crawlers to fetch site content, feed it to their models, and use it for retrieval and citation.

It made me stop and think: should blog content be open to AI at all? And if I do open it, what can I actually do so those crawlers understand what I wrote?

This post isn’t about configuring a specific blog framework. It’s about a few framework-agnostic ideas, useful whether I run Hugo, WordPress, or a site I built myself.

Decide Whether to Let AI Crawl You

The default reaction of many site owners is to block. They worry about content being “stolen,” or about AI answers stealing their traffic.

Flip it around: when AI cites my content, it’s exposure. Someone asks a question, AI says “read this article,” and I’ve just gained another entrance. Personal blogs don’t live off search engines alone, so being cited by AI is free referral traffic.

Unless my content sits behind a paywall, or I explicitly don’t want it appearing in AI answers, letting crawlers in is a good deal. This question is worth thinking through because the default behavior of robots.txt has probably already decided for me — most blogs configure nothing, which means everything is allowed.

Once I’ve made that decision, the rest actually matters.

Make the Content Itself AI-Friendly

AI reads my blog differently from search engine crawlers. Search engines care about keywords and links. AI cares about whether it can understand what I wrote.

A few plain principles that work regardless of the tech stack:

• Say what the post is about up front. AI often reads only the first few paragraphs to make a summary. If the opening is pleasantries and filler, the summary it produces will be filler too.
• Write plainly, avoid beating around the bush. Humans can decode clever titles and vague phrasing. AI just guesses.
• Make titles that describe the content. A “my journey with X” title is less useful than “how to fix X.”

None of this requires writing code. It just requires the awareness.

Offer AI an Index

More sites are shipping an extra index file for AI to read — a plain list of “what’s on this site.” The emerging de facto standard here is llms.txt. Some AI tools, when they reference a site, first request https://site/llms.txt, get a structured table of contents, and decide whether to dig deeper.

If my content is already Markdown or clean text, this is the easiest thing to do. It’s like handing AI a table of contents. I may not even need to maintain it by hand — many static blog frameworks can generate it at build time. And even manually, it’s just a short list.

The benefit goes both ways: AI gets a clear catalog and is more likely to cite me, and my content gets read in full instead of being reduced to its opening paragraphs.

Do Agent Discovery While I’m At It

llms.txt solves “let AI understand my content.” But how AI products hook into websites has moved beyond crawling. More agents now probe a site first: are there standard entry points? What resources are available? Only then do they decide how to connect. This batch of community-convention discovery points is usually called “Agent Discovery.”

The approach is stack-agnostic. Here’s what I did for this Hugo + Cloudflare Pages site, using only plain static techniques:

• Declare Content-Signal in robots.txt. A single line, Content-Signal: ai-train=no, search=yes, ai-input=no, spells out my content-usage preferences instead of leaving them unstated.
• Link headers in responses. An _headers file adds a Link header to every page, pointing at machine-readable resources such as llms.txt, RSS, and sitemap. AI gets the HTML and, in the same response, sees what other structured entry points exist.
• /.well-known/ standard endpoints. Added /.well-known/api-catalog (a linkset-format resource catalog), /.well-known/agent-skills/index.json (a site skills index), and auth.md (declaring the content fully public with no auth). These locations are community conventions that agents proactively probe.

No dynamic services are involved — just a few static files and a few lines of config. But to AI, my site goes from “a blob of HTML” to “an object with clear interfaces and documentation.”

Awareness Beats Technique

Technical solutions keep changing. What’s popular today may be replaced tomorrow. But these things stay relevant:

1. Decide whether my content should be open to AI.
2. Make the content itself easy to understand.
3. Offer AI a structured index.
4. Offer Agent Discovery entry points while I’m at it.

None of them are hard, and none require knowing a particular framework. The hard part is simply knowing they exist — and remembering them when I build a site or write content.

Article Link: