Key Takeaways
- AI crawlers have different purposes, such as search, training and user-triggered systems, which may follow different rules.
- robots.txt still matters. Platforms such as OpenAI and Google provide controls for certain automated crawlers and AI-related uses.
- llms.txt is useful, but not magic; it can give AI agents a cleaner guide to your content, but it does not guarantee rankings or citations.
- Site structure comes first, and clear HTML, internal links, headings and accessible content remain more important.
- Audit AI crawlability separately. A site can be open to Googlebot while restricting AI-related crawlers.
AI crawlers read your site by requesting pages, interpreting accessible content and following technical signals that determine what they can access. Some work similarly to search engine crawlers, while others are used for AI search, model training or user-triggered browsing.
That matters for Singapore businesses. When we work on technical SEO in Singapore, we are no longer checking only whether Googlebot can reach a page. We also need to understand whether AI systems can discover important content and whether robots.txt, firewalls or security tools are blocking them.
Then there is llms.txt.
The proposed standard received a major Version 2 update in August 2026 and is attracting growing interest. But it is not an “AI SEO sitemap” that automatically gets your company cited by ChatGPT.
Here is where it actually fits.
Table of Contents
How Do AI Crawlers Differ From Googlebot?
Googlebot mainly crawls websites so Google can discover and index pages for Google Search.
AI crawling is more fragmented.
OpenAI, for example, separates several functions:
- OAI-SearchBot helps surface websites in ChatGPT search.
- GPTBot crawls content that may be used to improve and train OpenAI’s generative AI models.
- ChatGPT-User handles certain user-triggered requests and is not an automatic web crawler in the same sense.
OpenAI notes that robots.txt rules may not apply to ChatGPT-User requests in the same way they apply to automated crawlers.
Google has similar distinctions. Google-Extended lets publishers control certain uses of content by Gemini-related systems without affecting inclusion or ranking in Google Search.
One technical detail matters here: Google-Extended is a robots.txt product token, not a separate HTTP crawler user agent.
So the assumption that:
“If Google can crawl me, AI can crawl me.”
is not necessarily true.
What Is llms.txt And Do I Need One?
llms.txt is a proposed Markdown-based standard that gives AI agents a concise overview of a website and links to important resources.
Think of it as a curated guide rather than a replacement for sitemap.xml.
The original proposal appeared in September 2024. Version 2, updated in August 2026, allows an llms.txt file at /llms.txt or within a particular section of a website.
A file can include:
- An H1 naming the site or project
- A short summary
- Additional context
- Sections containing links to important resources
- Optional descriptions explaining those links
The key word is proposed.
llms.txt is not a Google ranking directive, and there is currently no established evidence that publishing one automatically improves Google rankings or guarantees citations inside AI answers.
Would we implement one for every client?
Not automatically.
For documentation-heavy sites, SaaS companies, large publishers or businesses actively investing in AI visibility, it can be a sensible low-cost addition.
For a small service website with poor navigation or broken internal links, those fundamentals come first.
How Do I Let AI Bots Crawl My Site?
Start with robots.txt.
Before adding AI-specific files, check whether your existing setup allows the crawlers you want.
OpenAI documents OAI-SearchBot and GPTBot separately, which means a publisher can allow OAI-SearchBot for ChatGPT search while blocking GPTBot for model-training use.
A typical audit should check:
- robots.txt: See which AI crawlers or control tokens are allowed or blocked.
- Server responses: Important pages should return proper 200 status codes.
- Firewall and CDN rules: Security tools can block crawlers even when robots.txt allows them.
- HTML accessibility: Important information should be available in crawlable page content.
- Internal links: Crawlers need clear paths to priority pages.
- Canonicalisation: Avoid competing URLs for the same content.
For Singapore businesses using Cloudflare, WordPress security tools or aggressive bot protection, firewall rules are particularly important.
Your robots.txt file may say “allow” while the server still rejects the request.
Does Blocking AI Bots Hurt Visibility?
Potentially, but it depends on what you block.
Blocking a model-training crawler does not automatically mean disappearing from every AI search product.
OpenAI separates GPTBot from OAI-SearchBot. Its documentation says sites opting out of OAI-SearchBot will not appear as sources in ChatGPT search answers, although navigational links may still appear.
At Google, blocking Google-Extended does not affect inclusion or ranking in Google Search.
This is why copying a “block every AI bot” robots.txt template without understanding what each control does can be risky.
The better question is:
What are you trying to prevent, and what visibility are you willing to give up?
A publisher concerned about model training may make a different decision from a B2B company that wants its services surfaced in AI search.
What Structure Helps AI Read My Content?
Before creating AI-specific files, make the website itself easy to interpret.
Many supposed “AI optimisation” issues are really ordinary technical SEO and content architecture problems.
Compare a vague service page with one structured like this:
- H1: SEO Services in Singapore
- Intro: What the service is and who it serves
- H2: What Is Included?
- H2: How Does the Process Work?
- H2: How Much Does It Cost?
- FAQ: Direct answers to common questions
The second version creates clearer relationships between topics.
For both search crawling and AI retrieval, we generally want:
- Clear heading hierarchy
- Descriptive internal links
- Useful structured data
- Consistent business information
- Direct answers before longer explanations
Structured data can help search systems interpret entities and relationships, but it should be treated as supporting markup rather than a guaranteed AI-visibility signal.
Likewise, llms.txt cannot rescue poorly organised pages.
Where Does llms.txt Fit Into The Stack?
We see llms.txt as an additional discovery and context layer, not the foundation of AI SEO.
A sensible order is:
- Crawlability
- Content Structure
- Internal Linking
- Entity Signals
- Structured Data
- AI-Specific Enhancements
llms.txt belongs near the end.
Its strongest use case is giving an AI agent a curated entry point into a large website instead of making it interpret hundreds or thousands of pages blindly.
Version 2 also introduces ways to connect normal pages with Markdown alternatives and corresponding llms.txt files through standard link relationships.
That makes the idea more useful than simply uploading a text file and forgetting about it.
Still, claims such as “llms.txt improves ChatGPT rankings” should be treated cautiously. Different AI platforms use different crawlers, indexes and retrieval systems. There is no universal AI ranking algorithm controlled by one file.
How Do I Check AI Crawl Access?
Start by visiting:
yourdomain.com/robots.txt
Look for crawler or control tokens such as GPTBot, OAI-SearchBot and Google-Extended.
Remember that Google-Extended is a robots.txt product token rather than a separate crawler user agent.
Then check:
Server logs: Are AI crawlers actually requesting your URLs?
CDN or WAF logs: Are requests being blocked, rate-limited or challenged?
Live URLs: Do priority pages load correctly without authentication or unexpected redirects?
Source code: Is important information present in crawlable HTML?
llms.txt: If you use one, does /llms.txt load correctly and do its links work?
The useful question is no longer simply “Can Google index the website?”, but also “Who can access the site, what can they access, and what information do they receive?”
That is a better way to think about AI search visibility, retrieval and crawlability.
What Should Singapore Businesses Prioritise Right Now?
There is no need to rebuild your website around AI bots.
A practical priority order is:
First: Fix traditional crawlability. Important pages should be accessible, internally linked and canonicalised correctly.
Next: Review AI crawler permissions. Decide separately how you want search, training and user-triggered systems handled.
Then: Improve your content structure. Use clear service descriptions, useful headings, factual answers and appropriate structured data.
Finally: Test emerging standards. Add llms.txt where it makes sense and monitor how relevant crawlers behave.
That is less exciting than installing an “AI SEO plugin” and expecting instant citations, but it is far closer to sustainable technical SEO.
Read More: Common Technical Mistakes That Block Your AI Visibility
Getting AI Crawlers Right
AI crawlers add another layer to website crawlability, but the fundamentals have not changed.
Singapore businesses still need technically accessible pages, sensible site architecture and clear information. The difference is that we now need to consider more than Googlebot when deciding who can access that content.
At Rankpage, we treat AI crawler accessibility and emerging standards such as llms.txt as part of the wider technical SEO picture, not as shortcuts.
If your website needs a stronger foundation for both traditional search and emerging AI discovery, our technical SEO in Singapore can help identify the gaps worth fixing.
Sources
- OpenAI — Publishers and Developers FAQ
- OpenAI — ChatGPT Search
- OpenAI — Advertiser Guidance for Allowing OpenAI Web Crawlers
- Google Crawling Infrastructure — Google’s Common Crawlers
- Google Crawling Infrastructure — User-Triggered Fetchers
- Google Search Central — URL Canonicalization
- llms.txt — Specification
- llms.txt — Version 2 Changes
Frequently Asked Questions About How AI Crawlers Read Your Site
What Is llms.txt?
llms.txt is a proposed Markdown-based standard that provides AI agents with a concise overview of a website and links to important resources. It is designed to complement standards such as robots.txt and XML sitemaps rather than replace them.
Does llms.txt Improve SEO Rankings?
There is currently no established evidence that publishing llms.txt directly improves Google rankings. Its purpose is to help compatible AI agents discover and interpret important website resources.
What Is GPTBot?
GPTBot is an OpenAI crawler used to access content that may be used to improve and train OpenAI’s generative AI models. Website owners can manage GPTBot through robots.txt.
What Is The Difference Between GPTBot And OAI-SearchBot?
GPTBot is associated with model-training use, while OAI-SearchBot is used for surfacing websites within ChatGPT search. OpenAI provides separate robots.txt controls for the two.
Can I Block AI Crawlers Without Blocking Google?
Yes. Automated crawlers can generally be managed separately through robots.txt. Google-Extended is a product token rather than a separate crawler, and Google states that using it does not affect inclusion or ranking in Google Search.
How Can I Make My Website Easier For AI Crawlers To Read?
Start with strong technical SEO: allow appropriate crawlers, use clear HTML and headings, maintain logical internal links, remove crawl barriers and make important business information explicit. llms.txt can then be considered as an additional discovery layer.
