Key takeaways
- Allow Meta-WebIndexer: Meta says it is used to improve Meta AI search results and to cite and link your content in Meta AI responses.
- Meta-ExternalAgent is used for AI training and indexing — allow or block based on your policy.
- Meta AI lives inside social and messaging apps, so your Instagram and Facebook presence matters.
- Clear, mobile-first pages with direct answers suit how people use Meta AI on their phones.
- Meta AI is especially relevant for consumer, D2C, local and lifestyle brands.
How Meta AI finds and cites content
When a question needs current information, Meta AI can run a web search and include linked sources in its response. Meta runs a specific crawler for this, Meta-WebIndexer, which it describes as improving Meta AI search result quality and helping it cite and link to your content.
Meta also operates Meta-ExternalAgent, used to train foundation models or improve products by indexing content directly, and Meta-ExternalFetcher, which fetches individual links when a user asks. Each can be controlled separately in robots.txt.
Because Meta AI sits inside WhatsApp, Instagram and Facebook, users frequently ask about products, places, recipes, travel and recommendations. Brands with clear websites and active, consistent profiles on Meta’s platforms are better placed for these questions.
Let Meta AI read your site
These are the user agents Meta uses. Check them in robots.txt and in your CDN or firewall — bot protection often blocks them silently.
| User agent | What it does | For visibility |
|---|---|---|
| meta-webindexer | Improves Meta AI search results and helps cite and link your content in Meta AI responses. | Allow |
| meta-externalfetcher | Fetches individual links when a user requests them, for agentic AI features. | Allow (user-triggered) |
| meta-externalagent | Crawls for AI model training or to improve products by indexing content directly. | Your choice |
| facebookexternalhit | Builds link previews (title, description, image) when your URL is shared in Meta apps. | Allow |
Example robots.txt rules
User-agent: meta-webindexer
Allow: /
User-agent: meta-externalfetcher
Allow: /
User-agent: meta-externalagent
Allow: /
User-agent: facebookexternalhit
Allow: /Meta notes its crawlers may cache robots.txt for up to 24 hours, so changes can take a day to apply.
Step-by-step: how to rank on Meta AI
- 1
Allow Meta-WebIndexer and check your CDN
Add an explicit allow rule for meta-webindexer and confirm your firewall or bot protection is not blocking it. This is the crawler tied to Meta AI search citations.
- 2
Get your link previews right
Set Open Graph title, description and image on every important page. When your links are shared in WhatsApp or Instagram DMs, a clean preview builds trust and clicks.
- 3
Write mobile-first, direct answers
Meta AI is used on phones. Keep answers near the top, use short paragraphs and scannable lists, and make sure pages load fast on mobile networks.
- 4
Keep Instagram and Facebook profiles complete
Use consistent brand names, categories, descriptions, websites and contact details across your Meta profiles and your site. Consistency helps AI systems connect the dots.
- 6
Cover consumer questions in your category
Publish guides that answer practical, everyday questions buyers ask — sizing, how to choose, how to use, where to buy, price comparisons. These map closely to how people prompt Meta AI.
Common mistakes that keep brands out of Meta AI
- Blocking all Meta crawlers to stop training, which also removes you from Meta AI citations.
- Missing or broken Open Graph tags, leading to poor link previews in WhatsApp and Instagram.
- Inconsistent business details between your site and your Meta profiles.
- Desktop-first pages that are slow or awkward on mobile.
How to measure your Meta AI visibility
- Monthly Meta AI prompt tracking in WhatsApp or the Meta AI app for your key questions.
- Server-log activity from meta-webindexer on key pages.
- Referral traffic from meta.ai, facebook.com, instagram.com and WhatsApp-shared links.
- Engagement and review volume on your Meta profiles.
Meta AI ranking FAQ
Which Meta crawler should I allow for Meta AI?
Allow meta-webindexer. Meta describes it as the crawler that improves Meta AI search results and helps cite and link your content in Meta AI responses.
Can I block Meta AI training but stay visible?
Yes. You can disallow meta-externalagent (training and product indexing) while allowing meta-webindexer, which handles Meta AI search citations.
Does Meta AI appear inside WhatsApp?
Yes. Meta AI is available inside WhatsApp, Instagram, Facebook and Messenger, which is why it is an important channel for consumer brands.
Is Meta AI relevant for B2B?
Less than for consumer brands, but it can matter for SMB-focused B2B and for markets where WhatsApp is the primary communication channel.
How long does it take for robots.txt changes to apply?
Meta says its crawlers may cache robots.txt for up to 24 hours, so allow a day for changes to take effect.