llms.txt checker

See whether a site publishes llms.txt, whether the file follows the proposed format, and how it sits alongside the other things agents can find.

Try:

lookupowl.com

Results for lookupowl.com

Checked 2026-11-15live lookup496 ms

Checklist, not a score

Ready

19 of 21 checks pass. 2 are information only. Every group passes, so the level is the top one.

Counted, not weighted

19 pass, 0 warning, 0 fail, 2 information.

Can AI read it?5 pass, 1 info

Whether the page answers, robots.txt and which AI crawlers it lets in.

What does it see before JavaScript?3 pass

The text, heading and links in the first response, before any script runs.

Does it describe itself?5 pass

Title, description, canonical link and structured data.

Can it be found?2 pass, 1 info

Sitemap, llms.txt and how fast the first byte arrives.

Can an agent use it?4 pass

Markdown, an API catalog, an MCP server card and Link headers.

llms.txt

This site publishes an llms.txt file. It also has llms-full.txt.

llms.txt is optional. Google says it does not use it for Search, and no search engine requires it. Some assistants and coding agents read it. A missing file is information, not a failure. We read the file; we do not follow its links.

Where we looked: https://lookupowl.com/llms.txt

Checklist21

Failures first, then warnings, passes and information. Every line says what we saw and where.

GoodThe page answers page-status

https://lookupowl.com/ answered 200.

Where we looked: https://lookupowl.com/

Goodrobots.txt robots-present

robots.txt found (200, text/plain), 9 groups of rules.

Where we looked: https://lookupowl.com/robots.txt

GoodAI crawlers are allowed ai-crawlers

GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended and 9 more are allowed.

Where we looked: robots.txt

GoodContent signals content-signals

Content-Signal: ai-train=yes, search=yes, ai-input=yes

Where we looked: robots.txt

GoodPage-level directives page-directives

The page sends index, follow, which does not block reading.

Where we looked: https://lookupowl.com/

GoodText before JavaScript prejs-text

About 1,606 words are in the first response, before any JavaScript runs.

Where we looked: https://lookupowl.com/

GoodMain heading prejs-headings

One h1 in the first response.

Where we looked: https://lookupowl.com/

GoodLinks a crawler can follow prejs-links

155 links in the first response.

Where we looked: https://lookupowl.com/

GoodPage title title

"LookupOwl: free DNS, WHOIS and network tools"

Where we looked: https://lookupowl.com/

GoodMeta description meta-description

"Free WHOIS, RDAP, DNS, IP, SSL, SPF and DMARC checks in seconds. No signup, no captcha, no ads, plus a free API and MCP …"

Where we looked: https://lookupowl.com/

GoodCanonical link canonical

https://lookupowl.com/

Where we looked: https://lookupowl.com/

GoodOpen Graph title og-title

"LookupOwl: free DNS, WHOIS and network tools"

Where we looked: https://lookupowl.com/

GoodStructured data json-ld

Found 2 JSON-LD blocks (WebSite, FAQPage).

Where we looked: https://lookupowl.com/

GoodSitemap sitemap

Named in robots.txt (200, application/xml).

Where we looked: https://lookupowl.com/sitemap.xml

GoodResponse speed ttfb

First byte in 0.20 s from one cold request.

Where we looked: https://lookupowl.com/

GoodMarkdown for agents markdown

Accept: text/markdown returns text/markdown, about 2,777 tokens.

Where we looked: https://lookupowl.com/

GoodAPI catalog api-catalog

/.well-known/api-catalog answers 200 (application/linkset+json, RFC 9727).

Where we looked: https://lookupowl.com/.well-known/api-catalog

GoodMCP server card mcp-card

/.well-known/mcp/server-card.json answers 200 (application/json).

Where we looked: https://lookupowl.com/.well-known/mcp/server-card.json

GoodLink headers link-headers

The Link header advertises api-catalog, service-desc, service-doc, describedby.

Where we looked: https://lookupowl.com/

NoteSEO crawlers seo-bots

AhrefsBot, SemrushBot, MJ12bot, DotBot and 3 more are asked to stay out. These are not AI crawlers.

Where we looked: robots.txt

Notellms.txt llms-txt

Found. 47 links in 10 sections, titled "LookupOwl". Optional: Google says it does not use it; some assistants and coding agents read it.

Where we looked: https://lookupowl.com/llms.txt

Crawler access14

GPTBotAllowedTraining
OpenAI. Nothing in robots.txt keeps it out of this page.
Deciding line: Allow: /
ClaudeBotAllowedTraining
Anthropic. Nothing in robots.txt keeps it out of this page.
Deciding line: Allow: /
Google-ExtendedAllowedTraining
Google. Nothing in robots.txt keeps it out of this page.
Deciding line: Allow: /
CCBotAllowedTraining
Common Crawl. Nothing in robots.txt keeps it out of this page.
Deciding line: Allow: /
Applebot-ExtendedAllowedTraining
Apple. Nothing in robots.txt keeps it out of this page.
Deciding line: Allow: /
Meta-ExternalAgentAllowedTraining
Meta. Nothing in robots.txt keeps it out of this page.
Deciding line: Allow: /
BytespiderAllowedTraining
ByteDance. Nothing in robots.txt keeps it out of this page.
Deciding line: Allow: /
OAI-SearchBotAllowedSearch
OpenAI. Nothing in robots.txt keeps it out of this page.
Deciding line: Allow: /
Claude-SearchBotAllowedSearch
Anthropic. Nothing in robots.txt keeps it out of this page.
Deciding line: Allow: /
PerplexityBotAllowedSearch
Perplexity. Nothing in robots.txt keeps it out of this page.
Deciding line: Allow: /
ChatGPT-UserAllowedAssistant
OpenAI. Nothing in robots.txt keeps it out of this page.
Deciding line: Allow: /
Claude-UserAllowedAssistant
Anthropic. Nothing in robots.txt keeps it out of this page.
Deciding line: Allow: /
Perplexity-UserAllowedAssistant
Perplexity. Nothing in robots.txt keeps it out of this page.
Deciding line: Allow: /
MistralAI-UserAllowedAssistant
Mistral. Nothing in robots.txt keeps it out of this page.
Deciding line: Allow: /

Training collects pages to train models. Search builds an AI search index. Assistant fetches a page because a person asked. We read robots.txt only; we never pretend to be these crawlers.

What AI sees

The first 600 characters a crawler reads before JavaScript runs (1,606 words in all).

Skip to content Lookup Owl Tools Domain Who owns a name, and is it set up? WHOIS lookup Domain report Domain availabilit…

Structured data found in the first response:

WebSite
FAQPage

Structured result (JSON) · 11 KB

{
  "target": "lookupowl.com",
  "finalUrl": "https://lookupowl.com/",
  "redirectedTo": null,
  "blockedUs": false,
  "blockedStatus": null,
  "summary": {
    "pass": 19,
    "warn": 0,
    "fail": 0,
    "info": 2,
    "unchecked": 0,
    "total": 21,
    "level": "ready"
  },
  "groups": [
    {
      "id": "access",
      "title": "Can AI read it?",
      "result": "pass",
      "checks": [
        {
          "id": "page-status",
          "title": "The page answers",
          "status": "pass",
          "couldntCheck": false,
          "evidence": "https://lookupowl.com/ answered 200.",
          "source": "https://lookupowl.com/",
          "fix": null,
          "snippet": null
        },
        {
          "id": "robots-present",
          "title": "robots.txt",
          "status": "pass",
          "couldntCheck": false,
          "evidence": "robots.txt found (200, text/plain), 9 groups of rules.",
          "source": "https://lookupowl.com/robots.txt",
          "fix": null,
          "snippet": null
        },
        {
          "id": "ai-crawlers",
          "title": "AI crawlers are allowed",
          "status": "pass",
          "couldntCheck": false,
          "evidence": "GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended and 9 more are allowed.",
          "source": "robots.txt",
          "fix": null,
          "snippet": null
        },
        {
          "id": "content-signals",
          "title": "Content signals",
          "status": "pass",
          "couldntCheck": false,
          "evidence": "Content-Signal: ai-train=yes, search=yes, ai-input=yes",
          "source": "robots.txt",
          "fix": null,
          "snippet": null
        },
        {
          "id": "seo-bots",
          "title": "SEO crawlers",
          "status": "info",
          "couldntCheck": false,
          "evidence": "AhrefsBot, SemrushBot, MJ12bot, DotBot and 3 more are asked to stay out. These are not AI crawlers.",
          "source": "robots.txt",
          "fix": null,
          "snippet": null
        },
        {
          "id": "page-directives",
          "title": "Page-level directives",
          "status": "pass",
          "couldntCheck": false,
          "evidence": "The page sends index, follow, which does not block reading.",
          "source": "https://lookupowl.com/",
          "fix": null,
          "snippet": null
        }
      ]
    },
    {
      "id": "prejs",
      "title": "What does it see before JavaScript?",
      "result": "pass",
      "checks": [
        {
          "id": "prejs-text",
          "title": "Text before JavaScript",
          "status": "pass",
          "couldntCheck": false,
          "evidence": "About 1,606 words are in the first response, before any JavaScript runs.",
          "source": "https://lookupowl.com/",
          "fix": null,
          "snippet": null
        },
        {
          "id": "prejs-headings",
          "title": "Main heading",
          "status": "pass",
          "couldntCheck": false,
          "evidence": "One h1 in the first response.",
          "source": "https://lookupowl.com/",
          "fix": null,
          "snippet": null
        },
        {
          "id": "prejs-links",
          "title": "Links a crawler can follow",
          "status": "pass",
          "couldntCheck": false,
          "evidence": "155 links in the first response.",
          "source": "https://lookupowl.com/",
          "fix": null,
          "snippet": null
        }
      ]
    },
    {
      "id": "self",
      "title": "Does it describe itself?",
      "result": "pass",
      "checks": [
        {
          "id": "title",
          "title": "Page title",
          "status": "pass",
          "couldntCheck": false,
          "evidence": "\"LookupOwl: free DNS, WHOIS and network tools\"",
          "source": "https://lookupowl.com/",
          "fix": null,
          "snippet": null
        },
        {
          "id": "meta-description",
          "title": "Meta description",
          "status": "pass",
          "couldntCheck": false,
          "evidence": "\"Free WHOIS, RDAP, DNS, IP, SSL, SPF and DMARC checks in seconds. No signup, no captcha, no ads, plus a free API and MCP …\"",
          "source": "https://lookupowl.com/",
          "fix": null,
          "snippet": null
        },
        {
          "id": "canonical",
          "title": "Canonical link",
          "status": "pass",
          "couldntCheck": false,
          "evidence": "https://lookupowl.com/",
          "source": "https://lookupowl.com/",
          "fix": null,
          "snippet": null
        },
        {
          "id": "og-title",
          "title": "Open Graph title",
          "status": "pass",
          "couldntCheck": false,
          "evidence": "\"LookupOwl: free DNS, WHOIS and network tools\"",
          "source": "https://lookupowl.com/",
          "fix": null,
          "snippet": null
        },
        {
          "id": "json-ld",
          "title": "Structured data",
          "status": "pass",
          "couldntCheck": false,
          "evidence": "Found 2 JSON-LD blocks (WebSite, FAQPage).",
          "source": "https://lookupowl.com/",
          "fix": null,
          "snippet": null
        }
      ]
    },
    {
      "id": "found",
      "title": "Can it be found?",
      "result": "pass",
      "checks": [
        {
          "id": "sitemap",
          "title": "Sitemap",
          "status": "pass",
          "couldntCheck": false,
          "evidence": "Named in robots.txt (200, application/xml).",
          "source": "https://lookupowl.com/sitemap.xml",
          "fix": null,
          "snippet": null
        },
        {
          "id": "llms-txt",
          "title": "llms.txt",
          "status": "info",
          "couldntCheck": false,
          "evidence": "Found. 47 links in 10 sections, titled \"LookupOwl\". Optional: Google says it does not use it; some assistants and coding agents read it.",
          "source": "https://lookupowl.com/llms.txt",
          "fix": null,
          "snippet": null
        },
        {
          "id": "ttfb",
          "title": "Response speed",
          "status": "pass",
          "couldntCheck": false,
          "evidence": "First byte in 0.20 s from one cold request.",
          "source": "https://lookupowl.com/",
          "fix": null,
          "snippet": null
        }
      ]
    },
    {
      "id": "agent",
      "title": "Can an agent use it?",
      "result": "pass",
      "checks": [
        {
          "id": "markdown",
          "title": "Markdown for agents",
          "status": "pass",
          "couldntCheck": false,
          "evidence": "Accept: text/markdown returns text/markdown, about 2,777 tokens.",
          "source": "https://lookupowl.com/",
          "fix": null,
          "snippet": null
        },
        {
          "id": "api-catalog",
          "title": "API catalog",
          "status": "pass",
          "couldntCheck": false,
          "evidence": "/.well-known/api-catalog answers 200 (application/linkset+json, RFC 9727).",
          "source": "https://lookupowl.com/.well-known/api-catalog",
          "fix": null,
          "snippet": null
        },
        {
          "id": "mcp-card",
          "title": "MCP server card",
          "status": "pass",
          "couldntCheck": false,
          "evidence": "/.well-known/mcp/server-card.json answers 200 (application/json).",
          "source": "https://lookupowl.com/.well-known/mcp/server-card.json",
          "fix": null,
          "snippet": null
        },
        {
          "id": "link-headers",
          "title": "Link headers",
          "status": "pass",
          "couldntCheck": false,
          "evidence": "The Link header advertises api-catalog, service-desc, service-doc, describedby.",
          "source": "https://lookupowl.com/",
          "fix": null,
          "snippet": null
        }
      ]
    }
  ],
  "crawlerMatrix": [
    {
      "bot": "GPTBot",
      "vendor": "OpenAI",
      "kind": "training",
      "access": "allowed",
      "byChoice": false,
      "matchedRule": "Allow: /"
    },
    {
      "bot": "ClaudeBot",
      "vendor": "Anthropic",
      "kind": "training",
      "access": "allowed",
      "byChoice": false,
      "matchedRule": "Allow: /"
    },
    {
      "bot": "Google-Extended",
      "vendor": "Google",
      "kind": "training",
      "access": "allowed",
      "byChoice": false,
      "matchedRule": "Allow: /"
    },
    {
      "bot": "CCBot",
      "vendor": "Common Crawl",
      "kind": "training",
      "access": "allowed",
      "byChoice": false,
      "matchedRule": "Allow: /"
    },
    {
      "bot": "Applebot-Extended",
      "vendor": "Apple",
      "kind": "training",
      "access": "allowed",
      "byChoice": false,
      "matchedRule": "Allow: /"
    },
    {
      "bot": "Meta-ExternalAgent",
      "vendor": "Meta",
      "kind": "training",
      "access": "allowed",
      "byChoice": false,
      "matchedRule": "Allow: /"
    },
    {
      "bot": "Bytespider",
      "vendor": "ByteDance",
      "kind": "training",
      "access": "allowed",
      "byChoice": false,
      "matchedRule": "Allow: /"
    },
    {
      "bot": "OAI-SearchBot",
      "vendor": "OpenAI",
      "kind": "search",
      "access": "allowed",
      "byChoice": false,
      "matchedRule": "Allow: /"
    },
    {
      "bot": "Claude-SearchBot",
      "vendor": "Anthropic",
      "kind": "search",
      "access": "allowed",
      "byChoice": false,
      "matchedRule": "Allow: /"
    },
    {
      "bot": "PerplexityBot",
      "vendor": "Perplexity",
      "kind": "search",
      "access": "allowed",
      "byChoice": false,
      "matchedRule": "Allow: /"
    },
    {
      "bot": "ChatGPT-User",
      "vendor": "OpenAI",
      "kind": "assistant",
      "access": "allowed",
      "byChoice": false,
      "matchedRule": "Allow: /"
    },
    {
      "bot": "Claude-User",
      "vendor": "Anthropic",
      "kind": "assistant",
      "access": "allowed",
      "byChoice": false,
      "matchedRule": "Allow: /"
    },
    {
      "bot": "Perplexity-User",
      "vendor": "Perplexity",
      "kind": "assistant",
      "access": "allowed",
      "byChoice": false,
      "matchedRule": "Allow: /"
    },
    {
      "bot": "MistralAI-User",
      "vendor": "Mistral",
      "kind": "assistant",
      "access": "allowed",
      "byChoice": false,
      "matchedRule": "Allow: /"
    }
  ],
  "llmsTxt": {
    "present": true,
    "full": true,
    "url": "https://lookupowl.com/llms.txt",
    "errors": []
  },
  "page": {
    "status": 200,
    "firstByteMs": 205,
    "words": 1606,
    "excerpt": "Skip to content Lookup Owl Tools Domain Who owns a name, and is it set up? WHOIS lookup Domain report Domain availabilit…",
    "jsonLd": [
      {
        "types": [
          "WebSite"
        ],
        "valid": true
      },
      {
        "types": [
          "FAQPage"
        ],
        "valid": true
      }
    ]
  },
  "notes": [],
  "partial": false
}

Next step

HTTP headers and redirectsSee every header the page sends
Check lookupowl.com in HTTP headers and redirects
Security headersGrade its security headers
Check lookupowl.com in Security headers
Is it down?Is it up from other countries?
Check lookupowl.com in Is it down?

What llms.txt is, and what it is not

llms.txt is a plain Markdown file at the root of a site, proposed at llmstxt.org as a short guide for language models: who you are, and where your best documents live. Coding agents and some assistants read it to find documentation without crawling everything. It is a proposal, not a standard that any search engine requires.

Be careful with claims in either direction. Google has said it does not use llms.txt for Search, and it neither helps nor harms there. No one has shown that adding one makes assistants cite you more. So this page treats a missing file as information, never as a failure, and never lets it change a level.

The format we check against

The first line is a single H1 with the name of the site or project: # Name.
An optional blockquote follows with a one-sentence summary.
Optional paragraphs of plain notes may come next, with no headings.
Then H2 sections, each holding a list of links in the form - [Title](https://example.com/page): what it is.
A section called Optional marks links a reader short of room may skip.

Text

# Example Co

> Example Co makes a small DNS tool and publishes its docs for developers.

## Docs

- [Getting started](https://example.com/docs/start): install and first query
- [API reference](https://example.com/docs/api): every endpoint

## Optional

- [Changelog](https://example.com/changelog): release notes

We report whether the address answers with a Markdown file and not a web page, whether it starts with a title, how many sections and links it has, and any list item that is a bare address instead of a link. We also note whether llms-full.txt, the long version, exists. We read the file only; we do not open the links inside it.

What else this page shows

The same run covers the rest of the AI readiness checklist, so you can see llms.txt in context. Open the other tabs for the full list of checks, the crawler matrix and the first text a crawler reads. The pieces that matter more than llms.txt for most sites are an accessible robots.txt, content in the first response and a sitemap.

Writing a useful one

Keep it short. Link to the pages a newcomer needs first, not to everything.
Write each note for a reader who has not seen your site, in plain words.
Link to Markdown pages where you have them. They cost a reader far fewer tokens than HTML.
Keep it current. A file that points to removed pages is worse than none.

Questions people ask

Do I need an llms.txt file?

No. It is optional. Google has said it does not use the file for Search, and no search engine requires it. It can help coding agents and some assistants find your documentation, so it is worth having if you publish docs for developers.

Does Google use llms.txt?

Google has said it does not use llms.txt for Search. Other tools and assistants may read it, but nobody has shown that it changes what they cite, so treat it as a convenience for readers and not as an optimisation.

What is the difference between llms.txt and llms-full.txt?

llms.txt is a short index with links. llms-full.txt, when a site has one, holds the full text of those documents in one file so an agent can read it in a single request. This checker tells you whether the long file exists.

Why does it say my file has a bare address?

In a section, each list item should be a Markdown link with a title, then a colon and a short note. An item that is only an address works for a person but gives a reader no title. Free-form text before the first section is fine.

Why is a missing llms.txt not a failure here?

Because the file is optional and unproven for search. Marking it as a failure would imply a benefit nobody can show. A missing file is reported as information and the level is unaffected.

Use it from AI, the API or the command line

Every LookupOwl tool is also a free API and an MCP tool, with no key and no account. This one is ai_readiness_check. Please be kind to the rate limits.

curl

curl "https://lookupowl.com/api/v1/tools/ai-readiness-check?q=lookupowl.com"

MCP

claude mcp add --transport http lookupowl https://lookupowl.com/mcp

Description for AI agents · OpenAPI · llms.txt

Domain

DNS

Email

Network

IP and subnets

Web

Security

Make and convert

llms.txt checker

Results for lookupowl.com

Ready

Next step

What llms.txt is, and what it is not

The format we check against

What else this page shows

Writing a useful one

Questions people ask

Use it from AI, the API or the command line

Ready

Next step

What llms.txt is, and what it is not

The format we check against

What else this page shows

Writing a useful one

Questions people ask

Use it from AI, the API or the command line

You might also need