Why I Built astro-agent-ready: Generating llms.txt Without the HTML-to-Markdown Roundtrip

Autonomous coding agents (Google Antigravity, Cursor, Claude) and AI search engines (Perplexity, ChatGPT Search) are quickly becoming the primary consumers of technical blogs and documentation.

Yet when an AI agent fetches a page on a modern Astro site, over 80% of its context window is wasted on layout shells, navigation sidebars, SVG icons, cookie banners, and syntax-highlighted <span> tags.

The llmstxt.orgspecification solves this by serving a clean /llms.txt index and a /llms-full.txt knowledge base at the root of your site. When I went looking for an Astro plugin to generate these files forezinner.com, I hated how existing packages were built—so I wrote my own and published it as astro-agent-ready.

Almost every existing llms.txt generator for Astro makes the exact same architectural blunder: the HTML-to-Markdown roundtrip.

Here is what they do during a build:

1. You write clean Markdown or MDX in your repository.
2. Astro compiles your Markdown into full HTML pages with layout wrappers and Shiki syntax spans.
3. The plugin pulls in a dozen heavy dependencies (jsdom,unified, rehype, remark-stringify) to parse the rendered HTML back into Markdown.

"Converting Markdown to HTML only to scrape it back into Markdown is completely backwards. It slows down builds, mangles complex tables, corrupts code block escaping, and drags megabytes of transitive dependencies into node_modules just to emit plain text."

Just like I did with astro-strict-csp, I built astro-agent-ready with zero runtime dependencies ("dependencies": ), relying purely on native node:fs and node:path APIs.

Instead of scraping compiled HTML, it goes straight to your source files:

• Zero Conversion Loss: It reads .md,.mdx, and .astro files directly from your source tree, strips YAML frontmatter and <*Layout> wrappers, filters out unpublished drafts (draft: true), and emits your exact original Markdown. Code fences and tables remain 100% untouched.
• Universal Content Discovery: Whether you store posts in standard Astro Content Collections (src/content/**) or standalone.astro page directories (extracting export const title, descriptions, and tags via lightweight AST regex parsing), it discovers and categorizes them automatically.
• Dual Spec-Compliant Output: It generates both/llms.txt (a compact, categorized map with pinned core pages like About, CV, or Scrap) and /llms-full.txt (your entire site's content concatenated into a single file for deep reasoning ingestion).

When you are feeding documentation or articles into an LLM context window, file size in kilobytes matters much less than token count.

During astro build, astro-agent-ready runs an internal subword token estimator across your content and prints the exact context window cost right in the terminal:

[astro-agent-ready] Generated /llms.txt with 24 documents (2.2k tokens, 10.1 kB)
[astro-agent-ready] Generated /llms-full.txt full context (54.1k tokens, 256.5 kB)

Running it across ezinner.com made the architectural difference obvious immediately:

An agent can now ingest every technical article I have ever written in a single HTTP request while consuming less than 3% of a modern 2M-token context window.

I deliberately named the package astro-agent-ready rather than locking the name to astro-llms-txt because static Markdown dumps are only the first step in making a website first-class infrastructure for autonomous agents.

Instead of forcing developers to install three separate integrations as agent standards evolve, my roadmap for this zero-dependency package includes:

• JSON-LD Agent Cards (/.well-known/agent.json):Generating structured capability discovery manifests for agent-to-agent (A2A) routing.
• WebMCP Integration: Helping Astro sites expose interactive client-side tools and forms directly to WebMCP-compatible browser agents.

If you run an Astro blog or documentation site and want to give AI crawlers and coding agents a clean feed without bloating your package.json:

# Install the zero-dependency integration
npm install astro-agent-ready

Add it to your astro.config.mjs:

// astro.config.mjs
import { defineConfig } from 'astro/config';
import agentReady from 'astro-agent-ready';

export default defineConfig({
  site: 'https://example.com',
  integrations: [
    agentReady({
      title: 'My Engineering Site',
      description: 'Technical notes, systems architecture, and distributed systems.',
      generateFull: true,
      sources: [
        {
          name: 'articles',
          heading: 'Technical Articles',
          dir: 'src/pages/articles',
          urlPrefix: '/articles',
          sortBy: 'date-desc',
        },
      ],
    }),
  ],
});

If you run into a custom Astro folder structure or have ideas for upcoming WebMCP and Agent Card features, feel free to open an issue or pull request onGitHub.