Crawl4AI

Open-source LLM-friendly web crawler that outputs clean Markdown and JSON

Hand-reviewed by the Toolglade team·Updated Aug 2026

web-scraping#open-source#llm-ready#web-crawler#python

Free plan Claimed API Self-hosted

Toolglade’s take

Crawl4AI has become a go-to open-source crawler for feeding LLMs, and its Markdown/JSON output is exactly what RAG pipelines need. The Playwright foundation plus deep-crawl strategies make it capable, but it is a developer tool that you self-host and operate, not a hosted service with support. Proxies and scaling are your responsibility. Best for engineers building AI agents and RAG systems who want control and no per-request fees.

About Crawl4AI

Crawl4AI is a free, open-source Python web crawler that outputs LLM-ready Markdown and structured JSON, with deep crawling, CSS/XPath/LLM extraction, proxy rotation, and Docker deployment.

Crawl4AI is designed for developers building RAG pipelines and AI agents that need web content in a format LLMs can consume directly. It automatically converts HTML to Markdown, extracts structured data via CSS/XPath or LLM-based strategies, and handles dynamic pages through Playwright with proxy and stealth controls. Recent versions added deep crawling (BFS/DFS/BestFirst strategies), a memory-adaptive dispatcher, multiple crawling strategies (Playwright and HTTP), a CLI, browser profiling, and PDF processing. Apache 2.0 licensed and self-hostable via Docker with a FastAPI service, Crawl4AI is a free, developer-first alternative to hosted scraping APIs, with 70,000+ GitHub stars reflecting strong adoption.

TL;DR

Crawl4AI is a free, open-source Python web crawler that outputs LLM-ready Markdown and JSON for RAG and AI agents, self-hosted via library or Docker.

Company overview

Crawl4AI is a community open-source project (led by developer unclecode) focused on making web content consumable by LLMs. With 70,000+ GitHub stars, it has become a widely-used building block for AI data pipelines.

Rather than a commercial service, it is developer infrastructure distributed under Apache 2.0, maintained by an active open-source community.

Product features

Crawl4AI crawls pages with Playwright and asyncio, converting HTML to Markdown and extracting structured JSON via CSS/XPath or LLM strategies. It supports deep crawling, a memory-adaptive dispatcher, proxy rotation, and PDF processing.

A CLI, browser profiler, and Docker/FastAPI deployment make it flexible for both scripts and self-hosted services, targeting engineers who need clean, LLM-ready web data at scale.

Target market

Crawl4AI targets AI engineers, data teams, and agent builders who need programmable, self-hosted web crawling with LLM-ready output and no per-request pricing.