Skip to content

Entry

Crawl4AI

Appears in 5 awesome lists

Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key. Join our Discord https://discord.gg/VuxzM8mtf

Open github.comunclecode/crawl4ai

Found in these lists

Awesome Data Analysis

Section: Tools · Advanced web crawling framework designed for AI and data extraction tasks.

FreshScore 79

Awesome Generative AI

Section: Everything to Markdown to LLMs · Open-source LLM Friendly Web Crawler & Scraper

SlowScore 58

Awesome Open Source AI

Section: 5. Retrieval-Augmented Generation (RAG) & Knowledge · LLM-friendly web crawler that turns websites into clean Markdown for RAG and agentic workflows.

FreshScore 87

awesome-python

Section: Other · Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key. Join our Discord https://discord.gg/VuxzM8mtf

FreshScore 79

Awesome Python

Section: Web Scraping · An open-source, LLM-friendly web crawler that provides lightning-fast, structured data extraction specifically designed for AI agents.

FreshScore 100

LlamaIndex

(MIT) provides modules for structured outputs at different levels of abstraction, including output parsers for text completion endpoints, Pydantic programs for mapping prompts to structured outputs using function calling or output parsing, and pre-defined Pydantic programs for specific output types.

In 14 listsDetails

Milvus

Milvus is a cloud-native, open-source vector database built to manage embedding vectors generated by machine learning models and neural networks.

In 14 listsDetails

Haystack

Open-source AI orchestration framework for building context-engineered, production-ready LLM applications. Design modular pipelines and agent workflows with explicit control over retrieval, routing, memory, and generation. Built for scalable agents, RAG, multimodal applications, semantic search,…

In 13 listsDetails

browser-use

Open-source browser agent (Python and TypeScript) whose loop injects serialized DOM state each step, exposes browser actions as registered tools, and runs the same code against a local Chrome or a cloud browser. It has grown from a minimal reference into a full framework (~70k lines of Python…

In 11 listsDetails

FAISS

is a library for efficient similarity search and clustering of dense vectors. It contains algorithms that search in sets of vectors of any size, up to ones that possibly do not fit in RAM. It also contains supporting code for evaluation and parameter tuning. Faiss is written in C++ with complete…

In 9 listsDetails

MarkItDown

Python tool for converting files and office documents to Markdown. Supports PDF, PowerPoint, Word, Excel, images, audio, HTML, and more with OCR and transcription capabilities. MIT licensed.

In 9 listsDetails

Selenium

Portable software testing framework for web applications. License: Apache 2 , .

In 9 listsDetails

DocsGPT

Private AI platform for building intelligent agents and assistants with enterprise search. Features Agent Builder, deep research tools, multi-format document analysis, and multi-model support. MIT licensed.

In 10 listsDetails