Awesome Data Analysis
Section: Tools · An open-source and collaborative web crawling framework for Python.
FreshScore 80
Entry
Scrapy
Appears in 5 awesome lists
is a fast high-level web crawling and web scraping framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing.
Found in these lists
Awesome ETL
Section: Libraries · "a fast high-level web crawling & scraping framework for Python."
ActiveScore 72
Awesome Python Data Science
Section: Web Scraping · Fast and extensible scraping library. Can write rules and create customized scraper without touching the core
ActiveScore 71
Awesome list of FREE tools Startups should be using
Section: Web Scraping Tools
FreshScore 78
Table of Contents
Section: Python Frameworks, Libraries, and Tools · is a fast high-level web crawling and web scraping framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing.
SlowScore 51
Often listed next to it
Dask
| Python | - Parallel computing with task scheduling in Python with a Pandas like API
In 11 listsDetails
Polars
| Rust, Python | - Polars is a blazingly fast DataFrames library implemented in Rust using Apache Arrow Columnar Format as memory model.
In 10 listsDetails
browser-use
Minimal browser-automation agent harness with clean separation of tool registration, DOM state injection, action loop, and error recovery. Small codebase, clear structure — the best "minimal viable harness" reference for understanding core loop mechanics.
In 10 listsDetails
Hamilton
| Python | - A scalable general purpose micro-framework for defining dataflows. You can use it to build dataframes, numpy matrices, python objects, ML models, etc. Embed Hamilton anywhere python runs, e.g. spark, airflow, jupyter, fastapi, python scripts, etc.
In 10 listsDetails
Selenium
Portable software testing framework for web applications. License: Apache 2 , .
In 9 listsDetails
DuckDB
A fast in-process analytical database that has zero external dependencies, runs on Linux/macOS/Windows, offers a rich SQL dialect, and is free and extensible.
In 8 listsDetails
dbt-core
Project banner, super clear description (friendly to people brand new to the product), screenshot of the docs the tool generates, and concise links to other comprehensive pages explaining Getting Started, Joining the dbt Community, Bug Reports, Code Contribution, and Code of Conduct
In 7 listsDetails
playwright-python
Python version of the Playwright testing and automation library.
In 6 listsDetails