A fast high-level screen scraping and web crawling framework.

Creating Scrapy scrapers via the Django admin interface.

Redis-based components for Scrapy.

Uses Redis and Kafka to create a distributed on demand scraping cluster.

Uses scrapy,redis, mongodb,graphite to create a distributed spider.

A powerful spider system.

A versatile web crawler built using modern tools and concurrency.

A distributed crawling framework.

PyQuery-based scraping micro-framework.

A pure-python HTML screen-scraping library.

Universal feed parser.

Dumb downloader that scrapes the web.

A Python library for automating interaction with websites.

Visual scraping for Scrapy.

Pythonic Crawling / Scraping Framework based on Non Blocking I/O operations.

A simple, Pythonic library for browsing the web without a standalone web browser.

A simple ,easy spider using gevent and js render.

A lightweight web data extracting framework.

A simple spider frame in Python3.

Web crawling framework based on asyncio for everyone.

Minimalist and powerful Web Crawler.

The simple, easy to use command line web crawler.

News, full-text, and article metadata extraction in Python 3

An async web scraping micro-framework based on asyncio.

An easy to use web crawler for domain-specific search.

Highly extensible, highly scalable web crawler for production environment.

A plugin for Apache Nutch to crawl semantic annotations within HTML pages.

Simple and lightweight web crawler.

Scrapes, parses, manipulates and cleans HTML.

Website-Specific Processors for HTML information extraction.

A full set of search functions. Build your own indexing strategy. Parsers extract full-text data. The crawlers can…

A easy to use lightweight web crawler

Simple interfaces for crawling the Web,you can setup a multi-threaded web crawler in less than 5 minutes.

A scalable crawler framework.

A scalable ,extensible, multi-threaded web crawler.

A distributed web crawler framework,support js render.

Extensible, web-scale, archival-quality web crawler project.

An agile, distributed crawler framework.

An open source collection of resources for building low-latency, scalable web crawlers on Apache Storm

Evolving Apache Nutch to run on Spark.

A DFS web spider.

A visual spider framework, it's so good that you don't need to write any code to crawl the website.

Norconex HTTP Collector is a full-featured web crawler (or spider) that can manipulate and store collected data into a…

Built in C# 3.5 version. it contains a simple extension of web content categorizer, which can separate between the web…

Simple spider base on mutithreading, regluar expression.

This is a cross platfrom, ligth spider develop by C#.

C# web crawler built for speed and flexibility.

Advanced Crawler and ETL tool written in C#/WPF.

An asynchronous web scraper / web crawler using async / await and Reactive Extensions.

A simple but powerful web crawler library in C#.

A complete and versatile web scraper.

A Node.js scraper for humans.

Event driven web crawler.

Node-crawler has clean,simple api.

Web crawler for Node.JS, both HTTP and HTTPS are supported.

A reliable web crawling framework which can scrape ajax and js rendered content in a web page.

Web scraper with pagination and crawler support.

HTML/XML parser and web scraper for Node.js.

Web data extraction tool implemented as chrome extension.

Define custom handlers to parse content. Obeys robots.txt, rate limits and concurrency limits.

Headless Chrome crawls with jQuery support

High fidelity, user scriptable, archival crawler that uses Chrome or Chromium with or without a head

A web scraping and browser automation library for Node.js that helps you build reliable crawlers. Fast.

A screen scraping and web crawling library for PHP.

Laravel 5 Facade for Goutte.

The DomCrawler component eases DOM navigation for HTML and XML documents.

The progressive PHP crawler framework.

Parallel web crawler written in PHP.

A configurable and extensible PHP web spider.

An easy to use, powerful crawler implemented in PHP. Can execute Javascript.

Crawlzone is a fast asynchronous internet crawling framework for PHP.

PHPScraper is a scraper & crawler built for simplicity.

A distributed open source search engine and spider/crawler written in C/C++.

Copy websites to your computer.

A Rubygem providing HTML, XML, SAX, and Reader parsers with XPath and CSS selector support.

A batteries-included framework for easy web-scraping. Just add CSS(Or do more).

Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages.

RubyRetriever is a Web Crawler, Scraper & File Harvester.

Spider a site, multiple domains, certain links or infinitely.

Web crawler with very flexible crawling options, standalone or using sidekiq.

Automated web interaction & crawling.

The fastest web crawler and indexer.

A gRPC web indexer turbo charged for performance.

Simple web scraping for R.

A scalable, distribuited and highly configurable web cawler.

Web Scraping Toolkit using HTML and CSS Selectors or XPath expressions.

A distributed, high concurrency and powerful web crawler.

Polite, slim and concurrent web crawler.

A simple and flexible web crawler that follows the robots.txt policies and crawl delays.

An awesome Go concurrent Crawler(spider) framework.

BitTorrent DHT Protocol && DHT Spider.

A open source, distributed, restful crawler engine in golang.

A simple, higher level interface for Go web scraping.

The Next Generation Crawler Framework (Go).

Fast and Elegant Scraping Framework for Gophers.

Declarative web scraping.

Extract structured data from web pages. Web sites scraping.

Simple, fast web crawler designed for easy, quick discovery of endpoints and assets within a web application

Scala DSL for web crawling.

Scala crawler(spider) framework, inspired by scrapy.

Ferrit is a web crawler service written in Scala using Akka, Spray and Cassandra.