Skip to content

Entry

scrapy

Appears in 6 awesome lists

Fast, high-level web crawling and scraping framework for Python. Extract structured data from websites at scale with built-in support for handling common challenges like pagination, cookies, and concurrent requests. BSD-3-Clause licensed.

Open github.comscrapy/scrapy

Found in these lists

Awesome-crawler

Section: Python · A fast high-level screen scraping and web crawling framework.

StaleScore 52

Awesome First Pull Request Opportunities

Section: Python · (label: good first issue) A fast high-level web crawling & scraping framework for Python.

FreshScore 91

Awesome Open Source AI

Section: 1. Core Frameworks & Libraries · Fast, high-level web crawling and scraping framework for Python. Extract structured data from websites at scale with built-in support for handling common challenges like pagination, cookies, and concurrent requests. BSD-3-Clause licensed.

FreshScore 87

awesome-python

Section: Scraping and Crawling · Scrapy, a fast high-level web crawling & scraping framework for Python.

FreshScore 79

开发实战资源整合

Section: Web 后端 · 一个全能型的爬虫框架

StaleScore 46

Awesome Python

Section: Web Scraping · A fast high-level web crawling and scraping framework.

FreshScore 100

TensorFlow

How to use the Hexagon Delegate to speed up model inference on mobile and edge devices. Also see blog post Accelerating TensorFlow Lite on Qualcomm Hexagon DSPs.

In 23 listsDetails

PyTorch

(label: good first issue) PyTorch is an open source machine learning library based on the Torch library, used for applications such as computer vision and natural language processing.

In 16 listsDetails

Luigi

Python module for building complex pipelines of batch jobs. Handles dependency resolution, workflow management, visualization, and Hadoop integration. Built at Spotify and battle-tested in production. Apache 2.0 licensed.

In 15 listsDetails

transformers

(formerly known as pytorch-transformers and pytorch-pretrained-bert) provides state-of-the-art general-purpose architectures (BERT, GPT-2, RoBERTa, XLM, DistilBert, XLNet, CTRL...) for Natural Language Understanding (NLU) and Natural Language Generation (NLG) with over 32+ pretrained models in…

In 14 listsDetails

mem0

The most-deployed memory layer for AI agents — drop-in add/search/update/delete fact storage with LLM-extracted long-term / short-term / semantic memory: automatic fact consolidation (dedup, contradiction resolution, decay) so agents keep "context that persists" without context bloat;…

In 14 listsDetails

Colossal-AI

(from Hpcaitech) - A Unified Deep Learning System for Large-Scale Parallel Training (1D, 2D, 2.5D, 3D and sequence parallelism, and ZeRO protocol).

In 14 listsDetails

Apache Airflow

"Use airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The airflow scheduler executes your tasks on an array of workers while following the specified dependencies. Rich command line utilities make performing complex surgeries on DAGs a snap. The rich user interface makes it…

In 13 listsDetails

mitmproxy

Inspect and record HTTP traffic with an interactive proxy, web interface, or scriptable command-line tools. Tool · Also: App, Protocol · Full entry

In 12 listsDetails