📰 Fetch daily trending repositories information on GitHub Trending Page by script writen in JavaScript and executed with GitHub Actions Service.

Weather Data Scraper

基于python的网页自动化工具。既能控制浏览器，也能收发数据包。可兼顾浏览器自动化的便利性和requests的高效率。功能强大，内置无数人性化设计和便捷功能。语法简洁而优雅，代码量少。

A list of AI agents and robots to block.

🤖/👨‍🦰 Detect bots/crawlers/spiders using the user agent string

Kennedy: Crawler and Search Engine for Gemini space. Leverages techniques and architecture from early WWW crawlers like Mercator, Archive.org, and GoogleBot

Norconex Crawlers (or spiders) are flexible web and filesystem crawlers for collecting, parsing, and manipulating data from the web or filesystem to various data repositories such as search engines.

A bot to help people with their rental real-estate search. 🏠🤖

Artificial General Intelligence Infrastructure of "The Sacred Computer" AGI Institute : Custom Intelligent Selective Internet Archiving and Exploration/Crawling; Information Retrieval, Media Monitoring, Search Engine, Smart DB, Data Preservation, Knowledge Extraction,Datasets creation,AI Generative models building and testing,Experiments etc.

Crawler program for popular Chinese social media Sina Weibo (mobile site). It is often used to build unstructured and image datasets.

Open source SEO auditing tool.

Raven is a powerful and customizable web crawler written in Go.

I describe myself as a passionate developer who loves coding, open-source, and the web platform

dysdera web crawler

Desktop app that crawls urls from Google's search engine results

Scrapy dappy doo crawler for proxy sites

Simple robots.txt template. Keep unwanted robots out (disallow). White lists (allow) legitimate user-agents. Useful for all websites.

A highly performant and versatile crawling engine, designed with scalability and extensibility in mind.

Wget-AT is a modern Wget with Lua hooks, Zstandard (+dictionary) WARC compression and URL-agnostic deduplication.