Automation, agents & web scraping

firecrawl/firecrawl

API to search and scrape the web at scale

169k
AGPL-3.0
TypeScript

What it is and why it matters

A context API to search, scrape and interact with the web at scale, built to feed data to agents. It started as a page fetcher handling proxying and has grown with search, whole-site crawling and periodic monitoring. It is a paid (open-core) service with a free tier to try.

Key points

Scrapes single pages or crawls a whole site following links
Periodic monitoring with email or webhook alerts
Offloads work to the cloud to avoid local blocks and rate limits

Where we saw it

The Next New Thing 2026-07-24 The guest describes it as one of the better systems for crawling web content: it began fetching pages and now adds search and a monitoring feature that checks sites every few hours. Advantage over doing it locally: running in the cloud avoids blocks and rate limits. Free ai video creation + OCR full books + private photos + more Github finds

Same category

n8n-io/n8n Fair-code workflow automation with native AI Open detail browser-use/browser-use Let AI agents drive the browser Open detail gitroomhq/postiz-app Open-source social scheduling built for agents Open detail