From a017fa3592098287432c57db3ad98e8a5b3646d4 Mon Sep 17 00:00:00 2001 From: Vinta Chen Date: Fri, 2 Oct 2026 13:52:48 +0800 Subject: [PATCH] docs: rewrite Web Scraping category intro The old intro recommended html2text, which the 2026-10-02 audit removed from the README section. Co-Authored-By: Claude --- website/data/category_intros/web-scraping.md | 36 +++++++++++--------- 1 file changed, 20 insertions(+), 16 deletions(-) diff --git a/website/data/category_intros/web-scraping.md b/website/data/category_intros/web-scraping.md index 0409e072..e1996c0a 100644 --- a/website/data/category_intros/web-scraping.md +++ b/website/data/category_intros/web-scraping.md @@ -1,28 +1,32 @@ -To crawl a whole site, use Scrapy as your Python web scraping library; to feed pages to an LLM, Crawl4AI; to pull an article's main text, Trafilatura. +Most crawls you build with a Python web scraping library fit Scrapy: you write selectors, it schedules requests. Crawl4AI hands pages to an LLM as Markdown. How to choose: -- Crawling a whole site, or many sites: Scrapy -- Web pages as clean Markdown for an LLM: Crawl4AI -- An article's main text and metadata: Trafilatura -- An AI agent that does a task in the browser for you: Browser Use -- Browser automation in your own code, with plain-language steps: Stagehand -- A browser agent that picks each action from a table of page elements: jev-ultrafast +- Crawling a site with selectors you write: Scrapy +- Pages an LLM will read, as Markdown: Crawl4AI +- A goal in plain words that an LLM carries out in a browser: Browser Use +- A browser script with plain-word steps where selectors would break: Stagehand +- A browser agent where TypeSafe's Jev picks each action: jev-ultrafast +- A whole HTML page or fragment as Markdown: markdownify - RSS, Atom, and JSON feeds: feedparser -- A whole HTML page as Markdown text: html2text +- An article's main text, without menus and ads: Trafilatura -Scrapy is a framework [for crawling websites and extracting structured data](https://docs.scrapy.org/en/latest/intro/overview.html) from them, and it sends requests asynchronously. Create a project with `scrapy startproject` and [write your spiders in it](https://docs.scrapy.org/en/latest/intro/tutorial.html). When a page loads its data with JavaScript, [find the request that returns the data](https://docs.scrapy.org/en/latest/topics/dynamic-content.html) and send it yourself. Use a headless browser only when that fails. To run your spiders in production, deploy them to [Scrapyd or Zyte Scrapy Cloud](https://docs.scrapy.org/en/latest/topics/deploy.html). +Scrapy is [an application framework for crawling web sites and extracting structured data](https://docs.scrapy.org/en/latest/intro/overview.html). It schedules and processes requests asynchronously, so one slow page doesn't hold up the crawl. Start with [`scrapy startproject`](https://docs.scrapy.org/en/latest/intro/tutorial.html#creating-a-project), write a spider class whose callbacks yield items, and send those items through [item pipelines](https://docs.scrapy.org/en/latest/topics/item-pipeline.html) that clean, validate, deduplicate, or store them. -Crawl4AI [turns websites into clean, LLM-ready Markdown](https://docs.crawl4ai.com/). It runs Chromium headless by default, so run `crawl4ai-setup` once to install the browser. Open an `AsyncWebCrawler` in an `async with` block and [call `arun()` for each URL](https://docs.crawl4ai.com/core/quickstart/). When pages share one layout, extract the data with CSS or XPath schemas: they [do exactly what you specify](https://docs.crawl4ai.com/extraction/no-llm-strategies/), while an LLM's output can vary. Keep [LLM extraction](https://docs.crawl4ai.com/extraction/llm-strategies/), which is slower and costlier, for content an AI has to interpret. +Crawl4AI [turns any website into clean, LLM-ready Markdown](https://docs.crawl4ai.com/) for RAG, AI agents, and data pipelines. Its `AsyncWebCrawler` [launches a headless browser](https://docs.crawl4ai.com/core/quickstart/) and converts each page's HTML to Markdown. Give `arun()` one URL, or give [`arun_many()`](https://docs.crawl4ai.com/advanced/multi-url-crawling/) a list, which it crawls with concurrency control. -Trafilatura [extracts a page's main text and metadata](https://trafilatura.readthedocs.io/en/latest/) and skips boilerplate like headers and footers. Call `extract()`, and pass `favor_precision=True` or `favor_recall=True` to [tune what it keeps](https://trafilatura.readthedocs.io/en/latest/usage-python.html). It works on raw HTML, so [render JavaScript pages first](https://trafilatura.readthedocs.io/en/latest/troubleshooting.html) with a browser and pass the result to `extract()`. It pairs with Scrapy: [Scrapy crawls, Trafilatura extracts](https://trafilatura.readthedocs.io/en/latest/faq.html#how-does-trafilatura-compare-to-beautifulsoup-or-scrapy). +Browser Use [runs its agent locally from Python](https://github.com/browser-use/browser-use#user-content-path-3-python-library), with your choice of model and a local or cloud browser. Create an `Agent` with a task and an LLM, then `await agent.run()`. -Browser Use is an AI browser agent: you [give it a task and an LLM](https://docs.browser-use.com/open-source/quickstart), and it runs the task in a browser. Its docs say to [be specific about the actions](https://docs.browser-use.com/open-source/customize/agent/prompting-guide) you want. Pass a Pydantic model as `output_model_schema` to get structured results. Don't take the agent's word for it: `is_successful()` is [only its own assessment](https://docs.browser-use.com/open-source/customize/agent/output-format), so check important results yourself, like whether a form really got submitted. Pass logins as `sensitive_data`, so the model [sees only placeholders](https://docs.browser-use.com/open-source/examples/templates/sensitive-data) in the text it reads. Set `use_vision=False` too, or the real values can leak through screenshots. +Stagehand lets you [mix AI steps with deterministic browser control](https://docs.stagehand.dev/v4/first-steps/introduction) in one script and decide how much AI each step uses. Call `page` methods like `goto` and `click` when you know the selector, and the self-healing `act()`, `extract()`, and `observe()` where a selector would break when the site changes. -Stagehand leaves the steps to your code: you [mix plain-language actions with regular page calls](https://docs.stagehand.dev/first-steps/introduction) in one script, and decide how much AI each step uses. Give each `act()` call [one focused action](https://docs.stagehand.dev/best-practices/prompting-best-practices), and pass credentials as variables, so their values never reach the model. To get data out, pass `extract()` a Pydantic model, and Stagehand [validates the result against it](https://docs.stagehand.dev/basics/extract). +jev-ultrafast [takes one goal](https://github.com/browser-use/jev-ultrafast): TypeSafe's Jev picks an operation and a page element, and a small LLM writes text only when the operation is typing. Its default loop reads structured page state, not screenshots. Running it takes a [TypeSafe API key](https://github.com/browser-use/jev-ultrafast#user-content-try-it) along with the text model's. -feedparser parses RSS, Atom, and JSON feeds with [one function, `parse()`](https://feedparser.readthedocs.io/en/latest/introduction/), which takes a URL, a file, or a string. When you poll a feed, send back the [ETag and Last-Modified values](https://feedparser.readthedocs.io/en/latest/http-etag/) from the last response. Otherwise you download unchanged feeds again, and the publisher may ban you. Content feedparser marks as `text/plain` [hasn't been sanitized](https://feedparser.readthedocs.io/en/latest/html-sanitization/), so escape it before you render it. +markdownify [converts HTML to Markdown](https://github.com/matthewwithanm/python-markdownify#user-content-usage): call `markdownify()` on an HTML string. It converts all the HTML you give it, so for an article's main text, use Trafilatura. -html2text [converts a page of HTML into Markdown](https://github.com/Alir3z4/html2text), with options such as [`ignore_links`](https://github.com/Alir3z4/html2text/blob/master/docs/usage.md). It's GPL, while Trafilatura is [Apache](https://trafilatura.readthedocs.io/en/latest/). +feedparser has [one main function, `parse()`](https://feedparser.readthedocs.io/en/latest/introduction/), which takes a URL, a local file, or a string and reads RSS, Atom, and JSON feeds. When you poll a feed, [pass its last ETag and Last-Modified values back](https://feedparser.readthedocs.io/en/latest/http-etag/): without them you download unchanged feeds again, and the publisher may ban you. -Whatever you pick, tell sites who you are and go easy on them. Scrapy's docs say to [set `USER_AGENT` to a value that identifies you](https://docs.scrapy.org/en/latest/topics/practices.html#avoiding-getting-banned) and space out your requests. The feedparser docs say to [set the User-Agent to your app's name and URL](https://feedparser.readthedocs.io/en/latest/http-useragent/), and Trafilatura's to [throttle per domain and follow robots.txt](https://trafilatura.readthedocs.io/en/latest/downloads.html). +Trafilatura [returns a page's main text](https://trafilatura.readthedocs.io/en/latest/faq.html#how-does-trafilatura-compare-to-beautifulsoup-or-scrapy) without the navigation, ads, and boilerplate, with no selectors to write. Its FAQ pairs it with Scrapy: Scrapy crawls, Trafilatura extracts. [`extract()` is the default choice](https://trafilatura.readthedocs.io/en/latest/faq.html#which-extraction-function-should-i-use) among its functions. + +Let sites know who's crawling. Scrapy's tutorial says to [set `USER_AGENT`](https://docs.scrapy.org/en/latest/intro/tutorial.html#creating-a-project) to a project name plus a URL or an email address, so site owners can ask you to adjust your crawler instead of blocking it. feedparser's docs also say to [set the User-Agent](https://feedparser.readthedocs.io/en/latest/http-useragent/) to your application's name and URL. + +With a browser agent, keep passwords away from the model: Browser Use [shows the LLM only placeholders](https://docs.browser-use.com/open-source/examples/templates/sensitive-data), and Stagehand [fills in variables after the model picks the action](https://docs.stagehand.dev/v4/best-practices/prompting-best-practices#protect-sensitive-data). Then check the agent's work: Browser Use's docs say to [verify important outcomes against the destination system](https://docs.browser-use.com/open-source/customize/agent/output-format), and jev-ultrafast's README says a `DONE` [still needs independent verification](https://github.com/browser-use/jev-ultrafast#user-content-evidence-and-limits).