diff --git a/website/data/category_intros/web-scraping.md b/website/data/category_intros/web-scraping.md index e1996c0a..af5c73d5 100644 --- a/website/data/category_intros/web-scraping.md +++ b/website/data/category_intros/web-scraping.md @@ -6,7 +6,7 @@ How to choose: - Pages an LLM will read, as Markdown: Crawl4AI - A goal in plain words that an LLM carries out in a browser: Browser Use - A browser script with plain-word steps where selectors would break: Stagehand -- A browser agent where TypeSafe's Jev picks each action: jev-ultrafast +- Selectors that survive a site redesign, without AI: Scrapling - A whole HTML page or fragment as Markdown: markdownify - RSS, Atom, and JSON feeds: feedparser - An article's main text, without menus and ads: Trafilatura @@ -19,7 +19,7 @@ Browser Use [runs its agent locally from Python](https://github.com/browser-use/ Stagehand lets you [mix AI steps with deterministic browser control](https://docs.stagehand.dev/v4/first-steps/introduction) in one script and decide how much AI each step uses. Call `page` methods like `goto` and `click` when you know the selector, and the self-healing `act()`, `extract()`, and `observe()` where a selector would break when the site changes. -jev-ultrafast [takes one goal](https://github.com/browser-use/jev-ultrafast): TypeSafe's Jev picks an operation and a page element, and a small LLM writes text only when the operation is typing. Its default loop reads structured page state, not screenshots. Running it takes a [TypeSafe API key](https://github.com/browser-use/jev-ultrafast#user-content-try-it) along with the text model's. +Scrapling is [an adaptive web scraping framework that handles everything from a single request to a full-scale crawl](https://scrapling.readthedocs.io/en/latest/). Turn on `adaptive` in its fetcher and select elements with `auto_save=True`. When a redesign breaks a selector, run it again with `adaptive=True`, and Scrapling [finds the element most similar to the one it saved](https://scrapling.readthedocs.io/en/latest/parsing/adaptive.html), without AI. [Use `Fetcher`](https://scrapling.readthedocs.io/en/latest/fetching/choosing.html#fetchers-overview) when HTTP requests alone can get the page, and a browser fetcher for dynamically loaded sites. For a crawl, [write a `Spider`](https://scrapling.readthedocs.io/en/latest/spiders/getting-started.html#your-first-spider) with `start_urls` and an async `parse()` that yields items, a pattern familiar from Scrapy. markdownify [converts HTML to Markdown](https://github.com/matthewwithanm/python-markdownify#user-content-usage): call `markdownify()` on an HTML string. It converts all the HTML you give it, so for an article's main text, use Trafilatura. @@ -29,4 +29,4 @@ Trafilatura [returns a page's main text](https://trafilatura.readthedocs.io/en/l Let sites know who's crawling. Scrapy's tutorial says to [set `USER_AGENT`](https://docs.scrapy.org/en/latest/intro/tutorial.html#creating-a-project) to a project name plus a URL or an email address, so site owners can ask you to adjust your crawler instead of blocking it. feedparser's docs also say to [set the User-Agent](https://feedparser.readthedocs.io/en/latest/http-useragent/) to your application's name and URL. -With a browser agent, keep passwords away from the model: Browser Use [shows the LLM only placeholders](https://docs.browser-use.com/open-source/examples/templates/sensitive-data), and Stagehand [fills in variables after the model picks the action](https://docs.stagehand.dev/v4/best-practices/prompting-best-practices#protect-sensitive-data). Then check the agent's work: Browser Use's docs say to [verify important outcomes against the destination system](https://docs.browser-use.com/open-source/customize/agent/output-format), and jev-ultrafast's README says a `DONE` [still needs independent verification](https://github.com/browser-use/jev-ultrafast#user-content-evidence-and-limits). +With a browser agent, keep passwords away from the model: Browser Use [shows the LLM only placeholders](https://docs.browser-use.com/open-source/examples/templates/sensitive-data), and Stagehand [fills in variables after the model picks the action](https://docs.stagehand.dev/v4/best-practices/prompting-best-practices#protect-sensitive-data). Then check the agent's work: Browser Use's docs say to [verify important outcomes against the destination system](https://docs.browser-use.com/open-source/customize/agent/output-format).