How do I add a hosted fallback to Crawl4AI?
Pass an UpRock fetch as Crawl4AI's fallback_fetch_function. When Crawl4AI's own attempts end blocked, it calls your function, which loads the page on a real device in the country you choose and hands the HTML back to Crawl4AI's normal markdown and extraction pipeline, so you pay $0.50 per 1,000 pages only for the fallbacks.
Updated · Read as markdown
What Crawl4AI does well
Crawl4AI is an open-source crawler for LLM pipelines, written in Python on Playwright, with about 86,000 GitHub stars and a release every few weeks (0.9.5 came out on 14 Oct 2026). You run it yourself, so you control everything: browser profiles and sessions, hooks at every step, deep crawls (breadth-first, depth-first or best-first), an adaptive crawler that stops when it has learned enough, CSS and XPath extraction without an LLM, and "fit markdown" filters that strip menus and footers before your model sees the page. It ships a Docker server with a REST API and an MCP endpoint, and there's a hosted version, Crawl4AI Cloud.
If you want to own your crawler, Crawl4AI is a great choice. This page is about one gap that comes with owning it.
Where a self-hosted crawler runs out
A crawler you host fetches from your server's IP address. Plenty of sites treat requests from cloud and datacenter addresses differently: they rate-limit them, show a challenge page, or answer 403. Crawl4AI detects these responses (403 or 429 with an empty body, Cloudflare, Akamai and other challenge pages) and can retry, rotate through proxies you supply, and switch on stealth mode. Its docs are candid that none of that is guaranteed.
The same goes for location. locale, timezone_id and GeolocationConfig change what the browser reports, but the request still leaves from your server. A site that prices or ranks by IP country will still see your server's country.
For the pages that are left, Crawl4AI has a built-in last resort: fallback_fetch_function, an async function that takes a URL and returns raw HTML. Its docs suggest it for exactly this: "useful when you have access to a scraping API". That's where UpRock goes.
Crawl4AI and UpRock side by side
Crawl4AI Cloud's prices and limits were checked 24 Oct 2026; the rest comes from its README and docs, checked 19 Oct 2026.
| Crawl4AI | UpRock | |
|---|---|---|
| Price | Library: open source (Apache-2.0 with an attribution clause); you pay for your server and proxies. Cloud: from $10 a month for 10,000 credits. A plain page is half a credit ($0.50 per 1,000 on that plan), 2× when a browser renders it and 4× when the site fights back, set per page (live price table, marked soft launch) | $0.50 per 1,000 pages by card or USDC, $0.40 with UPT |
| Where requests come from | Library: your machine, or proxies you configure. Cloud: its network, with ISP or residential proxies on request | Real desktop and mobile devices whose owners opted in |
| Country | Library: browser-level locale, time zone and coordinates; the IP doesn't change. Cloud: a country option, with proxies at 2× (ISP) or 4× (residential) the page price | Every request: 63 countries, or a region |
| Control over the crawl | Full: hooks, sessions, deep crawl, extraction strategies, your own browser | One page per call, returned as markdown and HTML |
| Fallback hook | fallback_fetch_function | This page plugs into it |
| MCP server | In the Docker server (SSE and WebSocket) | Yes, remote (streamable HTTP) |
| Video | Collects the video URLs it finds on a page | Search public video by moment, $1 per 1,000 searches |
| Rate limits | Yours to set. Cloud: 120 requests a minute on Starter | Not published yet |
Where Crawl4AI is better: you own the code and the data path, it costs nothing to run beyond your server, it does far more than fetch a page, and its Cloud charges the same as UpRock for a plain page. Where UpRock fits: the pages that end blocked from your server, a request that really comes from the country you need, and paying per fallback instead of keeping a proxy contract for the few pages that need one.
Plug UpRock into fallback_fetch_function
The function below asks UpRock to load the page on a device in the country you pick and return the rendered HTML: one POST /v1/crawl request with format: html and your key in an Authorization: Bearer header. The CLI tab does the same with uprock ai crawl, then uprock ai fetch on the HTML resource. Crawl4AI runs that HTML through the same markdown generator and extraction strategy as every other page, so the rest of your pipeline doesn't change. max_retries=0 skips Crawl4AI's retry rounds and goes straight to the fallback, as its docs suggest for a safety net.
# pip install crawl4ai requests && crawl4ai-setup env: UPROCK_API_KEYimport asyncioimport osimport sysimport requestsfrom crawl4ai import AsyncWebCrawler, BrowserConfig, CacheMode, CrawlerRunConfigdef uprock_html(url: str, country: str) -> str: # One call: UpRock loads the page on a real device in that country, tries a fresh device # every 7 s (up to 3 tries) and returns the rendered HTML. Failed calls aren't billed. res = requests.post( "https://edge.uprock.com/v1/crawl", headers={"Authorization": f"Bearer {os.environ['UPROCK_API_KEY']}"}, json={"url": url, "country": country, "format": "html"}, timeout=90, ) page = res.json() if not res.ok: raise RuntimeError(f"UpRock {res.status_code} {page.get('code')}: {page.get('error')}") return page["html"]def uprock_fallback(country: str): """A fallback_fetch_function: takes a URL, returns raw HTML fetched by UpRock in `country`.""" async def fetch_html(url: str) -> str: return await asyncio.to_thread(uprock_html, url, country) return fetch_htmlasync def crawl(url: str, country: str = "US"): config = CrawlerRunConfig( cache_mode=CacheMode.BYPASS, max_retries=0, # once Crawl4AI detects a block, go straight to the fallback fallback_fetch_function=uprock_fallback(country), ) async with AsyncWebCrawler(config=BrowserConfig(headless=True)) as crawler: result = await crawler.arun(url=url, config=config) # resolved_by is "direct", "proxy" or "fallback_fetch", or None if every path failed print( "resolved by:", result.crawl_stats.get("resolved_by"), "| success:", result.success, ) return resultif __name__ == "__main__": # python crawl.py https://developer.mozilla.org/en-US/docs/Web/HTTP DE url = sys.argv[1] if len(sys.argv) > 1 else "https://developer.mozilla.org/en-US/docs/Web/HTTP" result = asyncio.run(crawl(url, sys.argv[2] if len(sys.argv) > 2 else "US")) print( result.markdown.raw_markdown[:300] if result.success else result.error_message )# pip install crawl4ai && crawl4ai-setup; npm i -g uprock env: UPROCK_API_KEYimport asyncioimport jsonimport sysfrom crawl4ai import AsyncWebCrawler, BrowserConfig, CacheMode, CrawlerRunConfigasync def run(*args: str) -> bytes: proc = await asyncio.create_subprocess_exec( *args, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE ) out, err = await proc.communicate() if proc.returncode != 0: raise RuntimeError(f"UpRock: {err.decode().strip()}") return outdef uprock_fallback(country: str): """A fallback_fetch_function: takes a URL, returns raw HTML fetched by UpRock in `country`.""" async def fetch_html(url: str) -> str: page = json.loads(await run("uprock", "ai", "crawl", url, "--country", country)) return ( await run("uprock", "ai", "fetch", page["content"]["html"]["resource"]) ).decode() return fetch_htmlasync def crawl(url: str, country: str = "US"): config = CrawlerRunConfig( cache_mode=CacheMode.BYPASS, max_retries=0, # once Crawl4AI detects a block, go straight to the fallback fallback_fetch_function=uprock_fallback(country), ) async with AsyncWebCrawler(config=BrowserConfig(headless=True)) as crawler: result = await crawler.arun(url=url, config=config) # resolved_by is "direct", "proxy" or "fallback_fetch", or None if every path failed print( "resolved by:", result.crawl_stats.get("resolved_by"), "| success:", result.success, ) return resultif __name__ == "__main__": # python crawl.py https://developer.mozilla.org/en-US/docs/Web/HTTP DE url = sys.argv[1] if len(sys.argv) > 1 else "https://developer.mozilla.org/en-US/docs/Web/HTTP" result = asyncio.run(crawl(url, sys.argv[2] if len(sys.argv) > 2 else "US")) print( result.markdown.raw_markdown[:300] if result.success else result.error_message )From the Docker server, in TypeScript
If you run Crawl4AI's Docker server and call it over REST, the Python hook isn't available, so do the fallback in your own code. The /md endpoint returns { markdown, success }, and answers HTTP 502 when the crawl fails. The HTTP version calls POST /v1/crawl; the MCP tab does the same through UpRock's MCP server.
// Crawl4AI's Docker server first, UpRock when the crawl fails.// Node 18+ env: CRAWL4AI_API_TOKEN, UPROCK_API_KEYconst CRAWL4AI = process.env.CRAWL4AI_URL ?? "http://localhost:11235";async function viaUpRock(url: string, country: string) { // One call: UpRock loads the page on a real device in that country, tries a fresh device // every 7 s (up to 3 tries) and returns markdown. Failed calls aren't billed. const res = await fetch("https://edge.uprock.com/v1/crawl", { method: "POST", headers: { Authorization: `Bearer ${process.env.UPROCK_API_KEY}`, "Content-Type": "application/json", }, body: JSON.stringify({ url, country }), signal: AbortSignal.timeout(90_000), }); const page = await res.json(); if (!res.ok) throw new Error(`UpRock ${res.status} ${page.code}: ${page.error}`); return page.markdown ?? "";}// POST /md answers { markdown, success } when the crawl works, 502 when it fails, 504 on timeout.async function viaCrawl4ai(url: string): Promise<string | null> { const res = await fetch(`${CRAWL4AI}/md`, { method: "POST", headers: { Authorization: `Bearer ${process.env.CRAWL4AI_API_TOKEN}`, "Content-Type": "application/json", }, body: JSON.stringify({ url, f: "fit" }), }); if (res.status === 502 || res.status === 504) return null; if (!res.ok) throw new Error(`Crawl4AI ${res.status}: ${await res.text()}`); // token, request or server problem const body = await res.json(); return body.success ? body.markdown : null;}export async function markdown(url: string, country = "US") { const md = await viaCrawl4ai(url); if (md) return { via: "crawl4ai", markdown: md }; console.warn(`Crawl4AI couldn't crawl ${url}; trying UpRock from ${country}`); return { via: "uprock", markdown: await viaUpRock(url, country) };}// node md.ts https://developer.mozilla.org/en-US/docs/Web/HTTP DEconst [url = "https://developer.mozilla.org/en-US/docs/Web/HTTP", country = "US"] = process.argv.slice(2);const page = await markdown(url, country);console.log(page.via, page.markdown.slice(0, 200));// Crawl4AI's Docker server first, UpRock when the crawl fails.// npm i @modelcontextprotocol/sdk (Node 18+) env: CRAWL4AI_API_TOKEN, UPROCK_API_KEYimport { Client } from "@modelcontextprotocol/sdk/client/index.js";import { StreamableHTTPClientTransport } from "@modelcontextprotocol/sdk/client/streamableHttp.js";const CRAWL4AI = process.env.CRAWL4AI_URL ?? "http://localhost:11235";const uprock = new Client({ name: "crawl4ai-fallback", version: "1.0.0" });await uprock.connect( new StreamableHTTPClientTransport( new URL("https://mcp.uprock.ai/mcp"), { requestInit: { headers: { Authorization: `Bearer ${process.env.UPROCK_API_KEY}` } } }, ),);const textOf = (content: unknown) => (content as { type: string; text?: string }[]).find((c) => c.type === "text")?.text ?? "";async function viaUpRock(url: string, country: string) { const res = await uprock.callTool( { name: "crawl_fetch", arguments: { url, country } }, undefined, { timeout: 300_000 }, ); if (res.isError) throw new Error(`UpRock: ${textOf(res.content)}`); const uri = JSON.parse(textOf(res.content)).content?.markdown?.resource; if (!uri) throw new Error(`UpRock: no markdown for ${url}`); return textOf( (await uprock.callTool({ name: "resource_fetch", arguments: { uri } })).content, );}// POST /md answers { markdown, success } when the crawl works, 502 when it fails, 504 on timeout.async function viaCrawl4ai(url: string): Promise<string | null> { const res = await fetch(`${CRAWL4AI}/md`, { method: "POST", headers: { Authorization: `Bearer ${process.env.CRAWL4AI_API_TOKEN}`, "Content-Type": "application/json", }, body: JSON.stringify({ url, f: "fit" }), }); if (res.status === 502 || res.status === 504) return null; if (!res.ok) throw new Error(`Crawl4AI ${res.status}: ${await res.text()}`); // token, request or server problem const body = await res.json(); return body.success ? body.markdown : null;}export async function markdown(url: string, country = "US") { const md = await viaCrawl4ai(url); if (md) return { via: "crawl4ai", markdown: md }; console.warn(`Crawl4AI couldn't crawl ${url}; trying UpRock from ${country}`); return { via: "uprock", markdown: await viaUpRock(url, country) };}// node md.ts https://developer.mozilla.org/en-US/docs/Web/HTTP DEconst [url = "https://developer.mozilla.org/en-US/docs/Web/HTTP", country = "US"] = process.argv.slice(2);try { const page = await markdown(url, country); console.log(page.via, page.markdown.slice(0, 200));} finally { await uprock.close();}Tested 19 Oct 2026 with Crawl4AI 0.9.5 and the live UpRock API: on a test URL that always answers 429, Crawl4AI called the fallback and UpRock returned the page (resolved_by: fallback_fetch). We couldn't run the Docker server, so the TypeScript version was tested against a stand-in that answers /md the way the server's code does. On 6 Nov 2026 we ran the TypeScript HTTP and MCP versions again with an UpRock API key in an Authorization header, and the Python HTTP version's UpRock fallback on its own (without Crawl4AI installed); each got the page.
FAQ
When does Crawl4AI call fallback_fetch_function?
After every crawl attempt (and every retry round and proxy in proxy_config) has ended with a detected block, such as a 403 or 429 with an empty body, or a challenge page. Crawl4AI then calls the function with the URL and processes the HTML it returns like any other page. result.crawl_stats["resolved_by"] tells you which path won.
Does Crawl4AI come with proxies?
The open-source library doesn't. You configure your own in proxy_config. Crawl4AI Cloud, the hosted version, can fetch from a chosen country through ISP or residential proxies, at 2× and 4× its page price.
Do Crawl4AI's locale and geolocation settings change the country a site sees?
They change what the browser reports (language, time zone, coordinates for the Geolocation API). The request still leaves from your server's IP address unless you add a proxy. UpRock's country setting picks a device that is actually in that country; when we fetched an IP lookup page with it set to DE, JP and BR, the page reported Berlin, Nagoya and Teresina.
Can I use this from the Crawl4AI Docker server?
Not as a hook. fallback_fetch_function is a Python callable, and since 0.9.0 the Docker server rejects proxy settings in request bodies. Call the server, and if a URL fails, fetch it from UpRock in your own code, as in the TypeScript example on this page.
Is Crawl4AI free?
The library is open source under Apache-2.0 with an added attribution clause; you pay for the machine it runs on and any proxies. Crawl4AI Cloud includes 1,000 credits a month at no cost and has paid plans from $10 a month.
Other use cases to explore
- See a page as it looks in another country: one store page from five countries, with five prices, two languages and a cookie banner in one country only.
- Decide which failed pages are worth retrying: a diagnosis table from status code to the legitimate fix, and when to stop.
- Fetch from countries your crawler doesn't cover: such as South Korea and Indonesia, sent straight to a device in that country.
- Search public video by moment: find the second something is shown or said, across platforms.
References
- Crawl4AI on GitHub (README, license, releases), checked 19 Oct 2026
- Crawl4AI license (Apache-2.0 with an attribution clause), checked 19 Oct 2026
- Crawl4AI docs, Anti-bot detection and fallback, checked 19 Oct 2026
- Crawl4AI docs, Proxy and security, checked 19 Oct 2026
- Crawl4AI docs, Identity-based crawling (locale, timezone, geolocation), checked 19 Oct 2026
- Crawl4AI docs, Self-hosting (Docker server, REST and MCP), checked 19 Oct 2026
- Crawl4AI Docker migration notes (0.9.0 request-body rules), checked 19 Oct 2026
- Crawl4AI Cloud docs (plans, country and proxy options), checked 24 Oct 2026
- Crawl4AI Cloud, live price table (credits per page, multipliers, plans), checked 24 Oct 2026
- Crawl4AI on PyPI (latest release), checked 19 Oct 2026
- UpRock, everything in one file (API, CLI, MCP, pricing), checked 24 Oct 2026
Get started
Add UpRock in one paste.
Paste this into Claude Code, Codex, Cursor or any agent with a terminal. It reads our docs, sets up the MCP server or CLI, and tells you the cost before it spends anything.
Set up UpRock for me: read https://uprock.ai/agents.md, connect the UpRock MCP server, and run one test crawl. Tell me what it costs before you spend anything.