# How do I add a hosted fallback to Crawl4AI?

Pass an UpRock fetch as Crawl4AI's fallback_fetch_function. When Crawl4AI's own attempts end blocked, it calls your function, which loads the page on a real device in the country you choose and hands the HTML back to Crawl4AI's normal markdown and extraction pipeline, so you pay $0.50 per 1,000 pages only for the fallbacks.

Updated 24 Oct 2026. Web page: https://uprock.ai/compare/crawl4ai

## On this page

- [What Crawl4AI does well](#what-crawl4ai-does-well)
- [Where a self-hosted crawler runs out](#where-a-self-hosted-crawler-runs-out)
- [Crawl4AI and UpRock side by side](#crawl4ai-and-uprock-side-by-side)
- [Plug UpRock into fallback_fetch_function](#plug-uprock-into-fallback_fetch_function)
- [From the Docker server, in TypeScript](#from-the-docker-server-in-typescript)
- [FAQ](#faq)
- [Other use cases to explore](#explore)
- [Related articles](#related)
- [References](#references)
- [Get started](#get-started)

## What Crawl4AI does well

Crawl4AI is an open-source crawler for LLM pipelines, written in Python on Playwright, with about 86,000 GitHub stars and a release every few weeks (0.9.5 came out on 14 Oct 2026). You run it yourself, so you control everything: browser profiles and sessions, hooks at every step, deep crawls (breadth-first, depth-first or best-first), an adaptive crawler that stops when it has learned enough, CSS and XPath extraction without an LLM, and "fit markdown" filters that strip menus and footers before your model sees the page. It ships a Docker server with a REST API and an MCP endpoint, and there's a hosted version, Crawl4AI Cloud.

If you want to own your crawler, Crawl4AI is a great choice. This page is about one gap that comes with owning it.

## Where a self-hosted crawler runs out

A crawler you host fetches from your server's IP address. Plenty of sites treat requests from cloud and datacenter addresses differently: they rate-limit them, show a challenge page, or answer 403. Crawl4AI detects these responses (403 or 429 with an empty body, Cloudflare, Akamai and other challenge pages) and can retry, rotate through proxies you supply, and switch on stealth mode. Its docs are candid that none of that is guaranteed.

The same goes for location. `locale`, `timezone_id` and `GeolocationConfig` change what the browser reports, but the request still leaves from your server. A site that prices or ranks by IP country will still see your server's country.

For the pages that are left, Crawl4AI has a built-in last resort: `fallback_fetch_function`, an async function that takes a URL and returns raw HTML. Its docs suggest it for exactly this: "useful when you have access to a scraping API". That's where UpRock goes.

## Crawl4AI and UpRock side by side

Crawl4AI Cloud's prices and limits were checked 24 Oct 2026; the rest comes from its README and docs, checked 19 Oct 2026.

| | Crawl4AI | UpRock |
|---|---|---|
| Price | Library: [open source](https://github.com/unclecode/crawl4ai/blob/main/LICENSE) (Apache-2.0 with an attribution clause); you pay for your server and proxies. [Cloud](https://crawl4ai.com/docs): from $10 a month for 10,000 credits. A plain page is half a credit ($0.50 per 1,000 on that plan), 2× when a browser renders it and 4× when the site fights back, set per page ([live price table](https://api.crawl4ai.com/v1/prices), marked soft launch) | $0.50 per 1,000 pages by card or USDC, $0.40 with UPT |
| Where requests come from | Library: your machine, or [proxies you configure](https://docs.crawl4ai.com/advanced/proxy-security/). Cloud: its network, with ISP or residential proxies on request | Real desktop and mobile devices whose owners opted in |
| Country | Library: [browser-level](https://docs.crawl4ai.com/advanced/identity-based-crawling/) locale, time zone and coordinates; the IP doesn't change. Cloud: a `country` option, with proxies at 2× (ISP) or 4× (residential) the page price | Every request: 63 countries, or a region |
| Control over the crawl | Full: hooks, sessions, deep crawl, extraction strategies, your own browser | One page per call, returned as markdown and HTML |
| Fallback hook | [`fallback_fetch_function`](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/) | This page plugs into it |
| MCP server | [In the Docker server](https://docs.crawl4ai.com/core/self-hosting/) (SSE and WebSocket) | Yes, remote (streamable HTTP) |
| Video | Collects the video URLs it finds on a page | Search public video by moment, $1 per 1,000 searches |
| Rate limits | Yours to set. Cloud: 120 requests a minute on Starter | Not published yet |

Where Crawl4AI is better: you own the code and the data path, it costs nothing to run beyond your server, it does far more than fetch a page, and its Cloud charges the same as UpRock for a plain page. Where UpRock fits: the pages that end blocked from your server, a request that really comes from the country you need, and paying per fallback instead of keeping a proxy contract for the few pages that need one.

## Plug UpRock into fallback_fetch_function

The function below asks UpRock to load the page on a device in the country you pick and return the rendered HTML: one `POST /v1/crawl` request with `format: html` and your key in an `Authorization: Bearer` header. The CLI tab does the same with `uprock ai crawl`, then `uprock ai fetch` on the HTML resource. Crawl4AI runs that HTML through the same markdown generator and extraction strategy as every other page, so the rest of your pipeline doesn't change. `max_retries=0` skips Crawl4AI's retry rounds and goes straight to the fallback, as its docs suggest for a safety net.

### Python (HTTP)

```python
# pip install crawl4ai requests && crawl4ai-setup   env: UPROCK_API_KEY
import asyncio
import os
import sys

import requests
from crawl4ai import AsyncWebCrawler, BrowserConfig, CacheMode, CrawlerRunConfig


def uprock_html(url: str, country: str) -> str:
    # One call: UpRock loads the page on a real device in that country, tries a fresh device
    # every 7 s (up to 3 tries) and returns the rendered HTML. Failed calls aren't billed.
    res = requests.post(
        "https://edge.uprock.com/v1/crawl",
        headers={"Authorization": f"Bearer {os.environ['UPROCK_API_KEY']}"},
        json={"url": url, "country": country, "format": "html"},
        timeout=90,
    )
    page = res.json()
    if not res.ok:
        raise RuntimeError(f"UpRock {res.status_code} {page.get('code')}: {page.get('error')}")
    return page["html"]


def uprock_fallback(country: str):
    """A fallback_fetch_function: takes a URL, returns raw HTML fetched by UpRock in `country`."""

    async def fetch_html(url: str) -> str:
        return await asyncio.to_thread(uprock_html, url, country)

    return fetch_html


async def crawl(url: str, country: str = "US"):
    config = CrawlerRunConfig(
        cache_mode=CacheMode.BYPASS,
        max_retries=0,  # once Crawl4AI detects a block, go straight to the fallback
        fallback_fetch_function=uprock_fallback(country),
    )
    async with AsyncWebCrawler(config=BrowserConfig(headless=True)) as crawler:
        result = await crawler.arun(url=url, config=config)
    # resolved_by is "direct", "proxy" or "fallback_fetch", or None if every path failed
    print(
        "resolved by:",
        result.crawl_stats.get("resolved_by"),
        "| success:",
        result.success,
    )
    return result


if __name__ == "__main__":
    # python crawl.py https://developer.mozilla.org/en-US/docs/Web/HTTP DE
    url = sys.argv[1] if len(sys.argv) > 1 else "https://developer.mozilla.org/en-US/docs/Web/HTTP"
    result = asyncio.run(crawl(url, sys.argv[2] if len(sys.argv) > 2 else "US"))
    print(
        result.markdown.raw_markdown[:300] if result.success else result.error_message
    )
```

### Python (CLI)

```python
# pip install crawl4ai && crawl4ai-setup; npm i -g uprock   env: UPROCK_API_KEY
import asyncio
import json
import sys

from crawl4ai import AsyncWebCrawler, BrowserConfig, CacheMode, CrawlerRunConfig


async def run(*args: str) -> bytes:
    proc = await asyncio.create_subprocess_exec(
        *args, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE
    )
    out, err = await proc.communicate()
    if proc.returncode != 0:
        raise RuntimeError(f"UpRock: {err.decode().strip()}")
    return out


def uprock_fallback(country: str):
    """A fallback_fetch_function: takes a URL, returns raw HTML fetched by UpRock in `country`."""

    async def fetch_html(url: str) -> str:
        page = json.loads(await run("uprock", "ai", "crawl", url, "--country", country))
        return (
            await run("uprock", "ai", "fetch", page["content"]["html"]["resource"])
        ).decode()

    return fetch_html


async def crawl(url: str, country: str = "US"):
    config = CrawlerRunConfig(
        cache_mode=CacheMode.BYPASS,
        max_retries=0,  # once Crawl4AI detects a block, go straight to the fallback
        fallback_fetch_function=uprock_fallback(country),
    )
    async with AsyncWebCrawler(config=BrowserConfig(headless=True)) as crawler:
        result = await crawler.arun(url=url, config=config)
    # resolved_by is "direct", "proxy" or "fallback_fetch", or None if every path failed
    print(
        "resolved by:",
        result.crawl_stats.get("resolved_by"),
        "| success:",
        result.success,
    )
    return result


if __name__ == "__main__":
    # python crawl.py https://developer.mozilla.org/en-US/docs/Web/HTTP DE
    url = sys.argv[1] if len(sys.argv) > 1 else "https://developer.mozilla.org/en-US/docs/Web/HTTP"
    result = asyncio.run(crawl(url, sys.argv[2] if len(sys.argv) > 2 else "US"))
    print(
        result.markdown.raw_markdown[:300] if result.success else result.error_message
    )
```

## From the Docker server, in TypeScript

If you run Crawl4AI's Docker server and call it over REST, the Python hook isn't available, so do the fallback in your own code. The `/md` endpoint returns `{ markdown, success }`, and answers HTTP 502 when the crawl fails. The HTTP version calls `POST /v1/crawl`; the MCP tab does the same through UpRock's MCP server.

### TypeScript (HTTP)

```ts
// Crawl4AI's Docker server first, UpRock when the crawl fails.
// Node 18+   env: CRAWL4AI_API_TOKEN, UPROCK_API_KEY

const CRAWL4AI = process.env.CRAWL4AI_URL ?? "http://localhost:11235";

async function viaUpRock(url: string, country: string) {
  // One call: UpRock loads the page on a real device in that country, tries a fresh device
  // every 7 s (up to 3 tries) and returns markdown. Failed calls aren't billed.
  const res = await fetch("https://edge.uprock.com/v1/crawl", {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.UPROCK_API_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({ url, country }),
    signal: AbortSignal.timeout(90_000),
  });
  const page = await res.json();
  if (!res.ok) throw new Error(`UpRock ${res.status} ${page.code}: ${page.error}`);
  return page.markdown ?? "";
}

// POST /md answers { markdown, success } when the crawl works, 502 when it fails, 504 on timeout.
async function viaCrawl4ai(url: string): Promise<string | null> {
  const res = await fetch(`${CRAWL4AI}/md`, {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.CRAWL4AI_API_TOKEN}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({ url, f: "fit" }),
  });
  if (res.status === 502 || res.status === 504) return null;
  if (!res.ok) throw new Error(`Crawl4AI ${res.status}: ${await res.text()}`); // token, request or server problem
  const body = await res.json();
  return body.success ? body.markdown : null;
}

export async function markdown(url: string, country = "US") {
  const md = await viaCrawl4ai(url);
  if (md) return { via: "crawl4ai", markdown: md };
  console.warn(`Crawl4AI couldn't crawl ${url}; trying UpRock from ${country}`);
  return { via: "uprock", markdown: await viaUpRock(url, country) };
}

// node md.ts https://developer.mozilla.org/en-US/docs/Web/HTTP DE
const [url = "https://developer.mozilla.org/en-US/docs/Web/HTTP", country = "US"] =
  process.argv.slice(2);
const page = await markdown(url, country);
console.log(page.via, page.markdown.slice(0, 200));
```

### TypeScript (MCP)

```ts
// Crawl4AI's Docker server first, UpRock when the crawl fails.
// npm i @modelcontextprotocol/sdk   (Node 18+)   env: CRAWL4AI_API_TOKEN, UPROCK_API_KEY
import { Client } from "@modelcontextprotocol/sdk/client/index.js";
import { StreamableHTTPClientTransport } from "@modelcontextprotocol/sdk/client/streamableHttp.js";

const CRAWL4AI = process.env.CRAWL4AI_URL ?? "http://localhost:11235";
const uprock = new Client({ name: "crawl4ai-fallback", version: "1.0.0" });
await uprock.connect(
  new StreamableHTTPClientTransport(
    new URL("https://mcp.uprock.ai/mcp"),
    { requestInit: { headers: { Authorization: `Bearer ${process.env.UPROCK_API_KEY}` } } },
  ),
);

const textOf = (content: unknown) =>
  (content as { type: string; text?: string }[]).find((c) => c.type === "text")?.text ??
  "";

async function viaUpRock(url: string, country: string) {
  const res = await uprock.callTool(
    { name: "crawl_fetch", arguments: { url, country } },
    undefined,
    { timeout: 300_000 },
  );
  if (res.isError) throw new Error(`UpRock: ${textOf(res.content)}`);
  const uri = JSON.parse(textOf(res.content)).content?.markdown?.resource;
  if (!uri) throw new Error(`UpRock: no markdown for ${url}`);
  return textOf(
    (await uprock.callTool({ name: "resource_fetch", arguments: { uri } })).content,
  );
}

// POST /md answers { markdown, success } when the crawl works, 502 when it fails, 504 on timeout.
async function viaCrawl4ai(url: string): Promise<string | null> {
  const res = await fetch(`${CRAWL4AI}/md`, {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.CRAWL4AI_API_TOKEN}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({ url, f: "fit" }),
  });
  if (res.status === 502 || res.status === 504) return null;
  if (!res.ok) throw new Error(`Crawl4AI ${res.status}: ${await res.text()}`); // token, request or server problem
  const body = await res.json();
  return body.success ? body.markdown : null;
}

export async function markdown(url: string, country = "US") {
  const md = await viaCrawl4ai(url);
  if (md) return { via: "crawl4ai", markdown: md };
  console.warn(`Crawl4AI couldn't crawl ${url}; trying UpRock from ${country}`);
  return { via: "uprock", markdown: await viaUpRock(url, country) };
}

// node md.ts https://developer.mozilla.org/en-US/docs/Web/HTTP DE
const [url = "https://developer.mozilla.org/en-US/docs/Web/HTTP", country = "US"] =
  process.argv.slice(2);
try {
  const page = await markdown(url, country);
  console.log(page.via, page.markdown.slice(0, 200));
} finally {
  await uprock.close();
}
```

Tested 19 Oct 2026 with Crawl4AI 0.9.5 and the live UpRock API: on a test URL that always answers 429, Crawl4AI called the fallback and UpRock returned the page (`resolved_by: fallback_fetch`). We couldn't run the Docker server, so the TypeScript version was tested against a stand-in that answers `/md` the way the server's code does. On 6 Nov 2026 we ran the TypeScript HTTP and MCP versions again with an UpRock API key in an `Authorization` header, and the Python HTTP version's UpRock fallback on its own (without Crawl4AI installed); each got the page.

## FAQ

### When does Crawl4AI call fallback_fetch_function?

After every crawl attempt (and every retry round and proxy in proxy_config) has ended with a detected block, such as a 403 or 429 with an empty body, or a challenge page. Crawl4AI then calls the function with the URL and processes the HTML it returns like any other page. result.crawl_stats["resolved_by"] tells you which path won.

### Does Crawl4AI come with proxies?

The open-source library doesn't. You configure your own in proxy_config. Crawl4AI Cloud, the hosted version, can fetch from a chosen country through ISP or residential proxies, at 2× and 4× its page price.

### Do Crawl4AI's locale and geolocation settings change the country a site sees?

They change what the browser reports (language, time zone, coordinates for the Geolocation API). The request still leaves from your server's IP address unless you add a proxy. UpRock's country setting picks a device that is actually in that country; when we fetched an IP lookup page with it set to DE, JP and BR, the page reported Berlin, Nagoya and Teresina.

### Can I use this from the Crawl4AI Docker server?

Not as a hook. fallback_fetch_function is a Python callable, and since 0.9.0 the Docker server rejects proxy settings in request bodies. Call the server, and if a URL fails, fetch it from UpRock in your own code, as in the TypeScript example on this page.

### Is Crawl4AI free?

The library is open source under Apache-2.0 with an added attribution clause; you pay for the machine it runs on and any proxies. Crawl4AI Cloud includes 1,000 credits a month at no cost and has paid plans from $10 a month.

## Other use cases to explore

- [See a page as it looks in another country](https://uprock.ai/use-cases/page-from-another-country): one store page from five countries, with five prices, two languages and a cookie banner in one country only.
- [Decide which failed pages are worth retrying](https://uprock.ai/use-cases/agent-gets-403): a diagnosis table from status code to the legitimate fix, and when to stop.
- [Fetch from countries your crawler doesn't cover](https://uprock.ai/compare/firecrawl#when-a-second-crawler-helps): such as South Korea and Indonesia, sent straight to a device in that country.
- [Search public video by moment](https://uprock.ai/video-semantic-search): find the second something is shown or said, across platforms.

## Related articles

- [How do I add a backup crawler to Firecrawl?](https://uprock.ai/compare/firecrawl.md): When to send a page from Firecrawl to UpRock, published prices side by side, and a tested fallback in TypeScript and Python.
- [How do I add a backup for Tavily Extract?](https://uprock.ai/compare/tavily.md): Retry the URLs in Tavily Extract's failed results on a real device, and read pages as they appear in a chosen country. Prices side by side and tested code.
- [How do I add a fallback for Exa contents errors?](https://uprock.ai/compare/exa.md): Read the per-URL statuses in Exa's /contents response and retry the right ones live on a real device. Error tags explained, prices side by side, tested code.
- [How do I add a backup to Jina Reader, and when is it cheaper?](https://uprock.ai/compare/jina-reader.md): Spot a failed read in Reader's JSON (it still answers 200), retry it on a real device, and see what six real pages cost in Reader tokens next to a flat per-page price.
- [How do I add a backup to Bright Data's Web Unlocker?](https://uprock.ai/compare/bright-data.md): Which Web Unlocker errors to retry, which to send to a second network and which to respect, with Bright Data's published prices next to ours and tested fallback code.
- [How do I add a fallback to an Apify or Crawlee crawler?](https://uprock.ai/compare/apify.md): Hand the requests Crawlee gives up on to a second network with failedRequestHandler, see what a fetched page costs on Apify's own Actors, and run the tested code.
- [When should a Browserbase agent read a page without opening a browser?](https://uprock.ai/compare/browserbase.md): What 1,000 page reads cost on Browserbase three ways, when to send a read to UpRock instead (including when every browser is busy), and tested fallback code.
- [How do I cap ScrapingBee's credit cost and send the rest to a backup?](https://uprock.ai/compare/scrapingbee.md): ScrapingBee charges 1 to 75 credits a page depending on the options. Cap it with Auto-Mode and max_cost, watch the country fallback, and send the rest to UpRock with tested code.
- [Spider costs less per page than UpRock. When is a backup still worth it?](https://uprock.ai/compare/spider.md): Spider is cheaper per page, so keep it as your primary. How Spider reports a page it couldn't get (from our own tests), and tested code that sends only those pages to UpRock.
- [How do I add a fallback to Parallel's Extract API?](https://uprock.ai/compare/parallel.md): Make Parallel Extract report a failed live fetch instead of quietly serving an old cached copy, then retry those URLs on UpRock. Prices side by side and tested code.
- [How do I add a backup to ZenRows?](https://uprock.ai/compare/zenrows.md): Which ZenRows error codes to send to a second network and which to respect, what a page from one country costs once premium proxies are required, and tested fallback code.
- [How do I retry a ScraperAPI batch's failed jobs on another network?](https://uprock.ai/compare/scraperapi.md): Send the jobs ScraperAPI's async batch gives up on to UpRock, see which countries each ScraperAPI plan can target, and compare credit costs. Tested code for both.
- [How do I add a backup to Oxylabs' new Web API?](https://uprock.ai/compare/oxylabs.md): Which Oxylabs Web API failures to send to a second network and which to respect, what a generic page really costs after its October 2026 pricing change, and tested fallback code.
- [How do I add a backup to Context.dev's scrape API?](https://uprock.ai/compare/context-dev.md): Context.dev reports a failed scrape inside an HTTP 200. Which of its error codes to retry on a second network, what a page costs by plan, and tested fallback code.
- [How do I add a backup to Scrapfly instead of paying for residential?](https://uprock.ai/compare/scrapfly.md): A Scrapfly page costs 1 to 30 credits depending on proxy pool and browser. What that is per 1,000 pages on each plan, which errors to send on, and tested code.
- [How do I keep Crawlbase's site tiers from surprising my bill?](https://uprock.ai/compare/crawlbase.md): Crawlbase bills each site at 1 to 20 times its base rate. What that does to the cost per 1,000 pages, and tested code that moves pricey domains to UpRock.
- [Which unlocker API fits: Bright Data, Oxylabs, Zyte or UpRock?](https://uprock.ai/compare/web-unlocker-apis.md): Bright Data's Web Unlocker, Oxylabs' Web API and Zyte API priced per 1,000 hard pages, with what each bills and when a second network helps.
- [Compare: all articles](https://uprock.ai/compare.md)

## References

1. [Crawl4AI on GitHub (README, license, releases)](https://github.com/unclecode/crawl4ai), checked 19 Oct 2026
2. [Crawl4AI license (Apache-2.0 with an attribution clause)](https://github.com/unclecode/crawl4ai/blob/main/LICENSE), checked 19 Oct 2026
3. [Crawl4AI docs, Anti-bot detection and fallback](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/), checked 19 Oct 2026
4. [Crawl4AI docs, Proxy and security](https://docs.crawl4ai.com/advanced/proxy-security/), checked 19 Oct 2026
5. [Crawl4AI docs, Identity-based crawling (locale, timezone, geolocation)](https://docs.crawl4ai.com/advanced/identity-based-crawling/), checked 19 Oct 2026
6. [Crawl4AI docs, Self-hosting (Docker server, REST and MCP)](https://docs.crawl4ai.com/core/self-hosting/), checked 19 Oct 2026
7. [Crawl4AI Docker migration notes (0.9.0 request-body rules)](https://github.com/unclecode/crawl4ai/blob/main/deploy/docker/MIGRATION.md), checked 19 Oct 2026
8. [Crawl4AI Cloud docs (plans, country and proxy options)](https://crawl4ai.com/docs), checked 24 Oct 2026
9. [Crawl4AI Cloud, live price table (credits per page, multipliers, plans)](https://api.crawl4ai.com/v1/prices), checked 24 Oct 2026
10. [Crawl4AI on PyPI (latest release)](https://pypi.org/project/crawl4ai/), checked 19 Oct 2026
11. [UpRock, everything in one file (API, CLI, MCP, pricing)](https://uprock.ai/llms-full.txt), checked 24 Oct 2026

## Get started

Get an API key: https://uprock.ai/app/api-keys

Paste this into your agent:

```text
Set up UpRock for me: read https://uprock.ai/agents.md, connect the UpRock MCP server, and run one test crawl. Tell me what it costs before you spend anything.
```

Setup for every client (MCP, skill, CLI): https://uprock.ai/agents.md