# How do I add a fallback to an Apify or Crawlee crawler?

Use Crawlee's failedRequestHandler. Once a request has used up its retries, hand the URL to UpRock, which loads it on an opted-in device in the country you choose for $0.50 per 1,000 pages, and keep Apify for everything around it, like scheduling, storage and the Actor store.

Updated 21 Oct 2026. Web page: https://uprock.ai/compare/apify

## On this page

- [What Apify does well](#what-apify-does-well)
- [Where a Crawlee crawler gives up](#where-a-crawlee-crawler-gives-up)
- [What a fetched page costs on Apify](#what-a-fetched-page-costs-on-apify)
- [Apify and UpRock side by side](#apify-and-uprock-side-by-side)
- [Hand Crawlee's failed requests to UpRock](#hand-crawlees-failed-requests-to-uprock)
- [FAQ](#faq)
- [Other use cases to explore](#explore)
- [Related articles](#related)
- [References](#references)
- [Get started](#get-started)

## What Apify does well

Apify is a platform for running scrapers and automations, which it calls Actors. Its store lists over 94,000 of them, from Google Maps and Amazon scrapers to tools built for AI agents, and you can run any of them on a schedule, store the results in datasets, and plug them into LangChain, n8n, Zapier, Make and others. It also maintains Crawlee, an open-source crawling library for Node.js and Python (Apache-2.0), and an official MCP server that gives agents its RAG Web Browser and Web Fetch Actors out of the box.

If you build crawlers, Apify takes care of the parts around them: running, scaling, scheduling, storing. Keep it.

## Where a Crawlee crawler gives up

Crawlee already tries hard before it gives up. When a site answers 401, 403 or 429, it marks the session as blocked, rotates to a fresh one and retries, up to `maxRequestRetries` times. Then the request is marked as failed, and the URL drops out of your results.

That last step is the place for a backup. Crawlee calls `failedRequestHandler` with the request and the final error. Hand the URL to UpRock there, and the page gets one more try, from a different network and from a device in the country you choose.

## What a fetched page costs on Apify

If you'd rather not run a crawler at all, Apify sells page fetching through its own Actors, priced per result. These are Apify's published prices, checked 21 Oct 2026:

| How you fetch on Apify | Price per 1,000 pages | Notes |
|---|---|---|
| [RAG Web Browser](https://apify.com/apify/rag-web-browser/pricing), Standby mode | $2.28 Free, $1.90 Starter, $1.62 Scale, $1.33 Business | Platform usage included; failed pages not charged |
| RAG Web Browser, one run per URL | About $1.80 more | A start fee of $0.000225 per GB of run memory; the default is 8 GB |
| [Web Fetch](https://apify.com/apify/web-fetch) | $1.50 Free and Starter, $1.25 Scale, $1.00 Business | Nothing for failed requests; 10 MB and 2-minute cap |
| [Website Content Crawler](https://apify.com/apify/website-content-crawler) | About $0.20 raw HTTP, $0.50 to $5 with a headless browser | Apify's estimate at $0.25 per compute unit; plans now list $0.13 to $0.20 |
| Any of the above, from a chosen country | Plus residential proxy traffic | [$8 per GB](https://apify.com/pricing) on Free and Starter, down to $7 |
| UpRock | $0.50 by card or USDC, $0.40 with UPT | Country included; one credit per page |

## Apify and UpRock side by side

| | Apify | UpRock |
|---|---|---|
| What it is | A platform for running scrapers, with a store of ready-made Actors | An API that fetches one page per call |
| Price per 1,000 pages | See the table above: $1.00 to $2.28 for Apify's fetch Actors | $0.50 by card or USDC, $0.40 with UPT |
| Scheduling, storage, datasets | Yes | No |
| Open source | [Crawlee](https://github.com/apify/crawlee), Apache-2.0 | No. The device runtime (Fuel Engine) is |
| Where requests come from | [Datacenter, residential or SERP proxy groups](https://docs.apify.com/platform/proxy); how residential IPs are sourced isn't stated | Devices whose owners opted in |
| Country | Per proxy configuration; fails if there's no proxy in that country | Every request: 63 countries, or a region |
| MCP server | [Official](https://docs.apify.com/platform/integrations/mcp), remote, OAuth or token | Yes, remote (streamable HTTP) |
| Video | YouTube scraper (maintained by Apify) with paid transcript and per-video description add-ons | Search public video by moment, $1 per 1,000 searches |
| To try it | $5 of platform credit a month on the Free plan | Prepaid credits |
| API rate limits | [Published](https://docs.apify.com/api/v2): 60 requests a second per resource | Not published yet |

Where Apify is better: everything around the fetch, from the Actor store to scheduling, storage and integrations, plus an open-source crawler and a monthly credit to try it. Where UpRock fits: the requests your crawler gives up on, a page from a chosen country without buying proxy traffic by the gigabyte, and a lower price per page than Apify's fetch Actors.

## Hand Crawlee's failed requests to UpRock

Every version crawls two URLs: one that loads normally and a test URL that always answers 403. Crawlee reads the first itself, retries the second on fresh sessions, and when it gives up, `failedRequestHandler` sends the URL to UpRock. The TypeScript versions use Crawlee for Node.js and the Python versions Crawlee for Python. UpRock gets one `POST /v1/crawl` request with your key in an `Authorization: Bearer` header; the MCP and CLI tabs do the same through UpRock's MCP server (TypeScript) or CLI (Python).

### TypeScript (HTTP)

```ts
// A Crawlee crawler (Apify's open-source library; runs locally or as an Apify Actor)
// that hands pages it couldn't get to UpRock.
// npm i crawlee   (Node 18+)   env: UPROCK_API_KEY
import { CheerioCrawler } from "crawlee";

const COUNTRY = process.argv[2] ?? "US"; // where UpRock should load the pages from

async function viaUpRock(url: string) {
  // One call: UpRock loads the page on a real device in that country, tries a fresh device
  // every 7 s (up to 3 tries) and returns markdown. Failed calls aren't billed.
  const res = await fetch("https://edge.uprock.com/v1/crawl", {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.UPROCK_API_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({ url, country: COUNTRY }),
    signal: AbortSignal.timeout(90_000),
  });
  const page = await res.json();
  if (!res.ok) throw new Error(`UpRock ${res.status} ${page.code}: ${page.error}`);
  return page.markdown ?? "";
}

const crawler = new CheerioCrawler({
  maxRequestRetries: 2, // Crawlee retries first, on a fresh session each time
  async requestHandler({ request, $ }) {
    console.log("crawlee", request.url, $("head > title").text().trim());
  },
  // Runs after a request has used up its retries: blocked (401, 403, 429), 5xx or a network error.
  async failedRequestHandler({ request }, error) {
    try {
      const markdown = await viaUpRock(request.url);
      console.log(
        "uprock ",
        request.url,
        `${markdown.length} chars (Crawlee: ${error.message.slice(0, 60)})`,
      );
    } catch (err) {
      console.log("failed ", request.url, (err as Error).message);
    }
  },
});

// node crawl.ts DE
await crawler.run(["https://developer.mozilla.org/en-US/docs/Web/HTTP", "https://postman-echo.com/status/403"]);
```

### Python (HTTP)

```python
# A Crawlee crawler (Apify's open-source library; runs locally or as an Apify Actor)
# that hands pages it couldn't get to UpRock.
# pip install 'crawlee[beautifulsoup]' requests   env: UPROCK_API_KEY
import asyncio
import os
import sys

import requests
from crawlee.crawlers import (
    BasicCrawlingContext,
    BeautifulSoupCrawler,
    BeautifulSoupCrawlingContext,
)

COUNTRY = (
    sys.argv[1] if len(sys.argv) > 1 else "US"
)  # where UpRock should load the pages from


def via_uprock(url: str) -> str:
    # One call: UpRock loads the page on a real device in that country, tries a fresh device
    # every 7 s (up to 3 tries) and returns markdown. Failed calls aren't billed.
    res = requests.post(
        "https://edge.uprock.com/v1/crawl",
        headers={"Authorization": f"Bearer {os.environ['UPROCK_API_KEY']}"},
        json={"url": url, "country": COUNTRY},
        timeout=90,
    )
    page = res.json()
    if not res.ok:
        raise RuntimeError(f"UpRock {res.status_code} {page.get('code')}: {page.get('error')}")
    return page.get("markdown", "")


async def main() -> None:
    crawler = BeautifulSoupCrawler(max_request_retries=2)  # Crawlee retries first

    @crawler.router.default_handler
    async def handler(context: BeautifulSoupCrawlingContext) -> None:
        title = context.soup.title.get_text(strip=True) if context.soup.title else ""
        print("crawlee", context.request.url, title)

    # Runs after a request has used up its retries: blocked (401, 403, 429), 5xx or a network error.
    @crawler.failed_request_handler
    async def failed(context: BasicCrawlingContext, error: Exception) -> None:
        try:
            markdown = await asyncio.to_thread(via_uprock, context.request.url)
            print(
                "uprock ",
                context.request.url,
                f"{len(markdown)} chars (Crawlee: {str(error)[:60]})",
            )
        except Exception as err:
            print("failed ", context.request.url, err)

    await crawler.run(["https://developer.mozilla.org/en-US/docs/Web/HTTP", "https://postman-echo.com/status/403"])


if __name__ == "__main__":
    # python crawl.py DE
    asyncio.run(main())
```

### TypeScript (MCP)

```ts
// A Crawlee crawler (Apify's open-source library; runs locally or as an Apify Actor)
// that hands pages it couldn't get to UpRock.
// npm i crawlee @modelcontextprotocol/sdk   (Node 18+)   env: UPROCK_API_KEY
import { CheerioCrawler } from "crawlee";
import { Client } from "@modelcontextprotocol/sdk/client/index.js";
import { StreamableHTTPClientTransport } from "@modelcontextprotocol/sdk/client/streamableHttp.js";

const COUNTRY = process.argv[2] ?? "US"; // where UpRock should load the pages from
const uprock = new Client({ name: "crawlee-fallback", version: "1.0.0" });
await uprock.connect(
  new StreamableHTTPClientTransport(
    new URL("https://mcp.uprock.ai/mcp"),
    { requestInit: { headers: { Authorization: `Bearer ${process.env.UPROCK_API_KEY}` } } },
  ),
);

const textOf = (content: unknown) =>
  (content as { type: string; text?: string }[]).find((c) => c.type === "text")?.text ??
  "";

async function viaUpRock(url: string) {
  const res = await uprock.callTool(
    { name: "crawl_fetch", arguments: { url, country: COUNTRY } },
    undefined,
    { timeout: 300_000 },
  );
  if (res.isError) throw new Error(`UpRock: ${textOf(res.content)}`);
  const uri = JSON.parse(textOf(res.content)).content?.markdown?.resource;
  if (!uri) throw new Error(`UpRock: no markdown for ${url}`);
  return textOf(
    (await uprock.callTool({ name: "resource_fetch", arguments: { uri } })).content,
  );
}

const crawler = new CheerioCrawler({
  maxRequestRetries: 2, // Crawlee retries first, on a fresh session each time
  async requestHandler({ request, $ }) {
    console.log("crawlee", request.url, $("head > title").text().trim());
  },
  // Runs after a request has used up its retries: blocked (401, 403, 429), 5xx or a network error.
  async failedRequestHandler({ request }, error) {
    try {
      const markdown = await viaUpRock(request.url);
      console.log(
        "uprock ",
        request.url,
        `${markdown.length} chars (Crawlee: ${error.message.slice(0, 60)})`,
      );
    } catch (err) {
      console.log("failed ", request.url, (err as Error).message);
    }
  },
});

// node crawl.ts DE
await crawler.run(["https://developer.mozilla.org/en-US/docs/Web/HTTP", "https://postman-echo.com/status/403"]);
await uprock.close();
```

### Python (CLI)

```python
# A Crawlee crawler (Apify's open-source library; runs locally or as an Apify Actor)
# that hands pages it couldn't get to UpRock.
# pip install 'crawlee[beautifulsoup]'; npm i -g uprock   env: UPROCK_API_KEY
import asyncio
import json
import subprocess
import sys

from crawlee.crawlers import (
    BasicCrawlingContext,
    BeautifulSoupCrawler,
    BeautifulSoupCrawlingContext,
)

COUNTRY = (
    sys.argv[1] if len(sys.argv) > 1 else "US"
)  # where UpRock should load the pages from


def via_uprock(url: str) -> str:
    run = subprocess.run(
        ["uprock", "ai", "crawl", url, "--country", COUNTRY, "--content"],
        capture_output=True,
        text=True,
        timeout=330,
    )
    if run.returncode != 0:
        raise RuntimeError(f"UpRock: {run.stderr.strip()}")
    return json.loads(run.stdout).get("inlined_markdown", "")


async def main() -> None:
    crawler = BeautifulSoupCrawler(max_request_retries=2)  # Crawlee retries first

    @crawler.router.default_handler
    async def handler(context: BeautifulSoupCrawlingContext) -> None:
        title = context.soup.title.get_text(strip=True) if context.soup.title else ""
        print("crawlee", context.request.url, title)

    # Runs after a request has used up its retries: blocked (401, 403, 429), 5xx or a network error.
    @crawler.failed_request_handler
    async def failed(context: BasicCrawlingContext, error: Exception) -> None:
        try:
            markdown = await asyncio.to_thread(via_uprock, context.request.url)
            print(
                "uprock ",
                context.request.url,
                f"{len(markdown)} chars (Crawlee: {str(error)[:60]})",
            )
        except Exception as err:
            print("failed ", context.request.url, err)

    await crawler.run(["https://developer.mozilla.org/en-US/docs/Web/HTTP", "https://postman-echo.com/status/403"])


if __name__ == "__main__":
    # python crawl.py DE
    asyncio.run(main())
```

Tested 21 Oct 2026 with Crawlee 3.18 (Node.js) and 1.11 (Python) and the live UpRock API, no Apify account needed: Crawlee read example.com, retried a test URL that always answers 403 on new sessions, then handed it to UpRock. On 6 Nov 2026 we ran the HTTP and MCP versions again with an UpRock API key in an `Authorization` header: Crawlee read the MDN page itself and handed a page that answered 403 to UpRock.

If your agent calls Apify's RAG Web Browser instead, the same idea applies to its output: a page that failed comes back with `crawl.requestStatus` set to `failed` and no markdown. Send those URLs to UpRock.

## FAQ

### When does Crawlee call failedRequestHandler?

After a request has failed every attempt allowed by maxRequestRetries. Blocked statuses (401, 403 and 429) count as failures and make Crawlee rotate to a fresh session first; 5xx responses and network errors count too. The handler gets the request and the last error, so it can hand the URL to UpRock.

### Does this only work on the Apify platform?

No. Crawlee is open source (Apache-2.0) and the code on this page runs on your own machine. The same crawler runs unchanged as an Apify Actor.

### Can't I just use Apify's residential proxies?

You can, at $7 to $8 per GB on Apify's plans, and with a country set. Apify's docs say that if there are no proxy servers in the country you ask for, the connection fails. UpRock is a second network with one price per page, for the requests that still fail.

### What does Apify charge for a page that fails?

Nothing, on its pay-per-result Actors. RAG Web Browser's README says a page that fails to load isn't charged, and Web Fetch charges nothing for failed requests. A fallback adds one UpRock credit.

### Is UpRock in the Apify Store?

No. UpRock is its own API, so the crawler calls it directly, over MCP in the TypeScript example and through the CLI in the Python one.

## Other use cases to explore

- [Add a fallback to a Python crawler you host](https://uprock.ai/compare/crawl4ai#plug-uprock-into-fallback_fetch_function): Crawl4AI has its own fallback hook; send it the pages your server can't load.
- [Work out what a browser read really costs](https://uprock.ai/compare/browserbase#what-1000-page-reads-cost): per-minute billing, proxy megabytes and a fetch API, side by side for 1,000 pages.
- [Decide which failed pages are worth retrying](https://uprock.ai/use-cases/agent-gets-403): a diagnosis table from status code to the legitimate fix, and when to stop.
- [Search public video by moment](https://uprock.ai/video-semantic-search): find the second something is shown or said, across platforms.

## Related articles

- [How do I add a backup crawler to Firecrawl?](https://uprock.ai/compare/firecrawl.md): When to send a page from Firecrawl to UpRock, published prices side by side, and a tested fallback in TypeScript and Python.
- [How do I add a backup for Tavily Extract?](https://uprock.ai/compare/tavily.md): Retry the URLs in Tavily Extract's failed results on a real device, and read pages as they appear in a chosen country. Prices side by side and tested code.
- [How do I add a fallback for Exa contents errors?](https://uprock.ai/compare/exa.md): Read the per-URL statuses in Exa's /contents response and retry the right ones live on a real device. Error tags explained, prices side by side, tested code.
- [How do I add a backup to Jina Reader, and when is it cheaper?](https://uprock.ai/compare/jina-reader.md): Spot a failed read in Reader's JSON (it still answers 200), retry it on a real device, and see what six real pages cost in Reader tokens next to a flat per-page price.
- [How do I add a hosted fallback to Crawl4AI?](https://uprock.ai/compare/crawl4ai.md): Plug UpRock into Crawl4AI's fallback_fetch_function so blocked pages are fetched on a real device in the country you choose, then run through Crawl4AI's own markdown pipeline.
- [How do I add a backup to Bright Data's Web Unlocker?](https://uprock.ai/compare/bright-data.md): Which Web Unlocker errors to retry, which to send to a second network and which to respect, with Bright Data's published prices next to ours and tested fallback code.
- [When should a Browserbase agent read a page without opening a browser?](https://uprock.ai/compare/browserbase.md): What 1,000 page reads cost on Browserbase three ways, when to send a read to UpRock instead (including when every browser is busy), and tested fallback code.
- [How do I cap ScrapingBee's credit cost and send the rest to a backup?](https://uprock.ai/compare/scrapingbee.md): ScrapingBee charges 1 to 75 credits a page depending on the options. Cap it with Auto-Mode and max_cost, watch the country fallback, and send the rest to UpRock with tested code.
- [Spider costs less per page than UpRock. When is a backup still worth it?](https://uprock.ai/compare/spider.md): Spider is cheaper per page, so keep it as your primary. How Spider reports a page it couldn't get (from our own tests), and tested code that sends only those pages to UpRock.
- [How do I add a fallback to Parallel's Extract API?](https://uprock.ai/compare/parallel.md): Make Parallel Extract report a failed live fetch instead of quietly serving an old cached copy, then retry those URLs on UpRock. Prices side by side and tested code.
- [How do I add a backup to ZenRows?](https://uprock.ai/compare/zenrows.md): Which ZenRows error codes to send to a second network and which to respect, what a page from one country costs once premium proxies are required, and tested fallback code.
- [How do I retry a ScraperAPI batch's failed jobs on another network?](https://uprock.ai/compare/scraperapi.md): Send the jobs ScraperAPI's async batch gives up on to UpRock, see which countries each ScraperAPI plan can target, and compare credit costs. Tested code for both.
- [How do I add a backup to Oxylabs' new Web API?](https://uprock.ai/compare/oxylabs.md): Which Oxylabs Web API failures to send to a second network and which to respect, what a generic page really costs after its October 2026 pricing change, and tested fallback code.
- [How do I add a backup to Context.dev's scrape API?](https://uprock.ai/compare/context-dev.md): Context.dev reports a failed scrape inside an HTTP 200. Which of its error codes to retry on a second network, what a page costs by plan, and tested fallback code.
- [How do I add a backup to Scrapfly instead of paying for residential?](https://uprock.ai/compare/scrapfly.md): A Scrapfly page costs 1 to 30 credits depending on proxy pool and browser. What that is per 1,000 pages on each plan, which errors to send on, and tested code.
- [How do I keep Crawlbase's site tiers from surprising my bill?](https://uprock.ai/compare/crawlbase.md): Crawlbase bills each site at 1 to 20 times its base rate. What that does to the cost per 1,000 pages, and tested code that moves pricey domains to UpRock.
- [Which unlocker API fits: Bright Data, Oxylabs, Zyte or UpRock?](https://uprock.ai/compare/web-unlocker-apis.md): Bright Data's Web Unlocker, Oxylabs' Web API and Zyte API priced per 1,000 hard pages, with what each bills and when a second network helps.
- [Compare: all articles](https://uprock.ai/compare.md)

## References

1. [Apify, Pricing (plans, compute units, proxies)](https://apify.com/pricing), checked 21 Oct 2026
2. [Apify Store, RAG Web Browser pricing](https://apify.com/apify/rag-web-browser/pricing), checked 21 Oct 2026
3. [Apify Store, RAG Web Browser (README, input and output)](https://apify.com/apify/rag-web-browser), checked 21 Oct 2026
4. [Apify Store, Web Fetch](https://apify.com/apify/web-fetch), checked 21 Oct 2026
5. [Apify Store, Website Content Crawler](https://apify.com/apify/website-content-crawler), checked 21 Oct 2026
6. [Apify docs, Proxy (groups and country selection)](https://docs.apify.com/platform/proxy), checked 21 Oct 2026
7. [Apify docs, MCP server](https://docs.apify.com/platform/integrations/mcp), checked 21 Oct 2026
8. [Apify docs, API rate limits](https://docs.apify.com/api/v2), checked 21 Oct 2026
9. [Crawlee on GitHub (Apache-2.0)](https://github.com/apify/crawlee), checked 21 Oct 2026
10. [UpRock, everything in one file (API, CLI, MCP, pricing)](https://uprock.ai/llms-full.txt), checked 21 Oct 2026

## Get started

Get an API key: https://uprock.ai/app/api-keys

Paste this into your agent:

```text
Set up UpRock for me: read https://uprock.ai/agents.md, connect the UpRock MCP server, and run one test crawl. Tell me what it costs before you spend anything.
```

Setup for every client (MCP, skill, CLI): https://uprock.ai/agents.md