How do I add a fallback to an Apify or Crawlee crawler?
Use Crawlee's failedRequestHandler. Once a request has used up its retries, hand the URL to UpRock, which loads it on an opted-in device in the country you choose for $0.50 per 1,000 pages, and keep Apify for everything around it, like scheduling, storage and the Actor store.
Updated · Read as markdown
What Apify does well
Apify is a platform for running scrapers and automations, which it calls Actors. Its store lists over 94,000 of them, from Google Maps and Amazon scrapers to tools built for AI agents, and you can run any of them on a schedule, store the results in datasets, and plug them into LangChain, n8n, Zapier, Make and others. It also maintains Crawlee, an open-source crawling library for Node.js and Python (Apache-2.0), and an official MCP server that gives agents its RAG Web Browser and Web Fetch Actors out of the box.
If you build crawlers, Apify takes care of the parts around them: running, scaling, scheduling, storing. Keep it.
Where a Crawlee crawler gives up
Crawlee already tries hard before it gives up. When a site answers 401, 403 or 429, it marks the session as blocked, rotates to a fresh one and retries, up to maxRequestRetries times. Then the request is marked as failed, and the URL drops out of your results.
That last step is the place for a backup. Crawlee calls failedRequestHandler with the request and the final error. Hand the URL to UpRock there, and the page gets one more try, from a different network and from a device in the country you choose.
What a fetched page costs on Apify
If you'd rather not run a crawler at all, Apify sells page fetching through its own Actors, priced per result. These are Apify's published prices, checked 21 Oct 2026:
| How you fetch on Apify | Price per 1,000 pages | Notes |
|---|---|---|
| RAG Web Browser, Standby mode | $2.28 Free, $1.90 Starter, $1.62 Scale, $1.33 Business | Platform usage included; failed pages not charged |
| RAG Web Browser, one run per URL | About $1.80 more | A start fee of $0.000225 per GB of run memory; the default is 8 GB |
| Web Fetch | $1.50 Free and Starter, $1.25 Scale, $1.00 Business | Nothing for failed requests; 10 MB and 2-minute cap |
| Website Content Crawler | About $0.20 raw HTTP, $0.50 to $5 with a headless browser | Apify's estimate at $0.25 per compute unit; plans now list $0.13 to $0.20 |
| Any of the above, from a chosen country | Plus residential proxy traffic | $8 per GB on Free and Starter, down to $7 |
| UpRock | $0.50 by card or USDC, $0.40 with UPT | Country included; one credit per page |
Apify and UpRock side by side
| Apify | UpRock | |
|---|---|---|
| What it is | A platform for running scrapers, with a store of ready-made Actors | An API that fetches one page per call |
| Price per 1,000 pages | See the table above: $1.00 to $2.28 for Apify's fetch Actors | $0.50 by card or USDC, $0.40 with UPT |
| Scheduling, storage, datasets | Yes | No |
| Open source | Crawlee, Apache-2.0 | No. The device runtime (Fuel Engine) is |
| Where requests come from | Datacenter, residential or SERP proxy groups; how residential IPs are sourced isn't stated | Devices whose owners opted in |
| Country | Per proxy configuration; fails if there's no proxy in that country | Every request: 63 countries, or a region |
| MCP server | Official, remote, OAuth or token | Yes, remote (streamable HTTP) |
| Video | YouTube scraper (maintained by Apify) with paid transcript and per-video description add-ons | Search public video by moment, $1 per 1,000 searches |
| To try it | $5 of platform credit a month on the Free plan | Prepaid credits |
| API rate limits | Published: 60 requests a second per resource | Not published yet |
Where Apify is better: everything around the fetch, from the Actor store to scheduling, storage and integrations, plus an open-source crawler and a monthly credit to try it. Where UpRock fits: the requests your crawler gives up on, a page from a chosen country without buying proxy traffic by the gigabyte, and a lower price per page than Apify's fetch Actors.
Hand Crawlee's failed requests to UpRock
Every version crawls two URLs: one that loads normally and a test URL that always answers 403. Crawlee reads the first itself, retries the second on fresh sessions, and when it gives up, failedRequestHandler sends the URL to UpRock. The TypeScript versions use Crawlee for Node.js and the Python versions Crawlee for Python. UpRock gets one POST /v1/crawl request with your key in an Authorization: Bearer header; the MCP and CLI tabs do the same through UpRock's MCP server (TypeScript) or CLI (Python).
// A Crawlee crawler (Apify's open-source library; runs locally or as an Apify Actor)// that hands pages it couldn't get to UpRock.// npm i crawlee (Node 18+) env: UPROCK_API_KEYimport { CheerioCrawler } from "crawlee";const COUNTRY = process.argv[2] ?? "US"; // where UpRock should load the pages fromasync function viaUpRock(url: string) { // One call: UpRock loads the page on a real device in that country, tries a fresh device // every 7 s (up to 3 tries) and returns markdown. Failed calls aren't billed. const res = await fetch("https://edge.uprock.com/v1/crawl", { method: "POST", headers: { Authorization: `Bearer ${process.env.UPROCK_API_KEY}`, "Content-Type": "application/json", }, body: JSON.stringify({ url, country: COUNTRY }), signal: AbortSignal.timeout(90_000), }); const page = await res.json(); if (!res.ok) throw new Error(`UpRock ${res.status} ${page.code}: ${page.error}`); return page.markdown ?? "";}const crawler = new CheerioCrawler({ maxRequestRetries: 2, // Crawlee retries first, on a fresh session each time async requestHandler({ request, $ }) { console.log("crawlee", request.url, $("head > title").text().trim()); }, // Runs after a request has used up its retries: blocked (401, 403, 429), 5xx or a network error. async failedRequestHandler({ request }, error) { try { const markdown = await viaUpRock(request.url); console.log( "uprock ", request.url, `${markdown.length} chars (Crawlee: ${error.message.slice(0, 60)})`, ); } catch (err) { console.log("failed ", request.url, (err as Error).message); } },});// node crawl.ts DEawait crawler.run(["https://developer.mozilla.org/en-US/docs/Web/HTTP", "https://postman-echo.com/status/403"]);# A Crawlee crawler (Apify's open-source library; runs locally or as an Apify Actor)# that hands pages it couldn't get to UpRock.# pip install 'crawlee[beautifulsoup]' requests env: UPROCK_API_KEYimport asyncioimport osimport sysimport requestsfrom crawlee.crawlers import ( BasicCrawlingContext, BeautifulSoupCrawler, BeautifulSoupCrawlingContext,)COUNTRY = ( sys.argv[1] if len(sys.argv) > 1 else "US") # where UpRock should load the pages fromdef via_uprock(url: str) -> str: # One call: UpRock loads the page on a real device in that country, tries a fresh device # every 7 s (up to 3 tries) and returns markdown. Failed calls aren't billed. res = requests.post( "https://edge.uprock.com/v1/crawl", headers={"Authorization": f"Bearer {os.environ['UPROCK_API_KEY']}"}, json={"url": url, "country": COUNTRY}, timeout=90, ) page = res.json() if not res.ok: raise RuntimeError(f"UpRock {res.status_code} {page.get('code')}: {page.get('error')}") return page.get("markdown", "")async def main() -> None: crawler = BeautifulSoupCrawler(max_request_retries=2) # Crawlee retries first @crawler.router.default_handler async def handler(context: BeautifulSoupCrawlingContext) -> None: title = context.soup.title.get_text(strip=True) if context.soup.title else "" print("crawlee", context.request.url, title) # Runs after a request has used up its retries: blocked (401, 403, 429), 5xx or a network error. @crawler.failed_request_handler async def failed(context: BasicCrawlingContext, error: Exception) -> None: try: markdown = await asyncio.to_thread(via_uprock, context.request.url) print( "uprock ", context.request.url, f"{len(markdown)} chars (Crawlee: {str(error)[:60]})", ) except Exception as err: print("failed ", context.request.url, err) await crawler.run(["https://developer.mozilla.org/en-US/docs/Web/HTTP", "https://postman-echo.com/status/403"])if __name__ == "__main__": # python crawl.py DE asyncio.run(main())// A Crawlee crawler (Apify's open-source library; runs locally or as an Apify Actor)// that hands pages it couldn't get to UpRock.// npm i crawlee @modelcontextprotocol/sdk (Node 18+) env: UPROCK_API_KEYimport { CheerioCrawler } from "crawlee";import { Client } from "@modelcontextprotocol/sdk/client/index.js";import { StreamableHTTPClientTransport } from "@modelcontextprotocol/sdk/client/streamableHttp.js";const COUNTRY = process.argv[2] ?? "US"; // where UpRock should load the pages fromconst uprock = new Client({ name: "crawlee-fallback", version: "1.0.0" });await uprock.connect( new StreamableHTTPClientTransport( new URL("https://mcp.uprock.ai/mcp"), { requestInit: { headers: { Authorization: `Bearer ${process.env.UPROCK_API_KEY}` } } }, ),);const textOf = (content: unknown) => (content as { type: string; text?: string }[]).find((c) => c.type === "text")?.text ?? "";async function viaUpRock(url: string) { const res = await uprock.callTool( { name: "crawl_fetch", arguments: { url, country: COUNTRY } }, undefined, { timeout: 300_000 }, ); if (res.isError) throw new Error(`UpRock: ${textOf(res.content)}`); const uri = JSON.parse(textOf(res.content)).content?.markdown?.resource; if (!uri) throw new Error(`UpRock: no markdown for ${url}`); return textOf( (await uprock.callTool({ name: "resource_fetch", arguments: { uri } })).content, );}const crawler = new CheerioCrawler({ maxRequestRetries: 2, // Crawlee retries first, on a fresh session each time async requestHandler({ request, $ }) { console.log("crawlee", request.url, $("head > title").text().trim()); }, // Runs after a request has used up its retries: blocked (401, 403, 429), 5xx or a network error. async failedRequestHandler({ request }, error) { try { const markdown = await viaUpRock(request.url); console.log( "uprock ", request.url, `${markdown.length} chars (Crawlee: ${error.message.slice(0, 60)})`, ); } catch (err) { console.log("failed ", request.url, (err as Error).message); } },});// node crawl.ts DEawait crawler.run(["https://developer.mozilla.org/en-US/docs/Web/HTTP", "https://postman-echo.com/status/403"]);await uprock.close();# A Crawlee crawler (Apify's open-source library; runs locally or as an Apify Actor)# that hands pages it couldn't get to UpRock.# pip install 'crawlee[beautifulsoup]'; npm i -g uprock env: UPROCK_API_KEYimport asyncioimport jsonimport subprocessimport sysfrom crawlee.crawlers import ( BasicCrawlingContext, BeautifulSoupCrawler, BeautifulSoupCrawlingContext,)COUNTRY = ( sys.argv[1] if len(sys.argv) > 1 else "US") # where UpRock should load the pages fromdef via_uprock(url: str) -> str: run = subprocess.run( ["uprock", "ai", "crawl", url, "--country", COUNTRY, "--content"], capture_output=True, text=True, timeout=330, ) if run.returncode != 0: raise RuntimeError(f"UpRock: {run.stderr.strip()}") return json.loads(run.stdout).get("inlined_markdown", "")async def main() -> None: crawler = BeautifulSoupCrawler(max_request_retries=2) # Crawlee retries first @crawler.router.default_handler async def handler(context: BeautifulSoupCrawlingContext) -> None: title = context.soup.title.get_text(strip=True) if context.soup.title else "" print("crawlee", context.request.url, title) # Runs after a request has used up its retries: blocked (401, 403, 429), 5xx or a network error. @crawler.failed_request_handler async def failed(context: BasicCrawlingContext, error: Exception) -> None: try: markdown = await asyncio.to_thread(via_uprock, context.request.url) print( "uprock ", context.request.url, f"{len(markdown)} chars (Crawlee: {str(error)[:60]})", ) except Exception as err: print("failed ", context.request.url, err) await crawler.run(["https://developer.mozilla.org/en-US/docs/Web/HTTP", "https://postman-echo.com/status/403"])if __name__ == "__main__": # python crawl.py DE asyncio.run(main())Tested 21 Oct 2026 with Crawlee 3.18 (Node.js) and 1.11 (Python) and the live UpRock API, no Apify account needed: Crawlee read example.com, retried a test URL that always answers 403 on new sessions, then handed it to UpRock. On 6 Nov 2026 we ran the HTTP and MCP versions again with an UpRock API key in an Authorization header: Crawlee read the MDN page itself and handed a page that answered 403 to UpRock.
If your agent calls Apify's RAG Web Browser instead, the same idea applies to its output: a page that failed comes back with crawl.requestStatus set to failed and no markdown. Send those URLs to UpRock.
FAQ
When does Crawlee call failedRequestHandler?
After a request has failed every attempt allowed by maxRequestRetries. Blocked statuses (401, 403 and 429) count as failures and make Crawlee rotate to a fresh session first; 5xx responses and network errors count too. The handler gets the request and the last error, so it can hand the URL to UpRock.
Does this only work on the Apify platform?
No. Crawlee is open source (Apache-2.0) and the code on this page runs on your own machine. The same crawler runs unchanged as an Apify Actor.
Can't I just use Apify's residential proxies?
You can, at $7 to $8 per GB on Apify's plans, and with a country set. Apify's docs say that if there are no proxy servers in the country you ask for, the connection fails. UpRock is a second network with one price per page, for the requests that still fail.
What does Apify charge for a page that fails?
Nothing, on its pay-per-result Actors. RAG Web Browser's README says a page that fails to load isn't charged, and Web Fetch charges nothing for failed requests. A fallback adds one UpRock credit.
Is UpRock in the Apify Store?
No. UpRock is its own API, so the crawler calls it directly, over MCP in the TypeScript example and through the CLI in the Python one.
Other use cases to explore
- Add a fallback to a Python crawler you host: Crawl4AI has its own fallback hook; send it the pages your server can't load.
- Work out what a browser read really costs: per-minute billing, proxy megabytes and a fetch API, side by side for 1,000 pages.
- Decide which failed pages are worth retrying: a diagnosis table from status code to the legitimate fix, and when to stop.
- Search public video by moment: find the second something is shown or said, across platforms.
References
- Apify, Pricing (plans, compute units, proxies), checked 21 Oct 2026
- Apify Store, RAG Web Browser pricing, checked 21 Oct 2026
- Apify Store, RAG Web Browser (README, input and output), checked 21 Oct 2026
- Apify Store, Web Fetch, checked 21 Oct 2026
- Apify Store, Website Content Crawler, checked 21 Oct 2026
- Apify docs, Proxy (groups and country selection), checked 21 Oct 2026
- Apify docs, MCP server, checked 21 Oct 2026
- Apify docs, API rate limits, checked 21 Oct 2026
- Crawlee on GitHub (Apache-2.0), checked 21 Oct 2026
- UpRock, everything in one file (API, CLI, MCP, pricing), checked 21 Oct 2026
Get started
Add UpRock in one paste.
Paste this into Claude Code, Codex, Cursor or any agent with a terminal. It reads our docs, sets up the MCP server or CLI, and tells you the cost before it spends anything.
Set up UpRock for me: read https://uprock.ai/agents.md, connect the UpRock MCP server, and run one test crawl. Tell me what it costs before you spend anything.