How do I retry a ScraperAPI batch's failed jobs on another network?
Submit your URLs to ScraperAPI's async batch endpoint as usual, and when its own retries give up on a job, or it's still running at your deadline, send that URL to UpRock. Country matters too, because ScraperAPI's two cheapest plans can target only the US or the EU, not individual countries, while UpRock takes any of 63 countries on every request.
Updated · Read as markdown
What ScraperAPI does well
ScraperAPI is a scraping API that handles proxies, retries and rendering behind one endpoint. Beyond single pages, it has an async API that keeps working on a job for up to 24 hours, batches of up to 50,000 URLs, webhooks, structured-data endpoints for Amazon, Google, Walmart and others, DataPipeline for scheduled runs, a crawler, an AI parser, markdown and text output, and rendering instructions for clicks and scrolls. It integrates with LangChain, LlamaIndex, n8n and Zapier and has an official MCP server.
For large batches you can hand off and collect later, ScraperAPI is built for the job. Keep it.
Where a batch leaves work undone
ScraperAPI retries each request itself for up to 70 seconds before giving up with a 500, and its FAQ says to expect 1 to 3% of requests to fail anyway. In an async batch, those show up as jobs that end as failed, or that finish with a non-2xx statusCode. Some jobs also just take a long time: the async API will keep going for up to 24 hours.
A fallback closes both gaps. Poll each job's statusUrl (no key needed), keep the ones that finished with a 2xx, skip 404 and 410 (the site's answer, and billed), and send the rest, plus anything still running at your deadline, to UpRock.
What each plan can target, and what it costs
ScraperAPI bills in credits, and what a country costs depends on your plan (checked 21 Oct 2026):
| Plan | Price, credits | Geotargeting | Plain page | Rendered | Premium + rendered |
|---|---|---|---|---|---|
| Hobby | $49, 100,000 | US and EU only | $0.49 | $4.90 | $12.25 |
| Startup | $149, 1 million | US and EU only | $0.15 | $1.49 | $3.73 |
| Business | $299, 3 million | 68 countries | $0.10 | $1.00 | $2.49 |
| Advanced (largest self-serve) | $1,975, 21.5 million | 68 countries | $0.09 | $0.92 | $2.30 |
| Premium geotargeting | 155 more countries, with premium=true (10 credits) | ||||
| UpRock | 63 countries, or a region, every request | $0.50 | $0.50 | $0.50 |
Prices are per 1,000 pages. A plain request is 1 credit, rendering or premium proxies 10, both 25, and ultra premium with rendering 75. Some domains cost more whatever you choose: e-commerce sites 5 credits, Google and Bing 25, LinkedIn 30. Unused credits don't roll over.
ScraperAPI and UpRock side by side
| ScraperAPI | UpRock | |
|---|---|---|
| Price per 1,000 pages | $0.09 to $36.75 depending on plan, options and domain | $0.50 by card or USDC, $0.40 with UPT |
| Failed requests | Not charged (200 and 404 are) | Not documented yet |
| Async batches | Up to 50,000 URLs, worked on for up to 24 hours, webhooks | No. One page per call |
| Country | US and EU only on Hobby and Startup; 68 countries from Business up; 155 more with premium | Every request: 63 countries, or a region |
| Where requests come from | Its proxy pools; premium is a private pool of residential and mobile IPs | Devices whose owners opted in |
| Structured data, crawler, AI parser | Yes | No |
| MCP server | Official, hosted or local | Yes, remote (streamable HTTP) |
| Video | No (YouTube pages as text) | Search public video by moment, $1 per 1,000 searches |
| To try it | 5,000 credits over 7 days, then a free plan with 1,000 a month | Prepaid credits |
Where ScraperAPI is better: big async batches with webhooks, structured data for major sites, failed requests that cost nothing, very cheap plain pages on bigger plans, and a free trial. Where UpRock fits: the jobs a batch gives up on or takes too long with, a specific country on any plan, rendered pages at one flat price, and video.
The fallback, in TypeScript and Python
Every version submit a batch to async.scraperapi.com/batchjobs with markdown output, poll each job every five seconds until a deadline (two minutes here), keep the finished 2xx results, skip 404 and 410, and send every other URL to UpRock in parallel. If the batch can't be submitted at all (a bad key, no credits, too many requests), every URL goes to UpRock. They call ScraperAPI's HTTP API directly, because its Node SDK has no async methods and neither SDK has a batch method. UpRock gets one POST /v1/crawl request with your key in an Authorization: Bearer header; the MCP and CLI tabs do the same through UpRock's MCP server (TypeScript) or CLI (Python).
// Submit a batch to ScraperAPI's async API, then send the jobs that failed (or are still// running at the deadline) to UpRock.// Node 18+ env: SCRAPERAPI_KEY, UPROCK_API_KEYconst ASYNC = "https://async.scraperapi.com";type Job = { id: string; status: string; statusUrl: string; url: string; response?: { statusCode: number; body: string };};async function viaUpRock(url: string, country: string) { // One call: UpRock loads the page on a real device in that country, tries a fresh device // every 7 s (up to 3 tries) and returns markdown. Failed calls aren't billed. const res = await fetch("https://edge.uprock.com/v1/crawl", { method: "POST", headers: { Authorization: `Bearer ${process.env.UPROCK_API_KEY}`, "Content-Type": "application/json", }, body: JSON.stringify({ url, country }), signal: AbortSignal.timeout(90_000), }); const page = await res.json(); if (!res.ok) throw new Error(`UpRock ${res.status} ${page.code}: ${page.error}`); return page.markdown ?? "";}async function submitBatch(urls: string[]): Promise<Job[]> { const res = await fetch(`${ASYNC}/batchjobs`, { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ apiKey: process.env.SCRAPERAPI_KEY, urls, apiParams: { output_format: "markdown" }, }), }); // 401 bad key, 403 out of credits, 429 too many concurrent requests. if (!res.ok) throw new Error(`ScraperAPI ${res.status}: ${(await res.text()).slice(0, 200)}`); return res.json();}export async function scrapeAll(urls: string[], country = "US", waitMs = 120_000) { const pages = new Map<string, { via: string; markdown: string }>(); let pending: Job[] = []; try { pending = await submitBatch(urls); } catch (err) { console.warn(`${(err as Error).message}; sending every URL to UpRock`); } const retry: string[] = pending.length ? [] : [...urls]; const deadline = Date.now() + waitMs; while (pending.length && Date.now() < deadline) { await new Promise((r) => setTimeout(r, 5_000)); const still: Job[] = []; for (const job of pending) { const state: Job = await (await fetch(job.statusUrl)).json(); // the status URL needs no key const code = state.response?.statusCode ?? 0; if (state.status === "running") still.push(job); else if (state.status === "finished" && code >= 200 && code < 300) { pages.set(job.url, { via: "scraperapi", markdown: state.response!.body }); } else if (code === 404 || code === 410) { console.warn(`${job.url}: the page is gone (${code})`); // billed, and not worth a retry } else retry.push(job.url); // failed after ScraperAPI's own retries } pending = still; } retry.push(...pending.map((j) => j.url)); // still running at the deadline const retried = await Promise.allSettled(retry.map((u) => viaUpRock(u, country))); retried.forEach((r, i) => { if (r.status === "fulfilled") pages.set(retry[i], { via: "uprock", markdown: r.value }); else console.warn(`UpRock also failed for ${retry[i]}: ${r.reason}`); }); return pages;}// node batch.ts DE https://developer.mozilla.org/en-US/docs/Web/HTTP https://example.orgconst [country = "US", ...urls] = process.argv.slice(2);for (const [url, page] of await scrapeAll(urls, country)) { console.log(page.via.padEnd(10), url, page.markdown.length, "chars");}# Submit a batch to ScraperAPI's async API, then send the jobs that failed (or are still# running at the deadline) to UpRock.# pip install requests env: SCRAPERAPI_KEY, UPROCK_API_KEYimport osimport sysimport timefrom concurrent.futures import ThreadPoolExecutorimport requestsASYNC = "https://async.scraperapi.com"def via_uprock(url: str, country: str) -> str: # One call: UpRock loads the page on a real device in that country, tries a fresh device # every 7 s (up to 3 tries) and returns markdown. Failed calls aren't billed. res = requests.post( "https://edge.uprock.com/v1/crawl", headers={"Authorization": f"Bearer {os.environ['UPROCK_API_KEY']}"}, json={"url": url, "country": country}, timeout=90, ) page = res.json() if not res.ok: raise RuntimeError(f"UpRock {res.status_code} {page.get('code')}: {page.get('error')}") return page.get("markdown", "")def submit_batch(urls: list[str]) -> list[dict]: res = requests.post( f"{ASYNC}/batchjobs", timeout=60, json={ "apiKey": os.environ.get("SCRAPERAPI_KEY"), "urls": urls, "apiParams": {"output_format": "markdown"}, }, ) # 401 bad key, 403 out of credits, 429 too many concurrent requests. if not res.ok: raise RuntimeError(f"ScraperAPI {res.status_code}: {res.text[:200]}") return res.json()def scrape_all( urls: list[str], country: str = "US", wait_s: int = 120) -> dict[str, dict]: pages, retry, pending = {}, [], [] try: pending = submit_batch(urls) except Exception as err: print(f"{err}; sending every URL to UpRock", file=sys.stderr) retry = list(urls) deadline = time.time() + wait_s while pending and time.time() < deadline: time.sleep(5) still = [] for job in pending: state = requests.get( job["statusUrl"], timeout=30 ).json() # the status URL needs no key code = (state.get("response") or {}).get("statusCode", 0) if state["status"] == "running": still.append(job) elif state["status"] == "finished" and 200 <= code < 300: pages[job["url"]] = { "via": "scraperapi", "markdown": state["response"]["body"], } elif code in (404, 410): print( f"{job['url']}: the page is gone ({code})", file=sys.stderr ) # billed else: retry.append(job["url"]) # failed after ScraperAPI's own retries pending = still retry += [job["url"] for job in pending] # still running at the deadline with ThreadPoolExecutor(max_workers=8) as pool: futures = {u: pool.submit(via_uprock, u, country) for u in retry} for url, future in futures.items(): try: pages[url] = {"via": "uprock", "markdown": future.result()} except Exception as err: print(f"UpRock also failed for {url}: {err}", file=sys.stderr) return pagesif __name__ == "__main__": # python batch.py DE https://developer.mozilla.org/en-US/docs/Web/HTTP https://example.org country, *urls = sys.argv[1:] or ["US"] for url, page in scrape_all(urls, country).items(): print(page["via"].ljust(10), url, len(page["markdown"]), "chars")// Submit a batch to ScraperAPI's async API, then send the jobs that failed (or are still// running at the deadline) to UpRock.// npm i @modelcontextprotocol/sdk (Node 18+) env: SCRAPERAPI_KEY, UPROCK_API_KEYimport { Client } from "@modelcontextprotocol/sdk/client/index.js";import { StreamableHTTPClientTransport } from "@modelcontextprotocol/sdk/client/streamableHttp.js";const ASYNC = "https://async.scraperapi.com";const uprock = new Client({ name: "scraperapi-fallback", version: "1.0.0" });await uprock.connect( new StreamableHTTPClientTransport( new URL("https://mcp.uprock.ai/mcp"), { requestInit: { headers: { Authorization: `Bearer ${process.env.UPROCK_API_KEY}` } } }, ),);type Job = { id: string; status: string; statusUrl: string; url: string; response?: { statusCode: number; body: string };};const textOf = (content: unknown) => (content as { type: string; text?: string }[]).find((c) => c.type === "text")?.text ?? "";async function viaUpRock(url: string, country: string) { const res = await uprock.callTool( { name: "crawl_fetch", arguments: { url, country } }, undefined, { timeout: 300_000 }, ); if (res.isError) throw new Error(`UpRock: ${textOf(res.content)}`); const uri = JSON.parse(textOf(res.content)).content?.markdown?.resource; if (!uri) throw new Error(`UpRock: no markdown for ${url}`); return textOf( (await uprock.callTool({ name: "resource_fetch", arguments: { uri } })).content, );}async function submitBatch(urls: string[]): Promise<Job[]> { const res = await fetch(`${ASYNC}/batchjobs`, { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ apiKey: process.env.SCRAPERAPI_KEY, urls, apiParams: { output_format: "markdown" }, }), }); // 401 bad key, 403 out of credits, 429 too many concurrent requests. if (!res.ok) throw new Error(`ScraperAPI ${res.status}: ${(await res.text()).slice(0, 200)}`); return res.json();}export async function scrapeAll(urls: string[], country = "US", waitMs = 120_000) { const pages = new Map<string, { via: string; markdown: string }>(); let pending: Job[] = []; try { pending = await submitBatch(urls); } catch (err) { console.warn(`${(err as Error).message}; sending every URL to UpRock`); } const retry: string[] = pending.length ? [] : [...urls]; const deadline = Date.now() + waitMs; while (pending.length && Date.now() < deadline) { await new Promise((r) => setTimeout(r, 5_000)); const still: Job[] = []; for (const job of pending) { const state: Job = await (await fetch(job.statusUrl)).json(); // the status URL needs no key const code = state.response?.statusCode ?? 0; if (state.status === "running") still.push(job); else if (state.status === "finished" && code >= 200 && code < 300) { pages.set(job.url, { via: "scraperapi", markdown: state.response!.body }); } else if (code === 404 || code === 410) { console.warn(`${job.url}: the page is gone (${code})`); // billed, and not worth a retry } else retry.push(job.url); // failed after ScraperAPI's own retries } pending = still; } retry.push(...pending.map((j) => j.url)); // still running at the deadline const retried = await Promise.allSettled(retry.map((u) => viaUpRock(u, country))); retried.forEach((r, i) => { if (r.status === "fulfilled") pages.set(retry[i], { via: "uprock", markdown: r.value }); else console.warn(`UpRock also failed for ${retry[i]}: ${r.reason}`); }); return pages;}// node batch.ts DE https://developer.mozilla.org/en-US/docs/Web/HTTP https://example.orgconst [country = "US", ...urls] = process.argv.slice(2);try { for (const [url, page] of await scrapeAll(urls, country)) { console.log(page.via.padEnd(10), url, page.markdown.length, "chars"); }} finally { await uprock.close();}# Submit a batch to ScraperAPI's async API, then send the jobs that failed (or are still# running at the deadline) to UpRock.# pip install requests; npm i -g uprock env: SCRAPERAPI_KEY, UPROCK_API_KEYimport jsonimport osimport subprocessimport sysimport timefrom concurrent.futures import ThreadPoolExecutorimport requestsASYNC = "https://async.scraperapi.com"def via_uprock(url: str, country: str) -> str: run = subprocess.run( ["uprock", "ai", "crawl", url, "--country", country, "--content"], capture_output=True, text=True, timeout=330, ) if run.returncode != 0: raise RuntimeError(f"UpRock: {run.stderr.strip()}") return json.loads(run.stdout).get("inlined_markdown", "")def submit_batch(urls: list[str]) -> list[dict]: res = requests.post( f"{ASYNC}/batchjobs", timeout=60, json={ "apiKey": os.environ.get("SCRAPERAPI_KEY"), "urls": urls, "apiParams": {"output_format": "markdown"}, }, ) # 401 bad key, 403 out of credits, 429 too many concurrent requests. if not res.ok: raise RuntimeError(f"ScraperAPI {res.status_code}: {res.text[:200]}") return res.json()def scrape_all( urls: list[str], country: str = "US", wait_s: int = 120) -> dict[str, dict]: pages, retry, pending = {}, [], [] try: pending = submit_batch(urls) except Exception as err: print(f"{err}; sending every URL to UpRock", file=sys.stderr) retry = list(urls) deadline = time.time() + wait_s while pending and time.time() < deadline: time.sleep(5) still = [] for job in pending: state = requests.get( job["statusUrl"], timeout=30 ).json() # the status URL needs no key code = (state.get("response") or {}).get("statusCode", 0) if state["status"] == "running": still.append(job) elif state["status"] == "finished" and 200 <= code < 300: pages[job["url"]] = { "via": "scraperapi", "markdown": state["response"]["body"], } elif code in (404, 410): print( f"{job['url']}: the page is gone ({code})", file=sys.stderr ) # billed else: retry.append(job["url"]) # failed after ScraperAPI's own retries pending = still retry += [job["url"] for job in pending] # still running at the deadline with ThreadPoolExecutor(max_workers=8) as pool: futures = {u: pool.submit(via_uprock, u, country) for u in retry} for url, future in futures.items(): try: pages[url] = {"via": "uprock", "markdown": future.result()} except Exception as err: print(f"UpRock also failed for {url}: {err}", file=sys.stderr) return pagesif __name__ == "__main__": # python batch.py DE https://developer.mozilla.org/en-US/docs/Web/HTTP https://example.org country, *urls = sys.argv[1:] or ["US"] for url, page in scrape_all(urls, country).items(): print(page["via"].ljust(10), url, len(page["markdown"]), "chars")Tested 21 Oct 2026 against the live UpRock API with an invalid ScraperAPI key: the batch endpoint answered 401, and both versions sent every URL to UpRock. Without an account, the polling path follows ScraperAPI's documented job format. On 6 Nov 2026 we ran the HTTP and MCP versions again with an UpRock API key in an Authorization header: with an invalid ScraperAPI key, each fell back to UpRock and got the page.
FAQ
How often do ScraperAPI requests fail?
ScraperAPI's own FAQ says to expect 1 to 3% of requests to fail. That's after its automatic retries, which run for up to 70 seconds before it answers with a 500. Those are the requests a fallback is for.
Does ScraperAPI charge for failed requests?
No. Its pricing page says you're charged only for successful responses (200 and 404) and for requests you cancel after 70 seconds. Its terms add that requests to sites unreachable because of geo-restrictions are chargeable.
Which countries can ScraperAPI target?
It depends on the plan. Hobby and Startup support only US and EU targeting, not individual countries. Business and higher get 68 country codes, and premium geotargeting adds 155 more but needs premium=true, which costs 10 credits a request.
Why use the async batch API at all?
For lots of URLs, it does the waiting for you. ScraperAPI keeps working on each job for up to 24 hours, takes up to 50,000 URLs per batch and stores results for at least 24 hours. The fallback on this page adds a deadline, so jobs that are still running don't hold your pipeline up.
Does ScraperAPI have an MCP server?
Yes, an official one, hosted at mcp.scraperapi.com/mcp or installed locally.
Other use cases to explore
- Add a fallback to a crawler you run: hand the requests Crawlee gives up on to a device in the country you choose.
- See what a country costs elsewhere: another provider where choosing a country means paying for premium proxies.
- Estimate what 1,000 pages will cost: token counts for six real pages, and the length where a flat per-page price is cheaper.
- Search public video by moment: find the second something is shown or said, across platforms.
References
- ScraperAPI, Pricing (plans, credit costs, billing rules), checked 21 Oct 2026
- ScraperAPI docs, Credits and request costs, checked 21 Oct 2026
- ScraperAPI docs, Async batch requests, checked 21 Oct 2026
- ScraperAPI docs, Async job handling, checked 21 Oct 2026
- ScraperAPI docs, Standard geotargeting, checked 21 Oct 2026
- ScraperAPI docs, Premium geotargeting, checked 21 Oct 2026
- ScraperAPI docs, API status codes, checked 21 Oct 2026
- ScraperAPI docs, General FAQ, checked 21 Oct 2026
- ScraperAPI docs, MCP server, checked 21 Oct 2026
- UpRock, everything in one file (API, CLI, MCP, pricing), checked 21 Oct 2026
Get started
Add UpRock in one paste.
Paste this into Claude Code, Codex, Cursor or any agent with a terminal. It reads our docs, sets up the MCP server or CLI, and tells you the cost before it spends anything.
Set up UpRock for me: read https://uprock.ai/agents.md, connect the UpRock MCP server, and run one test crawl. Tell me what it costs before you spend anything.