How do I keep my AI agent's web costs under control?
Work out what one task costs (its searches plus its page reads), multiply by your monthly volume, then cut the page reads, which are usually the bigger part of the bill. Cache pages by URL and country, cap the pages each run may fetch, and send only the pages your cheapest fetch can't get, or that need a country, to a second network. In our worked example, caching and opening fewer results cut the monthly bill by 30 to 55 percent, depending on the setup.
Updated · Read as markdown
Price one task, then multiply
An agent's web bill is two counts: searches and page reads. Price them per task, not per call, because the task is what you can count and change. Our example is a research task that runs 4 searches and opens the top 5 results of each, so 20 page reads. At 5,000 tasks a month, that's 20,000 searches and 100,000 page reads.
Your own numbers will differ, so count them from your logs. Our agent cost calculator does the same sum per user per day, with presets from 2 searches and 6 pages for a light user to 30 searches and 150 pages for a heavy one. The prices below use the same model as the calculator: each vendor's cheapest search type, the cheapest plan for the volume, and the monthly free allowance subtracted. We re-checked every price on the vendor's own page on 25 Oct 2026.
The same month on five setups
| Setup | 20,000 searches | 100,000 page reads | A month |
|---|---|---|---|
| Firecrawl search and scrape | 40,000 credits (2 per search of 10 results) | 100,000 credits | $183: Standard plan, $83 billed yearly, plus 40,000 extra credits at $2.50 per 1,000 |
| Tavily basic search and extract | 20,000 credits | 20,000 credits (1 per 5 pages) | $312: 39,000 credits after the free 1,000, at $0.008 |
| Exa lite search and text | $80 | $100 | $170, after the $10 monthly credit |
| Parallel fast search and extract | $20 | $100 | $115, after 5,000 free requests |
| Parallel search, UpRock pages | $20 | $50 ($0.50 per 1,000) | $65, after Parallel's 5,000 free requests |
What the table says:
- Page reads are most of the bill in four of the five setups. Tavily is the exception: its searches cost $8 per 1,000 and its extracts $1.60 per 1,000 pages, so searches are half its total.
- The mixed setup is cheapest here. That's real money at this volume, but not a reason to run two vendors if you don't need what the second one adds.
- Billing period matters. Firecrawl's Standard plan is $99 billed monthly, which makes its row $199.
- Others are cheaper per page at higher volume. ScraperAPI's Startup plan is $149 a month for 1,000,000 credits, and a standard page costs 1 credit: about $0.15 per 1,000 pages if you use them all, against $0.50 on UpRock (checked 25 Oct 2026). It doesn't include search, and harder sites cost more credits.
The calculator's UpRock column prices searches with UpRock's own web research, at $1 per 1,000, which would put the 20,000 searches at $20 instead of Parallel's $15. We paired UpRock pages with a separate search API here because, when we tested UpRock's web_research on 24 Oct 2026, most of its results were off-topic. Use a dedicated search API for now.
Cache, and read fewer pages
Two changes cut the page reads without touching what the agent can answer.
Cache what you've already read. Agents re-read the same pages more than you'd expect: the same docs, the same news story across tasks, the same product page in a retry. Say 30% of reads repeat within a day (measure your own rate). A cache that serves them skips the call. A provider's cache doesn't do the same job: Firecrawl's docs say cached results "still cost 1 credit per page" (checked 25 Oct 2026). Key the cache by URL and country, and give it a max age.
Open fewer results. Opening the top 3 results of each search instead of 5 cuts reads by 40%. Better still, let the model choose which results to open from their titles and snippets, and skip the page entirely when the snippet already answers.
| Page reads | Firecrawl | Tavily | Exa | Parallel | Parallel search, UpRock pages | |
|---|---|---|---|---|---|---|
| As above | 100,000 | $183 | $312 | $170 | $115 | $65 |
| Cache 30% repeat reads | 70,000 | $108 | $264 | $140 | $85 | $50 |
| Then open 3 results, not 5 | 42,000 | $83 | $219 | $112 | $57 | $36 |
That's a 30% to 55% cut, depending on the setup. Two things stand out. Firecrawl's bill drops to its plan fee and stops there, because its plans are bought in blocks: fewer reads only help once you can move down a plan. And Tavily and Exa are now mostly search spend, so cache search results too, keyed by query and country with a short max age.
Route each page to the cheapest path that works
Most pages load fine with a plain fetch or your usual crawler. A few don't: a 403, a timeout, or a page that has to be seen from a particular country. Routing means trying the cheap path first and paying for the expensive one only on those pages.
Say 5% of the 42,000 remaining reads fail on your primary or need a country. That's 2,100 pages. Sent to UpRock, they're 2,100 credits, $1.05 a month at $0.50 per 1,000. The alternative is moving every request to a pricier tier. On ZenRows, for example, a basic request is 1 credit and premium proxies are 10, or 25 with JavaScript, and choosing a country requires premium proxies (checked 25 Oct 2026). Escalating only the pages that need it keeps that multiplier on 5% of your reads instead of all of them.
Two details keep routing honest. Count a failed attempt as a page, because some providers bill it: Firecrawl charges 1 credit for a page that comes back with a 403. And don't retry a page that answered 404 or 410; it isn't there.
The wrapper: cache, page cap and router
Every version does the same three things. A cache keyed by URL and country, with a max age (24 hours by default). A cap on pages fetched per run, counting every attempt and not counting cache hits, that stops the run with an error when it's reached. And a router: your primary fetch first, UpRock when the primary fails, and UpRock directly when the page needs a country. The primary here is a plain HTTP fetch; replace it with your Firecrawl, Exa or Parallel call. UpRock gets one POST /v1/crawl request with your key in an Authorization: Bearer header; the MCP and CLI tabs do the same through UpRock's MCP server (TypeScript) or CLI (Python).
The demo uses a cap of 3 so you can see it stop: example.com (fetched), example.com again (served from the cache), a URL that always answers 403 (the primary fails and UpRock fetches it, so that one page costs two attempts), then example.org (over the cap).
The Python versions keep their cache in a SQLite file, so it survives between runs: run it twice and the second run reads example.com and the 403 page from the cache.
// Node 18+ env: UPROCK_API_KEY// A cache keyed by URL and country with a max age, a cap on pages per run, and a router:// your primary fetch first, UpRock only when it fails or the page needs a country.type Page = { via: "cache" | "primary" | "uprock"; text: string };async function viaUpRock(url: string, country?: string): Promise<string> { // One call: UpRock loads the page on a real device in that country, tries a fresh device // every 7 s (up to 3 tries) and returns markdown. Failed calls aren't billed. const res = await fetch("https://edge.uprock.com/v1/crawl", { method: "POST", headers: { Authorization: `Bearer ${process.env.UPROCK_API_KEY}`, "Content-Type": "application/json", }, body: JSON.stringify({ url, country }), // no country: any device signal: AbortSignal.timeout(90_000), }); const page = await res.json(); if (!res.ok) throw new Error(`UpRock ${res.status} ${page.code}: ${page.error}`); return page.markdown ?? "";}// Your primary: a plain fetch here. Swap in Firecrawl, Exa, Parallel or your own crawler;// it should return the page's text and throw when it can't.async function primary(url: string): Promise<string> { const res = await fetch(url, { signal: AbortSignal.timeout(30_000) }); if (!res.ok) throw new Error(`HTTP ${res.status}`); return res.text();}// Shared by every run in this process. Use Redis or a database to share it wider.const cache = new Map<string, { at: number; text: string }>();export class WebBudget { pages = 0; // pages fetched this run; cache hits are free and don't count readonly maxPages: number; readonly maxAgeMs: number; // maxPages: per run, e.g. twice what a normal task reads. // maxAgeMs: how long a cached page stays good. constructor(maxPages = 40, maxAgeMs = 24 * 3600_000) { this.maxPages = maxPages; this.maxAgeMs = maxAgeMs; } private spend() { if (this.pages >= this.maxPages) throw new Error(`page cap reached (${this.maxPages} this run)`); this.pages++; } async read(url: string, country?: string): Promise<Page> { const key = `${country ?? "any"} ${url}`; // the same URL can differ by country const hit = cache.get(key); if (hit && Date.now() - hit.at < this.maxAgeMs) return { via: "cache", text: hit.text }; let page: Page | undefined; if (!country) { // A plain fetch can't choose a country, so country pages go straight to UpRock. this.spend(); try { page = { via: "primary", text: await primary(url) }; } catch (err) { console.warn(`primary failed (${(err as Error).message}); trying UpRock`); } } if (!page) { this.spend(); page = { via: "uprock", text: await viaUpRock(url, country) }; } cache.set(key, { at: Date.now(), text: page.text }); return page; }}// node budget.ts (a cap of 3 pages, to show it stopping the run)const run = new WebBudget(3);const urls = [ "https://example.com", "https://example.com", // read again: served from the cache "https://postman-echo.com/status/403", // always 403: the primary fails, UpRock runs "https://example.org", // over the cap];try { for (const url of urls) { const page = await run.read(url); console.log(`${page.via.padEnd(7)} ${url} (${page.text.length} chars)`); }} catch (err) { console.log(`stopped: ${(err as Error).message}`);} finally { console.log(`pages fetched: ${run.pages} of ${run.maxPages}`);}# No packages to install env: UPROCK_API_KEY# A cache keyed by URL and country with a max age, a cap on pages per run, and a router:# your primary fetch first, UpRock only when it fails or the page needs a country.import jsonimport osimport sqlite3import sysimport timeimport urllib.errorimport urllib.request# Shared by every run on this machine, so a page read by one task is free for the next.CACHE = sqlite3.connect("web-cache.sqlite")CACHE.execute( "CREATE TABLE IF NOT EXISTS pages (key TEXT PRIMARY KEY, at REAL, text TEXT)")class PageCapReached(RuntimeError): passdef primary(url: str) -> str: """Your primary: a plain fetch here. Swap in Firecrawl, Exa, Parallel or your own crawler; it should return the page's text and raise when it can't.""" request = urllib.request.Request(url, headers={"User-Agent": "my-agent/1.0"}) with urllib.request.urlopen(request, timeout=30) as res: # raises on 4xx and 5xx return res.read().decode("utf-8", "replace")def via_uprock(url: str, country: str | None) -> str: # One call: UpRock loads the page on a real device (in that country, if you pass one), # tries a fresh device every 7 s (up to 3 tries) and returns markdown. Failed calls # aren't billed. body = {"url": url, "country": country} if country else {"url": url} req = urllib.request.Request( "https://edge.uprock.com/v1/crawl", data=json.dumps(body).encode(), headers={ "Authorization": f"Bearer {os.environ['UPROCK_API_KEY']}", "Content-Type": "application/json", }, ) try: with urllib.request.urlopen(req, timeout=90) as res: return json.load(res).get("markdown", "") except urllib.error.HTTPError as err: page = json.load(err) raise RuntimeError(f"UpRock {err.code} {page.get('code')}: {page.get('error')}") from Noneclass WebBudget: def __init__(self, max_pages: int = 40, max_age_s: float = 24 * 3600): self.max_pages = max_pages # per run, e.g. twice what a normal task reads self.max_age_s = max_age_s # how long a cached page stays good self.pages = 0 # pages fetched this run; cache hits are free and don't count def _spend(self) -> None: if self.pages >= self.max_pages: raise PageCapReached(f"page cap reached ({self.max_pages} this run)") self.pages += 1 def read(self, url: str, country: str | None = None) -> tuple[str, str]: key = f"{country or 'any'} {url}" # the same URL can differ by country row = CACHE.execute( "SELECT at, text FROM pages WHERE key = ?", (key,) ).fetchone() if row and time.time() - row[0] < self.max_age_s: return "cache", row[1] via, text = "uprock", None if country is None: # A plain fetch can't choose a country, so country pages go straight to UpRock. self._spend() try: via, text = "primary", primary(url) except Exception as err: # any failure of your primary print(f"primary failed ({err}); trying UpRock", file=sys.stderr) if text is None: self._spend() via, text = "uprock", via_uprock(url, country) CACHE.execute( "INSERT OR REPLACE INTO pages VALUES (?, ?, ?)", (key, time.time(), text) ) CACHE.commit() return via, textif __name__ == "__main__": # python budget.py (a cap of 3 pages, to show it stopping the run; run it twice) run = WebBudget(max_pages=3) urls = [ "https://example.com", "https://example.com", # read again: served from the cache "https://postman-echo.com/status/403", # always 403: the primary fails "https://example.org", # over the cap on the first run ] try: for url in urls: via, text = run.read(url) print(f"{via:<7} {url} ({len(text)} chars)") except RuntimeError as err: # the page cap, or UpRock failing print(f"stopped: {err}") print(f"pages fetched: {run.pages} of {run.max_pages}")// npm i @modelcontextprotocol/sdk (Node 18+) env: UPROCK_API_KEY// A cache keyed by URL and country with a max age, a cap on pages per run, and a router:// your primary fetch first, UpRock only when it fails or the page needs a country.import { Client } from "@modelcontextprotocol/sdk/client/index.js";import { StreamableHTTPClientTransport } from "@modelcontextprotocol/sdk/client/streamableHttp.js";type Page = { via: "cache" | "primary" | "uprock"; text: string };const uprock = new Client({ name: "web-budget", version: "1.0.0" });await uprock.connect( new StreamableHTTPClientTransport( new URL("https://mcp.uprock.ai/mcp"), { requestInit: { headers: { Authorization: `Bearer ${process.env.UPROCK_API_KEY}` } } }, ),);const textOf = (content: unknown) => (content as { type: string; text?: string }[]).find((c) => c.type === "text")?.text ?? "";async function viaUpRock(url: string, country?: string): Promise<string> { const res = await uprock.callTool( { name: "crawl_fetch", arguments: country ? { url, country } : { url } }, undefined, { timeout: 300_000 }, ); if (res.isError) throw new Error(`UpRock: ${textOf(res.content)}`); const uri = JSON.parse(textOf(res.content)).content?.markdown?.resource; if (!uri) throw new Error(`UpRock: no markdown for ${url}`); return textOf( (await uprock.callTool({ name: "resource_fetch", arguments: { uri } })).content, );}// Your primary: a plain fetch here. Swap in Firecrawl, Exa, Parallel or your own crawler;// it should return the page's text and throw when it can't.async function primary(url: string): Promise<string> { const res = await fetch(url, { signal: AbortSignal.timeout(30_000) }); if (!res.ok) throw new Error(`HTTP ${res.status}`); return res.text();}// Shared by every run in this process. Use Redis or a database to share it wider.const cache = new Map<string, { at: number; text: string }>();export class WebBudget { pages = 0; // pages fetched this run; cache hits are free and don't count readonly maxPages: number; readonly maxAgeMs: number; // maxPages: per run, e.g. twice what a normal task reads. // maxAgeMs: how long a cached page stays good. constructor(maxPages = 40, maxAgeMs = 24 * 3600_000) { this.maxPages = maxPages; this.maxAgeMs = maxAgeMs; } private spend() { if (this.pages >= this.maxPages) throw new Error(`page cap reached (${this.maxPages} this run)`); this.pages++; } async read(url: string, country?: string): Promise<Page> { const key = `${country ?? "any"} ${url}`; // the same URL can differ by country const hit = cache.get(key); if (hit && Date.now() - hit.at < this.maxAgeMs) return { via: "cache", text: hit.text }; let page: Page | undefined; if (!country) { // A plain fetch can't choose a country, so country pages go straight to UpRock. this.spend(); try { page = { via: "primary", text: await primary(url) }; } catch (err) { console.warn(`primary failed (${(err as Error).message}); trying UpRock`); } } if (!page) { this.spend(); page = { via: "uprock", text: await viaUpRock(url, country) }; } cache.set(key, { at: Date.now(), text: page.text }); return page; }}// node budget.ts (a cap of 3 pages, to show it stopping the run)const run = new WebBudget(3);const urls = [ "https://example.com", "https://example.com", // read again: served from the cache "https://postman-echo.com/status/403", // always 403: the primary fails, UpRock runs "https://example.org", // over the cap];try { for (const url of urls) { const page = await run.read(url); console.log(`${page.via.padEnd(7)} ${url} (${page.text.length} chars)`); }} catch (err) { console.log(`stopped: ${(err as Error).message}`);} finally { console.log(`pages fetched: ${run.pages} of ${run.maxPages}`); await uprock.close();}# npm i -g uprock env: UPROCK_API_KEY# A cache keyed by URL and country with a max age, a cap on pages per run, and a router:# your primary fetch first, UpRock only when it fails or the page needs a country.import jsonimport osimport sqlite3import subprocessimport sysimport timeimport urllib.request# Shared by every run on this machine, so a page read by one task is free for the next.CACHE = sqlite3.connect("web-cache.sqlite")CACHE.execute( "CREATE TABLE IF NOT EXISTS pages (key TEXT PRIMARY KEY, at REAL, text TEXT)")class PageCapReached(RuntimeError): passdef primary(url: str) -> str: """Your primary: a plain fetch here. Swap in Firecrawl, Exa, Parallel or your own crawler; it should return the page's text and raise when it can't.""" request = urllib.request.Request(url, headers={"User-Agent": "my-agent/1.0"}) with urllib.request.urlopen(request, timeout=30) as res: # raises on 4xx and 5xx return res.read().decode("utf-8", "replace")def via_uprock(url: str, country: str | None) -> str: cmd = ["uprock", "ai", "crawl", url, "--content"] if country: cmd += ["--country", country] run = subprocess.run(cmd, check=False, capture_output=True, text=True, timeout=330) if run.returncode != 0: # Keep the key out of logs: some CLI errors include the MCP URL, which holds it. error = run.stderr.strip().replace(os.environ["UPROCK_API_KEY"], "***") raise RuntimeError(f"UpRock: {error}") return json.loads(run.stdout).get("inlined_markdown", "")class WebBudget: def __init__(self, max_pages: int = 40, max_age_s: float = 24 * 3600): self.max_pages = max_pages # per run, e.g. twice what a normal task reads self.max_age_s = max_age_s # how long a cached page stays good self.pages = 0 # pages fetched this run; cache hits are free and don't count def _spend(self) -> None: if self.pages >= self.max_pages: raise PageCapReached(f"page cap reached ({self.max_pages} this run)") self.pages += 1 def read(self, url: str, country: str | None = None) -> tuple[str, str]: key = f"{country or 'any'} {url}" # the same URL can differ by country row = CACHE.execute( "SELECT at, text FROM pages WHERE key = ?", (key,) ).fetchone() if row and time.time() - row[0] < self.max_age_s: return "cache", row[1] via, text = "uprock", None if country is None: # A plain fetch can't choose a country, so country pages go straight to UpRock. self._spend() try: via, text = "primary", primary(url) except Exception as err: # any failure of your primary print(f"primary failed ({err}); trying UpRock", file=sys.stderr) if text is None: self._spend() via, text = "uprock", via_uprock(url, country) CACHE.execute( "INSERT OR REPLACE INTO pages VALUES (?, ?, ?)", (key, time.time(), text) ) CACHE.commit() return via, textif __name__ == "__main__": # python budget.py (a cap of 3 pages, to show it stopping the run; run it twice) run = WebBudget(max_pages=3) urls = [ "https://example.com", "https://example.com", # read again: served from the cache "https://postman-echo.com/status/403", # always 403: the primary fails "https://example.org", # over the cap on the first run ] try: for url in urls: via, text = run.read(url) print(f"{via:<7} {url} ({len(text)} chars)") except RuntimeError as err: # the page cap, or UpRock failing print(f"stopped: {err}") print(f"pages fetched: {run.pages} of {run.max_pages}")The TypeScript version printed this on its first run. The Python version printed the same, except that its warning, sent to stderr, reads "HTTP Error 403: Forbidden":
primary https://example.com (577 chars)cache https://example.com (577 chars)primary failed (HTTP 403); trying UpRockuprock https://postman-echo.com/status/403 (30 chars)stopped: page cap reached (3 this run)pages fetched: 3 of 3And the Python version's second run, with the cache from the first:
cache https://example.com (577 chars)cache https://example.com (577 chars)cache https://postman-echo.com/status/403 (30 chars)primary https://example.org (577 chars)pages fetched: 1 of 3The 403 test URL answers 403 on every network, so what UpRock brought back is that error body ({"status":403}); the test shows the routing, not a rescued page. In your own code, check the text before caching it, so an error page doesn't sit in the cache for a day.
Tested 25 Oct 2026 against the live UpRock API (MCP SDK 1.32.2, uprock CLI 0.0.37), with the code blocks copied from this page and the primary as written: both versions produced the output above, and the Python cache file served the second run. Prices were re-checked on each vendor's page the same day. We didn't test the country path (a call with a country goes straight to crawl_fetch with that country, as on our other pages) or a Firecrawl, Exa or Parallel primary, since that needs their keys. On 6 Nov 2026 we ran the HTTP and MCP versions again with an UpRock API key in an Authorization header: the blocked page went to UpRock and the run stopped at its page cap.
FAQ
Does my provider's own cache save me money?
Not always. Firecrawl serves cached pages by default (up to two days old), but its docs say cached results still cost 1 credit per page; the cache makes them faster, not cheaper. A cache in your own code skips the call, so a repeat read costs nothing.
How long should I keep a page in the cache?
As long as its content stays true for your task. Hours for news and prices, a day for most pages, a week for documentation. Put the country in the cache key, because the same URL can show different prices, languages or results in different countries.
Am I charged for pages that fail?
It depends on the provider. Firecrawl charges 1 credit for a page that comes back with an error status such as 403 or 404, and nothing when it returns no document. Tavily says you're never charged when an extraction fails. ZenRows charges only successful requests, though 404 and 410 count as successful. The wrapper on this page counts every attempt against the cap either way.
What should the page cap be?
About twice what a normal task reads, so 40 for the 20-page task in the example. Tasks that need more stop early instead of looping, and your worst month has a known ceiling of tasks times the cap.
Is UpRock the cheapest way to read pages?
Not always. At $0.50 per 1,000 pages it's below the per-page prices of the agent APIs in the example, but plain scraping APIs on large plans cost less per page, such as ScraperAPI at about $0.15 per 1,000 on its $149 plan if you use all its credits. Use the cheaper one as your primary and UpRock for the pages it can't get or that need a country.
Other use cases to explore
- Work out the cost per user: pick how active your users are and compare a month of searches and page reads across five APIs.
- Decide what to do with a failed page: which refusals to respect, when to slow down, and when a second network is the right fix.
- Cut the model tokens each page costs: HTML and markdown measured on six pages, and what the markdown dropped.
- See what a country costs on a credit-based scraper: from 1 to 25 credits a page, and how a chosen country moves you up the scale.
References
- UpRock, Agent cost calculator (the model behind the numbers), checked 25 Oct 2026
- Firecrawl, Pricing (plans and extra credits), checked 25 Oct 2026
- Firecrawl docs, Billing (search and scrape credits, error pages), checked 25 Oct 2026
- Firecrawl docs, Faster scraping (cache maxAge, cached pages still billed), checked 25 Oct 2026
- Tavily docs, API credits, checked 25 Oct 2026
- Exa, Pricing, checked 25 Oct 2026
- Parallel, Pricing, checked 25 Oct 2026
- ScraperAPI, Pricing, checked 25 Oct 2026
- ZenRows docs, Pricing (credit weights), checked 25 Oct 2026
- ZenRows docs, Proxy country (requires premium proxies), checked 25 Oct 2026
- UpRock, everything in one file (API, CLI, MCP, pricing), checked 25 Oct 2026
Get started
Add UpRock in one paste.
Paste this into Claude Code, Codex, Cursor or any agent with a terminal. It reads our docs, sets up the MCP server or CLI, and tells you the cost before it spends anything.
Set up UpRock for me: read https://uprock.ai/agents.md, connect the UpRock MCP server, and run one test crawl. Tell me what it costs before you spend anything.