I scraped Google SERPs with proxies, at Serper's price

By Muninder Adavelli. Scraping, SEO, and proxies for over a decade.

1. Why this page exists

Honestly, scraping Google is getting harder every year. The 10-results SERP shift, the /goto redirect that broke every click-through parser I'd written, the slow but steady upward creep of client-side JS challenges on /search: all of it adds up to a world where small-scale SEO curiosity costs real money. If you want to see what the SERP looks like for 500 variations of a long-tail, you are either paying Serper roughly $0.001 per query, DataForSEO roughly $0.002 per query, or you are building your own stack. For a solo SEO or a small agency, those managed-API numbers are not catastrophic but they are a tax on being curious. The moment you want to test a weird hypothesis (what does the SERP look like for 400 variants of this long-tail?), you are thinking about the invoice instead of the data.

I decided to build the open stack. The whole reason this page and the six writeups behind it exist is to publish what I learned while doing it, including the parts where I was wrong for weeks and had to walk the conclusion back. Each of the posts linked in the sidebar documents one measurement. This page is the condensed read: the whole thing in roughly ten minutes.

The point I am making is this: I am not pretending to have solved Google. I have measured a stack that reliably works on one residential-proxy provider, at one point in time, for one query shape. That stack is at parity with managed APIs on cost and well above them on the "do I own my own pipeline?" axis. There are open questions behind it, and I'll keep updating this page as I measure them.

2. TL;DR, the stack I'm running right now

• Browser: Camoufox, a patched Firefox fork with a coherent fingerprint bundle. Not stealth-Chromium, which fails silently at the TLS handshake on residential proxies.
• Mechanism: mint one Google session, park the tab, replay every subsequent query as an in-page fetch() from inside the parked tab. The parked-tab writeup is where this is measured.
• Proxy: Decodo residential, rotating port :10000. Rotating-with-a-held-connection beats both pure rotating and sticky. How I got past the blocks covers the sticky-vs-rotating finding.
• Measured pass rate on this stack: 20/20 on Decodo rotating. ~1.0s per query, ~138 KB wire per query (post-gzip, what the proxy actually bills).
• Cost per successful SERP: ~$0.0007 steady state, ~$0.0015 all-in with the one-time mint amortized over 20 queries. Serper is $0.001; we're at parity.
• What this doesn't fix: wholly-burned-pool days, where the mint itself fails and the whole run dies (the pool-burn writeup covers why that happens). Cross-provider portability is still open; I've measured this on Decodo only.

So, what is this stack actually for? Rotating residential pools on Google /search where you want to amortize one JS-challenge pass over many queries without earning a re-check. Not sticky-session providers, not datacenter pools without JS-challenge survival, not non-Google targets where a lighter approach would do.

3. What I measured, in order

The six posts linked below are the full story of how I got here. Each one is standalone-readable; this page is the shortcut.

3.1 How I got past Google's scraper blocks

The opening story. I started on plain httpx, got nowhere, tried curl_cffi with 17 different Chrome and Firefox TLS impersonation profiles, got 30/30 blocked on all of them with Google's enablejs interstitial, then pivoted to Playwright and finally pulled a real SERP on my home IP (597 KB, 8 h3 results, status 200). I thought I had a working scraper. I did not. Read this one if you want the full "what works and what doesn't" ladder across HTTP clients and browsers, and the specific stealth-Chromium recipe that gets you to a working home-IP scraper in ~30 lines of code.

3.2 Stealth-Chromium beat Google on my home IP. On residential, it didn't even get through the handshake.

What happened the moment I pointed the home-IP scraper at Decodo residential. Stealth-Chromium went 0/3: not with /sorry/, with a silent hang at the TLS handshake. Same proxy, same minute, plain curl returned a clean 200 in 1.2 seconds. The diff was the TLS stack. I then ran Firefox on the same pool and got 3/3 at the handshake layer but 3/3 /sorry/ on the actual SERP, which is the finding this post is really about: Google's /search has at least two independent walls, and switching browsers only unlocks the first one. Read this if you have a scraper that works on your home IP and are about to be very confused by what residential proxies do to it.

3.3 The honest Chromium + Decodo baseline

The measured floor of what my scraper actually ships at. 4/9 ≈ 44% pass rate on Decodo rotating :10000, with the full recipe (code block included), the day-to-day pass-rate matrix, the proxy-wire bandwidth number (~1.95 MB per query, four times the number my own logging was reporting), and the cost math that honestly loses to Serper by 15-30× at this baseline. Read this if you want the receipts on what a plain-vanilla stealth-Chromium scraper does on residential against Google, plus the git tag discipline for not accidentally shipping a regression on top.

3.4 Does humanizing a scraper actually work?

The full A/B on the single most-recommended "trick" in the scraping discourse. I added random dwells, scroll gestures, and page.mouse.move() on top of the baseline and ran it interleaved against a no-humanization control on the same keywords. Result: zero pass-rate improvement, 68% more bandwidth, 3-6× latency. The mechanism-level reason it can't work is that Google's /search decides pass/fail before any of the gestures get a chance to run, plus page.mouse.move() dispatches events with event.isTrusted === false, which any hostile detector reads as "synthetic." Read this if you've been told to "slow it down" or "make it look human" and want the data on why that doesn't move the number against Google specifically.

3.5 The pool burn I didn't see coming

The reality check that reframed the whole project for me. Six days after measuring 4/9 on Decodo rotating, I ran the exact same probe (same code, same keywords, same port) and got 0/9. Three different browser configurations on the same pool the same day all came back zero. The common denominator was the pool, not my code. This post is about why pass rate is a function of pool reputation at a point in time, not a function of your script's quality, and why "pick the best proxy provider" is probably the wrong question. Read this if you've been staring at a scraper that "worked yesterday" and wondering what you broke.

3.6 The parked-tab trick that finally worked

The mechanism change that beat all the previous failures. Camoufox, one minted identity, a parked tab, in-page fetch() replay. 20/20 pass rate on the same Decodo rotating pool that had been serving the old loop 6-7/10. One exit IP held for the whole session, ~1.0s per query, ~138 KB wire per query. The two mechanisms I'd been missing: Google's session cookies are IP-bound (you can't extract them and replay from a different process), and Decodo rotates per TCP connection, not per request. A parked tab holding one HTTP/2 connection pins the exit IP for free. Read this if you want the one thing in the whole run that actually survived its own measurement.

4. The cost math right now

With the stack above measured on Decodo rotating :10000 at $5/GB residential pricing:

Two things worth saying honestly about this table. First, the mint is a fixed cost: 3.5 MB ($0.017) per minted identity. The lever that moves the per-SERP number is how many queries one identity serves before you recycle. 20 queries → $0.0015 all-in. 100 queries → ~$0.0009. The ceiling of one identity is still an open question (see below), but at 25 queries per identity the number is already below Serper.

Second, this table is the measurement on working-pool days. The pool-burn writeup is the asterisk: on wholly-burned-pool days the mint fails and the per-SERP cost goes undefined because the denominator is zero. I don't have a clean number for what percentage of days are burned. From a sample size of two measurements six days apart, I observed one usable day and one zero day, so probably somewhere between 10% and 40% of days are burned on any given residential pool. I think. I need more data.

For comparison purposes it is still the honest version of the pitch: on a working-pool day, the self-scrape stack is at parity with Serper and roughly 3× cheaper than DataForSEO. The pitch does not survive a 50%-burn-day distribution; it does survive a 20%-burn-day distribution. The real cost per successful SERP across a month is somewhere in between, and I'll put a number on it once the long-duration probe has run.

5. The stack I'm actually running

The thing I'm running on my own keywords is a small open-source packaging of this stack: Camoufox plus minted identity plus in-page fetch, Decodo rotating by default, with the engine from the parked-tab writeup. I keep it on GitHub as openserpchecker; the domain openserpchecker.org is where it will land once the install is one command. It is not polished. I'm iterating on it in public because I'd rather have the pool-burn fraction and the cross-provider questions answered by three people running it than by me running it alone. If you want to pull it today you can. It works for me, nothing more honest to claim than that.

6. What is still open

In rough order of how much each one could change the recommendation if the data came back surprising:

• Cross-provider validation. The parked-tab mechanism depends on Decodo's per-connection rotation. If Oxylabs rotates per request instead, a held connection won't hold an IP and the mechanism collapses. If Oxylabs rotates per connection the way Decodo does, this ports cleanly. I have measured this on Decodo only. If you have Oxylabs rotating credentials and 20 minutes to run the probe (scripts/probes/minted_identity_inpage_fetch.py --provider oxylabs once the probe is generalized), that is the single most valuable data point I'm missing.
• Identity lifetime. How many queries can one parked tab serve before Google forces a reconnect or revokes the cookie? Measured lower bound: 20. Engine default: 25 with proactive recycle. Actual ceiling: unknown. Could be 100, could be 500. The lower this number, the worse the amortized cost; the higher it is, the better.
• Datacenter pool viability. If a datacenter IP (Webshare at ~$0.10/GB vs Decodo residential at $5/GB) can host a minted identity, the cost per SERP drops roughly 50×. If Google's residential-vs-datacenter discriminator fires at mint time, it doesn't. I don't know which world I live in yet.
• Pool-burn correlation across providers. If Decodo and Oxylabs burn simultaneously (because Google's abuse detection flags residential-IP ranges upstream of any single provider), multi-provider failover doesn't help. If they burn independently, it does. This matters more for ops than for the mechanism, but it is the biggest unknown in the "can I run this reliably?" question.

Note: I'll keep this page updated as each of these gets measured. New findings land in the related-posts list in the sidebar; retired results get a line in this page noting what changed.

7. Who this is for, and who it isn't

It is for SEOs tracking a few hundred to a few thousand keywords who want to own their pipeline and keep their query budget out of the Serper invoice. It is for devs building rank trackers who want a residential-proxy mechanism that survives Google's edge in 2026 and later. It is for anyone who thinks the current "pay a managed scraper API or go without" dichotomy is not the only option.

It is not for production jobs that need a 99% SLA today. The pool-burn risk is real and the mechanism doesn't fix it; it just makes the working-pool days much cheaper. It is also not for teams who would rather pay a flat managed-API bill than keep up with the mechanism's open questions. If your preference is "I will pay someone else to care about this," Serper or DataForSEO is probably the right call and nothing I've written will change your mind.

8. If you want to compare notes

Two reader types I would genuinely like to hear from. The first is anyone running a self-hosted Google scraper on a provider other than Decodo, especially Oxylabs rotating, who can report whether a held connection pins the exit IP the way it does on Decodo. The second is anyone who has measured identity lifetime past 50 queries and can tell me whether the ceiling is in the hundreds or the low thousands. Those are the two data points that would most change the recommendation. If you've got either, the repo is probably the easiest place to open an issue, or the author contact on proxy.report works too.