Google blocks SERP scrapers at two layers: TLS and /sorry/

By Muninder Adavelli. Scraping, SEO, and proxies for over a decade.

1. Where this picks up

In how I got past the blocks, the ending was clean enough that I thought I was mostly done. Headless Chromium, a six-line stealth init script (navigator.webdriver, window.chrome, navigator.plugins, navigator.languages), the --disable-blink-features=AutomationControlled launch flag, a homepage warm-up before the search, block images/media/fonts but not stylesheets. On my home IP that pulled a real SERP: 597 KB, 8 h3 results, status 200. Scraper working, in principle.

The thing I had not tested yet was production shape. Nobody ships a rank tracker that scrapes Google from the author's home IP. The whole point of the product is to run from residential proxies at some scale. I use Decodo as primary, 1-minute sticky TTL, the works.

So I pointed the exact same stealth-Chromium script at gate.decodo.com:10000 and expected a dip. What I got was something I'd never actually seen before.

2. TL;DR

• Stealth-Chromium that got 10/10 on my home IP went 0/3 on Decodo rotating on the same day, and not as /sorry/, as a silent hang. The TLS handshake never completed. The browser just sat there.
• I ran Firefox on the same pool, same keywords, same day. 3/3 at the TLS layer: handshake clean, 200 OK, response body returned.
• Then I actually looked at the response body. Firefox was 3/3 /sorry/. The handshake passed, the SERP did not.
• Adding a Firefox stealth init script (navigator.webdriver === false verified in page) did not move it. Still 3/3 /sorry/.
• The lesson I missed earlier: the block isn't one wall. It's at least two. Fixing the TLS fingerprint tears down layer 1 and exposes layer 2 (IP reputation on /sorry/). You cannot patch layer 2 from inside the renderer.

So what does "stealth your Chromium" actually buy you on residential? Not nothing, but less than I thought. It buys you one working layer of a two-layer stack, with no observability of the second layer until you clear the first.

3. The silent hang (and why it's not what you'd guess)

Here is what Chromium through Decodo rotating looked like on 2026-09-06. Same script that pulled 597 KB on my home IP, now with proxy=... on the browser context pointing at Decodo:

[q1] goto https://www.google.com/search?q=foxy+ai+promo+code
     ... (30s elapsed, no response, no error)
     ... (60s elapsed)
     TimeoutError: page.goto: Timeout 60000ms exceeded

Three queries, three identical timeouts. No status code. No /sorry/ redirect. No "unusual traffic" page. The connection opens, Chrome sends the TLS ClientHello, and the response never comes back.

I almost dismissed this as a proxy issue. If a residential IP is dead or rate-limited, the connection should still resolve: a 429, a reset, a 5xx, something. A silent hang that only shows up through a proxy felt like a Decodo problem.

To rule it out I ran plain curl through the same Decodo proxy at the same minute:

curl --proxy http://user:[email protected]:10000 'https://www.google.com/' -o /dev/null -w '%{http_code} %{time_total}'
200 1.2

200 in 1.2 seconds. Same proxy, same credentials, same minute. curl sailed through. Chromium hung.

That's the diff you'd ignore at your peril. The only thing changing between those two attempts was the TLS stack. curl on macOS uses SecureTransport; Chromium ships BoringSSL with a very specific extension order and cipher-suite preference. Google's edge was dropping Chromium's ClientHello and completing curl's. Same IP, same TCP endpoint, different handshake, different outcome.

The point I am making is this: Google's residential-proxy defense in 2026 doesn't necessarily respond when it decides to block you. It can just stop responding. A silent hang is a block. If you're monitoring for 403/429//sorry/ you'll miss it entirely.

4. The Firefox test, one hour, cleanest signal of the week

Firefox has a completely different TLS stack. Different cipher-suite ordering, different extension layout, different ALPN behavior, different JA3/JA4 fingerprint. If my read on the silent hang was right, Firefox should breeze through where Chromium hangs.

I copied the Chromium probe to scripts/probes/isolate_hang_firefox.py, swapped pw.chromium.launch(...) for pw.firefox.launch(...), set the UA to Firefox 122 desktop, and reran. Same three keywords, same Decodo rotating port, same minute.

[q1] goto https://www.google.com/search?q=foxy+ai+promo+code
     status=200  time=1.4s  body=92KB
[q2] goto https://www.google.com/search?q=heygen+promo+code
     status=200  time=1.1s  body=91KB
[q3] goto https://www.google.com/search?q=apify+promo+code
     status=200  time=1.3s  body=92KB

3/3 at the TLS handshake. Not a hang in sight. The response came back fast, consistent, well-shaped. That is as clean an A/B as I have run on this project: same proxy, same pool, same keywords, same day, only Chromium → Firefox changed, and the handshake behavior flipped from 0/3 to 3/3.

So the hypothesis checked out. Chromium's TLS ClientHello, coming out of a residential-proxy exit, is the specific thing Google's edge is dropping. From a home ISP the same handshake looks normal enough that it goes through. Through a residential proxy, something (maybe just that Google has seen this exact ClientHello from this exact IP range too many times) pushes it over the deny threshold.

I was almost ready to switch the whole engine to Firefox and call it a day.

Then I actually read the 92 KB response body.

5. The second wall nobody warned me about

Here is what I expected to see in a passing Firefox response: a 400-700 KB SERP with 7-10 h3 result links. Here is what I actually saw: a 92 KB page titled "Something's wrong on our end," with "unusual traffic from your computer network detected" in the body, and the final URL rewritten to https://www.google.com/sorry/index?continue=....

Three queries. Three identical /sorry/ captcha walls.

The TLS handshake had passed. The TCP connection had completed. Google had served me a response. The response just happened to be a captcha page instead of a SERP. From the Playwright side, this reads as a successful goto with status=200. From the actual-did-I-get-a-SERP side, this is 0/3.

I almost missed this. If you're grepping for errors (timeouts, non-200s, exceptions) Firefox looked like a complete win. The failure only shows up if you look at (a) the final URL after redirects or (b) whether the body contains any h3 elements. Status 200 is not a signal of success for Google scraping. It's barely a signal of anything.

OK, maybe Firefox needs its own stealth patches. The equivalent of the Chromium init script. I wrote firefox_stealth_check.py: patched navigator.webdriver to false, patched navigator.plugins and navigator.languages with Firefox-shaped values, confirmed each patch took effect in a about:blank self-check:

[self-check] navigator.webdriver = false
[self-check] navigator.plugins.length = 5
[self-check] navigator.languages = ['en-US', 'en']
[self-check] navigator.userAgent = 'Mozilla/5.0 ... Firefox/122.0'

Patches applied. Same three keywords. Same Decodo rotating port.

3/3 /sorry/.

The renderer-level patches do not touch whatever signal Google uses to decide a residential-proxy request earns a captcha. They probably matter for the subset of defenses that do read navigator.webdriver: some Cloudflare rules, DataDome, PerimeterX. On Google specifically, in 2026, through Decodo residential: not the discriminator.

Dead-on-arrival signal.

6. What's actually happening (my best read)

I don't have a packet trace from Google's edge and I never will. These are the parts I'm reasonably sure about and the parts that are still guesswork.

Reasonably sure:

• Google's /search endpoint has at least two independent gates between you and a SERP. The first is a TLS/handshake-level gate that reads some composite signal of (handshake shape + source IP range). The second is an IP-reputation / request-pattern gate that returns /sorry/ even on 200.
• Chromium's TLS ClientHello through a residential proxy exit trips the first gate. Firefox's doesn't. Nothing about navigator.webdriver or --disable-blink-features could matter at the TLS layer: those are renderer-side flags, and the handshake happens before the renderer boots. That whole class of fix lives inside the wrong wall.
• The second gate, the /sorry/ wall, is not a browser-fingerprint check in any useful sense. I know this because adding Firefox stealth patches didn't move the pass rate off 0/3. Something upstream of the renderer decided this request earned a captcha, and no amount of renderer-side cosmetic patching changes that decision.

Still guesswork:

• I think the second gate is primarily IP reputation: "we've seen too much weird traffic from this residential pool in the last window, so new requests from these IPs get /sorry/ on arrival." But I can't prove it from the client side. It could also be a request-pattern thing (shape of headers + timing + user-agent coherence). I don't know how to disambiguate without seeing Google's side of the decision.
• I don't know whether the first gate (TLS) is purely fingerprint-based (JA3/JA4) or whether it's TLS+IP joint, i.e. "Chromium handshake from residential pool X is dropped, Chromium handshake from home ISPs is fine." Session 3.2's curl_cffi probe already showed that spoofing the Chrome handshake from the server-side didn't help, so pure JA4 isn't the whole story either.

The honest version is: I can describe what I observed and have a working theory for the first wall. The second wall I only know by its name and its behavior.

Note: next thing I want to test on this axis is whether the TLS gate ever fires for Firefox under worse pool conditions. If Firefox-through-residential starts hanging silently on a bad pool day, the "Chromium bad, Firefox good" framing is actually "Chromium always bad, Firefox good until the pool is bad enough", which is a different story.

7. Where this leaves the project

Honestly, worse off than I was at the end of my first writeup.

• At the end of my first writeup, I had a stealth-Chromium script that worked on my home IP and I believed would basically work with proxies swapped in.
• At the end of this session I have a Firefox probe that gets past the TLS gate but 0/3 on the actual SERP, no renderer patch moves it, and a strong suspicion that the thing I need to fix is not inside the browser at all.

That is not progress toward "ship a working rank tracker." That is progress toward understanding what the actual problem is, which is a different and necessary step but doesn't feel the same. If you only count working features, this session was a regression. If you count "fewer wrong assumptions in my head," it was the single most useful day on the project.

A few things that fell out of this that are worth keeping even if you never scrape Google:

• If your scraper hangs silently through a proxy, suspect the TLS layer. Not the proxy, not the DNS, not the SSL cert chain, the specific ClientHello shape your HTTP library sends. Different libraries and different browsers produce measurably different handshakes, and some edge services drop handshakes they don't like without responding.
• Status 200 is not a success signal. You need a layer-7 check (specific DOM element, URL path, body-string pattern) or you will cheerfully count captcha walls as passes.
• Renderer-side stealth patches have a specific scope. They defeat defenses that read navigator.webdriver, window.chrome, navigator.plugins, etc.: the "is this browser being driven by Puppeteer/Playwright/Selenium?" family. They do not defeat defenses that read anything before the renderer boots (TLS, HTTP/2 settings, source IP, cookie state). Know which wall you're trying to fix.

8. Why this is a separate post from the fix

Because the next thing I did was switch to Firefox stealth and run a proper N=10 dashboard check through Decodo. 4/9 pass, roughly consistent with the "6-7/10 rotating ceiling" I'd measured earlier in how I got past the blocks. Not great, but a working scraper. I thought I'd found the floor.

Three days later I ran the same probe again. Same code, same pool, same keywords. 0/9.

That is the pool-burn writeup. It is about why pass rate is not actually a function of your script. It is a function of something you do not control.

If you're building the same thing and you've hit the TLS-fingerprint wall and the IP-reputation wall and want to compare notes before the next post lands, reach out. I'm especially curious whether anyone else has observed Chromium silent-hangs through residential proxies on providers other than Decodo. One pool is an anecdote; three pools is a pattern.