A Benchmark Is a Photograph, Not a Measurement

By Muninder Adavelli. Scraping, SEO, and proxies for over a decade.

This is a just a thought, a by product of testing procedures I have for my opensource tools. In this case OWS - Opensource Web Scraping API

I have been seeing a lot of benchmarking stuff on the internet for everything, starting from hosting to hair dryers. BIG thanks to Claude and Codex.

But how good is that whole concept of Benchmarking?

My thought:

Not sure, if that statement applies to everything under the sun, but in quite a few industries I am familiar with benchmarking is more like a photograph with a timestamp.

It is all correct

but, with respect to time!

Especially, things related to servers, networks and computers . Industries, products or services working these things usually are multi-variate equations. And a lot of those variables are often new.

How or Why I am SAYING this?

I am in the middle of building an opensource web scraping api that anyone can set up with one terminal command and start scraping HTML of websites open on the internet.

As a part of testing my opensource web scraping api, I started with a very simple test: Scraping Wikipedia. the easiest scraphing task by design.

I started with this to just to test the curl, httpx and log them compare them to other sophisticated engines like Camoufox or Playwright.

I Know its a dead easy one. It was easy for me to with almost every engine I had in my scraper.

But the logs I had for these tests, the numbers, they were full of surprises. Atleast for my preconcieved ideas of scraping and scraping Wikipedia.

Let me paint the picture of how I did it it:

1. Date: 2026-11-03, I first ran Wikipedia multiple fetch tests with just n=3 (three times in less than a second, for every engine like camoufox). Those three times in less than second, I hit en.wikipedia.org/wiki/Web_scraping

2. Date: 2026-11-0-4, I was doing n=5 with same set up. same ip, same router, almost same internet spped.

And here are the results:

Three things jumped out at me.

Run 2 and run 3 should look like run 1. But they don't.

httpx took 1345ms, then 232ms, then 317ms.

Same site, same laptop, two days apart. That's a 5x swing in a number most benchmarks present as a fact.

My first instinct was "my internet was slow on run 1." It wasn't, really.

The gap between run 2 and run 3 is only ~85ms. That's noise.

But, the 1345ms on R1 which is 5x bigger and the possible reasons could be different Wikipedia edge node, TLS negotiation, or something else was in the way.

The browsers are the calm ones, and for a reason.

camoufox: 16070, 16272, 15871ms. cookie_replay: 20168, 16745, 16540ms. Under 1% variance across all three runs.

That pattern is the thesis of this whole article in one number: what's expensive in a browser engine is the stuff the internet isn't involved in. Booting Firefox, loading the anti-detect profile, the fingerprint model, rendering the page, waiting for JS. That work takes ~16 seconds regardless of whether you're on fiber or a bad hotel connection. Bandwidth is a rounding error next to it.

Patchright tells the same story from the other side — it's a browser that got fast (~930ms) because it skips most of that boot work, and it still delivers the rendered page (258KB, like camoufox).

So the fast engines swing with the network. The slow engines are stable because they barely care about the network. That's the observation I want to keep, but it comes with a hedge: I only ran this on Wikipedia, and only 3 to 5 samples per engine.

Pass rate is the honest column. The medians hide the failures.

Look down the primp row: 67%, 40%, 60%. The medians look fast — 383ms, 230ms. But the pass rate tells you what's actually happening: Wikipedia rate-limited us on the fast engines. 429s, three of five requests in run 2. When HTTP engines fire in a burst, the site notices, and the burst is exactly what a benchmark does.

nodriver flips the same way: 100%, 40%, 100%. Its run-2 failures were transport errors — the CDP connection dying mid-scrape — and its byte count jumped from 234KB to 417KB across runs, which I've seen happen with CDP before. The pass rate column is where every one of those scars shows up. Median latency alone would have told you nodriver is fine.

So: for a benchmark, the smallest n that still shows you thumbs down when things fail is worth more than the prettiest median.

What this means if you're scraping Wikipedia at scale

Concrete advice from 35 logged requests:

• Use httpx, expect 429s. It passed 100% every run, dropped to sub-second on warm days. But Wikipedia rate-limits bursts, and a benchmark is a burst. At 100 pages you'll hit it; at 1,000 pages you'll hit it a lot. Sleep between requests or pace your RPM.

• Don't trust your first latency number. Budget with the slow day, not the fast one. The 1345ms run and the 232ms run are both Wikipedia.

• The browsers are predictable in the worst way. camoufox at 16s/page is a dead certainty. If you need rendered HTML and not the 258KB, patchright gave us nearly the same bytes at ~40x less time.

• Every failure reason is logged — 429s, transport errors, empty responses — so the pass column isn't guesswork. That's the column to watch.

Note: I haven't run this at 100, 500, or 1,000 pages yet. That's the stress test this article is heading toward — throughput, concurrency, where the knee is.

This article is a series now

I'm treating this as a living benchmark, not a one-off. Each run in this folder (articles/data/tier-tests/) is dated and numbered. This page gets a new row when I run it again, so the trend replaces the snapshot. That's the honest shape for this kind of data — logging it, not just reporting it.

If you're scraping Wikipedia or any rate-limited site and your numbers disagree with one of mine, or you've seen the 429 behavior flip — I want to hear it. That's half the point of printing this publicly.

Next run: stress test at 100 → 500 → 1,000 pages. It drops on this page when it ships.

P.S. OWS is MIT licensed. Built in public. Every benchmark result published.