2 Seconds for the Rankings, 15 for the AI Overview: What It Takes to Read Google Reliably

The system, in one paragraph

A client sends one POST to serp.hilgard.cz with a query, a country and a language. A small Next.js gateway checks the API key, the quota and the rate limits, then calls my search agent over MCP — two instances of it, one on a rented server and one on a 2018 Mac mini at home, picked by whichever has fewer calls in flight, with a breaker that takes a failing instance out of rotation and one retry on the other. Each agent instance drives its own browsers, and every request leaves through the LTE proxy pool from the first part of this series, so Google sees mobile carrier IPs. The organic results, People Also Ask and ads come back in about two seconds. The AI Overview takes about fifteen, because Google renders it progressively and the service waits until the answer has stopped growing. Every result link is unwrapped from Google's redirect back to its real URL. Every outcome is classified honestly — an AI Overview Google didn't show is a different answer from one the service couldn't get, and a parser that suddenly finds zero results is an error, not an empty page. Only successful responses are billed.

That's the system. It looks like one HTTP request from the outside. The rest of this post is about why it isn't one: the architecture, the way bot detection shapes it, and ten problems from production that all have the same nasty property — when they go wrong, they produce data that looks plausible and is wrong.

Why this exists

This is the third part of a series about the unglamorous infrastructure underneath AI products. The first part was a pool of LTE modems that puts requests on mobile carrier IPs, and it named querying Google at scale as the clearest workload that pool makes possible. The second part was the parser layer, built around one rule: no silent errors. Google search is where those two layers meet the same target every day.

The people who need SERP data share one assumption: that the input is correct. A rank tracker turns it into position charts. A GEO or AI-visibility tool turns it into "how often does an AI Overview cite your brand." An SEO platform turns it into share of voice; an AI agent turns it into grounding for an answer. Each of them builds a product on top of the search results and, reasonably, doesn't re-check every field underneath. That is exactly why the failures below matter: they don't announce themselves. They show up downstream as a chart that dips, a brand that "disappears," a market that looks quieter than it is.

The architecture

Three layers, each with one job:

client
  │  POST /v1/search  (Bearer key, JSON)
  ▼
serp.hilgard.cz — Next.js gateway behind nginx
  │  keys · quotas · limits · billing
  │  MCP · least in flight · breaker · 1 retry
  ├──────▶ agent 1   rented server
  └──────▶ agent 2   2018 Mac mini at home
              │  own browsers · own profile pool
              ▼
       LTE proxy pool (part one) ──▶ Google

The gateway is deliberately boring. It owns everything that has to do with customers: keys, the free quota, per-minute and daily limits, concurrency caps, which rate a response is billed at, and the shape of the JSON. It doesn't touch Google itself. Its one piece of real logic is choosing an agent instance: it sends each call to the instance with the fewest calls in flight, treats a network error or an nginx error page with nothing behind it as "instance down," takes an instance out of rotation after a run of such failures, and retries a failed call once on the other instance. A busy instance and a dead one look different — a timeout means alive but loaded, a connection error means gone — and they're handled differently.

The agent is the part that actually reads Google. It's a single codebase deployed twice, exposed as an MCP server, with the same configuration on both instances. Each instance runs its own browsers and keeps its own pool of browser profiles, which started from the same state and have since lived separate lives. The second instance is a much weaker machine — fewer concurrent browsers, slower answers — and least-in-flight picking handles that without any special casing: a slower instance accumulates calls in flight and simply receives fewer new ones.

The proxy pool is shared. Both instances leave through the same LTE modems, and that sharing produced the most useful capacity lesson in the whole system. Adding the second instance doubled browser capacity and CPU. It did not double anything about the mobile lines. So the pace of queries towards Google is one budget shared by both instances, while concurrency and the profile reserve are planned per instance. It's the kind of thing that's obvious on a diagram and easy to get wrong in a config file, where each instance happily assumes the lines are its own.

How Google decides you're not a person

Every problem below sits on top of one fact: Google actively tries to tell automated traffic from people, and it is very good at it. Its bot detection isn't one check. It's several layers that each have to agree:

• The network. What kind of address a request comes from, who it belongs to, and how much traffic Google has already seen from it. Datacenter ranges are easy to classify. Shared mobile carrier ranges are not — more on that in section 3.
• The browser. Whether the client is a real browser and behaves like one: runs the page's JavaScript, renders it, and presents a consistent picture of itself. A raw HTTP client fails this layer immediately. An automated browser leaves smaller traces, and consistency between them matters more than any single one.
• The session. Whether the visitor has a past. A brand-new client with no cookies and no history, issuing a commercial query, looks very different from a browser that has been using Google for weeks.
• The behavior. Rate, rhythm and sequence: how fast queries arrive, how regular the spacing is, how many come from one identity, and what they look like as a pattern rather than one at a time.

The rule that falls out of this is that the weakest layer decides. A clean mobile IP doesn't help a browser that gives itself away. A perfect browser doesn't help an address sending queries at a machine's rhythm. A long-lived profile doesn't survive being used from a datacenter range. There is no single trick that solves detection.

In this system, that translates into a handful of general choices:

• real browsers rather than raw HTTP requests — AI Overviews need a full browser to render anyway, so this was never optional;
• persistent browser profiles with their own history, a separate pool per agent instance, rather than fresh stateless clients for every query;
• mobile carrier IPs from the LTE pool;
• pacing as one shared budget across both instances, because they share the same mobile lines;
• a challenged request is never an answer — it's retried or reported as a failure, never passed through as data.

What I won't publish is the specific part: the rates, the rotation, how profiles are managed and rested. That is the part that changes most often, because the other side keeps learning too. Which is, in the end, the real cost of doing this yourself — not building the layers, but keeping all of them consistent while Google moves.

Even with all of that in place, requests still get challenged. Google regularly answers proxy traffic with a CAPTCHA or its /sorry/ page instead of results. The goal of the setup above isn't zero blocks — it's a block rate low enough to be cheap, plus a system that never mistakes a block for an answer.

1. Rankings in two seconds, the AI Overview in fifteen

The first surprise is that one results page has two very different latencies.

The organic results — the ranked links, People Also Ask, the ads — are there in about two seconds. The AI Overview is not. Google renders it progressively: the answer appears and keeps growing, paragraph by paragraph and source by source, and it is complete only after roughly fifteen seconds. The service reads it only once the answer has stopped growing, so it never returns a truncated AI Overview. A fixed delay would be the wrong tool here: too short for a long answer, wasted time on a short one. The signal to wait for is the page itself going quiet.

This is the cleanest illustration of why "just download the HTML" isn't enough. What you download at any given moment is a snapshot of a page that is still being written. Read it at two seconds and you have the rankings and either no AI Overview or the opening of one. A truncated answer is worse than a missing one, because it looks complete: a sentence or two of fluent text, with the later list items missing — and with them the citations attached to the parts you didn't wait for. For an AI-visibility tool, citations are the entire metric. Counting them on a half-rendered answer gives you a number that is wrong in a way nobody will ever notice.

Once the answer is complete, there's a second job: turning a piece of interactive UI into data. The AI Overview text is interleaved with citation chips, embedded video cards and a source panel. The service strips those out of the text, returns the full answer as markdown and as plain text, and returns the citations separately — in the order Google cites them, each with its title, URL and domain, plus a per-domain roll-up that counts how many times each domain is cited and where it first appears. That roll-up is the number most AI-visibility products actually want; it's the difference between "a source was cited" and "your domain was cited three times, first in position one."

The practical consequence is that these are two different products with two different costs in time. The playground on the landing page fires the organic request and the AI Overview request in parallel, which is why the rankings render first and the answer fills in later. Through the API, a request with include_ai_overview: true returns everything together after about fifteen seconds; with false it returns the organic results alone in about two. If all you need is rankings, there's no reason to wait for an answer you'll throw away.

2. No AI Overview is not the same as being blocked

Google doesn't show an AI Overview for every query. When it doesn't, that is a genuine observation about that query in that market, and it belongs in the data: "no AI Overview here" is an answer.

A blocked request is something else entirely. When every attempt for a query lands on a CAPTCHA or on /sorry/, you haven't learned anything about the AI Overview. You have no observation at all.

The distinction sounds pedantic until you see what happens without it. A tracker that collapses both cases into "no AI Overview" produces false data with a perfectly plausible shape:

• a brand appears to drop out of AI Overviews on the day the proxies had a bad hour;
• a market looks like it has low AI Overview coverage when what it actually has is a high block rate;
• a week-over-week trend line moves because the blocking moved, not because Google did.

Nothing in that output is obviously broken. The JSON is valid, the fields are filled, the chart renders. It's simply measuring the scraper instead of Google.

So the states have to stay separate all the way to the customer, and the API makes them explicit. search_metadata.ai_overview_status is one of four values:

The billed rate comes back on every response, both as billed_as in the metadata and as an X-Billed-As header, so a client can reconcile its own usage without trusting a monthly invoice. A request that was blocked end to end doesn't come back as data at all: it's a 502 or 504 with a JSON error body, it isn't billed, and the client knows to retry.

3. The IP decides what you get to see

Of the four detection layers, the network is the one where the choice of infrastructure matters most, and here the economics from the first part apply directly.

The reputation of a shared mobile carrier IP is shaped mostly by the ordinary people using it. Carriers put large numbers of real subscribers behind the same public addresses through carrier-grade NAT, and whatever Google concludes about such an address, it concludes mostly from their behavior. Blocking it aggressively would mean blocking those people. Datacenter IPs sit at the other end of the scale: they're registered to hosting providers, they're easy to classify, and nobody's grandmother browses from them. Even the expensive tiers of commercial proxies are, in the end, recognizable as what they are: infrastructure.

That is why serp.hilgard.cz runs on my own mobile proxy infrastructure — the LTE pool from part one — rather than on rented proxy capacity. It's also a large part of why the service is cheap to run: the dominant ongoing cost of that pool is flat-rate SIM plans, not traffic billed by the gigabyte.

But it's worth being precise about what good IPs buy you. They change the odds, not the nature of the problem. Individual requests still get challenged, which is exactly why the distinction in the previous section has to exist, and why the other three layers still have to hold. A pipeline that assumes good IPs mean no blocks is one bad afternoon away from section 2's false data.

4. Every link is a Google link now

Look at the links on a current results page and you'll notice that the destination isn't what the link points to. Google wraps the targets of organic results — and the sources cited in the AI Overview — in an opaque redirect through its own /goto endpoint. The page shows you a domain; the link itself goes to Google.

For a human clicking through, this is invisible. For anyone measuring, it's fundamental. A rank tracker matches results by URL and domain. An AI-visibility tool counts which domains an AI Overview cites. If the links aren't unwrapped, every result in your dataset points to google.com and every domain-level metric collapses into one meaningless bucket. If they're unwrapped carelessly, the failure is worse: a result attributed to the wrong site, which nobody downstream has any reason to question.

So the service turns every wrapped link back into the real target URL, and internally every result carries a flag recording whether that worked. That flag has consequences. A cited source whose real URL couldn't be recovered is left out of the AI Overview's citation list rather than counted against the wrong domain — a missing citation is a known gap, a misattributed one is a silent error. The breadcrumb Google displays under a result is returned as its own field, because it's what a person sees, but it's a label, not a link. In the six-query sample further down, none of the 50 organic results came back pointing to a Google domain.

5. The parser that returns zero

As the previous part put it, a scraper that crashes is the lucky case. Google search is where that lesson is most expensive.

When Google changes the markup of the results page, a parser doesn't necessarily throw an exception. It runs, finds nothing where it used to find results, and returns an empty list. Zero results is a perfectly valid-looking answer. It's what a genuinely empty query would look like. And it flows straight into a customer's dashboard as "you lost all your rankings overnight."

For a popular query, zero organic results isn't an answer — it's a symptom. The service treats a page that rendered as a normal results page but yielded nothing as a parser failure, not as "Google found nothing," and reports it that way. It's the same rule the whole series rests on: no silent errors. For the consumer of the data, the difference is between an error you can retry or alert on and a false ranking drop recorded as fact.

Google makes this harder than most sites, because it runs layout experiments on its results page. A change doesn't necessarily arrive as one clean redesign on a Tuesday; it can arrive as a share of responses that look slightly different from the rest. A parser that handles the majority layout and silently returns nothing on the minority one has an error rate that never trips an alarm on its own.

The deeper point is about where the effort goes. Writing a parser that works on today's page is the easy part. Knowing, on every single request, whether its output is still telling the truth is the actual job — and it never ends, because the page it reads keeps changing.

6. Location: UULE is dead

For years, the standard way to ask Google for results "as seen from" a particular city was to encode the location in the uule parameter. That no longer works: Google ignores it. I last verified this on 26 October 2026, and the API doesn't accept uule codes for that reason — accepting a parameter that silently does nothing would just move the problem to the customer.

Reliable city-level results now require something heavier. A located request runs in a dedicated browser session that reports the requested position to Google as the device's own location. The client names a place — "Austin,Texas,United States", or "Praha,Czechia" when it's unambiguous, or raw coordinates — and an ambiguous name comes back as a 400 listing the candidates, rather than a guess. All of that costs time: a located request takes roughly 10 to 40 seconds longer for the organic results and up to a minute for the AI Overview, and it has its own, much smaller rate limit.

And it still isn't guaranteed. Google can decide not to use the location it was given. So the result is verified every time against where Google itself says it searched from, and the response carries both sides — the requested place, the resolved coordinates, and what Google reported. If Google didn't apply the location, the API returns 501 location_unavailable. The alternative — returning results for some other place, labeled as the city the customer asked for — is the worst kind of output: geographically wrong, internally consistent, and indistinguishable from the real thing.

7. Ads follow the IP, not the country parameter

Ads on a results page are targeted by the exit IP of the request, not by the gl country parameter. A query with gl=us sent from a Czech IP typically returns zero ads.

That makes "0 ads" one of the most misleading values in search data. It doesn't mean nobody is bidding on the keyword. It may only mean the request came from the wrong country for the advertisers' targeting. For competitive PPC intelligence — who advertises on which terms, in which market — the exit IP has to match the market being measured. Otherwise the ads column is systematically empty, and an empty column looks like a finding.

8. AI Mode is not AI Overview

It's tempting to treat Google's generative surfaces as one thing. They aren't. AI Mode and the AI Overview are different products, and they cite differently.

A concrete example from the Czech market (gl=cz): AI Mode has returned answers with no sources at all, while the AI Overview cites around nine. AI Mode also tends to get blocked more often than the regular results page, so a pipeline that reads both sees noticeably different failure rates on each.

For anyone building AI-visibility metrics, the implication is simple and easy to miss: be explicit about which surface you're measuring. A brand's citation rate in AI Overviews says little about its presence in AI Mode, and the reverse is just as true. A number that silently mixes the two describes neither.

9. num=100 is gone

For a long time, rank trackers relied on a URL parameter, num=100, that asked Google for 100 results on a single page. One request gave you the whole top 100. In September 2025 Google stopped honoring it.

Since then, a deeper position means more separate requests: the first page, then the next, then the next, up to ten pages for what used to be one. That raised the cost of rank tracking for the entire market at once. Nothing about the data became more valuable; it simply became several times more expensive to collect.

Two consequences follow. Depth is now a real cost decision rather than a free default — tracking positions 50 to 100 for every keyword is something to choose deliberately. And pagination multiplies every other problem on this list: each extra page is another request to pace, another chance to be challenged, another response to validate, another place for a silent failure to hide. The API reports has_next_page on every response, so a client can decide whether the next request is worth paying for.

10. Who pays for a failed request

Every problem above has a price. A CAPTCHA, a retry, a request that times out, a located session that Google ignored — each one consumes mobile bandwidth, browser time and waiting, and produces nothing a customer can use. Someone pays for that.

If a provider bills per attempt, the customer pays for the provider's block rate, and the provider has no particular incentive to bring it down. That's the arrangement I wanted to avoid. serp.hilgard.cz bills only successful requests — HTTP 200 with data. Blocked requests, failed attempts, timeouts and retries are on me. The AI Overview rate applies only when an AI Overview is actually delivered; if Google shows none, or the extraction fails, the request is billed at the organic rate.

That isn't generosity, it's alignment. The cost of failure sits with the only party that can reduce it — and it's the reason every layer above is worth the effort. Each block avoided, each retry not needed, each parser failure caught early is cost that never reaches the price.

The numbers

A small sample, run on 29 October 2026 through the public playground — six queries across four markets, all with the AI Overview requested. Latencies are the ones the API itself reports:

Sources are the citations in the AI Overview, Domains the distinct domains behind them. Every query also returned 7 to 9 organic results (50 in total), and five of the six came with four People Also Ask questions.

A few things worth reading out of it:

• The two-latency shape holds everywhere. Organic results came back in 1.8 to 2.5 seconds in five of six queries; the AI Overview took 15.3 to 18 seconds every time, across all four markets.
• Tails exist. One organic request took 15.3 seconds. That's the reason the timeouts on the client side should be sized for the tail, not the median, and the reason a slow answer has to stay distinguishable from no answer.
• Sources and domains are different metrics. For "best project management software," five cited sources came from just two domains; for "project management software," fourteen sources came from thirteen. Counting sources without the per-domain roll-up would make the first query look far more diverse than it is.
• AI Overviews move. The same query, "best project management tools," cited nine sources from seven domains on 26 October and six sources from three domains on 29 October. An AI-visibility metric measured once is a snapshot, not a fact.
• No Google links leaked. None of the 50 organic results in the sample pointed to a Google domain — every one came back as its real target.

Six queries is a sample, not a benchmark. But it's real, and you can reproduce any row of it yourself in the playground.

The weekly job nobody budgets for

None of these ten problems is solved once. Google changes the markup of the results page, the way it renders AI Overviews, the redirect wrapper around links, what it does with location, which parameters it honors, and how it decides who is a person. Each change shows up first not as an error but as data that is subtly wrong — and that is the part that makes it expensive. You don't just have to fix things; you have to notice that they broke, on every request, every week.

What makes the service cheap and reliable isn't any single one of the layers above. It's that all of them exist at once and are owned by one person who runs the proxies, the browsers and the parsers end to end. The mobile pool keeps the cost of the network layer flat. The two agent instances and the gateway's failover keep it available. The detection layers are kept consistent with each other. The status model keeps every failure visible. And billing only on HTTP 200 keeps the incentive pointed the right way.

Building this yourself is a legitimate choice. If search data is core to your product, if your volume is large enough that owning the infrastructure pays for itself, and if someone on the team can treat it as their actual job, the economics from part one can work in your favor. I built it myself for exactly those reasons.

For everyone else, the question is simpler: is reading Google your product, or an input to it? If it's an input, the rational move is to buy that input from someone whose job is to keep it correct.

See it run

That's what serp.hilgard.cz is: everything above, behind one API call. The landing page has a live playground — type a query, pick a country and language, and watch it run against live Google. A request looks like this:

curl -X POST https://serp.hilgard.cz/v1/search \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "best enterprise crm",
    "country": "us",
    "include_ai_overview": true
  }'

In short:

• Organic results, People Also Ask and ads as structured JSON in about 2 seconds.
• The complete AI Overview — the full answer as markdown, every cited source, and a per-domain roll-up of the citations — in about 15 seconds.
• Own mobile proxy infrastructure, the LTE pool from part one.
• You pay only for HTTP 200. Blocked requests, failed attempts and retries are never billed.
• 5,000 free test requests, no credit card.
• From $0.80 per 1,000 queries for organic results and $1.20 per 1,000 including the AI Overview, cheaper at higher volume.

If you're building a rank tracker, a GEO or AI-visibility tool, an SEO platform or an agent that grounds its answers in search, run a few of your own queries in the playground before you decide anything. The thing to watch isn't only the speed. It's whether the answer you get is complete, and whether the API tells you when it isn't.

This is the third part of a series on production infrastructure for AI products. The first was the LTE proxy pool that gets requests through; the second was the self-healing parser and its rule of no silent errors. This one is where both of them meet Google search.