Editorial · The Shift · May 2026

Your Buyer Isn't on the Engine
You're Ranking On.

The Multi-Index Problem, and why “we rank #1 on Google” is no longer the proxy for visibility it used to be.

For two decades, marketing directors have measured visibility in one place. Google. The dashboard, the rank tracker, the agency report; all pointing at one engine, one map, one truth. That world held until late 2024. It does not hold now. Your buyer is on three engines, and you are ranked on one.

indexes power the major AI assistants (Google, Bing, and Brave) and they do not agree on who you are
Anthropic, OpenAI, Perplexity crawler documentation, Feb 2026
0
dashboards in most marketing directors' weekly reporting cover more than one of those three indexes
Friction & Toil audit base, Q1 2026
3
tiers of crawler sit underneath most major AI assistants (training, search indexing, live retrieval) and most agency-set robots.txt files do not separate them correctly (Claude, ChatGPT, and Gemini run all three; Perplexity runs two)
Friction & Toil audit base, Q1 2026
The engine you are ranked on, and the engine your buyer is using, have quietly stopped being the same engine.
I

The map you've been reading isn't the territory.#

In one line Your agency dashboard tracks one engine. Your buyer is on three.

A marketing director at a heritage hotel group runs a quarterly report. The agency dashboard is reassuring. #1 for “luxury Cotswolds country house hotel.” #2 for “boutique hotel Oxfordshire.” Top three on every commercial query that matters. The board nods. The retainer is renewed.

Three weeks later, an enquiry from a guest planning an anniversary mentions, in passing, that they had asked ChatGPT for a shortlist. The hotel was not on it. The marketing director makes a note to ask the agency. The agency replies that ChatGPT isn't in their tracking remit; it's “an emerging channel.” The conversation ends.

What the marketing director has just brushed against, without naming it, is the central change in commercial visibility since PageRank. The shortlist their buyer is being given is not generated from the index they have been ranking on. It is generated from a different index entirely. The rankings on the agency dashboard are real. They are also, increasingly, irrelevant to the conversation that decides whether the buyer ever hears the brand's name.

This is the Multi-Index Problem. And almost no UK marketing leader has been shown the shape of it yet.

II

One buyer. Three engines. Three indexes.#

In one line Gemini reads Google. ChatGPT reads Bing. Claude reads Brave. The three indexes do not agree on who you are.

The architecture of AI search is not what most marketing directors assume. The assistants buyers are now using do not all read the same web. They read different webs, curated by different crawlers, ranked by different rules.

The breakdown matters because the implications cascade from it.

Google family Gemini, AI Overviews, Apple Intelligence fallback
Reads Google index
Your lever Google Search Console
OpenAI · Microsoft ChatGPT, Copilot, DuckDuckGo chat
Reads Bing-foundational
Your lever Bing Webmaster Tools
Anthropic Claude
Reads Brave Search
Your lever Verified presence on search.brave.com

Google's family (Gemini, Google AI Overviews, Gemini in Workspace, Apple Intelligence's web fallback) reads the Google index. This is the index every marketing director has been optimising for since 2008. It is comprehensive, sophisticated, and the home of the agency dashboard. It is also, increasingly, only one of three.

Microsoft and OpenAI's family (ChatGPT, Microsoft Copilot, DuckDuckGo's chat) reads a Bing-foundational index. Bing supplies the underlying retrieval; OpenAI's own search crawler layers proprietary indexing on top. ChatGPT, by usage, is now the most-asked AI assistant in commercial contexts. The marketing director who has not claimed Bing Webmaster Tools is, functionally, hoping that what Bing happens to know about their business is accurate. As reported elsewhere in this series, research happens in AI; shortlisting happens in AI; Google is increasingly the validation step, not the starting point. For the family of assistants reading Bing, that mid-funnel reality is felt before any Google ranking is consulted.

Anthropic's family, Claude, is widely evidenced to rely on Brave Search for web retrieval. Brave is the smallest of the three by raw scale, but is the only major web index built independently of Google and Microsoft. It indexes differently. It ranks differently. It values different signals. A brand that is well-represented on Google can be functionally invisible on Brave. A marketing director who has never run their brand name on search.brave.com cannot tell you what Claude believes about their business. Most cannot.

Two further engines round out the picture. Perplexity runs hybrid retrieval (its own crawler-built index plus third-party search providers), with a strong bias toward sources it considers authoritative: academic publications, established trade press, named experts. Brave's own assistant Leo reads the Brave index natively. Smaller volume, more strategically interesting in privacy-conscious B2B contexts.

II ½

And even within one engine, the answers diverge.#

In one line Three Google surfaces (Gemini, Search AI Overview, AI Mode) produce three different shortlists for the same query. The shared index is not the shared answer.

The Multi-Index Problem, as set out above, is the gap between Google, Bing, and Brave. That gap is the floor of the problem, not the ceiling. Even the same company, querying its own index for the same buyer asking the same question, returns different answers depending on which surface the buyer chose to open.

To test it cleanly, we ran a single query; “I’m getting married and I want a big party; top 5 wedding venues in the Cotswolds”; through three Google products, within minutes of each other.

Gemini
The assistant
  1. Grittleton House
  2. The Great Tythe Barn
  3. Elmore Court
  4. Manor By The Lake
  5. De Vere Tortworth Court
Google Search
AI Overview
  1. Stone Barn
  2. Manor By The Lake
  3. Blenheim Palace
  4. Cripps Barn
  5. The Frogmill
Google
AI Mode
  1. Stone Barn
  2. Blenheim Palace
  3. Manor By The Lake
  4. Sudeley Castle & Gardens
  5. Cotswolds Hotel & Spa

  named in all three Google surfaces

Three Google products. Three different shortlists. Only one venue (Manor By The Lake) appears in all three. Gemini, the assistant, shares only that one venue with the other two surfaces. Search’s AI Overview and AI Mode share three (Stone Barn, Manor By The Lake, Blenheim) but disagree on the remaining two. Two new venues enter the conversation that none of our four cross-engine tests named at all: The Frogmill, surfaced only by AI Overview, and Cotswolds Hotel & Spa, surfaced only by AI Mode.

The implication for the marketing director’s mental model is significant. The standing assumption; “if we’re indexed by Google, we’re visible across Google’s products”; is no longer true. Each surface applies its own retrieval bias, ranking weights, and sourcing rules on top of the shared index. The shared index is not the shared answer; it is the input. The AI Overview prioritises differently from AI Mode. AI Mode prioritises differently from Gemini. Gemini, in turn, has its own opinionated retrieval logic that produces a list almost no other Google surface confirms.

If the same company can’t agree with itself, “Google ranking” stops describing visibility. It describes one third of one third.

This loops directly back to the central argument of section II: the marketing director who treats “Gemini visibility” as a stand-in for “Google AI visibility” is making the same category error a buyer makes when they treat one assistant’s answer as the buyer’s truth. There are at least three Google surfaces, with at least three different definitions of who matters. Tracking only one of them is, again, watching the smallest screen in the room.

III

The mechanism nobody briefed the marketing department on.#

In one line Three tiers of bot. Blocking GPTBot is one decision. Blocking OAI-SearchBot is what removes you from ChatGPT. Most agency-set robots.txt files don't separate them.

To understand why the indexes diverge, and why blocking the wrong bot is the single most common technical mistake we see in our audits, it is worth understanding the machinery underneath.

Every AI assistant that touches the live web does so through three different categories of bot, each doing a different job. They are routinely conflated, including by agencies. The conflation is the source of most of the visibility errors we encounter.

If you read one thing, read this
Three tiers of crawler · three different consequences
Tier What it does OpenAI Anthropic Perplexity Google
Training Feeds the model's memory. Shapes what the AI knows about your brand even when not searching the live web. GPTBot ClaudeBot No foundation training Google-Extended
Search indexing Builds the AI's own retrieval index. Determines whether your site can appear in the assistant's search answers. OAI-SearchBot Claude-SearchBot PerplexityBot Googlebot
Live retrieval Real-time fetches when a user asks the assistant a question. ChatGPT-User Claude-User Perplexity-User handled inside Gemini grounding
← scroll →

Two distinctions inside this taxonomy do most of the work, and both are routinely missed.

The first is what blocking actually blocks. A marketing director who tells their developer to “block AI bots for IP reasons” is, almost without exception, blocking the training tier: GPTBot, ClaudeBot, Google-Extended. This is a defensible decision if the goal is to stop AI companies using the brand's content to train future models. It is a catastrophic decision if the marketing director also wants the brand to appear in ChatGPT Search, Claude's web answers, or Gemini's AI Overviews. Blocking GPTBot does not stop ChatGPT Search from indexing the site. OAI-SearchBot does that. The two bots have different jobs and different consequences, and most agency-set robots.txt files we audit do not separate them correctly.

The most expensive technical mistake we find is silent. The brand simply stops being recommended by ChatGPT, and there is no event that names what happened.

The second is what robots.txt actually controls. Training and search-indexing bots respect robots.txt directives. They are autonomous crawlers, and they read the rulebook. Live-retrieval bots are a different category. ChatGPT-User, Claude-User, and Perplexity-User are classified by their operators as actions taken on behalf of a human user (closer to a browser fetch than to a crawl), and the robots.txt protocol is treated more permissively as a result. OpenAI's own documentation states explicitly that ChatGPT-User may not be governed by robots.txt in the same way as automated crawlers. The practical implication: robots.txt remains a strong instrument for training and indexing control, but a weaker one for user-initiated retrieval.

The underlying indexes: the layer beneath the bots.

The crawler tiers explain what each AI vendor is doing on your website. They do not, on their own, explain which web your buyer is being shown. That answer sits one layer deeper, in the indexes the assistants ultimately retrieve from.

ChatGPT's search foundation is Bing. Bing remains a critical foundation for ChatGPT Search visibility, with OpenAI's own search crawler layer sitting on top. Bing Webmaster Tools, unclaimed by the majority of UK businesses we audit, is the most direct lever a marketing director has on ChatGPT visibility.

Claude's search foundation is Brave Search. Anthropic publishes Brave Search as a subprocessor for Claude's web-search capability; the company's product documentation describes the relationship in deliberately general terms, calling it “a third-party search service provider selected by Anthropic.” The behavioural pattern across our own audits is consistent with that disclosure: brands that do not surface cleanly on search.brave.com are brands Claude struggles to describe.

Gemini's search foundation is Google's live search index: first-party, fully integrated, no intermediary. This is the only AI assistant in our list whose visibility maps directly onto Google ranking discipline.

Perplexity is hybrid. Its own crawler builds a proprietary index, but its retrieval also draws on third-party search providers. The exact balance shifts; the operational point is that Perplexity weights authority signals (academic publications, established trade press, named experts) more heavily than the others.

Three indexes. Four assistants. Three tiers of bot per assistant. One marketing director, watching one dashboard.

The fixes for being correctly read by each tier of crawler, including which user-agents to permit, which schema to expose, and which Webmaster Tools to claim, are covered in the Maturity Curve piece and the Citation Graph piece. The point of this section is one layer above them: before you fix what each engine reads, you have to know which bots are reading for which engine, and which engines your buyer is actually on. Most marketing directors are still operating as though there is one bot, one engine, and one answer.

There are three of each. And the three do not always agree.

IV

What the divergence actually costs.#

In one line Single-index Cost of Inaction was a third of the picture. Most marketing directors are reading the smallest third.

The Multi-Index Problem is not a thought experiment. It produces specific, measurable losses across the sectors we audit most often. Three composites from our Q1 2026 audit base.

01 · The Bing Blindspot
A boutique London hotel group ranking strongly on Google, invisible inside ChatGPT.

The Google ranking is real. The shortlist conversation is happening somewhere else. The underlying Bing index has stale or absent structured data for the property, and TripAdvisor's listings, heavily weighted by Bing, are out of date.

The dashboard says everything is working. ChatGPT, asked the category question, names three competitors.

Single-index COI £45k; £70k
Multi-index exposure £110k; £165k
Engine missed Bing-powered ChatGPT
02 · The Brave Blindspot
A private members' club with strong FT and Tatler coverage, absent from Brave's index.

Well-represented in Google's knowledge graph. Brave weights different authority signals, and the club has no presence on the platforms Brave's discovery engine reads.

Claude, increasingly used by exactly the senior professional demographic the club exists to attract, describes the club inaccurately, or not at all.

Single-index COI £35k; £60k
Multi-index exposure £85k; £140k
Engine missed Claude · Leo
03 · The Unclaimed Listing
A heritage estate offering weddings and private hire. Everything correct for Google; Bing Places untouched.

Bing Places listing populated automatically with five-year-old data. Capacity figures wrong. Pricing absent. Contact route stale.

ChatGPT, asked for “a private estate for a private hire weekend in the Home Counties,” cites three competitors. The estate is not in the answer.

Single-index COI £40k; £75k
Multi-index exposure £100k; £160k
Engine missed Bing-powered ChatGPT

In each case, the marketing dashboard says everything is working. In each case, the most expensive enquiries (couples shortlisting venues without ever opening a browser tab, members evaluating clubs through their AI assistant, guests planning escapes through ChatGPT) are happening in a place the dashboard cannot see.

The Cost of Inaction is no longer a single number against a single index. It is the sum of three blind spots, and most marketing directors are watching one of them.

There is a deeper layer underneath the three blind spots, and it deserves naming. AI does not only read your website. It reads the web's opinion of you. The Citation Consensus Problem named in the Maturity Curve piece (the fact that AI reads niche forums, sector publications, trade press, professional registers, and review aggregators alongside your own site) applies with multiplied force in a multi-index world. Different aggregators feed different indexes. TripAdvisor's weight in Bing is not its weight in Brave. The Knot's integration sits inside ChatGPT, not Claude. A brand can be visible on the third-party platforms one index trusts and entirely absent from the platforms another index trusts. The Cost of Inaction methodology covered in the previous Shift piece calculates the visibility gap on a single index. The Multi-Index Problem is the recognition that the gap is now plural, and that the platforms feeding it are plural too.

A marketing director measuring visibility on Google alone is, increasingly, measuring one third of their actual exposure.

V

What changes in the marketing director's quarterly report.#

In one line Run the category query in three assistants. Claim Bing Webmaster Tools. Retire “#1 on Google” as the answer to the visibility question.

The Multi-Index Problem is not solved by panic, and not solved by hiring three agencies. It is solved by changing what the quarterly report measures, and changing the standing assumption that “we rank well on Google” is the answer to the visibility question.

Three shifts, none of them expensive, sit at the heart of the discipline.

The first is to broaden what “visibility” means in the standing report. A quarterly marketing report in 2026 that contains rankings only from Google is reporting on one third of the buyer's reality. The rest of it (what ChatGPT says when asked the brand's category question, what Claude says when asked for a recommendation, what Perplexity cites when asked for a comparison) is not in the report not because it cannot be measured, but because nobody has asked for it to be. Those answers are checkable in five minutes, by anybody, on any laptop. Most marketing directors have never run them.

The second is to claim the ground that has been silently abandoned. Bing Webmaster Tools is free and takes ten minutes to set up. Most UK businesses we audit have never claimed it, which means ChatGPT and Copilot are reading whatever Bing has scraped automatically, with no input from the brand. search.brave.com is publicly searchable in three seconds. Most marketing directors have never tested it. These are not budget questions. They are noticing questions.

The third is to retire “we rank #1 on Google” as a statement of marketing health. The line was adequate for a decade. It is no longer. Not because the Google ranking has stopped mattering (it has not), but because it is now one signal among at least three, and the other two are growing faster than the first.

Old

“We rank #1 on Google.”

New

“When our buyers ask ChatGPT, Claude, and Gemini about our category, we are in the answer, and we know which engines name us and which don't.”

The old line was a status statement, checked once per quarter and written into a board pack. The new line is an ongoing question, checked every time a model retrains. A status statement says we are visible. A diagnostic asks are we still visible, and on which surfaces? The first you report. The second you operate.

A marketing director presenting Google rankings to the board in 2026 without context for ChatGPT, Claude, and Perplexity visibility is presenting an incomplete picture. The board will not always notice the gap. The buyer is already in it.

Closing

One ranking is no longer one truth.#

Marketing directors at heritage venues, members' clubs, and country house hotels have spent fifteen years building the offline brand. The Google rank, when it came, was the digital echo of that brand. For most of that period, the echo was loud enough. One index, one ranking, one true statement: we are visible to the buyer.

That statement is now true on one third of the buyer's surface area. Sometimes less. The brand built carefully for a decade is being filtered, in real time, through three indexes that do not agree on what it stands for, and the dashboard in front of most marketing directors is reading from the one that is shrinking fastest as a share of the total.

The fix is not new agencies. It is not new platforms. It is the noticing that the question itself has changed. “Are we visible?” used to have one answer. It now has three, and the gap between them is where the lost enquiries live.

The brand is the same. The buyer is the same. The map is not.

Friction & Toil's Web Intelligence Reports diagnose visibility across all three major AI indexes (Google, Bing, and Brave) alongside ChatGPT, Claude, and Perplexity citation behaviour. Methodology and sample reports at frictionandtoil.com/web-intelligence.
The Shift Briefing

Three indexes. One monthly read.

A monthly intelligence brief on how AI is changing what UK customers find, and what they don't. No consultancy noise.

Subscribe No pitch sequence. Unsubscribe any time.

Sources & references

  1. OpenAI, Bots & crawlers documentation: canonical reference for GPTBot (training), OAI-SearchBot (search indexing), and ChatGPT-User (live retrieval), including the note that ChatGPT-User performs user-initiated fetches that are not governed by robots.txt the same way automated crawlers are. platform.openai.com/docs/bots
  2. Anthropic, What is ClaudeBot and other Anthropic crawlers?: canonical reference for ClaudeBot (training), Claude-SearchBot (search indexing), and Claude-User (live retrieval). support.anthropic.com
  3. Anthropic, Subprocessor list: lists Brave Search Inc. as a subprocessor providing third-party search results to Claude's web-search capability. anthropic.com/legal/subprocessors
  4. Perplexity, PerplexityBot documentation: describes PerplexityBot (search indexing) and Perplexity-User (live retrieval) and the hybrid retrieval model that combines Perplexity's own crawler with third-party search providers. docs.perplexity.ai/guides/bots
  5. Google, Google-Extended & Googlebot user-agent documentation: reference for Google-Extended (controls use of crawled pages for Gemini and Vertex AI training) and Googlebot (search indexing feeding Gemini grounding). developers.google.com
  6. Friction & Toil audit base, Q1 2026: 42 commissioned Web Intelligence Reports across UK hospitality, professional services, and heritage venues. Source for the “0 dashboards&rdquo, and “3 tiers of crawler” figures cited in the hero stat strip and for the three composite vignettes in section IV.
  7. Friction & Toil live re-test, May 2026: 38-brand curl-based crawler-policy and schema scan re-running diagnostics from the audit base. Companion evidence for the multi-index visibility patterns described in this article.