Same Question, Same Engine, Two Devices: Measuring Retrieval Asymmetry in China's Biggest AI Assistant

Technical Note 01 · Bo Wang · GeoCheckTool · July 2026

Part of an ongoing measurement program on Doubao, the AI assistant most Chinese consumers actually use. Every claim below is backed by an archived evidence packet — question text, screenshots, search terms, source panel, and raw answer — committed to a version-controlled repository.

---

TL;DR

We ran 30 frozen question-entity cells across 5 industries, each executed on both the desktop web version and the mobile app of the same AI assistant, in fresh sessions, paired within a 30-minute window where possible. 60 first answers total.

Three things came out of it:

1. Surface asymmetry. Among the 16 pairs that met the 30-minute window, the web surface displayed more search terms in 15 of 16 and more sources in 14 of 16. Not once did mobile exceed web. We cannot yet explain why — and we ruled out the most obvious artifact.
2. Brand-defense queries expanded in this matrix. In 20 of 20 first answers, the engine introduced a third competitor we never named. This is a word-for-word audit of 20 hashed answer files, but each cell remains a single run.
3. Entity resolution fails in four distinct ways, one per industry, all reducing to the same root: the retrieval layer does not preserve entity boundaries.

Raw data, method, and the things we could not verify are all published below.

---

Why a Chinese engine should interest you

Two reasons, neither of them "China is interesting."

It's a preview. Consumer AI-assistant adoption for commercial decisions — which vendor, which clinic, which supplier — is further along here than in most Western markets, and so is the industry that has grown up to game it. Whatever equilibrium AI search reaches, this market is closer to it.

It's a blind spot. Every Western AI-visibility platform tracks ChatGPT, Gemini, Perplexity, Copilot. None of them can reach Doubao — it requires Chinese IP, Chinese accounts, and Chinese-language source judgment. To our knowledge this is the first English-language measurement of how this engine selects and weights sources. If you own a brand with a China market, no tool you currently pay for can see this.

---

Method

• 5 industries, chosen to span different decision mechanics rather than different verticals: local experience (campsites), local services (moving companies), credential-gated professional services (construction labor), product purchase (family camping tents), B2B procurement (AGV/AMR robotics).
• 2 real entities per industry — one with a strong public record, one with visible gaps.
• 3 base question types per industry: open category recommendation, constrained scenario, brand defense / competitive comparison.
• 30 frozen cells × 2 surfaces = 60 first answers. Question wording frozen in version control before collection; any change logged as a deviation.
• Each surface run in a fresh session; paired runs executed within 30 minutes where operationally possible.
• Every sample archives: exact question text, submission screenshot, the search terms the interface displayed, the full source panel, the answer, and any anomaly.

Evidence grading. We separate four levels and never let them blur: L1 interface facts (visible in a screenshot), L2 verified facts (source body text checked), L3 aggregate patterns (with n, window, and confounds stated), L4 hypotheses — including every explanation the model gives for its own behaviour. The model's account of why it searched, ranked, or excluded something is treated as a hypothesis to test, never as a root cause. This one rule eliminates most of what currently passes for AI-search research.

---

Finding 1 — The surface asymmetry

Among the 16 pairs that satisfied the 30-minute window (credential-gated services, product purchase, B2B procurement):

Measure	Web higher	Tie	Mobile higher
Search terms displayed	15	1	0
Sources in panel	14	2	0

Mobile never exceeded web on either measure.

What we ruled out. The obvious objection is that the mobile app simply shows fewer sources — a collapsed list, a "see all" control the collector never tapped. We checked: 31 source-panel screenshots across all 30 mobile samples, no expand control anywhere; 6 samples show the list terminating naturally; and we re-opened 2 historical mobile conversations and scrolled the panel to its end — 23 of 23 and 24 of 24 sources visible, answer body immediately following. The counts are real counts.

What we cannot rule out. Account state is not fully controlled — the web account label is known, the mobile account label was not verified, and account history may differ. Session load and index freshness within the window are uncontrolled. We are therefore reporting a count pattern, not a causal claim. "Web retrieves more" is not something this data can support; "web never displayed fewer search terms or sources in the 16 pairs that met our window" is.

This is the part we would most like someone else to break. If you have access to any AI assistant with both a web and a native mobile client, run the same question on both, in fresh sessions, minutes apart, and count the sources. If the asymmetry doesn't replicate on ChatGPT or Gemini, that's a more interesting result than ours.

---

Finding 2 — Brand-defense queries widened the field in this matrix

20 of 20 brand-defense answers introduced a competitor that was not named in the question. Not most. All of them.

To be precise about what that denominator is: 10 distinct brand-defense questions, each run once on each surface. So this is 10 independent question-entity conditions observed twice each, not 20 independent trials — and each condition was run once, not repeated. What the data supports is generality across conditions: every condition we tested did it. What it does not yet test is whether re-asking the same question always does it. Both answers were verified against the recovered verbatim text, not from screenshots — the third brand is in the body of the answer in every case.

It is already known that competitor-authored content — "Top 5 alternatives to X" — surfaces in the citations when a buyer evaluates a brand. That's been documented in Western data. Two things here are new:

The within-matrix count is 20 of 20. We rechecked all 20 complete verbatim answer files (10 web, 10 mobile), including the answers recovered from synchronized account history, and each first answer expanded the consideration set. File-level SHA-256 hashes and coding decisions are included in the release. Because every cell is still n=1, this is not yet a repeat-stability claim.

The added party differed within the matrix. The third competitor was not the same across the observed answers and surfaces; in the product category, the added model also differed. With one run per cell, this is a cross-sample difference, not a stability estimate.

The practical consequence within this observed matrix is that naming two brands did not keep the answer to those two brands, and the added comparison party differed across observed surfaces. Whether that behavior repeats reliably is a separate test.

---

Finding 3 — Four industries, four flavours of the same failure

Each industry produced a distinct entity-resolution failure, and they rhyme:

• Local services — a near-identical company name was substituted for the target. The source body text describes a different legal entity whose name differs by two characters. Verified against the source body (L2). In one unreplicated follow-up, adding an explicit disambiguation requirement coincided with a correctly distinguished answer; this remains a hypothesis for repeated single-variable testing, not evidence about a default mechanism.
• Credential-gated services — registered business scope, filing records, and safety permits were repeatedly merged into a single chain of reasoning, as though they were one credential. They are not.
• Product purchase — brand-level stability, model-level drift. Specific model numbers, waterproof ratings, and pole materials moved between runs while the brand stayed put.
• B2B procurement — a vendor's product capability and its systems-integration and delivery capability were written as one thing. For a procurement decision, that is the distinction that matters most.

Underneath all four: the retrieval layer does not preserve entity boundaries. Names that are close get treated as the same thing; attributes that belong to adjacent categories get merged. Chinese corporate naming conventions generate near-identical names at industrial scale, which makes this market a stress test — but nothing about the failure mode is China-specific.

We also recorded two verified cases of retrieved-but-not-adopted: a page entered the source panel and did not inform the answer. "Getting cited" and "getting into the source panel" are different outcomes and should be measured separately.

---

An emerging pattern we are not yet calling a finding

Across several industries we observed answers making claims stronger than their sources support — a source establishing that a company has registered a given business scope, and an answer reporting that the company is licensed, staffed at a certain headcount, accident-free, or shipping at volume.

The academic literature has documented citation-support failure in general terms. What we have not seen documented is this behaviour in a commercial recommendation context, where the escalated claims are precisely the ones a buyer would act on. We are adding a dedicated code for it and will report it properly once it's counted rather than noticed.

---

What this data cannot support

Stating this clearly is part of the method, not a disclaimer:

• No causal claims. Nothing here establishes that any condition causes inclusion or exclusion. That requires single-variable reversal with repetition, which we have not yet run.
• No veto or bonus factors. We have candidates. We do not have proof.
• Mobile answer text was recovered after the fact, not captured live. For 24 of the 30 mobile samples, the full verbatim answers were recovered afterwards from the same account's synchronized conversation history and hashed on recovery. They are flagged as recovered rather than captured.
• Mobile source URLs were read from the desktop-synced view of the same conversation, not tapped through on the handset. This is worth being explicit about, because our headline finding is a mobile-versus-web count: the counts themselves come from mobile screenshots and nothing else. The desktop-synced view was used only to resolve which URLs sat behind those already-counted entries, so that we could fetch and read the source bodies. In two method checks, the synchronized view preserved 21 of 22 and 23 of 23 mobile source positions (44/45 combined across n=2 panels). That supports this recovery method for those panels, not a platform-wide completeness guarantee.
• The Sohu near-name evidence page has no third-party timestamped archive. Wayback returned HTTP 520 and archive.today returned HTTP 429/CAPTCHA. We did not bypass the CAPTCHA; the page is retained as local HTML plus SHA-256 under a disclosed archive waiver.
• Source-body verification is partial and in progress. Of the mobile source URLs recovered so far, most returned a fetchable body; only a small number have been promoted to verified (L2) after human review. Download success is recorded as a retrieval fact and is never auto-promoted to verification.
• Only 16 of 30 pairs met the 30-minute window. The surface asymmetry is n=16. Pairs outside the window are reported separately and are not used for the pattern.
• One engine, one market, one collection window. No generalization to other engines is offered.

---

Data

→ github.com/bobuilds/doubao-ai-search-data — everything below is in that repository, live now, CC BY 4.0.

Published alongside this note:

• The master sample table — every frozen sample plus logged exceptions, all 47 fields, one row per question × entity × surface × session. Values we cannot publish are marked [withheld] rather than silently dropped, so the schema itself stays fully transparent.
• The frozen question bank for this batch, released now that collection is closed. (The question bank for our quarterly index is a separate, sealed instrument and stays sealed — a published bank is a gameable bank.)
• The deviation log — including the rows we excluded because the sampled entity turned out to be a client of our consulting practice. Our protocol bars client entities from the public sample pool; the exclusions are disclosed, the identities are not.
• The brand-defense verbatim audit — one redacted row per first answer, with file SHA-256, third-competitor coding result, and the explicit same-cell-n=1 boundary.
• Three exemplar evidence packets (question, displayed search terms, source panel, answer) with third-party identifiers redacted.

Naming policy: platforms and major brands appear under their real names; small local businesses are pseudonymized with stable coded IDs — a constraint of publishing from mainland China, not of the data. Researchers who want the verbatim identities or the full packet behind any specific row can ask; each packet needs manual redaction, so we prepare them per request.

One thing you will notice immediately on opening the table: a lot of cells say pending. Site-penetration status is unresolved on 57 of 61 rows, and archive_ref on 54. That is deliberate. Verification state is a published column, not a private one — you can see exactly which claims in this note rest on verified ground and which are still interface-level observation, without taking our word for either.

Suggested citation:

Wang, B. (2026). Surface Asymmetry in China's Biggest AI Assistant: Batch-1 Dataset. GeoCheckTool. https://github.com/bobuilds/doubao-ai-search-data

Disclosures. We sell GEO consulting services. No entity measured in this study paid for inclusion, and measurement cannot be purchased. Entities that turned out to be clients of our practice were removed from the public sample pool and the removal is logged.

---

The next note covers query expansion — what the engine silently turns your question into before it retrieves anything.

Bo Wang — geochecktool.com · github.com/bobuilds/doubao-ai-search-data