GA4 LLM Benchmark: A Harder Test With Recent Data

Our first November GA4 audit gave the models a clue that made the answer too easy: an event name identified the demo generator. The published scores of 92-100 also measured valid GA4 fields and query structure. They left us with little information about how carefully the models reasoned from the reports.

We reran the same eight models and added GPT-6 Astra, using recent data and a new evidence rubric. The task asks for calculations with the right denominators, checks across reports, a credible production explanation and a measurement that would distinguish it from synthetic traffic. A correct label alone earns no points.

The new evidence score is a different metric from the earlier field score. These results cannot establish that a model became worse after we removed the clue. The data window, report interface, task and grading changed together. The original results remain below for reference.

Retest results

GPT-6 Astra earned 100/100 in all three attempts. GPT-6.1 Sol scored 90, 97 and 99, for a mean of 95.3. Across the 27 planned attempts, 15 produced final analyses, six ended without a final answer and six Gemini attempts remained unscored after provider errors. A late GA4 revision affected Astra's last attempt; the methodology explains the changed values and the checks against each answer's returned reports.

Scores of zero represent genuine attempts without a final answer. N/A represents an infrastructure failure and is excluded from the mean. Time and cost means cover scored attempts, including genuine no-final attempts. Provider time excludes GA4 calls; the download also records end-to-end time.

Claude Fable 5.2 and Qwen3.8 Max used the fixed response allowance without producing a final answer in all three attempts. Both Gemini endpoints returned repeated upstream MALFORMED_FUNCTION_CALL errors in this setup; the exact cause was not established. Provider-reported usage for the 21 scored attempts totaled $4.482, including genuine no-final attempts. Preserved interrupted or unscored work added a known $0.082, for a known total of $4.564. One earlier interrupted Carbon request has no saved usage record, so its additional charge is unknown. An exhausted key allowance interrupted the initial pass; only infrastructure failures were recovered, with the failed recordings retained.

Download the per-attempt scores, criteria and usage.

Compare the interactive evidence table and historical results.

What the answers revealed

The useful distinctions were often in the details of an otherwise plausible answer. In its first attempt, GPT-6.1 Sol checked the latest week's 415 purchase events against 415 reported transactions and the corresponding revenue. It treated that agreement as evidence of internal consistency, while keeping open whether those purchases represented real orders. Its proposed follow-up was to compare a sample with independent order records.

Grok 4.7's first answer also reconciled purchase counts, but made an unsupported inference about devices. The grouped user counts added up to the overall reported user total, which it treated as proof that nobody used multiple devices. Google documents unique-count approximation and reporting-identity effects; device-based identity does not join a person's different devices. Our reading is that equal aggregate totals cannot establish zero real cross-device overlap. The device criterion lost credit even though the individual counts were copied correctly.

GPT-6.1 Terra's first answer chose inconclusive and identified which questions it had not answered. Its attribution requests had failed, and it had not obtained the prior week's events or the standard commerce metrics. That was a useful boundary to state, but acknowledging missing evidence does not earn the points for performing those checks.

The signup question tests a similar distinction. A ratio of signup completion events to start events cannot tell us which people completed after starting, or whether they did so within 24 hours. That requires linked identities, order and timestamps. The report interface does not supply them, and the rubric rewards recognizing the limitation.

Data and methodology

The primary window is November 7-13, 2026, with October 31-November 6 as its comparator, inclusive in America/Los_Angeles. These were the latest seven complete calendar days when the batch was prepared on November 14. Every model and resumed attempt uses those same dates.

The property contains synthetic demo traffic. The model-facing interface exposes standard aggregate activity, daily patterns, device categories and selected business events. It omits generator context, property and custom metadata, transaction IDs, page labels and hostnames. Acquisition labels are reduced to assigned and (not set). Returned reports have IDs that the answer can cite.

The generator no longer sends the recovery probe or its explicit diagnostic parameters to GA4. A narrowly scoped historical deletion request is in its preview period. Google retains the event count; this is not complete erasure of the event. The benchmark also excludes the identifying event name from its queries. Once GA4 suppresses that name, the filter cannot identify and subtract the retained count.

The prompt, settings and rubric were fixed before paid outputs. Transport fixes made during recovery stopped errored provider output from entering the conversation and stopped the run on access or budget refusals. Those changes did not alter the task, scoring, report surface or response budget; earlier captures were preserved.

The acquisition interface had an implementation restriction the model-facing guide did not fully explain: it accepted only sessions and engaged sessions, without a second dimension. Several otherwise valid-looking requests were rejected. This is a limitation of our test interface and can cost an investigation useful evidence within the response allowance. It should not be read as proof that a model invented a GA4 field.

How the evidence score works

Numerical claims need support from an actual returned report. A missing final answer, or a conclusion without any nonempty reports, receives zero. Provider and connector failures are unscored. An evidence-supported inconclusive answer can earn substantial credit; it receives one fewer classification point than a supported synthetic conclusion. The three individual scores belong beside the mean.

For paired four-point quantitative checks, each supported week earns two points. Grading includes judgment about explanations and uncertainty, so these scores are an operational assessment of this task rather than a general measure of model intelligence.

What this batch cannot establish

This is one synthetic property. It cannot estimate how often a model would misclassify a real production property with broken tracking. Three attempts give only a limited view of repeatability, and the fixed response budget and report interface are part of what is being tested.

The dates are pinned, but GA4 remains live. Google notes that recent reports can change during processing. We retained the preparation reference and graded each claim against the rows that attempt received. The comparator also spans a generator repair and a daylight-saving transition; neither is a cause established by these aggregates alone.

The post-run check at 20:52 UTC on November 14 found revisions to the current week's totals, daily rows, acquisition and device reports. Weekly sessions changed from 7,800 to 7,780 (-0.26%) and engaged sessions from 4,776 to 4,794 (+0.38%); period-distinct users remained 4,159. November 13 changed from 1,151 sessions and 676 engaged sessions to 1,152 and 674. The comparator and the current named-event and commerce references matched at the checked precision. Astra's third attempt received the revised weekly and daily totals and calculated its rates correctly; its device claims were checked against the device rows that particular request returned. Rechecking those claims did not change any awarded points. This is a comparison of date-pinned live reports, not an identical immutable snapshot, and it cannot isolate all effects of report timing on model behavior. The downloadable results retain a structured account of this revision.

Future batches use newer dates. Within-batch comparisons are stronger than comparing scores from different datasets as though only the model changed.

Where to add more difficulty

Astra's first attempt earned all the available points. We kept that score: changing the rubric after seeing the answer would make the comparison less useful. The next step is a set of controlled cases that includes collection defects, changes within a segment, shifts in device mix and duplicate commerce events. Models would have to distinguish those explanations using additional evidence, with the transformations and expected answers fixed before testing.

Ordered signup cohorts would add another useful challenge: repeat events, completions across midnight and starts without a full 24-hour observation window. That suite is a design for future work, not a test these models have already passed. Simulated tracking-defect cases would also need to be labeled as simulations; they would not establish a false-positive rate on production traffic.

Original November 13 results

These are the original field-validation results, published November 14 before the revised evidence test. All eight models completed three scored runs. The runner retained 12 full conversations, all of which included a synthetic conclusion after GA4 returned rows. The table covers 24 scored attempts; it does not imply that all 24 had a saved full transcript.

The explicit recovery-event label was the clearest clue in several saved answers. Some models also queried older periods with very different traffic characteristics. Those findings describe the original test and are not evidence gathered through the restricted recent-data interface.

Related results

• Evergreen GA4 LLM leaderboard
• Choosing an LLM for analytics
• The June synthetic-data benchmark