How to track Discover in real-time

header image

by Valentin Pletzer - August 1st, 2019 (updated August 5th, 2019)

“Knowledge is power, France is Bacon” is one of my favourite internet memes, a misunderstanding which came to be because someone didn't dare to ask. For me it is the same with Google's referrers, we know there is something off but somehow nobody dares to digg deeper. This article is an attempt to fix this for Google Discover and Google Search.

I will focus on Google Analytics (GA) but it’s pretty much the same for a lot of the other web analytics services. Some “intelligent” aggregation will skew your perspective and therefor mislead you. Google Analytics for example will aggregate pretty much all referred clicks from www.google.com (and other TLDs) as “google / organic”. In the end you won't know if your traffic came via Google Search or Discover.

After applying the filter described in figure 1 you will see something similar to the screenshot on top of this article. Users from Google Discover will show up as www.google.com and com.google.android.googlequicksearchbox/ while traffic from www.google.com/ is Google Search.

compare Discover traffic from GSC with GA
it is by no means perfect but good enough to work with

Before you proceed some important information:

Changing your Google Analytics settings might result in data loss and/or interrupt an otherwise continuous row of data. Proceed at your own risk! To mitigate risk I would recommend to clone your current in-production analytics view and apply the filter only on the copy.

How to prepare Google Analytics for Discover tracking

figure 1 showing the GA filter
figure 1: click to view
As described above Google Analytics by default does aggregate all www.google.xyz-referrers into a single line “google” which results in dropping a lot of information which proves precious for differentiating Google Search and the Google Discover app. That is why my recommendation is to setup a filter which overwrites the “campaign source” field.

Figure 1 shows the filter I applied in our Google Analytics view. The important part is the regex applied to the unmodified referrer (named “referral” in GA).

(.*)\:\/\/([A-Za-z0-9\-\.]*\/?)

The regex basically groups a referrer into several groups like protocol (e.g. “https”), hostname (e.g. “www.google.com”) and the most important part: It extracts the first slash “/” if available. The filter itself then writes hostname plus slash into the “campaign source” field. You might want to write this data into a custom variable but overwriting the source does enable you to monitor the data in the real-time section of Google Analytics.

Why the slash is so important

When analysing the full referrer-data I noticed something very curious: Some looked like this https://www.google.com/ (which you might expect) while others were missing a trailing slash https://www.google.com. While RFC 3986 states that an empty path (aka no slash) is OK, this is highly unusual for browsers which all do send at least a slash as URL path. So when you get an click from the Google Search webpage you can expect to get https://www.google.com/ as referrer.

The only likely candidates for https://www.google.com without slash are apps. And after some research I was able to confirm that indeed the official Google Android app is sending https://www.google.com as a referrer – but only if a user clicks on a recommendation card (aka Google Discover) since a search will always open a browser/webview with a typical result page and this will send a “normal” referrer.

The Google iOS app sadly doesn’t send a referrer at all. So this will show as “(direct) / (none)” in Google Analytics. You can improve detection a tiny bit by look at the AMP viewer (see the “Even more tracking extravaganza” part of this article).

A complete list of Google referrer mess

After some promising discoveries I went to build a complete list of Google Services and their referrers. As it turns out Google referrer are a complete and utter mess. This is what it looks like at the beginning of June 2019.

full referrer
with filter
iOS Google App (Discover) no referrer
(direct)
iOS Google App (Search) https://www.google.com/search?q=.... (not cut)
www.google.com/
iOS Google News App https://news.google.com/
news.google.com/
iOS Google Chrome (Articles for you) https://www.googleapis.com/auth/chrome-content-suggestions
www.googleapis.com/
iOS Google Chrome (Search) https://www.google.com/
www.google.com/
Android 9 Google App (Discover) android-app://com.google.android.googlequicksearchbox/https/www.google.com
com.google.android.googlequicksearchbox/
Android Google App (Discover) https://www.google.com (no trailing slash)
www.google.com
Android 9 Google App (Search) android-app://com.google.android.googlequicksearchbox (no trailing slash)
com.google.android.googlequicksearchbox
Android 9 Floating Search Bar android-app://com.google.android.googlequicksearchbox (no trailing slash)
com.google.android.googlequicksearchbox
Android Floating Search Bar https://www.google.com (no trailing slash)
www.google.com
Android Google Chrome (Search) https://www.google.com/
www.google.com/
Android Google Chrome (Articles for you) https://www.googleapis.com/auth/chrome-content-suggestions
www.googleapis.com/
Android Google News Widget https://news.google.com/
news.google.com/

If you look into the history of Google referrers you might find a lot more mess. For example: Google News used to be news.url.google.com but there was a bug for several weeks and it changed into plus.url.google.com and then back again. At the same time you might have seen news.google.com as a referrer as well. Mobile Chrome Recommendations did change into discover.google.com in November 2018 and then back into www.googleapis.com in December. Also: It is quite difficult to get a clear answer from anyone at Google but it looks like Chrome suggested articles are a seperate beast and not part of Google Discover.

Even more tracking extravaganza

Another filter you might want to setup does need some client side preparation, meaning you will have to enhance the data-collecting javascript – specifically the AMP code. If your website does not use AMP this doesn’t apply to your setup.

Including tracking in Accelerated Mobile Pages can be achieved by using the amp-analytics component. Your implementation might look different (e.g. setting up the tracking within an iframe) but either way you’ll probably already add some custom variables to you tracking pixel and then this should be easy. If not, have a look at the documentation specifically the section on extra url parameters.

Not very well known but very useful are AMP HTML URL Variable Substitutions in our case it is worth to have a look at the “viewer” variable. The documentation states “[viewer] provides an identifier for the viewer that contains the AMP document. An empty string is provided when the document is loaded directly in the browser or if the id is not found.” Meaning if your document is loaded not from your server but via a cache (e.g. https://foo-com.cdn.ampproject.org/c/s/foo.com/amp_document.html) a variable called “viewer” is most probably available telling you which cache specifically is used.

If you don’t know how AMP caches work you might want to have a look at this great explanation: How AMP pages are cached

figure 2 showing the GA filter
figure 2: click to view
Figure 2 does show our filter setup after the viewer variable is available as a value in custom dimension (in this case custom dimension 157 is named ampViewer_h157). The filter does push the data from the custom dimension into “campaign term” and appending the name of the operating system of the client. The reasoning behind this is to a) have this data available in “real-time” as well as b) to differentiate between Android and iOS users.

Depending on your setup you might not use “campaign term” anyways since Google doesn’t send the full referrer for a very long time now and the “q”-parameter, which includes the search query/keyword, isn’t available to parse. At least that is the official story. There is an exception to this: On iOS Google has to use the webview as a browser within it’s Google App. People who visit your page from within this webview will send the full referrer since Apple doesn’t seem to honor the meta tag which should cause a cutoff of all the GET-parameters in the url.

So what's the deal with AMP viewer? If you look at the table above you will notice that traffic from the iOS Google Discover app doesn't send a referrer, but it does set an amp viewer which is https://www.google.com/. So if you use AMP and Google Discover does indeed send you iOS traffic this is how you will know.

please send feedback

I hope this article proves useful to you. If you have any comments, suggestions or improvements please do not hesitate and contact me on Twitter @VorticonCmdr. If you have have additional data I am more than happy to add this to this article.

In case someone from Google does read this: Please help us and improve your referrer data so we can differentiate between all of the Google services we enjoy.

If you read everything until here you seem really interested and you might want to continue and read this great article about 7 Examples of Untrackable Clicks From Google’s Ecosystem of Search, Chrome, and Feeds by Glenn Gabe.