Retrieving Search Analytics Data from the Google Search Console API for Bulk Keyword Research

Last year I blogged about using 855 properties to retrieve all your Search Analytics data. Just after that Google luckily released that the limits on the API to retrieve only the top 5000 results had been lifted. Since then it’s been possible to potentially pull all your keywords from Google Search Console via their API (hint: you’re not able to get all the data).

Since I’ve started at Postmates now well over two months ago one of the biggest projects that I started with was getting insights into what markets + product categories we’re already performing OK in from an SEO perspective. With over 150.000 unique keywords weekly (and working on increasing that) it is quite hard to easily get a good grasp on what’s working or not as we’re active in 50+ markets that influence the queries that people are searching for (for example, show me all the queries over a longer period of time with only Mexican in the title across all markets, impossible from the interface). That’s why clicking through the Search Analytics feature in Google Search Console was nice for checking specific keywords quickly, but overall it wouldn’t help in getting detailed insights into what’s working and what’s not.

Some of the issues I was hoping to solve with this approach:

  • Pull all your data on a daily basis so you can get an accurate picture of the number of clicks and how that changes over time for a query.
  • Hopefully get some insights into the actual number of impressions. Google Adwords Keyword Tool data is still vary valuable but as it’s grouped it can be off on occasion. Google Search Console should be able to provide more accurate data on a specific keyword level.
  • Use the data as a basis for further keyword research and categorization.

Having used the Google Search Console API a bit before I was curious to see what I could accomplish pulling in the data on a daily basis and making sense of it (and combining it with other data sets, maybe more on that in later blog posts).

The process:

  • Daily pull in all the keywords, grouped by landing page so you know for sure you get all the different keyword combinations and your data isn’t filtered by the API.
  • Save the specific keyword if we haven’t saved it before, so we know if the keyword was a ‘first-hit’ for the first time.
  • For every keyword that you return do another call to the API to get the country, landing pages and metrics for that specific query.

In our case we categorize the keywords right after we pull them in to see if it’s matching a certain market or product category. So far this has been really useful for us as it’s providing way better ways for dashboarding.

Some of the things that I ran into while building out this:

What to look out for?

  • The API is very much limiting the keywords that you get to see with only impressions. I was able to retrieve some of the data but on a daily basis the statistics for impressions are off with 50% from what I’m seeing in Google Search Console. However clicks seems to only have a small difference, win!
  • Apparently they’re hiding some of the keywords as they qualify them as highly personal. So you’ll miss a certain percentage because of that.
  • The rate limits of the Google Search Console aren’t very nice, for over 5k keyword it’s taking quite long to pull in all the data as you have to deal with their rate limits.

Most of these items aren’t really being an issue for us, we have better sources for volume data anyway. In the future we’re hoping to gather more data around different sources to extend that. I’m hoping to blog about somewhere in the future.


Making a move … what’s next!?

Last week was my last one at The Next Web as their Director of Marketing. For the last four years I’ve worked alongside great people: publishing the best content (TNW), organising the best + biggest tech conferences (TNW Conferences), selling the craziest drones (TNW Deals), creating the most beautiful workspace (TQ) and them who collect a ton of data on the global tech industry (Index.co). But still… it’s time for me to move on to something new:

Project: ‘New adventures’

As of Monday I’ll be joining Postmates to help out/lead their SEO strategy, which means I’m already in San Francisco to join the team from there.

For the last 7 years I’ve been maintaining a list of goals that I update every day/week/month with the things I’d like to achieve on a personal and business level in the (near) future. Since my first trip to the US many years ago one of the goals that I created was to move to the Bay Area for x months > x years, mainly to see if my skills would hold up in the more competitive and global area that I consider the Bay Area to be. Having had the ability to spend 6 weeks in/around San Francisco at the end of last year it made it even easier to decide that I wanted to move towards that area the sooner the better.
Having had some very positive changes in my life over the course of the last year it made the choice even easier ? #analyticspowercouple.

In talking to Postmates I found that my passion for SEO, Growth, Analytics, Innovation could even be more stimulated so joining them is a great opportunity to develop myself even more.

Some highlights and numbers from my time at TNW: 8 TNW conferences, 35+ conferences, 60+ flights, xx blog posts, 415 million users, 342 A/B tests, 1465 commits, 231 Gitlab tickets, 111827 messages on Slack.

What I’m not going to miss from my time at TNW:

  • The most ridiculous PR pitches from companies that aren’t relevant to TNW 😉
  • Product & startup pitches in my inbox that aren’t relevant or not ready for the scale of TNW 😉
  • Endless analysis on what content is supposed to attract more engagement + traffic 😉 – Still haven’t found the answer if your curious.

What I’ll be missing though:

  • The great opportunity that I got from Boris & Patrick in having me built out the marketing team.
  • Very passionate people trying to improve TNW every day just a little bit more.
  • A great team, that I’ll truly miss working with.

Overall, I’m ready for the new challenge at Postmates and very enthusiastic about working with a new (growing) team and trying to reach world domination. If you’re around in the future, please let me know. I’ll definitely be sticking to my current strategy: trying to meet with great people across the industry to learn from. So, drinks on me!
If you want to reach out to me, you can find me at: martijn.scheijbeler@gmail.com


Why giving back is so important, help out

Getting more experience can be hard when you’re just starting with your career. You’re either trying to hope to get into an internship, your first job or if you’re making a career move you just need the experience to keep up. But you’re living in a great time to get there as there are many options available to you to get started, which would almost deserve it’s own blog post. Today I’d like to talk about the idea of giving back to help you get more experience.

Giving back, even if you’re more experienced, is really important in my opinion. There are way too many NGOs, foundations that don’t have the resources (time, tools, people, knowledge, budget) to hire the experts in the field of SEO, Growth, Analytics and CRO. But there are many more opportunities for all of us to give back (our time). I’d like to highlight two initiatives that I’m working with in helping them out but that allow you as well to participate:

  • Analysis Exchange: Started by Analytics Demystified to give NGOs the ability to hire a mentor + student to help them with their web analytics projects. The perfect opportunity for people starting in web analytics to get taught by an industry expert in improving their skills. But also a great way for NGOs to get the help from two people during a couple weeks to help them out with their questions.
  • MeasureCamp (Amsterdam): A great initiative that MeasureCamp Amsterdam is starting during the next edition is reserving 8 time slots for a foundation/NGO to help them out with ideas from 100+ experts in different areas. That’s basically a hundred hours that day from top experts in the area of data, analytics, SEO and CRO that will get contributed and should help the chosen foundation in improving their web presence.

How are you helping!? What great initiatives am I missing that you & I could contribute our time + knowledge to?


Introducing: the Google Tag Manager & Google Analytics for AMP – WordPress Plugin

Today it’s time to make it easier for sites running AMP on WordPress to track & measure their web site traffic. Over the past weeks I’ve been working on a WordPress Plugin which supports adding Google Tag Manager and Google Analytics to your AMP pages. As AMP itself is quite a hard new platform to get completely right, I thought it was time to make it easier for developers, marketers and analysts to do more with the platform.

Over the last year AMP had as massive increase in support and their WordPress plugin is rising to the charts of the most downloaded plugins on WordPress but overall the support for web analytics & tag management is lacking. The documentation is available, but as the abilities within WordPress are still limited I thought it would be pretty straight forward to come up with a plugin to enable users to start using GTM & GA on AMP pages.

So that’s what this plugin will help you with, the ability to add Google Analytics or Google Tag Manager to your pages, start tracking more advanced features (outbound click tracking) and provide 10 (for now) custom variables that you can use as custom dimensions.

You’re curious how to start? Check this out:


Moving away from onclick events, data attributes it is

Onclick attributes are amazing to easily sent some data to the dataLayer. It will make sure that on every click the data is being sent, let’s for example look at this:

<a href=”http://example.com” onclick=”dataLayer.push({‘eventCategory’:’Click’, ‘eventAction’:’This Link’, ’eventLabel’:’Anchor Text’});”>Link</a>

Looks valid right? It is and it will work for you. But the problem is that with external links the dataLayer push won’t hit the browser + Google Tag Manager before the browser moves on to the external link (not taken into account links that open in a new tab/window). Which means that you’ll lose the onclick event in Google Tag Manager and won’t be able to record it.

But there is a great alternative, which we switched to for The Next Web when we found out this was happening. Data attributes can be very useful instead. If you’d use them like this:

<a href=”http://example.com” data-eventCategory:’Click’, data-eventAction:’This Link’, data-eventLabel:’Anchor Text’”>Link</a>

Why this works? The link attributes on click will always be sent automatically to Google Tag Manager. So making sure that the data attributes are part of the link and not in a separate dataLayer.push will improve it.

How about React/Angular? With these platforms there is obviously never an actual refresh of the page happening and most likely you can still send a dataLayer push when you’re updating the state of the application. That’s why in most cases you won’t have to go with the technique described in this blog post. Lucky you!

Google Tag Manager

Next step is to create a variable in Google Tag Manager for all three attributes:

Make sure to set the default value to undefined. In this case Google Tag Manager won’t take any actions in the next step if there is no data, which will be defining the Trigger. Repeat this two more times for the Event Action and Event Label variables that we need to use in our Tag.

This trigger will check all incoming links, with a data-event-category attribute that doesn’t equal undefined and that has any values. In the next step for your Tag you can use all three variables that you’ve defined before.

You’re all set! You can now use data-attributes for your links to make sure that you won’t lose any data when the browser isn’t ready to receive dataLayer pushes and you just made your tracking a bit more efficient!