Exporting Amplitude Data to Google BigQuery

Exporting Amplitude Data to Google BigQuery

I’ve written about using Amplitude before on this blog (in Dutch), but what if you want to combine the huge amount of data that you have in Amplitude with your other big data. The Enterprise version gives you the ability to export your data to a Redshift cluster. But a lot of companies these days are also on Google Cloud Platform and want to use Google BigQuery, which is similar in its setup to Redshift.

Amplitude

The Export API from Amplitude lets you download all your events data (regardless of your account plan) for free. There are some limits to the data that they can export but most startups/companies that use them are likely to stay under that on a daily/hourly basis. Which means that you can export the data. So basically everything that you need to do is setup a cronjob that either every hour or day can retrieve the data. It will parse the data and prepare new files that will be temporarily stored in Google Cloud Storage (this will ensure that the data is easier/faster available for upload into Google BigQuery). The next step is loading the data from GCS to GBQ.

Google BigQuery

Over the last years I wanted to do more with Google BigQuery and this project was perfect for it. As BigQuery has many connectors with multiple Google products (Analytics, Data Studio) and other vendors like Tableau it should give companies the ability to analyze their data and connect to other sources.

Schemas

Within Google BigQuery we’re going to save the data in two tables:

  • Events: As everything is an event in Amplitude that’s also one of the tables that you’ll need to use in Google BigQuery, that’s why every event will end up as being it’s own row in Google BigQuery.
  • Properties: Every event can have properties in a few different ways: events, users, groups, group properties and an actual data property. We are connecting them to the data from the events table.

FAQ

  • Do I need to be a paying customer for Amplitude? No, you don’t. The free plans, which I started to use this on will have the support for the Export API as well.
  • What is the cost of the integration? The costs that are associated with this setup are related to Google Cloud Platform. You’ll have to pay for storage in Google Cloud Storage and the storage of Google BigQuery. For the setup that I’m running we’re saving millions of rows monthly and the costs are in total less than 10 dollar.
  • What do I need to do to get this up and running? Check out the README in the repository on Github, it will give you a getting started checklist to ensure that you can run this script.

 

Feedback? / Contribute?

I haven’t been the first person to work on an integration with BigQuery. I’ve seen other solutions for Java and Python, but they all work a bit different. If you have any feedback on the setup, leave an Issue on Github, submit a Pull Request with your proposed changes. In the end, I can code, but I don’t consider myself to be an engineer 😉

Sitemaps; Setup, Monitoring & Metrics for Analysis

Sitemaps; Setup, Monitoring & Metrics for Analysis

Last updated on March 20th, 2018 at 05:14 pm

In my effort to write longer posts on a specific topic I thought it was time to shed some light on something that we’ve been working on during the last months at Postmates and something that I never thought of as a topic that could become interesting: sitemaps. They’re pretty boring in itself, it’s a technology where you give search engines basically all the URLs for a site, that you want them to know about (indexed) and you take it from there. Even more so, as most sites these days run on a CMS like WordPress where tons of plugins can take care of this for you. Don’t get me wrong, do use them if you are on one! But as I work mainly for companies that don’t have a ‘standard’ CMS I worked multiple times on creating sitemaps and having their integrations work flawless. Over time that taught me a ton of things and recently we discovered that certain additional features in the process can help speed up the process. That’s why I think it was time to write a detailed essay on sitemaps ;). (*barf: definitive guide).

TLDR; How can sitemaps help you get better insights, how to set them up?

  1. Sitemaps will provide you with insights on what pages are submitted and which ones are indexed.
  2. You create create sitemap files by uploading XML or TXT files with dumps of URLs
  3. All your different content on pages can be added to sitemaps: images, video, news.
  4. Different fields for priority, last modified and frequency can give search engines insights in the priority for certain URLs to be crawled.
  5. Create multiple sitemaps with segments of pages, for example by product category.
  6. Add your sitemap index file to your robots.txt so it’s easy to find for a search engine.
  7. Submit your sitemap and ping sitemap files to search engines for quick discovery.
  8. Make sure all URLs in your sitemaps are working and returning a 200 status code, think twice: do you all want them to be discovered?
  9. Monitor your data and crawls through log files and Google Search Console.

Goals

When you start working on sitemaps there is a few things to keep in mind. The ideas that you have around them and the goal: what problem that you have are they solving? For small sitemaps (100 pages) I’m honestly not sure if I would support sitemaps. There is probably a lot of other projects that would have more impact on SEO/the business.

If you’re thinking about setting up sitemaps there is a few goals that it will help you accomplish:

  • Get better insights into what pages are valuable to your site.
  • Provide search engines with the URLs that you want them to index, the fastest way to submit pages at scale.

Overall this means that you want to support the best sitemap infrastructure you can as that will help you get the best insights ever, the quickest way to get these insights and most of all get your pages indexed + submitted as fast as possible.

Setup

Sitemap

Format? XML/Text? Does the format matter, for most companies probably not as they’re using a plugin to support their sitemaps. If you want to go more advanced and get better insights I would go with the XML format myself. From time to time we’re using text file sitemaps where we just dump all the URLs. They’ll help in getting you a sitemap quick and dirty if you don’t have the time or resources quickly.

Types: There are multiple formats for sitemaps to support different content types.

  • Pages: In there you’ll dump all the actual URls that you have on the site and that you want a search engine to know about. You can add images for these specific pages to that Schema as well to ensure that the search engine understands what images are an important part of the page.
  • Images: For both image search as making an impact with the pages you can add sitemaps for images.
  • Videos: Video sitemaps used to have a bigger impact back in the days as the video listings were a more prominent part of the search results page. These days you mostly want to let search engine know about them as they’re usually part of an individual page.
  • News: News is not really its own format as they’re just individual pages. But Google News sitemaps do have their own format. Creating a News Sitemap – Google.
  • HREFLang: This is not really a type of content but it’s still important to think about. If your pages have a translated version, you want to make sure they’re being listed as the ‘duplicate’ version of that. Read more information about that here in Google’s support.

Fields

  • Frequency: Does the page change on a regular basis? Some pages are going to be dynamic and will always change. But for some of them they will change only daily, weekly, monthly. It’s likely worth it to include this as a good signal in combination with the Last Modified field and the header.
  • Last Modified: We do want to let a search engine know what kind of pages have been updated/modified and which ones aren’t. That’s why I’d always recommend to organizations that they should include this in their sitemap. In combination with the Last Modified header, we’ll talk about that in the next step it will be a good enough signal to assess if the page has been modified or not.
  • Priority: This is a field that I wouldn’t spend too much time thinking about. On multiple occasions, Google has mentioned that they don’t put any value or effort into understand this field. Some plugins use it and it won’t hurt. But for custom setups it’s not something that I would recommend adding.

Last Modified

Has the actual sitemap changed since the last time it’s been generated? Yes or No? In some cases your sitemap won’t change. You didn’t add any new products/articles. Have you ever run this in your terminal:

curl -I https://www.example.com/sitemap/sitemap_index.xml

Look at the headers, if you see a Last Modified header, it will be a signal to see when the page has been last modified. We use it to tell the last time it was updated. We combine this with serving a Last Modified Header at the URLs that are in the sitemaps. Sometimes this won’t always work as pages can change momentarily (based on availability of products for example).

Segmenting Pages

For better insights it’s really useful to segment your sitemaps. The limit per sitemap is in the end 50.000 URLs, but there is basically not a required minimum. The way you’ll see sitemaps being segmented is in multiple ways. Based on these you can get more segmented insights, is 1 category of pages better indexed then another one.

Categories: Most companies that I work with are segmenting there pages by the categories they’ve defined themselves. This could be based on region or for example by product categories for an ecommerce site.

Static Pages: Something that most people with custom build sites don’t realize is that there is usually still a ton of pages that aren’t backed up by a database that you you want insights on too. Think about: contact, homepage, about us, services, etc. List all these pages in a different sitemap (static_sitemap.xml) and include this file in your sitemap index too.

Sitemap Index

If you have multiple sitemaps (10-25+) you want to look into creating a sitemap index file, with this you can just submit 1 file and with that the search engine will be able to find all the underlying files that are part of the sitemap. This saves you adding multiple sitemap URLs to Google Search Console/Bing Webmaster Tools and will also give you the ability to add only 1 line to your robots.txt file. In the end it’s another sitemap technically which lists all the different URLs of the other sitemaps.

Robots.txt

You want to make sure that on first entry a search engine will know about your sitemaps. Usually one of the first files a search engines’ crawler will look at is the robots.txt file as it needs to know what it can/can’t look at on a site. As we just talked about the sitemap index, we’re going to list that one in the robots.txt file for your site which should live on https://www.domain.com/robots.txt. It’s just as simple as adding this one line to it:

Sitemap: https://www.domain.com/sitemap/sitemap_index.xml

Obviously the URL can be different based on where you have hosted your sitemap index file.

GZIP

If you’re a big site you likely have servers that won’t go down and can take quite a hit but if you have extensive sitemap files they could easily get up to +50MB that is not a file transfer that can be done in a matter of two seconds. Also it can just slow down things on both your end and the end of the search engine. That’s why we’ve started GZipping our sitemap files to make for a faster download and speed up that process, at the same time you make it 1 step more complicated for people to copy paste your data.

PING Search Engines

Guess what, it has an affect. I thought it was crazy too, but we found a tiny bit of proof that actually pinging a search engine will result in something. As you mostly will likely only care about Google and Bing we still have a way of letting them know about a page:

Submit your sitemap

Probably not worth explaining, you need to make sure that you can get insights into your XML sitemaps and the URLs that are listed in there. So make sure to submit your sitemaps to Google Search Console and Bing Webmaster Tools.

Pubsubhubbub

One of the projects that is very unknown is the PubSubHubbub project, it will let, mostly publishers, be instantly notified (through a specific push protocol) when new URLs are published in a feed. This protocol works through an ATOM feed (do you still know about that protocol?) that you provide. Once you have registered the feed with the right services you can make it easier for them to be notified of new pages.

XSLT

XML Sitemaps aren’t easy to read for a regular person. If you’re not familiar with the format of XML it might be uncomfortable. Luckily a while back people invested XSLT. This will let you ‘style’ the output of XML files to something that is more readable. This would make it easier to see certain elements in the sitemaps that you’ve listed. If you want to make them more readable I would advise looking into: https://www.w3schools.com/xml/xsl_intro.asp.

Quality Signals

Search engines like sites that are of high quality. The pages are the best, the URLs are always working and your site never goes down. Chances are high that all of this doesn’t always apply to your sitemaps as some pages might not be great. Some things to consider when you’re working on this:

  • 301/302/404: Are all URLs in your sitemap responding like they should with a 200 response? In the best case scenario none of your URLs should be responding with another response code then that. In reality most sitemaps always contain some errors.
  • NoIndex: Have you included URLs in your sitemap that are actually excluded by a noindex meta tag or header? Make sure that it’s not the case.
  • Robots.txt: An even bigger problem, are you telling the search engine about URLs that you actually don’t want them to look at?
  • Canonical Pages: Is the actual URL that you’re listing the canonical URL/original URL or are you listing the pages that are still ‘stealing’ the content from another page, like a filter page. Do you really want to list these URLs in your sitemap?

With all of these signals, some might have a big/small impact others won’t matter at all. But at least think about the implications that they might have when you’re building out your sitemaps.

Airflow

Lately I’ve been working a ton with Apache Airflow, it’s the framework that we use at Postmates, invented by the great folks at Airbnb and mostly use for dealing with data pipelines. You want to do X, if X succes you want it to go on to task Y. We’re using that for the generation of sitemaps, if we can generate all sitemaps we want to have them pinged with the search engines, if that succeeds we want to run some quality scripts, if that is done we want to be notified on both email and Slack to tell us at what time the script succeeded.

For some sitemaps we want it to run everyday, for a specific segment we want to have it run on an hourly basis. The insights from Airflow will give us the details to see if it’s failing or not and will notify us when it succeeds/fails. With this setup, we have constant monitoring in place to ensure that sitemaps are being generated daily/hourly.

Monitoring

Eventually you only want to know if your pages are of good enough quality that they’re being indexed by the search engine. So let’s see how can see this in Google Search Console.

Index coverage

A useful report in Google Search Console is the Index Status report (Google Index > Index Status). It will show for the property that you’ve added how many pages have been indexed and what pages have been crawled. As the main goal for a sitemap is driving up the number of pages being submitted for the Google index the following step is making sure that they’re being indexed. This report will give you that first high level overview.

Sitemap Validation: Errors & Amount of URLs

But what about the specifics of the sitemap, are the URLs being crawled properly and are the URLs being submitted to the index. The sitemap reports give you this level of detail (in this case 98% is indexed, which makes sense, the 2% missing are some products that were test ones that Google seemed to have ignored, luckily!). Remember what we talked about before regarding segmenting your pages? If you would have done that you would have seen in this particular example what percent of pages in that sitemap was submitted / indexed. Very useful if you work on big sites where the internal link structure for example is lacking and you want to push that. These reports can (they not always) give you insights into what the balance could be between them.

Quality Assurance

  • Are the URLs working (200 status code)? An unknown fact, but Google doesn’t like following redirects or finding broken URLs in your sitemaps. Spend some time on making sure that these pages aren’t in there or add the right monitoring to prevent it from happening. Since we’ve starting Gzipping our sitemaps that’s become a tiny bit harder as you first need to unpack them. But for quality testing we still have scripts in place that on demand can run a crawl of the sitemap to see if all URLs in there are valid.
  • Page Quality: Honestly, is this page really worth it to be indexed in Google? Some pages are just not of the quality that they should be and so sometimes you should take that into account when building up sitemaps. Did you filter out the right pages?

Metrics & Analysis

So far we’ve talked about the whole setup and how to monitor results. Let’s go a little step further before we close this subject and look at the information in log files. It’s a topic that I became more familiar with and have worked closely with over the last months too:

Log Files

As log files can be stored on the web server that you’re also using for your regular pages you can get additional insights into how often your sitemaps are being viewed and if there are any issues with sitemaps. As we work on them on a regular basis it could be that they break. That’s why we make sure that for example we monitor the status codes for the URLs so that we can see when a certain sitemap doesn’t hit a successful 200 status code.

Proving that pinging works

A while back we started to ping our sitemaps to Google and Bing, both make it clear (Google) that if you have an existing sitemap and you want to resubmit it this is a good way to do it. This sounds weird, Google got rid of their ‘submit a URL’ feature for the index years ago. So we were skeptic to see if this had any impact. As it was really easy to implement, you just fire a GET request to a Google URL with the sitemap URL in there. What we noticed is that we saw Google almost immediately try to look at these URLs. As we refresh this specific sitemap every hour, we also ping it every hour to Google. Guess what happens, every hour for the last weeks they look at the sitemap by now. Who says you can’t influence crawlers? Result? If you want to ensure that Google is actually looking at a page and actively crawling it, pinging seems to prove that, that is actually happening.

Screenshot of this from a Kibana dashboard where we log server requests

What if you can’t ping? Usually I would only recommend pinging a search engine if your whole sitemap generation process is fully automated, it doesn’t make sense to open your browser or have a tiny script for this. If you still want to basically experience the same, use the Resubmit button in Google Search Console > Sitemaps to achieve the same.

Future

This is not all of it and I’ve gone over some topics briefly, I didn’t want to document everything as there’s already a ton of information from Google and other sites about how you can specifically setup sitemaps. In my case, we’re on a route to figure out how we can make our sitemap setup near perfect, what I’m still wanting to investigate or analyze:

  • Adding a Last Modified Header to pages in the sitemap, what is the effect of pinging a sitemap and Google looking at all pages or just the ones that are modified?
  • Segmenting them even further, let’s say I only add 100/1000 pages to a sitemap and start creating just more of them, does that influence crawling, do we get better insights?

Resources

You want to learn more about sitemaps, look into the following resources to learn more about the concept, the idea behind it and the technical specification:

Next steps?

When I started writing I didn’t plan to have this become everything I know about sitemaps. But what did I miss? What optimizations can we apply to sitemaps in order to get better insights, speed up the crawling of pages. This is just one of the areas of technical SEO but probably an important one if you’re looking for deeper insights into what Google or Bing think about your site. If you have questions or comments, feel free to give a shout on Twitter: @MartijnSch

Using Amplitude for Product & Web Analytics

I’ve previously published this blog post in Dutch on Webanalisten.nl.

What if you are looking for a product for web analytics but have a lot of events, a complicated product and sending more and more data over time. Sometimes it wouldn’t just work to go with Google Analytics (360), Adobe Analytics and maybe integrating your custom build solution or Snowplow might be too complicated for your organization. In this post I’d like to show you another tool that might be interesting to you: Amplitude. I’ve been working with it for the last year and it provides some great flexibility over other products in the industry.

What is Amplitude?

“Analytics for modern product teams”

Or in normal language, you can track any event and connect that to a user. All the events that you send can have properties, just like the user can have properties. You can filter by all these data points and move your data around to turn it into multiple types of charts: funnels, event hits, revenue, etc. In this article we’ll be running through how you can be using Amplitude and what it’s good for. Let’s go ahead and dive in!

Why would you be using Amplitude?

You want to measure what is happening within your product and what users are doing. Keeping in mind that all this data can help you improve their workflows and measure the impact certain changes have on their behaviour. In that aspect, Amplitude is directly competing with mostly tools outside of web analytics, like: Google Analytics for Firebase, Snowplow, KISSmetrics, Mixpanel, etc. In the next section we’ll explain why, as a lot of features are interpreted differently from regular web analytics but can still help you a lot in your daily work:

What’s the difference?

 

  • Instant Reporting/DIY: While most analytics tools provide you with a lot of pre configured dashboards. Amplitude will let you do this all on your own, which can be a time consuming task but in my opinion it also lets you think a bit more about the way you set-up your analytics infrastructure.
  • No default metrics: Bounce rate doesn’t exist as any event can be triggered to influence it (plus, would that be your most useful metric anyway?)
  • Funnels: Anything can be a  funnel, in my opinion that makes it very powerful as it doesn’t require you to create any predefined goals and also will make sure you can create funnels retroactively (recognize this pain point in Google Analytics?). If you added events a few weeks ago and now is the time to start creating a funnel of it, you’re able to. Want to change the funnel and add more/less events. You can.

 

 

  • User/Session: Sessions (on Web) don’t really exist in Amplitude. While in my opinion this is a metric that has a very loosely defined definition anyway it can come in handy from time to time to measure retention. It will provide this data on mobile where sessions are way easier to be defined (app open/close).
  • Channel/Traffic Source: If you’re looking for an easy way to track where your visitors are coming from with detailed reports that would be associated with costs. That’s just not what Amplitude is for. While it can easily save the UTM data that you’re sending along it won’t offer your great reporting around it. That’s why there focus is mostly on product analytics.
  • Merging Events/Linking Events: At the beginning of this section we talked about the need for setting up all the dashboards yourself. As you won’t have a very defined plan with your tracking for what’s to come over the next few years it can be hard to follow a certain naming convention from scratch. Usually turning your analytics data into an unstructured chaos over time. Within Amplitude you’re able to merge certain event names and link them together. So you can easily change your old event names to something new and still connect the data together. One of the features I really miss sometimes in other tools when I’m trying to redefine naming conventions and clean them up.

Why data governance is even more important

The role of data governance is becoming more important by using tools like this in combination with having the need for good documentation. If you come into an organization that is already sending hundreds of different events it can be really hard to get started with making a more deep analysis as you’re not always familiar with the

    • Naming conventions: You want to make sure that you’re using the right names for the events and that you’re making sure that their logical in order to send the data. It would be good to give this article, on creating measurement plans, that I wrote for online-behavior.com a read. We’ll talk later about how Amplitude can still help you if you would like to make changes to the events you sent.

 

  • Segments/Cohorts: As most of the data for users can be saved in event or user properties this will also mean that you need to make sure that the data in there doesn’t change too often as it might affect how you’ve set up your segments and cohorts.

 

  • Also funnels and many reports can be impacted by the way you save data.

Overview of features

  • Dashboarding/Charts: As we talked about the flexibility that Amplitude can provide you with, this mostly shows in the way you’re working with charts and adding them to dashboards. You can create dozens of different charts and add them to a dashboard. The dashboards will then, for example, give you the ability to change the date range. If you don’t like that you can still make all the changes from the actual chart.
  • A/B Testing – Significance Calculator: Are you running an A/B test on your site and sending the data to Amplitude. Within certain charts you can segment out the results and immediately calculate if they’re significant for what you’re analyzing. Saves time trying to find a significance calculator.

      • Custom Metrics: Just as many other web analytics tools, Amplitude will give the ability to create custom formulas within a chart to calculate other metrics.

     

  • Retroactive reporting: You have added tracking months ago but only today you’ve figured out that an event should be measure as a goal? You can set up a goal funnel really easily with old data and have all the old data being available to you.
  • Realtime: The fact that all of the events that you send to Amplitude are processed in real time makes it very powerful. Basically within minutes of launching a new feature you can start analyzing the data to see what’s going on or if it’s broken. Gone are the times were you need to wait for hours to have all the data that you are collecting be fully available.
  • Unlimited event/user properties & ‘dimensions’: Every event can have properties that can be related to the event. In addition to that a user can have properties that can be used too. So if I want to mark certain users with an action I can easily send that along with an event to update the records for this.
  • CLTV: Measuring the lifetime value of users will obviously require you to start identifying users (relatively easy to set up). But this will enable you to look into how you’re users are performing over time and if you have high retention for what that means for their customer lifetime value. This is an example report that would provide me with the performance of a segment of users over the last 12 weeks and what they’re worth to the business.
Chart for CLTV

What’s missing?

Google integrations? Obviously some things are missing, while the Cohort feature’ abilities are very powerful and Amplitude can provide you with some cool integrations with other software it still can’t make the connection with the audience data from Google. Which is obviously always going to be a big upside of the Google Analytics Suite.

Transactions/Purchase: The way Amplitude is tracking a conversion is a bit weird. You send all the products that you purchase as different revenue events. There is no concept of a purchase, which seems strange. Also it’s really hard to identify what the Xth purchase was, these are events that you need to setup yourself.

UTM/Traffic Source Reporting: It does exist but it isn’t great and definitely not as powerful as you’re used to in your regular web analytics tools. Does it get the job done for product analytics, Yes it does I’d say. If you’re looking to do anything more advanced with the data you should be building additional capabilities on your own end.

Use Cases

  • Funnels: Every event can be part of a funnel and that makes it very flexible and useful if you want to compare user behaviour. For example connecting certain user actions before a purchase funnel can be the case too.
  • Customer Lifetime Value/Retention:
  • Cohorts: Where you would have segments & audiences in Google Analytics you have the ability to also create cohorts of users to measure the impact of certain properties/events on their behaviour over time. For example this is a cohort that we used often at Postmates where we would look at what users that have come in with the sign up referrer that includes google, yahoo, bing (an organic search user). We would use this cohorts either to export them from other marketing purposes (email/push campaigns) or to analyze their behaviour against other cohorts.
    • How do organic search users in the last month behave different if they have used x feature?
    • How do users who have touched feature x find the other features?
Segmenting users with its Cohort feature.

Conclusion

Overall I’m pretty satisfied with Amplitude and how it can help you with its flexibility in adding/creating events and figuring out later what kind of dashboarding/charts you’ll create on top of this. But it’s likely (for now) not going to replace most of the data that you’re used to in web analytics as that would require a lot of additional setup and tracking. You can use it very effectively within organizations to track certain aspects and user behaviour. All in all a great addition to most analytics teams. All in all I would advise most companies to use these tools together as they can be very useful in providing more insights into what the user is doing specifically.

If you’ve worked with Amplitude and want to share more about your experiences, leave a comment! Currently I’m working on exporting Amplitude data to Google BigQuery for ‘big data’ analysis, in a future post I hope to share with you on how you can set that up yourself.

Dealing with SEO within your company / internally

Dealing with SEO within your company / internally

Last updated on March 16th, 2018 at 07:30 pm

“My company/manager/CEO doesn’t understand SEO, my engineers have no idea on how to implement X, I don’t get the buy-in that I need.” Just some of the comments that I hear in real life and see pass by on Twitter. That’s why in late 2017 I asked this question on Twitter. So that’s why I thought it would be time for a write up on the scenarios that I’ve seen over the last few years in SEO and the other ways that I’ve seen that help for getting input. I’ll try to share some insights into how I’m/we’re dealing with explaining and dealing with SEO internally.

Is there any believe in SEO?

For a successful SEO strategy and great results the first thing that you’ll need is somebody in the organization needs to support SEO. If that person or role isn’t there it’s going to be really hard to get things moving forward. But how do you know that support is there. In my case I’ve been lucky, my first job at Springest I had a guy who started a SEO agency himself, TNW was such a big online tech player I didn’t even need to explain what SEO was and these days at Postmates I got specifically asked if I wanted to focus on SEO when I joined. All with the believe that SEO would help move the business forward. But in the end you don’t always need to have people that know what SEO is and know everything inside out. It will help if they’re able to help you out when you have questions of SEO and at least know the good and bad parts about it. If the person that’s asking you to help out is also talking about link buying I would probably reconsider my decisions to work for them a few times  (and probably decide not to).

Explaining the value

In most organizations you’re going to need to explain the value of SEO, in the end it remains a black/dark grey box in which it’s hard to explain what kind of impact you can deliver on. But what you can explain in my opinion is the following:

  • What is the opportunity, how big is your industry/nice in terms of search volume?
  • What is our current position? How good/bad are we performing against our competition?
  • Who’s our competition really? It’s most likely not the companies that offer the same product but probably the ones that are beating your *ss in the search results.
  • Increase conversion rate while they’re at it, increase awareness while they’re at it, increase referral traffic while they’re at it.

These are just a few that additional values that you can bring to the table as an SEO (team) in a company. Most of all, if you do a great job on keyword research you can tell your internal organization a lot about the keywords and intent of the users within your industry/niche.

But how do you proceed, to excel even more. In the end you want your whole organization to be supportive and help the cause of SEO. The more people that work on it the more you can hopefully grow your visibility and with that your relevant business metrics (clicks, leads, sign ups, etc.).

Creating Buy In

What kind of support do you need, why do you need it and what can you do to get it? Let’s talk a little bit about that:

Team Support

Does your team know what SEO is, how it can help the company and what their contribution to it could be/mean? Very often I don’t see SEOs talk with the IT team/engineers/developers or whatever job title they’ll have in your organization. The often talked about phrase: “Sometimes you should just have a beer with them to build up a good relation” most often is incorrect. You’ll build up a better relationship, that I agree with. But that doesn’t always cover an actual understanding of the problem which is still going to be essential.

Building up understanding on what you’re/they’re working on

Does your boss, his boss and the CEO what you work on for SEO? Pretty sure that they don’t. So it’s not surprising that you don’t have all the support or resources that you would need. Start educating them, on what you’re working on and what the results are. If you’re in doubt about something there are multiple paths to go: run an experiment, launch an MVP as soon as possible or create the business case/technical documentation so you’re aware that you know what you’re building.

My {fill in random job title} doesn’t understand SEO

“Why didn’t my engineer think of adding redirects?”, “Why didn’t our content team use the right keywords?”, “Why doesn’t my boss understand what SEO is really about?”. Questions that you must have asked yourself and I can’t blame you. But the answer to all of them is easy: “Because you haven’t told them”. In the end all these things matter to SEO and your success, so why don’t you explain it once more. Repetition makes it easier to have these answers printed on their minds.

What can help me build up an understanding of SEO?

Besides working with your team it’s even more important to work in a nice way with the other teams in a company. It’s very likely that there are more people outside of SEO in your company then there are working with it on a daily basis. All of these folks can help you out with SEO too. I remember the times where I was in desperate need for a copywriter after hiring dramas that continued for months and our receptionist turned out to be an English major and able to help out immediately.

Internal Decks/Meeting

The previous two companies that I’ve worked for were relatively small (<75) and as I was an ‘early’ employee at both I saw a lot of people come in. In the end that made it easier to explain SEO to them as I either hired them or they had a manager that I already worked with within the organization. At Postmates that is quite different, I came in years after founding the company at the point where we had over 400+ people and a growing organization.

That’s why early on, when I was formulating the SEO strategy I started creating a slide deck explaining SEO for the rest of the organization and also telling them more about the projects that we already worked on or would be working on in the next months. Whenever a new team would be formulated or somebody would join the Growth team I tried to keep up with setting up a meeting with them and seeing if there would be any overlap or room to work together. In the end your Comms teams, Support teams have probably some interest in SEO or you can help them with their work with the tools, resources and/or products that you have available.

Weekly Status Meeting/Email/Update

When you work with multiple people on a team it’s hard to keep them all up to date with everything that is going on. Regular status updates, either in person or via VC, email can help with that. As we have multiple engineers working on SEO at the same time they can already get behind on what’s going on easily. That’s why on a weekly basis we send out updates on the work that they’ve done to the organization but also to the the bigger team. With this it’ makes it easier to show progress, list down what we’re planning on working on and provide early results. So things that we list in the email are:

  • Experiments started/finished: Do we have any results, or what did we launch this week?
  • What did we do last week? What are the tickets that we worked on, what kind of early results do we have, is this already being picked up by a search engine?
  • What are we going to work on this week? What issues will we work on, what kind of results might that provide, why do we work on this.
  • What did we learn last week? What kind of results die we see, what kind of growth did we achieve.

Monthly Updates

On top of that we send some headlines once a month for the bigger projects that we launched so we know what kind of progress we have made for the quarterly targets that we have. This will give a more birds eye view on what we’re achieving and if that’s on track with what we planned upfront. It’s a similar update to the Monthly one, but a bit more high level and readable for the whole team and people that don’t work with SEO on a day to day basis.

What’s next?

This is still not good enough, even when you have internal support you always have new questions rise and even when they all support you it’s probably going to happen that they start asking deeper questions that you need to keep explaining. This is something I now endure, questions basically are more relevant to your own work. Which is awesome, as it makes up for great debates that in the end will only improve products and SEO strategy. Even better, this will all help streamline the process of SEO and usually speed up the output.

What is SEO Experimentation?

What is SEO Experimentation?

Last updated on November 4th, 2018 at 10:45 pm

If you’ve been reading some of my blog posts in the past you’ll have noticed that I worked a lot on analytics, experimentation, and SEO. Having been able to combine these areas together has led to the point that for both Postmates and The Next Web previously, we worked on setting up a framework for SEO Experimentation. In my opinion, over time you’d like to learn more about what Google appreciates and how it can help you future wise, to think about what you should be focusing your attention on == prioritization. Lately, I read a blog post on the SEMrush blog with the title: SEO Experiments that will blow your mind (clickbait alert! so I won’t link to it). Hoping there would be a lot of great ideas in that blog post I started reading realizing that over 50% of examples weren’t even close to being an experiment. They were just research, from over a year ago (which was alarming too!).

Which pushed me to write this essay to tell you more about what SEO experimentation really is and how you can learn more about it as it’s still a relatively new area in SEO that is only been explored and exposed by a few big companies.

Update: since the original publication date I’ve become even more excited about SEO Experimentation and its possibilities. That’s why I’ve updated this post with more information and some frequently asked questions that I get about the subject.
Last update: November 4, 2018 – Added an additional resource from the Airbnb Team.

What really is, SEO Experimentation?

You’re testing 2-X variations of a template on your site (all product pages, all category pages) and measure over a period of time what the impact is in organic search traffic of the pages (usually a template) that have seen changes in your experiment. You want to isolate your changes as much as possible, you set a certain level of significance and you calculate in a mathematical way how your results have changed in the period before and after the change.

It’s not:

  • Research (I  compare data between periods too, but it doesn’t make it an experiment).
  • Guesswork: I’ve seen my pageviews go up with X% after optimizing my title on this page.

I would encourage you to also read this post on Moz by Will Critchlow in which he shared how they built Distilled ODN (we worked with their platform when I was at TNW) and how you can be testing with SEO yourself.

How is SEO Experimentation different from user testing?

Measurement: Instead of measuring conversion rates or other business metrics you’re mostly focused on tracking the number of sessions that are coming in from organic search. If the variants that you’ve worked on increase you’ll start calculating the impact on this. On the side, you’re still making sure that the conversion rate of these pages doesn’t decline as, in the end, it will still be the business metrics that will count.

Bucketing: Instead of bucketing users and sending user A into bucket A you’re doing this with pages. In the end, you want to make sure that the same pages (just like you do with users) end up in the same bucket. How you usually do that is that you sort by alphabet or category to have some kind of logic. What’s important though is to make sure that these buckets are in balance. Which is harder than you would do with user testing.

The difference between bucketing for users experimentation and SEO experiments
Bucketing works differently with SEO Experimentation

What are some examples of SEO Experiments?

Examples of things that you could be testing? Let’s list a few examples that I can think of:

  • Updating Title tags: does a certain keyword help, do I need to make them longer/shorter?
  • Adding/Removing Schema.org, what is the impact of adding structured data?
  • Adding content: does adding more content to a page help with its ranking? What if I add an additional heading to my content.
  • Internal link structures: do you want to add more relevant links to other pages?
  • Testing Layouts, what layout does help better for the SEO on this page? Ever noticed why Amazon has so many different product pages 😉

How do I start doing SEO Experimentation?

Let me give you a very much shortened and a simplified idea of how an SEO experiment works:

  1. Think about what you want to be testing (adding descriptions, setting a canonical), write down a hypothesis, we’ll talk about this shortly. 
  2. Figure out if you have enough traffic (500+ per variation per day) would be a minimum I’d say.
  3. Figure out how you can serve different content/HTML per page. Are you able to serve different areas on your site based on a randomized factor, still making sure that your buckets are just as big (50/50)?
  4. Setup the experiment and let it run for as long as needed to see valid results (usually at least 3-4+ weeks). You want to make sure that Google has picked up on all the changes on your page and has re-indexed the impacted pages.
  5. Analyze the results after this period and look at how these buckets of pages performed before your experiment and after. Are these results valid, you didn’t make any other changes to these pages in the meantime? Are the results per variant significant?
  6. You have found the winner + loser is. Time to start iterating and launch another experiment.

How to document an SEO Experiment?

You want to make sure that what you do around SEO experimentation is well documented, this will help you in the future with figuring out what kind of experiments are working and what you learned. When you can run over 25 experiments a year you probably won’t know after a year how many of these were successful and how to interpret the results. For documenting SEO experiments I’ve created a template that we filled in with the data on the actual experiment.  You can find it here and copy it in your own Google Drive for own use:

How to analyze an SEO Experiment?

You want to make sure before an SEO experiment is running that you know what has happened with it before it starts. It’s basically the ‘control’, you want to make sure your bucket is providing stable results so you can clearly analyze the difference when the new variants are being launched.

Creating buckets for SEO Experimentation
Bucketing needs to ensure that there are additional buckets so you can measure the baseline and take care of anomalies in your data sets.

Bucketing: Your bucketing needs to make sure that it’s sending the right variant to your analytics account (most often used: Google Analytics). When the experiment starts, make an annotation so you know from when you start analyzing the results.

Logs: Logs can come into play when you start analyzing the results. In most cases, your experiment won’t generate results in the first week as the changes in your variant haven’t been picked up in the experiment. That’s why you want to look at your log files to ensure that the new variants have been crawled.

Measuring & Analyzing impact: For measuring the impact you’re segmenting down the buckets and measure what happened before the experiment and after. To see if the changes are significant or not, you need to rely on the CausalImpact library to see what has happened or not. You want to send the data for different buckets in a way that can be visualized like this:

Sending dataLayer events for measuring SEO experiments
Send data about the buckets and elements (de)activated to web analytics

Anomalies: Analyze your buckets individually! Do you see any spikes in traffic that might be hurting the data quality? You need to be sure that is not the case. For example, what if one of the buckets contain pages about political topics that all of a sudden see a spike in search volume. This doesn’t mean that your page has been performing better, it means there was just more demand so the data for that variant might be invalid.


Examples of SEO Experiments

As I mentioned both at The Next Web and Postmates I was responsible for running SEO experiments. Most of them were around optimizing <title> tags. As changes to this have, in most cases, a direct connection to the CTR within the SERPS. The title is, in the end, used as the headline for a specific search result. So let me walk you through an example of an SEO experiment as we ran it at Postmates.

Updating Titles with additional keywords

The problem: We noticed a lot of search traffic for terms around food delivery in which a zip code, like 91615 was mentioned. As we could ‘easily’ create pages for zip codes we wanted to know if that was worth it, so: “What can we do to drive more additional searches around zip codes without building new landing pages and wasting valuable engineering resources doing so”.

The solution: As we knew for restaurants in what specific zip codes they were active we had the ability to mention the zip code in the title. As we were doing this across tens of thousands of restaurants we knew that we had enough of a sample size.

  • Old:
    • {Restaurant Name} {Street Address} in ({City}} – Postmates On-Demand Delivery
    • Paxti’s Pizza 176 Fillmore Street in San Francisco – Postmates On-Demand Delivery
  • New:
    • {Restaurant Name} {Street Address} in ({City}} ({Zip Code}) – Postmates On-Demand Delivery
    • Paxti’s Pizza 176 Fillmore Street in San Francisco (97521) – Postmates On-Demand Delivery

The result: It was inconclusive, in the end, that wasn’t likely the outcome that you were hoping for. But I want to paint a realistic picture of what can happen when you run experiments. In a lot of cases, you don’t see enough changes in the data to be certain that it’s an improvement. In this case, we expected that a title change wasn’t good enough to actually compete for zip code related queries. The food delivery industry is one of the most competitive in the world for SEO so we knew it was always possible to have an outcome like this.


Frequently Asked Questions + Answers

There are a lot of questions that come up when I talk about this subject with people. So I’ll try to keep up this blog post with any new questions that might arise:

Isn’t this cloaking? Doesn’t this hurt for my Google rankings?

Not really, you’re not changing anything based on who’s looking at the page. You’re changing this only on certain pages that are being served and the search engine + user will see the same thing. Ever looked at Amazon’s product pages and wonder why they all have a different layout? Because they’re testing for both user experience as SEO.

Do you want to learn more about this subject?

Great, when we were setting up our own SEO experimentation framework at Postmates about 6 months ago I tried to find all the articles related to it and talked to as many people as possible on this. These were mostly the articles I would refer you to if you want to learn more.

Resources

When I wanted to learn more about SEO experimentation I started to figure out what was already written on the web, most of these resources are from teams & companies that I worked with before. So if you’re enthusiastic about this subject, read more here:


Let’s really start innovating in the SEO industry and let’s get rid of terrible clickbait headlines. SEO Experimentation is like I mentioned something that we should be embracing as it’s a more scientific approach that is going to lead to finding new insights. If you want to talk more about this, feel free to reach out. Like I said, I have been updating this post for over 6+ months now, and will keep on doing so in the future.

What books am I reading in 2018?

What books am I reading in 2018?

For the last two years I wrote blog posts (2017 & 2016) listing the books that I read in the past year and that I wanted to be reading in that specific year. As always, the past year I didn’t read all the books that I’ve listed out in the blog post as I discovered some new ones and changed my focus during the year. Also moving to another country (hi San Francisco!) made it tough to keep up with the goals I set for myself. So that’s why I didn’t make it to the goal to read 20+ books last year and had to leave it at 14.

So what will I (at least) be reading in 2018:

So what does this tell you? The guy wants to know more about branding in 2018 and is in desperate need for some cool new personal development books. Over the last year I read a lot of popular books (Elon Musk, High Output Management, etc.) that have provided me with a lot of inspiration on great managers + techniques. In 2018 I’d like to dive a bit more into brand building, although I have an SEO job most of what we think about everyday is building out the Postmates brand and luckily we get a ton of freedom to do that + in the end I remain a marketer.

As always, leave your recommendations in my Twitter feed (@MartijnSch) as I’d love to know from others what I should be reading and what you recommend should be on the list or removed from the list.

Finding & Dealing with Related Keywords

Finding & Dealing with Related Keywords

Last updated on September 26th, 2018 at 04:47 am

How do you go from 1 keyword and find another 10.000 that might also be relevant to your business/site. One of the things that I’ve been thinking about and worked on for some sites recently. It’s fun as with smaller sites it makes it easy to get more insights into what an estimated size can be of an industry/niche that a company operates in. This ain’t rocket science and hopefully, after this blog posts, you’ll get some new ideas on how to deal with this.

How to get started?

Pick 1 keyword, preferably short-head: coffee mug, black rug, Tesla Roadster. They’re keywords that can create a good start for your keyword research as they’re more generic. In the research itself, we’ll talk about ways to get more insights into the long tail based on this 1 keyword.

From 1 to 10.000

Start finding related keywords for the keyword(s) you picked that you consider relevant. Use the tools that we’re going to talk about after this and repeat the process for all the keywords that you get back after the first run: 1 = 100 results = 10.000 results. Depending on the industry/niche that you operate in you might be able to find even more keywords using this method. When I started doing research for a coffee brand within 30 mins I ended up with data for 3 big niches within that space and over 25k keywords.

What tools are out there?

Obviously, you can’t do this without any tools. For my own research, I use the tools that are listed beneath. They’re a mix of different tools but they have the same output eventually. Getting to know more keywords but at the same time also get different input on intent. Focused on search (I’m looking for.. {topic_name}) and other search intent (I have a question around {topic_name}).

Besides the tools that I’ve listed there are many more that you could be using that I want you to benefit from:

    • Google Adwords Keyword Tool: The best source for related keywords by a keyword.
    • SEMRush: The second best source likely as they’re using all sorts of ways to figure out what keywords are related to each other. Also a big database of keywords.
    • AnswerThePublic: Depending on why/what/where/who you’re looking for AnswersThePublic can help you find keywords that are related to a user question.

Suggested Searches:

    • Google, Bing, Yahoo: The biggest search engines in the world are all using different ways to calculate related searches through their suggestions. So they’re all worth looking into.
    • Google Trends: Is a keyword trending or not and what keywords are related to a trending topic. Mostly useful when you’re going after topics that might have (had) some popularity.
    • YouTube: Everything video related, need I say more.
    • Wikipedia: You really are looking for some in-depth information in the topic, Wikipedia can likely tell you more about the topic and the related topics that are out there.
    • Instagram: Everything related to pictures and keywords, their hashtags might mislead you from time to time.
    • Reddit: The weirdest place to find keywords and topics.
    • Quora: Users have questions, you can answer them. The most popular questions on Quora on a topic are usually the biggest questions on your customer’s minds too.
    • Yahoo Answers: Depending on the keyword the data can be a bit old, who still uses Yahoo? But it can be useful to get the real hardcore keywords with a question intent.
    • Synonyms: The easiest relevance, find the keywords that have the same intention.
    • Amazon: Find keywords that people are using in a more transactional intent and that you might search for when you’re looking for a product. Great for e-commerce.

Grouping Keywords

When you’ve found your related keyword data set it’s time for the second phase, grouping them together. In the end, 1 keyword never comes alone and there is a ton you can do with them if you group them together in a way that makes sense for you….

By name/relevance/topical: Doing this at scale is hard, but I’m pretty sure that you see the similarity between the keywords: coffee mug and: black coffee mug. In both ‘coffee mug’ is the keyword that is overlapping (bigram). If you start splitting up keywords with different words relatively fast you’re able to find the top words and word combinations that your audience is using most. If you’re wanting to find out more on how to group them, check out KeywordClarity.io where you can group keywords together based on word groupings.

By keyword volume: If you have the right setup you can retrieve the keyword volumes for all of these keywords and start bucketing the keywords together based on short-head and the long tail. This will enable you to get better insights into the total size of the volume in your industry/niche.

By ranking/ aka opportunity: It would be great if you can combine your keywords with data from rankings. So you know what opportunity is and for what words you’re still missing out on some additional search volume.

What’s next?

Did you read the last part? What if you would start combining all three ways of grouping them? In that case, you’ll get more insights into the opportunity, your current position in the group and what kind of topical content you should be serving your audience. Food for thought for future blog posts around this topic.

Using Keyword Rankings In SEO

A few weeks ago I gave a talk at an SEO Meetup in San Francisco. It was a great opportunity to get some more feedback on a product/tool that I’m working on (and that we are already using at Postmates). You’ll hear more on this in the upcoming months (hopefully). In a previous blog post at TNW I talked about using dozens of GBs of data to get better insights in search performance. Over the last years I kept working on the actual code around this to also provide myself with more insights into the world around a set of keywords.

Because billions of searches are done on a daily basis and ~20% of queries haven’t been searched for in the past 30-90 days it means that there is always something new to find out. I’m on the hunt to explore these new keyword areas/segment & opportunities as fast as possible to get an idea on how important they can be.

That means two things:

  1. The keyword might be absolutely new and has never been searched for.
  2. The keyword has never come up on the radar of the company, it was never a related keyword or never got an impression simply because content didn’t rank for it.

Usually the next thing you want to know is what their ranking is so you can start improving on it, obviously that can be done in thousands of ways. But hopefully the process would usually work something like this. Moving up from an insane ranking (read: nowhere to be found) to the first position within a dozen weeks (don’t we all wish that can happen in that amount of time?).

Obviously what you’re looking for is hopefully a graph for a keyword that will look something like this:

What am I talking about?

Back at TNW my team was tracking 30.000 keywords on a weekly basis to get better insights into what was happening with our search volume & our rankings. It has multiple benefits:

  1. Get insights into your own performance for specific keywords.
  2. Get insights in your actual performance in search engines (are 100 keywords increasing/stable/decreasing?).
  3. Get insights into your competitors performance.

Besides that there is a great opportunity to learn more about the flux/delta of changes in the search results. You’re likely familiar with Mozcast & SERPMetrics Flux and other ‘weather’ radars that monitor the flux in rankings for tons of keywords to see what is changing and if they’re noticing an update. With your own toolset you’ll be able to get insights into that immediately. I started thinking about this whole concept years ago after this Mozcon talk from Martin McDonald in 2013. One of the things that are particularly interesting:

Share of Voice

You’ve also likely heard of the concept of Share of Voice in search. In this case we’re talking about it in the concept of rankings. If you rank #100 in the search results, you’ll get 1 point. If you’ll rank #1 you would assign it 100 points. Which basically means that you will get more points the higher you’ll rank. If you bundle all the keywords together, let’s say 100 you can get: 100 x 100 = 10.000 in total. Over time this will help you to see how a lot of rankings will be influenced and where you’re growing instead of being focused on just the rankings of 1 keyword (always a bad idea in my opinion).

In addition to measuring this for yourself, there will also be other useful ways you can use Share of Voice:

  • Who are my competitors: Obviously you know your direct competitors, but most of the times that doesn’t mean that they’re the same as you’re going against in search results. Get the top 10-20-50-100 (whatever works for you) and count the URLs for the same domain in all of the keywords in a group and multiply that by their Share of Voice. The ones that raise to the top will be the competitors that are annoying you most.
  • Competitors: You’re familiar now with the concept, so if you apply the same thing to your competitors you’re able to figure out how they’re growing compared to you and what their coverage is in search for a set of keywords. Basically providing you with the data you otherwise would have to dig up somewhere else.

How can you combine it with other data sets?

In a future blog posts I’m hoping to tell you more about how to do the actual work to connect your data to other sets in order for it to make sense. But the heading I’m going for right now is to also look more at competitors/ or at least other people in the same space. There is probably a big overlap with them but there also will be a lot of keywords missing.

What’s next?

I’m nearing the end of the first alpha version to use, it will enable users to track their rankings wherever they want. Don’t dozens of tools already do that? Yes! I’m just trying to make the process more useful for bigger companies and provide users with more opportunities to expand their keyword arsenal. All with the goal to increase innovation in this space and to lower costs. It doesn’t have to be expensive to track thousands of keywords whenever you want.

20 Reasons Why Most Experiment Programs Are Setup for Failure

Last updated on November 8th, 2017 at 06:28 pm

Over the course of the last few years I worked on over 200+ experiments, from a simple change to a Call To Action (CTA) up to complete design overhauls and full feature integrations into products. So far it taught me a lot about how to set up an experiment program and what you can mess up along the way that could have a major impact (good and/or bad). As I get a lot of questions these days on how to set up a new testing program or people asking me how to get started I created a slide deck that I gave a couple times this year at conferences about all the failures that I see & made myself running an experimentation program.

The (well known) process of A/B Testing

You’ve all seen this ‘circle’ process before. It shows the different stages of an experiment, you start with a ton of ideas, you create their hypothesis, you go on to designing & building them (with or without engineers), you do the appropriate Quality Assurance checks before launching, you run an analyze the results of your test. If all goes well you’re able to repeat this process endlessly. Sounds relatively easy, right? It could be, although along the way I’ve made mistakes in all of these steps. In this blog post I’d like to run you through the top 20 mistakes that I’ve (seen being) made.

You can also go through the slidedeck that I’ve presented at LAUNCH SCALE and Growth Marketing Conference:

Ideas

1. They just launch, they just test.

One of the easiest to spot mistakes, as you’re basically not experimenting but putting features/products live without figuring out if they’re really going to have an impact on what you’re doing. That’s why you basically always want to give a certain feature a test run on a small percentage of your traffic, if your audience is big enough that could be just as little as 1% or for smaller companies run it 50%/%50. In that case it’s easier for you to isolate what the impact is, that’s the solution to this problem.

2. Companies that believe they’re wasting money with experimentation.

One of the most fun arguments to run into in my opinion. Whenever organisations think that by running so many experiments that don’t provide a winner it might kill their bottom line there are still some steps to take that will help them better understand experimentation. Most of the times this is easy to over come, ask them what they think the right way to go is with experimentation and let them pick the winners for a few experiments. Chances are about 100% that at least one of their answers will be proven wrong. Point being that whenever they would have made the decision based on gut feeling or experience it also would cost the organization money (and in most cases even way more money). That’s why it’s still important to quickly overcome this argument and get the buy-in of the whole organization to make sure people believe in experimentation.

3. Expect Big Wins.

It depends in what stage you are with your experimentation program, at the beginning it’s likely that you’ll pick up a lot of low hanging fruit that will provide you with some easy wins. But I promise it won’t get easier of time (read more about the local maximum here). You won’t be achieving big results all the time. But don’t give up, if you can still achieve a lot of small wins over time it will also sum up to a lot of results. If you expect that every test will double your business as you might read in (bad) blog posts, you won’t.

4. My Competitor is Doing X, so that’s why we’re testing X.

Wrong! Chances are your competitor also has no clue what they’re doing, just like you! So focus on what you should be doing best, know your own customers and focus on your own success. Even when you see your competition is running experiments, chances are high that they’re also not sure what will become a winner and what will be a loser. So focusing on repeating their success will only put you behind them as you need to spend maybe even longer then them figuring out what’s working and what’s not.

5. Running tests when you don’t have (enough) traffic.

Probably the most asked question around experimentation: How much traffic do I need to run a successful experiment on my site? Usually followed by: I don’t have that much traffic, should I still be focused on running experiments. What I’d recommend most of the time is figure out if you can successfully launch more than ~20 experiments yearly. If you have to wait too long on results for your experiments you might run into trouble with your analysis (see one of the items on this laster). This is combined most of the time with the fact that these teams are relatively small and don’t always have the capacity to do more with this it might be better to focus first on converting more users or focus on the top of the funnel (acquisition).

Hypothesis

6. They don’t create a hypothesis.

I can’t explain writing a hypothesis better than this blog post by Craig Sullivan. Where he lays out the frameworks for a simple and more advanced hypothesis. If you don’t have a hypothesis, you can’t use it to verify later on that your test has been successful or not. That’s why you want to make sure that you have documented how you are going to measure the impact and how you’ll be evaluating that the impact was big enough that you’ll deploy it.

7. Testing multiple variables at the same time, 3 changes require 3 tests.

Great, you realize that you need to test more. That’s a good step in the right direction. But over time changing too many elements on a specific page or across pages can make it hard to figure out what is leading to an actual change in results for an experiment. But if you need to show real results in an experiment you could turn this failure into a winner by running 1 experiment where you change a lot and seeing what the impact is. Which after you do you run more experiments that will prove what specific element brought most of the value. I’d like to do this from time to time, sometimes when you make small incremental changes time after time it could be that there is no clear winner. Running a big experiment will help in that case to see if you can impact the results with that. Once you do that, go back and experiment with smaller changes to see what exactly led to that result so you know going forward what potential areas are for experimentation that will provide big changes.

8. Use numbers as the basis of your research, not your gut feeling.

We like our green buttons more than our red ones. In the early days of experimentation an often heard reply. These days I still hear many variations of the same line. But what you want to make sure is that you use data as the basis for your experiment instead of a gut feeling. If you know based on research that you need to improve the submission rate for a form. You usually won’t be asking more questions but want to make sure that the flow of the form is getting more optimal to boost results. If you noticed in your heat maps or surveys that users are clicking in a certain area or can’t find the answer on a particular question they have you might want to add more buttons or a FAQ. By adding and testing you’re building on top of a hypothesis, like we discussed, before that is data driven.

Design & Engineering

9. Before and After is not an A/B test. We launched, let’s see what the impact is.

The most dangerous way of testing that I see companies do is testing: before > after. You’re testing what the impact is of a certain change by just launching it, which is dangerous considering that many surrounding factors are changing with that as well. With experiments like this it’s near impossible to really isolate the impact on the change, making it basically not an experiment but just a change where you’re hoping to see what the impact is.

10. They go over 71616 revisions for the design.

You want to follow your brand and design guidelines, I get that. It’s important as you don’t want to run something that is not going to open up to the world if it’s a winner. But if you’re trying to figure out what the perfect design solution is to a problem you’re probably wasting your time as that’s exactly why you’re running an experiment, to find the actual best variant. That’s why I would advise to come up with a couple of design ideas that you can experiment with and run the test as soon as possible to learn and adapt to the results as soon as possible.

Quality Assurance

11. They don’t Q&A their tests. Even your mother can have an opinion this time.

Most of the time your mother shouldn’t be playing a role in your testing program. The chances that she can tell you more about two tiered tests and how you should be interpreting your results then you do as an upcoming testing expert are very minimal. But what she can help you with is make sure that your tests are functionally working. Just make sure she’s segmented in your new variant and run her through the flow of your test. Is everything working as expected? Is nothing breaking? Does your code work in all browsers and across devices? With more complex tests I noticed that usually at least 1 element when you put it through some extensive testing, that’s why this step is so important in your program. Every test that is not working can be a waste of testing days in the years and one not spend on actually optimizing for positive returns.

Run & Analysis

12. Running your tests not long enough, calling the results early.

Technically you can run your test for 1 hour and achieve significance if you had the right amount of users + conversions in your tests. But that doesn’t always mean you should call the results of the test. A lot of business deal with longer lead/sales times which could influence the results, also weekends, weekdays whatever can influence your business is something that might have your results be different. You want to take all of this into account to make sure your results are as trustworthy as possible.

13. Running multiple tests with overlap.. it’s possible, but segment the sh*t out of your tests.

If you have the traffic to run multiple experiments at the same time you’ll likely run into the issue that your tests will overlap. If you run a test on the homepage and at the same time one on your product pages it’s likely that a user might end up in both experiments at the same time. Most people don’t realize that this is influencing the results of the experiment for both tests as theoretically you just ended running a Multivariate test across multiple pages. That’s why it’s important to also use this in your analysis, by creating the right segments where you audience is overlapping in multiple experiments but also by isolating the users in 1 segment.

14. Data is not sent to your main analytics tool, or you’re comparing your A/B testing tool to analytics, good luck.

You’re likely already using a tool for your Web Analytics; Google Analytics, Clicky, Adobe Analytics, Omniture, Amplitude, etc.. chances are that they’re tracking the core metrics that matter to your business. As most A/B testing tools are also measuring similar metrics that are relevant for your tests you’ll likely run into a discrepancy between the metrics, either on revenue (sales, revenue, conversion rate)  or regular visitor metrics (clicks, session, users). They’re loading before/after your main analytics tool and/or the definition of the metrics are different, that’s why you’ll always end up with some difference that can’t be explained. What I usually tried was making sure that all the information on an experiment is also captured in your main analytics tool (GA was usually the tool of my liking). Then you don’t have to worry about any discrepancies as you’re using your main analytics tool (which should be tracking everything related to your business) to analyze the impact of an experiment.

15. Going with your results without significance.

Your results are improving with 10% but the significance is only 75%. That’s a problem, it means that 25% of the time you don’t know for sure that the experiment is going to provide the results that you have so far (although you still would never know for sure as reaching 100% is impossible). With experimentation it’s a problem, in simple words: it basically means that you can’t trust the results of your experiment as they aren’t significant enough to say it’s a winner or a loser just yet. When you want to know if your results are significant make sure that you’re using a tool that can calculate this for you, one of these tools is this significance calculator. You enter the data from your experiment and you’ll find out what the impact was.

16. You run your tests for too long… more than 4 weeks is not to be advised, cookie deletion.

For smaller sites that don’t have a ton of traffic it can be hard to reach significance, they just need a lot of data to make a decision that is supported by it. But also for smaller sites that are running experiments on a smaller segment this could become an issue. If your test is running for multiple weeks, let’s say 4+ weeks, it’s going to be hard to measure the impact for this in a realistic way as it could be that people are deleting their cookies and a lot of surrounding variables might be changing during that period of time. What that means is that over time the context of the experiment might change too much which could have an effect on how you’re analyzing the results.

Repeat

17. Not deploying your winner fast enough, it takes 2 months to launch.

One of the aspects of experimentation is that you have to move fast (and not break things). When you find a winning variant in your experiment you want to have the benefits from it as soon as possible. That’s how you make a testing program worth it for your business. Too often I see companies (usually the bigger ones) having to deal with the rough implementation process to get something implemented for production purposes. A great failure because they can’t get the upside of the experiment and likely by the time they can finally launch the winning variant circumstances have changed so much that it might already need a re-test.

18. They’re not keeping track of their tests. No documentation.

Can you tell me what the variants looked like of the test that ran two months ago and what the significance level was for that specific test? You probably can’t as you didn’t keep track of your testing documentation. Definitely in bigger organizations and when you’re company is testing with multiple teams at the same time this is a big issue. As you’re collecting so many learnings over time it can be super valuable to keep track of them, so document what you’re doing. You don’t want to make the mistake that another team is implementing a clear loser that you’ve tested months ago. You want to prove to them that you’ve already ran the test before. Your testing documentation will help you with that, in addition it can be very helpful in organizing the numbers. If you want to know what you’ve optimized on a certain page it can probably tell you over time changing what elements brought most return.

19. They’re not retesting their previous ideas.

You tested something 5 months ago, but as so many variables changed it might be time to come up a new experiment that is re-testing your original evaluation. This also goes for experiments that did provide a clear winner, over time you still want to know if the uplift that noticed before is still going on or if the results have flattened over time. A retest is great for this as you’re testing your original hypothesis again to see what has been changed. It will provide you usually with even more learnings.

20. They give up.

Never give up, there is so much to learn about your audience when you keep on testing. You’ve never reached the limits! Keep on going whenever a new experiment doesn’t provide a new winner. The compound effect: incremental improvements is what lets most companies win!

That’s it, please don’t make all these mistakes anymore! I already made them for you..

What did I miss, what kind of failures did you have while setting up your experimentation program and what did you learn from them?

Measuring SEO Progress: From Start to Finish – Part 2: From Creation to Getting Links

Measuring SEO Progress: From Start to Finish – Part 2: From Creation to Getting Links

How to measure (and over time forecast) the impact of features that you’re building for SEO and how to measure this from start to finish. In this series I already provided some more information on how to measure progress: from creation to traffic (part 1). This blog post (part 2) will go deeper into another aspect of SEO: getting more links and how you can measure the impact of that. We’ll go a bit more into depth on how you can easily (through 4 steps, 1 bonus step) get insights into the links that you’ve acquired and how to measure their impact.

1. Launch

You’ve spent a lot of time writing a new article or working on a new feature/product with your team, so the last thing you want is not to receive search traffic for it and not start ranking. For most keywords you’ll need to do some additional authority building to make sure you’ll get the love that you might be needing. But it’s going to be important to keep track of what’s happening around that to measure the impact of your links on your organic search traffic.

2. Monitor

So the first thing you’d like to know if your new page is getting any links, there are multiple ways to track this. For this you can use the regular link research tools, that we’ll talk about more in depth later in this piece. But one of the easiest ways for a link to show real impact is to figure out if you’re receiving traffic from it and when that time was. Just simple and easy to figure out in Google Analytics. Head to the traffic sources report and see for that specific page if you’re getting any referral traffic. Is that the case? Then try to figure out when the first visit was, you’ll be able to monitor more closely then since when you’ll have this link or look at the obvious thing, the published date if you can find it.

How to measure success?

Google Alerts, Mention, Just-Discovered Links (Moz) and as described Google Analytics. They’re are all tools that can be used to identify links that are coming in and might be relatively new. As they’re mentions in the news media or just the newest being picked up by a crawler. It’s important to know more about that as you don’t’ want to be dependent on a link index that is updating on an irregular basis.

3. Analyze

Over a longer period of time you want to know how your authority through links is increasing. While I’m not a huge fan of the ‘core metrics’ like Domain Authority, Page Authority, etc. as they can change without providing any context I rather look at the graphs and new and incoming root domains to see how fast that is growing. In the end it is a numbers game (usually more quality + quantity) so that’s the best way to see it. One of my favorite reports in Majestic is the cumulated links + domains so I can get an easy grasp of what’s happening. Are you rapidly growing up and to the right or is progress slow?

How to measure success?

One suggestion that I would have is to look at the cached pages for your links: So by now you’ve figured out what kind of links are sending traffic, so that’s a good first sign. But are they also providing any value for your SEO? Put the actual link into Google and see if the page is being indexed + cached. It is? Good for you, that means the page is of good enough quality and being cached for Google’s sake. It’s not, hmm then there is work to do for no and your actual page might need some authority boosting on its own.

4. Impact

Are you links really impacting what’s happening to the authority and ranking of the page. You would probably want to know. It’s one of the harder tasks to figure out as you have a lot of variables that can be playing a role in this. It’s basically a combination of the value of these links, which you could use one of the link research tools’ metrics for or just looking at the actual changes for search traffic for your landing page. Do you see any changes there?

5. Collect all the Links

In addition to getting insights into what kind of links might be impacting your rankings for a page you’ll likely want to know where all of your links can be find. That’s relatively simple, it’s just a matter of connecting all the tools together and using them in the most efficient way.

So sign up for at least the first three tools, as Google Search Console and Bing Webmaster Tools are free, you can use them to download your link profiles. When you sign up for Majestic you’re able to verify your account with your GSC account and get access to your own data when you connect your properties. So you just unlocked three ways of getting more data.

That’s still not enough? Think about getting a (paid) account at three other services so you can download their data and combine it with the previous data sets, you’re not going to be able to retrieve much more data and get a better overview as you’re now leveraging 6 different indexes.

(P.S. Take notice that all of them grow their indexes over time, a growing link profile might not always mean that you’re getting more links, it might be that they’re just getting better at finding them.)

How to measure success?

Download all the data on a regular basis (weekly, monthly, quarterly) and combine the data sets, as they’re all providing links and root domains you can easily add the sheets together and remove the duplicate values. You won’t have all the metrics per domain + link that way but still can get a pretty good insight into what your most popular linking root domains + links are.
In the previous part I talked more about measuring the impact from creation to getting traffic. Hopefully the next part will provide more information on how to measure business impact & potentially use the data for forecasting. In the end when you merge all these different areas you should be able to measure impact in any stage independently. What steps did I miss in this analysis and could use some more clarification?