What is SEO Experimentation?

Last updated on June 14th, 2018 at 06:40 pm

If you’ve been reading some of my blog posts in the past you’ll have noticed that I worked a lot around analytics, experimentation and SEO. Having been able to combine these areas together has led to the point that for both Postmates and The Next Web previously, we worked on setting up a framework for SEO Experimentation. In my opinion over time you’d like to learn more about what Google appreciates and how it can help you future wise, to think about what you should be focusing your attention on == prioritization. Lately I read a blog post on the SEMrush blog with the title: SEO Experiments that will blow your mind (clickbait alert! so I won’t link to it). Hoping there would be a lot of great ideas in that blog post I started reading realizing that over 50% of examples weren’t even close to being an experiment. They were just plain research, from over a year ago (which was alarming too!).

Which pushed me to write this blog post to tell you more about what SEO experimentation really is and how you can learn more about it as it’s still a relatively new area in SEO that is only been explored and exposed by a few big companies.

Update: since publishing this post I’ve became even more excited about SEO Experimentation and its possibilities. That’s why I’ve updated it with more information and some frequently asked questions that I get about the subject. Last update: June 4, 2018 – Added some new sources of information that talk in depth about SEO experimentation and how it’s applied at Pinterest.

What really is, SEO Experimentation?

You’re testing 2-X variations of a template on your site (all product pages, all category pages) and measure over a period of time what the impact is in organic search traffic of the pages that have seen changes in your experiment. You want to isolate your changes as much as possible, you set a certain level of significance and you calculate in a mathematical way how your results have changed in the period before and after the change.

It’s not:

  • Research (I  compare data between periods too, but it doesn’t make it an experiment).
  • Guess work: I’ve seen my pageviews go up with % after optimizing my title on this page.

I would encourage you to also read this post on Moz by Will Critchlow in which he shared how they built Distilled ODN (we worked with their platform at TNW) and how you can be testing with SEO yourself.

How is SEO Experimentation different?

Measurement: Instead of measuring conversion rates or other business metrics you’re mostly focused on tracking the number of sessions that is coming in from organic search. If the variants that you’ve worked on increase you’ll start calculating the impact on this. On the side you’re still making sure that the conversion rate of these pages doesn’t decline as in the end it will still be the business metrics that will count.

Bucketing: Instead of bucketing users and sending user A into bucket A you’re doing this with pages. In the end you want to make sure that the same pages (just like you do with users) end up in the same bucket. How you usually do that is that you sort by alphabet or category to have some kind of logic. What’s important though is to make sure that these buckets are in balance. Which is harder than you would do with user testing.

Bucketing works different with SEO Experimentation

What are some examples of SEO Experiments?

Examples of things that you could be testing? Let’s list a few examples that I can think of:

  • Updating Title tags: does a certain keyword help, do I need to make them longer/shorter?
  • Adding/Removing Schema.org, what is the impact of adding structured data?
  • Adding content: does adding more content to a page help with its ranking?
  • Testing Layouts, what lay-out does help better for the SEO on this page? Ever noticed why Amazon has so many different product pages 😉

How do I start doing SEO Experimentation?

Let me give you a very much shortened and simplified idea of how an SEO experiment works:

  1. Think about what you want to be testing (adding descriptions, setting a canonical), write down a hypothesis.
  2. Figure out if you have enough traffic (500+ per variation per day) would be a minimal I’d say.
  3. Figure out how you can serve different content/HTML per page. Are you able to somehow serve different areas based on a randomized factor, still making sure that your buckets are just as big?
  4. Setup the experiment and let it run for as long is needed to see valid results (usually at least 3-4+ weeks).
  5. Analyze the results after this period and look how these buckets of pages performed before your experimen and after. Are these results valid, you didn’t make any other changes to these pages in the meantime? Are the results significant?
  6. You have found what the winner + loser is. Time to start another experiment.

How to document an SEO Experiment?

You want to make sure that what you do around SEO experimentation is well documented, this will help you in the future with figuring out what kind of experiments are working and what you learned. When you can run over 25 experiments a year you probably won’t know after a year how many of these were successful and how to interpret the results. For documenting SEO experiments I’ve created a template that we filled in with the data on the actual experiment.  You can find it here and copy it in your own Google Drive for own use:


How to analyze an SEO Experiment?

You want to make sure before an SEO experiment is running that you know what has happened with it before it starts. It’s basically the ‘control’, you want to make sure your bucket is providing stable results so you can clearly analyze the difference when the new variants are being launched.

Bucketing needs to ensure that there are additional buckets so you can measure the baseline and take care of anomalies in your data sets.

Bucketing: Your bucketing needs to make sure that it’s sending the right variant to your analytics account (most often used: Google Analytics). When the experiment starts, make an annotation so you know from when you start analyzing the results.

Logs: Logs can come into play when you start analyzing the results. In most cases your experiment won’t generate results in the first week as the changes in your variant haven’t been picked up in the experiment. That’s why you want to look at your log files to ensure that the new variants have been crawled.

Measuring & Analyzing impact: For measuring the impact you’re segmenting down the buckets and measure what happened before the experiment and after. To see if the changes are significant or not, you need to rely on the CausalImpact library to see what has happened or not. You want to send the data for different buckets in a way that can be visualized like this:

Send data about the buckets and elements (de)activated to web analytics

Anomalies: Analyze your buckets individually, do you see any spikes in traffic that might be hurting the data quality. You need to be sure that is not the case.

Frequently Asked Questions + Answers

There are a lot of questions that come up when I talk about this subject with people. So I’ll try to keep up this blog post with any new questions that might rise:

Isn’t this cloaking? Doesn’t this hurt for my Google rankings?

Not really, you’re not changing anything based on who’s looking at the page. You’re changing this only on certain pages that are being served and the search engine + user will see the same thing. Ever looked at Amazon’s product pages and wonder why they all have a different layout? Because they’re testing for both user experience as SEO.

You want to learn more about this subject?

Great, when we were setting up our own SEO experimentation framework at Postmates about 6 months ago I tried to find all the articles related to it and talked to as many people as possible on this. These were mostly the articles I would refer you to, if you want to learn more.


When I wanted to learn more about SEO experimentation I started to figure out what was already written on the web, most of these resources are from teams & companies that I worked with before. So if you’re enthusiastic about this subject, read more here:

Let’s really start innovating in the SEO industry and let’s get rid of terrible clickbait headlines. SEO Experimentation is like I mentioned something that we should be embracing as it’s a more scientific approach that is going to lead to finding new insights. If you want to talk more about this, feel free to reach out, at some point I might write a longer post explaining what you should be thinking off while building your own framework for SEO experimentation.

Help me out, share this article:Tweet about this on Twitter
Share on LinkedIn
Email this to someone
Share on Facebook
Share on Google+
Buffer this page

Leave a Reply

Your email address will not be published. Required fields are marked *