What is SEO Experimentation?

By Martijn Scheijbeler Published January 16, 2018

If you’ve been reading some of my blog posts in the past you’ll have noticed that I worked a lot on analytics, experimentation, and SEO. Having been able to combine these areas together has led to the point that for both Postmates and The Next Web previously, we worked on setting up a framework for SEO Experimentation. In my opinion, over time you’d like to learn more about what Google appreciates and how it can help you future wise, to think about what you should be focusing your attention on == prioritization. Lately, I read a blog post on the SEMrush blog with the title: SEO Experiments that will blow your mind (clickbait alert! so I won’t link to it). Hoping there would be a lot of great ideas in that blog post I started reading realizing that over 50% of examples weren’t even close to being an experiment. They were just research, from over a year ago (which was alarming too!).

Which pushed me to write this essay to tell you more about what SEO experimentation really is and how you can learn more about it as it’s still a relatively new area in SEO that is only been explored and exposed by a few big companies.

Update: since the original publication date I’ve become even more excited about SEO Experimentation and its possibilities. That’s why I’ve updated this post with more information and some frequently asked questions that I get about the subject.

Last update: January 2020 – Updated the blog post with some new learnings and rewrote some parts.

What really is, SEO Experimentation?

You’re testing 2-X variations of a template on your site (all product pages, all category pages) and measure over a period of time what the impact is in organic search traffic of the pages (usually a template) that have seen changes in your experiment. You want to isolate your changes as much as possible (example: change Title tag template), you set a certain level of significance and you calculate in a mathematical way how your results have changed in the period before and after the change.

It’s not:

Research (I compare data between periods too, but it doesn’t make it an experiment).
Guesswork: I’ve seen my pageviews go up with X% after optimizing my title on this page. This is just guess work as you’re not using any data that is statistically siginificant to prove the statements.

I would encourage you to also read this post on Moz by Will Critchlow in which he shared how they built Distilled ODN (we worked with their platform when I was at The Next Web) and how you can start testing SEO yourself.

How is SEO Experimentation different from user testing (aka A/B Testing)?

Measurement: Instead of measuring conversion rates or other business metrics you’re mostly focused on tracking the number of sessions that are coming in from organic search. If the variants that you’ve worked on increase you’ll start calculating the impact on this. On the side, you’re still making sure that the conversion rate of these pages doesn’t decline as, in the end, it will still be the business metrics that will count.

Bucketing: Instead of bucketing users and sending user A into bucket A you’re doing this with pages. In the end, you want to make sure that the same pages (just like you do with users) end up in the same bucket. How you usually do that is that you sort by alphabet or category to have some kind of logic. What’s important though is to make sure that these buckets are in balance. Which is harder than you would do with user testing.

The difference between bucketing for users experimentation and SEO experiments

Bucketing works differently with SEO Experimentation

What are some examples of SEO Experiments?

Examples of things that you could be testing? Let’s list a few examples that I can think of:

Updating Title tags: does a certain keyword help, do I need to make them longer/shorter?
Adding/Removing Schema.org, what is the impact of adding structured data to a page?
Adding content: does adding more content to a page help with its ranking? What if I add an additional heading to my content.
Internal link structures: do you want to add more relevant links to other pages (how many, what anchor texts)?
Testing Layouts, what layout does help better for the SEO on this page? Ever noticed why Amazon has so many different variations of product pages 😉

How do I start doing SEO Experimentation?

Let me give you a very much shortened and a simplified idea of how an SEO experiment works:

Think about what you want to be testing (adding descriptions, setting a canonical), write down a hypothesis, we’ll talk about this shortly.
Figure out if you have enough traffic (500+ per variation per day) would be a minimum I’d say.
Figure out how you can serve different content/HTML per page. Are you able to serve different areas on your site based on a randomized factor, still making sure that your buckets are just as big (50/50)?
Setup the experiment and let it run for as long as needed to see valid results (usually at least 3-4+ weeks). You want to make sure that Google has picked up on all the changes on your page and has re-indexed the impacted pages.
Analyze the results after this period and look at how these buckets of pages performed before your experiment and after. Are these results valid, you didn’t make any other changes to these pages in the meantime? Are the results per variant significant?
You have found the winner + loser is. Time to start iterating and launch another experiment.

How to document an SEO Experiment?

You want to make sure that what you do around SEO experimentation is well documented, this will help you in the future with figuring out what kind of experiments are working and what you learned. When you can run over 25 experiments a year you probably won’t know after a year how many of these were successful and how to interpret the results. For documenting SEO experiments I’ve created a template that we filled in with the data on the actual experiment. You can find it here and copy it in your own Google Drive for own use:

How to analyze an SEO Experiment?

You want to make sure before an SEO experiment is running that you know what has happened with it before it starts. It’s basically the ‘control’, you want to make sure your bucket is providing stable results so you can clearly analyze the difference when the new variants are being launched.

Creating buckets for SEO Experimentation

Bucketing needs to ensure that there are additional buckets so you can measure the baseline and take care of anomalies in your data sets.

Bucketing: Your bucketing needs to make sure that it’s sending the right variant to your analytics account (most often used: Google Analytics). When the experiment starts, make an annotation so you know from when you start analyzing the results. I would recommend going with at least 4 buckets, of which A + AA are the same and so are B + BB. This way you can ensure that there won’t be any anomalies in your bucket due to 1-2 pages outperforming others due to other reasons. See my comments below on anomolies.

Logs: Logs can come into play when you start analyzing the results. In most cases, your experiment won’t generate results in the first week as the changes in your variant haven’t been picked up in the experiment. That’s why you want to look at your log files to ensure that the new variants have been crawled.

Measuring & Analyzing impact: For measuring the impact you’re segmenting down the buckets and measure what happened before the experiment and after. To see if the changes are significant or not, you need to rely on the CausalImpact library to see what has happened or not. You want to send the data for different buckets in a way that can be visualized like this:

Sending dataLayer events for measuring SEO experiments

Send data about the buckets and elements (de)activated to web analytics

Anomalies: Analyze your buckets individually! Do you see any spikes in traffic that might be hurting the data quality? You need to be sure that is not the case. For example, what if one of the buckets contain pages about political topics that all of a sudden see a spike in search volume. This doesn’t mean that your page has been performing better, it means there was just more demand so the data for that variant might be invalid.

Examples of SEO Experiments

As I mentioned both at The Next Web and Postmates I was responsible for running SEO experiments. Most of them were around optimizing <title> tags. As changes to this have, in most cases, a direct connection to the CTR within the SERPS. The title is, in the end, used as the headline for a specific search result. So let me walk you through an example of an SEO experiment as we ran it at Postmates.

Updating Titles with additional keywords

The problem: We noticed a lot of search traffic for terms around food delivery in which a zip code, like 91615 was mentioned. As we could ‘easily’ create pages for zip codes we wanted to know if that was worth it, so: “What can we do to drive more additional searches around zip codes without building new landing pages and wasting valuable engineering resources doing so”.

The solution: As we knew for restaurants in what specific zip codes they were active we had the ability to mention the zip code in the title. As we were doing this across tens of thousands of restaurants we knew that we had enough of a sample size.

Old:
- {Restaurant Name} {Street Address} in ({City}} – Postmates On-Demand Delivery
- Paxti’s Pizza 176 Fillmore Street in San Francisco – Postmates On-Demand Delivery
New:
- {Restaurant Name} {Street Address} in ({City}} ({Zip Code}) – Postmates On-Demand Delivery
- Paxti’s Pizza 176 Fillmore Street in San Francisco (97521) – Postmates On-Demand Delivery

The result: It was inconclusive, in the end, that wasn’t likely the outcome that you were hoping for. But I want to paint a realistic picture of what can happen when you run experiments. In a lot of cases, you don’t see enough changes in the data to be certain that it’s an improvement. In this case, we expected that a title change wasn’t good enough to actually compete for zip code related queries. The food delivery industry is one of the most competitive in the world for SEO so we knew it was always possible to have an outcome like this.

Frequently Asked Questions + Answers

There are a lot of questions that come up when I talk about this subject with people. So I’ll try to keep up this blog post with any new questions that might arise:

Isn’t this cloaking? Doesn’t this hurt for my Google rankings?

Not really, you’re not changing anything based on who’s looking at the page. You’re changing this only on certain pages that are being served and the search engine + user will see the same thing. Ever looked at Amazon’s product pages and wonder why they all have a different layout? Because they’re testing for both user experience as SEO.

Do you want to learn more about this subject?

Great, when we were setting up our own SEO experimentation framework at Postmates about 1,5 years ago I tried to find all the articles related to it and talked to as many people as possible on this. These were mostly the articles I would refer you to if you want to learn more.

Resources

When I wanted to learn more about SEO experimentation I started to figure out what was already written on the web, most of these resources are from teams & companies that I worked with before. So if you’re enthusiastic about this subject, read more here:

Let’s really start & keep innovating in the SEO industry and let’s get rid of terrible clickbait headlines. SEO Experimentation is like I mentioned something that we should be embracing as it’s a more scientific approach that is going to lead to finding new insights.

If you want to talk more about this, feel free to reach out. Like I said, I have been updating this post for over two years now, and will keep on doing so in the future.