Experimentation for Product instead of Conversion improvements

Over the last years, I’ve had many (healthy) debates with product, brand and growth teams on what experiments to run and for what reason. In some cases, it was easier to run brand or product experiments just like a regular experiment to improve conversion rate. But in the some of them there was fear that either brand or product changes might hurt conversion rate. The main question that usually came up was: what is acceptable? And at the same time for product improvements: is this improvement going to impact anything? And do we decide that speed of testing is more important than the actual learnings? That’s what I want to cover in this blog post.

Experiments: Isolation & Exploration

So when do you run multiple experiments to explore what combination of features is working and when do you focus on one specific feature to isolate what really makes your audience tick. I think the majority of discussion around testing for conversion rate optimization is focused around that.

Combining Product Changes

A discussion that I’ve had with multiple teams over the years, what if we add feature X, Y, Z to the page. But what then, in some cases it’s easy to add all of these new features to the same page and run individual experiments or it’s a product improvement and in any case, it would be good to have that feature.  In most cases, I would recommend running 1 experiment and stacking all these changes together.

Exploration: You explore if the product changes are going to have an impact on the user and what they’re doing. The argument against it is that you basically haven’t learned much and that you only know if the impact has been positive/negative. The upside is that you have good insights into if the product changes have an effect at all by rolling them out. The downside is that it might be hard to analyze the individual changes that you have implemented.

Isolation: In this case, I really want to know what the impact of a CTA, Image, text is on the user and if making changes to it will have any impact on the users’ behavior. If that’s the case, great! You really learned something that you can leverage again for future tests. And in case of the multiple changes, you would run multiple experiments to test the effect of all the changes on each other. The big downside could be in this case that it will take longer to achieve the results that you want.

What other methods have you seen, or what could be improved? How do you test faster when you do not always have enough traffic for multiple experiments.


Joining RVshare!

I’m joining RVshare, a two-sided marketplace for RVs and motorhomes, as their VP Marketing! Over the last few weeks, I’ve talked extensively with the founders and part of the team. It became clear that this was a great opportunity for a few reasons.

Why RVshare?

  • They have great product-market fit and proved this concept clearly works.
  • They’re wanting to build out a world class (marketing) team.
  • They’re a marketplace business as well, connecting the supply of RV owners to the demand of renters who want to explore the world.
  • They had backing from a great investment partner making it possible to grow fast.

All of this got me really excited, after Postmates I decided that I wanted to work not just on one channel (SEO), but on multiple to create a strong diverse set of acquisition channels and build out a world class marketing organization somewhere. That’s what I’m mainly going to focus on for the next year (which means I’m going to be hiring soon for various roles, reach out to me if you’re interested in joining RVshare). In addition, I’ve always wanted to work for a company that is operating in the travel industry, as I consider it one of the most competitive industries and a joyful one, who doesn’t love bringing more travel to people. And what better ways are there to test your own skills than in an industry like that.

What is the challenge?

  • Build out a world-class marketing organization.
  • Create a diversified acquisition strategy to drive more new RVs/Motorhomes on the platform and acquire more renters who want to explore the world with an RV.
  • Trust data! Build and improve everything that the team has built out in the last years and help it grow to the next level.
  • Create an even stronger brand and build out renting RVs as an option for people traveling.

I’ll do my best to try to document parts of the journey here on this blog and on Twitter (@MartijnSch), if you want to get a hold of me, reach out to me via martijn@rvshare.com


Measuring Content Performance: Content Engagement Metrics

What is the effectiveness of our content, how well does our content work? Who is writing the best content? What should we be writing about next? These were the top X questions I received on a weekly basis while working at The Next Web. And I’m probably not the only one, I hear a lot of companies that are asking the same questions. We were a publisher, with a big editorial team (~10 FTE editors) and publishing around 30-40 blog posts on a daily basis and had so for the past ~10 years. Which meant that there were over 65.000 articles in our archive that we could learn from. One of the hardest projects in the end.

“Tons of page views doesn’t always mean this post is doing well or is of the highest quality/the best journalism”. If one of our post hit Reddit, HackerNews or was a top story in Google News we for sure celebrated this but the impact on the business wasn’t always that big as we were hoping. Traffic from most of these sources has very low engagement and basically only brought in money through display advertising (with incredibly low CPMs). Overall, this meant that we wanted to find a better metric to score the performance of editors and find out what kind of quality really worked well for the business, not just for engagement & visitor metrics. So we got started looking into this…

Usage Metrics, Useful & Useless

What I still see a lot of companies do, and in all honestly you can’t always blame them for that. Is look at the basic usage metrics of content. How much time has a user spend reading this article, what is the bounce rate, etcetera. Which isn’t great as most of the time you don’t know the context. Is a good blog post one that has been read for 2 minutes, or 1 minute. Obviously the length of the blog post and if it has videos/images has an impact on this. All data that is usually easily overseen when analyzing content performance. Even with taking this into account it’s making it hard to come up with recommendations while working for a publisher. Are you really always going to recommend to use at least 2 ¾ images in a blog post and require a video that is exactly 23 seconds long so we know for sure people will watch it? Probably not.

Business results

Most of all, these metrics don’t align with your business goals. How often does the CEO/COO/CMO ask you what the bounce rate is of your articles so they can calculate the return on investment. Never right? That’s why over time we need better metrics that align with the business and that provide useful insight into what content really helps and what content is just good for vanity metrics like page views. You’ll be surprised in the end how many companies are still driven by these.

Capturing Goals in Google Analytics is important to measure impact of performance.

Creating a Content Engagement Metric

It’s going to be about the monetary value that content represents for your business but we won’t be setting it up that way. In the end you need to be able to evaluate the way you calculate this and over time this will likely change with your business. That’s why it’s important that we can make the decisions in an open and honest way so we can make changes later when we think they’re needed.

Setting up Goals

If you want to get started and do this on your own it’s going to be important to know what kind of events are important to your business and site. In our case we looked at, for example: newsletter subscriptions, article shares on social media (they bring in more visits, so useful for more revenue), ticket sales, ecommerce sales, etc. But also in general we wanted to know what the value for us was with a pageview. I’m not going to walk you through the setup of all these goals in this post, but if you’re unfamiliar with setting up goals in Google Analytics, read & watch this.

Custom & Calculated Metrics

So now you have determined what the goals are that matter to you. What is going to be important in this step is making sure that you assign ‘points/value’ to all of the goals. You don’t have to edit the goals for that (probably even better not to, to not interfere with the page value metric). So for example, think of it like this:

  • Pageview: 1 point
  • Article share: 5 points
  • Newsletter subscription: 10 points
  • Ecommerce sale: 150 points (or varying on the product value)
  • Ticket sale: 250 points

Now it’s time to set up a calculated metric to take this into account. Why a calculated metric? By using this we’re able over time to adjust the formula to our needs but also individually assign values to the goals if we wanted to (in that case we’d use goal completions as well).

When you get started with this, know what numbers you are using for the goals that you have setup. You can find that in the Goals interface within Google Analytics.

  1. Name: Content Engagement Score.
  2. Formatting Type: Integer (you can change this to Float or Currency if that’s more applicable)
  3. Formula: {{Goal1 Completions}} * 1 + {{Goal2 Completions}} * 5 + {{Goal3 Completions}} * 10…

In the end that should look something like this:

Reporting & Dashboarding for Content Quality

Now you have successfully created a calculated metric that we can use for reporting and informing other teams in the company about the content performance.

Trial & Error: Formulas should always be up for discussion

This formula isn’t going to be good the first time, you’ll have to tweak it and assign values that make more sense in your case. That’s why I also didn’t share the actual values of TNW (they were different from the example) as it matters to your specific business and goals.

To talk a little bit more about how things worked out for us, we tweaked it 3 to 4 times over a period of two months to get closer to our actual goal of measuring performance. In the end we could have editors who still would write for a ton of pageviews so get hundreds of thousands of points through that but editors who did great on business results weren’t rewarded value through that. So we upped some of the points assigned to our business goals to align them better.

Custom Reporting

Just setting up the goals probably already gets you quite far. It will allow you to create a page + goals report in which you can see how many goals have been hit for certain pages. Very useful if you want to measure the performance.

Dashboarding: Google DataStudio/Chartio versus (Google) Sheets

You need to share the reporting around this in an easy way, depending on your teams it will depend what works best for them. Sharing spreadsheets or making it easy for people to immediately take a look at a dashboard in a tool like Chartio and/or Google DataStudio.

Performance by day/editor/topic

Now you have all this data you can hopefully combine this with the data that you also have gathered through your custom dimensions. Previously I have blogged on The Next Web about all the ideas that I have around custom dimensions. Give it a go, it will be surprising to see how much easier it will become to report & analyze the performance across topics, time of day, team members when you can really align this with your business goals.


This is just an example of how we were measuring the performance of content for the business. There are many other ways to do this and you will have to customize the formula for your own business. What other metrics and ideas would you take into account to analyze and report this? Leave a comment or reach out on Twitter: @MartijnSch


Why & What alerts in SEO are becoming more important

We have all been there, haven’t we? Quotes like: “SH*T, my sitemaps are broken”, “I have no-indexed half my pages” or: “I have been kicked out of a search engine with way too many pages” sound familiar? Honestly, I can’t blame you. It’s getting harder and harder to keep track of all the changes that are being made regarding SEO on your site and you’re likely the only person involved with SEO for your company while also trying to work on driving traffic through other channels. So let me give you a quick insight in what I am usually tracking within bigger companies and where there is an actual team to react on issues that come up.

Continue Reading


Growing as an SEO (1/4) – Writing better Job Descriptions for SEO Roles

Writing a resume isn’t fun (IMHO) and writing job descriptions is probably even less fun. Over the last years I’ve written many of them, usually following a similar template that would help us define what the role is about. Which isn’t always a good thing, depending on the seniority of the role you want to make sure you use the right approach to hire and make it as personal as possible. Which usually makes for better hiring, most of my best hires came through my network of people I was at least ware of. Over the last months I’ve received many requests if I wanted to take a look at an SEO job description, if I knew people that were looking for a job and wanted to share it with my network, you get it. But what I started noticing is that most SEO job descriptions are incredibly generic and don’t really seem inviting too many people.

“We’re looking for somebody to set up or SEO strategy, we’re looking for somebody to work with our engineering and design team to create content. You’ll pick the right keywords for us to focus on”. Yada yada yada. You’ve seen and heard it all before. Obviously when you’re on a job search in SEO you’ll come across all of these requirements and responsibilities easily. But I think companies need to do better, definitely in an area like Silicon Valley, to hire the right SEO talent or to get them even interested. There isn’t that many of us, but the information you give ‘us’ isn’t always great. That got me thinking on what information should be mentioned in job descriptions for SEOs. But I also wanted to take a look at what job descriptions look like right now:

Saving job descriptions

I must admit, I have a weird obsession, if I see well written (or really poor) job descriptions, for whatever type of role in digital, growth, marketing, you name it, I have a tendency to save them (in Evernote). Over the years that has build up to a nice archive (150+ JDs) that I can use for writing new job descriptions that I’ve used for hiring. The list of 16+ companies that are amongst them: Airbnb, Uber, Groupon, Booking, Zillow, Hulu, Porch, Tesla and the descriptions range from SEO Assistants to more senior positions like Senior Director of SEO. Fill that up with all the job descriptions that you can easily find on most job sites (LinkedIn, Glassdoor) and you can get a good enough understanding of what managers + recruiters are thinking about while sourcing/hiring for SEO roles.

Almost unfortunately, Postmates didn’t have a job description for me. As my previous boss asked me to fill this need within the Growth team, otherwise I would have loved to share that original one.

What companies are looking for?

It doesn’t exist, even when you’re in the right position and you might be able to write your own job description. But most of them have some issues, so I decided to look at all the SEO job descriptions that I could find and see if there are any patterns in what companies are looking for. So let’s look at the two main areas of job descriptions:

Responsibilities

Tag clouds are good for something I guess, that’s why I just threw in all the requirements for a dozen job descriptions and these were the main keywords that came up in the tagcloud. Some of the ones that stood out for me:

    • Performance: This keyword was interesting to me so I did some digging on the context, I expected it to be a requirement to know about performance marketing. Turns out the overwhelming majority of companies wants better performance reporting around their SEO strategy.
    • Content: People in SEO need to have a solid understanding of content, know how to create it and maybe even more important, know how to improve it.
    • Technical: Guess what, these days SEOs need to be technical. As most of the job descriptions are from Bay Area companies, that doesn’t surprise me at all considering that the work with product managers (or in some orgs are even PMs) and engineers most of the day. This is also important regarding technical audits that are usually performed inhouse.
    • Strategies/Initiatives: SEOs need to be able to make strategic decisions. For most companies they’re one of the people working on usually the biggest traffic channel for the site so they need to be able to think strategic as they can make changes to a platform that have a bigger impact than just SEO.
    • Team(s): They either need to be great working in teams (aka teamplayer) and in the more senior positions they need to be great at building up their own teams, or building out.

Missing?

While analyzing this there were a few things that I was missing that I thought were interesting so at least I wanted to mention them.

  • Agencies: A good portion of SEOs that I know work with agencies, but there was barely a mention in job descriptions about working with agencies, finding them, etc.
  • ASO: Most companies that I went through had mobile apps, but ASO was never really part of the job description.

Requirements / Qualifications

  • Experience in SEO: For starter roles this is usually not a requirement, as they can only have experience with the work that they’ve done on the side and not in an actual job/company.
  • Experience in Analysis: Most SEOs needs to be at least familiar on a basic level with a web analytics tool like Google Analytics, Omniture, Adobe Analytics so they can analyze their performance (one of the core responsibilities).
  • Tools: Often I see experience with Google Search Console being mentioned, but I’d love to see more companies mention the other tools in their toolset too. In the end you won’t share that much information with your competition by telling them what tools you’re using.
  • Delivering results: Although you can’t guarantee that your work will help you need to be able to show the progress that you’ve made on other sites and the work that you’ve done there. If it didn’t result in an uplift, at least you’re able to provide answers on the why not and what your original hypothesis was.

Missing?

What I feel is missing in the list of requirements & qualifications is a few things, what about the setup that you already have, or are they diving into a new field of opportunity. Are you going to expand your business, are you operating in new niches? For some companies the future manager will already know what projects (s)he wants work to be done for.

  • Tools? What is your current toolset, if somebody has exceptional expertise with a certain tool that for sure would help. Anybody can learn more about a tool, but experience is important too.
  • How often have they played ‘this’ game before? How many sites have you worked on, what was the scale/business model of the sites? I have way more experience then on average with publishers and marketplace models then probably other SEOs. While somehow I have barely worked for ecommerce sites and SaaS companies thus far. This also gives better insights if they have a certain ‘play book’ on how to approach certain issues.

Writing the Ultimate Job Description

I’m on a journey to change the world. OK slowly. And one by one. But I believe we can do better, making people find the right jobs will make them happier and increase the productivity and output for the company. The first step to get that started would be to improve job descriptions so people have a better idea on what they’re getting into then setting up a very generic one. Not all bullet points will apply to every job description, but you likely get the point:

Responsibilities

  • Define the SEO strategy: we’re wanting to grow (X metric) with approximately XX% this year. SEO is one of the channels that we depend on, so we’re looking for somebody who could build out the channel after an intensive audit and figure out what opportunities we really have.
  • Reporting: be able to use our analytics infrastructure to dive into customer & traffic data to find new insights and opportunities for us to grow SEO as a traffic channel.
  • Reporting Up: be able to talk to our stakeholders and peers in the company about the performance and opportunities that you see within SEO. Be able to communicate the results of the work that we‘ve done.
  • Analytical: be analytical and data driven, are you able to write SQL and work with large amounts of data? Great! We have some of our analysts ready to work with you in supporting the insights that you need to gather.
  • Technical: we have developers ready to work with you, so it would help if you could code and be able to explain in detail what your wishes are for implementations regarding SEO and new features.
  • Content: we’ve been wanting to create & produce more and better content. It would be great if you have worked with copywriters and are able to take our blog & content marketing efforts to the next level. We have copywriters that we work with and also our PR specialists.
  • Build out the team: be a team leader and builder. Currently the team is 2 people that will be supporting you, but we hope to build out the team with your support. So we’d like to see experience leading people & teams.
  • Performance: you need to be able to identify opportunities, build out the resources needed and along the way have a ton of fun while always striving for better results.

Requirements / Qualifications

  • You have X years working experience in online/digital marketing and you know what channels are important for our type of business to be successful.
  • You have worked on (multiple) big sites regarding SEO before, it is important to us that you can show experience building out a strategy for a bigger site (50.000+ pages).
  • You have worked with web analytics tools and understand how you can use these insights to further improve user experience and optimize pages for search engines. Preferred tools would be: Google Analytics, Amplitude Analytics, Adobe Analytics, …, etc..
  • Do you have experience writing or have worked with copywriters before, great! This will help push forward our ideas on content marketing.
  • You have experience managing different products/projects at the same time, our teams are divided between products/projects and some are cross functional (designers, engineers).
  • You have worked before with tools that we already have in our toolset: Google Search Console, Bing Webmaster Tools, Majestic SEO, Screaming Frog, … , etc. but you’re free to look into other SEO tools (up to enterprise budget) and evaluate needs for our organization.

This is not even good enough but hopefully a good start, in the job descriptions that I usually write I also try to give insights into the company, mention what the team looks like and what the perks & benefits are of the role. But most important what type of person we’re looking for and how we think this role will help the bigger team grow & support. In the end it’s a two way stream and we want to make that clear from the start. You need somebody’s skills but you also want them to feel welcome and appreciated!

What’s missing?

What do you think is really missing in job descriptions these days that should be reflect. What are you looking for in a next or first SEO role? Let me know, I’d love this post to become the ultimate SEO job description for the rest of the world. Hit me up on @MartijnSch on Twitter for feedback!

Growing as an SEO

In this series I’ve also blogged about: