The practice of A/B testing follows the Lean methodology of Build, Measure, and Learn (BML).  By using tools capable of performing these tests, we can build a quick test, measure the interaction with the tests, and learn from the results gathered.  This is fundamental to improving your content and your application.

If you’ve been following my company’s Nonlinear Thinking blog series on A/B testing, you may have seen the article that my colleague Gavin and I put together on the Top 10 reasons to A/B Test everything you publish.  We initially started this from a Sitecore angle, gathering all the reasons we wanted to do A/B testing with Sitecore.  As we pared down to the top 10, however, it became obvious that this was something that applied to any tool capable of A/B testing.  The reasons for doing it were not tool-specific, they held up on their own.

However, for those of you interested in using Sitecore as your content management tool of choice, you may be wondering how you can use Sitecore for tackling these items. So here we go… tackling the Top 10!

10. Learn about your audience

Sitecore’s capability to build up personas for your users based on interactions with content that you have tagged allows you to begin to learn more about each user.  My colleague Randy recently wrote about this in his article on A/B testing and persona analysis:

A marketing-enabled CMS like Sitecore CEP allows authors to specify into which persona a visitor to the page they have published should fall. Sitecore not only achieves the implementation of this concept, but succeeds at doing it subtly as well. Any visit to any page can add “points” to multiple personas. So an author publishing an overview of a software product’s features might give visitors 5 points each in the “technical buyer” and “business buyer” personas. When publishing the technical specs page, they might assign 10 points to “technical buyer”.

The key point is that application of the persona-categorization rules is:

• Marketing driven – it’s performed by those creating and publishing content
• Intuitive – no coding is required

9. Enhance content

When setting up an A/B test in Sitecore, your data sources for the test are already pieces of content that are live and being read by your audience.  By testing which variants work better and choosing a winner, your content can be continually enhanced with very little effort and no coding required.  Your ‘losing’ tests can even be retained to be used for personalization purposes for specific personas, while your winner can be your default content that is presented to users that have no specific persona yet.

8. Small changes can have a big impact

Since Sitecore drives the testing based on using data sources as your variations, this means that content editors can quickly copy an existing data source to create a new variation and make a small change within minutes.  This provides a marketing team with the capability of quickly rolling out a small change to a high-traffic page with no down-time and no deployments.  Just follow these quick steps:

1. Find your existing data sources for the component you want to test.
2. Create a copy of one of the existing data sources.
3. Give the new data source its own item name.
4. Edit the new data source to provide the variation in content.
5. Open Page Editor on the page where you wish to test the new content.
6. Select the component to test.
7. Open the ‘Testing’ dialog for the component.
8. Add a new variation, selecting the newly published data source.
9. Save the page.
10. If governing publishing via workflow, submit the changes to the next step in the workflow.
11. When all the content is approved and ready to go, publish the content with Sitecore’s publishing functionality.

7. Tests don’t have to be perfect every time

Because no downtime is required, and no code is involved, the marketing team can use Sitecore to constantly create minor variations, select winners, and move onto another variation.  If a data source isn’t working, change it and publish a new edit.  Avoid going through weeks of content approval and expensive meetings, and just get your content flowing!  Also, Sitecore’s Page Editor allows for adding more tests as you go, or altering the tests on the fly as you see things aren’t working as expected.

Since Sitecore allows you to be continuously changing and editing your content without expensive deployments or outages, you can do cheaper, less perfect tests and constantly adjust.

6. Personalization

When editing components, Sitecore provides the ability to drive personalization of the component’s datasource, in a similar manner to how tests are configured.  The content author is able to configure rules based on profiles and pattern cards, attributes of the visitor, DMS data gathered about the users’ traffic through other content, etc.   These rules are then leveraged to decide which datasource should be shown to the user.

Results from testing should be used to build new personalization rules, and the data source that was used for testing can be immediately used as the data source for the personalization variation.

5. Drive Investment

The key decision makers in your organization need data upon which to drive your investment decisions.  By perform A/B tests in Sitecore, you begin to gather that data for analysis by these groups.  Additionally, by integrating further into your other analytics capabilities, such as sending DMS data to Google Analytics, you begin to build an even better view of how your website is performing.  This allows for better and quicker decisions.

Finally, since the testing results are per component, the team is able to make decisions on a smaller scale for the investment, rather than needing to make large-scale investment decisions all the time.

4. Small increase in conversion rate has large impact on baseline

By architecting your Sitecore solution to use components driven by data sources, it provides the marketing team with the capability of performing A/B tests across the entirety of the site, and not just select areas.  By performing small tests on all components, the marketing team can fine-tune conversions across the entire flow of a user’s visit.

Additionally, by leveraging goals, personas, and the rest of the DMS toolkit, the marketing team has additional tools available to them to further fine-tune their engagement plans and conversion funnels.  The key to all of this is architecting the solution correctly from the beginning to provide these opportunities to the marketing team.

3. Keeping pace with market

Due to Sitecore’s ability to perform variation tests directly in the tool and publish changes immediately, the marketing team has a shorter cycle between learning about a change in the market and being able to react.  With A/B tests in place, the team can observe which content variations are performing better, and use these observations to decide on new default content, new personas, or new content personalization rules.

The key here is to make sure that somebody on the marketing team is active and involved with monitoring the results and tweaking the content.  The tool will provide the opportunity, but the team needs to seize the control.

2. Return on investment

If you have already invested in Sitecore as your content management system of choice, and want to begin leveraging A/B tests, using DMS provides a cheap alternative that is natively built-in to the application.  Once architected correctly, the marketing team can spend very little time creating the A/B tests, analyzing the results, and adjusting website content.

The key costs here will be up-front investment in the tool, and in architecture of the site.  Many organizations new to Sitecore may overlook the planning for DMS and may not architect their content pieces with a component-based approach.  Make sure when dealing with your Sitecore development partner that they are familiar with how to build a DMS-ready application, and that they understand component-based architecture.  Getting the architecture of the site correct is a key to a better return on the investment.

1. You can’t predict results

Due to the unpredictability of the marketplace, it is necessary to have a tool that will allow the team to react quickly to unexpected behaviour of site visitors.  With BML, the Learning stage is key to being able to start the next cycle of improvement.  Follow these BML steps to handle the unpredictability:

1. Build. After creating a new content data source for a component, immediately create a copy of the data source.
2. Build. Analyze the copy, and make subtle variations.
3. Build. Immediately implement a test using these two variants.
4. Build. Publish the changes to the site.
5. Measure. Monitor results until you have enough data for statistical relevance.
6. Learn. Analyze the results and decide on a course of action.

By immediately testing everything being published, the marketing team will be ready to react!

What’s next?

I strongly recommend reading some of my colleagues’ insights into A/B Testing from the last month, as it will cover most of the topics you need to know about with regard to A/B Testing in general, and specifically with Sitecore.  For readers looking at Sitecore resources only, here are some recommended readings:

• Screencast: Setting up A/B tests in Sitecore
• Statistical relevance of A/B tests in Sitecore DMS
• Personalization with Profiles and Pattern Cards in Sitecore DMS
• Sitecore DMS case study: How the Rotman School is driving personalized web experiences
• Sitecore personalization: Use Google Analytics to measure results
• Sitecore DMS and Google Analytics: 3 very good reasons to integrate
• Accessing Sitecore analytics with Microsoft BI Tools