Top 7 Best Practices for A/B testing and D365 Marketing

Every marketer knows that when it comes to sending emails, getting clicks and having your customers or target customers engage with your content can be challenging. That’s why testing your content and seeing how it performs is super important with Dynamics 365 Marketing A/B testing!

So, what is A/B testing? A/B testing (or sometimes known as a split test) is used to ensure you are creating engaging and useful content. A/B testing is used to help marketers observe how one version of content is performing alongside another. To run an A/B test, you need to create two versions of one piece of content, with changes to a single variable.

It sounds simple enough, but there are some key elements and best practices with A/B testing, if not followed, can lead to your skewed or inconclusive testing results...

1. Pick one Variable to Test

As you optimize your web pages and emails, you might find there are a number of variables you want to test. However, to evaluate how effective a change is, you'll want to isolate one "independent variable" and measure its performance. To pick a variable, you must look at the various elements in your marketing resources and their possible alternatives for design, wording, and layout. Other elements you might test could be email subject lines, sender names, and different ways to personalize your emails. Keep in mind that even simple changes, like changing the image in your email or the words on your call-to-action button, can drive big improvements.

2. Identify your Goal

Choose a primary metric to focus on. In Dynamics 365 Marketing, this is either your open rate or your click-through rate. This means the email with the most opens or most clicks is deemed the winner. Eventually, you’ll be able to measure based on a GOAL/Conversions. (Coming in a future D365 Marketing release)

3. Create a Control and a Challenger

Build out your Version A (control) in your marketing email and then your Version B (challenger). Version A and Version B MUST be different by only one independent variable to have the best results for your A/B test.

4. Split your Sample Groups Equally

Typically, an email A/B test performs best if it’s sent to at least 100+ people. In Dynamics Marketing for example, if a customer is sending an email to 1000 people, you can send Version A to 10% of the list, Version B to 10% of the list as test groups, and send the winning version to the remaining 80%. D365 marketing does this easily and automatically when you follow a few simple steps.

5. Make sure you’re only running one test at a time

Testing more than one variable for a single campaign can lead to inconclusive results. If you desire to test more than one, it is best practice to run multiple single variable tests so your results are easier to track.

6. Give the A/B test Enough Time

Make sure to let your test run long enough to analyze results properly. A general rule of thumb is 24 hours, but it really depends on how big your marketing list is and what type of content you’re testing. For example, if you are sending a newsletter to your client list, and you know they are mostly located in a single province/state, it is safe to use a testing time of 24 hours as everyone would be on the same time zone and most likely interacting with the email once they’ve received it. However, if you were running a nurture campaign, over a long period of time, it would be best to conclude results over a longer period of time.

7. Take actions on your Results

If one variation is statistically better than the other, then of course you have a winner! Dynamics will automatically send the winning email to the rest of your segment. If neither variation is statistically better, you've just learned that the variable you tested didn't impact results, and you'll have to mark the test as inconclusive. If this is the case, stick with the original variation— or run another test. You can use the failed data to help you figure out a new iteration on a new test. If the test has insightful results, use those results as learnings and be sure to create more content like that in the future!

To learn how to set up and deploy an A/B test in D365 Marketing watch our video or Talk to an Expert to learn more!

About CRM Dynamics an HSO Company

We are thrilled to announce HSO, a leading global transformation partner has acquired CRM Dynamics! As these two leaders in the industry join forces, we can look forward to the same firm commitment to our customers, partners, employees, and the Canadian Business Community.