We run a lot of multivariate tests on donation landing pages throughout the year. Frankly, I’m always urging our clients to run even more—since the more things they test, the more they learn (and the more money they raise!).
But when Summer rolls around, virtually every organization we work with sees a big seasonal drop-off in their visitor traffic and conversions on their top donation pages. And while a breaking news story or natural disaster can produce a traffic surge, those events are impossible to predict and testing is too difficult to pull off when time is tight.
What can you do in the dead of summer to keep your testing program on track? For most, the answer is simple. Forget about multivariate testing and run an A/B test instead.
An A/B test, also called a split test, is where a single alternative version of a landing page (which features one or more changes) is tested against the baseline page. This is the best choice when your traffic volume is low and/or you want results in a short time-period.
In contrast, a multivariate test allows you to run more than one variable at a time, and evaluate how each variable performs individually as well as in combination with other variables—but requires a large amount of visitor traffic and conversions to execute and can take many months to reach statistical significance.
When traffic is sluggish, it makes sense to focus on your busiest page, which almost certainly still receives enough visitors and conversions to run a basic split test. (Reality check: if your donation page gets at least 250 conversions a month or about 8 per day, go ahead and split test).
The trick to designing an effective A/B test is creating a new treatment that incorporates enough change to make a meaningful impact on the end user. Split tests are not the place to experiment with subtle design changes, e.g. the elimination of a few words of copy or removing 1-2 fields on a long form, since minor tweaks are likely to have little to no observable impact on visitor behavior.
This is the time to shake things up a little. Test an idea (or several) that add up to something genuinely new, such as adding a photo to the page or adding new content that amplifies your value proposition.
While combining several meaningful changes into one test treatment doesn’t permit analysis of how each change individually affects user behavior, A/B tests can lead to bigger optimization gains over a shorter time period than multivariate testing—even during those months when your donors may be off hiking the Inca Trail!
Dawn Stoner is Donordigital’s Director of Analytics & Testing and works with clients to help them increase online revenues with web usability best practices and landing page testing. Dawn speaks regularly about testing and optimization at industry conferences and publishes papers highlighting what’s working and not working with our testing clients.