A/B Testing for Associations

By Octavio Peralta – July 26, 2019

A/B testing is not related to a blood donation activity that some associations do. Also referred to as bucket test or split-run testing, A/B testing is a method marketers use in conducting controlled and randomized experiments to improve a web site metric, such as clicks, form completions and purchases.

As the name implies, two versions (A and B) are compared. Both are identical, except for one variation that might affect a user’s behavior. Version A might be the currently used version (control) while version B is modified in some respect (treatment). The test determines a winner between the two competing messages or design concepts. By dividing an audience into segments and testing two different messages, marketers will be able to see which communication received the most clicks, opens, likes or shares.

Associations can use A/B testing as a marketing and communications tactic, too, as the experience of the American Art Therapy Association  tells us. When AATA wanted to find the most effective way to bring lapsed members back into the fold, it used A/B testing.

Tim Ebner, a senior editor for the Associations Now magazine of the American Society of Association Executives (ASAE), wrote that AATA applied A/B testing to a rudimentary membership question: Do lapsed members rejoin for instant gratification or for long-term benefits?

Kat Michel, AATA’s membership manager, in consultation with Rita Santelli, president and CEO of the Savvy Org Llc., tested this question with two subsets of former members who have lapsed membership status more than six months ago. Group A was offered a $10 Starbucks gift card for renewing, while Group B was given a three-month extension on their annual membership.

“The good news was that our members seemed to respond more to the long-term benefits of an extended membership offer,” Michel says. The difference was clear: Group B was twice as likely to renew as Group A. This finding now guides AATA’s win-back strategy which emphasizes new membership benefits like a recently launched online member community.

“We were really focused on trying to get those lapsed members who have, for whatever reason, decided that maybe membership wasn’t for them,” Santelli says.

A/B testing sounds simple, but it takes time to set up. Before you try it, check whether your e-mail marketing and association management systems can handle a two-pronged test. “Right now, our AMS doesn’t allow for us to use discount codes or coupon codes, which was a major drawback and added to the staff time for this project,” Santelli says.

This is not to say you shouldn’t give A/B testing a try. You can use manual configurations like e-mail segmentation and suppression, but Michel recommends looking to technology to help automate the process. “Don’t be afraid,” she says. “Just be ready for a few extra turns if you don’t have the right tools in place.”

Here’s how to conduct A/B testing: (1) Pick one variable to test; (2) Identify your goal; (3) Create a control and competing version; (4) Split your sample groups equally and randomly; (5) Determine your sample size, if applicable; (6) Decide how significant your results need to be; and (7) Make sure you’re only running one test at a time on any campaign.

Leave a Reply

Your email address will not be published. Required fields are marked *