Perils of A/B Testing

I’ve been enamored with SquareSpace lately after testing it for my wife’s business site. It’s an amazing tool. Once everything was in place using their free trial, I went into the billing section to purchase the service. That’s where my confusion began.

During the free trial I’ve looked at SquareSpace’s pricing plans several times and they’ve always had an $8 Personal plan, $16 Professional plan, and $24 Business plan, billed annually.


But this time their plans didn’t show any sign of the annual discount: plans were now $10, $20, and $30. I accepted this as the normal cost of doing business—I waited too long to purchase and they’ve raised their rates.

A few days later SquareSpaces rolls out a new feature and posts about it on their blog—Commerce for All (emphasis added):

… That’s why we’ve added the ability to accept donations or cash gifts, as well as sell a single item, to our new Personal plan, which starts at just $8 per month

… For those folks who have basic e-commerce needs without integrations, we’ve created our new Professional plan – with the ability to sell up to 20 products – starting at $16 per month.

For more information on each of our plans, please visit our pricing page. Please note that with this update, our plan pricing remains the same.

Now I’m starting to question my purchase, and for that matter, SquareSpace. I paid 20% more for my product, but a company communication posted after my purchase advertises a lower price and reinforces that prices didn’t change. Hmm.

I decide to go into my SquareSpace Admin section to see if changing my plan gives me the current prices listed on their website. But the Billing section doesn’t even list the correct plans (Personal, Professional, Business) available:


Even more confusing, I singed up for the Business plan which is advertised to have unlimited pages, products, and users. If the Business plan gives you unlimited everything, then what is the Unlimited plan? And why is it cheaper than my current plan?

After a few emails back and forth with SquareSpace, everything was settled. It seems that I purchased in the middle of their A/B testing and just happened to be the lucky winner of a higher rate: $360 vs $288 for the year.

That in itself is not the problem. Like I said earlier, I accepted that the rates raised. But everything on the site pointed to lower rates than what I paid, and no reference to the higher rates I was charged. This led me to believe it was a genuine billing mistake.

The issue is the lack of consistency:

  • Different prices depending on the day
  • No explanation of pricing changes or testing in the support area
  • New blog posts referencing the ‘old’ prices
  • Admin panel showing plan names that I had never seen before

Suggestions for Smoother Testing

  1. Don’t reference constants (such as a price) in blog posts. Sure this isn’t possible all the time, and you certainly don’t want to alienate readers by being coy with key information such as price. However, things change; especially prices. In SquareSpace’s case, they could have referenced plan names rather than prices to future-proof their content.
  2. Perform a content audit before the A/B testing phase. This audit gives visibility to times where you might have talked about an item/topic that is the subject of an A/B test. This also segues into the topic of modifying content of previous blog posts or writings. Is it okay to do? I don’t know, but I think it heavily depends on the company culture, topic, and method of communication. In the case of SquareSpace, I think it would be perfectly fine to edit past posts to remove all references to prices.
  3. Keep updated internal documentation on where key data is referenced. That way when a company decides to move forward with A/B testing, they could reference their internal wiki page on that topic. A page about pricing might say something to the affect of ‘Prices and plan names are listed on …/pricing/ and plan names are listed on …/admin/billing/’. This gives you a checklist of things to update before rolling out the test.
  4. Keep user’s experiences consistent. I signed up for SquareSpace’s free trial when the $8/16/$24 plans were active. But when I clicked the ‘Trial Account — Upgrade Now’ button I saw different prices; I should have seen the same prices that convinced me to create my free trial in the first place. The same thing goes for the email SquareSpace sent saying ‘Your 14 day trial is almost up… upgrade now’. Only new users should be funneled into the A/B test, not users in the middle of the trial/conversion process.
  5. Communicate the test outcomes to your customer service team. Your customer service team is the first line of defense, and often the face of the company for most customers. They should be kept up to date or every customer-facing decision. This cuts down confusion and enables your service reps to solve customer issues at lightening fast speeds. It seems as SquareSpace got this one right!

When All Else Fails

Even if you have the best execution plan ever planned in the history of planning, hiccups will inevitably occur. And when they do, you need to have a top-notch customer service team at the ready to deal with issues. SquareSpace had exactly that.