As designers, we often face a tough challenge. We create a new design we're proud of, but the A/B test results don't look good. What do we do then?
The Problem
Imagine you've just redesigned your company's homepage. The old version was cluttered, hard to use, and visually dated. Your new design is clean, modern, and intuitive. But then, A/B test results show a dip in some metrics, and your boss—focused only on the numbers—wants to revert to the old design.
This is a common challenge: how do we balance design instincts with data that doesn't immediately support our changes?
Solutions
1. Think Long-Term
A/B tests offer short-term data, but good design often shows its impact over time. Explain that users need time to adjust to a new interface. Ask for a longer test period to assess the full effects.
2. Broaden the Metrics
Don't focus solely on conversion rates. Expand the metrics to include:
- User engagement: Are people interacting more with key elements of the new design? Maybe you've purposefully moved elements so it's expected that some interaction metrics would drop in-favour of others.
- Time on page and bounce rates: Are users staying longer or exploring more pages?
These behavioural metrics often reveal the true value of a design.
Also understanding that some metrics might drop in favour of others is important. For example, if you've moved a call-to-action further down the page, it's expected that the click-through rate might drop, in favour of a new route or action you're trying to encourage.
3. Incorporate Qualitative Data
Data tells you what is happening, but not why. Introduce qualitative feedback by:
- Conducting user surveys or usability tests to see how people feel about the new design.
- Collecting comments or feedback from users to identify benefits that numbers might miss.
4. Show It's Not Just Personal Opinion
Back your design choices with evidence. Reference industry standards or competitor benchmarks, showing that the design isn't just trendy—it follows best practices. Jakob's Law, for example, demonstrates that users prefer familiar design patterns.
Users spend most of their time on other sites. This means that users prefer your site to work the same way as all the other sites they already know.
—Jackob's Law
5. Highlight Technical Improvements
Your redesign might bring benefits beyond aesthetics, such as:
- Faster load times
- Better mobile responsiveness
- Improved SEO performance
These are often invisible to users and your A/B test results, but crucial for long-term growth.
6. Address the Cost of Inaction
Sticking with an outdated design has its risks. It can make your brand seem outdated and cause you to fall behind competitors with more modern interfaces. Additionally, not updating could mean missing out on new technology or industry trends.
Conclusion
Navigating the tension between data and design is challenging. But by focusing on long-term value, broadening success metrics, and backing your decisions with evidence, you can make a strong case for design changes. Remember, good design often takes time to show its full benefits, and data should be just one part of the conversation, not the final word.