"Before enlightenment, chop wood, carry water. After enlightenment, chop wood, carry water."
Okay! That's the Zen monk mantra.
Now, tell me, what should be the techie mantra?
I know you know it!
Of course, it's A/B Test, A/B Test, and A/B Test. Before the launch of your website/app, A/B Test. After the launch of your website/app, A/B test.Why? Simply because relying on your guts could prove to be extremely suicidal for your business.
Not surprisingly, the big tech world is hugely betting on A/B tests and experimentations like their life depended on them.
With technology evolving rapidly these days, products are turning obsolete the moment it hits the shelves. This means the inventor company has to keep reinventing its existing products repeatedly, and that too in a fraction of time. This means new variants are to be launched back to back, backed by all the latest bells and whistles time and again. (Think of Apple's iOS upgrades).
And, no, companies can't ever think of "taking breaks" from A/B testing. (Breaks could mean bankruptcy.) Quite the contrary, they must be continually on their toes Ab testing products and services, and thus continually adapt to changing customer tastes to ensure they stay in the race.
Okay! It's time for the rubber to meet the road!
Here I provide you with a roadmap, tips, and tools required for startups to keep up with times and a few incredible A/B testing examples.
But first, let's start with the basic definition of A/B testing.
What is A/B Testing?
If you think A/B testing is about ‘going through the motions', relying on your gut instincts, or following the HIPPO (Highest Paid Person’s Opinion) for testing various versions of your product or services, then sorry, you’ve got it wrong.
Quite the contrary, A/B testing is a field-tested, science-backed testing method that helps engineers compare multiple versions - A (controlled version) from B (challenger version) - by showing them to a predefined percentage of site visitors who help them determine the ideal variant that could leave the most significant impact on the user and drive conversions.
The best part: With each test, engineers get to evaluate a ton of data and make course corrections until they meet the requirements of their target audience. This helps businesses bring their A-game to the table as they get to discard versions that don’t work.
Incidentally, the idea of continual testing is old-school and is not a new-age experiment. Remember how Edison tested his inventions 10,000 times? If you go back in time, there are umpteen examples of outstanding innovators who tirelessly tested their products until they matched ever-changing user expectations.
Culture of Experimentation: How Testing Colonized Edison’s Mind
“I have not failed 10,000 times—I've successfully found 10,000 ways that will not work” - Edison.
Thomas Alva Edison ( the Elon Musk of yesteryears) seemed to have geeked out on his light bulb invention so much so that he even tested the hair of one of his men’s beards, much less 6000 filaments before he got closer to finding the one filament that ultimately worked.
Nikola Tesla, the famed inventor, who worked at Edison’s lab during the light bulb invention, mocked the latter’s perpetual patience by saying, “If Edison wanted to find a needle in a haystack, he would do it at once by examining straw after straw until he found the object of his search.” Today light bulb exemplifies the power of deliberate experimentation that profited humanity in incalculable and unimaginable ways. And that is exactly why Edison’s example is so crucial for driving across the point that matters the most when A/B testing: Success takes time and tries.
To emulate Edison’s success, a start-up needs to be willing to iterate and test just as much as he did, if not more. A/B testing isn’t a one-time task. Instead, it is a process, a work style that lets you get through the 10,000 versions that do not work before you get to the one that does.
And this isn’t just true for light bulbs; if we dig into history a bit, there’ll be numerous examples that demonstrate the power of deliberate experimentation or A/B testing.
More Continuous A/B Testing Examples from Recent History
- Einstein published 248 research papers; however, his research paper on relativity got world attention.
- Paul Erdos co-authored 1,500+ mathematical research articles; however, only a handful of them earned him the reputation of being the top influential mathematician of the 20th century!
- The iconic cartoons that appear in The New Yorker are the result of a taxing in-house process followed by the magazine editors wherein over 50 freelancers are requested to submit up to 10 sketches each for consideration every week.
The above examples testify to the fact that A/B testing is a tried and tested formula that has worked for individuals and companies alike in the past and present and would most likely in the future.
The same idea of rigorous testing that inspired the great thinkers of the past applies to the world of tech and product development.
In fact, top tech businesses bring their ultimate game to businesses using an unstoppable A/B and Beta testing process. For the skeptics out there, here are a few examples.
Continuous A/B Testing Thereby Building Culture Of Experimentation: A Proven Game Plan Of Big Tech Titans
The tech world is not a friendly place. Technology expansions are happening at a lightning pace. Think Chat GTP. And the only thing we now know for sure is that the rate of change will be faster and faster in the months and years ahead. Little wonder, product and engineering teams are facing the heat day in and out. Thankfully, there’s a way out. The continuous A/B testing rule helps businesses navigate and adapt to this precarious technology plain by helping them conduct bite-sized experiments at a faster rate.
SpaceX: SpaceX was in the news lately when its super heavy rocket - touted to be the largest and most powerful rocket ever built - exploded midair. Despite the heavy loss in terms of money and manpower, SpaceX engineers were upbeat about the event as they had anticipated the explosion, and their only goal for the test flight was to gather data for the next launch regardless of the current mission’s results.
Facebook: The Disappearing Messages and Stories feature of Facebook wasn’t launched in a day. In fact, it was the result of continuously failed experiments that the social media giant set in motion around disappearing messages and then silently killed.
“One of the things I’m most proud of that is really key to our success is this testing framework … At any given point in time, there isn’t just one version of Facebook running. There are probably 10,000.” - Mark Zuckerberg
Netflix: If atypical Netflix user doesn't find anything worthwhile in the app within 60-90 seconds, she might get bored and move on to something else. Little wonderNetflix researchers have to be on their toes continuously A/B testing every bit of their content and even the loading speed to optimize their UI.
“By following an empirical approach, we ensure that product changes are not driven by most opinionated and vocal Netflix employees but instead by actual data, allowing our members themselves to guide us toward the experiences they love.” - Netflix
The key takeaway here is that most experiments fail. According to a study that reviewed the success rates of experiments at Amazon, Microsoft, and other big tech companies, less than 50% of the experiments proved to be successful. In fact, these failures may have cost Amazon a bomb, but Bezos dismisses them as the ‘cost of innovation.’
“One area where I think we are especially distinctive is failure. I believe we are the best place in the world to fail (we have plenty of practice!), and failure and invention are inseparable twins. To invent, you have to experiment, and if you know in advance that it’s going to work, it’s not an experiment. Most large organizations embrace the idea of invention, but are not willing to suffer the string of failed experiments necessary to get there.”- Jeff Bezos
A Glimpse into A/B Testing Statistics Run By Other Top Businesses Annually
- Even in 2008, when Google was a much smaller company, it ran 50–200 experiments with the help of search engine users.
- Booking.com runs 1,000+ rigorous tests simultaneously and 25,000+ tests a year. At any point in time, quadrillions of landing-page options are live. Simply put, no two customers from the same location are likely to see the same version.
- At any given point in time, brand Expedia works on 50 different versions of the website and provides daily updates to its customers. Leveraging data analytics, the company’s engineering team can figure out small changes and nail the version that consumer really likes and then quickly adapt to it.
Long Story Short: Businesses are dialing in on continuous A/B testing in a big way, so I don’t see any reason why startups should shy away from it. On the contrary, with 99% of startups failing these days, it’s high time businesses gear themselves toward the continuous A/B testing formula, just like the tech behemoths, before they opt for a full-fledged launch.
“According to the Law of Cause and Effect, if you do what other successful people do (the causes), you will soon get the same results (the effects) that they do,” Brian Tracy.
So far, we’ve discussed what A/B testing is and why it matters, as well as seen various examples of A/B testing being applied in history and tech. But how do you go with A/B testing as a start-up? Here is a roadmap to get you started!
How to do A/B Testing for Startups
As mentioned above, A/B testing is not the exclusive province of big tech. Even for startups, it could act as a GPS system, pointing to the direction of your target audience and taking feedback from them, and making course corrections until you achieve your goal. Without much much ado, here’s a brief breakdown for startups interested in A/B testing.
- Know your Goal: Ask yourself:Why do you plan to conduct an A/B test in the first place: are you looking for more sign-ups for your newsletter or additional traffic on your landing pages? Figure out your conversion metric in short. This ensures that test results are relevant.
- Frame Hypotheses: Based on user behavior, frame your assumptions. Your assumption should clearly state what you plan to change, the expected outcome, and your justification. Running the experiment could either validate or invalidate your hypothesis.
- Find an A/B Testing Tool: Most likely, your email marketing tool provides this facility. However, if you are looking for a dedicated free tool, then try Google Optimize, if you have budget limitations (more on this below). If not, you can check out a few options here.
- Design Variations: Create variant B or more variants with the changes based on your hypothesis. The variants could be for a design element such as color, layout, size, and so on, or a feature, for example, checkout process, pricing, and more. No matter what, test only one variable at a time because testing multiple variants won’t help with conclusive results.
- Set up the Experiment Group: Divide your user base into groups to determine who will see which variant. However, one thing you need to consider here is that you need to determine how many participants should be part of the group so that test results are statistically significant. According to conventional wisdom, the larger the sample size, the more reliable the results. But then, more data doesn’t mean better data.
- Set up the Test Duration: Determine how long you want your test to continue to ensure that users get enough time to interact with both or all the versions of your product or website and don’t rush through the entire process.
- Run the Experiment and Analyze Results: Once the test is over, it’s time to gather data and take action based on it. The changes could go either way: you may stick with the A version of your website or app or make changes as the B version demands.
- Be patient. It takes time to collect data and arrive at meaningful conclusions.
5 Ways Culture of Experimentation Power Startups
From 2015 to 2018, 4,645 startups adopted A/B testing the technology on their websites. On average, those firms had roughly 10 percent more weekly page views, a 5 percent greater likelihood of raising VC funding, and launched 9 to 18 percent more products. - Research by Rembrand Koning
#1. Scale Faster, Fail Faster
Scale Faster, Fail Faster: That’s the startup mantra.
It has been observed that startups, particularly technology companies, must scale up their operations more quickly or face stiff competition. This is where rapid A/B testing comes in handy as it allows them to separate the chalk from the cheese in terms of what features or functionalities would resonate with the audience and which should be omitted. And in some cases, only through repeated A/B tests of a product or feature can they arrive at this differentiation.
Put differently, the more wood they chop and water they carry, the more likely their ultimate product or website will be successful.
Say, for instance, you are planning to launch a clothing website.
Version, A of the website features large images of glamorous models wearing your brand’s apparel, while Version B plays up on the pricing and the product description partsrather than focusing on the models.
Variant A of your website could be like this - complete focus on models and products.
Variant B of your website could be something like this – a balanced approach to product and pricing with less emphasis on models.
After a few weeks of A/B testing, you find that version B is 10x times more popular with your targeted users than version A. The success of the B version proves that users were more interested in the look and feel of the product and its pricing over the other model.
The key takeaway here is that the testing process enables you to fail faster by quickly identifying and discarding the underperforming version and optimizing the version that works.
Pro tip: To scale faster and stay ahead of the competition, startups should A/B test their CTA buttons, product descriptions, and marketing emailers. Even the color of the CTA button matters. So, A/B tests that as well. Yes, chopping wood, and carrying water, should be followed relentlessly regardless of what you are launching.
#2. Compounds Performance via Incremental Changes
The firms that experiment continually learn and grow faster and gain a unique edge over competitors. Plus, they are able to arrive at quicker solutions to complex problems and also develop newer products.
Consider Dulingo’s 1% improvement every week strategy. Since its inception, the education app has run over 3,000 A/B tests to gauge user response regarding its app features. In fact, it has been reported that the company runs almost 200 tests at any given time, which apparently is in keeping with one of its operating principles: Test Everything.
All those experiments helped the company generate an avalanche of data that guides the company’s decisions. Little wonder, the company’s experimental culture has aided it in boosting the app from 13% D1 retention in 2011 to 55% D1 retention today.
#3. Facilitates Hiring of a Skilled Workforce, Increased Idea Generation, and Increased Number of Tests
As opposed to design-thinking and lean startup experimentations, A/B testing is cost-effective. The money saved from carrying could be diverted into several complementary practices, be it hiring a skilled workforce to help companies chart new growth paths or, more importantly, encouraging companies to conduct more tests.
John Cline, Director of Engineering at Blue Apron, revealed in Rembrand Koning’s research how A/B testing platforms gave his company the flexibility to perform more tests and generate more product ideas.
“Now that we have this capability, other groups have started using it. We went from one or two teams doing one or two tests a quarter to now when we probably have at least 10 tests live at any given moment. A large number of tests every quarter is run by every product team.“ -John Cline, Director of Engineering at Blue Apron
In the early days, companies behaved like typical copycats when executing tactics. They simply imitate their competitors. But then, when you think about it, their approach wasn’t entirely wrong. Due to a lack of data, they played safe, as executing any idea or concept could result in hara-kiri.
No more! Now with A/B and manual tests being the order of the day, companies can use insightful, intelligent data to implement alternative strategies.
Allan Willie, a co-founder of business dashboard startup Klipfolio, explains how A/B testing helped them choose a suitable pricing model for their startup instead of relying on generic templates used by competitors.
“Take the time to understand what pricing model works best for you. Most people don’t. I’d say that most companies merely copy their competitors and for lack of a scientific process, just guess. I feel strongly that a business needs to actively test various models to make sure the price it sets is strategic and aligns with its objectives.” - Allan Willie, a co-founder of business dashboard startup Klipfolio
Further, A/B testing silences the impact of failed ideas. How? As more and more MVPs get rolled out and tested, failed ideas are replaced by successful ideas in no time.
Thus, A/B testing promotes overall organization learning by encouraging idea generation and offering a rigorous scientific process to test assumptions before implementation.
#4. Boosts Confidence and Facilitates Radical Changes
Small and incremental changes are good enough for startups. But small changes are insufficient in specific scenarios when expectations are quite high.
In the following cases, radical A/B testing should be done:
- When the site traffic is low
- Minor tweaks are not yielding positive results
- Current changes seem half-baked and unsatisfactory
- When significant gains are on your agenda
# When the site’s traffic is low
Yes, when the site traffic is low, expecting instant results from minor tweaks is pure foolishness. Instead, if you A/B test two radically different designs, you can arrive at conclusive results faster and in much less duration, and more than anything else, you could find answers to your low-traffic rate.
# Small tweaks are not yielding positive results
You may have tested every element on your website. However, these constant incremental changes might reach a point when these changes might not yield any results and sometimes even negative results. This is because your current design may have reached a maximum conversion potential called “local maximum.”
In such cases, try testing your fully-optimized page against a radical new design and see how it’s performing.
# Current changes seem half-baked and unsatisfactory
Minor tweaks and changes work when your company is new to A/B testing or when you want to convince your management of the importance of A/B testing.
On the other hand, if you are aware of the benefits of A/B testing and you are a seasoned pro who sees that current changes are not yielding good results, then bold tests are the way forward.
The bottom line is that if you are making minor changes to a website that demands a complete overhaul is a sheer waste of time.
# When significant gains are on your agenda
Unarguably, minor tweaks are helpful and enable small wins. But if you are eyeing colossal gains, you must replace small tests with radical redesigns.
Is it a risky proposition? Yes, it is. But if you handle it right, you can surely hit the jackpot.
As per industry practitioners, rather than incremental changes, radical overhauls yield more successful outcomes. The reason is that radical changes are backed by clear hypotheses, unlike gradual changes—incremental changes, on the other hand, slightly tilt towards the speculative.
Google Optimize - A Free A/B Testing Software to Help Startups Get Started
As a startup, it’s much more obvious that you’ll be bootstrapping your business. So, here’s a free A/B testing software from Google called ‘Google Optimize’ that could help you get started. Read about its core features and extensive customer reviews on GoodFirms.
When Startups Shouldn’t Think About A/B Testing
So far, so good! A/B testing is, of course, a valuable tool in a startup's arsenal. But there's no denying that many startups should only adopt it if they have 1000 transactions, says conversion specialist Peep Laja. If your signups, leads, or purchases total less than 1000, you better focus on other stuff.
And, no, he continues, micro conversions aren't synonymous with higher revenue. By overpromising and setting false expectations, you are just transferring the problem to the next page, where the users may be required to pay, but they won't.
My point: So, if you are still unsure whether your micro-conversions are contributing to your bottom line, you should spend your mental energies on bettering your product.
Make Chopping Wood, and Carrying Water A Standard Operating System at Your Company
Simply put, make A/B testing a standard operating system at your company, just as chopping wood and carrying water would be for anyone training to survive in the woods. There are no two ways about it! A/B tests your product and succeed. or, be rest assured, the loss is around the corner.
Placing bedrock belief in this experimentation process assists businesses in continually iterating the product and services to meet evolving user expectations.
Nonetheless, implementing deliberate experimentation at every step of the product development journey requires you first to foster an experimentation culture within your organization. This you can do by viewing every failure as an opportunity and offering a better satisfying user experience. But then, yes, A/B testing tools only can help you so far; instead, roping in an A/B testing company will render better results. This will make you feel more empowered and confident to carry on with such tests.
