Creating a good test of professional knowledge takes work. This kitchen has its subtleties. Not owning them, even the most qualified expert risks making many mistakes. We must deal with these mistakes regularly, introducing the methodology for developing and conducting professional tests. Testing will fail even due to one error by the webmaster in setting up or running an experiment. This article will list the popular, but only partially obvious mistakes web admins and optimizers make.
Launched a sequential page check
Some webmasters set up page rendering by running the main page for X time, stopping it, starting a new version for the same time, and then measuring the difference. This is a mistake.
If something happens during the test, it will only affect one page. It can get a surge of new traffic, resulting in pages having different results for reasons beyond their control.
For a pure AB test, it is important to split traffic from one channel between two versions and set up page display simultaneously so that external factors do not affect the result.
Set to show at different times
Some tools allow you to test different times or other days of the week to see how traffic performs over different periods. Useful if you want to know when your site has the most visitors. But it will only hurt in cases where you show two groups of audiences on different pages.
For example, a business blog gets less traffic on weekends. If you run a test with a control page from Monday to Wednesday and from Friday to Sunday with an updated one, then the second one will have less traffic and different results.
The test compares pages, with the only difference being the updated element. Everything else should be the same.
Run tests during seasonal events or major site changes
Tests should be carried out during something other than Google or Yandex core updates, major world events, sales, and holidays. These events can bring down the results; waiting until everything calms down is better.
The exception is if you want to test the change in audience behavior at this particular time.
I didn’t check if everything works
This is the simplest mistake, but testing is often launched with broken buttons, old links, and layouts.
Check the points:
- successfully passed from entering the site to conversion;
- pages load quickly;
- the design looks as it should; the layout and fonts have not moved out;
- all buttons work;
- the page opens correctly on different devices and in other browsers;
- have you set up conversion tracking;
- you have error reporting set up if something breaks;
- you tested the same on non-cached devices, as sometimes the information in the cache doesn’t match what the page looks like.
These are worth checking out before running the test and driving traffic. Similarly, before launching, you need to check the updated version of the page.
Launched a test for a closed or incorrect URL
A simple but common mistake is to run an experiment on a “test site” where the webmaster has made changes.
Check which pages you are using. A webmaster, out of habit, can go to a secure page with his access, check it, and run a test. Only the audience will not open it.
Conducted a test without a hypothesis
Some site owners run a campaign and see what changes without thinking about the hypothesis to be tested. They consider the test sample successful if the new page shows some conversion.
You can only improve a page by analyzing what results it currently has. The updated development may reduce the conversion, but the webmaster will only know about it if he tracks the starting results.
It is important to formulate a hypothesis about where the problem is, its cause, and how to solve it. You will get more leads, conversions, or sales if you know which element you want to improve.
Focused on the surface
Not every increased indicator indicates the effectiveness of the updated page. Avoid indicators that are unrelated to and do not lead to measurable results.
For example, an increase in Facebook page reposts does not mean an increase in sales. You can spend resources to remake pages in the version that showed a rise in shares, but you will waste your energy. Remove social media buttons and see how many leads you get.
Be careful with “vanity metrics” likes, followers, views, and reposts. If they don’t affect conversions, you may target the wrong audience or forget to sell to them.
We paid attention only to quantitative data
Not only quantitative test data are important. For example, the test shows that X people did not click on the button, but one can only guess why:
- Is the button invisible? Is it too low?
- I need to understand why to connect.
- Does the offer match what the user wants?
- Does the button look unclickable?
- Does the button not work at all?
Quantitative data cannot always tell the reasons for such results. Testers need to learn from the audience what they need, what motivates them to take action on the site, and what holds them back and repels them. This information is useful for formulating new ideas, hypotheses, and tests.
Focused on the little things
Take on small, high-impact tasks first that will bring big results.
A webmaster can test the fifth iteration of a page with a new button design when more important pages lead to a conversion. Prioritize first:
- Will this page directly affect sales?
- Are there other pages on the conversion path that need to catch up?
Focus on them first.
It’s fine if you’ve achieved a 1% increase in conversions on a sales page, but it’s better to increase conversions by 20% on page users explore before buying. This may be more important, especially if you lose most of your audience on this page.
Tested several changes at the same time
There are radical tests when the webmaster changes many elements or redoes the entire page altogether. It might work, but you won’t know which page change worked.
Most often, during the test, one thing is changed, for example:
- title;
- image;
- content layout;
- prices;
- registration of discounts;
- registration of tariffs;
- CTA buttons and more.
Tested on traffic not suitable for the target
Ideally, a webmaster should test both pages on an audience from the same segment. They usually try on new visitors to see how they react when they visit the site for the first time. Sometimes you may need to test on repeat visitors, email subscribers, or paid traffic.
You only need to test one segment at a time to get an accurate picture of that group’s interaction with the page. Select the audience you want to work with and remove all others when setting up.
Did not exclude repeat visitors from the test
If a visitor sees a page on the site, closes it, comes back, and sees another version, he will react differently than if he got the same performance on both visits. He needs clarification, sows suspicions about the site’s security, or may already know where to click on the first visit.
The results will become less objective due to these additional interactions. Use a tool that shows the user a random version of the page but keeps it the same on repeat visits until the test is over.
Running a test too short
There are three factors to consider when testing:
- statistical significance;
- sales cycle;
- sample size.
Many site owners end tests when they see that one page is better. In a short period, the excess of one of the pages in terms of conversions may be accidental.
Sales and attendance may fluctuate depending on the day of the week or month. If the test falls on a day when many companies pay salaries, you will have a lot of sales.
Focusing on the test duration in two to four weeks is better. During this time, you can get enough traffic for the results to be accurate. Decide in advance what sample size you need; wait to stop testing until you reach it.
Taking the test too long
Delaying the test can also be harmful. If the test runs for over a month, users’ cookies will likely be lost. If these users return to the site, they will be counted as new ones and corrupt the sample data.
Spying on the progress of the experiment
Some testers peek at the test while it is running. In this case, there is a great temptation to correct something, to complete it. Ideally, one should only look at the progress of the experiment once it reaches statistical significance and a sufficient sample.
On the other hand, no one would like to find out a month after the launch that there was a failure on the first day or something broke on the page. To prevent this from happening, 24 hours after the launch, check if everything is working and if there are visits and conversions.
All decisions are made after testing is completed. The only change that can be made during the test is to fix what is broken.
Did not stop the test with accurate results
There were times when webmasters forgot to stop the test. He continued to work and feed 50% of the audience to the weaker page and only gave 50% to the clear winner.
Changed decision time
Another thing to consider when testing is that new elements can affect the time it takes for a user to make a purchase decision.
Example: A company’s leads typically have a 30-day or even longer sales cycle. The webmaster is testing a new call to action that affects decision time. For example, it creates a shortage or offers bonuses for immediate purchases. Then a new CTA might skew the results. The control page might have the same number of conversions, but due to the longer sales cycle, purchases go beyond the testing period and don’t count.
Review your analytics during and after the test to ensure you get everything.
Abandoned the hypothesis without testing other versions of it
If the idea failed during the test, it might mean that its implementation was unsuccessful. The idea itself may be correct.
Try new CTAs, different designs, layouts, images, and text. You have an idea and can choose the best shape for it.
Didn’t look at segment results
The new version of the pages can show low conversions on desktop but give a 40% increase on mobile. This can only be known by segmenting the results. Look at the information on your devices and generally explore the different channels.
Didn’t scale successful solutions to other pages
Changes that perform well in the test may also work on other pages. Found a winning sales page option – try it as a landing page in advertising. You’ve found a great lead magnet style — test it out across your site.
But only make big changes with testing. What works in one area may fail in others, so everything is worth checking.
Stuck on one page
The page you’re testing on may reach its “local maximum.” This situation is when it comes to a plateau and the webmaster fails to increase its performance. You don’t have to fight for continued improvements on one page; you can move on to others participating in the conversion chain.
An increase in conversion from 10% to 11% on a sales page may be less significant than an increase from 2% to 5% on a page that sends traffic. It may even turn out that her growth helps this stuck page by giving her more leads.
If you can’t make a page even stronger, find the next most important page and work on it.
Not tracked other important results
The ultimate goal of a company is sales. Before determining the winner of the test, you need to compare different indicators. For example, a new call to action on a test page gets fewer clicks. But the clicks it does get lead to more sales from motivated users.
Tests not documented
Creating an internal database of tests can keep you from repeating mistakes. You will be able to learn from old tests and not run the risk of testing that you have already taken. The database should contain data about the page, hypothesis, successful and unsuccessful decisions, growth volume, and other indicators.
Conclusion
Conducting tests always require constant monitoring and improvements. If you are faced with quickly obtaining results useful for business and not wasting time manually setting up an experiment, take a closer look at test automation.
