The typical mindset for testing content on a website is to find which variation of site elements performs the best, and then implement those changes permanently: make a test, discover the winner, turn the test into what 100% of visitors see, and then move onto the next test.
There is a big flaw with this mindset: it assumes visitors don’t change preferences over time. The winning variation was good this week but that is no reason to think that it will be the winning combination the following week. One of the least understood aspects of e-Commerce is how much web visitor behavior changes from one time interval to another.
Giving the test more time to run doesn’t really solve this problem because all visitors who visit once the test has been declared over could be different than the ones that visited during the test. A winning variation is only winning during the time it was running.
The solution is to have the variables run constantly. A winner is never declared and implemented permanently because there isn’t only one clear cut winner. As the seasons and visitor preferences change, the system changes with it. For example: the combination of variables that are successful in the summer, when the products are full price and customers weigh pros and cons, read reviews and analyze costs to benefits, will be very different than the variables that succeed during holiday, when visitors are under a time crunch, want nothing more than a low price and to get in, buy and get out.
For this reason saying something like, “we need to do some testing to make sure our site is in tip top shape for holiday,†can cause some serious problems. How visitors act now is not how they will act then.