Marketing in a Minute – Episode 22 | Ahmed Moaaz

Version “A” beat version “B” by a small margin – is it a real difference or just noise? Before declaring a winner in any test, make sure the sample is large enough to be significant.

The question that precedes every decision

You tried a new ad against the old one, and the new one achieved a 3.2% conversion rate versus 2.8% for the old. Do you build on this difference and change everything? Or is the difference statistical luck that would have appeared even comparing the old ad to itself? This is the question of statistical significance – what separates a decision built on data from one built on random fluctuations.

Why do small samples deceive you?

Imagine flipping a coin ten times and getting seven heads. Is the coin biased? No; that happens by chance often. But flip it ten thousand times and get seven thousand heads, and the coin is biased almost without doubt. The same logic in your marketing: an ad that made 4 sales from 100 visits (4%) and another that made 3 sales from 100 visits (3%) – the difference between them is a single sale, comfortably inside the zone of chance. But 400 sales from 10,000 visits versus 300 from 10,000 is a real difference worth a decision. The rule: the smaller your numbers, the greater the probability that the difference you see is random noise, not truth.

Two concepts determine the answer

• Sample size: how many visits or views entered the comparison? There is no single magic number, but hundreds of visits per version are the practical minimum for conversion decisions, and thousands are better when differences are small.
• Confidence level: the common standard is 95%, meaning that if there were no real difference between the two versions at all, the probability of your data showing a gap of this size or larger by chance is 5% or less. Some big decisions deserve 99%, and some quick small experiments can accept 90%.

Free significance calculators are everywhere: enter the visits and conversions for each version and they tell you whether the difference is significant. Use one before every comparison-based decision.

Fatal experiment mistakes

• Stopping the test at the first result you like: results swing in the early days; whoever watches daily will often see an early fake “winner”. Set the test duration in advance and commit to it.
• Changing several things together: you changed the image, headline and price at the same time, so you will never know which made the difference. One variable per test.
• Unequal time periods: comparing a week containing a holiday with a normal week, or month-end (salary time) with month-start. The period’s circumstances themselves may be the real “variable”.
• Ignoring days of the week: buyer behavior differs day to day; make the test cover whole weeks, not fragments.
• Judging a new campaign early: the platform’s own algorithm is still learning in the first days and results are unstable. Give it the learning period before comparing.

How do you run a clean test step by step?

1. Write your hypothesis first: “I expect the video ad to beat the image in conversion rate because the product needs visual explanation.” A written hypothesis stops you from moving the goalposts after seeing results.
2. Set the success number in advance: what difference would actually make you adopt the new version? A tiny difference may not be worth the cost of change.
3. Split the audience randomly and fairly: the platforms’ testing tools do this automatically; do not interfere manually in the distribution.
4. Let the test run: usually two to four weeks, until you reach a sufficient sample size – not until you see a result you like.
5. Judge with a significance tool: enter the numbers in a statistical significance calculator, and do not build on the raw difference alone.
6. Document the result: record what you tried and what you learned even if the hypothesis lost; a documented loss saves you from repeating the same test a year later.

A practical example

A store tested two landing pages for two weeks: version A received 2,000 visits and 58 orders (2.9%), and version B received 2,000 visits and 74 orders (3.7%). The difference looks tempting, but the significance calculator says confidence is only about 84% – below the 95% standard. The right decision: extend the test instead of adopting immediately. After four more weeks at the same visit rate, the sample reached 6,000 visits per version and B kept its lead with confidence above 98%, so adoption became a grounded decision, not a gamble.

When do you not need a statistical test at all?

Statistical significance is an important tool, but it is not a tax you pay on every decision:

• Obvious errors: a button that does not work on mobile, a wrong price, a painfully slow page – do not wait for a sample; fix immediately.
• Huge differences: a version achieving double the other with reasonable numbers needs no calculator to name the winner.
• Easily reversible decisions: a button color or section order can be tried and observed without complexity, because the cost of error is nearly zero.

Save testing rigor for costly decisions: changing pricing strategy, rebuilding the product page, moving budget between channels. Only there is significance worth its time, because the cost of a wrong decision is genuinely large.

Finally, remember that the goal of testing is not proving you were right, but knowing the truth early and at the lowest cost. The experiment that disproves your hypothesis saved you from a big wrong decision – and that makes it a success, never a failure, however disappointing it looks.

Summary

Every difference you see in your numbers falls into one of two possibilities: a truth worth building on, or luck that will punish you if you build on it. The decider is sample size and confidence level. Write your hypothesis in advance, change one variable, wait for a sufficient sample, and judge by the significance calculator, not by impression. The professional marketer is not the one who runs the most tests, but the one who tells their real results from their noise.

Keep following Marketing in a Minute: one marketing concept per episode. Have a question about this number? Send it on WhatsApp.