Creative testing and audience testing
Settle one experiment and change the other, and the account starts answering questions.
You changed the video and you changed who it was shown to, in the same week, because both felt overdue and running one test seemed more efficient than running two.
Something then happened, up or down, and you now have an explanation for it. So does everybody else in the conversation, and all of the explanations are equally supportable, which is the same as none of them being supportable.
There are two experiments hiding in that week. One asks whether this creative is better than that creative. The other asks whether these people respond better than those people. Run together, they cannot be separated afterwards by any amount of analysis, because the information required was never collected.
What are you actually asking?
Before anything is set up, write down the question in one sentence, and notice which of the two sentences it is.
"Does opening on the result beat opening on the problem?" is a creative question. The answer is useful for years and it changes what gets filmed. "Do people who already follow us respond differently from people who have never heard of us?" is an audience question. The answer changes where the money goes and it usually expires sooner, because audiences move.
Both are worth knowing. Neither can be learned from a test that changes both.
Why can a mixed test not be read afterwards?
Because the result has two possible causes and the data contains no way to separate them.
Suppose the new creative shown to the new audience does better than the old creative shown to the old audience. That is consistent with the creative being better. It is equally consistent with the creative being worse and the new audience being much better. It is also consistent with both being slightly worse and something outside the account, a season, a competitor, a news story, moving the whole thing.
Nothing in the numbers distinguishes between those. People resolve it by picking the explanation that matches what they already believed, which is not analysis. It is confirmation with a spreadsheet attached.
Which one do you settle first?
Settle the audience, then test creative against it.
Two reasons. Creative answers last longer: a hook that works tends to keep working, and what you learn feeds directly back into what gets filmed. Audience answers are more perishable and depend on how the platform is behaving that month.
The second reason is practical. Testing audiences properly needs several distinct groups running side by side for long enough to separate them, which spreads a modest budget very thin. Testing creative can be done inside one audience, so the whole spend is doing one job.
If you already have a defensible view of who your customers are, take that view, hold it still, and put everything into creative. That is the order that suits a small budget and it is the order most small accounts should use.
What does a clean creative test look like?
Same audience. Same placements. Same period. Same offer. Same everything, except one deliberate difference.
The difference should be big enough to matter. Two openings that make genuinely different promises will teach you something. Two openings that use different word orders to say the same thing will not, because they are the same advertisement with a rewrite.
Give each version enough of a run to be more than a rumour, and do not stop one early because it looks bad on day one. Early numbers swing wildly, and a version killed at the first wobble was never tested, it was just judged faster.
Then write the answer down somewhere permanent. The point of a test is not the campaign it was run in. It is that the next twelve videos get filmed differently.
When is a difference not a difference?
Most of the time, at modest spend, and this is the part nobody says out loud.
When the number of people involved is small, results bounce around on their own. Two identical versions of the same ad, shown to the same kind of people, will not produce identical numbers, and the gap between them is not information. It is noise wearing a decimal point.
The practical rule: if you have to squint to see the winner, there is no winner. Take the one you can produce most consistently and move on to a bigger question. Chasing a small gap is how an account spends its entire budget on tests that resolve nothing, and it is a more expensive mistake than running no tests at all.
A difference worth acting on is usually visible without arithmetic. That is a low bar and it is the right one.
What to write down before it starts
Four lines, before anything goes live. The question, in one sentence. The one thing that differs between versions. What is being held still. And what result would make you change what you film next.
That last line is the one that gets skipped and it is the one that turns a test into a decision. If no outcome would change anything, the test is a way of feeling rigorous and the money would do more sitting in the better-performing version.
Take the last test you ran and try to write those four lines about it retrospectively. If you cannot, you now know what the next one has to look like, and that is worth more than whatever the last one appeared to show.
Keep reading
This is a service, and the method is written down.
Everything above came out of doing the work rather than writing about it. If you want the method instead of the story, it runs in order on one page.
Start with a free audit
Tell us where your content is now. We will come back with what we would change and what result to expect.
A person reads the channel and writes the audit by hand: a considered read typically takes three working days. That is the usual shape, not a promised turnaround. We use these details only to reply to you: no lists, no lurking.
What you will get
A fit snapshot: where your channel stands, and whether we are a match.
Two to three opportunities: specific, prioritised, yours to keep.
A recommended next step, even if that step is not us.
The audit is free and commits you to nothing: nobody follows up with a call you did not ask for.