Click for a Printer-friendly Version
- Adobe PDF
Identifying Millions in Lost Revenue
By Jim Wheaton
Principal, Wheaton Group
Original version of an article that appeared in the
January 7, 2002 issue of "DM News"
Most direct marketers find statistical formulas difficult to understand.
Many respond by ignoring them. Or, they take the easy road
and heed the advice of consultants who say things such as, "You're
fine as long as you have 50 responders."
This widespread lack of sophistication costs direct marketers dearly
in squandered opportunity, in the form of unrealized revenue.
In this article, we'll illustrate the magnitude of the problem
via a two part-statistical formula. This formula, which was
first presented in "How Big Should My Test Be?" (DM
News, October 1, 2001), assists in determining the quantity of names
to include in test panels. We'll see how inputting this
formula into a spreadsheet, and investing an hour or two in experimentation,
might just generate millions of dollars for your direct marketing
A Quick Review of the October Article
The following is the
formula for determining how many names to include in a test panel:
Part 1: (Expected Response Rate X (1 - Expected Response Rate)
X Z2) — Precision 2
Part 2: Answer to Part 1 — (1 + (Answer to Part 1 —
Rollout Universe Quantity))
Understanding "Z" would require a statistics lesson.
All we need to know for our purposes, however, is that it corresponds
to the level of confidence that we have in the accuracy of our test
panel response rate. For example, a given test panel quantity
might result in a confidence level of 90% that a test panel response
rate of 0.8% will translate to a rollout rate of between 0.72% and
The following are six combinations of Z and confidence levels:
a Z of 1.96 corresponds to a confidence level of 95%, a Z of 1.645
to 90%, 1.282 to 80%, 1.04 to 70%, 0.84 to 60%, and 0.67 to 50%.
Precision describes the degree of "plus/minus" uncertainty
around a test panel response rate. We can never know for sure
from a test panel response rate what the "true" rollout
rate will be. For example, with a test panel response rate
of 0.8% and a universe size of 100,000, a test panel size of 5,273
will result in our being 50% confident that the rollout response
rate will be between 0.72% and 0.88%. In other words, one
out of every two times the rollout rate will be within ten percent
of the test panel rate.
You can check this yourself by substituting "0.67" for
"Z" in the formula, and "0.08%" for "Precision."
You'll know that you've done it correctly when you get
an answer of "5,273."
We've just established that one half of the time the true
rollout response rate will be between 0.72% and 0.88%. By
definition, one out of every two times it will be outside of this
range. Therefore, by extension, one out of every four times
the rollout rate will be less than 0.72%.
Generating Millions of Dollars
For most direct marketers,
the effort involved in understanding this formula is less pleasant
than tasks such as sourcing new merchandise and working on promotional
layouts. However, it's just as important. We'll build
upon the example from the previous section to illustrate why this
formula might just generate millions of dollars for your direct
In the example, we established that one out of every four times
the true rollout response rate will be under 0.72%, one out of two
times it will be between 0.72% and 0.88%, and one out of four
times it will be over 0.88%. Of course, in the real world
there is no way to know for sure what the true rollout response
rate is without going through the effort of contacting everyone.
However, let's assume for the sake of illustration that we
magically know in advance that the true rollout response rate is
identical to the test panel rate of 0.8%. Although extremely
rare in direct marketing testing, it does happen on occasion that
the two are the same. Let's also assume that 0.72%
— or, ten percent less than 0.8% — is the minimum test
panel response rate that is required for rollout.
Armed with this information, we know that one out of every four
times a test panel size of 5,273 will result in our failing to roll
out the list select, because the test panel response rate will be
below the required 0.72%. This is a missed opportunity because
the true rollout response rate of 0.8% is comfortably above our
minimum. The financial ramifications of this missed opportunity
are profound because rollouts generally are repeated many times.
Let's assume that we contact proven rental lists three times
a year. By failing to roll out the 100,000 list select, we
will have failed to cost-effectively generate 300,000 promotions
a year, or 1.5 million over a five-year period. Using our
response rate assumption, that's 12,000 missed customers!
Now, let's assume that each new customer will, on average,
order one additional time, and that the size of each order is $90.
That translates into 24,000 missed orders and $2.16 million of missed
revenue over the five years. And, that's from just a
single missed rollout!
There's no right or wrong answer when deciding on test panel
quantities. In other words, there's no one size that
will be optimal for every direct marketer. The appropriate
quantity will depend on factors such as the amount of money available
for testing, and the level of risk the direct marketer is willing
to assume that the rollout response rate will be significantly different
from the test rate. Nevertheless, most direct marketers are
appalled when they are made to understand the profound financial
ramifications of small test panel quantities.
Fortunately in the example earlier, it's possible to minimize
the risk of failing to identify the $2.16 million in revenue.
What has to be done is to increase the size of the test panel.
Let's explore the effects of various panel quantities on the
accuracy of our test reads. Throughout, we'll assume
that promotional costs, including list rental, are $1.00 per thousand:
If we increase our test panel quantity from 5,273 to 8,046, we'll
spend an extra $2,763. With a response rate of 0.8%, we'll
increase our responder quantity from 42 to 64. As a result,
we'll fail to identify the $2.16 million dollar opportunity
just one out of every five times rather than one out of every four.
If we move from 5,273 to 11,826, we'll invest an extra $6,543
in order to fail to identify the $2.16 million just fifteen percent
of the time. That's going from 42 to 95 responders.
Likewise, with 16,930, we'll spend an incremental $11,647
to fail just one out of ten times. Finally, with 25,124, we'll
invest an extra $19,841 to fail just one out of twenty times.
You'll have to decide for yourself which
of these quantities and corresponding costs is the best balance
for you. As the test quantities increase, the chances of failing
to identify the $2.16 million decrease. At the same time,
the costs of testing increase. There is a specific point that
reflects your personal equilibrium.
Be mindful throughout that the incremental costs presented earlier
for increased certainty are based on gross rather than net calculations.
Orders will be generated from the additional promotional quantities,
which will defray a portion of the costs. And, when you are
more confident in your results, you can proceed to full rollout
Most direct marketers have no idea of the extent of the missed opportunities
that result from small test panels. This is understandable
because missed opportunities are — by definition — a hidden
phenomenon. Out of sight, out of mind, so to speak.
Nevertheless, direct marketers who are unwilling to understand the
basics of statistical sampling theory are paying a steep tax for
their lack of knowledge.
Unfortunately, the list brokerage industry — those professionals
whose job it is to make test recommendations to the direct marketing
community — generally do little to educate their clients on
these issues. This is because most list professionals are
no more comfortable with sampling theory than their clients.
However, that's the subject for another article!
Jim Wheaton is a Principal at Wheaton Group, and can be reached
at 919-969-8859 or email@example.com. The firm
specializes in direct marketing consulting and data mining, data
quality assessment and assurance, and the delivery of cost-effective
data warehouses and marts. Jim is also a Co-Founder of Data