Click for a Printer-friendly Version
- Adobe PDF
Explaining Disappointing Rollouts
By Jim Wheaton and Cynthia Baughan Wheaton
Original version of an article that appeared in the
May 6, 2002 issue of "DM News"
Have you ever wondered why so many rollouts are disappointing?
Three of the authors' recent DM News articles established
that a significant part of the answer lies with statistical sampling
theory. However, there is more to the story than sampling
theory! Two other articles have explored how merge/purge can
affect reported list performance. We will continue to focus
on the non-statistical side this month, but will branch out beyond
Statistical testing theory is predicated on the assumption that
all things are equal except for the test parameters. Unfortunately,
in the direct marketing business nothing is ever equal. Some
of the problem is caused by the poor decisions of circulation professionals,
who often have not been trained in proper experimental design.
This is something that we will focus on in a future article.
A significant portion, however, is driven by fluctuating environmental
influences, both competitive and non-competitive. And, you
as a direct marketer must understand and be mindful of these phenomena
in order to correctly interpret rollout behavior.
Non-Competitive Environmental Influences
Consider the following
article, "Piece's Eerie Timing Shatters Response Goal," that appeared
in the January 25 issue of DM News:
"The question, 'Can Islam and democracy mix,'
and the photo of a Muslim woman covered except for her eyes on a
direct mail piece from The Economist that dropped Sept. 4 contributed
to a response rate 30 percent greater than expected·
'Our September piece did better than any campaign
we did in the last three years, both gross and net,' said
Hilde Sprung, promotion marketing director for North America at
The Economist, New York·
"The 1.2 million pieces began arriving in mailboxes Sept.
" 'Our acquisition costs will be very low on this,'
Sprung said. 'I didn't anticipate the response. It blew all
the other creative packages we've tested it against out of the water·'
The Economist should be thankful for its fortuitous timing.
However, it should be mindful of the fact that similar response
boosts were experienced by certain direct marketers during last
decade's Persian Gulf War, and that these boosts did not hold
up over time.
Consider, for example, the veteran's organization that built
a prospecting model off a fundraising mailing that dropped during
the Gulf War. As the organization learned when the model flopped
during rollout, response patterns during this atypical time in our
nation's past were not representative of future behavior!
Competitive Environmental Influences
What the competition
does, and when it does it, can have a major impact on the performance
of your direct marketing promotions. The following case study
illustrates this by focusing on the significant overlap that can
exist between the customer universes of you and your competitors:
Several years ago, a statistics-based model was built to predict
the future purchase volume of a direct marketing client's
customers. This model continues to be very effective in determining
which individuals to promote, and how often.
Subsequent to the construction of the model, the client purchased
its largest competitor. Currently, certain administrative
functions have been consolidated. However, when it comes to
customer and prospect contact strategies, the two direct marketing
companies continue to operate as if they are still standalone operations.
We will refer to the original business as the "original division,"
and to the purchased business as the "sister division."
Recently, the overlap between the two divisions was quantified,
with a mind to determining the level of cannibalization. The
study focused on the significant number of individuals who are customers
of both divisions. The goal was to determine if historical
purchase information pertaining to the original division is predictive
of purchase behavior from the sister division. Specifically,
the predictive model for the original division — which by definition
employs only historical information from the original division —
was measured for its accuracy in estimating order activity from
the sister division. Here's how:
First, the overlapping customers were scored by the original-division
model. Then, the customers were sorted by score and broken
into ten equal groups called deciles. With this approach,
Decile 1 corresponds to the best performing ten percent of the file,
Decile 2 to the second best, and so forth. Finally, responses
received by the sister division within a six-week timeframe were
appended to the file, and tabulated for each of the ten deciles.
It was remarkable just how effective the original-division model
was in identifying which customers were likely to make a sister-division
Original-Division Model Deciles:
Percent Purchasing Sister Division Merchandise
The results illustrate the strong interconnectivity that can exist
between you and your competition.
Controlling for Environmental Influences
To the fullest
extent possible, you as a direct marketer must anticipate and control
for environmental influences when evaluating results. Without
this perspective, you are evaluating list performance in a vacuum.
Clearly, it is more difficult to do this for competitive than for
non-competitive environmental influences. However, it is not
impossible. As long as you are receiving promotions from the
competition, it is a straightforward matter to estimate the degree
You can increase the probability of receiving the competition's
promotions to its house files by ordering multiple times from the
opposition camp. As just about anyone who has ever ordered
from a direct marketing channel has learned, a purchase or two pretty
much guarantees multiple follow-up contacts.
You can also achieve perspective on the competition's prospecting
efforts by ordering from firms whose rental lists have consistently
performed well for you in the past. The list business is very
incestuous, and what works for you is likely to have also been effective
for the competition!
Jim Wheaton and Cynthia Baughan Wheaton are Principals at Wheaton
Group, and can be reached at 919-969-8859 or firstname.lastname@example.org.
The firm specializes in direct marketing consulting and data mining,
data quality assessment and assurance, and the delivery of cost-effective
data warehouses and marts. Jim is also a Co-Founder of Data