Click for a Printer-friendly Version
- Adobe PDF
Common Errors in the Use of Overlay Data
By Jim Wheaton
Principal, Wheaton Group
Original version of an article that appeared in the
September 12, 1994 issue of "DM News"
Individual and household overlay data often play major roles in
descriptive as well as predictive research. But the careless
use of this information can result in more harm than good when it
comes to making accurate marketing decisions. However, by
adhering to certain methods of incorporating overlay data into research
and by properly interpreting the results, many of these errors can
Descriptive Research Applications
Handling Missing Data
One common error in descriptive
research applications results from the fact that individual and
household overlay data invariably cannot be applied to a significant
percentage of a given file. The portion for which a specific
data element cannot be applied generally ranges from 20% to 95%.
Therefore, whenever marketing decisions are based on a given demographic
or lifestyle variable, often it is implicitly assumed that those
individuals for whom data coverage does not exist have the identical
Consider a file in which the average age of the codeable records
is 44. Any marketing decisions that result from this information
will be appropriate only if the uncodeable portion of the file also
has an average age that approximates 44.
Unfortunately, uncodeable individuals almost always are demographically
different from the codeable, because representation on major overlay
databases is skewed towards older, more stable individuals.
I call this the "Ozzie and Harriet factor." The
extent to which an individual has a mortgage, children, credit cards,
and the like is the extent to which this individual is likely to
be represented on a given overlay database. Conversely, those individuals
who cannot be matched to an overlay database tend to be young renters
who move frequently. These people generally also are not affluent
and not married.
Let's get back to our example, in which the average age of the codeable
records is 44. This is exactly what happened to the client
of a major data compiler who, finding this average to be counter-intuitive,
sought a second opinion. Fortunately, techniques exist to
adjust demographic and lifestyle profiles for the systematic bias
that is inherent in missing data. Application of one such
adjustment algorithm shifted the average age of the file from 44
to 30! This lower estimate agreed exactly with the client's
"gut instinct," as well as with extensive survey research.
Hazards of "Marketing to the Mean"
mistake is what I'll call "marketing to the mean." It is critical
to look beyond the average to the distributions of a given variable.
A real-life example is a well-known fashion magazine whose average
adjusted subscriber age is 36, but who in actuality has two target
- New-to-the-workforce 18 to 22 year old women who view the magazine
as a "wish book.
- Affluent women in their late-40s to mid-50s who reference the magazine
when making purchase decisions.
In fact, individuals who are the average age of 36 are very poor
prospects because many are parents with mortgages and little discretionary
power for high-priced fashion merchandise.
Multiple Overlay Variables
Another common error is
the assumption that the demographic and lifestyle overlay variables
that stand out or "pop" on a file all describe the same group of
individuals. Assume, for example, that the following characteristics
are over-represented on a file of diamond ring buyers: "young,"
"male," "affluent" and "married." It could be hazardous to
conclude that the target audience is young, affluent, married
males. There just as likely could exist multiple audiences,
- Young (single) males (of various income levels) who purchased an
- Affluent couples (of various ages) who bought a ring to commemorate
an important wedding anniversary.
This distinction has profound marketing implications. Fortunately,
multivariate statistical techniques such as CHAID (Chi-Square Automatic
Interaction Detection) have the power to identify situations in
which multiple target audiences exist.
Predictive Research Applications
Problems with Static Data
A frequent problem when
overlay demographics are incorporated into predictive models is
that static data — sources purchased outright and not periodically
updated — change meaning over time. This occurs because
of the large percentage of individuals who move every year.
This, in turn, results in an ever-increasing overlay rate for older,
more stable people compared with their younger, more mobile counterparts.
A real-life example of this phenomenon is a regression model in
which two "political affiliation" overlay variables,
"conservative" and "liberal," both "popped"
positively. The reason is that the variables were several
years old and rapidly were becoming surrogates for the target audience
— stable individuals in their 40's and 50's.
Hazards of Short-Term Data Fluctuations
even non-static data can change meaning over time. An excellent
example is "length of residence," a common overlay variable.
Because of peculiarities in the update cycle of at least one major
data compiler, for three months every year essentially no one on
its file shows a "length of residence" of less than one year.
In the absence of an adjustment to reflect this phenomenon, this
would be problematic for a model developed for "new mover" merchandise
such as window treatments.
Predictive Power of Missing Data
are unaware of the often remarkable explanatory power that is inherent
in missing data. Sometimes, for a given individual, the inability
to apply specific demographic or psychographic information is more
predictive than the information itself. This has to do with
the missing data bias discussed earlier.
As an example, let's revisit the "length of residence"
variable, which is created in significant part by comparing names
at specific addresses in phone directories from one year to another.
Besides the usual problem of younger, mobile individuals having
lower hit rates, we have additional bias because of those demographic
groups that have a higher probability of opting for an unlisted
telephone number. With the unlisted-number group, it is very
likely that the information required to calculate "length
of residence" cannot be obtained. These people generally
fall into one of the following categories:
- Single women, urban residents, and the very affluent (who opt for
unlisted numbers for security reasons).
- The very poor (who cannot afford phones).
Therefore, the absence of "length of residence" information
increases the probability that a given individual belongs to one
or more of the groups listed above. This might very well be
more predictive than the knowledge that a given individual, for
example, has resided at his or her address for three years.
In order to capture the predictive power of missing demographic
and psychographic information, it is critical that missing data
for a given predictor variable be assigned its own value when building
a model. This is contrary to the practice of many statisticians,
who set missing data to the mean of all the observations for which
information exists. Others default to the equivalent Census-level
variable, which is an improvement but still not optimal.
A wonderful example of the missing data's potential predictive
power is what I refer to as "The Unmodel," which was
constructed to segment several large outside rental lists.
The top decile was driven largely by the absence of information
on multiple overlay elements. This is because the target audience
was comprised of "un-Ozzie and Harriets" — single,
downscale renters of apartment units.
Consider, for example, the univariate relationship to response of
several "Unmodel" predictor variables, where the "Missing"
categories all correlate very highly with response:
Response Rate by Income
Response Rate by Credit Card
Response Rate by Age
The resulting performance was quite good for a prospecting model,
with "lift" — top 10% to average — of 209 (and "lift"
— top 10% to bottom 10% — of 475).
The use of overlay data can have a powerful
impact on direct marketing research, if applied properly.
To ensure the effective incorporation of overlay data and the correct
interpretation of results, there are several rules to keep in mind.
First, for descriptive research, demographic and lifestyle profiles
must be adjusted to reflect the "Ozzie and Harriet"
bias that is inherent in major overlay databases. It is also
important to consider profile distributions rather than means when
drawing marketing conclusions. And finally, never assume that
multiple overlay variables that "pop" on a file all
describe the same group of individuals.
For predictive research, incorporate static data into models with
caution, recognizing that their meanings will change over time as
they become surrogates for older, more stable individuals.
Also, be mindful of the fact that even non-static data can change
meaning as suppliers update their databases. And finally,
recognize and take advantage of the fact that missing data often
can provide remarkable explanatory power.
Jim Wheaton is a Principal at Wheaton Group, and can be reached
at 919-969-8859 or firstname.lastname@example.org. The firm
specializes in direct marketing consulting and data mining, data
quality assessment and assurance, and the delivery of cost-effective
data warehouses and marts. Jim is also a Co-Founder of Data