Project:
A Cautionary Tale: Using Propensity Scores To Estimate the Effect of Food Stamps on Food Insecurity
Year: 2004
Research Center: Institute for Research on Poverty, University of Wisconsin-Madison
Investigator: Gibson-Davis, Christina M., and E. Michael Foster
Institution: Duke University
Project Contact:
Christina M. Gibson-Davis
Public Policy Studies
226 Sanford Institute Building
Duke University, NC 27708
Phone: 919-613-7364
E-mail: cgibson@duke.edu
Summary:
In 2003, the Food Stamp Program (FSP) provided assistance to 9.2 million
households, including 5 million households with children. It is the largest
Federal food program and is the cornerstone of Federal food assistance. FSP
attempts to ensure that low-income families have sufficient resources to
purchase a nutritionally adequate diet. Food insecurity is an FSP outcome
measure. A module designed by the U.S. Department of Agriculture (USDA)
and the Department of Health and Human Services (HHS) consists of 18 items
that classify families as either food secure or food insecure. Reducing levels of
food insecurity is an important goal, particularly for children: those who are
food insecure are more likely to suffer from a range of academic and behavioral
deficits.
The impact of the FSP on food insecurity is difficult to analyze since unmeasured
or unobserved characteristics may be correlated with both program
participation and food insecurity. This correlation introduces statistical bias,
which may either understate or overstate program impact. Most research indicates
that those who use food stamps have measurable disadvantages relative to
income-eligible persons who do not participate. These disadvantages may
increase the likelihood that these families are also food insecure. Simple
comparisons between those who use food stamps and those eligible persons
who do not may understate the program's impact if there are unmeasured
disadvantages that prompt the most food insecure households to become FSP
participants. However, the direction of the statistical bias may operate in the
opposite direction. Eligible families who apply and participate may be better
organized or otherwise advantaged in comparison to eligible nonparticipants.
In that case, the program impact may be overstated.
Recent developments in nonexperimental methodology provide new techniques
for evaluating a nonrandomized program such as the FSP. The use of propensity
scores is one such method. Under key assumptions, propensity scores
approximate a randomized experiment by creating a “matched” treatment and
control group who are, save for treatment status, comparable. When the two
matched groups are compared on an outcome, any resulting differences should
reflect the treatment and not unmeasured characteristics. This method depends
heavily on the ability to control for observed determinants of both program
participation and food insecurity.
This research uses propensity scores to examine the effect of the FSP on
food insecurity. Data come from the first and second waves of the Early
Childhood Longitudinal Survey-Kindergarten Cohort (ECLS-K), a nationally
representative dataset of over 21,000 children. Propensity scores were
developed to create equivalent groups, with one receiving the treatment
while the other group does not. Propensity scores represent the predicted
probability of participating in the treatment, based on the observed and
measured characteristics used in the prediction equation. The literature does
not provide definitive guidance on how propensity scores should be calculated,
so this research used several models, with each model varying the
number of covariates.
The study found no effect of the FSP on the likelihood that a household will
be classified as food insecure: the estimates were small and not consistent
across the model specifications. As a further step, however, the study estimated
the effect of food stamps on the level of food insecurity. Among
households that indicated some amount of food insecurity, FSP participation
reduced the amount of food insecurity.
This research makes two contributions:
- It uses statistically rigorous methods to evaluate the potential impact
of food stamps among a sample of households with young children.
- The advantages and disadvantages of propensity scores are
compared to more traditional linear regression models.
The use of the method is illustrated, highlighting how it may be applied in
other research efforts. The study demonstrates limitations to the use of
propensity scores based on their underlying assumptions. In order to help
attain unbiased estimates, scores should be based on a rich array of covariates.
This research found that estimates using regular linear regression
methods were similar to results of the propensity score models. It is possible
that a rich dataset such as ECLS-K, where many potentially confounding
factors can be controlled for, could be sufficient for estimating the program's
effect. Propensity scores should be used with caution. To examine the
impact of a program like the FSP, where a randomized experiment cannot
take place because eligible recipients cannot be denied benefits, using
propensity scores in conjunction with more traditional linear regression
models may provide informative results on program impact.