A Cautionary Tale: Using Propensity Scores To Estimate the Effect of Food Stamps on Food Insecurity

Year: 2004

Research Center: Institute for Research on Poverty, University of Wisconsin-Madison

Investigator: Gibson-Davis, Christina M., and E. Michael Foster

Institution: Duke University

Project Contact:
Christina M. Gibson-Davis
Public Policy Studies
226 Sanford Institute Building
Duke University, NC 27708
Phone: 919-613-7364


In 2003, the Food Stamp Program (FSP) provided assistance to 9.2 million households, including 5 million households with children. It is the largest Federal food program and is the cornerstone of Federal food assistance. FSP attempts to ensure that low-income families have sufficient resources to purchase a nutritionally adequate diet. Food insecurity is an FSP outcome measure. A module designed by the U.S. Department of Agriculture (USDA) and the Department of Health and Human Services (HHS) consists of 18 items that classify families as either food secure or food insecure. Reducing levels of food insecurity is an important goal, particularly for children: those who are food insecure are more likely to suffer from a range of academic and behavioral deficits.

The impact of the FSP on food insecurity is difficult to analyze since unmeasured or unobserved characteristics may be correlated with both program participation and food insecurity. This correlation introduces statistical bias, which may either understate or overstate program impact. Most research indicates that those who use food stamps have measurable disadvantages relative to income-eligible persons who do not participate. These disadvantages may increase the likelihood that these families are also food insecure. Simple comparisons between those who use food stamps and those eligible persons who do not may understate the program's impact if there are unmeasured disadvantages that prompt the most food insecure households to become FSP participants. However, the direction of the statistical bias may operate in the opposite direction. Eligible families who apply and participate may be better organized or otherwise advantaged in comparison to eligible nonparticipants. In that case, the program impact may be overstated.

Recent developments in nonexperimental methodology provide new techniques for evaluating a nonrandomized program such as the FSP. The use of propensity scores is one such method. Under key assumptions, propensity scores approximate a randomized experiment by creating a “matched” treatment and control group who are, save for treatment status, comparable. When the two matched groups are compared on an outcome, any resulting differences should reflect the treatment and not unmeasured characteristics. This method depends heavily on the ability to control for observed determinants of both program participation and food insecurity.

This research uses propensity scores to examine the effect of the FSP on food insecurity. Data come from the first and second waves of the Early Childhood Longitudinal Survey-Kindergarten Cohort (ECLS-K), a nationally representative dataset of over 21,000 children. Propensity scores were developed to create equivalent groups, with one receiving the treatment while the other group does not. Propensity scores represent the predicted probability of participating in the treatment, based on the observed and measured characteristics used in the prediction equation. The literature does not provide definitive guidance on how propensity scores should be calculated, so this research used several models, with each model varying the number of covariates.

The study found no effect of the FSP on the likelihood that a household will be classified as food insecure: the estimates were small and not consistent across the model specifications. As a further step, however, the study estimated the effect of food stamps on the level of food insecurity. Among households that indicated some amount of food insecurity, FSP participation reduced the amount of food insecurity.

This research makes two contributions:

  1. It uses statistically rigorous methods to evaluate the potential impact of food stamps among a sample of households with young children.
  2. The advantages and disadvantages of propensity scores are compared to more traditional linear regression models.
The use of the method is illustrated, highlighting how it may be applied in other research efforts. The study demonstrates limitations to the use of propensity scores based on their underlying assumptions. In order to help attain unbiased estimates, scores should be based on a rich array of covariates. This research found that estimates using regular linear regression methods were similar to results of the propensity score models. It is possible that a rich dataset such as ECLS-K, where many potentially confounding factors can be controlled for, could be sufficient for estimating the program's effect. Propensity scores should be used with caution. To examine the impact of a program like the FSP, where a randomized experiment cannot take place because eligible recipients cannot be denied benefits, using propensity scores in conjunction with more traditional linear regression models may provide informative results on program impact.