A Method for Estimating Design-based Sampling Variances for Surveys with Weighting, Poststratification, and Raking
Hao Lu and Andrew Gelman
It is common practice to use weighting, poststratification, and raking to correct for sampling and nonsampling biases and to improve efficiency of estimation in sample surveys. In general, the sampling variances of the resulting estimates depend on the weighting procedures, not just on the numerical values of the weights.
In this article we develop a method for estimating the sampling variance of survey estimates with these adjustments, using three ideas: (1) a general notation that unifies the different forms of weighting adjustment, (2) a variance decomposition to estimate sampling variances conditional and unconditional on sample sizes within poststratification categories, and (3) the delta method applied to uncertainties in sample sizes within poststrata. The resulting variance estimates are design-based and comparable to those obtained by the jackknife. We focus on estimation of population and subgroup means but also discuss more complicated summaries such as ratios and regression coefficients.
We apply our approach to the problem that motivated this research, the New York City Social Indicators Survey, a telephone survey that uses inverse-probability weighting, poststratification, and raking to correct for sampling design and nonresponse. Our variance estimates systematically differ from those obtained using methods that do not account for the design of the weighting scheme. Assuming simple random sampling leads to underestimating the sampling variance, and treating all weights as inverse-probability causes variances to be overestimated.
Inverse-probability weighting; iterative proportional fitting; poststratification; raking; ratio estimate; regression estimate; sample survey; variance estimation.