Reporting Sources of Error in U.S. Federal Government Surveys
Daniel Kasprzyk and Lee Giesbrecht
In 1996 the United States Federal Committee on Statistical Methodology (FCSM) established a subcommittee to review the measurement and reporting of data quality in federal data collection programs. The issues contained within this broad mandate are complex. Data user goals, and consequently their concept of ``quality,' are varied. Similarly, reporting about quality can be implemented in various ways depending on the data product.
The FCSM subcommittee, whose membership represents the experiences of twelve U.S. statistical agencies, approached this topic by focusing on the accuracy dimension of quality and asking: 1) What measurement methods do federal data collection programs use to assess sources of error?; 2) To what extent do federal data collection programs report information on sources of error to the user community?; 3) How does reporting about error sources vary across different types of publications and dissemination media?; and 4) What information on sources of error should federal data collection programs provide and how should they provide it? The subcommittee completed three studies that focused on reporting sources of error in each of three types of data dissemination products; short-format reports, analytic reports, and the Internet. The studies provided information for the subcommittee about U.S. government statistical agencies' practices with regard to reporting the accuracy of their data (McMillen and Brady 1999; Atkinson, Schwanz, and Sieber 1999; Giesbrecht, Miller, Moriarity, and Ware-Martin 1999).
The subcommittee completed a report that discusses quality in terms of the measurement and reporting of various error sources that affect data quality: sampling error, nonresponse error, coverage error, measurement error, and processing error. The report discusses the measurement of each source of error - the measurement techniques and methods used; then it presents current practices for reporting information about the error source; and, finally, it presents recommendations for measuring and reporting survey error. This article summarizes the results of the studies of current reporting practices and provides recommendations to improve reporting of information on the various sources of error.
Survey error sources; nonsampling; measurement; coverage; processing; nonresponse; sampling.