New Strategies for Pretesting Survey Questions

Lois Oksenberg, Charles Cannell, and Graham Kalton

Abstract:
An experimental study has been conducted to evaluate some new techniques for improving the effectiveness of standard pretesting in identifying problems with questions in survey questionnaires. This paper reports the results obtained from the use of behavior coding and of special probe questions. Coding interview behavior was found to uncover a sizeable number of question problems that would go undetected with the usual pretesting procedures. Special probes were found to be valuable for identifying comprehension problems, but the probes employed for other types of problems were found to be less useful. The addition of these new techniques to standard pretesting leads to a more effective and systematic evaluation of a survey questionnaire without an appreciable effect on the cost or the complexity of the pretest.

Keywords:
Behavior coding; probe questions; pilot surveys; pretests; questionnaire design.