Lars Lyberg, Kristen Cibelli Hibben and Beth-Ellen Pennell
Surveys in multinational, multiregional and multicultural contexts (or “3MC” surveys) are becoming increasingly important to global and regional decision-making and theory…
Abstract
Purpose
Surveys in multinational, multiregional and multicultural contexts (or “3MC” surveys) are becoming increasingly important to global and regional decision-making and theory building. To serve this purpose, the surveys need to be well managed, with an awareness of key sources of survey error and how to minimize them, mechanisms in place to control the implementation process and an ability to intervene in that process when necessary in a spirit of continuous improvement (Pennell et al., 2017). One key approach for managing and assessing the quality of 3MC surveys is the total survey error (TSE) framework and associated survey process quality. This paper aims to examine the application of the TSE framework and survey process quality to the Programme for the International Assessment of Adult Competencies (PIAAC).
Design/methodology/approach
The authors begin with a background on TSE and discuss recent adaptations of TSE and survey process quality for 3MC surveys. They then presents a TSE framework tailored with examples of potential contributions to error for PIAAC and ways to address those through effective quality assurance (QA) and quality control (QC) approaches.
Findings
Overall, the authors find that the design and implementation of the first cycle of PIAAC largely reflect the current best practice for 3MC surveys. However, the authors identify several potential contributions to error that may threaten comparability in PIAAC and ways these could be addressed in the upcoming cycle.
Originality/value
With a view toward continuous improvement, the final section draws on the survey process quality approach adapted from Hansen et al.’s study (2016) to summarize the recommendations in terms of additional QA elements (inputs and activities) and associated QC elements (measures and reports) for PIAAC’s consideration in the next cycle.
Details
Keywords
Kristen Cibelli Hibben, Beth-Ellen Pennell and Lesli Scott
At the invitation of the Programme for the International Assessment of Adult Competencies (PIAAC), this paper aims to examine advances in survey interviewer monitoring and make…
Abstract
Purpose
At the invitation of the Programme for the International Assessment of Adult Competencies (PIAAC), this paper aims to examine advances in survey interviewer monitoring and make recommendations on minimizing the effect of interviewers on survey results.
Design/methodology/approach
The authors first provide an overview of the most recent literature on interviewer effects, quality assurance and quality control. Here, they draw upon recent publications such as the cross-cultural survey guidelines (www.ccsg.isr.umich.edu) and newly published or in-press material specifically addressing these issues in multicultural, multinational and multiregional (3MC) contexts.
Findings
The authors discuss trends and innovations in quality assurance and quality control in 3MC studies and draw upon examples from international surveys that are using cutting-edge and innovative approaches to monitor interviewer behavior and minimize interviewer effects.
Originality/value
With a view to continuous quality improvement, the authors conclude with concrete recommendations for PIAAC to consider for the next cycle. Many of the recommendations have general relevance for other large-scale cross-national surveys.