Kentaro Yamamoto and Mary Louise Lennon
Fabricated data jeopardize the reliability of large-scale population surveys and reduce the comparability of such efforts by destroying the linkage between data and measurement…
Abstract
Purpose
Fabricated data jeopardize the reliability of large-scale population surveys and reduce the comparability of such efforts by destroying the linkage between data and measurement constructs. Such data result in the loss of comparability across participating countries and, in the case of cyclical surveys, between past and present surveys. This paper aims to describe how data fabrication can be understood in the context of the complex processes involved in the collection, handling, submission and analysis of large-scale assessment data. The actors involved in those processes, and their possible motivations for data fabrication, are also elaborated.
Design/methodology/approach
Computer-based assessments produce new types of information that enable us to detect the possibility of data fabrication, and therefore the need for further investigation and analysis. The paper presents three examples that illustrate how data fabrication was identified and documented in the Programme for the International Assessment of Adult Competencies (PIAAC) and the Programme for International Student Assessment (PISA) and discusses the resulting remediation efforts.
Findings
For two countries that participated in the first round of PIAAC, the data showed a subset of interviewers who handled many more cases than others. In Case 1, the average proficiency for respondents in those interviewers’ caseloads was much higher than expected and included many duplicate response patterns. In Case 2, anomalous response patterns were identified. Case 3 presents findings based on data analyses for one PISA country, where results for human-coded responses were shown to be highly inflated compared to past results.
Originality/value
This paper shows how new sources of data, such as timing information collected in computer-based assessments, can be combined with other traditional sources to detect fabrication.
Details
Keywords
Joyce Payne and Aurelia Stephen
If you are 30 or older, you are middle‐aged by someone's criteria. When the college students of the 1970s declared “Don't trust anyone over 30,” did you think they would be…
Abstract
If you are 30 or older, you are middle‐aged by someone's criteria. When the college students of the 1970s declared “Don't trust anyone over 30,” did you think they would be someday talking about you? And what about those who say “Life begins at 40”? Did you ever believe them?
Dan Fleming and Shaun Nicholson
This chapter “unpacks” a poster from the CEAD (Contemporary Ethnography Across the Disciplines) conference 2010 and re-situates it within an autoethnographic narrative. The poster…
Abstract
This chapter “unpacks” a poster from the CEAD (Contemporary Ethnography Across the Disciplines) conference 2010 and re-situates it within an autoethnographic narrative. The poster presented a project that combined Evocative and Analytic modes in a visual ethnography focused on a collection of tourist photographs taken on Rarotonga in the South Pacific. The framing autoethnography finds in this project evidence of a distinctive tension in contemporary informationalized life between embodied life and data coordinates, plots, or maps of the spaces and times where life takes place. The chapter aligns two sets of terms: on one hand embodied life and the Evocative mode, on the other hand data coordinates/plots and the Analytic mode. With its focus on the photographic image, the chapter also suggests two further terms for both Evocative and Analytic investigation: the image as fantasy and the visual moment. The chapter takes the form of a layered performance text in order to explore these matters.