Prelims
ISBN: 978-1-80043-003-7, eISBN: 978-1-80043-002-0
Publication date: 21 October 2021
Citation
Ventista, O.M. (2021), "Prelims", How to Evaluate the Effectiveness of a School-Based Intervention: Evaluating the Impact of the Philosophy for Children Programme on Students' Skills, Emerald Publishing Limited, Leeds, pp. i-xviii. https://doi.org/10.1108/978-1-80043-002-020211013
Publisher
:Emerald Publishing Limited
Copyright © 2021 Ourania Maria Ventista
Half Title Page
How to Evaluate the Effectiveness of a School-Based Intervention
Title Page
How to Evaluate the Effectiveness of a School-Based Intervention: Evaluating the Impact of the Philosophy for Children Programme on Students’ Skills
BY
OURANIA MARIA VENTISTA
United Kingdom – North America – Japan – India – Malaysia – China
Copyright Page
Emerald Publishing Limited
Howard House, Wagon Lane, Bingley BD16 1WA, UK
First edition 2021
Copyright © 2021 Ourania Maria Ventista. Published under exclusive license by Emerald Publishing Limited.
Reprints and permissions service
Contact: permissions@emeraldinsight.com
No part of this book may be reproduced, stored in a retrieval system, transmitted in any form or by any means electronic, mechanical, photocopying, recording or otherwise without either the prior written permission of the publisher or a licence permitting restricted copying issued in the UK by The Copyright Licensing Agency and in the USA by The Copyright Clearance Center. Any opinions expressed in the chapters are those of the authors. Whilst Emerald makes every effort to ensure the quality and accuracy of its content, Emerald makes no representation implied or otherwise, as to the chapters’ suitability and application and disclaims any warranties, express or implied, to their use.
British Library Cataloguing in Publication Data
A catalogue record for this book is available from the British Library
ISBN: 978-1-80043-003-7 (Print)
ISBN: 978-1-80043-002-0 (Online)
ISBN: 978-1-80043-004-4 (Epub)
Dedication Page
To my parents and to Grigorios
Contents
List of Tables and Flowcharts | xiii |
Abbreviations | xv |
Foreword | xvii |
Chapter 1: Introduction | 1 |
1.1. The Two Themes | 1 |
1.2. Two Premises | 3 |
1.2.1. Evidence-based Education | 3 |
1.2.2. Skills-based Curricula | 4 |
1.3. The Outline of this Book | 6 |
Chapter 2: How to Conduct a Multidimensional Programme Evaluation | 9 |
2.1. The Purpose of the Evaluation | 9 |
2.2. Dimensions of Evaluation | 10 |
2.2.1. Pedagogical Evaluation | 11 |
2.2.2. Systematic Literature Review | 11 |
2.2.3. Identify Ways to Fill in the Literature Gaps | 11 |
2.2.4. Synthesising the Evidence | 14 |
2.2.5. Offer Recommendations | 15 |
Chapter 3: Philosophy for Children: Discussing About Pedagogy | 17 |
3.1. Introduction | 17 |
3.2. Philosophy for Children: A School-based Dialogic Intervention | 17 |
3.3. The Purpose of the Evaluation | 18 |
3.3.1. A Questioning Mind | 19 |
3.3.2. Was There a Need for Evaluating Philosophy for Children? | 20 |
3.4. Pedagogical Evaluation | 22 |
3.4.1. Identify the Stance and Justify It | 22 |
3.4.2. Investigate the Programme Rationale | 23 |
3.4.3. Evaluate the Main Elements of the Programme: Learning and the Role of the Student and the Teacher | 25 |
3.4.4. Examine Oppositional Views | 29 |
3.4.5. Search for Inconsistencies and Areas for Improvement | 32 |
3.4.6. Reach to a Conclusion | 33 |
Chapter 4: Systematic Literature Review: Evaluating the Available Evidence | 35 |
4.1. Conducting a Systematic Literature Review | 35 |
4.1.1. Specify the Research Question and the Inclusion Criteria | 35 |
4.1.2. Decide on the Way the Search Will Be Conducted and Identify the Relevant Literature | 36 |
4.1.3. Evaluate the Quality of the Evidence | 37 |
4.1.4. Calculating the Impact of the Programme | 39 |
4.2. Philosophy for Children: A Systematic Literature Review | 40 |
4.3. The Impact of Philosophy for Children on Students’ Skills | 43 |
4.4. Limitations of the Review | 46 |
4.5. A Summary of the Findings of the Systematic Literature Review | 47 |
Chapter 5: Generating New Evidence | 49 |
5.1. Conducting an Evaluation Study | 50 |
5.1.1. Research Design | 50 |
5.1.2. Pre-test and Post-test | 52 |
5.1.3. Process Evaluation | 52 |
5.1.4. Analysing the Data | 53 |
5.1.5. Recognising and Reporting the Limitations | 54 |
5.2. The Method of Conducting the Philosophy for Children Study | 54 |
5.2.1. Intervention Group | 55 |
5.2.2. Comparison Group | 55 |
5.3. Response Rate and Missing Data | 56 |
5.4. Process Evaluation and Analysis | 56 |
5.5. Ethics | 58 |
5.6. Limitations of the Research Design | 58 |
Chapter 6: Evaluating the Impact on Thinking Skills: Working Definitions | 61 |
6.1. Defining Critical Thinking | 61 |
6.1.1. Lipman’s Definition of Critical Thinking | 61 |
6.1.2. Critical Thinking as a Guide to Action | 62 |
6.1.3. Critical Thinking or Critical Thinker? | 64 |
6.1.4. Critical Thinking as Problem-solving | 66 |
6.1.5. Critical Thinking as an Active Process | 67 |
6.2. Two Important Debates | 67 |
6.2.1. Is Critical Thinking Value-neutral? | 67 |
6.2.2. Should Critical Thinking Be Considered a General or a Subject-specific Skill? | 69 |
6.3. Critical Thinking: Working Definition | 71 |
6.4. Defining Creativity | 73 |
6.4.1. Person | 74 |
6.4.2. Process | 75 |
6.4.3. Product | 76 |
6.4.4. Environment | 77 |
6.5. Debates | 77 |
6.5.1. Is Creativity Value-neutral? | 77 |
6.5.2. Is Creativity a Domain-specific Skill? | 78 |
6.6. Creativity: Working Definition | 78 |
6.7. The Relationship Between Critical and Creative Thinking | 80 |
6.8. Can Critical Thinking and Creativity Ever Be Improved? | 81 |
6.9. Transparency | 83 |
Chapter 7: Assessing Students’ Thinking Skills | 85 |
7.1. Introduction | 85 |
7.2. Critical Thinking Assessments | 85 |
7.2.1. Halpern Critical Thinking Assessment (2010) | 86 |
7.2.2. Cornell Critical Thinking Test (2005) | 86 |
7.2.3. Ennis–Weir Critical Thinking Test Essay (1985) | 87 |
7.2.4. Test of Appraising Observations (1984) | 87 |
7.2.5. Watson Glaser Critical Thinking Appraisal (2002) | 88 |
7.2.6. New Jersey Test of Reasoning Skills (1983) | 89 |
7.3. Creativity Assessments | 89 |
7.3.1. Multi-trial Creative Ideation | 89 |
7.3.2. Latent Semantic Analysis | 90 |
7.4. Measurement Tools for the Comparative Evaluation Study | 90 |
7.5. Assessing Students’ Creativity | 92 |
7.6. Marking Students’ Creativity | 93 |
7.6.1. First Activity | 94 |
7.6.2. Second Activity | 96 |
7.6.3. Calculating the Overall Creativity Score | 96 |
7.7. Designing the Critical Thinking Assessment | 98 |
7.7.1. Purpose | 99 |
7.7.2. Construct | 99 |
7.7.3. The Form of the Assessment | 99 |
7.7.4. Why Three Alternatives in the Multiple-choice Questions? | 100 |
7.7.5. Guidelines for Constructing Good Multiple-choice Items | 101 |
7.8. Sample Items of Critical Thinking Assessments | 102 |
7.8.1. Evaluation of an Argument and Credibility of Sources | 102 |
7.8.2. Deduction | 104 |
7.8.3. Assumption Identification | 105 |
7.9. Marking | 106 |
7.10. Psychometric Properties | 107 |
7.10.1. Reliability | 107 |
7.10.2. Validity | 108 |
7.11. Piloting of the Assessments Used in the Philosophy for Children Evaluation | 110 |
Chapter 8: The Findings of the Quasi-experimental Study | 111 |
8.1. The Impact of the Philosophy for Children Programme on Critical Thinking | 111 |
8.2. The Impact on Different Critical Thinking Skills | 112 |
8.3. The Impact of the Philosophy for Children Programme on Creativity | 114 |
8.4. The Impact on Different Aspects of Creativity | 115 |
8.5. A Summary of the Findings of the Quasi-experimental Study | 117 |
Chapter 9: Secondary Data Analysis: Evaluating the Long-term Impact of the Programme | 119 |
9.1. Secondary Data Analysis as an Evaluation Method | 119 |
9.2. Cases | 120 |
9.3. Missing Data | 122 |
9.4. Analysis | 123 |
9.5. Findings of the Secondary Data Analysis | 123 |
9.5.1. Results: Impact on Attainment | 124 |
9.5.2. Results: Impact on Disadvantaged Students’ Attainment | 125 |
9.6. Discussion | 125 |
Chapter 10: Combining the Evidence: Deciding About the Programme Effectiveness | 127 |
10.1. Philosophy for Children: Synthesising the Evidence | 128 |
10.2. Should the Programme Be Implemented? | 133 |
Chapter 11: Recommendations: When and How Should the Programme Be Implemented in the Classrooms? | 135 |
11.1. Recommendations for Practitioners | 135 |
11.1.1. Thinking Skills | 136 |
11.1.2. Social Skills | 136 |
11.1.3. Self-esteem and Self-confidence | 138 |
11.1.4. Well-being | 139 |
11.1.5. Recommendations on Programme Fidelity | 140 |
11.1.6. Time Allocated | 140 |
11.2. Closing the Gap Between Advantaged and Disadvantaged Students: Recommendations for Policy Makers | 141 |
11.3. Recommendations for Teacher Education | 141 |
11.4. Recommendations for Researchers | 142 |
Chapter 12: Concluding Thoughts | 143 |
12.1. Future Directions | 143 |
12.2. Teaching and Assessing Skills in Schools | 144 |
12.2.1. Critical and Creative Thinking | 145 |
12.2.2. How Can Schooling Support the Development of Students’ Thinking Skills? | 146 |
12.2.3. Assessing Creativity and Critical Thinking | 147 |
12.3. Final Thoughts | 148 |
References | 149 |
Index | 163 |
List of Tables and Flowcharts
Tables | ||
Table 1. | Rating Scale to Evaluate the Trustworthiness of a Study Which Evaluates the Impact of a School-based Programme. | 38 |
Table 2. | Quality of Research Design and Reporting of the Studies Included in the Systematic Literature Review. | 41 |
Table 3. | Quality of Studies in Relation to the Calculated Effect Sizes (P4C Impact). | 42 |
Table 4. | Matrix with Inter-item Correlations for the Pre-test. | 97 |
Table 5. | Matrix with Inter-item Correlations for the Post-test. | 97 |
Table 6. | Simple Version of the Test Blueprint for the Critical Thinking Assessments. | 102 |
Table 7. | Impact of P4C on Critical Thinking. | 112 |
Table 8. | Impact of P4C on Students’ Critical Thinking Skills. | 113 |
Table 9. | Impact of P4C on Creativity. | 114 |
Table 10. | Impact of P4C on Students’ Creative Thinking Skills. | 115 |
Table 11. | Number of Students Eligible for FSM and Non-FSM Per Group. | 122 |
Table 12. | Frequency of Pupils’ Assessment Data in Maintenance Schools. | 123 |
Table 13. | Impact of P4C on Attainment. | 124 |
Table 14. | The Synthesis of Evidence. | 129 |
Table 15. | Weighing the Evidence. | 134 |
Flowcharts | ||
Flowchart 1. | Participant Flow Chart for the Quasi-experimental Study. | 57 |
Flowchart 2. | Participant Flow Chart for the Secondary Data Analysis. | 122 |
Abbreviations
P4C | Philosophy for Children |
RCT(s) | Randomised Controlled Trial(s) |
TTCT | Torrance Tests of Creative Thinking |
HCTA | Halpern Critical Thinking Assessment |
CCTT | Cornell Critical Thinking Test |
SD | Standard Deviation |
NPD | National Pupil Database |
FSM | Free School Meals |
Foreword
Professor Chris Brown, Durham University School of Education
The world we inhabit, COVID-19 aside, has had very mixed fortunes over the last decade. On the one hand, technology has enabled us to achieve things our ancestors would never have believed possible: redefining the way people communicate, collaborate, shop, travel, read, research, watch films, gather information, book holidays, bank and so much more (Greengard, 2015). Digital photography and social media have enabled us to capture, represent and share the world in previously unimaginable ways, while tools such as 3D printers allow us to make real our designs for anything, from sculptures to bridges, at the touch of a button. In terms of the economy, artificial intelligence (AI) and mass automation have been transformative, with much routine and low skilled work now undertaken by robots or algorithms. This use of AI is set to continue, with most commentators agreeing that AI will eventually take over many of the tasks machines can perform equally or better than humans: everything from processing insurance claims to space exploration. And this is problematic: with pre-COVID estimates indicating that, over the next 20 years, some 47% of jobs in the United States and 54% of those in Europe will be lost to machines (Bregman, 2018; Frey & Osbourne, 2013).
But what separates humans from machines is our ability to engage in creative thinking – which along with its alter ego, critical problem solving – is about constructing more or less novel ideas, objects or even worlds (du Sautoy, 2019). It is ‘imaginative activity fashioned so as to produce outcomes that are both original and of value’ (National Advisory Committee on Creative and Cultural Education (NACCCE), 1999): in other words, activity designed to produce ideas that are not only innovative, but also fit for purpose (Durham Commission on Creativity and Education, 2019). What is novel, fit for purpose and indeed somehow satisfying, will depend on different domains: in what Victorian designer William Morris describes as the lesser arts (such as interior decoration), there can be a strong emphasis on novelty. In fields such as architecture, fitness for purpose is generally likely to receive more attention. In all cases, however, our acts of creation elevate, expand and transform what it means to be human.
But if it’s creativity that is keeping us one step ahead of the machines, then education systems globally now need to be focussed on actively supporting future citizens to be able to collaboratively engage in critical and creative thought. In other words, we need education systems to now arm students with the capacities required to explore, experiment, try and re-work, make and re-make, explore and value difference, overcome obstacles and develop and apply knowledge (Newton & Newton, 2018). But we still need practical suggestions for how to make this happen. Also, to have the confidence that if we are to embrace approaches to creative problem solving, that these will make a concrete and positive difference. With this extraordinary new book, Dr Rania Ventista has managed to meet these twin goals. Taking us on a powerful journey of exploration, not only does Dr Ventista showcase, with some considerable skill, how to evaluate educational interventions effectively; she also illustrates in detail why one intervention in particular – Philosophy for Children – matters.
The results are a framework that can and should be used by teachers and school leaders when understanding how to allocate scarce resources. But at the same time, how Philosophy for Children can enhance creative problem solving and why educators should be embracing this programme to arm their students with the skills they need to navigate the perils and pitfalls the twenty-first century holds. With this, her first book, Dr Ventista has announced to the world her skills as a scholar and leading thinker in this field. I can’t wait for what’s to come!
References
Bregman, R. (2018). Utopia for realists and how we can get there. London: Bloomsbury.
du Sautoy, M. (2019). The creativity code: How AI is learning to write, paint and think. London: 4th estate.
Durham Commission on Creativity and Education. (2019). Durham Commission on Creativity and Education. Retrieved from https://www.artscouncil.org.uk/sites/default/files/download-file/Durham_Commission_on_Creativity_04112019_0.pdf. Accessed on March 5, 2021.
Frey, C., & Osborne, M. (2013). The future of employment: How susceptible are jobs to computerisation. Retrieved from https://www.oxfordmartin.ox.ac.uk/downloads/academic/The_Future_of_Employment.pdf
Greengard, S. (2015). The internet of things. Cambridge, MA: MIT Press.
National Advisory Committee on Creative and Cultural Education (NACCCE) (1999). All our futures: Creativity and cultural education. London: DfEE.
Newton, L., & Newton, D. (2018). Making purposive thought productive. Ulm: The International Centre for Innovation in Education.
- Prelims
- Chapter 1: Introduction
- Chapter 2: How to Conduct a Multidimensional Programme Evaluation
- Chapter 3: Philosophy for Children: Discussing about Pedagogy
- Chapter 4: Systematic Literature Review: Evaluating the Available Evidence
- Chapter 5: Generating New Evidence
- Chapter 6: Evaluating the Impact on Thinking Skills: Working Definitions
- Chapter 7: Assessing Students’ Thinking Skills
- Chapter 8: The Findings of the Quasi-experimental Study
- Chapter 9: Secondary Data Analysis: Evaluating the Long-term Impact of the Programme
- Chapter 10: Combining the Evidence: Deciding about the Programme Effectiveness
- Chapter 11: Recommendations: When and How Should the Programme Be Implemented in the Classrooms?
- Chapter 12: Concluding Thoughts
- References
- Index