Guest editorial: The uses and misuses of the evaluation of cities and capitals of culture

Citation

Oancă, A., Bianchini, F., Simpson, J., Tommarchi, E. and Wright, D. (2024), "Guest editorial: The uses and misuses of the evaluation of cities and capitals of culture", Arts and the Market, Vol. 14 No. 1, pp. 1-13. https://doi.org/10.1108/AAM-02-2024-064

Evaluation has become a central feature of policymaking, used and often misused to shape and influence policy goals, priorities and practices. In the case of cultural mega events such as the European Capital of Culture (ECoC) and the UK City of Culture (UKCoC), which are the focus of this special issue, evaluation is a significant mechanism to shape, justify and inform urban cultural policies. Since the publication of the Myerscough (1994) and Palmer and Palmer/Rae Associates (2004) reports, the reputation of the ECoC initiative has been consolidated as an effective catalyst and accelerator for culture-led urban regeneration. After the emergence of the ECoC, many other international and national City of Culture (CoC) initiatives were established across the globe, including the UKCoC. Many policymakers highlight the benefits outlined in evaluation studies at different stages, ranging from the participation in competitive bidding and the implementation of a programme of cultural activities to post-event and legacy actions.

However, within the Cities of Culture Research Network (CCRN), we noted a research gap in critical studies on the evaluation of CoC initiatives (Bianchini et al., 2022). As suggested in the previous volume of this special issue (2023), CCRN was established in 2019 and funded by the UK’s Arts and Humanities Research Council until 2021. Its main aim was to create an interdisciplinary space where academics, postgraduate researchers and policymakers could pursue a better collective understanding of CoCs and their evaluation. It included ECoC, UKCoC and London Borough of Culture projects delivered by British cities and connected UK researchers with their counterparts in Aarhus (Denmark) and Galway (Ireland). The network’s members and activities attempted to interrogate the often-problematic relationships between policy and evaluation and to explore the conditions and procedures that are required to create productive links between research and new policy developments, including a better acknowledgement and discussion of ambivalences and failures (Jancovich and Stevenson, 2021). The idea for this special issue arose as part of the activities of the network.

We noted that in evaluation studies and impact assessments, CoCs are often portrayed as producing positive socio-economic effects in areas including tourism, city branding and attracting inward investment. This often leads to a vicious circle in the relationship between evaluation and policymaking. Evaluation studies justify the implementation of policies which produce more evaluation studies that continue to enable policy development. It is also evident that there are many isolated studies about the impacts of CoC programmes, and few that explore medium and long-term effects (with the possible exception of Garcia and Cox, 2013). In studies about the impacts of CoCs, there is often a lack of clarity about processes of evaluation, their main practices and organising principles, the key actors involved and the effects of evaluation itself. More importantly, in a landscape increasingly informed by truth claims of “evidence-based policy” in which all policies are rhetorically imagined to be rational interventions based on evidence (Belfiore, 2009; Jancovich and Stevenson, 2021), studies that explore the often complicated, entangled, contentious and problematic relationship between evaluation and policymaking, and the ambivalent effects of evaluation, are urgently needed.

To recognise these entanglements between knowledge production and power, it is helpful to conceptualise evaluation not as a straightforward, “rational” data collection exercise but as a process of governance that is negotiated and contested and that can have complex and ambiguous effects (Espeland and Sauder, 2016; Lamont, 2012; Porter, 1995; Shore and Wright, 2011, 2015; Merry, 2015, 2016a, b). Thus, the aim of the special issue is to critically explore the various uses and misuses of evaluation and to begin to develop a theoretical basis and scope, in conversation with the sociology of evaluation, the anthropology of policy and critical policy studies. Our aim is also to extend the discussion to emerging topics and themes. To achieve this, we invited our contributors to reflect on the following key questions that this special issue aims to explore: what are the meanings which policymakers attach to ideas of cultural value, cultural policy and evidence-based policymaking and how are these understandings manifested in evaluating CoCs? How did the need for evaluation within CoC bids and projects historically emerge? How can we understand the multiplicity of interests, priorities and values at stake in CoC evaluation processes?

Overview of the special issue

This special issue on Evaluating Cities of Culture is published in two volumes and includes nine research articles, a shorter editorial and this longer editorial piece. As the idea for the issue arose within CCRN, network members contributed as (co-)authors of six articles. Through an open Call for Papers, we selected three additional contributions to enrich our debates. The contributors discuss the politics and practices of evaluation, data and evidence, and the conditions, procedures and mechanisms under which CoC evaluations have been produced.

The first part was published in 2023 as Issue 3 of Volume 13 of Arts and the Market and contained four articles by Stephen Crone and Rafaela Ganga, by Michael Howcroft, by Charlie Ingram and by Jessica Whitfield. A more complete summary of these articles is included in the editorial of that volume (Oancă et al., 2023), but we would like to briefly reiterate here the main contributions made by these authors.

Crone and Ganga (2023) offer a critical reflection of their experience of “Impacts18”, a study focussing on the long-term effects of Liverpool ECoC 2008. While exploring delicate issues in the methodological design, and in the management of data and of stakeholder relationships within the Impacts18 project, Crone and Ganga underline the risk of the emergence of “policy-led evidence”, as opposed to “evidence-led policy”, although policymakers and researchers widely state their commitment to the latter.

While focussing on the cultural politics of civic pride of Hull’s tenure as UKCoC 2017, Michael Howcroft (2023) argues that policymakers crafted and controlled a singular pride narrative as opposed to the perceived pride deficit of the local population that was meant to create the feeling of change brought about by Hull2017 but not to create actual bottom-up processes of change. This dominant local pride development strategy ended up foreclosing any critical, ambivalent perspectives regarding the supposed benefits of the UKCoC. Similar to Crone and Ganga, Howcroft argues for the importance of critical perspectives on CoC evaluation studies and their political uses.

In a study on the use of theatre arts and of headphone verbatim theatre as part of the evaluation of Coventry UKCoC 2021, Charlie Ingram (2023) makes a compelling case for the need to include creative and artistic production into the methodological toolkits of evaluation practices. Ingram argues that in contrast to mainstream, econometric modes of evaluation, inclusive artistic production can both produce and disseminate research and evaluation, including research that offers insights into the lives of people participating in cultural activities.

In a research article on the intangible benefits of the Hull UKCoC 2017 volunteer programme, Jessica Whitfield engages extensively with the experiences and perceptions of volunteers and provides a longitudinal perspective on the role and impacts of such programmes, which is much needed in evaluation studies. While volunteering programmes are becoming an increasingly common aspect of the management of City/Capital of Culture initiatives across Europe, the voices of volunteers are often ignored within longitudinal evaluation.

The second part of the special issue is published as the current volume of Arts and the Market, as Issue 1 of Volume 14 (2024), and includes five research articles.

In the first article, Daniel Ashton, Ronda Gowland-Pryde, Silke Roth and Fraser Sturt focus on data relations and frictions in the creation of a baseline against which the wider socio-economic impacts of UKCoC programmes are assessed. The authors examine published evaluations of Derry-Londonderry 2013; Hull 2017, as well as studies produced as part of Southampton’s unsuccessful bid for the 2025 title. Their analysis explores the definition of data morsels, seen as segments of quantitative and qualitative information from a range of sources that contribute to the understanding of a city, the local histories and infrastructure of data generation and sharing (e.g. longitudinal city-wide surveys), and the capacity and expertise that are deployed in data generation and evaluation processes. Ashton and colleagues propose three compelling arguments. First, they highlight that existing evaluation criteria assume that a baseline exists; however, when this is not the case, the unavailability of such data has an impact on evaluation efforts. Second, the authors reflect on the fact that creating a baseline for measuring culture is a creative and exploratory task itself, which generates tensions between data and measurement. Finally, the authors recognise that the creation of a baseline is a generative process, as it is also about creating new organisations, relationships and practices of evaluation. The work by Ashton and colleagues is pivotal to discussions on data infrastructure, governance and production as part of the evaluation of City/Capital of Culture events and stresses the importance of adopting a clear and systematic approach to data collection long before the event itself.

In the second article, Hans-Peter Degn, Steven Hadley and Louise Ejgod Hansen propose an “Evaluation Dilemmas Model” for ECoCs drawing on the experience of the organisation rethinkIMPACTS 2017 that worked on the evaluation of Aarhus ECoC 2017. Taking the evaluation of Aarhus 2017 and the perceived suitability of the model for Galway ECoC 2020 as the starting point of the analysis, Degn, Hadley and Hansen elaborate a set of “dilemmas” that capture the main challenges arising from the process of evaluation of ECoCs. These dilemmas include: breadth vs. depth; formative vs. summative evaluation modes; analysis of outputs vs. analysis of outcomes and impacts; short term vs. long term; evaluation of results vs. evaluation of processes and commissioned vs. independent modes of evaluation. The authors highlight the substantial differences between the bidding and the delivery phases of ECoC projects. They identify the “propensity to exaggerate” which characterises bid books and the fact that the strategic objectives of ECoC programmes often change in the transition from bidding to delivery, despite the contractual status of bid books for the European Commission, which awards the ECoC title. This raises the difficult question of which sets of objectives evaluation should be based on. The authors conclude that in the case of ECoCs “any framework for evaluation is only ever tentative” (p. 13) and that careful consideration by cultural managers, policymakers and evaluators of the dilemmas discussed in the article could help spend more wisely the limited resources available for evaluation research.

In the third article, Barbara Grabher provides a methodological proposal for a relational comparative approach as a response to the widespread call for more comparative approaches to the study of City/Capital of Culture events. Her proposal builds on her research on how cultures of gender equality emerge in the politics, practices and perceptions of these initiatives. Grabher’s relational comparative approach is grounded on the idea, emerging in recent literature on comparative urbanism, to explore cities through other places, rather than simply comparing them – possibly against a benchmark – to search for similarities and differences. This means putting case studies in conversation with one another, allowing them actively to shape the interpretative framework that guides comparative efforts. Starting from the observation that knowledge production in and about CoCs has been either composed of single case studies or “traditional” comparisons, Grabher argues that research on such cultural mega-events would benefit from a relational comparative reading. Rather than looking at Donostia-San Sebastián ECoC 2016; DSS 2016; Hull UKCoC 2017 as two separate events, Grabher employs the former as the relational framework for the investigation of the latter. Grabher makes use of the metaphor of “a pair of glasses” to reflect on how her ethnographic study of DSS 2016 informs her subsequent analysis of Hull2017. Insights and understandings from the Basque case study contribute to shaping her interpretative framework, thus helping highlight different approaches in the conceptualisation of culture in the two host cities/events.

In the fourth article, Szilvia Nagy reflects on the application of Participatory Action Research to shape the evaluation framework for the ECoC, although the approach she proposes can be applied to City/Capital of Culture events more broadly. She explores the use of Participatory Action Research from two perspectives. First, as a tool to encourage participatory evaluation to foster community engagement and development. Second, to connect ex ante and expost evaluation exercises, which are commonly conducted by different academic, policy or practice communities along the ECoC policy cycle. She focusses on the case of Valletta Design Cluster (VDC), a legacy project of Valletta ECoC 2018, in the light of her role in a series of community engagement workshops aimed at involving cultural practitioners and residents in the evaluation of the event, and at building a participatory planning approach to urban regeneration. Nagy’s paper provides an innovative and much needed contribution looking at Participatory Action Research as a means to link evaluation exercises, which often are top-down processes, with a participatory planning approach, and also to set up democratic, empowering, collaborative and continuous evaluation processes that last through the whole duration of the policy cycle of City/Capital of Culture events.

In the fifth article, Mark Scott, Jonothan Neelands, Haley Beer, Ila Bharatan, Tim Healey, Nick Henry, Si Chun Lam, and Richard Tomlins address UKCoC claims to create cultural “value” and how this might be measured. Their article explores significant challenges related to defining how and by what means cultural mega-events may enable “value” to be experienced across a broad range of constituent audiences and communities. A bottom up or “scaling-down” approach, they argue, calls for a paradigm shift. Such a shift is needed to develop new frameworks for and approaches to conceptualising the “benefits” of the arts, which would be more inclusive than those adopted by the investment models and mechanisms used by the UK Government’s Department for Digital, Culture, Media and Sport (DCMS). They cite Arts Council England (ACE) and its initiatives, “Let’s Create” and “Creative People and Places”, which have tended to prioritise hierarchical constructs of heritage “value” and the “public good” as delivered through ACE funding. By taking Coventry UKCoC 2021 as the key testing ground for their critique of the “value” claims of these kinds of cultural strategy missions, Scott and colleagues propose instead a re-conceptualisation of cultural mega-events in terms of their capabilities to deliver measurable democratic, co-creative benefits. They conceive these “values” and “benefits” as enacted in the design of Coventry’s UKCoC 2021 through co-creative and “change”-directed initiatives. The main characteristics which underpinned Coventry 2021’s status as a mega-event – its promotion of cultural diversity, youthfulness, intercultural dialogue and community-led participation – offered scope for greater cultural democracy, leading in specific instances to broader social impacts (through smaller community events coming together) and to more meaningful approaches to their evaluation. In short, the article’s key take-away finding is the innovative potential of a “change-led” cultural strategy such as the one adopted by Coventry UKCoC 2021, linked to more effective capture and evaluation of its “benefits” beyond the short term. In turn, this is seen as offering a paradigm shift, challenging models of the mega-event that delivers stratified concepts of “benefit” or “public good”.

Challenges in the valuation and evaluation of Capitals/cities of culture

The nine articles gathered in the two volumes help take us in productive directions and point to further areas of research, praxis and intervention. The papers highlight interesting cross cutting themes, ranging from the critique of dominant conceptualisations of evaluation to the critique of mainstream econometric and quantitative approaches, to emphasising the value and benefits of qualitative and mixed methods in CoC evaluation.

Who counts? From a linear to a processual conceptualisation of evaluation, data and evidence

One of the key insights from the two volumes of the special issue is the importance of conceptualising evaluation as an evolving practice, as a process of governance that is negotiated and contested and that can have complex and ambivalent effects. The papers in the two volumes remind us that it is important to critique the dominant perspective on evaluation as a straightforward, “rational” data collection exercise that produces “evidence” and that informs a linear or cyclical model of policymaking.

Evaluation is often imagined as the last step in policymaking processes. At best, this linear model of policymaking is turned into a cyclical model in which evaluation is conceptualised as a tool meant to improve the future efficiency of policy (Shore and Wright, 2011). As Shore and Wright highlight, in dominant accounts of policymaking, “the work of policy consists of analysing the problem and appraising the range of possible responses, selecting a response on sound, and rational grounds, implementing the chosen course of action, evaluating whether the action produced the desired outcome and, in the light of that, revising the policy to be more effective in the future” (2011, p. 5–6). Thus, practitioners imagine policy as something “out there” that needs to be managed efficiently and instrumentally and evaluation is meant to work as the natural barometer of success and clinical fine tuning. Yet these dominant models are misleading and normative. They tend to enforce a linear coherence and temporality on what are otherwise complex, often messy and ambivalent processes, relations and (dis)connections between local, national, regional and international actors and institutions.

Capitals/Cities of Culture initiatives are often described through a linear narrative: from the writing of the bid book, the bidding process itself, the preparation for the year of culture, the management of the whole year (or the management of failure after an unsuccessful bidding competition), to evaluation studies of a particular title, to policy recommendations and studies assessing the CoC programmes undertaken for the European Commission or for DCMS in the UK and Ministries of Culture in other countries (such as Italy) which have national Capital of Culture competitions. At best, evaluation studies are caught into a cyclical narrative in which they are meant to lead to improved “evidence-led policy”.

For Szilvia Nagy, Participatory Action Research-based Evaluation (PAR-E) could offer “a continuous participatory framework for the whole ECoC cycle, as well as serving as a tool for empowerment and community development” (p.1). In her case study of Valletta Design Cluster (VDC) Nagy concludes that the recommendations emerging from participatory processes were not followed. The VDC project instead focussed on commercialised design and private events for local entrepreneurs, without fully engaging with the implications of the results of the application of PAR-E to ECoC evaluations. Her work shows the limits of self-styled participatory approaches to evaluation, and the gulf between the rhetoric of participation and what actually happens in participatory processes (Cooke and Kothari, 2001; Tommarchi et al., 2018).

This also indicates, as Crone and Ganga note for the Impacts18 longitudinal study of the effects of Liverpool ECoC 2008, that there are tensions between critical and advocacy-driven evaluation research. For Crone and Ganga, there were contradictory rationales between researchers interested in exploring the longer-term impacts of a supposedly successful culture-led urban regeneration initiative such as Liverpool ECoC 2008 and the local stakeholders who were interested in reinforcing the boosterist narratives of Liverpool”s successful transformation. Crone and Ganga argue that this “pervasive rhetoric of impact (…) risks undermining the truth and knowledge-seeking functions of evaluation” (p. 132). Thus, for Crone and Ganga, the stakeholder relationships and the incentive structures which gave rise to the Impacts18 re-study initiative were incompatible and led, as suggested earlier, to a slippage from “evidence-led policy” to “policy-led evidence”.

What counts? Measuring change and success

In a context in which all policies are rhetorically imagined to be rational interventions based on evidence, there is a further interesting point to be made about the relations between the rise of the CoC initiatives and the parallel rise of “data” and “evidence” as a means of evaluating the success of these programmes.

As Ashton et al. demonstrate, data requires modes of classification and aspirations to “measure” change. This implies a need to establish a solid, underlying reality – a baseline – against which change – hopefully, improvement or at least (given the identification of CoCs with places subject to the need for investment of various kinds) the slowing down of decline – can be measured. In the context of the pragmatic evaluation requirements of funding bodies this means the identification or design of specific metrics and modes of measurement to achieve this and to “fix” a moment in data/evidence-determined time against which the intervention will be judged. Scott et al. similarly highlight the significance of existing levels of data infrastructure, not just for cultural participation but also for all the domains that cultural interventions such as CoCs are expected to have impacts on (including health and well-being, civic pride and social integration).

Ashton et al. and also Degn et al. argue that once the monies have been secured and the event has begun, the evidence for success does not become easier to gather. Eventually there is a fading of both the imperative to establish success and of the research infrastructure and accompanying personnel required to continue to track and map metrics over longitudinal timescales, that would allow success in such a complex, multi-faceted and contested field as urban regeneration to be assessed. The pragmatic policy imperative is thus reduced to something familiar to anyone who has ever controlled a grant or held a budget. Simply to spend the money in a way that is transparent and auditable. The question here is, is it easier to “audit” – i.e.: to measure how and on what things financial resources are spent – than to evaluate, i.e. to identify the impacts and effects of the spending in meeting particular strategic goals?

Degn et al. make a related point in their discussion of the distinction between formative and summative forms of evaluation. A formative mode – that responds reflexively to the unfolding complexity of an event like a CoC – might be recognised as “best practice” in a social research sense. However, the pragmatic demands of reporting and narrating strategic impacts mean that a summative approach (attempting to answer some version of a question like “was it worth the effort?”) becomes more of a default. Given the potentially febrile debates that surround public funding (for anything, but especially for the arts) it is hard to anticipate the political circumstances in which an evaluation answering that question might conclude “no, it wasn’t worth it and no, we haven’t delivered what we set out to do”. The point made by Degn et al. about learning, on the one hand and evidencing “control”, on the other, points to a key question about whether evaluations are ritualistic and performative, rather than actually committed to finding out “what really happened”.

This also highlights the role of universities in these processes. Key relationships between universities and CoC organisers are evident in a number of recent CoC projects including Liverpool ECoC 2008, Hull UKCoC 2017, Aarhus ECoC 2017, Galway ECoC 2020; Coventry UKCoC 2021. Universities are powerful local institutions and economic actors, and as such, are often involved in bidding and delivery (as in the cases of the Hull, 2017 and Coventry, 2021UKCoCs) as well as being co-opted into evaluation teams on the basis of, as Degn et al. suggest, their ability to provide rigour – meaning independence and critical thinking – in research terms. The involvement of university teams in itself can provide no guarantee that such aspirations are actually met, often due to the tensions between the position of universities as, on the one side, supposedly independent evaluators and, on the other, key stakeholders and partners in CoC projects. Universities often play their own PR games around the success of their investments and partnerships, and there are tensions between claims to knowledge by universities that tend in some cases to fetishise their critical thinking while at the same time promoting their positions in rankings determined by opaque methodologies. As such, beyond the simple dichotomy between “advocacy” vs. academic research, universities are entangled and often complicit in the production of knowledge and in policymaking.

How to count? Conceptualising the tension between quantitative and qualitative methods of evaluation

While quantification and the impetus for “objectivity” became even more central to the governance and evaluation of CoCs, it is critical to examine the frameworks and practices within which these systems of evaluation are developed. In Trust in Numbers, Porter strongly argues that the pursuit of objectivity and quantification – at the expense of political negotiations, for example – “derives its impetus from cultural contexts, (…) where elites are weak, where private negotiation is suspect, and where trust is in short supply” (1995). Metrics depoliticise and render the political technical; they (are made to) appear as neutral, instrumental, objective rather than political (Lyall and Havice, 2018), by concealing and circumscribing political debates. Debates about urban change are sublimated as debates over metrics.

The solidity of quantitative indicators masks their emergence from specific research choices and processes – as much a part of their “construction” as those that Ingram describes in relation to verbatim theatre or as those that Howcroft focusses on in relation to pride narratives. From the creation of a baseline for CoCs as considered by Ashton et al. to the fostering of emotions and civic pride as discussed by Howcroft, it becomes evident that both qualitative and quantitative indicators are “produced” and “fabricated.”

There is evidence in the papers (notably in the efforts by Scott et al.), that – at the margins but also as part of the narrative-driven or advocacy/PR elements of communicating the success of a CoC project to local constituencies – there is acknowledgement within the teams tasked with this kind of work of the value of a wider methodological toolkit than the purely econometric one used in evaluation practices. This includes the embedding of qualitative work and even action and performance-oriented modes of research. These might align with the interests and commitments of academic researchers recruited to these kinds of projects, including researchers from the arts and humanities and from cultural sector organisations.

Ingram’s paper is a good example of this. The conventional mode of evaluation conceptualises a CoC project as involving culture being done to a place, accompanied by “before and after” measurements through which the relative success of cultural interventions can be estimated. Ingram’s approach instead recognises that creative, artistic production itself both involves research and is a mode of presenting research – including research which has claims to offer insights into the reality of the lives of participants. The production of inclusive artistic responses in this case is part of the cultural intervention and part of the evaluation.

Degn et al. reflect on this in their distinction between formative and summative modes of evaluation – a term which Scott et al. also use in their identification of a dynamic process of evaluation of activities within a CoC event feeding into the future programming of other activities. In this move, there is a rhetorical acknowledgement of the long-standing impasses between “the quantitative” and “the qualitative”, and between the “positivistic” and the “interpretivist”. There is also recognition of the fact that these epistemological framings reflect and contain differing commitments to instrumental forms of reason and to the significance of lived experience. The increasingly well-established complication of “cultural value” which has attempted to bring arts and humanities modes of understanding and scholars into the process of evaluation (or at least used funding pots once ear-marked for arts and humanities-oriented research for econometric style research on culture) as a way of justifying cultural funding can also play a role here. But in the bidding for cultural mega-events in the UK – where funding comes directly from the central government – there is the weight of legal obligation behind a single metric: “Return on Investment”, as articulated by the Treasury Green Book (HM Treasury, 2023; Walters et al., 2019).

In examining the effectiveness of evaluation approaches and tools for assessing what constitutes cultural “value”, or indeed value for investment in delivery of CoCs, the articles concerned with specific UKCoC case studies, such as Hull (2017) and Coventry (2020–21), raise broader concerns related to the ambitions of future cultural mega-events. The papers by Whitfield, Ingram and Scott et al. also investigate the evolution of broader cultural policy ambitions and evolving methods that have shifted the evaluation of CoCs from a focus principally on measurable economic gains to social impacts and qualitative cultural benefits. Indeed, an overarching theme is how such future events and their underpinning policy objectives, may be more comprehensively and effectively evaluated in terms of delivering benefits that would prioritise cultural investment as a key driver of social equity.

The papers by Whitfield (2023) and Ingram (2023) focus on key examples of how and where such “hidden” cultural benefits of CoCs can be optimised in terms of their potential to deliver qualitative impacts relating to cultural and social value - that move beyond narrower measures of economic gain/legacy. Putting the spotlight on Hull 2017, Whitfield sheds light on what she calls the “intangible legacies” of the Hull UKCoC volunteer programme. Examined in the contexts of the broader investment imperatives of an evolving UKCoC strategy and its requirements to deliver demonstrable “value” for public cultural expenditure, Whitfield’s paper addresses the so-called soft “benefits” of participatory programmes. The paper’s focus on evaluating legacy through a longitudinal approach considers important intangible benefits derived from the Hull 2017 volunteering programme, notably experiences of personal well-being, positive perceptions of the city countering those of “crappiness” or of isolation, and the emergence of an enhanced idea of “community”. Based on findings captured from two volunteer focus groups in 2019 and interviews conducted in 2021, Whitfield’s approach allows for significant points of data comparison. These, and the extended lens on tracking volunteer perceptions, offer developed insights into how and in what ways such responses may contribute new contexts for building conceptions of cultural legacy (and policy tools) that could help enlarge the terrain of cultural “value” and “worth” in terms of their potential to create “real-world”, ongoing benefits. However, as the paper argues, such benefits, ergo a marked increase in awareness of Hull’s diverse historic and maritime heritage, civic pride and “confidence in the city”, do not necessarily produce an amplified story of “impact”. Indeed, Whitfield’s key findings, set against the devastating effects of the COVID-19 pandemic for the UK’s cultural sector, point to the need for a more nuanced assessment of the value of cultural attitude shifts and perceptions (including negative ones) as potentially pivotal for improving the local sense of community and “worth”. In turn, these “intangible benefits”, as Whitfield shows, offer scope for sustaining, and building cultural legacy benefits, and for a more holistic approach to “cultural impact” evaluation, in ways that would contribute to future CoC agendas.

The importance of more holistic indicators for assessing cultural worth is also a key concern for Ingram. His paper examines the uses and potential benefits of theatre performance practices in the evaluation of the UKCoC. Taking Coventry UKCoC 2021 as his case-study, Ingram’s analysis of theatre “verbatim” offers new perspectives on modes of performance conceived experientially, trialled in the context of Coventry 2021’s shift to modes of delivery with an emphasis on co-creative and community-led events (see also Scott et al.). As the paper argues, putting into practice a verbatim performance of participants’ experiences unmediated by rehearsal (reflecting changing perceptions of the city and of cultural identity) was a means to capture a more direct and diverse sense of community. In turn, such performances using “raw” data by consent, highlighted a deeper connection with embodied stories of place, community “voice” and citizen perceptions of civic pride – perspectives which, as Ingram contends, are rarely captured in UKCoC evaluation exercises or factored into models to assess “social impacts”. Whilst drawing attention to the potential challenges of “researcher mediation” in the uses of “verbatim” performance, the paper’s findings point to the demonstrable benefits of co-creative practices with capacity to unlock a greater democracy of citizen voices and experiences of place-making, seen as pivotal to the Coventry UKCoC 2021 model.

Ingram’s conclusions resonate with Whitfield’s spotlight on the hidden intangible benefits that should be considered as a key legacy of Hull UKCoC 2017. That is, both papers propose important case studies and methods for a broader, more diversified approach to evaluating cultural mega-events in terms of their “benefits” and “worth”. Further, both papers suggest scope for paradigm shifts in which universities and policymakers share models and methods with cultural practitioners and citizens, in evolving approaches to future UKCoC mega-events and their legacies.

Scott et al. clearly go to some lengths to develop new modes of measurement and indicators of success and to explore ideas of cultural participation informed by a democratic critique of traditional concepts that elide it with a publicly funded culture that still arguably reflects a residue of traditional cultural hierarchies of “high” and “low”. It is important to consider the extent to which these residual cultural hierarchies reflect actual aesthetic commitments on the part of people working in cultural institutions today, as opposed to a kind of institutional inertia tied to “where the money comes from”. In any case, the search for “the right method” might be futile. Can the broader ontological complexities (of culture, of participation and of the highly contested notion of “impact”, for example) really be addressed by any method?

At the same time, the kinds of metrics used to establish economic impacts such as GVA (Gross Value Added), job creation including employment within the cultural and creative sectors, number of businesses started and amount of inward investment into a city are themselves constructs of research processes and modes of classification that are contested and contingent. This also belies the extent to which causality can be established. Even when causality can be claimed, it is not clear whether increases or decreases in any of such metrics are unproblematically “good things” over the long term. To take two examples of concerns shared by cultural policy researchers: to what extent do the volumes of jobs created within a city and/or sector indicate the quality of those jobs or how such new employment opportunities might be an improvement on the kinds of jobs that might otherwise be created? How sustainable will such jobs be in the longer term? Is the measurement of cultural value, in other words, part of a process of identifying how value can be extracted, or is it mainly concerned with creating the conditions in which it can be produced and shared over the long term? These questions problematise the inherently “good things” that are measured by metrics and evaluation practices.

Concluding observations

Papers across the two volumes of the special issue have examined a range of broad social outcomes, mirroring the efforts of recent CoC evaluation schemes.

One of the key insights from the contributions is dissatisfaction with the linear/cyclical understanding of evaluation, in tandem with the overemphasis on the economic outputs of CoC events at the expense of other types of impacts.

In a context in which all CoCs are rhetorically imagined as renaissance stories, the special issue touched on the complicated relationships between failure and success, and on the discrepancies between the apparently positive framing of evaluation and the more ambivalent realities on the ground. The rise of CoC initiatives and the parallel rise of “data” and “evidence” as a means of evaluating the success of these programmes implies the lurking possibility of failure. Indeed, “failure seems to be the hardest word to say”, as Jancovich and Stevenson (2021, p. 967) note in the FailSpace research project on Cultural Participation: Stories of Success, Histories of Failure (FailSpace, 2018). Can some cultural projects be seen as “failure” and what are the implications of this? What are the factors producing “failure” in processes of evaluation?

Failed bids for City/Capital of Culture events may well be perceived as traumatic occurrences and seen as the failure of years of planning and initial investment in cultural programmes. Failing may lead to a loss of momentum or interest in cultural policy (Wilson and O’Brien, 2012), and to political attempts to downplay the failure and move on. However, unsuccessful applicants have in some cases benefitted from their investment in arts and culture even without hosting the mega event itself. For example, Newcastle-Gateshead, the “big favourite” for the ECoC 2008 title (Hetherington, 2003) which instead went to Liverpool, gained media attention in the following years as the cultural capital of the North of England. Investment might not be returned due to the costs of bidding in the event of failure and ever increasing “bidding wars” (Oancă, 2018). However, bidding can also be galvanising for local and regional cultural organisations and other stakeholders despite failure. A more critical and productive process of evaluation would also openly discuss ambivalence and failure, as there is value in failing forward.

It is also likely that collaboration between different disciplines will become more important in the evaluation of future CoCs (Bianchini et al., 2023). For example, interdisciplinary collaborations will be required to assess the impacts of CoCs in terms of environmental sustainability and digitalisation. Following the pandemic, CoCs have developed growing and more sophisticated digital cultural programmes, which support and extend the offer of in presence activities. Future evaluations of CoCs will have to develop more specific methodologies and indicators to assess the strengths and weaknesses of online cultural programmes (Bianchini and Simjanovska, 2022).

The aftermath of the COVID-19 pandemic has seen the emergence of a more “curative” approach to culture-led urban regeneration, with the growing prominence of “arts and health” and “cultural welfare” objectives in urban cultural policies and in CoC programmes. In particular, there is growing recognition of the positive contributions of cultural activities in attempting to tackle the mental health crisis. It is therefore likely that within future CoC evaluation programmes researchers in the arts and humanities and in the social sciences (including economists) will collaborate more closely with public health specialists and epidemiologists.