Quality outcomes in NHS library and knowledge services

Abstract

Purpose

This paper aims to demonstrate the approach taken in delivering the quality and impact elements of Knowledge for Healthcare, the strategic development framework for National Health Service (NHS) library and knowledge services in England. It examines the work undertaken to enhance quality and demonstrate the value and impact of health library and knowledge services. It describes the interventions developed and implemented over a five-year period 2015–2020 and the move towards an outcome rather than process approach to impact and quality.

Design/methodology/approach

The case study illustrates a range of interventions that have been developed, including the outcomes of implementation to date. The methodology behind each intervention is informed by the evidence base and includes professional engagement.

Findings

The outcomes approach to the development and implementation of quality and impact interventions and assets provides evidence to demonstrate the value of library and knowledge staff to the NHS in England to both high-level decision-makers and service users.

Originality/value

The interventions are original concepts developed within the NHS to demonstrate system-wide impacts and change. The Evaluation Framework has been developed based on the impact planning and assessment (IPA) methodology. The interventions can be applied to other healthcare systems, and the generic learning is transferable to other library and knowledge sectors, such as higher education.

Citation

Edwards, C. and Gilroy, D. (2021), "Quality outcomes in NHS library and knowledge services", Performance Measurement and Metrics, Vol. 22 No. 2, pp. 106-116. https://doi.org/10.1108/PMM-07-2020-0040

Introduction

Health Education England (HEE) is the steward of development and investment in library and knowledge services on behalf of the National Health Service (NHS). HEE's Knowledge for Healthcare (HEE, 2014) is the strategic development framework for NHS-funded library and knowledge services in England, which has set out an ambitious vision to ensure the use of the right knowledge and evidence at the right time. It calls for service transformation, redesign and collaboration (Lacey Bryant et al., 2018). One of the strategic work streams underpinning Knowledge for Healthcare is quality and impact.

The initial plans for the quality and impact work stream were “to develop tools and resources to empower library and knowledge staff to demonstrate value, and to support the collection of evidence to raise the profile and impact in delivery of decision making and healthcare”(Edwards and Ferguson, 2015). This article demonstrates the progress in delivery and development of this work focussing on four key interventions:

1. Value and Impact Toolkit

2. Metrics for Success

3. Evaluation Framework

4. Quality Assurance Framework

Grieves and Pritchard (2018) note the importance of an outcomes and impact-centred model in developing an agile evidence base. This enables the library and knowledge staff to demonstrate to stakeholders that they “fully understand the value our customers place upon services, the contribution we make to strategic objectives, our value for money and the longer-term impact”. An outcomes focus has been adopted in the implementation of these interventions.

Methodology

The HEE Library and Knowledge Services team uses driver diagrams as a strategic planning tool. A driver diagram (Figure 1) informed the development of interventions to be implemented by quality and impact group (QIG) to achieve the vision of Knowledge for Healthcare. In most instances, each intervention resulted in a project to oversee the development and implementation of a tool to improve service quality and demonstrate value and impact. In some instances, such as with the research and innovation aim, the intervention consisted of a range of projects and initiatives. Intervention and project were overseen by task and finish groups which included representatives from HEE, NHS-employed library and knowledge staff, higher education healthcare librarians and subject matter experts. An evidence-based approach was followed, including a literature review, piloting and evaluation processes.

Value and Impact Toolkit

Our next steps are to refresh the “impact” tool, promote widespread adoption and publish case studies in order to attract more decision makers to make the best use of the service (HEE, 2014).

An initial intervention was to enhance the existing NHS library and knowledge services impact toolkit. This toolkit was based on sound evidence (Weightman et al., 2009); however, it was acute hospital focussed, and the evidence showed that many library and knowledge staff were not routinely measuring the impacts made by their service. Library and knowledge staff also often confused impact and user feedback (Ayre et al., 2018). A refresh of the original tool was required, grounded in current evidence of the type of positive impacts being demonstrated by health librarians (Brettle et al., 2016) and applicable to all healthcare settings.

A Value and Impact task and finish group was set up to review and update the Impact toolkit. The methodology used included a literature search, a baseline survey on the use of the existing toolkit and development and piloting of a questionnaire (Ayre et al., 2018).

The task and finish group used the standard methods and procedures for assessing the impact of libraries BS IS0 16439:2014 to define impact for NHS libraries as a difference or change in an individual or group resulting from the contact with library services. The group adopted Saracevic and Kantor’s (1997) definition of value as the perceived value approach which relies on an individual's own perception of the value of an impact.

The refreshed Impact Toolkit (HEE, 2016b) includes a generic questionnaire, an impact case study template and a resource that brings together a range of material useful in measuring value and impact. The main change to the questionnaire is the recommendation that impact is gathered in relation to one specific incident (or use) rather than overall use of the library (Ayre et al., 2018). The questionnaire focusses on impact as both immediate and probable future outcomes. Knock et al. (2017) provided an overview of the collaborative approach used in the development of the toolkit, its contents and intention to collate the outcomes of both the questionnaire and case studies nationally to provide a clear picture of the impact of health libraries in the NHS in England.

Since the launch of the toolkit, in 2016, there has been a considerable increase in the generation and sharing of impact evidence across England to demonstrate the value of the library and knowledge services to the NHS. Examples of this include Warrington and Halton Teaching Hospitals, where an impact mural was added in a prominent location within the teaching space, and University Hospitals Coventry and Warwickshire's Clinical Evidence-Based Information Service (CEBIS) which regularly use social media to tweet about their impact work.

A survey was carried out in spring 2019 to determine how many services had implemented the Impact toolkit. With a response rate of 100%, this survey showed that 75% of library and knowledge services were using the toolkit and of these, 80% were using the generic questionnaire, demonstrating progress towards the initial target in Knowledge for Healthcare of 95% of services using the toolkit.

The development of impact case studies has become the most powerful means of demonstrating the impact and value of services across England. Library and knowledge staff are encouraged to submit case studies to a national repository managed by the HEE team, with over 350 accepted to date. Many of these narratives have been developed into impact vignettes (Plate 1).

Impact data only deliver their full potential value where they are used to evidence the critical functions that library and knowledge staff fulfil in the NHS and healthcare environment. Gilroy and Turner (2018) demonstrated how library and knowledge staff were championing their organisational impact at the local level, using impact evidence in a variety of ways including in annual reports and promotional materials to highlight their value to stakeholders.

The use of quotes from individual stakeholders and opinion leaders who have benefited from the use of services is an important part of the case study development. This recommendation is used in promotional material at the national and local level, ensuring the role of health library and knowledge services is visible to high-level decision-makers influencing thinking and policy.

Quality and Improvement Outcomes Framework (Outcomes Framework)

We will refresh the Library Quality Assurance Framework to ensure it continues to drive service improvement and is aligned with wider education and service monitoring (Knowledge for Healthcare, 2014).

An initial objective was to refresh the existing Library Quality Assurance Framework (LQAF), first introduced in 2010 (De la Mano and Harrison, 2012). However, to align with the needs of Knowledge for Healthcare, it was decided to completely review the existing quality process. Many of the reasons for review were the same as those highlighted by Reid (2019); standards of service delivery had improved (Lacey Bryant et al., 2018); context had changed and the perception that it was becoming “too easy” to attain the highest ranking of excellence.

A key emphasis of the review was the focus on outcomes rather than process. Grey et al. (2012) highlighted that any future strategic development of health library services should promote the importance of quality improvement outcomes (rather than processes) as the key to improving services.

The development has been grounded in an evidence-based approach using the quality improvement methodology of plan, do, study, act (PDSA). The piloting stage and use of a range of methods, both evaluative and knowledge gathering, have been important to ensure the development of a robust framework for implementation (Edwards and Gilroy, 2019).

The Outcomes Framework (HEE, 2019a) includes six outcomes, each with a related maturity model for service development and improvement:

1. All NHS organisations enable their workforce to freely access proactive library and knowledge services that meet organisational priorities within the framework of Knowledge for Healthcare.

2. All NHS decision-making is underpinned by high-quality evidence and knowledge mobilised by skilled library and knowledge specialists.

3. Library and knowledge specialists identify the knowledge and evidence needs of the workforce in order to deliver effective and proactive services.

4. All NHS organisations receive library and knowledge services provided by teams with the right skill mix to deliver on organisational and Knowledge for Healthcare priorities.

5. Library and knowledge specialists improve the quality of library and knowledge services using evidence from research, innovation and good practice.

6. Library and knowledge specialist demonstrate that their services make a positive impact on healthcare.

The maturity model has five levels from 0, which represents no development against the outcome, through to 4, a service that is highly developed and continually improving against the outcome.

Defining quality is difficult as there are multiple definitions which capture its myriad elements. Booth's (2003) analogy encourages consideration of as many different aspects, perspectives and types of evidence as possible to provide a realistic overview of quality. The evaluation of library and knowledge services against the levels for each of the outcomes therefore relies on a range of evidence to demonstrate progress. Based on the structure of the Public Library Improvement Model for Scotland (2017), the Outcomes Framework provides the scope, NHS strategic context, key questions to consider and examples of outcome-focussed evidence.

HEE's policy on NHS library and knowledge services in England (HEE, 2016a) emphasises the need for all NHS staff to be able to freely access library and knowledge services in order to use the right evidence and knowledge to deliver excellent healthcare and health improvement. To ensure that the NHS is engaged and delivering on this policy, the Outcomes Framework has taken an organisational approach. Outcomes 1 to 3 are focussed on the organisation to ensure that library and knowledge services are embedded and seen as business critical (Lacey Bryant et al., 2018). This ensures integration of the Outcomes Framework with the HEE Quality Framework (HEE, 2019b) covering the wider education and learning environment.

A baseline self-evaluation of the framework is being planned for submission by all NHS trusts across England in 2021. This differs from the LQAF for which only 70% of services carried out the initial baseline (De la Mano and Harrison, 2012). To ensure consistency, a single national process is being established to validate the self-evaluation submissions. The result of this will be the first truly national and comparable review of quality within NHS-funded library and knowledge services in England.

The Outcomes Framework provides a structure to ensure services evolve to meet the changing needs of organisations and individuals. The framework should lead to increasing satisfaction and improved outcomes for users of the services. Grey et al. (2012) noted that quality improvement systems produce valuable outcomes including a positive impact on strategic planning, promotion, new and improved services and staff development.

Metrics and the impact evaluation framework

Metrics will be reviewed, and additional meaningful measures introduced, as part of action planning to implement the strategic framework (Knowledge for Healthcare, 2014).

A task and finish group carried out an extensive review of what makes a good metric and how these have been applied to library and knowledge services in the NHS. The methodology included a survey with library and knowledge staff about current approaches; a review of the history of metrics in NHS-funded libraries and a scoping literature search.

The resulting Principles for Metrics Report and Recommendations (HEE, 2016c) identifies a set of principles for good metrics for health library and knowledge services, as meaningful, actionable, reproducible and comparable. It defines metrics as “criteria against which something is measured” (Showers, 2015). It also provides a template [1] to support the development and sharing of metrics that are adaptable across all service situations.

Fricker (EAHIL, 2017) emphasised how good metrics contribute to better engagement and understanding with stakeholders and highlighted the principles which will equip librarians to develop meaningful metrics in support of their service development and improvement.

This work has improved our understanding of metrics and has provided a major learning point since the production of Knowledge for Healthcare in 2014. This learning is now being used at the national level to inform the refresh of the strategy.

Working with Sharon Markless [2], an impact Evaluation Framework (HEE, 2017) was created to measure the progress and impact of delivery of the Knowledge for Healthcare vision. This used the impact planning and assessment (IPA) methodology and defined impact as “any effect of the service [or of an event or initiative] on an individual or group” (Streatfield and Markless, 2009). This aligns well with the definition in the Value and Impact Toolkit.

The Evaluation Framework is based on the premise that it is very difficult to provide clear evidence of impact in complex systems such as healthcare. Therefore, the approach taken is to identify a series of indicators which, when taken together, suggest that progress is being made (Streatfield and Markless, 2009). The overall emphasis in any evaluation framework is on achieving outcomes which show “changes in behaviour, relationships, activities or actions of people, groups and organisations with whom a programme works directly” (Earl et al., 2001).

A total of six impact objectives were identified, each offering a clear statement of what will be achieved and what will be different if delivery of the vision of Knowledge for Healthcare is successful:

1. Organisations are more effective in mobilising evidence and internally generated knowledge.

2. Patients, carers and the public are empowered to use information to make health and well-being choices.

3. Improved consistency and increased productivity and efficiency of healthcare library and knowledge services.

4. Enhanced quality of healthcare library and knowledge services.

5. Partnership working is the norm in delivering knowledge to healthcare.

6. Increased capability, confidence and capacity of library and knowledge services workforce.

It was important to ensure that appropriate evidence and data were available to demonstrate progress against the objectives. Most of the data and evidence required are generated by the activity carried out by the library and knowledge services delivering to the NHS at the local level, with additional evidence from national activity. A review resulted in the revision of existing NHS library service data sets and the development of some new processes.

A learning point has been to consider what data and evidence should be collected routinely compared to ad hoc requests for specific evidence requirements. A monitoring dashboard is being developed for review and reporting of the data sets against agreed metrics and to demonstrate trends and differences made against each of the objectives. The further goal is to have the significant longitudinal data and evidence to truly tell the story of the impact of the library and knowledge services in the NHS.

Overview of building the evidence base

As part of our commitment to quality, knowledge teams will continue to undertake and publish research in the field, thereby building the evidence base for service improvement and sharing best practice (Knowledge for Healthcare, 2014).

A range of initiatives have been taken forward by QIG towards achieving this aim with an emphasis on encouraging library and knowledge services to share good practice and innovation.

An important step had already been taken in promoting service improvement by recognising and rewarding innovation through the Sally Hernando Awards for Innovation [3] (De la Mano and Harrison, 2012). These awards have been refined with more focus on the evaluation and impact of service innovation.

A key aim of Knowledge for Healthcare is to increase the numbers of clinical librarians within the NHS. Brettle et al. (2016) urged future researchers to build a significant and comprehensive international evidence base about the effectiveness and impact of clinical librarian services. QIG supported a national project to contribute to this evidence base. A task and finish group continued this research, demonstrating the impact of clinical librarians in assisting in decision-making surrounding patient safety, quality of care and efficiency (Divall and James, 2019).

The Outcomes Framework also encourages library and knowledge staff to ensure developments are evidence based and to develop a research culture within the library service. This aligns to the work presented by Thorpe and Howlett (2019) on the development of an Australian maturity model for evidence-based practice.

Discussion

The work of the QIG has produced a range of streamlined interventions that are applied and implemented nationally in a single consistent way. This approach to development and implementation is enabling the creation of an outcomes-focussed national evidence base to demonstrate the value and impact of library and knowledge services across the NHS system. Aligned to this is the use of the evidence and the interventions to support service improvement.

The impact vignettes have been used successfully with stakeholders to raise the profile of library and knowledge staff, a primary example being the A Million Decisions campaign [4], delivered in partnership with the Chartered Institute of Library and Information Professionals (CILIP). Local services are using this effectively for both advocacy and promotion.

As Gann and Pratt (2013) conclude, there is a need for library and knowledge staff to identify ways to evaluate themselves and ensure current measures have meaning for those outside the library world and in the context of organisations' mission and objectives. Quality and impact tools facilitate this at a system level and allow for the development of policy. For example, the implementation of a library and knowledge staff ratio policy (HEE, 2020) has been underpinned by the evidence from the use of the impact tools and collection of metrics and impact. In 2014, an unrealistic target was set of an increase in clinical librarians, from 58 to 80%, at this time, we have only reached 63%. However, the library and knowledge workforce metrics, impact on clinical librarians’ research and the impact vignettes have formed the evidence, that speaks clearly to stakeholders and employers, to enable a recommendation on an improved staffing ratio to increase the number of embedded librarians and knowledge specialist. The policy states that organisations should “strive to achieve a ratio of at least 1 qualified librarian or knowledge specialist per 1,250 WTE NHS staff”.

Although the different interventions can be used separately, the collective outputs provide a powerful narrative of the value of library and knowledge staff to the NHS from an impact and patient outcomes perspective.

Conclusion

The QIG driver diagram illustrates the primary aim of enhancing the quality and demonstrating the value of healthcare library and knowledge staff. The interventions described in this paper are enabling this change and have been used to demonstrate the system-wide value and impact of library and knowledge staff to the NHS in England.

In 2020, Knowledge for Healthcare will be reviewed and refreshed. Originally drafted to cover a 15-year time span and published as a five-year strategy, this is a useful juncture to take stock. The next steps for QIG will be for further evaluation of all the interventions. Effective knowledge services are business critical for the NHS. It follows that there are two crucial next steps: embedding the use of these evidence-based interventions in a consistent way in local NHS-funded library and knowledge services; generating the evidence base that allows NHS executives, clinicians and managers, as well as librarians and knowledge specialists, to tell the impact story to a range of different audiences and for different purposes, be that advocacy, promotion or sharing good practice and innovation.

Figures

Figure 1

Driver diagram

Plate 1

Examples of impact case study vignettes

Notes

1. https://kfh.libraryservices.nhs.uk/value-and-impact-toolkit/tools/metrics/

2. Sharon Markless, https://kclpure.kcl.ac.uk/portal/sharon.markless.html

3. The Sally Hernando Awards are for innovation in NHS library services are named in memory of Sally Hernando (1957–2010), formerly head of knowledge management and e-learning at NHS south-west. Sally led on many innovative national developments and was a great supporter of developing library services to their fullest potential.

4. A Million Decisions is a joint campaign led by HEE's Library and Knowledge Services team and the CILIP Health Libraries Group and with CILIP https://www.cilip.org.uk/general/custom.asp?page=AMillionDecisions