Proposing a performance framework for innovation measurement: an exploratory case-based research

Abstract

Purpose

Measurement of the innovation process performance is critical for both managers and researchers. However, existing performance frameworks (PFs) neglect performance indicators (PIs) and dimensions relevant to the current innovation landscape in companies as well as lack support in the definition of action plans. Thus, this paper aims to introduce a new and updated PF for measuring innovation performance and defining improvement actions.

Design/methodology/approach

The proposed PF is developed from literature and action-oriented case studies in two European manufacturing companies. First, the literature review enabled the synthesis of framework elements into a “conceptual” PF capable of illustrating the current state of knowledge in the field. Then, this PF was applied in the case studies that enriched the conceptual form with empirical insights, resulting in a new and updated PF.

Findings

The review enabled the systematisation of nine dimensions and 259 PIs that were fragmented throughout the literature. In turn, empirical insights from the case studies gave rise to an actionable procedure for providing a comprehensive diagnosis of the company's situation considering the new trends as well as defining improvement actions. Although the results from the two cases cannot be generalised, the findings encourage broader applicability.

Originality/value

The novelty of this research resides on the fact that the PF consolidates elements from the literature but combined with empirical insights in a new actionable way that supports managers in performance measurement and provides researchers with an extensive systematisation of dimensions and PIs.

Citation

Nappi, V. and Kelly, K. (2022), "Proposing a performance framework for innovation measurement: an exploratory case-based research", International Journal of Productivity and Performance Management, Vol. 71 No. 5, pp. 1829-1853. https://doi.org/10.1108/IJPPM-06-2020-0332

1. Introduction

For most companies, innovation is a top priority. Many managers look at successful innovative companies in the market and wonder what they are doing right. To boost innovation, managers often use performance indicators (PIs) to benchmark best practices (Becheikh et al., 2006; Brattström et al., 2018; Richtnér et al., 2017). Several of these PIs measure innovation in terms of results, such as the number of new products launched and their sales (Adams et al., 2006). Others measure innovation as the number of patents developed. Some even measure input metrics such as the number of ideas generated, whereas others focus on the balance of the innovation portfolio, by examining the percentage of investments in radical projects versus product line extensions (Crossan and Apaydin, 2010).

A performance framework (PF) is a managerial tool that helps companies in the application of PIs to benchmark best practices to evaluate antecedents, activities and outcomes, thus ensuring that innovation is sufficiently supported and efficiently performed (Adams et al., 2006; Crossan and Apaydin, 2010; Dziallas and Blind, 2018). Prior research presents several innovation PFs that inform valuable learnings for managers and researchers in the field, such as the importance of including relevant performance dimensions to address the innovation process (Crossan and Apaydin, 2010), the selection of balanced PIs to cover those relevant dimensions (Chiesa et al., 1996; Werner and Souder, 1997) and support the identification of opportunities to improve and define action plans accordingly (Chiesa et al., 2009).

Nevertheless, existing PFs fail to address recent changes in how innovation is being performed by companies (Dziallas and Blind, 2018; Frishammar et al., 2019). These changes in the current innovation landscape are driven by recent performance dimensions that need to be included in the PFs, e.g. knowledge management and innovation environment considering new trends like openness, servitisation and sustainability (Henttonen et al., 2016; Lee and Markham, 2016). Such dimensions also need to be populated with relevant PIs systematised in a way that enables companies to generate a balanced selection (Dziallas and Blind, 2018). Thus, existing PFs make managers miss these changes or lack sound recommendations for how to address them in the context of performance measurement, as well as making information available for the development of action plans (Kahn et al., 2006; Lakiza et al., 2018; Sari et al., 2020).

Therefore, the goal of this paper is to propose a new and updated PF that addresses the opportunities emerging from the need to embrace recent performance dimensions and PIs, as well as critical demands from practice when managers need to make informed decisions and define improvement actions. To achieve this, we have developed an actionable step-by-step PF that emerged from a literature review followed by two action-oriented case studies in technology-intensive companies (an electric power transformers manufacturer and navigating and control equipment for transport manufacturer).

This exploratory research allowed us to substantiate the conceptual PF from the literature with practical issues from the case studies, which lead to a new and updated approach for the PF. This PF was validated with 25 senior and middle management employees from the companies, and because of its success, it is currently in use. As the core contribution for practice, this study provides an actionable procedure for managers reaching a comprehensive diagnosis of the company's current situation considering the new trends and defining improvement actions accordingly. For research, the PF provides a systematisation of nine dimensions and 259 PIs that were fragmented throughout the literature. Hence, this paper can be beneficial for researchers who may need a swift identification of relevant PIs and dimensions.

The remainder of this paper is structured as follows. Section 2 reviews the literature, whereas Section 3 describes the research method employed. Following this, Section 4 presents the results: the conceptual PF and the insights from the case studies. Section 5, in turn, illustrates the resulting PF proposal. Section 6 discusses the PF applicability. Finally, Section 7 concludes the study with theoretical and managerial implications followed by limitations.

2. Research background

In the past, the innovation process was treated as a linear process (Adams et al., 2006); however, this is no longer sufficient (Crossan and Apaydin, 2010). Nowadays, innovation is understood as iterative cycles of concurrent and/or sequential activities aiming to develop product–service systems, intertwined with decision moments, the stage gates (Lee and Markham, 2016).

Within the innovation process, a PF enables the definition of what is essential to the company in terms of relevant dimensions and PIs and how this information should be reviewed to identify weaknesses and, consequently, define improvement actions (Adams et al., 2006). To be able to perform all these functions, the PF relies on its critical elements. First, “database” elements refer to both PIs and dimensions (Franco-Santos et al., 2007). Indicators and dimensions are widely accepted as required elements; nonetheless, these are sometimes surprisingly neglected in the development of a PF (Dziallas and Blind, 2018). Further supporting elements can vary from simplistic manual processes for the acquisition, analysis and dissemination of performance information to sophisticated information software (Folan and Browne, 2005). These elements, regardless of their degree of sophistication, must include a procedure to support the measurement process (Folan and Browne, 2005; Medori and Steeple, 2000) and means to evaluate performance progress (Kahn et al., 2006) as well as define improvement actions (Chiesa et al., 1996; Tangen, 2004) to enable continuous improvement. In sum, the critical elements that constitute a PF are: database elements (dimensions and PIs), the process (i.e. procedure to apply the PF) and continuous improvement (means to evaluate the progress of performance and define improvement actions).

Existing PFs for measuring a company's innovation processes are presented in Table 1. These PFs can be found in either technology and innovation management (e.g. PFs by Crossan and Apaydin, 2010, and Kahn et al., 2006) or managerial accounting literature (e.g. PF audits by Chiesa et al., 1996, and Frishammar et al., 2019). Because of the multidisciplinary of innovation, these PFs may use distinct terms to address the innovation process, new product development, technology development or research and development (R&D). These PFs were identified in a previous review on frameworks by Henttonen et al. (2016), complemented with more recent publications. To provide an overview on innovation measurement, they are presented in a comparative analysis in terms of necessary elements for a PF, namely: database elements – PIs and dimensions, the process (procedure) and supporting elements related to continuous improvement – performance progression and support for improvement actions. Despite the many contributions of the PFs, shown in Table 1, they present some shortcomings that need to be addressed in new research.

One of the underlying issues of the existing PFs leads to underestimating the potential of innovation measurement (Brattström et al., 2018). It refers to measuring only the parts as opposed to the whole, as PFs either use too few dimensions or overlook relevant ones to the current innovation landscape (Dziallas and Blind, 2018; Frishammar et al., 2019). Indeed, many PFs overlook performance dimensions already proven to be indispensable in the measurement of the innovation process, e.g. knowledge management (Adams et al., 2006; Crossan and Apaydin, 2010). Others more recent ignore new trends addressed in the current landscape mostly related to the innovation environment, such as openness and servitisation (Frishammar et al., 2019; Lee and Markham, 2016). By contrast, our PF aims to provide a comprehensive compilation of performance dimensions relevant to the innovation process demonstrated to be significant in past research.

An additional underlying issue is that many PFs apply quantitative PIs but neglected qualitative ones (Loch and Tapper, 2002, see Table 1). Others overemphasised output-oriented PIs that are also known as lagging (e.g. new products sales) over leading input-oriented PIs (e.g. level of awareness and clarity of innovation goals) (Werner and Souder, 1997). This over-reliance on quantitative and lagging PIs confines the performance evaluation to past results, which are hard to influence and act upon and can hinder a more balanced set of PIs (Dziallas and Blind, 2018). Authors such as Costa et al. (2014) argue that managers see greater value in the systematisation of leading PIs to create a balanced set of PIs because they enable management to proactively act on the course of ongoing projects. Hence, studies will have a higher value for practice if they also focus on leading PIs (Dziallas and Blind, 2018). In this sense, this research promotes a systematisation that congregates several PIs in a database, considering qualitative and quantitative and leading and lagging characteristics.

A further issue relates to missing the potential benefits of providing a step-by-step procedure to measure innovation and define action plans to improve performance in a systematic way, according to Crossan and Apaydin (2010). In this sense, researchers emphasise the need to extend the PF beyond just measurement to encompass the evaluation of performance to produce essential performance information to be used in the definition of actions plans (Brattström et al., 2018; Dziallas and Blind, 2018; Henttonen et al., 2016). Action must always follow measurement; otherwise, there is no point in wasting efforts. The PF proposed in this research followed the future research directions addressed by Brattström et al., (2018), as well as Dziallas and Blind (2018) to explore PFs with company-oriented research, which enables the proposition of a procedure to cover performance measurement and the definition of action plans. Such an approach would allow a deeper understanding of performance measurement and evaluation of the innovation process in real-world settings (Dziallas and Blind, 2018).

3. Research method

The proposed PF is built on two main sources, a literature review followed by two case studies in European manufacturing companies to address the research gap: the need to embrace recent dimensions and PIs (Dziallas and Blind, 2018; Frishammar et al., 2019) and to provide recommendations covering both measurement of performance and definition of action plans (Brattström et al., 2018; Henttonen et al., 2016). First, the purpose of the review is to identify the database and supporting elements to build a “conceptual” PF. Then, this conceptual PF is further developed in two action-oriented case studies.

Our qualitative case studies focused on in-depth study rather than large distribution in a population. In qualitative research, the aim is not to cover all possibilities for a theoretical domain exhaustively, but to incorporate relevant understanding in a strategically defined sample, which may comprise of only a small number of units (Coughlan and Coghlan, 2002; Fraga et al., 2020; Lakiza and Deschamps, 2019). The unit of analysis for this study was selected based on the relevance of manufacturing companies in the innovation landscape. Technology-intensive companies, characterised by technological accumulation produced by the design, creation and operation of complex systems, have similarities when it comes to the innovation process (Tidd et al., 2005). Improvements are developed incrementally, often associated with the diffusion of best practices throughout the company, which resonates with the performance measurement praxis. In addition, three prerequisites were considered for the companies: a minimum level of formalisation of the innovation process, strategic alignment and commitment to innovation (as used in Lakiza et al., 2018).

The qualitative case studies were conducted for 12 months in 2018 and 2019, with nearly 900 h of work in a medium-sized manufacturer of electrical power transformers and a manufacturer of navigating and control equipment for rail transport. We followed pertinent action-oriented methodology advice and used action research protocols to collect “facts” and gain “insights” about experiences and results from implementing the PF in conjunction with key employees (Coughlan and Coghlan, 2002; Lakiza and Deschamps, 2019). Considering that the research goal is to increase the understanding and propose a PF, we handled the qualitative study with document analysis, semi-structured interviews, focus groups workshops and evaluation questionnaires for each company independently. Information gathered through the interviews, focus groups and questionnaires was treated under a non-disclosure agreement regarding the use of confidential information only for purposes related to the PF proposition. The analysis of these two sources of empirical data, the companies and the relevance of participants yielded sufficient information to gain a better perception into the innovation processes and measurement practices and, in turn, develop the proposal.

The case studies were conducted with the active participation of 25 key employees, mostly process owners from senior and middle management (directors of technology, R&D managers, innovation coordinators, technology specialists, new business managers, product and process engineers, marketing managers), with an average tenure of ten years in their companies (Table A1, Appendix). The document analysis in each company involved the study of reference models, roles, product documentation, financial reports and other documents. Following this, 17 semi-structured interviews to capture the context of the two companies were conducted based on a script designed in four blocks to capture in 136 items: level of formalisation of innovation activities, innovation drivers and goals; closed and open innovation systems; and the surrounding environment. Then, eight focus groups workshops were held to apply and improve the synthesised “conceptual” PF with the support of journal-keeping according to Coughlan and Coghlan (2002). Finally, the key employees participated in the application of evaluation questionnaires aiming to capture closing feedback. Their input was fundamental to the validation of the proposal and analysis of its applicability. More details on data gathering can be seen in Table A2, Appendix.

The dataset served its purpose. As an exploratory study, we needed to test the applicability of the framing we arrived from literature and practice (Voss et al., 2002). In this sense, capturing insights from relevant practitioners across two representative manufacturing companies provided crucial validation. To generalise the results from two case studies is not possible (Coughlan and Coghlan, 2002; Voss et al., 2002). However, the main purpose of this study is not to generalise but to explore and increase understanding in the research field. Moreover, even though the initial building blocks of the conceptual PF came from the literature, the final result is a novel and unique arrangement that was only possible with the practice and feedback from the case studies. Thus, the two qualitative case studies were important as a basic tester that the authors believe has been valuable.

Several researchers favour the use of a small sample when new management tools such as PFs and models are being developed (Pigosso et al., 2013). Overall, despite the small sample and the qualitative nature of this research, 1,783 data collection points were gathered during this research, e.g. in the 13 interviews capturing 136 items, providing a sound foundation for the results discussed next.

4. Results

The conceptual PF with the elements synthesised from the literature review is first presented in Section 4.1, followed by the key insights from the two case studies in Section 4.2. These empirical insights were then reintroduced into the framework to be reapplied in the workshops at the companies, with the results being captured with the interviews, focus groups and the evaluation questionnaire, culminating in the new and consolidated PF presented in Section 5 (Figure 1).

4.1 Conceptual framework

The conceptual PF is grounded in a triple foundation defined according to the literature review (Section 2). The first consists of the database elements, composed of PIs and their dimensions. The second refers to the measurement process itself, i.e. the procedure to apply the PF. The third, in turn, relates to continuous improvement, comprised of the support for performance progression and improvement actions.

4.1.1 Database elements

The initial fundamental step towards addressing innovation performance measurement is to identify and make explicit the existing dimensions. Putting them all together provides an overview and allows PIs to be categorised. After screening the relevant literature, nine dimensions are determined, as they empirically demonstrated to be significant in the innovation process. The dimension retrieved systematically from the literature needed to fulfil certain requirements. First, it needed to demonstrate a clear link to innovation success (increase in customer satisfaction or sales/profit) and then be cited several times to achieve “theoretical saturation” (as seen in Adams et al., 2006), i.e. the point of significant data recurrence and failure to surface new data. Together, these dimensions determine the innovation process and the resulting outcomes.

Two categories of dimensions can be set up because multiple aspects related to internal and external elements affect the ability of companies to implement innovations successfully (as applied in Dziallas and Blind, 2018). First, company-specific dimensions include those that are particular to a company that affect organisational innovation behaviour. In our study, PFs are defined as: innovation strategy, organisation and culture, knowledge management, portfolio management, project management and team management. Second, contextual dimensions are related to the company and its surrounding environment. Thus, the contextual dimensions can be defined as innovation environment (which includes the company's reaction to openness, servitisation and sustainability forces), technology management and market.

Most PFs would present some degree of performance dimensions (e.g. Chiesa et al., 1996; Loch and Tapper, 2002). For instance, Crossan and Apaydin (2010) present several dimensions in common with this study, such as innovation strategy, knowledge management, culture, project management and market. Even though their PF is the one with most dimensions from the literature (seven), they do not provide a complete overview dimension-wise. Interestingly, Frishammar et al. (2019) and Richtnér et al. (2017) already identified the need to include considerations about openness and servitisation (included in the innovation environment). Still, they did not include previously studied dimensions like knowledge management and market. Thus, the conceptual PF presenting the nine dimensions provide a more comprehensive take on the innovation process than existing PFs.

The identified indicators are classified according to the mentioned dimensions above. Therefore, 259 unique PIs were systematically retrieved from the literature (the PFs reviewed in Section 2 and more studies systematically identified). Table 2 displays their distribution among the dimensions. The project management dimension has the highest number of indicators (43 PIs), representing 17% of the total. The second most populated dimension is technology management (35 PIs), scoring 14% of the total. Both figures are understandable because of the role played by techno-centric R&D literature from the 1990s in the dissemination of project management and technology-related indicators. Then, knowledge-related indicators have gained attention for the past two decades (34 PIs), with a representation of 13%. Later studies also included indicators associated with the innovation environment dimension (32 PIs). In addition, each retrieved PI is categorised into quantitative or qualitative and leading (predictive and input-oriented) or lagging aspects (output-oriented). Examples of the PIs are presented in Table A3 (Appendix), and because of its considerable size, the full list containing the 259 indicators retrieved is available in an external database [1].

4.1.2 The process

The second foundation of the conceptual PF refers to the measurement process, which is supported by the procedure. The procedure is organised into two main stages: diagnosis of the company's current measurement practices and deployment of action plans to improve performance, based on existing procedures of the reviewed PFs (in Section 2). The first stage, diagnosis of the current situation, can be defined as follows:

1. Outline the application and set up specific goals: most PFs start with outlining the importance of measuring the innovation process to gain support from the main stakeholders and establish the primary needs of the company (e.g. Brown and Gobeli, 1992; Chiesa et al., 1996; Lakiza et al., 2018).

2. Meet the stakeholders involved: the participation of main stakeholders is essential to not only define the goals but also give access to further key employees and data inputs for measuring the PIs (Chiesa et al., 1996; Lakiza et al., 2018; Loch and Tapper, 2002).

3. Define the dimensions and PIs to be measured: this step aims to determine the relevant dimensions for the company and select the appropriate PIs for this. Furthermore, this step can be supported by a predefined compilation of PIs (Berg et al., 2009; Brown and Gobeli, 1992; Chiesa et al., 1996; Lakiza et al., 2018; Loch and Tapper, 2002; Werner and Souder, 1997).

4. Measure the PIs: here, all the measurements are made. Two main tasks are involved: the collection of data from the company's documents/systems and assessments from key employees involved in the process being measured (Chiesa et al., 1996; Lakiza et al., 2018; Loch and Tapper, 2002).

5. Evaluate performance: this step is the “black box” of most PFs. Despite many mentioning the need to define the desired performance, existing procedures do not include further guidelines. This means that the evaluation is solely based on the participants' know-how (with no support from the PFs), which entails putting more effort into finding ways to analyse the measurements and acquire this knowledge and experience, which, in turn, can lead to likely discouragements and unexpected results.

After the initial evaluation of performance, there is a natural pause to validate the findings and decide to go forward (or back and perform other cycles of diagnosis again), similar to a decision gate. The second stage of the conceptual PF, deployment of action plans, involves:

• (6)Identify where there are needs to be attended: the performance evaluation should help identify improvement gaps. However, as there is no structured support in the current literature, this step is rather superficial and almost exclusively based on the employees' know-how.

• (7)Propose action plans to improve performance: the majority of PFs indicate this step (e.g. Chiesa et al., 2009; Lakiza et al., 2018), but without major support, this step is often performed in an ad hoc basis. Still, managers find the definition of action plans especially difficult, even when they have identified the gaps that need improvement.

• (8)Prioritise actions and assign responsibilities: as companies have limited resources and time, the improvement gaps identified should not be tackled all at once. Existing PFs acknowledge the need to prioritise, but they leave at the discretion of the company to prioritise along with the task to assign who is responsible for implementing the actions (Chiesa et al., 1996; Lakiza et al., 2018; Loch and Tapper, 2002).

• (9)Support performance evaluation over time: the final step revolves around developing instruments to support performance progress in a continuous manner. Two PFs put to use distinct levels of performance going from poor to best practices to characterise the progression of the performance, but these do not relate with the PIs measured, leaving space for a better definition of this step in practice (Chiesa et al., 1996; Kahn et al., 2006). As they are intrinsically related to the foundation for continuous improvement, they are discussed next.

4.1.3 Continuous improvement

The third foundation of the conceptual PF relies on the support for continuous improvement, comprising the support for performance progression and improvement actions. For this, evolutionary approaches focusing on business process improvement that show the progression of performance across the considered dimensions are the go-to solution in other domains, e.g. in software management (Henttonen et al., 2016).

The evolutionary approach can be characterised by the application of distinguishing levels defined to portray the progression of components or the entirety of the process itself, across a number of dimensions to facilitate benchmark (Pigosso et al., 2013). To simplify, this approach is described with a limited number of levels, typically four, as seen in the PFs of Chiesa et al. (1996) and Kahn et al. (2006). These levels are sequentially ordered, from an initial level up to an ending level, considered the level of “excellence”. As a starting point, the PF adopts the four-level range, one to four, to indicate the progression of performance that a company might present. This range is chosen because it is easier to understand, but the final configuration depends on the case studies results.

The characterisation of each performance level can be qualitative with the description of practices of increasing sophistication as the levels of performance increase, or quantitative with benchmark values of the PIs, depicting gradually increasing performance (e.g. Chiesa et al., 1996; Kahn et al., 2006). These two types of characterisation together enable a more structured way to depict performance progress. Thus, for this, the database of PIs also compiles the benchmark values from the respective studies to help structure the evolutionary approach into the four levels.

Two final considerations must be done in the conceptual PF to address the definition of action plans. The first consideration revolves around the analysis of the current performance (Tangen, 2004). In this sense, the portrayal of the performance progression into increasing levels helps in the ponderation of the magnitude of how well the company is doing. Therefore, for the definition of where the company wants to go, i.e. the desired performance, the conceptual PF is envisioned with four performance levels, considering the implementation of one level at a time, targeting the gaps located at the lowest level and then moving up. The second consideration that needs to be addressed is the definition of action plans per se. For that, the literature presenting the PIs retrieved for the PF was further examined for key actions implemented that were cited by the respective authors to improve performance after the measurement. These cited actions were compiled in the same way as the PIs. The idea behind these action examples is to inspire managers and guide the definition of new action plans.

4.2 Empirical insights

This conceptual PF, with its three foundations, was applied in the two manufacturing companies, as explained in Section 3. The insights from the two companies are jointly analysed for the purpose of this research. Although they were captured separately, they were reintroduced into the PF and subsequently applied in further cycles at the two companies. In general, the feedback following the interviews and workshops supported some aspects of the conceptual PF. However, they also pointed out several additional improvements, which are summarised as follows. The first insights relate to the database elements:

1. Employees from both cases emphasised the need to carefully articulate PIs that help sustain a deliberate and explicit innovation focus. Otherwise, they risk measuring too much or too little. For this, the PIs was systematised into rapid assessment and in-depth indicators. The first are PIs from existing PFs used in the diagnosis of a company's current state, and because of this, they often present benchmark values, while the latter can be used for a more in-depth analysis after the diagnosis, whenever necessary.

2. The need for a simple visualisation of the company's current situation was brought up. Thus, a visual diagram of the rapid assessment PIs to portray the innovation capability profile was created and validated. It displays the 34 rapid PIs with four performance levels of cumulative stages, which are delimited by benchmark values.

In turn, the insights referring to the PF procedure can be highlighted as follows:

1. For the participants of the two companies, the PF needed a formal definition of the stakeholders that must be involved in the innovation measurement. This step involved the creation of a stakeholder map relating influence and interest, which, despite being a simple tool, was not applied in the context of a PF before. We recommend that stakeholders who have a moderate interest in the measurement process, but no strong influence, should be involved in the application of the PF.

2. A formal validation step leading to the second stage of the PF (the deployment of action plans) was introduced according to the participants' feedback to work as a mid-way milestone to generate awareness and dissemination among employees about the implementation of action plans to come.

3. An organised twofold approach to carry out the step of performance evaluation was proposed with the participation of the first company and replicated in the second, which yielded encouraging feedback. It was observed that different approaches to process improvement might be appropriate for companies depending on their innovation capability profile; those with lower levels (1 or 2) would typically require more direction and structured guidance, while those with higher levels (3 or 4) would naturally have more freedom to select the dimensions according to their drivers and strategic goals.

4. The suggestion of including criteria to help with the prioritisation of action plans was another insight worthy of note. The two companies were able to select the appropriate criteria they wanted to use and rank the action plans. The available criteria include implementation time, strategic alignment, top management support, resources availability, cost, competitive advantage, legal compliance and return on investment.

Lastly, the insights for improving the third foundation, performance progression and support for improvement actions, have been addressed in several focus groups by the participants in the following points:

1. Feedback from the participants indicated that a concise designation for the “overall” innovation profile would help disseminate measurement practices, i.e. if the company is level 1, 2, 3 or 4. Thus, the profile is informed by the lowest performance level with more PI measurements falling into the ranges delimited by the benchmark values.

2. A systematisation to help managers prioritise improvement actions was developed by applying the prioritisation criteria with the analytical hierarchic process (AHP). AHP is a decision-making approach that involves decomposing a decision into pairwise comparisons so stakeholders can make value judgements about alternatives that are later aggregated into a ranking (Saaty, 1990). With this systematisation, managers can prioritise “quick win” action plans and build momentum in the company.

3. The introduction of examples of companies applying the improvement actions from past studies was also of great help. The suggestion came from the first company and was exceedingly welcomed in the second. These examples helped define new plans that, in turn, included project charter templates and further in-depth PIs to monitor their implementation.

Based on the conceptual PF and the insights from the case studies, the main output of this research is the consolidated version of the framework to support companies in measuring their current performance and then defining action plans. It is important to note that this consolidated PF is currently in use in the two participating companies.

5. Consolidated framework

Drawing on both literature and practice, we have defined a two-stage PF that can help companies improve their innovation measurement in the: (1) diagnosis of the current situation and (2) deployment of action plans. A simplified illustration of the PF is shown in Figure 2, indicating the steps from prior literature and the ones emerging from the case studies, all discussed next.

5.1 Stage I: diagnosis

The aim of stage I is to assess the current state of the company in relation to the innovation measurement practices across the nine relevant dimensions. This diagnosis is accomplished through the eight steps outlined as follows.

5.1.1 Outline the initiative

The main objective of this step is to pitch the initiative to the top management to gain support and help secure traction for the following steps. One or more facilitators should take the lead on this step, making sure that the head(s) of departments are on board. This step was already foreseen and discussed in previous literature (Section 4.1.2).

5.1.2 Setting up a core team

The purpose of this step is to set up a core team within the company, who will be tasked with leading the implementation of the PF. Ideally, this team will be composed of four members with some experience in improvement initiatives (in other domains, for example). This step is not common in the literature. Still, Chiesa et al. (1996) indicated a similar step aiming to create a small team to lead the application of their PF (Table 1 in Section 2).

5.1.3 Definition of expectations and pre-requirements

This step aims to build a mutual understanding of the goals as well as to identify any prerequisites. For instance, this PF has a focus on the development and analysis of the performance and, therefore, the development of software is not within the scope of the framework. Previous PFs did not indicate similar activities performed in this step. Thus, this step emerged from the case studies findings.

5.1.4 Characterisation of the innovation process

The phases of the innovation process performed by the company even when not highly formalised are identified in this step. It includes a document analysis, as well as interviews of key employees to understand the day-to-day processes and to clarify issues of the analysed documents. This step was developed in action at the two participating companies to enable a better understanding of the company's current innovation process activities.

5.1.5 Adaptation of the data collection instruments to the company's vernacular

This step aims to adapt the instruments for collecting data to measure the 34 rapid assessment PIs (Table A3, Appendix) to the company's vernacular, to ensure that the questions collecting information will be effectively understood and the answers with data inputs for the indicators will be reliable. This step was an outcome of the action in the companies.

5.1.6 Joint definition of key employees to be interviewed

In this step, the core team identifies key employees to be interviewed, prioritising a variety of participants from different areas and hierarchical levels to ensure a broad view on how the innovation process is conducted in the company. The core team is responsible for organising the interviews, contacting the employees and providing the necessary resources. Similar steps can be found in previous PFs (Section 4.1.2). The new introduction is the use of stakeholder maps to help with the definition of the participants, which arose from the case studies.

5.1.7 Interviews for performance dimensions assessment

This step aims to measure the performance of each PI, based on the inputs of key employees and evidence collected. Face-to-face interviews are performed with the selected employees using a structured questionnaire with the 34 rapid assessment PIs adapted to the company's vocabulary. Interviews capturing performance data are well-applied in existing PFs (Table 1 in Section 2). Yet, the novelty here is the unique systematisation of 34 PIs out of 259 to facilitate the diagnosis.

5.1.8 Consolidation of the diagnosis

Finally, the performance levels (1–4 defined by the benchmark values of the 34 rapid PIs) are assigned and compared with the evidence collected to validate the accuracy of the answers and to ensure coherence. With the performance levels set, the company's current innovation capability profile is designed using the radar representation (see Figure 3 for the navigating and control equipment manufacturer, and Figure A1 for the power transformer manufacturer, Appendix). Existing PFs present activities of a similar nature to validate the findings and decide to go forward (Section 4.1.2).

5.2 Stage II: deployment of improvement projects

Once the profile of the company's innovation capability has been created, stage II can begin. The goal is to support the deployment of action plans. The PF indicates six steps to identify past examples of actions taken to prompt improvement plans based on the identification of gaps between current and desired performance within the dimensions where there are needs to be addressed.

5.2.1 Definition of the vision for the desired performance

This step begins with the preparation of the desired level of performance based on a gap analysis. The PF allows a degree of flexibility in the definition of the desired performance so that it can be aligned with the company's diagnosis. There are two possible approaches to structure action plans:

1. Staged approach: is an orderly way to define the vision for companies with a low-performance level (levels 1 and 2). It is based on the implementation of one level at a time, targeting the gaps located at the lowest level, and moving up.

2. Continuous approach: is a more flexible approach recommended only for companies characterised with a higher performance level, i.e. the performance level 3 or 4. In this case, the company can choose to focus on different levels related to one or several dimensions according to its own drivers.

This twofold approach that emerged from the case studies is a novelty in the proposition of frameworks, as such an approach has never been proposed for PF focused on the innovation process before.

5.2.2 Identification of appropriate action plans within the dimensions

Past actions collected from the literature to inspire and help structure plans to address the improvement gaps defined by either approach, staged or continuous, are identified in this step. Most PFs would have a step to define action. But, the difference here is that each gap is analysed with the help of the collected actions related to that specific PI. This introduction from the action in the companies enables the team to select the most beneficial past actions that can be used as guidelines for new action plans for the company.

5.2.3 Design of improvement projects for the innovation practices

In this step, the actions selected are specified into improvement projects. The use of improvement projects has long been used in designing action plans. For each gap, these projects are structured in terms of goals, deliverables, requirements, risks, time and resources, tailored to the company's context.

5.2.4 Selection of indicators to monitor the implementation

To track the implementation of the improvement projects, the database with its unique, comprehensive coverage of indicators from the literature can be further consulted to select additional PIs to monitor implementation.

5.2.5 Prioritisation of improvement projects

At this point, the projects should be prioritised. The company can use the systematisation of prioritisation based on the AHP (Figure A2, Appendix). Although AHP application in prioritisation can be found in the literature, their use in PFs is not usual. The inclusion of the prioritisation criteria – implementation time, strategic alignment, top management support, resources availability, cost, competitive advantage, legal compliance and return on investment – emerged from the activities performed in the companies.

5.2.6 Planning and assignment of responsibilities

As a final point, we suggest that the company adopts its usual procedure for planning the projects' implementation. Portfolio management concepts are often used in PFs and improvement initiatives to plan an implementation roadmap of the improvement projects. Schedule, work packages, champions and the teams involved should be defined following the company's usual project management practices.

Finally, during the implementation of the improvement practices, special care should be taken regarding change management, as people are the gatekeepers of change. The total duration of the PF application can vary according to top management support, resources available and the resistance encountered in the organisation.

6. Discussion of framework applicability

The findings of this research emphasise that a PF needs to encourage managers to recognise the importance of establishing a process for innovation measurement. By considering the literature and case-oriented research, the proposed PF has the potential for further applicability.

Applicability refers to the conditions of the study given by the context in which the new tool is to be used (Voss et al., 2002). Poor contextual conditions and lack of clear procedures can make a subsequent data analysis especially difficult. To overcome this, the research employed an intentional strategy for selecting the industry partners, taking into consideration sectorial and contextual particularities (Tidd et al., 2005). Besides, the choice of conducting action-oriented case studies in two companies to further develop and test the PF was key to leverage the experience and tacit knowledge of the practitioners.

This new and updated PF presents interesting theoretical implications. First, the identification of nine dimensions and 259 PIs can be beneficial for researchers who may need a swift identification of these relevant database elements, including the ones addressing recent trends in the innovation landscape, e.g. innovation environment with openness, servitisation and sustainability. Hence, the core contribution to the literature of this study relies on the systematisation of these PIs into nine dimensions and the identification of 34 rapid assessment ones. Indeed, indicators that are reliable and valid enable the accumulation of research in a scientific field and free further researchers from the need to redevelop them. Although, there are dimensions and PIs in common with existing PFs, the most recent PF presented only 37 PIs with no dimensions identified, and the one with the highest number of compiled indicators identified 78 PIs and seven dimensions (Section 2).

Implications for medium-sized manufacturing companies, like the ones from the case studies, relate to their advantage of being more flexible and adaptable to changing market needs than large companies due to fewer management layers and less bureaucratic. The empirical insights that allowed the creation of a step-by-step procedure, which includes two distinct approaches (staged and continuous) for defining the desired performance, can facilitate their adaptation to other companies similar in size or even smaller ones as they add flexibility to the process. By contrast, large companies are more likely to have a greater diversified product portfolio that can be distributed in distinct business units (BUs) and, hence, more bureaucratic. The application of the PF can be broken down into the two stages. The diagnosis should be applied to each BU responsible for the development of new products. Except when there are BUs that only manufacture the products designed by the headquarters; in this case, both units should be considered jointly. The second stage, deployment of action plans, should then involve a cross-functional team from the BUs to analyse the complementarities of the action plans to propose which ones should be implemented at the institutional level or locally.

In summary, the PF was applied in the two companies with similar success. The fact that both companies are currently using the PF, implementing the indicated improvement projects [2] and suggesting the PF to other companies from the same holding group demonstrates additional corroboration of this research contribution to practice. The success of implementing the PF was tangible to the companies, but there were also non-tangible benefits. Some examples include: the awareness of the two companies' personnel was raised – enabling a broad understanding of the current measurement practices, the PF gave management a “feel good” factor after the diagnosis when some of their suspicions of weaknesses were confirmed and it did not cause any drastic changes. Instead, the PF advocates the prioritisation of the improvement projects and, hence, was looked upon favourably by the management.

7. Conclusions

Companies are increasingly studying their innovation processes in the effort to become more innovative. Measuring innovation performance is central to these efforts (Brattström et al., 2018; Crossan and Apaydin, 2010). Within this context, PFs are management tools that enable managers to measure innovation performance. Our PF is designed to help manufacturing companies take control of their innovation measurement by supporting the analysis of strengths and weaknesses across nine up-to-date dimensions and the definition of suitable action plans, thereby allowing them to realise the full benefit of their innovation measurement efforts.

7.1 Implications

The paper highlights the PF implications for practice. The PF can provide a roadmap for improvement opportunities to achieve the desired situation, as well as establish a common language and a shared vision across the company. On the other hand, establishing a measurement process in a company is by no means straightforward. To avoid common mistakes, managers should take a holistic perspective on their company's innovation process. Our PF allows managers to not only measure innovation performance and identify gaps between current and desired performance into nine performance dimensions that address the current innovation landscape but also define action plans to improve performance. In short, we are confident that our PF can give companies a more efficient and practical approach to continuous improvement towards better innovation performance. The findings of the study have the potential to be applicable to any company and not just manufacturing companies, with more replications of the study.

This research also has theoretical implications. With the literature review, it was possible to provide a systematisation of PF elements: dimensions and PIs from the literature, into a database from which researchers can evaluate the applicability to other contexts. A compilation of such magnitude in terms of PIs and dimensions enable the update and accumulation of research in a scientific field and free researchers from the need to redevelop these measurement instruments. Moreover, the number of PIs dimensions indicates a more comprehensive view of the process than existing PFs. Further contributions involve the proposition of a procedure to deploy action plans based on continuous improvement, answering a call for more research on this subject (e.g. Brattström et al., 2018; Lee and Markham, 2016). Nevertheless, the extent to which the results of this study are transferable to other research remains open. Therefore, we conclude that our study represents a starting point for further research aiming at extending our findings to other companies and industries.

7.2 Limitations

The action-oriented case studies included a small rather than a large sample. Similar to other qualitative research, this study has a limited scope in terms of transferability of the findings (e.g. Pigosso et al., 2013). We tried to overcome such limitation by following a strategic selection of the industry partners, taking into consideration sectorial and contextual particularities. We selected the units of analysis based on the relevance of companies and participants. However, to gather more evidence and establish generalisations for the application of the PF, replications of the case studies must be performed in future research.

Figures

Figure 1

Illustration of research results

Figure 2

Simplified illustration of the PF

Figure 3

Innovation capability profile of the manufacturer of navigating and control equipment for rail transport. It shows the nine dimensions (from innovation strategy to market) with the 34 rapid assessment PIs (IS1, IS2, so on until MA15) arranged in a radar representation with four distinct levels

Figure A1

Innovation capability profile of the power transformer manufacturer

Figure A2

Prioritisation of improvement projects for the power transformer manufacturer based on AHP

Table 1

Comparative of existing PFs for innovation measurement

Performance frameworks	Database elements		The process	Continuous improvement
	PIs	Dimensions	Procedure	Performance progression	Improvement actions
Brown and Gobeli (1992)	“Top ten” quantitative PIs related to productivity, quality, timeliness and R&D costs (all lagging PIs)	Project management, people, innovation planning, new product development, outputs, and division results	It starts with steps to clarify the responsibilities, R&D activities and strategic goals. Then, the shortlist of PIs is defined. Following this, data are captured to assess performance	–	The improvements focus on the production processes
Chiesa et al. (1996)	40 qualitative and quantitative PIs across the dimensions (with leading and lagging PIs)	Strategy and leadership, tools and resources, organisational culture, and process management (concept generation, technology acquisition and product development)	It includes setting up facilitators and a small team, meet to discuss initial findings, key stakeholders' interviews, validation of the diagnosis, discussion and identification of improvement gaps, prioritisation of actions and assignment of responsibilities	Evolutionary levels 1 to 4 in each dimension characterise distinct levels of practice progression, together with a graphical representation of the improvement gaps	PF enables the identification of improvement actions, but it is not clear how they are identified and relate to the evolutionary levels
Werner and Souder (1997)	Eight top-cited PIs (mostly quantitative and lagging PIs)	Financial, project effectiveness, technical abilities and R&D productivity	–	–	–
Loch and Tapper (2002)	28 quantitative PIs formulated where the PF was applied (mostly lagging PIs)	New technologies and breakthrough concepts, customer support, knowledge repository and external reputation	A superficial procedure is presented in a case study	The PIs are organised into a radar graph that illustrates evolutionary levels of performance	–
Kahn et al. (2006)	–	Innovation strategy, portfolio management, process, market, people management and metrics and performance evaluation	–	Evolutionary levels 1 (poor practices) to 4 (best practices) are used to portray performance progression	Improvement gaps can be identified, but no further support is provided
Berg et al. (2009)	16 qualitative and quantitative PIs related to finance, patents, people, time, market and technical attributes (with leading and lagging PIs)	Market; process (opportunity identification, idea generation, project plan); outcomes and social/ physical environment	Five steps for the analysis of the companies' innovation environment, selection of measurement criteria, selection of data sources; data collection and analysis	It uses maturity levels to address the progression of performance (from the capability maturity model integration)	–
Chiesa et al. (2009)	55 qualitative and quantitative PIs used in the participating companies (mostly lagging PIs)	Financial, customer, innovation and learning and business processes	–	–	–
Crossan and Apaydin (2010)	78 qualitative and quantitative PIs addressing the dimensions (with leading and lagging PIs)	Innovation strategy, knowledge management; organisation/culture, portfolio; project management, people and market	–	–	–
Richtnér et al. (2017), Frishammar et al. (2019)	–	Innovation process, resources and capabilities, culture, business model and innovation performance (including openness and servitisation)	The steps begin with: identify existing practices, assess current innovation focus/priorities, develop or improve PIs for the process, portfolio and projects and set and implement measurement routines	–	Planning action plans is included in the PF
Lakiza et al. (2018)	37 PIs mostly based on productivity (mainly quantitative)	–	It starts with meeting main stakeholders, understanding the needs, selecting dimensions, clustering topics into the dimensions, proposing PIs, gaining feedback, assessing innovation process maturity, implementing PIs, proposing encouraged improvement practices	The study superficially discusses the progression of performance, but it does not go further into providing a proposal	There are steps to identify improvement gaps and define actions plans

Table 2

PIs distribution among the dimensions

Dimensions	Total PIs (%)	Nature			Type
		Qualitative	Quantitative	None*	Leading	Lagging	Both**
Innovation strategy	24 (9%)	17	6	1	15	5	4
Knowledge management	34 (13%)	12	15	7	27	3	4
Organisation and culture	25 (10%)	14	5	6	23	2	–
Portfolio management	16 (6%)	8	6	2	15	–	1
Project management	43 (17%)	13	26	4	32	3	8
Technology management	35 (14%)	16	15	4	20	11	4
Team management	29 (11%)	14	8	7	29	–	–
Market	21 (8%)	12	8	1	14	4	3
Innovation environment	32 (12%)	12	17	3	27	3	2
Total	259	118	106	35	202	31	26

Note(s): ^*Note that 35 PIs did not present their formula (but they had further definitions)

^**Sometimes, PIs can be both leading and lagging at the same time

Table A1

Participants profile. Note that although more employees joined occasionally, the following 25 participated throughout the study

Company	N°	Position	In-house experience (years)	Area/department
Power transformer	1	Coordinator	17	Innovation and technology management office (I&TMO)
	2	Specialist	7	(I&TMO)
	3	Manager	5	Product engineering
	4	Manager	20	Product engineering
	5	Department head	22	Product development
	6	Senior manager	30	Processes development
	7	Senior manager	24	R&D
	8	Senior manager	20	R&D
	9	Department head	21	Technology
	10	Manager	15	Commercial
	11	Manager	12	Marketing
	12	Director	1	BU
Navigating and control equipment for transport	13	Coordinator	17	Innovation management
	14	Coordinator	7	Quality
	15	Manager	1	Project management office (PMO)
	16	Manager	11	Technology radar
	17	Manager	12	Operation support systems
	18	Senior manager	18	Business development
	19	Department head	15	Product development
	20	Department head	15	Service
	21	Senior manager	17	Tendering
	22	Coordinator	4	R&D
	23	Department head	10	R&D
	24	Department head	16	Technology
	25	Director	5	BU

Table A2

Details on the data gathering of the action-oriented case studies

Data gathering methods	Data volume
Main activities	(Hours spent)
Document analysis	1,313 pages
Collection	(41.5)
Analysis	(81.8)
Key employees interviews	13
Planning	(89.3)
Interviews	(35.4)
Transcription and coding	(108.8)
Analysis	(155.6)
Focus-groups workshops	8
Planning	(85.9)
Execution	(33.3)
Analysis	(233.4)
Evaluation questionnaires	14
Planning and application	(13)
Analysis	(71)
Total	(949)

Table A3

Summary of the 34 rapid assessment PIs; dimension, title, nature, type and citations (#). Further PIs can be found in the database

Dimensions	[ID] Title	Nature	Type	#
Innovation strategy	[IS1] Level of awareness and clarity of innovation goals	Quantitative	Leading	7
	[IS2] Corporate goals for new product development program	Qualitative	Leading/lagging	5
	[IS3] Product planning horizon	Qualitative	Leading	3
	[IS16] Top management support for innovative ideas	Qualitative	Leading	4
Innovation environment	[IE1] Recognition that key problems must be solved with skills outside the organisation	Quantitative	Leading	3
	[IE2] Collaborative projects through an external vs internal open innovation focus	Quantitative	Leading/lagging	2
	[IE12] New product diversification as a strategy: goods, services or inseparable mix of both	Quantitative	Lagging	3
	[IE21] Sustainability criteria for innovation projects	Quantitative	Leading	3
Knowledge management	[KM1] Rate of generated ideas according to formal innovation process vs informal	Quantitative	Leading	7
	[KM8] Knowledge acquisition vs absorptive capacity	Qualitative	Leading/lagging	2
	[KM16] Importance of diversity of knowledge sources	Qualitative	Lagging	2
	[KM27] Time off for creative things	Qualitative	Leading	2
Organisation and culture	[OC1] Organisational climate for innovation projects	Quantitative	Leading	3
	[OC9] Work environment support for innovation projects	Qualitative	Leading	3
Portfolio management	[PFM1] Level of formalized portfolio management	Quantitative	Leading	3
	[PFM10] Portfolio decision-making effectiveness	Quantitative	Leading	3
	[PFM11] Innovation project portfolio alignment	Quantitative	Leading/lagging	4
Project management	[PM1] Level of resources commitment to innovation projects	Quantitative	Leading/lagging	3
	[PM23] Percentage of use of project management tools	Quantitative	Leading	3
	[PM24].Frequency post-launch evaluation procedures	Qualitative	Leading	3
	[PM31] Internal and external communication quality	Qualitative	Leading	6
	[PM32] Time-to-market management	Qualitative	Lagging	3
Technology management	[TM2] Level of monitoring new technologies	Qualitative	Leading	2
	[TM12] Intellectual property protection strategy effectiveness	Qualitative	Leading/lagging	2
	[TM13] Degree technology tools used	Quantitative	Leading	3
	[TM22] R&D intensity	Quantitative	Leading	2
Team management	[TEAM1] Level of cross-functionality in teams	Quantitative	Leading	3
	[TEAM2] Identifiable project leader	Quantitative	Leading	3
	[TEAM3] Frequency of cross-functional training	Quantitative	Leading	4
	[TEAM5] Dedicated group assigned to innovation tasks	Quantitative	Leading	3
	[TEAM17] Innovative team behaviour	Quantitative	Leading	2
Market	[MA1] Percentage of market research tools use	Quantitative	Leading/lagging	3
	[MA11] Product customer testing proficiency	Qualitative	Leading	2
	[MA15] Market launch proficiency	Qualitative	Leading/lagging	5

Notes

1. Due to its considerable size, the full list containing the 259 PIs can be accessed: (https://mfr.osf.io/render?url=https%3A%2F%2Fosf.io%2F7bp9q%2Fdownload).

2. The journal and improvement projects implemented cannot be published due to the non-disclosure agreements, therefore omitted from this document.