Student perceptions of Living Lab research internships in the COVID-19 pandemic – a Dutch case study

Abstract

Purpose

To effectively generate solutions to today's complex challenges, cooperation between governments, industry, civil society and academia is essential. To adequately prepare students for collaboration across academic and non-academic disciplines and stakeholders, Living Labs (LLs), unique research internships have emerged in the educational systems, which are focused on generating insights for society while embedding student learning in both practice and academia. To legitimise the LLs as a method of education in the academic curriculum, it is necessary to evaluate the experience of and potential benefits for students with regard to the development of their academic, professional and personal skills. Hence, this paper aims to investigate the outcomes of participating in LLs from the student's perspective via a case study at the University of Groningen in the Netherlands. A secondary aim is to evaluate the influence of the COVID-19 pandemic on the learning experience of the students.

Design/methodology/approach

The authors employed self-reported pre-and post-questionnaires into students' confidence levels with regard to their academic, professional and personal skills. A total of 35 questionnaires were conducted during the period February 2020–July 2021. The authors subsequently applied evaluation research, using a benchmarking approach, to analyse the data.

Findings

This study firstly indicates that students are most confident in their personal skills, both before and after conducting the LL, and that they further developed these due to being pushed outside of their growth zone by the various challenges posed during the LL, including the COVID-19 pandemic. Secondly, while students seem to have become more aware of their professional skills in the LL, this was the aspect on which no improvement was reported after conducting the LL, potentially due to an increased awareness of one's own room for improving professional skills outside of academia. Thirdly, students' reported academic skills improved the most during the LL, which highlights the importance of embedding academic learning both in theory and practice. Lastly, the impact of COVID-19 changed the setting of the LL which led to challenges but also opportunities with respect to research design, time investment and communication.

Practical implications

As LLs seem to contribute to transversal and academic skill development of students, we can legitimise their increasingly common place in higher education curricula. LLs are not only beneficial for stakeholders and society as they generate new insights into societal questions, they are also of added value to students who actively collaborate with the external organisations and researchers. The LLs can thus be seen as a method of education which contributes to students' preparation for future careers, which is one of the main tasks of higher education institutions. Due to the COVID-19 pandemic the LLs were conducted online, which resulted in disadvantages and advantages. Future LLs can be anticipated on more hybrid or even further online collaborations, which also opens the possibility of collaboration with international organisations located at different parts of the world. In this case, extra attention will need to be devoted to aligning expectations among the different stakeholders and specifically focussing on ways to ensure good communication.

Originality/value

This case-study is one of the first studies that specifically looks at the newly emerging concept of LL research internships and the perception of the student while conducting LLs with societal partners. Previous literature on the topic is scarce and, if existing, has mainly focused on the benefit of the partner organisation or society. Instead, we purposefully reflect on how the collaboration contributes towards students' professional development and employability. By doing so, this paper is one of the first to shed light on the benefits accrued to students’ development by participating in an LL. In addition, as the case study, unexpectedly, took place at the height of the COVID-19 pandemic, we were also able to evaluate the influence of COVID-19 on the LLs and draw insightful lessons learned for future collaborations with local and international partners in an online setting.

Citation

van Engelenhoven, A.M., Geiger, J.L., Ünal, A.B., Spinder, R.-N. and van der Zande, I.S.E. (2023), "Student perceptions of Living Lab research internships in the COVID-19 pandemic – a Dutch case study", Education + Training, Vol. 65 No. 10, pp. 65-80. https://doi.org/10.1108/ET-06-2022-0236

Introduction

Educational systems and teaching methods are dynamic constructs that are constantly in development. For example, the nineteenth century witnessed a convergence of the educational system to the technological requirements of the high industrial era (Müller et al., 1989). After the Earth Summit in 1992, in which (higher) education was recognised as one of the main drivers of sustainable development, the start of this century was marked by the emergence of an ecological driven agenda in educational policy with emerging frameworks and declarations such as Education for Sustainable Development (Elliott, 1999; UNESCO, 2004). Looking ahead, it can be noticed that educational programmes are currently influenced by practice driven trends, such as experiential learning (Lisko and O'dell, 2010) and practice- and work-based learning (Hynes et al., 2011; Alvarez and Rogers, 2006; Coll et al., 2003; Lester and Costley, 2010). The latter is only logical, as today's challenges reflect the dynamic and complex state of world affairs and cut across various disciplines. Solutions can no longer come from isolated improvements in one single area. They can only be addressed jointly by government, industry, civil society and academia: the quadruple helix. Collaboration between these stakeholders requires a shift towards inter- and transdisciplinary education and research: a learning environment where partners from various academic and non-academic backgrounds break down traditional boundaries and collaborate across fields to produce knowledge and innovation with social relevance (Klein, 2004; Nowotny et al., 2001; Carayannis and Campbell, 2009).

Future professional success of students in this changing labour market requires that they not only develop themselves intellectually, but also build future-proof and lifelong transversal competencies such as collaboration, creativity and leadership (Sá and Serpa, 2018; Rourke et al., 2018). Higher education institutions across the world have responded to the call to prepare future citizens for and provide training beyond transferrable content knowledge, for example through stimulating collaboration with societal partners in civic or global engagement projects (Grad and Van der Zande, 2022), or by the more recent focus on institutionalising graduate attributes. The idea of graduate attributes emerged in response to concerns about students being ill-equipped to access the changing labour market and involves learning objectives such as “stakeholder collaboration”, “being able to deal with change”, or “having a global outlook” (Harvey and Knight, 1996; Barrie, 2006; Rourke et al., 2018). This conception of education requires a transformation with regard to what is taught as well as how it is taught, i.e. new pedagogical methods. One increasingly emerging way in which the current educational system often aims to integrate theory and practice is through internships (here defined as gaining work experience at an organisation) or research internships (here defined as conducting research on behalf of an organisation). The newest trend in the area of experiential learning is coined Living Labs (LLs), which are particularly focused on generating insights for society while embedding student learning in both practice and academia (Hawk et al., 2012). In essence, LLs aim to integrate the learning experience of both traditional and research internships, thereby creating a unique opportunity for students to follow an inter- and transdisciplinary academic approach to address societal challenges. Research points towards the importance of collaboration between universities and businesses as an important tool to foster innovation and as a source of new ideas (Bravo-Biosca, 2020). At present, studies that have been devoted towards the investigation of LLs is, however, limited and mostly directed towards the benefits for firms and organisations. Such benefits for example include a superior match of innovations with user needs (Leminen et al., 2012), avoidance of path-dependencies and lock-ins in innovation (Niitamo, 2006) and enhanced collaboration between stakeholders (Fahy, 2007). Benefits accrued to students, who are at the centre of the LLs, have not yet been investigated.

To further enhance the understanding of LL outcomes and benefits, and thus legitimise the increasingly common practice of integrating LLs in the academic curriculum, this paper aims to firstly investigate the outcomes of participating in LLs from the students' perspective. To gain insight in this, a specific LL case study at the University of Groningen in the Netherlands is evaluated. This LL programme links students to research and practice over a period of 5 months, with the aim to foster co-creation between stakeholders to find novel solutions to societal challenges. The case study took place between February 2020 and July 2021, in which the LL programme was carried out twice. As such, the LLs took place in the middle of the COVID-19 outbreak which influenced the set-up and, potentially, the experience of the students. The influence of the pandemic on the LLs and the perception of students regarding this is therefore an important secondary outcome of this study. LLs were traditionally designed to be conducted onsite in a physical environment, having them completely online changed the way of interaction and communication in the labs. Lessons can potentially be learned with regard to future LLs, whether they are online out of need, for example due to the persisting pandemic, or whether they may deliberately be online, for example in an international collaboration, which could potentially broaden the scope of the LLs further.

Background

The concept of Living Labs

Traditional internships can be seen as the predecessor of LLs. These internships have been linked to employability of college graduates and gained increased importance in higher education with the prevalence of college graduates undertaking an internship growing from 17% in 1992 to over 50% in 2008 and colleges offering increasing numbers of experiential learning experiences (Lundsteen and Edwards, 2013; Gardner, 2020). The literature on internships, sometimes including research internships, is plentiful and offers broad insights in its benefits and outcomes for students. Often cited outcomes of internships include support for professional development (Beck and Halim, 2008), higher employability chances and job marketability (Gault et al., 2000, 2010) and greater monetary compensation (Saniter and Siedler, 2014; Gault et al., 2000). Over the last decade, internships have evolved from opportunities to gain experience at the workplace to a comprehensive co-creation process between students, researchers, organisations and, in recent years, civil society (Cupps and Olmosk, 2008). The LL concept is such a co-creation process.

Nowadays, the LL concept is also applied to teaching environments to develop students' professional and academic skills while viable knowledge is created for the company involved (Bourgault, 2012). The first Living Lab projects launched in 2005 were aimed at innovation in ICT and experimentation (Ballon et al., 2005). Early definitions of LLs hence focus on the organisation of innovation processes: “a research methodology for sensing, prototyping, validating and refining complex solutions in multiple and evolving real-life contexts” (p. 13; Eriksson et al., 2005) and “Experimentation environment in which technology is given shape in real-life contexts and in which (end) users are considered ‘co-producers’” (p. 3; Ballon et al., 2005). Since the initiation of LLs, the application has extended and the focus broadened to, for example, entrepreneurial practices (McPhee, 2014) and smart city development (Khomsi, 2016).

The fundamental difference between LLs and other types of internships lies in the fact that LLs constitute a triangular co-creation relationship between academia, students and practice (i.e. firms and organisations). LLs aim to establish embeddedness of student learning in both practice and academia. Indeed, the goal of the LLs is to not only advance practise or professional skills but also academic knowledge and student learning. However, student learning in LLs has not been tested yet. This new approach differs from previous internships, which have a strong focus on developing practical and personal skills for students and are generally focussed on creating value for organisations and students, whereas LLs aim to generate valuable insights for all involved parties, including academia, with the ultimate aim of contributing to societal challenges (Marsh, 2016; Holyoak, 2013). LLs thus go beyond the sole purpose of creating value for a single participant in the LL project, which is usually the employer. Instead, they are designed with the idea of co-creating value in an interactive and engaging way in mind (Franz, 2015) for all parties involved.

Living Labs from an educational perspective

Since the initiation of LLs, the application has extended and the focus broadened to, for example, entrepreneurial practices (McPhee, 2014), smart city development (Khomsi, 2016) and as mentioned above, also education (Bourgault, 2012). Some researchers made a first attempt to define LLs from an educational perspective. For example, Hawk et al. (2012) define LLs as an: “open innovation ecosystem serving to provide opportunities for local stakeholders to practice research and experiment with meaningful improvement for cities and other organisations” (p. 225). Here, local stakeholders who practise research can be classified as students. Karagiannidis (2008) define LLs as “regional innovation environments focussing on user communities embedded within real life’’ (p. 687). They suggest that LLs offer an effective framework for collaboration between experts and research but do not extend the definition of research to students specifically.

In addition to limited background on LLs definitions in the educational sphere, the application in an educational context is also not uniform. Whilst LLs inherently are based on co-creation between students, academia and practice, the way they are operationalised take different forms. One studied operationalisation method of LLs is building-based learning (BBL), in which business schools are used as LLs (Sroufe, 2020). In this setting, students collaborate with businesses and academic personnel to conduct research and put their findings into practice by implementing their solutions that make the buildings on campus more sustainable (Sroufe, 2020). In this setting, the campus is the LL and students work on an issue posed by the university with the help of professionals. Another setting is described by Gómez Zermeño and Alemán de la Garza (2020), where universities have laboratories in which students, academia and societal partners collaborate in social innovations quests. This setting is embedded in the concept of co-creation with society by involving citizen participants and is mostly applied to create social innovation.

Student learning in Living Labs

Following the definition of LL's as a triangular co-creation between academia, students and practice, we argue that the skills students develop in an LL's go beyond the ones of traditional internships. Specifically, we propose that students improve their personal, professional and academic skills. We define personal skills as being able to communicate well, including presenting and giving and receiving feedback, and as being able to work in a team and to build and keep partnerships and a network. Professional skills are here defined as communicating and collaborating with non-academic stakeholders, researchers and fellow students and as putting academic ideas into practice. Academic skills are related to research, analytical, critical thinking and complex problem analysis skills. Research examining traditional internships has found that students particularly develop in their professional and personal skill set, namely in the cognitive, intra- and interpersonal skills domains (Pietro Di, 2022). As students face similar professional and personal challenges in an LL project as in traditional internships, students in an LL are therefore expected to develop their professional and personal skills. As conducting research is a crucial part of the concept of LL's, we further expect students to improve their academic skills.

Interestingly, little is yet reported on the benefits that are specifically related to student-learning in LLs. One of the few case-studies that exist, Sroufe (2020) acknowledges the potential for student personal skill development in several areas such as teamwork, communication and problem solving, but these skills are not yet measured nor reported on. In a reflection by Gómez Zermeño and Alemán de la Garza (2020) students indicated that their personal skills, namely, problem-solving-, brainstorming- and entrepreneurship skills improved after being engaged in an LL project. This reflection is unfortunately not benchmarked. Hence, more research on students' skills development is needed to better understand the benefits of integrating LL's into the academic curriculum. To the best of our knowledge, this is the first paper that systematically examines students' development in personal, professional and academic skills before and after participating in an LL project. Such exploration of this skills set is important to legitimise the LLs as a method of education, and justify its increasingly common place in the academic curriculum where the research internship is said to contribute to the overall learning experience and preparation of students for future careers. In this paper we are, therefore, specifically interested in the experience of students with regard to the development of their academic, professional and personal skills as intended learning outcomes of LL participation. By focussing on student’ experience and learning in a systematic way, we aim at filling the research gap on how the LLs contribute to academic and professional outcomes for students.

Methodology

To investigate the outcomes of the LLs for students, this study applies an evaluation research approach (Peischl, 1995). Evaluation research is “the systematic assessment of the operation and/or the outcomes of a program or policy, compared to a set of explicit or implicit standards, as a means of contributing to the improvement of the program or policy’’ (Weiss, 1998, p. 4). One way by which evaluation research determines the effectiveness of a programme is benchmarking. In organisational sciences, benchmarking “represents a structured, proactive change effort designed to help achieve high performance through comparative assessment; it is a process that establishes an external standard to which internal operations can be compared’’ (Jurow, 1993, p. 120). This indicates that comparative data is retrieved over a given time period to investigate the effect of a program on the unit of observation.

As this research paper is concerned with the evaluation of an LL programme in relation to the student experience, evaluation research benchmarking is regarded as an appropriate method. In the case-study, we applied benchmarking by conducting a pre-questionnaire on self-reported confidence levels in three areas; namely personal, professional and academic skills. These three categories emerged based on the overall programme learning outcomes of the study programme and the subsequent course specific learning outcomes of the Living Labs, which were then categorised in the three areas by the researchers. After engagement in the LLs, comparative self-reported post-questionnaires were subsequently used and evaluated against the benchmark to investigate the perception of the LLs. Even though self-reported data is said to be more subjective than empirically standardised research, it provides researchers with large bandwidth and constitutes a practical source of information befitting the aim of this evaluation study, namely to investigate students' subjective perspectives on the LLs (Gonyea, 2005; Astin, 1993).

Data collection and sample

Data was collected through self-administered questionnaires which were distributed to second year students of the bachelor programme Global Responsibility and Leadership of the University of Groningen in the Netherlands. Students in this programme follow an interdisciplinary liberal arts and sciences curriculum in which the LL is a mandatory 10EC course which runs for 20 weeks in the second half of their second year. Students are allocated to research questions of host organisations on the basis of their preferences and work in groups of 2–4 throughout the entire LL, supervised by at least one scientific staff member from the degree programme and at least one supervisor form the host organisation. The host organisations span a wide range of stakeholders, as illustrated in Table 1 (see below).

Participants in this study received the online questionnaire via the course learning platform, accompanied with an introductory statement outlining the purpose of the questionnaire, confidentiality and anonymity. The questionnaire was sent before the start of the LL to establish the benchmark, and once more after completion of the LL. The questionnaire was conducted twice in total, with two different cohorts of students, one in 2020 and one in 2021. The results from both cohorts were merged for the purpose of this study.

A total of 54 respondents filled in the pre-LL questionnaire and 38 the post-LL questionnaire. After deleting non-responses, the final sample consisted of 35 respondents and 18 respondents respectively. Responses from students who did not answer all items related to one construct or more, were left out. For example, if all questions related to personal skills were blank, the response was omitted. Demographic characteristics are outlined in Table 2 below. 53 students provided qualitative answers (n = 35 in the pre-test and n = 17 in the post-test). As the Global Responsibility and Leadership programme was only established in 2018 and enrols a limited number of students due to its small-scale teaching pedagogy, the sample for this study constitutes relatively few students. Over the two years in which the questionnaire was administered, a total of 76 students were enrolled (23 in the first cohort, 53 in the second cohort) and hence the population of this study is limited. The main focus of analysis is therefore on the qualitative outcomes of the self-reported questionnaire, while the quantitative analysis serves as a secondary measure.

Measures

The self-reported questionnaire started with an open question about the students' general expectations of the LL before going into the specified topics (the starting question was: “Please describe below what your expectations are of the Living Lab:”). Next, students were asked to rate their confidence levels with regard to three main measures: academic skills, professional skills and personal skills. Questions were measured on a 7-point Likert scale, ranging from 1 = extremely bad to 7 = extremely good. A 7-point Likert scale was applied as it has been found that the chance of interpolation on a 7-point Likert scale is lower than on a 5-point Likert scale (i.e. choosing the middle value; Finstad, 2010). Hence seven-point Likert scales are considered more accurate in reflecting true evaluations of one's confidence levels with regard to the three skill types. We additionally included open questions for all three measures to gain further insight in their reasoning (“In what ways has the Living Lab contributed to the development of your academic/professional/personal skills?”, and “In what ways has the current COVID-19 pandemic influenced your Living Lab experience and developments, both positively and negatively?”).

During the pre-LL questionnaire students were asked to indicate how they would rate their current skills by using the same Likert scale. During the post-LL questionnaire students were asked to evaluate the same skills to investigate whether the LL had impacted their perception of these skills.

Personal skills were measured by using seven items, asking respondents how they would currently rate the following skills: communication, team working, networking, partnership building, ability to engage with multiple stakeholders, presenting and giving and receiving feedback.

Professional skills were measured using five items, asking respondents how they would rate their current ability with regard to: communicating with non-academic stakeholders, collaborating with academic researchers, collaborating with fellow students, putting academic ideas into practice and presenting themselves to non-academic stakeholders.

Academic skills were measured using six items, asking respondents how they would rate their current academic skills. This was asked in relation to: research skills (creating research proposals, collecting data and analysing data), analytical skills, ability to think critically, ability to investigate complex problems, ability to write an academic report, ability to formulate ideas into research questions and hypotheses.

First, to check for internal consistency in the applied measures, Cronbach alpha was calculated. All items exceed the benchmark of 0.7. For the pre-survey all scales showed high internal consistency: personal skills α = 0.92 (M = 2.58, SD = 0.80) academic skills α = 0.91 (M = 2.31, SD = 0.62, professional skills = 0.91 (M = 2.59, SD = 0.84). The post-LL survey indicated high internal consistency as well: personal skills α = 0.96 (M = 2,48, SD = 0.93), academic skills α = 0.94 (M = 2.49, SD = 0.62), professional skills = 0.91 (M = 2.42, SD = 0.66). This indicates that we can combine the items underlying the constructs of the pre-questionnaire which is the base for the benchmark as well as the post-questionnaire which was used to gain insight in the perceptions after completing the LLs.

Results

Below, we present and describe the findings per skill-set. In addition, Figure 1 provides a visual overview of how each skill was ranked in the pre- and post-questionnaire.

Personal skills

Pre-questionnaire

In the pre-questionnaire, students rated their personal skills the second highest (M = 2.58, SD = 0.93) in comparison with academic and professional skills. Interestingly though, only two students focussed on personal development when asked about expectations in the open starting question of the questionnaire. Here, one student expected to “improve planning skills (dividing work over six months with two people), meet new people with similar interests who are further in their career’’ (S 9).

Post-questionnaire

In the post-questionnaire, students continued to rate their personal skills the second highest and additionally reported a decrease of their personal skills (M = 2.48, SD = 0.93). Most students mentioned that teamwork and collaboration was challenging, but that they felt it improved anyway: “I believe this project has shown me another side of teamwork I had never encountered before and therefore brought me out of my comfort zone in some settings” (S 53). Besides the challenges of collaboration itself, students also found it sometimes challenging to balance different interests and personalities: “challenging teamwork has at least the advantage that you learn from it and learn to work together with all different kinds of characters” (S 52). And: “It can be hard to work on a project where multiple stakeholders are involved, as they have different opinions on the project” (S 42). Lastly, students indicated the LL had helped them to develop their networking skills, as is illustrated in the following quotations: “Cooperating with my fellow students has led to the development of my communication and networking skills”; “My networking skills have gotten a good boost” (S 38).

Professional skills

Pre-questionnaire

Visual inspection of the means reveals that before starting with the LL, students overall evaluated their current professional skills highest (M = 2.59, SD = 0.84). From the first open question about expectations of the LL, it could be observed that students mainly expected to gain practical experience and knowledge of working in a “real-life” organisation. For example, one student replied to the question about expectations that they hoped to gain “Practical experience, connecting themes from class to real-life applications …” (S 7). Others made reference to specifically working with an organisation, as illustrated by the following two quotations: “To get a taste of how it is to work with a real organisation’’ as well as: “Exciting experience to get an idea of the real working-life and environment’’ (S 38).

Post-questionnaire

The post-LL evaluation indicates a slight decrease in the confidence students have with respect to their professional skills (pre-questionnaire: M = 2.59, SD = 0.84, post-questionnaire: M = 2.42, SD = 0.66). Mostly, students described their experience with regard to the importance of communication: “It has made me realise how much work it is to communicate everything and keeping people up to date” (S 39). And a number of the students made specific reference to the importance of communication in a predominantly online environment:

During these corona times, communication was harder between me and my partner, which had an impact on both of our motivation. Additionally, there was hardly any communication between us and the host organisation, which also contributed to the loss of motivation. Therefore, it has taught me to keep up communication at all costs, even if you have the feeling that you would not need to speak to the supervisor, small updates can boost motivation already. (S 42).

Students additionally indicated that it was not only the communication with the organisations that was harder than expected, but also that at times their collaboration with peers proved to be more challenging than expected:

It can also be quite difficult working together with your colleague student. Of course we work together in this study a lot in different courses. However, for such a long period of time it was definitely difficult to keep spirit and motivation high (S 43).

Academic skills

Pre-questionnaire

In the closed questions, students evaluated their current academic skills the lowest (M = 2.30, SD = 0.62). In the open questions, students indicated that they hoped that the LL would enable them to link theoretical knowledge from class with real life situations: “link academic work with the ‘real’ life and especially public administration’’ (S 5), “connecting themes from class to real-life applications'’ (S 7) and “applying theoretical knowledge to real world problems” (S 9). Expectations regarding specific research skills were not mentioned frequently, only one student expected that the LLs would be helpful to develop research skills: “to learn how to research and solve real life problems independently’’ (S 10).

Post-questionnaire

After completion of the LL, students scored their academic skills higher than the other skills (M = 2.49, SD = 0.63). This is the category for which students perceived the biggest improvement (from M = 2.30 to M = 2.49) and which is now more in line with the other two skill categories.

In the open question about academic skills, students reported that they especially improved specific academic skills and felt more confident in applying certain research techniques such as conducting interviews, coding and report writing, or conducting questionnaires: “The Living Lab has contributed to the advancement of my academic writing skills. Also, conducting the survey and working with real data was a very interesting and exciting thing to do!” (S 38).

Additionally, students indicated they did not only learn how to apply a certain technique but also how to apply it for different stakeholders: “I have learned not only how to interview people with political functions but also to adapt the interview guide in a way that it effectively helps asking the right questions needed for the research” (S 36).

Influence of COVID-19

The pandemic often came back in the post-questionnaires. Students mentioned that it influenced communication with LL partners and that it was hard to ensure alignment with students and LL partners: “During these corona times, communication was harder between me and my partner, which had an impact on both of our motivation” (S 42). In addition, students mentioned that it was hard for them to stay motivated throughout the mostly online project and that the pandemic influenced their commitment and mental health: “Due to COVID-19, we were not able to meet any host supervisors, participants or other stakeholders in real life until now. However, also by online communication I was able to develop professional communication skills” (S 49).

At the same time, some things were also perceived to be easier due to the pandemic, such as organising meetings and not losing time on travelling to the host location. Additionally, the data collection sometimes was easier. For instance, one student mentioned that they changed their research from qualitative to quantitative research and were able to gather more data online. Nevertheless, the online environment in the end negatively influenced the fun of the project: “It was easier to do all the interviews quickly, as everything could be done by phone. The fun of the project was kind of taken away” (S 36). Another student indicated the same, while explaining that COVID-19 resulted in having to change from a field study to a questionnaire:

The Covid-19 crisis changed the way we collected our data. Therefore, it was not as interactive and lively as it would have been without Covid-19. I was really looking forward to collecting the data on the street. However, the digital form of data collection has made it easier and more numerous. We received far more replies than we would have gotten if we would have collected the data on the street’ (S 38).

Discussion

Our study is one of the first studies that aims to better understand LL's from the student perspective, by investigating their personal, professional and academic skills. Students in a research-based Living Lab experience seem to train their transversal personal and professional skills, while also improving their academic skills. This outcome is of importance to legitimise the participation of students in LLs as otherwise it will remain a concept of which the outcome to its participants (i.e. students) will remain unknown. By being one of the first studies to investigate outcomes accrued to students, this study enriches literature and also aims to make LLs a more legitimised concept in academia. In addition, it became clear that the COVID-19 pandemic highly influenced their overall experience of participating in a research internship. This is an important contribution to literature as well as to practise as previous research has not examined students' perspective on their LL experience.

Throughout the LL experience, students remained confident with regard to their personal skills, although a small decline was reported (pre-mean: 2.58; post-mean: 2.48). This development mainly occurred due to the fact that students perceived the LL as challenging, confirming that for growth to occur, students must step outside of their comfort zone into their growth zone (Blekkingh, 2015). This growth may also be stimulated further by the challenges posed by the COVID-19 pandemic which may have pushed students out of their comfort where they would do things on automatic pilot and instead push them towards a creation motive. In this creation motive they are encouraged by their environment to develop new personal skills that lead them towards their goal (Blekkingh, 2015). In relation to the LLs, this may indicate that students were pushed to develop their personal skills as the situation required other ways of networking, team working and communicating.

Contrary to our expectations, students rated their professional skills before conducting the LL higher than after completing the research internship (pre-mean: 2.59; post-mean: 2.42). One explanation could be that students have become more aware of their skills and potential room for improvement during the course of the research internship as they realise how a professional setting is different from what they were used to at university. Hence, participating in an LL may offer a unique opportunity for students to become aware of the professional skills that are needed later on in the professional field and that are traditionally not taught at university. Another explanation might be related to the fact that the LLs took place during the COVID-19 pandemic. As such, not only the LL itself was a new concept for students, the way in which the LL was conducted was new as well. Due to this, students' professional skills were required to develop in a different setting than anticipated (virtual versus physical). While students expected an onsite LL hosted by a local host organisation, most communication and collaboration with companies was conducted online. This online setting required a different professional skill set than they were used to. Additionally, the online setting might also have led to challenges for the organisations to optimally collaborate, connect and communicate with students. The COVID-19 pandemic has led to an accelerated and unplanned digitisation process in many companies which resulted in major transformations regarding companies' communication and collaboration tools (Almeida et al., 2020). While these changes are already challenging internally, it also spills over to external collaboration and communication with external partners like students, and scientific staff members, participating in the LL. Hence, this technological disruption and changed professional setting may have posed a challenge on the development of professional skills of students participating in the LL.

Finally, students reported their academic skills as the lowest of all three categories before completing the LL. This is an interesting finding, as the LL is a research concept and students have been trained in academic courses and research techniques before starting the LL. Of course, the LL is the first practical application of the student's academic skills so a potential explanation might be that they report them as rather low because they have never applied their research skills in an LL setting. Additionally, one could argue that, as the traditional teaching of universities is focused on academic skill development, students have the most realistic idea of the status quo of their academic skills. Furthermore, while academic skills were reported as the least developed before conducting the LL, they did in fact develop the most as compared to the other skill-sets after (pre-mean: 2.30; post-mean: 2.49). This indicates that there is a lot to gain with regard to academic skill development when applied outside of the academic curriculum and in practical environments like an LL. They allow students to successfully link theoretical knowledge from academic teaching with practical research and implement in-class taught methodologies and techniques like interviewing, coding and reporting. This emphasises the importance of the research-based element of the LLs as compared to traditional internships.

Research about traditional internships mainly focuses on the outcome for students in relation to professional development (Beck and Halim, 2008) and what the implications are for their employability (Gault et al., 2000, 2010; Silva et al., 2016). Moreover, current research on skills acquired during internships can be divided in three skill domains namely cognitive domain, intra- and interpersonal domain (Pietro Di, 2022). The development of these skills in these domains have been reported to bring similar results as we have found in our professional and personal skill set, such as communication, cooperation and planning skills (Pietro Di, 2022). The additional value from co-creation in LLs is the academic skill set gained by students. These skills have not been recognised by traditional internships. As shown in Figure 1, LLs contribute in a positive way to students' perception of their academic skills and which in fact increased mostly in our research. This indicates that there is a huge potential for students to develop their academic skills in LLs, when such academic components are embedded in the LLs. Our study therefore contributes to and extends previous research on Living Labs by showing how academic student learning can be improved via transdisciplinary research internship. This is not only of importance to research but also for practice. Before a new practice can be legitimised it is important to know its potential benefits. The current study has confirmed that participation in LLs can greatly contribute to students' development and therefore can legitimise the concept of LLs in academia.

Conclusion

Our study indicates that students who conducted an LL project report that they actively practised transversal personal and professional skills, and in addition improved their academic skills. As LLs thus seem to contribute transversal skills as well as to the academic development of students, we can legitimise their increasingly common place in higher education curricula. LLs are not only beneficial for stakeholders and society as they generate new insights into societal questions, they are also of added value to students who actively collaborate with the external organisations and researchers. The LLs can thus be seen as a method of education which contributes to students' preparation for future careers, which is one of the main tasks of higher education institutions. The study has also shown the importance of the setting of the LLs. The LLs were originally planned to be onsite with a local host organisation, but due to the COVID-19 pandemic the setting changed to an online environment which resulted in challenges, yet also created opportunities with respect to research design and time investment. Given these uncovered aspects, future LLs can be anticipated on more hybrid or even further online collaborations, which also opens the possibility of collaboration with international organisations located at different parts of the world. In this case, extra attention will need to be devoted to aligning expectations among the different stakeholders and specifically focussing on ways to ensure good communication.

Limitations and future research

This study has a number of limitations. First, given the small-scale teaching setting in which the research was conducted, the number of participants who carried out the pre and post questionnaires is relatively low, making it difficult to generalise the findings to other contexts. Future research should investigate the benefits of LLs that accrue to students based on a larger sample size. Research could also employ a longitudinal research design and ask students after they started working to reflect on their experiences in their LL's and the relevance for their current jobs. Second, due to the COVID-19 pandemic, we could not compare the data to a condition with students who carried out the LLs completely onsite, which would allow us to make firmer conclusions about the potential for online LLs.

Further, given that the current LL was subject to the COVID-19 pandemic, it would be interesting to see future research addressing the benefits for students in a setting which is not subject to a pandemic. This could potentially lead to interesting findings as the current study found that communication and research settings had to be adjusted. It would be interesting to compare the challenges students face in a non-pandemic setting as these challenges could potentially lead to different skill developments. For example, personal skills, such as networking, partnership building and presenting, might be challenged and thus developed more as these are more prevalent in face to face settings. It should also be noted that the unprecedented shift to virtual LL collaborations might still have contributed to the professionalisation of students in domains that were not measured in this study, such as hybrid and digital working and collaboration skills. As many organisations are experimenting with working from home post-pandemic, these skills could also provide a valuable basis for students later on. As such, future research might aim at investigating how such hybrid research internship collaborations help students develop various skills and in which domains, by employing scales that also measure hybrid-working skills.

Lastly, another aspect to improve the research would be to include the feedback of the host organisation and the academic partner. Currently skill development is solely investigated based on self-reported perceptions through a survey instrument. Incorporating 360° feedback from the organisation and the academic partner would allow future research to establish a broader and finer understanding of the students' development. Alternatively, skills could be assessed by using a test which measures the specific skill sets.

Figures

Figure 1

Pre and post-questionnaire

Table 1

Demographic characteristics of the sample

Age	% Of sample	Gender	% Of sample	Nationality	% Of sample
16–19	11	Male	28	Dutch	44%
20–23	78	Female	72	German	39%
24–27	11	Does not wish to disclose	0	Other*	17%
28–31	0

Note(s): * Other nationalities include Finnish, Dutch/Mexican, British, Canadian and Icelandic

Source(s): Authors' own work

Table 2

Living Lab partner overview

Organisation	Type of organisation	Year
Municipality Leeuwarden	Governmental institution	19/20
Global Center on Climate Adaptation	NGO	19/20
Amnesty International	NGO	19/20
Fries social planbureau	Research institute	19/20
House of Design	Art collective	19/20
Municipality Leeuwarden	Governmental institution	19/20
Wetterskip Fryslân	Governmental institution	19/20
Waterschap Zuiderzeeland	Governmental institution	19/20
Elodea	Advice bureau	19/20
Friesland College	Educational institution	19/20
Municipality Leeuwarden	Governmental institution	20/21
Global Center on Climate Adaptation	NGO	20/21
Amnesty International	NGO	20/21
Fries social planbureau	Research institute	20/21
House of Design	Art collective	20/21
Municipality Leeuwarden	Governmental institution	20/21
Wetterskip Fryslân	Governmental institution	20/21
Ekwadraat	Consultancy agency	20/21
Provence Fryslan	Governmental institution	20/21
8D-Games	Serious gaming company	20/21
JGM Serious experience	Serious gaming company	20/21
Rekenkamercommisie Waddeneilanden	Governmental institution	20/21
Bond Friese Vogelwachten	NGO	20/21
Alliancecpha	NGO	20/21
Solarfields	Consultancy agency	20/21
BugelHajema	Consultancy agency	20/21
Holwerd aan Zee	NGO	20/21
RCE Spark the Movement	NGO	21/22
Fries Social Planbureau	Research institute	21/22
JGM	Serious gaming company	21/22
Bond Friese Vogelwachten	NGO	21/22
Solarfields	Consultancy agency	21/22
De Afsluitdijk	Governmental institution	21/22
8D Games	Serious gaming company	21/22
Global Centre on Adaptation (1)	NGO	21/22
Global Centre on Adaptation (1)	NGO	21/22
Green Office UG	Educational institution	21/22
DIE	Research institute	21/22
VC4A	Consultancy agency	21/22
CJIB	Governmental institution	21/22
ISI	Educational institution	21/22
Arcadia	NGO	21/22
Deloitte	Consultancy agency	21/22
Province of Fryslan	Governmental institution	21/22
Municipality of Leeuwarden	Governmental institution	21/22
Municipality of Westerkwartier	Governmental institution	21/22
Municipality of Smallingerland	Governmental institution	21/22
Wetterskip Fryslân	Governmental institution	21/22
Data Fryslân	Research institute	21/22
The Alliance of child protection in humanitarian action	NGO	21/22

Source(s): Authors' own work

Ethics approval and consent to participate: The research study was approved by the Ethics Committee of the University of Groningen/Campus Fryslân. By completing the questionnaire, participants agreed to participate in the study.

Competing interests: There are no competing interests.