Living labs: concepts, tools and cases

Citation

Ballon, P. and Schuurman, D. (2015), "Living labs: concepts, tools and cases", info, Vol. 17 No. 4. https://doi.org/10.1108/info-04-2015-0024

Living labs: concepts, tools and cases

Article Type: Guest editorial From: info, Volume 17, Issue 4

Introduction

Praxis and theory

This special issue on "Living labs: concepts, tools and cases" comes 10 years after the first scientific publications that defined the notion of living labs, but more than 15 years after the appearance of the first living lab projects (Ballon et al., 2005; Eriksson et al., 2005). This five-year gap demonstrates the extent to which living labs have been a practice-driven phenomenon. Right up to this day, they represent a pragmatic approach to innovation (of information and communication technologies [ICTs] and other artefacts), characterised by a.o. experimentation in real life and active involvement of users.

While there is now a certain body of literature that attempts to clarify and analyse the concept (Følstad, 2008; Almirall et al., 2012; Leminen et al., 2012), living lab practices are still under-researched, and a theoretical and methodological gap continues to exist in terms of the restricted amount and visibility of living lab literature vis-à-vis the rather large community of practice (Schuurman, 2015). The present special issue aims to assist in filling that gap.

This does not mean that the development of living labs has not been informed by scholarly literature previously (Ballon, 2015). Cornerstones include von Hippel’s (1988) work on user-driven innovation because of its emphasis on the ability of so-called lead users, rather than manufacturers, to create (mainly ICT) innovations. Another cornerstone is Silverstone’s (1993) theory on the domestication of ICTs that frames technology adoption as an ongoing struggle between users and technology where the user attempts to take control of the technological artefact and the technology comes to be fitted to users’ daily routines. It has been said that, in living labs, von Hippel’s concept of user-driven design and Silverstone’s insights into the appropriation of technologies are coupled dynamically through experimentation (Frissen and Van Lieshout, 2006).

The concept of stigmergy, which refers to addressing complex problems by collective, yet uncoordinated, actions and interactions of communities of individuals, has gradually become the third foundational element, as social media have provided online platforms for stigmergic behaviour, which has subsequently been linked to the "spontaneous" emergence of innovations (Pallot et al., 2010; Kiemen and Ballon, 2012). A fourth cornerstone is the literature on open and business model innovation, which argues that today’s fast-paced innovation landscape requires collaboration between multiple business and institutional stakeholders, and that the business should use these joint innovation endeavours to find the right "business architecture" (Chesbrough, 2003; Mitchell and Coles, 2003).

Definitions

Combining these strands, living labs typically refer to co-creation and appropriation of innovations by users, often in a (online or offline) community setting, and also involving business stakeholders. Over the years, multiple definitions of living labs have been proposed.

Initial definitions included "a research methodology for sensing, prototyping, validating and refining complex solutions in multiple and evolving real-life contexts" (Eriksson et al., 2005) and an "experimentation environment in which technology is given shape in real-life contexts and in which (end) users are considered ‘co-producers’" (Ballon et al., 2005). Subsequently, these aspects were combined so that living labs were conceptualised as both a methodology and a milieu for organising user participation in innovation processes (Bergvall-Kåreborn et al., 2009).

Almirall and Wareham (2011) emphasise the (productive) tension between the characteristics of the real-life environment in which experiments and observations take place, and the research space that is grafted onto it. They see living labs as "semi-partitioned spaces in the form of innovation arenas integrated in real-life environments but separated by means of an innovation project structure that cultivate user-led insights" whereby the lab is able to "surface tacit, experiential and domain-based knowledge such that it can be further codified and communicated". According to whether the balance is tilted in favour of the real-life environment, in which events happen spontaneously, users have a lot of autonomy and the full context that is observable, or a more structured, formalised and controlled research environment, Følstad (2008) identifies two general living lab "archetypes": living labs supporting context research and co-creation and living labs as testbeds (cfr. infra).

Other authors have emphasised the institutional dimension of living labs, as entities which are part of an "innovation system". For instance, Fulgencio et al. (2012) define a living lab as a "human-technology interaction innovation entity utilizing a mix of methods, tools and principles drawn from known disciplines (design, science, ICT, etc.) and set in a real environment and on a local/societal scale". This has added to the proliferation of diverging uses of the term "living labs". Dutilleul et al. (2010) even mention five different meanings for which the concept is used:

1. an innovation system;

2. a real-life social setting;

3. an approach for user involvement in innovation;

4. an organisation facilitating living lab approaches; and

5. the European living lab "movement".

And Westerlund and Leminen (2014) identify no less than eight different research avenues researchers have taken to conceptualise living labs.

Therefore, the most effective way of characterising living labs is probably to analyse actual experiences, and how they evolved over time. Such analyses should not limit themselves to activities carrying the name of living labs, but they should also look at similar concepts and how they are put into practice. At least three important predecessors for the living labs movement as we know it today can be discerned (Schuurman, 2015). The cooperative design movement or the Scandinavian tradition of user involvement in information technology (IT) design processes (Ehn, 1989) can be traced back as far as the 1970s; in the 1980s, there were the European "social experiments" with IT (Oestmann and Dymond, 2001; Qvortrup, 1987); and from the 1990s onwards, "Digital City" projects started to blossom (Paskaleva, 2011). Then, towards the end of the 1990s, the proper living lab concept came into use, first in the USA setting (which mostly conformed to Følstad’s archetype of living labs as testbeds), but soon primarily in the European setting (which were mostly in line with Følstad’s archetype of living labs as means to research context and to enable co-creation).

Predecessors of living labs

Cooperative design: the Scandinavian tradition of user involvement

The tradition of cooperative design can be traced back to the 1970s, when research projects on user participation in systems development took place all over Scandinavia (Bødker, 1996). These early initiatives were supported by trade unions and involved workers in the design of IT applications in the workplace (Bjerknes et al., 1987). The "collective resource approach" that was used involved collaboration between the organisation’s workers and researchers, so that researchers were able to gather data and workers were empowered to influence the implementation of IT systems in their daily work context (Fowles, 2000). Key within cooperative design was the collective build-up of resources and knowledge. One of the aspects mentioned in Bødker and Grønbæk (1991) was the facilitation of trial use situations as part of the design process, so as to stage users’ hands-on experience with future applications.

This tradition has been extended and modified by other movements, which usually had in common a "social shaping of technology" focus, trying to steer and optimise innovations based on user needs and wants, and using sometimes a "socio-deterministic" view on innovation (Gasson, 2003).

The subsequent North American participatory design movement was one of these movements that partly built upon the Scandinavian initiatives in cooperative design (Greenbaum and Kyng, 1991). Participatory design was debuted by computer engineers and developed into a set of theories, practices and studies related to end users as full participants in activities leading to software and hardware computer products and computer-based activities (Muller and Kuhn, 1993; Bødker et al., 2004).

Because IT later became prevalent not only at work but also at home, in school and even while "on the move", participatory design has struggled somewhat to embrace the fact that much technology development no longer happens as design of isolated systems in well-defined communities of work (Gasson, 2003). As a result, the so-called user-centred design-approach (UCD) came to the fore. UCD can be characterised as a multi-stage, problem-solving process that not only requires designers to analyse and foresee how users are likely to use a product but also to test the validity of their assumptions with regards to user behaviour in real-world tests with actual users. This design methodology looks at the design of a product or service as a process in which the needs, wants and limitations of users are given extensive attention at each stage of the design process, including testing in field studies. The three principles of UCD became an early focus on users and tasks, an empirical measurement of product usage in field trials and iterative design (Gould and Lewis, 1985).

Commonly used methods include ethnographic studies, contextual inquiry, prototype testing, usability testing and generative methods. Contextual inquiry, as a part of the "contextual design" methodology, also evolved out of the cooperative and participatory design traditions, and focuses on the design of products or services within their actual usage context. Ethnographic techniques, such as observing and interviewing users during their normal daily life routines, were prevalent in these traditions (Wixon et al., 1990; Holtzblatt and Beyer, 1995).

Social experiments: field trials with IT in Europe

A second line of proto-living labs started in the 1980s when, all over Europe, various social experiments with IT were started. Social experiments originated in the field of psychology and refer to experiments taking place outside of laboratories and therefore with less physical isolation of subjects and materials, less procedural standardisation and longer-lasting treatments when compared to experiments in laboratory settings. They are usually designed to test an intervention or treatment whereby the test is seen as the evaluation of a global package of many components, rather than as a unidimensional causal construct (Cook and Shadish, 1994). Under an impulse of the European Commission and the FAST-programme, researchers started to use social experiments as a test and implementation methodology in the context of the developing field of ICT in the 1980s. Qvortrup (1987) defines these social experiments as specific forms of ICT implementation in which the primary goal is to establish new forms of organisation using IT, with a view to influencing the society at large. Some examples of European social experiments with ICT include field trials with Interactive Videotex (France, Germany, the UK and Denmark), Broadband Cable and Computer Conference Systems (Ancelin, 1987) and the implementation of so-called telecentres for rural ICT-development in various European countries (Oestmann and Dymond, 2001).

The appeal of social experiments, together with the European policy and funding support, resulted in a strong growth of initiatives being awarded the "social experiment" label without much further consideration (Qvortrup, 1987). In an attempt to delineate the concept somewhat, Ancelin (1987) added that social experimentation is an open-ended process in which there are several degrees of freedom and in which a mutual learning process for the promoter and for the user is facilitated. The key to the mutual learning process involving supplier and user is their consequent mutual adaptation. The responses of the users provide indications for the development of new technological applications, and these new applications, in turn, influence the users’ behaviour, giving rise to "social inventions" (Hartley,1987). This indicated a shift towards a "mutual shaping" view on ICT development instead of the dominant "social shaping" view of the cooperative design approach of the 1970s. The role of the end-user was considered important in these initiatives, but there was no conclusive stance about the nature of this role: users could function as mere testers or respondents, but could also be involved on equal footing (co-creators) or as innovators themselves.

Digital city initiatives

During the 1990s, the "digital city" concept took hold in Europe and elsewhere, referring to a number of digital initiatives undertaken by cities, especially related to digital representations of the city and the provision of Internet access for citizens. Digital cities can be seen as the counterpart of the telecentres that were set up for social experiments, which were often aimed at underprivileged, rural and remote areas. The digital city network infrastructure and the platforms used to disclose the large amount of digital information were of central importance within the digital city-discourse, causing the concept to carry a quite heavy technology-deterministic connotation (Ishida, 2000; Mechant et al., 2012).

Some examples of these early digital city-initiatives could be found in The Netherlands (Digital City Amsterdam, founded in 1994) and in Finland (Virtual Helsinki, founded in 1996). But also in the USA, AOL started a regional information service called "digital city" for several tens of major US cities, while, in Japan, the Digital City Kyoto Project was launched in 1998 to create a "social information infrastructure towards the 21st century". Many initiatives aimed to bring together digital information about cities and citizens and make it accessible in a public virtual space where citizens could consult this information but also interact with it and (increasingly) with each other (Loukis et al., 2011). This last aspect, interaction with each other, is also referred to in other definitions of digital cities that stress the connectivity between various stakeholders in a city context (Ergazakis et al., 2011; Middleton and Bryne, 2011).

Current smart city initiatives can be seen as either extensions of the large-scale technology infrastructure and platform initiatives, now also encompassing mobile technologies and the so-called Internet of things or as attempts to strengthen the cooperative and participatory strand, both present in the original Digital City concept (Breuer et al., 2014).

The digital city initiatives were, more than the previous two predecessors, explicitly multi-stakeholder, as they connect citizens (users), policymakers (public organisations) and private organisations (businesses) on a large scale. Thematically, the initiatives covered a broad range, albeit always with a link to city life. In terms of user involvement, the user was seen as a potentially innovative agent, with the technical infrastructure as a trigger for this creativity.

From home labs to living labs

The MIT vision

While there had been "accidental mentions" of the term living lab before – mostly as wordplays used to indicate the "in situ" nature of different types of biology, medicine and other research (Schuurman, 2015) – the actual birth of the concept is ascribed to MIT’s Prof Dr Mitchell, who used it to refer to a purpose-built lab where the routine activities and interactions of everyday home life can be observed, recorded for later analysis and experimentally manipulated, and where volunteer research participants individually live in, treating it as a temporary home (Eriksson et al., 2005). These labs had an initial focus on testing and adapting new technologies based on their fit with the daily home environment. This is not to say living labs were originally the same as so-called "smart home" projects, whereas the latter type of projects acts mostly as a showcase of the "home of the future", a living lab had as primary goal to research how ubiquitous computing technology can be designed to fit the daily lives of the living lab inhabitants (Markopoulos and Rauterberg, 2000). In these home labs, the user is mostly involved as a passive study object within a testbed setup.

In the MIT PlaceLab, a 1,000-square foot "living laboratory", with all facilities of a regular home, users are observed, logged and tracked with all sorts of devices, allowing to record their habits, activities and routines (Intille et al., 2006). Strong importance is placed on the technical infrastructure allowing the data gathering. In terms of methodology, this makes the living lab an extension of usability tests, aiming to get more accurate and realistic user information by having more long-term data, allowing observation of everyday activities and capturing tacit knowledge (Pierson and Lievens, 2005 ). In Europe, there are also some well-known examples of living labs similar to the MIT set-ups, i.e. the Philips Homelab in The Netherlands (which opened already in 2002) and the Fraunhofer InHaus in Germany. The StudioHome in the ID-StudioLab of the Delft University of Technology even moves the furniture and changes the interior to match the outlook of the individual user’s own home (Pasman et al., 2005).

European living labs

When living labs appeared in Europe in the first years of the 2000s, it became clear that the living lab notion that was to be predominant in Europe, and that built upon the earlier experiences with participatory design, social experiments and digital cities from the 1970s up until the 1990s, offered a fundamental reinterpretation of the US-originated home labs. A major divergence was that the user was to be studied in his or her everyday habitat instead of recreating a natural context in a laboratory setting (Niitamo et al., 2006). In terms of a methodological set-up, this implied bringing the testing facilities to the users instead of the other way round.

It can be argued that the European understanding of living labs as a set of methods and a milieu for leveraging user-technology reactions and interactions in the innovation process (cfr. supra) combined five basic elements that reflect a number of aims and characteristics of both the home labs and the living lab predecessors (Følstad, 2008; Schuurman et al., 2012). These elements are active user involvement (i.e. empowering end users to thoroughly impact the innovation process); real-life setting (i.e. testing and experimenting with new artefacts "in the wild"); multi-stakeholder participation (i.e. the involvement of technology providers, service providers, relevant institutional actors, professional or residential end users); a multi-method approach (i.e. the combination of methods and tools originating from a.o. ethnography, psychology, sociology, strategic management, engineering); and co-creation (i.e. iterations of design cycles with different sets of stakeholders). This combination of elements from previous experiences is summarised in Table I.

In 2006, the European living labs movement gained momentum through a set of European Union (EU) policy measures (Dutilleul et al., 2010), including the funding of "Corelabs" and "Clocks", two projects aimed at promoting and coordinating a common European innovation system for ICTs based on living labs and at hosting and promoting the establishment of the European Network of Living Labs (ENoLL). This initially consisted of 19 living labs located across the EU. Living labs were also explicitly supported in the "Strengthening innovation and investment in ICT research" pillar of i2010, the EU policy framework for the information society and media (Peltomäki, 2008).

The so-called Helsinki Manifesto (2006) described ENoLL as a platform for knowledge sharing and collaboration to foster common methodologies and tools across Europe that support, stimulate and accelerate co-creative innovation processes, relying on users involvement. The EU Commission allocated a significant budget to promote the development of ENoLL (Prime Minister’s Office, 2006). Originally, ENoLL consisted only of European initiatives that were admitted to the network after a benchmarking exercise, but it has grown to include members from a.o. Brazil, Colombia, Canada, Mexico, Australia, China and Egypt. The criteria used during the evaluation process for prospective members refer to the organisational set-up, openness, resources, user involvement and real-life facilities and value creation potential of the initiative (Dutilleul et al., 2010).

While the number of living labs that have been evaluated positively by ENoLL since 2006 amounts to over 300, not all of these initiatives are still around. Many living labs are only established to carry out a single innovation project (Ståhlbröst, 2012), while others have found it impossible to maintain a sustainable operation while being dependent on project-funding. It is estimated that between 35 and 40 per cent of the labs who successfully applied for the ENoLL benchmarking, are currently no longer active (Schuurman, 2015).

Recent developments and outline of the special issue

Despite the sustainability challenges of a number of individual initiatives, living labs have gradually become an established part of (ICT) innovation policy. Not only is the set-up and involvement of living labs by now requested in innovation programmes at the EU level in regular fashion, but they also appear in innovation policy measures and programmes at the level of individual member states and regions, including Finland, The Netherlands, all three official regions in Belgium, Spain, Slovenia and others. Also, the World Bank has started to promote living labs worldwide as an innovation policy tool for developing countries (Garcia et al., 2010; Hirvikovski, 2012; World Bank & ENoLL, 2014).

Despite the common frames of reference mentioned earlier, there is a still a large heterogeneity of living lab initiatives (Ballon, 2015; Schuurman, 2015). In terms of organisational set-up, living labs encompass a range of (semi-) permanent as well as temporary projects associated with academic institutes, large technology vendors, municipalities or non-profits, innovation consultants, design or marketing companies, industry clusters and so on. The number of users involved in living lab testing and experimentation ranges from a handful (e.g. in some homecare living labs that necessitate the installation of complex or costly equipment in users’ homes) to several thousands (e.g. in the case of large online or offline communities of practice involved in living lab trials). A number of living labs have turned predominantly to information retrieval and crowdsourcing online, using social media as a living lab environment. Others have extended their scope to treat entire (parts of) "smart cities" as living labs, making use of mobile, location-based social media and sensor network data. Many questions related to the nature and characteristics of living labs, their methods and effectiveness, their application domains and new avenues for research, are still open (Azzopardi and Balog, 2011; Schaffers et al., 2011; Ballon, 2015).

This special issue brings together a selection of expanded, rewritten and re-reviewed papers that were presented, in earlier form, during the annual European Network of Living Labs Research Day in Amsterdam, The Netherlands (September 2014). We especially thank Ana Garcia, the Head of Office of ENoLL, for the organisation of the event and the help in getting out the call for papers; the conference participants who provided feedback on the presented papers in relation to the Veli-Pekka Niitamo memorial award for best research contribution; and the many anonymous reviewers involved, both for the ENoLL Research Day and for this special issue.

The papers gathered here reflect some of the most salient issues related to the topic of living labs. The first two papers deal with some of the most important "traditional" challenges to living labs. The first paper, by Mastelic, Sahakian and Bonazzi, discusses ways to improve the sustainability of living labs. The paper explores how living labs might be evaluated based on criteria related to their business architecture. It finds that three elements are currently missing or under-represented in the ENoLL evaluation system, i.e. the cost structure, customer segments and the revenue stream. For the "sustainability" of a living lab, the paper argues that a strong business model is needed, based on a long-term strategy that considers funding structures, target audiences and revenue sources, all of which must be assessed not only at one moment in time, but over time, in a continuous and dynamic process involving different stakeholders.

The second paper, by Georges, Schuurman, Baccarne and Coorevits, treats the refinement of methods for active user participation of Living Labs. Based on the analysis of three living lab cases in which field trials were organised, the authors identify several factors playing a role in the engagement of users. An influential factor that emerged is the functional maturity of the innovation, i.e. the extent to which a prototype resembles the functionalities and the processes of the final, go-to-market product at the moment of the field trial. A "user engagement model for field trials" is proposed, as well as six concrete guidelines to better involve end-users in living labs.

The following two papers address case studies of specific living lab projects that expansion of knowledge related to specific application domains. The paper by Brankaert, den Ouden & Brombacher examines the case of several living lab tests related to the technological assistance of people suffering from dementia, and summarises the approach into a living lab protocol that allows for involving cognitively impaired people and their caregivers in their natural context. It also provides suggestions to overcome issues of informed consent/willingness to participate, safety and security and adequately addressing of needs for people with dementia.

The paper by Franz on living labs in urban research considers the application of living labs to large-scale social–spatial issues. It argues that living labs have the potential to be an instrument for the active inclusion of citizens in urban research projects investigating socio-spatial questions. It examines the possibilities and limitations of implementing living labs with public authorities as "producers" and citizens as "users", and it identifies current approaches and gaps in living lab design.

The final set of papers focuses on living labs at the service of specific private businesses. The paper by Lapointe and Guimont starts from the premise that many living labs aim to operate as open innovation ecosystems that include various private stakeholders, but that literature does not tell us much about their modus operandi or about the way in which they conceive open innovation. Lapointes’s paper focuses on the relationship of private stakeholders to open innovation in the context of in-situ activities and establishes a typology of businesses in relation to open innovation.

Lastly, Salminen, Rinkinen and Kahn examine how to support use of design in SMEs by developing a new design support service. The authors take two basic assumptions as a starting point:

1. using design is beneficial for small and medium enterprises’ (SMEs) business; and

2. business advisors are the best channels for reaching these SMEs.

Many countries have launched projects to address this issue and have devised solutions to support SMEs in this context. The paper analyses one such initiative, which focused on the potential of the business advisors to bridge the gap between SMEs and design service providers. Their study provides insights into the discussions on service development projects realised in the living lab environment that enhance the use of design services among SMEs. The case study and its findings presented in the paper highlight the need to focus on service users in the early phase of the service design process instead of utilising user knowledge only in the testing phase. Co-creation and co-design processes with users instead of only with assumed experts in the field provide wider possibilities and fruitful base for service development.

Together, we hope the papers gathered here provide an excellent contribution to the scholarly debate on living labs for anyone interested in user-driven and open approaches to ICT innovation.

Ballon Pieter and Schuurman Dimitri

Pieter Ballon is based at iMinds-SMIT, Vrije Universiteit Brussel, Brussels, Belgium. Dimitri Schuurman is a Senior Researcher based at iMinds-MICT, University of Ghent, Ghent, Belgium.

References

Almirall, E., Lee, M. and Wareham, J. (2012), "Mapping living labs in the landscape of innovation methodologies", Technology Innovation Management Review, Vol. 2 No. 9.

Almirall, E. and Wareham, J. (2011), "Living labs: arbiters of mid-and ground-level innovation", Technology Analysis & Strategic Management, Vol. 23 No. 1, pp. 87-102.

Ancelin, C. (1987), "Social experimentation with telematics", Social Experiments with Information Technology and the Challenges of Innovation: A Selection of Papers from the EEC Conference on Social Experiments with Information Technology, Springer, Odense, 13-15 January, p. 45

Azzopardi, L. and Balog, K. (2011), "Towards a living lab for information retrieval research and development", Multilingual and Multimodal Information Access Evaluation, Springer, Berlin Heidelberg, pp. 26-37.

Ballon, P. (2015), "Living labs", in Mansell, R. and Ang, P.H. (Eds), The International Encyclopedia of Digital Communication and Society, Wiley & Sons, Hoboken, NJ.

Ballon, P., Pierson, J. and Delaere, S. (2005), "Test and experimentation platforms for broadband innovation: examining European practice", available at: http://ssrn.com/abstract=1331557

Bergvall-Kåreborn, B., Ihlström Eriksson, C., Ståhlbröst, A. and Svensson, J. (2009), A milieu for innovation – defining living labs. In 2nd ISPIM Innovation Symposium, New York, pp. 6-9.

Bjerknes, G., Ehn, P. and Kyng, M. (1987), Computers and Democracy – A Scandinavian Challenge, Gower Publishing.

Bødker, K., Kensing, F. and Simonsen, J. (2004), Participatory IT Design: Designing for Business and Workplace Realities, MIT Press, Cambridge, MA.

Bødker, S. (1996), "Creating conditions for participation: conflicts and resources in systems development", Human-Computer Interaction, Vol. 11 No. 3, pp. 215-236.

Bødker, S. and Grønbæk, K. (1991), "Cooperative prototyping: users and designers in mutual activity", International Journal of Man-Machine Studies, Vol. 34 No. 3, pp. 453-478.

Breuer, J., Walravens, N. and Ballon, P. (2014), "Beyond defining the smart city. Meeting top-down and bottom-up approaches in the middle", TeMA Journal of Land Use, Mobility and Environment, Vol. 7 (Special Issue).

Chesbrough, H.W. (2003), Open Innovation: The New Imperative for Creating and Profiting from Technology, Harvard Business Press, Watertown.

Cook, T.D. and Shadish, W.R. (1994), "Social experiments: some developments over the past fifteen years", Annual Review of Psychology, Vol. 45 No. 1, pp. 545-580.

Dutilleul, B., Birrer, F.A. and Mensink, W. (2010), "Unpacking European living labs: analysing innovation’s social dimensions", Central European Journal of Public Policy, Vol. 4 No. 1, pp. 60-85.

Ehn, P. (1989), Work-Oriented Design of Computer Artifacts, Lawrence Erlbaum Associates, Hillsdale, NJ.

Ergazakis, E., Ergazakis, K., Askounis, D. and Charalabidis, Y. (2011), "Digital cities: towards an integrated decision support methodology", Telematics and Informatics, Vol. 28 No. 3, pp. 148-162.

Eriksson, M., Niitamo, V.P. and Kulkki, S. (2005), "State-of-the-art in utilizing living labs approach to user-centric ICT innovation – a European approach", Center for Distance-spanning Technology, Lulea University of Technology, Lulea.

Følstad, A. (2008), "Living labs for innovation and development of information and communication technology: a literature review", Electronic Journal of Virtual Organisations, Vol. 10 No. 5, pp. 99-131.

Fowles, R.A. (2000), "Symmetry in design participation in the built environment: experiences and insights from education and practice", in Collaborative Design, Springer, London, pp. 59-70.

Frissen, V. and van Lieshout, M. (2006), "ICT in everyday life", User Behavior and Technology Development, Springer, Rotterdam, pp. 253-262.

Fulgencio, H., Le Fever, H. and Katzy, B. (2012), "Living lab: innovation through pastiche", Proceedings of the eChallenges e-2012 Conference, IMC, Lisbon, pp. 1-8.

Garcia, A., Marsh, J., Trejo, J. and Switters, J. (2010), "Living labs and regional innovation policies in the Mediterranean area", Proceedings of the eChallenges Conference, Warsaw, Poland, October.

Gasson, S. (2003), "Human-centered vs user-centered approaches", Journal of Information Technology Theory and Application, Vol. 5 No. 2, pp. 29-46.

Gould, J.D. and Lewis, C. (1985), "Designing for usability: key principles and what designers think", Communications of the ACM, Vol. 28 No. 3, pp. 300-311.

Greenbaum, J.M. and Kyng, M. (1991), Design at Work: Cooperative Design of Computer Systems, Lawrence Erlbaum Associates.

Hartley, J. (1987), "Social experiments and the role of end-users", in Qvortrup, L., Ancelin, C., Frawley, J., Hartley, J., Pichault, F. and Pop, P. (Eds), Social Experiments with Information Technology and the Challenges of Innovation, D. Reidel Publishing Company, Dordrecht, pp. 261-269.

Helsinki Manifesto (2006), available at: http://elivinglab.org/files/Helsinki_Manifesto_201106.pdf

Hirvikovski, T. (2012), "Regional policy and design in user-driven open ecosystems", presented at Design Days, Brussels.

Holtzblatt, K. and Beyer, H.R. (1995), "Humanfactor", Communications of the ACM, Vol. 38 No. 5, p. 31.

Intille, S.S., Larson, K., Tapia, E.M., Beaudin, J.S., Kaushik, P., Nawyn, J. and Rockinson, R. (2006), "Using a live-in laboratory for ubiquitous computing research", Pervasive Computing, Springer, Berlin, Heidelberg, pp. 349-365.

Ishida, T. (2000), "Understanding digital cities", Digital Cities, Springer, Berlin, Heidelberg, pp. 7-17.

Kiemen, M. and Ballon, P. (2012), "Living labs and stigmergic prototyping: towards a convergent approach", Proceedings of the XXIII ISPIM Conference – Action for Innovation: Innovating from Experience, Barcelona.

Leminen, S., Westerlund, M. and Nyström, A.G. (2012), "Living labs as open-innovation networks", Technology Innovation Management Review, Vol. 2 No. 9.

Loukis, E., Charalabidis, Y. and Scholl, J. (2011), "Editorial of the special issue on digital cities", Telematics and Informatics, Vol. 28 No. 3, pp. 144-147.

Markopoulos, P. and Rauterberg, G.W.M. (2000), "Living lab: a white paper", IPO Annual Progress Report, Vol. 35 No. 35, pp. 53-65.

Mechant, P., Stevens, I., Evens, T. and Verdegem, P. (2012), "E–deliberation 2.0 for smart cities: a critical assessment of two’idea generation’cases", International Journal of Electronic Governance, Vol. 5 No. 1, pp. 82-98.

Middleton, C.A. and Bryne, A. (2011), "An exploration of user-generated wireless broadband infrastructures in digital cities", Telematics and Informatics, Vol. 28 No. 3, pp. 163-175.

Mitchell, D. and Coles, C. (2003), The Ultimate Competitive Advantage: Secrets of Continually Developing a More Profitable Business Model, Berrett-Koehler Publishers, San Francisco.

Muller, M.J. and Kuhn, S. (1993), "Participatory design", Communications of the ACM, Vol. 36 No. 6, pp. 24-28.

Niitamo, V.P., Kulkki, S., Eriksson, M. and Hribernik, K.A. (2006), "State-of-the-art and good practice in the field of living labs", Proceedings of the 12th International Conference on Concurrent Enterprising: Innovative Products and Services through Collaborative Networks, Milan, pp. 26-28.

Oestmann, S. and Dymond, A. (2001), "Telecentres – experiences, lessons and trends", in Latchem, C. and Walker, D. (Eds), Telecentres: Case Studies and Key Issues, The Commonwealth of Learning, London, pp. 1-15.

Pallot, M., Trousse, B., Senach, B. and Scapin, D. (2010), "Living lab research landscape: from user centred design and user experience towards user cocreation", presented at the First European Summer School "Living Labs", Paris.

Paskaleva, K.A. (2011), "The smart city: a nexus for open innovation?", Intelligent Buildings International, Proceedings of the SIGCHI Conference, University of Maribor, Maribor, Vol. 3 No. 3, pp. 153-171.

Pasman, G., Stappers, P.J., Hekkert, P. and Keyson, D. (2005), "The ID-studiolab 2000-2005", Proceedings of Design Research in The Netherlands, Eindhoven.

Peltomäki, A. (2008), "Foreword in living labs for user-driven open innovation: an overview of the living labs methodology, activities and achievements", Office for Official Publications of the European Communities, European Commission, Luxembourg.

Pierson, J. and Lievens, B. (2005), "Configuring living labs for a ‘thick’understanding of innovation", Proceeding of Ethnographic Praxis in Industry Conference, Blackwell Publishing, Redmond, pp. 114-127.

Prime Minister’s Office (2006), "The launch of a European network of living labs-co-creation of innovation in public, private and civic partnership", available at: http://vnk.fi/ajankohtaista/tiedotteet/tiedote/en.jsp?oid=149029

Qvortrup, L. (Ed.) (1987), Social Experiments with Information Technology and the Challenges of Innovation, Springer, Berlin, Heidelberg, Vol. 114.

Schaffers, H., Komninos, N., Pallot, M., Trousse, B., Nilsson, M. and Oliveira, A. (2011), Smart Cities and the Future Internet: Towards Cooperation Frameworks for Open Innovation, Springer, Berlin, Heidelberg, pp. 431-446.

Schuurman, D. (2015), "Bridging the gap between open and user innovation? Exploring the value of living labs as a means to structure user contribution and manage distributed innovation", Doctoral Dissertation, Ghent University, Vancouver.

Schuurman, D., Lievens, B., De Marez, L. and Ballon, P. (2012), "Towards optimal user involvement in innovation processes: a panel-centered living lab-approach", Proceedings of Technology Management for Emerging Technologies (PICMET), IEEE, Vancouver, pp. 2046-2054.

Silverstone, R. (1993), "Time, information and communication technologies and the household", Time & Society, Vol. 2 No. 3, pp. 283-311.

Ståhlbröst, A. (2012), "A set of key principles to assess the impact of living labs", International Journal of Product Development, Vol. 17 No. 1, pp. 60-75.

von Hippel, E. (1988), The Sources of Innovation, Oxford University Press, Oxford.

Westerlund, M. and Leminen, S. (2014), "The multiplicity of research on innovation through living labs", Proceedings of International Society for Professional Innovation Management Conference (ISPIM), Dublin, 8-11 June.

Wixon, D., Holtzblatt, K. and Knox, S. (1990), "Contextual design: an emergent view of system design", Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM, Seattle, pp. 329-336.

World Bank & European Network of Living Labs (2014), Citizen-Driven Innovation: A Guidebook for City Mayors and Public Administrators, World Bank & European Network of Living Labs, Brussels.

Further reading

Beyer, H. and Holtzblatt, K. (1997), Contextual Design: Defining Customer-Centered Systems, Elsevier, London.

Gregory, J. (2003), "Scandinavian approaches to participatory design", International Journal of Engineering Education, Vol. 19 No. 1, pp. 62-74.

Heckman, J.J. and Smith, J.A. (1995), "Assessing the case for social experiments", The Journal of Economic Perspectives, Vol. 9 No. 2, pp. 85-110.

Ishida, T. and Isbister, K. (Eds) (2000), Digital Cities: Technologies, Experiences, and Future Perspectives, Springer, Berlin, Heidelberg.

Levén, P. and Holmström, J. (2008), "Consumer co-creation and the ecology of innovation: a living lab approach", Public Systems in the Future: Possibilities, Challenges and Pitfalls, University of Maribor, Maribor.

Mensink, W., Birrer, F.A. and Dutilleul, B. (2010), "Unpacking European living labs: analysing innovation’s social dimensions", Central European Journal of Public Policy, Vol. 4 No. 1, pp. 60-85.