Open e-learning platforms and the design–reality gap: an affordance theory perspective

Abstract

Purpose

The dropout rates of open e-learning platforms are often cited as high as 97%, with many users discontinuing their use after initial acceptance. This study aims to explore this anomaly through the lens of affordances theory, revealing design–reality gaps between users' diverse goals and the possibilities for action provided by an open IT artefact.

Design/methodology/approach

A six-month case study was undertaken to investigate the design implications of user-perceived affordances in an EU sustainability project which developed an open e-learning platform for citizens to improve their household energy efficiency. Thematic analysis was used to reveal the challenges of user continuance behaviour based on how an open IT artefact supports users in achieving individual goals (e.g. reducing energy consumption in the home) and collective goals (lessening the carbon footprint of society).

Findings

Based on the findings, the authors inductively reveal seven affordances related to open e-learning platforms: informing, assessment, synthesis, emphasis, clarity, learning pathway and goal-planning. The findings centre on users' perception of these affordances, and the extent to which the open IT artefact catered to the goals and constraints of diverse user groups. Open IT platform development is further discussed from an iterative and collaborative perspective in order to explore different possibilities for action.

Originality/value

The study contributes towards research on open IT artefact design by presenting key learnings on how the designers of e-learning platforms can bridge design–reality gaps through exploring affordance personalisation for diverse user groups. This can inform the design of open IT artefacts to help ensure that system features match the expectations and contextual constraints of users through clear action-oriented possibilities.

Citation

McCarthy, S., Rowan, W., Kahma, N., Lynch, L. and Ertiö, T.P. (2022), "Open e-learning platforms and the design–reality gap: an affordance theory perspective", Information Technology & People, Vol. 35 No. 8, pp. 74-98. https://doi.org/10.1108/ITP-06-2021-0501

1. Introduction

Open e-learning refers to the delivery of educational content to students through open access platforms made freely available online (Daniel, 2012; Gregori et al., 2018; Labarthe et al., 2016). In contrast to “closed” e-learning platforms where access is contingent on the student satisfying eligibility criteria and paying registration fees, open e-learning platforms are typically available to anyone with an interest in the topic, on a voluntary and cost-free basis. Open e-learning platforms such as massive open online courses (MOOCs) are an increasingly prevalent medium of education and training delivery in which information, knowledge and instruction are openly disseminated through online media to diverse user groups (Cidral et al., 2018; Daniel, 2012; Gregori et al., 2018; Labarthe et al., 2016; Lin and Wang, 2012).

However, evidence suggests that open e-learning users often disengage and do not progress to completion, with dropout rates cited as high as 97% (Hew and Cheung, 2014). One explanation for this centres on the diversity of user profiles where individuals from different educational, geographic and socio-economic backgrounds are attracted to open e-learning by the low barriers to registration (Cidral et al., 2018; Daniel, 2012; Gregori et al., 2018; Labarthe et al., 2016; Lin and Wang, 2012). This results in numerous design challenges centred on the creation of an open e-learning platform which caters to the diverse learning goals of numerous user groups. Such challenges are unique to open e-learning as proprietary e-learning systems tend to target a narrower and more defined user group, for example, semi-qualified professionals. To better understand the acceptance–discontinuance anomaly in open e-learning, we adopt the lens of “functional affordances” which explores the possibilities for action that an open IT artefact provides different user groups (Strong et al., 2014; Markus and Silver, 2008). Functional affordances affect users' experiences of a system (their feelings or attitudes) by shaping whether it assists or constrains their ability to realise individual and collective goals (Lanamäki et al., 2016; Parchoma, 2014; Volkoff and Strong, 2013). This can in turn affect the perceived usefulness of an open e-learning platform and its continued usage.

There is no broad consensus on how user experience should be defined, but most scholars see it as something dynamic, context-dependent and subjective that emerges in individual encounters with a product, system, service or an object (Law et al., 2009). The main design principles considered in UX design can refer to the presentation of the user interface, that is, (1) visual design principles (alignment, balance, contrast, hierarchy, proximity and space) (Canziba, 2018, pp. 160–161, 168–173) but also knowledge on the actual use, in other words individual (2) experiential aspects such as the primarily evaluative feeling (good-bad), and the pragmatic and the hedonic quality. These qualities ultimately fulfil the user needs for autonomy, competency, stimulation, relatedness and popularity (Hassenzahl and Tractinsky, 2006; Hassenzahl, 2008).

While a significant body of research has focused on the critical success factors of implemented e-learning platforms (Choudhury and Pattnaik, 2020), less attention has been directed towards the design of open e-learning platforms where functional affordances are appropriated and tested with representative user groups to understand their experiences (Gregori et al., 2018; Tuunanen and Peffers, 2018). Given the potentially significant relationship between user perspectives and system success (Choudhury and Pattnaik, 2020), our ability to understand user and designers' perceptions of open e-learning affordances may be crucial for addressing the acceptance–discontinuance anomaly going forward (Strong et al., 2014). However, the literature has traditionally discussed affordances as a theoretical construct with empirical research on its relevance to information systems design only now emerging (Seidel et al., 2013; Markus and Silver, 2008). Our study is further motivated by calls for research on e-learning engagement following the coronavirus disease 2019 (COVID-19) pandemic and the recent pivot to online learning in higher education (Bhagat et al., 2020; Singh et al., 2021).

Against the background presented above, our case study focuses on the initial stages of platform design to explore the following research question: How do users perceive the functional affordances of open e-learning platforms? In order to address this research question, we draw on findings from the ECO2 (Energy Conscious Consumers) project which developed an open e-learning platform (ACT4ECO) to help European citizens implement concrete actions for saving energy in the household. As part of this study, we sought diverse user feedback, focussing on perceived functional affordances of the open e-learning platform in question.

This study makes two primary contributions. Firstly, we draw on affordance theory to gain a better understanding of the relationship between an open e-learning system's designed features, different user perceptions of these system features and the affordances that they provide users (Evans et al., 2017; Markus and Silver, 2008; Parchoma, 2014). Building on this understanding, we inductively reveal seven affordances related to open e-learning platforms: informing, assessment, synthesis, emphasis, clarity, learning pathway and goal-planning. We discuss open IT platform development from an iterative and collaborative perspective (Gregori et al., 2018), inviting diverse users to participate in the early stages of development in order to better understand the design implication of functional affordances. Secondly, we present key learnings form the case study to help designers address the design–reality gap through exploring affordance personalisation. We suggest that open e-learning platforms should be designed to support possibilities for action that are relevant, interesting and possible for different user groups by catering to their goals and contextual constraints (Tuunanen and Peffers, 2018). This can help ensure that the open platform better meets the reality of diverse user groups, avoiding a “one size fits all” approach to open e-learning design (Greenhalgh et al., 2010; Tuunanen and Peffers, 2018).

The paper is structured as follows: Section 2 reviews the relevant literature on user perceptions, the design–reality gap and affordance theory. Section 3 then introduces our case study and the gathering of user perspectives on affordances. Section 4 presents findings followed by a discussion in Section 5. Section 6 summarises contributions and avenues for future research.

2. Background

Open e-learning tends to attract users across a wide spectrum of learning profiles and backgrounds (Cidral et al., 2018; Daniel, 2012; Gregori et al., 2018; Labarthe et al., 2016; Lin and Wang, 2012). Consequently, designers are faced with the sizable challenge of developing open e-learning platforms that respond to diverse user needs (Walji et al., 2016). In this section, we begin by reviewing the literature on user perceptions of IT to explore its importance for systems success. We then shift attention towards the designers' perspective and explore the practices necessary for designing e-learning platforms that stimulate the interest among different user groups (Conole, 2013; Torres-Ramírez et al., 2014). As a way forward, we present an overview of affordance theory to outline how user retention might be improved by focussing design efforts on certain “possibilities for action” that enable open e-learning users to accomplish individual and collective goals.

2.1 User perceptions of IT

User perceptions centre on how individuals respond to an IT artefact by assessing the relationship between an IT artefact's features and the user's goal expectations (Deng et al., 2010; Pallud and Monod, 2010). The literature suggests that task–technology fit is paramount in open e-learning to ensure that the e-learning platform supports users' requirements during learning activities (Lin and Wang, 2012). The related concepts of perceived usefulness and perceived ease of use may similarly contribute towards positive user perceptions (Sun et al., 2008) as well as variety in the form of multimedia instruction, diversity of assessment and user interaction through the e-learning platform (Cidral et al., 2018; Lin, 2011; Sun et al., 2008). Information/instructional quality have also been identified as significant contributing factors towards user satisfaction in e-learning (Esteban-Millat et al., 2014; Liaw, 2008; Mohammadi, 2015).

Beyond task-oriented concerns, user experience then looks at users' emotive response to the open e-learning platform (Pallud and Monod, 2010). From an experiential perspective, user perceptions can go beyond task–technology fit alone to consider how the open e-learning platform's capabilities generate pleasure, enjoyment and empowerment for users (Deng et al., 2010) and potentially support positive emotions such as happiness and cheerfulness (Esteban-Millat et al., 2014). Research suggests that this in turn can improve student learning, lengthen online sessions, increase perceived usefulness and promote self-regulation (Liaw, 2008; Liaw and Huang, 2013; Mohammadi, 2015).

However, user perceptions are also shaped by negative critical incidents such as the disconfirmation of performance expectations where an e-learning system does not proceed normally, or challenges users' expectations (Deng et al., 2010; Lin, 2011). This includes users' expectations around the length of the learning unit, assessment issues and insufficient system-level feedback (Cappel and Hayen, 2004) which in turn causes the users to experience frustration with the e-learning platform (Saariluoma and Jokinen, 2014). User capabilities may moderate the relationship between negative incidents and perceived ease of use, with inexperienced users encountering greater challenges than experienced users (Lin, 2011). For instance, user's perception of their skills to overcome such challenges as self-efficacy and computer anxiety affect user satisfaction, as inexperienced users who are unfamiliar with e-learning need more time to develop the competencies to effectively exploit the technology (Esteban-Millat et al., 2014; Liaw and Huang, 2013). We next explore negative critical incidents from the designers' perspective, outlining the practices necessary to address these “design–reality gaps”.

2.2 Design–reality gaps

“Design–reality gaps” describe differences between the assumptions built into an IT artefact by designers and the real-life needs of users (Greenhalgh et al., 2010). Gaps can emerge when the features developed by designers do not match the goals of user groups, and therefore create “dissonance” between user expectations and systems delivery. To address these design–reality gaps, the literature suggests that designers must engage in continuous dialogue with users to understand their needs and bridge any perceived gaps which may arise (Damodaran, 1996; Iivari, 2009; Kautz, 2011). Design practice begins by utilising a collaborative approach to exploring, understanding and defining user needs by generating feedback on the system and perceptions of value (Goldkuhl and Lind, 2010; Gregor and Hevner, 2013; Hevner et al., 2004; Parchoma, 2014). This can be achieved through techniques such as user piloting which aims to understand user perceptions of a system through a combination of research methods, for example, focus groups, surveys, interviews (Liedtka, 2014).

Once user needs have been explored, defined and understood, design practice then shifts focus to develop and redevelop systems using an iterative approach (Dodgson et al., 2005). Prototyping is used to bridge the design–reality gap through “bold experimentation”, developing offerings that are empathetic and responsive to the users' needs (McCarthy et al., 2020; Rosenkranz et al., 2014). Several iterations of a system may be designed, built and evaluated to see how well each addresses the problem under investigation, and meets/exceeds user expectations (Goldkuhl and Lind, 2010; Gregor and Hevner, 2013). Design principles can be identified to describe the qualities or attributes most valued by users when adopting a solution, that is, physical, cognitive, aesthetical, and emotional parameters and constraints (Gregor et al., 2020).

The existing literature has typically focused on instances where designers work with a limited sample of “representative user” to explore a finite scope of features to be designed (Iivari and Iivari, 2006; Iivari, 2009; Mumford, 1983). Yet, openness as a concept intrinsic to open e-learning platforms brings additional challenges to the design process as the diverse user-base means that the platform must continually evolve over time to cater to user needs – especially for needs which could not be anticipated by designers (Tuunanen and Peffers, 2018). The emergent nature of open platforms suggests that exploring and validating user needs can extend well beyond the product launch stage (Feller et al., 2008; Feller and Fitzgerald, 2000; Wynn and Eckert, 2017). It is also suggested that aside from user needs, open platform design requires a context-specific approach whereby societal issues are considered during development to meet citizen needs (Ruijer et al., 2017). In the next section, we draw on affordance theory to explore how a system's designed features and the perceived goals of users might be aligned.

2.3 Affordance theory

One way of understanding the design–reality gap is through the lens of functional affordances, the “possibilities for action” offered by an IT artefact which allow users to accomplish individual or collective goals (Gibson, 1986; Hausvik and Thapa, 2017; Markus and Silver, 2008; Strong et al., 2014; Volkoff and Strong, 2013). As outlined by Markus and Silver (2008), affordance theory explores the relationship between a designed system feature, and users' perception of this system feature which is interpreted through the possibilities for goal-oriented action that it provides. Functional affordances therefore consider both the materiality of technical objects (e.g. an open e-learning platform) and their relationship with users as agents of change (Leonardi, 2013; Zammuto et al., 2007).

Examples of functional affordances include the comment feature of a decision support system which users perceive as offering possibilities for action in the form of an “idea recording” affordance (Markus and Silver, 2008), or the online check-in feature of an airline system which users perceive as allowing them to save time before their flight (“checking in 24 h in advance” affordance) (Balci et al., 2014). Of relevance to our case study is the work of Seidel et al. (2013) who investigate the contribution of functional affordances to environmental sustainability transformations. Based on findings from a revelatory case study, the authors identify two categories of functional affordances which contribute towards sustainable work practices: sensemaking affordances (reflective disclosure and information democratisation) and sustainable practicing affordances (output management and delocalisation). The authors call for further research on the design of functional affordances required for sustainable transformations.

Building on the work of Strong et al. (2014), Figure 1 provides an illustration of affordance theory. Given our interest in the design stages of open e-learning platform development, we focus attention on the potentials for action (affordances) provided by an IT artefact.

The existence of functional affordances does not rest on actualisation by a user, nor does perception of an affordance automatically result in actualisation (Du et al., 2019; Hausvik and Thapa, 2017; Strong et al., 2014). When designing IT systems, functional affordances are best understood as potential for actions embedded in a technical object. Possibilities for action are finite, and functional affordances can both enable and constrain user actions (Gibson, 1986; Hausvik and Thapa, 2017; Strong et al., 2014; Volkoff and Strong, 2013). Users may ignore certain functional affordances or create workarounds in order to achieve their goals (Balci et al., 2014; Parchoma, 2014; Robey et al., 2013). Capabilities of the user/user groups and their prior experience of using IT can also impact this (Balci et al., 2014; Lin, 2011; Parchoma, 2014). Prior research also suggests that users' interactions with affordances may depend on their age and education profile. Users may perceive functional affordances in unique ways and in turn derive different symbolic expressions which they attach to a technical object (Markus and Silver, 2008). For instance, functional affordances may create feelings of freedom for some users, efficiency or equality of participation for others, which users then begin to associate with the technical object (Balci et al., 2014). This can vary across individuals and collective user groups (e.g. experts vs less experienced users).

Moreover, functional affordances may be interpreted differently by designers and users. While designed system features offer potential for action, they are not necessary or sufficient conditions for actualisation (Balci et al., 2014; Markus and Silver, 2008). Indeed, the designers' intended affordance may often differ or potentially even conflict with the goal-oriented actions of users (Norman, 1988). Affordance theory can provide insights into how designers appropriate a system by changing the features to match user goals over time (Leonardi, 2013). It can also elucidate how the interrelated functional affordances which make up a technical object are perceived across contexts of use (Strong et al., 2014).

Nevertheless, further research is needed to explore how the functional affordances of an IT system dynamically shape individual and collective action. While prior studies on usability and user requirements have primarily focused on the design of system features, affordance theory shifts attention towards the possibilities for action that a system's feature provides users across different contexts of use. Affordance research is therefore distinct in its focus on user perceptions and goal-oriented actions. Further research on affordance theory can provide insights into how individual and collective goals shape use intention, offering a complementary (rather than contradictory) perspective to prior studies on usability and user requirements.

3. Methodology

The authors undertook a six-month case study (cf. Walsham, 2006) to explore users' perceptions of an open e-learning platform, and the “design–reality gaps” that emerged where the features developed by designers did not align with users' goals. Case study research was selected as an appropriate methodology, as it enabled the authors to explore the meanings and behaviours of users and designers, as well as the contexts in which they act. Concepts from affordance theory were empirically analysed, and potential extensions or modifications to their relationships were explored within the context of open e-learning (cf. Ridder, 2017). Our study centres on ECO2, a 36-month sustainability project funded by the EU Commission's Horizon 2020 programme. The aim of the project was to increase citizens' awareness of their energy consumption and improve energy efficiency in households through open e-learning. This is achieved by developing an open e-learning platform called ACT4ECO to help European citizens implement concrete actions for saving energy (see Figure 2). In total, the project aimed to reach 10,000 citizens across Europe who would sign up to the platform and learn how to complete actions that will increase their energy efficiency. The learning paths proposed by the open e-learning platform aim to support each user in climbing the “ladder of change” from “Motivation” to “Exploration” and finally to “Action”, where they make more environmentally sustainable choices (Kahma et al., 2021).

Our case study focuses on the piloting stage of the project (January 2020 to June 2020) when usability tests were run to provide in-depth insights into how users perceived an initial prototype of the platform and its e-learning content. During this time, the authors were “involved researchers” (cf. Walsham, 2006) working alongside designers of the prototype to draft content and information blocks for the e-learning platform. The wider project consortium consisted of private and public organisations from Denmark, Finland, Ireland, Italy, Belgium, Lithuania, Bulgaria, Portugal and Greece who were responsible for recruiting users in their respective countries.

Table 1 describes the themes used to structure users' learning experiences on ACT4ECO. The content was validated through a three-step process: (1) internal evaluation within the project team, (2) individual test runs with stakeholders, and (3) pilots with representative user groups. In this paper, we focus our attention on the pilots undertaken during step 3. The system's initial design was informed by research conducted by three of the co-authors (NK, TE, SMC) which explored the needs of target user groups based on a segmentation analysis of energy consumers in each of the partner countries (Kahma et al., 2021). The project team included three qualified designers who followed best practice guidelines in user interface design. The system may be categorised as an initial (ad-hoc) prototype designed to illicit initial feedback on the perceived affordances associated with features in the beta version of the platform.

3.1 Data collection and analysis

Data collection took place during the six-month piloting phase between January and June 2020. Piloting was completed in all nine countries. However, for the purposes of this paper, we will draw on representative findings from two countries in particular: Ireland and Finland. These countries were purposefully selected to provide in-depth findings on two diverse samples of users. While the level of energy awareness is quite high among citizens in Finland, Ireland comes from a lower base of understanding (Kahma et al., 2021). This is despite the comparable efforts that both national governments have undertaken to promote sustainable energy use (Kahma et al., 2021).

A total of 28 participants took part in the selected pilots: 15 participants from Ireland and 13 participants from Finland. Our study adopted a purposeful sampling approach and sought to recruit representative users from different socio-demographic backgrounds (e.g. age, gender, location, home ownership) and various levels of perceived competency (prior awareness of energy issues, technological savviness). Purposeful sampling allowed for the categorisation of participants based on identified similarities or differences (Patton, 2002). Our consideration of these criteria is briefly summarised in Appendix 1. In terms of socio-demographics, we aimed to recruit representative users for different gender and age categories across the two countries. While there was a relatively even split of male and female participants in the Irish study, the Finnish study had slightly more female participants in attendance. Across both countries, there was a slightly higher representation of users in the age category of 25–34 years. The perceived competency levels of participants are also outlined in Table 2. The inclusion of these variable was informed by the works of Esteban-Millat et al. (2014) and Liaw and Huang (2013) who suggest that perceived competencies impact users' ability to deal with negative critical incidents. According to the subjective evaluations, the Finnish pilot participants were more energy aware (76.9% compared to Ireland 46.6%), and more knowledgeable about energy (69.3% compared to Ireland 33.3%). In contrast, Irish pilot participants were more tech savvy (66.7% compared to Finland 46.2%). Challenges were faced in recruiting users with “very poor” levels of technology savviness due to a lack of Internet access and IT skills. Participants were self-selected and therefore represent a sub-sample of the population, already somewhat aware of energy efficiency issues and with some existing motivation, and willingness to act upon said motivation.

The pilots were structured as follows: (1) participants individually reviewed the ACT4ECO platform, (2) participants completed an online questionnaire (see Appendix 2), and (3) participants engaged in a focus group to discuss their perceptions of the designed features and e-learning content with the research team. Participants were also offered prompting questions by the research team to stimulate dialogue around different aspects of the platform (see Appendix 3). A combination of focus groups and individual surveys were used for piloting the open e-learning platform. The focus groups were conducted in a semi-structured manner and moderated by representatives of the project team in each country. Project team members were best placed to guide participants through the system features and elicit their feedback on perceived affordances given their prior knowledge of the system. Participants were invited to reflect on the “as is” system (alpha version) as well as the “to be” system (beta version) by providing comments on the existing design as well as recommendations on how to improve it. Focus groups lasted for approximately 1.5–2 h with focus groups in Finland conducted in both Finnish and English. In Ireland, the introduction of COVID-19 restrictions meant that some participants had to engage remotely as the pilots were scheduled just at the beginning of the pandemic. All participants in the focus groups were asked to complete the survey. A summary of the accompanying survey is included in Appendix 2.

Qualitative thematic analysis (cf. Braun and Clarke, 2006) and descriptive statistics were used by the authors to analyse participants' responses. Four authors coded the data over three phases, comparing and reconciling their findings where necessary. During the first phase, the authors began by continuously rereading the transcribed content to generate a set of initial codes which were judged as meaningful and important to the research question. Initial codes centred on ACT4ECO's features and the possibilities of actions (i.e. affordances) that these features provided users. The second phase of coding involved grouping the initial codes together to form overarching thematic categories of codes to help organise and refine the content according to similar types of affordances. We allowed for new affordances emerging from the data over time, with new categories also created as necessary to help further refine the content. The authors met regularly to make sense of the data and critique thematic categories to ensure this was representative of the data. Lastly, the authors generated a storyline around the research using the aggregated codes. Collective reasoning continued throughout this phase of thematic analysis until a point of saturation was reached, and further analysis did not contribute new interpretations, but rather supported existing ones (Braun and Clarke, 2006). Figure 3 presents a full list of codes across the three phases of our thematic analysis.

Recent studies have underlined the similarities of individual surveys and focus group as sources of data, emphasising how they investigate experiences, beliefs, and opinions in complementary ways. Nevertheless, there may be variation in the level of “depth” in terms of comparability and the frequency of themes (Guest et al., 2017). The data collection methods are, therefore, not interchangeable. Morgan (1996) suggests that the main difference between individual and group data collection methods is that whereas the former produces deeper knowledge, the latter offers a broader perspective on the theme studied (Crabtree et al., 1993; Morgan, 1996). The main strength of focus groups is that they allow for more speculation, as the group members encourage each other to elaborate on the discussed themes and consider a wide range of viewpoints beyond what is obvious (Heikkilä and Kahma, 2008; Kaplowitz and Hoehn, 2001). This moves beyond individual observations to produce synergistic insights from the collective as participants engage in conversation around emerging issues raised by the wider group (Braun and Clarke, 2013; Morgan, 1996). A weakness of focus groups is that findings may become positively or negatively skewed based on influence from other (sometimes more assertive) participants (Morgan, 1996). Surveys can work better from the viewpoint of some subjects, who perhaps feel more at ease sharing their thoughts in a more direct medium.

Combining the two techniques allows not only a comparison between two data sets, it may also be that in individual surveys there is more space for reflection than in a social group setting. The level of specificity also differs between the two types of data which can help gain a richer picture of functional affordances.

4. Results

As previously discussed, open e-learning platforms are typically aimed at large-scale diverse user groups. In this respect, the measurement of e-learning success shifts from traditional completion rates and certification to assessing levels of goal relevance.

Table 3 presents a matrix of systems features and the affordances perceived by users, as inductively revealed from our thematic analysis of qualitative data. Overall, we find that results from other countries involved in the pilots were largely consistent with those from the representative countries of Ireland and Finland despite differences in socio-economic backgrounds and skill levels.

4.1 Informing and assessment affordances

The informing affordance emerged as a central aspect of participants' open e-learning experience. Firstly, participants with limited prior knowledge of energy noted their appreciation when “information was given in manageable blocks”, as well-organised instructional content helped them to compare possible actions for improving energy efficacy in the household. These participants also noted the need for brevity when delivering instructional content, observing that sections which were “too wordy” impeded their learning. For instance, comments were made by participants with limited prior knowledge of energy that some technical content on retrofitting households needed to be reworded as the suggested actions did not make sense. One participant noted that “The issue/energy efficiency confuses me; I'm looking for plain language advice”, and shorter bullet points were suggested to support users' understanding of possible actions and maintain their attention: “short text descriptions kept my attention”.

Designers faced considerable challenges in crafting an accessible form of language for all users, as they also risked losing users with more advanced prior knowledge of energy. Similarly, creating assessments that were sufficiently challenging for all participants was an ongoing design challenge. While some participants felt assessments were not challenging enough, others thought it sufficiently tested the knowledge they had gained from the instructional content provided on the open e-learning platform. For instance, some e-learning users indicated that quizzes “were the most engaging part”, while others recommended that designers “make sure there's a clear purpose for quizzes (not just self-reflection) and ensure they're sufficiently challenging for all.” Participants with a good knowledge of energy were unsure about the purpose of assessments and felt “a bit confused about ‘what brought you here? ’. One focus group participant felt that the quizzes seemed pointless without a certificate for completion as they expected to “get a diploma on the next slide”. Focus group participants also noted the need for clear feedback on their progress over time and how well they were doing: “it would be nice … when I choose one of them, that I would get feedback on my choice. I mean ‘right! ’ or ‘wrong! ’.

Finally, a key challenge faced by designers was making the topic of energy awareness attractive for all users. For instance, one participant with low energy awareness felt that some content can be “a little boring”, and there is “nothing (that) drags you in”. Others observed that the use of guilt or shaming tactics to build user motivation for sustainable actions was ineffective for changing their energy consumption practices. It was recommended that designers should rewrite sections with a more positive tone using a “feel good mantra to tell your friends and neighbours”. The following quote illustrates the feelings of one survey participant on the tone on the platform: “people are being made to feel guilty for using appliances they always have… you want people to change their behaviour, not necessarily make them feel bad for their current behaviours”. The unintended consequence of guilt tactics according to participants was that they may give up entirely and decide it is not worth acting. Participants noted their preference for “Common sense ideas” about actions to improve their energy efficiency (e.g. draught proofing, changing to LED lightbulbs) and liked sections where the tone of the content was more conversational.

4.2 Emphasis and synthesis affordances

At the beginning of each theme, the e-learning platform presented a summary of the instructional content would be delivered to users and the potential impacts that they could realise through their engagement. When asked about how this emphasis affordance offered by ACT4ECO shaped their expectations, participants suggested that action-oriented phrases such as “saving energy”, “saving money” and “reducing my (carbon) footprint” drew them in and caught their attention. There were different perspectives on the meaning of some emphasised words however, with survey feedback indicating that some participants assumed that the term “Cost” refers to the expense of completing the course rather than the proposed actions. Participants also suggested that other emphasis affordances should be provided such as “a checklist or step–by-step approach to start the action ‘journey’”. Participants with low levels of energy awareness seemed to benefit most from the emphasis affordance as they indicated that the upfront statement of each theme's purpose increased their motivation for action. Social features were also suggested as means of emphasising actions: “I'd be motivated to use it if there was some purpose–a certification at the end, some reward for being involved like a leader board.”

In relation to the synthesis affordance, both groups suggested that the summative graphics provided by the open e-learning platform was useful for summarising learnings, and aided user's sense-making of possible actions. One participant with limited prior knowledge of energy noted that “The graphics are great … Once you start into actual pieces of advice it is actually really good”. A few participants also observed the importance of graphics for conveying messages, supporting learning and enhancing motivation. Graphics proved essential for summarising “a lot of information in one go [and helping] to spread it out”, particularly in the “Produce your own Energy” theme which participants indicated was “overly convoluted in places” and “over complicated”. Participants with limited prior knowledge also felt that graphics were useful for summarising messages and clarifying the key points: “once you make your way through the [language] used, I found pieces of useful information”. Similarly, participants with good prior knowledge noted that graphics could also convey more detail than text content alone and satisfied their desire for more knowledge: “everyone understands that electricity just does not come from the wall outlet whenever you plug something in, but there are the stages. You could maybe open up a little how that actually happens… Explaining it really fast [with] a stock photo”. However, some graphics conflicted with knowledgeable participants' understanding of the content, pointing to the importance of context and the placement of different graphics: “most of energy goes keeping your warm, but then there's a picture of a fridge… and an air-conditioner?”

4.3 Clarity affordance

A sign-in process was identified as a crucial feature by designers to track the successful recruitment of 10,000 users and maintain users' voluntary engagement with the open e-learning platform over time. However, both tech savvy and less tech savvy participants had mixed opinions on the sign-in process of ACT4ECO, with some stating that it made the prospect of returning to the platform on a regular basis “difficult” or “very difficult”. The main issue was that many participants were unclear on the value of the sign-in process for “self-paced learning” about actions and did not understand the platform's requirement for users to enable “local storage” in order to save progress. In particular, less tech-savvy participants felt there “should not be a sign-in process”, as it compromises the platform's attractiveness relative to other websites or blogs that do not require a log-in: “There are a number of sites you can find from Google instead of [using] this platform. So that personally I would not register unless the sign-in has some real advantages to me”. A more basic clarification concerned the emailed registration link, and lack of system feedback when users were redirected to the open e-learning platform's back-end content management system in order to input their login details. Participants also noted that the need for users “to return to the site and [sometimes] refresh the page” during login was “very off putting”. To improve clarity, both tech savvy and non-tech savvy participants asserted the need to “Provide more feedback”, “Make [the] sign-in easier” and “Make it seamless”.

Feedback related to the clarity affordance also centred on the layout of the open e-learning platform. One participant suggested the need for clearer guidance around how an e-learning platform would help users improve their energy efficiency, particularly less tech savvy users who had limited prior experiences of e-learning and may not understand its relevance towards promoting actions in the real world. It was recommended that a chat portal be provided if less tech savvy users had queries around the platform; however, resource constraints in the project meant that this would not be feasible in the short term. Meanwhile, comments on the “look and feel” of the platform were generally positive and the majority noted that the design of the open e-learning platform was important to make images and text easier to read. However, the design clashed with more knowledgeable participants' expectation of what an e-learning platform on sustainable energy use should look like: “there is the eco theme, so in terms of colours you could have something pointing to that direction, I mean people expect ecological colours, green and such?”

4.4 Learning pathway affordance

Another crucial challenge centred on the design of learning pathways that would effectively guide users through the “ladder of change” from “Motivation” to “Exploration” and finally to “Action”. Navigation within and across themes posed issues for different participants however, as some learning pathways were harder to follow than the designers had expected. When discussing the resulting learning pathways, less knowledgeable participants felt it was sometimes difficult to click through a lot of sections without a clear indication of “what is going on” relative to the overall learning outcome. In order to aid learning, participants recommended that the e-learning sections should flow seamlessly from one into another and avoid the platform's default setting of returning to the homepage once users finish a theme's sub-action. They asserted that this would help participants maintain their sense of orientation in the platform with a clear map to remind users of where they were.

Learning pathways were also designed to connect different themes together and ensure a cohesive narrative within the e-learning platform. Participants with limited prior knowledge however felt the navigation from routine topics (e.g. “Become a smart consumer”) to more complex topics (e.g. “Produce your own energy”) was unintuitive and noted the importance of system feedback and clearer indication on the “need to click/scroll through the headings in the ‘act now’ section for further content”. A scrolling slideshow was designed to capture users' attention on the homepage, but link mapping issues meant “some of the links to the action do not take you to the same start point every time” with one survey participant noting “the scrolling slideshow (top of page) should direct down to… ‘Explore the actions you can take’ instead of ‘Action Themes Archive’. Energy aware participants also called for a search function to overcome this issue, as they found it difficult to remember where content was housed.

4.5 Goals planning affordance

At the end of certain sections in the e-learning platform, users were provided with a goal-planning affordance when asked the following question: “what did you learn from ACTO4ECO today?” In answering this question, participants with low levels of energy awareness indicated that this stimulated reflection on the possibilities for action they had learnt based on the learning path provided. More than half of the survey participants stated that they “would make changes at home” to save energy. Participants also described the energy saving actions they had learnt and how these could be maintained, with one participant with limited prior knowledge detailing how the platform enabled them to become “more aware of my own behaviour and how I select and use more energy efficient appliances” as they realised “I'm not using my appliances correctly”.

Still, other participants indicated that they did not discover possibilities for action based on their engagement with the platform. This design challenge centred on gaps between possibilities for action and the resources available to different user groups. Some participants noted that many suggested goals were not feasible for them due to income constraints and their existing financial outlays. More energy aware participants also asserted the importance of options being framed in a real-world context: “I personally [liked] the last page, select the ones you're willing to do … I wish there could be some sort of story about someone and their daily habits … I have no idea what's going on in my electric bill. What is the actual impact, if it's like one euro per month, two euros per month, or is it bigger, what is the impact?” More knowledgeable participants also stated that while the information provided was useful, it would be beneficial for the platform to note when help should be sought from professional tradespeople in order to achieve more complex goals.

The next section provides a discussion of our findings relevant to academic and practitioner communities.

5. Discussion

The following section provides a discussion of findings in relation to our research question: How do users perceive the functional affordances of open e-learning platforms?

In contrast to traditional face-to-face learning environments, open e-learning is characterised by low barriers to registration, and self-paced independent study which allows diverse user groups to engage with open instructional material (e.g. teaching-materials, assignments, and quizzes) at their own convenience (Cappel and Hayen, 2004). However, the diversity of user profiles can make designing e-learning platform a challenge, particularly in light of the high dropout rates recorded in practice. To better understand this acceptance–discontinuance anomaly, we sought to elucidate differences in how users perceived the designed affordances of an open e-learning platform in order to better understand the “design–reality gap” (Greenhalgh et al., 2010; Tuunanen and Peffers, 2018). Based on our findings, we contribute insights into seven affordances provided by the open e-learning platform: informing, assessment, synthesis, emphasis, clarity, learning pathway and goal-planning. In the following section, we discuss each of these affordances in turn and present key learnings on how designers can bridge gaps between user-perceived affordances and an open e-learning feature going forward. We seek to answer the call by Ali et al. (2018) for further empirical studies into the barriers of e-learning platforms and how they might be addressed through more informed design choices. Specifically, we explore barriers related to the conceptual categories of individual and technological factors, for example, student knowledge prior to the course, software and interface design and technology experience (Ali et al., 2018).

The informing affordance calls on users' cognitive abilities to comprehend content, solve problems and make sense of possible actions (Bower, 2008; Cidral et al., 2018; Lin, 2011; Sun et al., 2008). We find that effectively designing for this affordance requires that designers cater to users with different levels of prior knowledge so they can comprehend content and engage with selected sections. Our findings suggest that avoiding wordy formulations is key to organising information, especially as literacy skills differ in adult populations. For instance, 17.9% (1 in 5) of adults in Ireland have a literacy level at or below Level 1, considered to be “very poor literacy skills” (OECD, 2015). Effectively designing for this affordance is necessary to provide gradated challenges in open e-learning and accommodate the diversity of users' backgrounds, learning profiles and tech savviness, so they can undertake the actions they may wish to pursue (Walji et al., 2016). Findings suggest that the assessment affordance can help here by strengthening users' comprehension of possible actions through quizzes and reflections on different action outcomes (Cidral et al., 2018; Lin, 2011; Sun et al., 2008). Less knowledgeable participants noted it was essential for instructional content to be supplemented by quizzes to provide feedback on learning, with errors explained through feedback to assist in user self-reflection (Cappel and Hayen, 2004). Given the complex nature of open e-learning design, informing and assessment affordances may need to be restructured each time new features are added or removed.

We also find that both the synthesis and emphasis affordances facilitated users' internalisation of knowledge and helped with managing their goal expectations (Bower, 2008; Deng et al., 2010; Lin, 2011). These affordances may in turn support improved information quality for both knowledgeable and less knowledgeable users, a key antecedent of user satisfaction in e-learning (Cidral et al., 2018; Esteban-Millat et al., 2014; Mohammadi, 2015; Sun et al., 2008). Results also show how the related affordances of learning pathway and clarity support ease of use by ensuring the user feels a sense of orientation in the platform and understands the purpose of actions. This ensures that users can concentrate on what they are doing (to the exclusion of other stimuli) and recover from any negative critical incidents that might arise (Deng et al., 2010; Esteban-Millat et al., 2014; Pallud and Monod, 2010). Lastly, we see how the goal-planning affordance transforms user learnings into possibilities for action, once content is tailored to the resources available to users (Strong et al., 2014). User perceptions are tightly bound to the interaction between IT features and their goal expectations (Deng et al., 2010; Pallud and Monod, 2010). We also find that users may also derive symbolic expressions (e.g. “helpful”, “useful”) through goal-planning which they in turn attach to an open e-learning platform (Markus and Silver, 2008; Pallud and Monod, 2010). This goes beyond technical considerations alone and requires equal consideration of users' positive and negative emotive responses to an e-learning platform (Pallud and Monod, 2010).

Table 4 presents a set of affordances which were operationalised in the ACT4ECO case, and summarises the learnings that designers garnered to better support the operationalisation of these affordances in future versions of the e-learning platform.

The design and development of different affordances is not a once off event, however, and ongoing adaptations are needed to sustain user engagement over time. This is of particular concern for the acceptance–discontinuance anomaly in open e-learning given the propensity for increased dropout rates over time. To ensure continuity of use beyond initial design, we suggest that open IT artefacts should remain in a state of perpetual evolution. Affordance redesign should seek to adapt existing features in a way that continually drives users towards more challenging actions with the promise of achieving more rewarding goals. This could take the form of learning pathways that re-enforce the need to take more energy efficient actions so as to realise greater cost saving and decreased CO₂ emissions. A more sustainable model of open e-learning can be supported by keeping a watchful eye on system analytics (e.g. user engagement and progress) and developing two-way communication between users and the design team. This ensures that feature scaling is informed by an ongoing evaluation of the design–reality gap and e-learning affordances.

Building on our findings and insights from affordance theory, we propose that designers must appropriate open IT artefacts for diverse user groups through affordance personalisation, ensuring that features accommodate user expectations and contextual constraints (cf. Leonardi, 2013). To close the design–reality gap in open e-learning, designers must therefore deliver affordances which are interesting, relevant and possible for diverse user groups. We suggest that the inclusion of personalised content for knowledgeable and less knowledgeable users can support continuance behaviour in open e-learning by helping different user groups to recognise how the system is facilitating the achievement of goals (Strong et al., 2014; Volkoff and Strong, 2013). Designers of open IT artefacts must seek to deliver clear action-oriented content aligned with user needs. In addition, our findings suggest that functional affordances may play a mediating role between the features of an open e-learning platform and users' satisfaction. Findings suggest that well-designed affordances may combine to improve user satisfaction by driving them towards the achievement of individual and collective goals, that is, more environmentally sustainable choices. This can also promote better user experiences through cognitive absorption in relevant content (Esteban-Millat et al., 2014). Figure 4 illustrates our extended conceptual model based on findings from the ACT4ECO case study.

Our extended conceptual model explores the implications of openness for affordance theory. It does so by considering how the low barriers to entry in open e-learning requires design teams to recognise and then accommodate the diverse levels of knowledge, topic awareness and technology skills of different user groups. In the absence of this recognition, some open e-learning users (e.g. those with limited prior knowledge or poor IT literacy) may end up being marginalised, as their actualisation of affordances would be inhibited in certain contexts. Our findings suggest that personalisation of open IT affordances is therefore required to adjust to the needs of different users (e.g. ensuring accessibility across learning pathway and navigation clarity).

In open e-learning, we therefore recommend that affordances should be designed with personalisation in mind to ensure alignment with the goals of diverse users (Lin and Wang, 2012). To that end, the openness of e-learning must be complemented with ongoing collaborative and iterative revision to ensure the needs of new users are embedded in the existing platform, in turn delivering an emergent and continuously realigned platform. We also recommend that differing piloting techniques be adopted to offer complementary insights from different user groups. Focus groups can be used both as a self-contained method, and as a complementary method to other ways of generating data such as individual surveys. As stated by one focus group from Finland: “it's good to discuss it in a group … it brings more in the end than just individual work … I only look at [it] from my perspective, but I do not know the other, so it's good to hear”. Descriptive statistics from survey data can also provide supportive comparative data on users' perceived competencies, for example, knowledge of energy and tech savviness.

Lastly, we suggest the need for cross-national studies to support open IT artefact design for diverse groups. As outlined by affordance theory, the interrelated functional affordances which make up a technical object may be perceived differently across context of use (Strong et al., 2014). Although methodological challenges exist when conducting cross-country studies, the involvement of diverse user groups is crucial for informing the development of open e-learning platforms. Going forward, it is hoped that users across Europe will engage with the open e-learning platform to undertake sustainable actions such as household renovations and/or behaviour changes, with the aim of increasing their energy efficiency.

6. Conclusion

Open e-learning platforms offer several unique advantages to users and instructors alike such as location flexibility, knowledge archival/storage and the sharing of digital content made freely available online (Lin, 2011; Zhang et al., 2004). Nevertheless, there are often differences between the perceptions of designers and users in open IT artefact development. Design–reality gaps can emerge between the designed features of an open e-learning platform and users' perceptions of these features as sufficiently interesting, relevant and possible. This can impede the primary objective of an open e-learning platform to direct users towards different possibilities for action through learning paths, for example, motivation, exploration and action.

In order to better understand design–reality gaps in open e-learning, our research draws on findings from “ACT4ECO”, an open e-learning platform which aimed to increase users' energy efficiency in the household. Based on our findings, we present two primary contributions which will be of interest to academia and practice. Firstly, we inductively developed a conceptual model of seven functional affordances which can help designers understand the possibilities for action provided by open e-learning platforms: informing, assessment, synthesis, emphasis, clarity, learning pathway and goal-planning. Our findings differ from prior studies on the usability of system features by focussing on the possibilities for actions that a system feature provides users to achieve their individual or collective goals. For instance, our findings on the affordance of “synthesis” looks at how representing knowledge in different ways can allow users to explore actions such as retrofitting their home to improve energy efficiency or reduce energy waste.

Secondly, we present key learnings for supporting the design of more user-centred open e-learning platforms through iterative and collaborative piloting with diverse user groups. The insights gained from this can help designers, developers, and instructors to enhance the delivery of open e-learning, and ultimately deliver more positive user experiences through affordance personalisation. We suggest the personalisation of open e-learning affordances may be pivotal for bridging gaps and accommodating different learning profiles (e.g. experts vs. less experienced users), demographics and contextual factors (Esteban-Millat et al., 2014; Sun et al., 2008). Our research therefore contributes practical insights into the importance of user engagement in open e-learning, not only for identifying affordance gaps during technical development, but also in gauging user satisfaction and continuance behaviour.

There are nevertheless limitations inherent in our study which future research can seek to address. Firstly, generalisability cannot be claimed from a single case study nor the current sample size of our focus group. Our study instead focused on collecting detailed findings from representative user groups. Future research efforts can seek to collect large-scale feedback from user groups across different contexts as well as other open e-learning platforms. This can also involve the measurement of the usage statistics associated with open e-learning platforms such as the number of actions recorded and quizzes attempted to further understand the impact of functional affordances. Another limitation is that the case study was primarily focused on the initial beta stages of designing the open e-learning platform. Our ability to study the long-term experiences of users along a UX curve was therefore limited as experience sampling methods require longitudinally engaged participant samples and wider testing. Future studies can seek to provide a longitudinal analysis of the impact of functional affordances from design to the implementation stages of development. The experience sampling method in particular could provide an interesting avenue for research on the evaluation of user experience over time using longitudinal user data and the surveillance of real-life users (cf. Kujala et al., 2011). We also suggest that the learnings presented in Table 4 for closing the design–reality gap could form the basis of design principles in open e-learning development going forward (cf. Gregor et al., 2020). While outside the scope of this paper, we believe the insights provided can act as a springboard for further research on the evaluation of design principles for open e-learning.

Figures

Figure 1

Affordance theory

Figure 2

Screenshots of the ACT4ECO platform

Figure 3

Full list of codes from thematic analysis

Figure 4

Extended conceptual model

Table 1

Themes of ACT4ECO

E-learning theme	Description
“Become a smart consumer”	Informs users on how to control their energy consumption by adjusting their thermostat, household appliances and heating/cooling systems
“Improve your home”	Educates users on how to create an energy efficient home by improving air tightness, home insulation and ventilation. Advice on how to finance these initiatives is also provided
“Sustain efficient energy use”	Ensures that users maintain energy efficiency by solidifying their behaviours over time. Users are educated on “energy rebound” where savings are negated by unconscious actions
“Produce your own energy”	Explores how users can generate energy at home by installing renewable energy systems such as solar photovoltaic and water heating systems
“Manage your energy consumption”	Educates users on their energy bill and the contribution of different appliances to increases users' awareness of energy consumption over time

Table 2

Pilot users' energy awareness, knowledge and technology savviness n (%)

Level	Energy awareness		Knowledge of energy		Technology savviness
	Ireland	Finland	Ireland	Finland	Ireland	Finland
Very good	2 (13.3)	8 (61.5)	0 (0.0	3 (23.1)	3 (20.0)	5 (38.5)
Good	5 (33.3)	2 (15.4)	5 (33.3)	6 (46.2)	7 (46.7)	1 (7.7)
Neutral	2 (13.3)	1 (7.7)	3 (20.0)	0 (0.0)	2 (13.3)	5 (38.5)
Poor	5 (33.3)	2 (15.4)	5 (33.3)	2 (15.4)	2 (13.3)	2 (15.4)
Very poor	1 (6.7)	0 (0.0)	2 (13.3)	2 (15.4)	1 (6.7)	0 (0.0)
Total	15 (100)	13 (100)	15 (100)	13 (100)	15 (100)	13 (100)

Table 3

Functional affordances and the associated system features of ACT4ECO

Affordance	Description of relevant ACT4ECO system feature(s)
Informing	Instructional content relates to the following themes: (1) become a smart consumer; (2) improve your home; (3) sustain efficient energy use; (4) produce your own energy and (5) manage your energy consumption
Assessment	Quizzes test user knowledge on the delivered instructional content related to each ACT4ECO theme
Emphasis	Information is provided on each module's estimated “duration”, “financial cost”, “difficulty” and “green” impact associated with each ACT4ECO theme
Synthesis	Graphics explain instructional content visually and aim to represent knowledge in different ways
Clarity	The homepage provides instructions on the sign-in process to access e-learning content. This aimed to allow self-paced study by saving progress across themes. Users could then return to previous pages of instructional content as required
Learning pathway	“Small circles” denote the users' progress through a learning section. “Next” and “back” buttons allow the user to navigate through the different learning sections relative to the platform's themes
Goal planning	At different points, users are asked: “what do you plan to do next?” This prompts users to reflect on their motivation for engaging with the theme and helps them develop a plan for goal-oriented action in the future

Table 4

Affordance implications for the design–reality gap in open e-learning

Affordance	Learnings from the ACT4ECO case
Informing	The informing affordance must cater to the diversity of users' backgrounds and learning profiles. Open e-learning must offer gradated levels of difficulty and complexity to appeal to a wide range of user abilities. Crucially, this requires information regarding previous knowledge, difficulty levels, outcomes and time required for completion to help users make informed decisions about the goals and actions they may wish to pursue
Assessment	The assessment affordance must strengthen both knowledgeable and less knowledgeable users' comprehension of possible actions through tailored quizzes. This provides a space for user reflection on different action outcomes and helps in bridging the gap with the informing affordances of open e-learning. Quiz designs should translate contents to everyday energy problem solving scenarios and offer possible courses of action
Synthesis and emphasis	The synthesis and emphasis affordances must support different user groups' internalisation of knowledge and manage their expectations. These affordances may in turn facilitate improved information quality as a key antecedent of user satisfaction in open e-learning
Clarity and learning pathway	Navigation must cater to users with different levels of tech savviness. Designing affordances for these different user groups can allow them to better achieve “flow” by concentrating on what they are doing, to the exclusion of other stimuli. The mix of difficulty levels ensures that previous knowledge and the completion of learning modules can be matched effectively; pathway affordances ensure seamless user progression in modules and across themes
Goal planning	The goal-planning affordance must transform the learnings of different users into possibilities for action, while being cognisant of the resources available to them. This fosters symbolic expressions which users then attach to an open e-learning platform. Each module should end in a call to action that helps users evaluate short and long term implications

Table A1

Overview of the Irish and Finnish participants; n (%)

Demographics		Ireland	Finland
Age range	18–24 years	0 (0)	5 (38)
	25–34	6 (40)	5 (38)
	35–44	3 (20)	1 (8)
	45–54	1 (7)	1 (8)
	55–64	4 (26)	1 (8)
	65+	1 (7)	0 (0)
Gender	Female	8 (53)	10 (76)
	Male	7 (47)	3 (24)

Background of Irish participants	Background of Finnish participants
• Administrator • Tradesman • Unemployed • Course designer • Retired • 3 University lecturers • IT salesperson • Homemaker • 2 researchers/PhD students • Nurse • English teacher • Programme director	• Researcher (economics) • Sustainability expert • Researcher (retired) • 10 university students

Table A2

Act4Eco survey instrument

Topics	Scales
Location and living
1. Where do you live?	1. Town/City/Countryside/Coast
2. What type of home do you live in?	2. Flat/Shared Accommodation/Terraced House/Semi-Detached/Detached
3. Who owns the property?	3. Private Landlord/Company/Me or Spouse/Relative
4. How is your home heated?	4. Electric/Gas/Oil
5. Other please specify?	5. (Free text)
Prior knowledge
6. Prior to using ACT4ECO, how would you rate your knowledge of energy efficiency?	6. Very Good – Very Poor
7. How would you rate your level of energy-awareness? (e.g. interest in home energy use, energy bills)	7. Very Good – Very Poor
8. How would you rate your technology skills?	8. Very Good – Very Poor
9. Any other comments?	9. (Free text)
e-learning and Act4Eco evaluation
10. The design of Act4Eco is appealing	10. Strongly Agree – Strongly Disagree
11. The Act4Eco homepage has a clear layout	11. Strongly Agree – Strongly Disagree
12. The sign-in for Act4Eco is straightforward	12. Strongly Agree – Strongly Disagree
13. How well did the content correspond to your needs?	13. Very well – Not at all
14. Regarding quizzes, how would you categorise the level of difficulty?	14. Very Easy – Very Difficult
15. What other improvements would you suggest for the platform, regarding both content and technical aspects?	15. (Free Text)
Motivation and changes
16. Did ACT4ECO inspire you to implement any changes in terms of energy consumption?	16. Yes – No
17. Can you tell us more about why act4eco did not inspire or help you make changes in terms of energy consumption?	17. (Free text)
18. What changes made by you were inspired by ACT4ECO?	18. (Free Text)
19. How motivated are you to maintain these changes or new behaviours in the future?	19. Very motivated – Not at all Motivated
Upscaling
20. Would you recommend the ACT4ECO platform to other people?	20. Yes, to friends/family/work colleagues – No

Appendix 1

Appendix 2

Appendix 3: Sample of prompts for focus groups.

1. What do you see/think/feel about the contents in Action section?

2. What helped to keep your attention during your use of ACT4ECO?

3. What made you disengage from ACT4ECO?

4. Did the content include relevant and real-life examples?

5. Which part of ACT4ECO was the most challenging for you?

6. How easy did you find it to sign into ACT4ECO?

7. What additional supports would you recommend?

8. What did you learn from ACT4ECO today?

9. Do you plan to make any changes at home to save energy after today?

10. Who do you feel you will benefit the most from using the Act4Eco platform?