Implementation and evaluation of a mobile application to support occupational therapy education

Abstract

Purpose

This study aims to evaluate the usability and effectiveness of a mobile application, ValTO, designed to enhance communication and professional reasoning skills in occupational therapy students through a case-based learning approach.

Design/methodology/approach

A descriptive exploratory study was conducted with 32 second-year occupational therapy students. The usability of the app was assessed using the Mobile Application Rating Scale and the System Usability Scale, complemented by the University of Oviedo’s Learning Satisfaction Survey.

Findings

The majority of students (77.8%) rated the app above average on the System Usability Scale, with 50% scoring it as excellent. Mobile Application Rating Scale scores revealed high ratings across functionality, aesthetics and information quality, with a significant correlation between app usability and user satisfaction. Increased student satisfaction was also observed in the Learning Satisfaction Survey compared to previous years.

Originality/value

ValTO integrates modern mobile health tools into occupational therapy education, enhancing students’ decision-making skills in an innovative, real-world context. This study contributes to the growing body of research on mobile health applications in educational settings, demonstrating their potential to improve both student engagement and learning outcomes.

Citation

Ruiz Fernández, M.L., Márquez-Álvarez, L.-J., Jiménez-Arberas, E. and Fernández Méndez, I. (2024), "Implementation and evaluation of a mobile application to support occupational therapy education", Irish Journal of Occupational Therapy, Vol. 52 No. 2, pp. 91-97. https://doi.org/10.1108/IJOT-03-2024-0005

Introduction

Clinical or professional reasoning in occupational therapy refers to the processes used by professionals to plan, manage, perform or contemplate their clients’ care. It is crucial for effective practice, as it influences professionals’ abilities in evaluation, planning and intervention. Recognized as pivotal in occupational therapy practice, professional reasoning profoundly impacts professionals’ competence in navigating complex domains such as evaluation, strategic planning, intervention implementation, problem-solving and decision-making (Cohn, 1991; Fleming, 1991; Kristensen et al., 2012; Márquez-Álvarez et al., 2019; Robertson, 1996).

However, for university students with limited clinical experience, organizing and synthesizing information in the initial stages of case analysis can be challenging (Moruno-Miralles et al., 2020; Rodríguez-Bailón et al., 2021). The way students manage their knowledge to analyze and summarize information in the early stages of a case is fundamental to acquire proper reasoning skills in academic education. Unlike experienced professionals who can readily identify critical cues and patterns, students may struggle to differentiate and prioritize information, often getting lost in the details (Chapparo and Ranka, 2008; Márquez-Álvarez, 2021; Moruno-Miralles et al., 2020; Pitonyak et al., 2020).

Case-based learning (CBL) has emerged as an effective methodology to address this challenge. This component of problem-based learning emphasizes the resolution of clinical problems presented as case studies. Students are presented with a variety of case formats including paper or video and simulated or real client cases to promote the development of clinical reasoning (VanLeit, 1995). This approach not only offers a reference case but also bridges the gap between theoretical intervention and clinical practice (Kamat et al., 2012; Kaur et al., 2020; Mahajan et al., 2016). Additionally, CBL promotes increased student autonomy and a departure from passive teaching methods, leading to more meaningful and long-term learning experiences (Ghosh, 2007; Kaur et al., 2020; McLean, 2016).

Understanding and improving problem-solving and decision-making skills have become paramount, with recent insights emphasizing the identification of indicators through thorough data analysis as a foundation for innovative approaches to evaluation and intervention planning (Doyle et al., 2014; Harries et al., 2012; Maruyama et al., 2021; Rassafiani et al., 2009; Schell and Schell, 2017; Unsworth and Baker, 2016). To address this need, the Occupational Therapy degree at the Faculty Padre Ossó has integrated these concepts into its curriculum through initiatives like the Bridging Horizons through Occupational Therapy project.

This project, which encourages voluntary participation from the first year, allows students and professors to collaborate on real cases in international cooperation, working with health centers in resource-limited countries in the global south (Jiménez-Arberas et al., 2021). By following the CBL philosophy, students engaged in this project acquire and apply a wide range of general and specific skills from the occupational therapy program, directly enhancing their professional reasoning.

However, the geographical distance and the complexity of international cooperation presented challenges that called for innovative solutions. As modern technology continuously provides new ways to reach clients (Hladik et al., 2023; Mitchell et al., 2023; Rhodus et al., 2023; Tkach and Earwood, 2023; Wielandt and Taylor, 2010), the development of a mobile application, ValTO, is a strategic response to overcome these barriers and enhance the educational value of the project. This app, available on Google Play, was designed to support the Bridging Horizons through Occupational Therapy project. It facilitates international tele-rehabilitation services by allowing students and professors to conduct evaluations and interventions with real-time feedback in a secure and confidential platform.

The creation of ValTO not only strengthens the project’s capacity to provide effective tele-rehabilitation but also expands its educational reach. The app offers a platform where students can engage in CBL more dynamically, with immediate feedback and direct communication with faculty. This integration of technology into the classroom enhances students’ ability to manage cases and ensures that their learning is both personalized and relevant to the challenges they will face.

The primary objective of this study is to assess the usability of the ValTO app as a valuable and innovative resource for occupational therapy education.

Material and methods

This study used a descriptive, exploratory quantitative methodology to obtain objective, measurable data regarding the performance and usability of the ValTO app. Given the educational nature of the study and the need to evaluate specific outcomes, such as app functionality and student satisfaction, a quantitative approach was ideal for systematically gathering information on key indicators. Although the sample size was relatively small, the quantitative design provided a structured framework to assess usability, app quality and student engagement using validated scales, ensuring the reliability and reproducibility of the results. The project obtained the approval of the Ethics Committee of the Principado de Asturias Public Health Services (ethics approval number code CEImPA 2020.091).

Participants

The participants consisted of students enrolled in the Geriatrics, Bone and Joint Pathologies course, which is part of their Occupational Therapy degree at the University of Oviedo during their second year. This course, valued at 6 ECTS credits, offers a foundational understanding of clinical and medical pathology, laying the groundwork for acquiring specific knowledge in occupational therapy within this domain. Inclusion criteria required students to be enrolled in the course during the academic year, actively participating in both theoretical and practical components and having access to an Android mobile device, as the ValTO app is currently only available on Google Play. Students who opted out of data collection, either because of personal choice or lack of consent, were excluded from the study sample.

The final sample was made up of a total of n = 32 students in the course. The participants were n = 27 women (84.4%) and n = 5 men (15.6%), with a mean age of 20.67 years (SD = 1.78). Only n = 9 of them (28.1%) had previously participated in the international cooperative project.

Measures

The performance and functionality of the app were assessed using two standardized scales: the Mobile Application Rating Scale (MARS) (Terhorst et al., 2020) and the System Usability Scale (SUS) (Vlachogianni and Tselios, 2021). These assessments primarily focused on evaluating app usability and user experience.

MARS is a widely recognized scale for assessing the quality and content of health science-related applications. It measures four distinct dimensions: engagement, functionality, aesthetics and quality of information. Each subscale is scored on a scale from 1 to 5, where higher scores indicate better performance in the respective dimension. The scale’s original validation studies demonstrated strong validity and reliability (α > 0.8) for both the overall scale and its individual subscales (Stoyanov et al., 2015).

The SUS is a quick and valid tool for assessing usability. It consists of ten items that students rate on a five-point Likert scale, ranging from “strongly disagree” (1) to “strongly agree” (5). The final SUS score, ranging from 0 to 100, corresponds to a percentile ranking based on established criteria. Systems with mean scores below 68 are considered below average, while those above 78 are rated as excellent. The research by Bangor et al. (2008) demonstrated that the SUS has a high Cronbach’s alpha, above 0.90, which indicates excellent reliability for this type of usability measure.

For the educational context, we relied on the University of Oviedo’s Learning Satisfaction Survey (LSS), administered at the end of each academic year (Unidad Técnica de Calidad de la Universidad de Oviedo, 2022). This survey gauges students’ perceptions regarding various aspects of their academic experience, including satisfaction with course materials, student guidance, teaching quality, course content and practical components. The use of the LSS allows us to compare student scores across different academic years, as it is a standardized survey provided by the university to assess the overall quality of teaching. As the score obtained reflects a global evaluation and is completed by all students anonymously, it ensures the confidentiality and validity of the responses, enabling reliable comparisons between cohorts. Additionally, the survey has a high reliability, with a Cronbach’s alpha of α = 0.899 (CI: 0.791–0.936), ensuring that the data obtained are consistent and valid.

Procedures

Before the app’s official introduction in the classroom, the ValTO application was made available to students participating in the Bridging Horizons through Occupational Therapy project. This early access provided an opportunity to test the app with real-world cases within the international cooperation framework. The objective was to later implement the app in the classroom environment, where students could evaluate its usability and effectiveness in an educational context.

To facilitate this, all participating students received training on CBL and a comprehensive introduction to the app. The focus of this training was aligned with the traumatological pathway, which was central to the course content. This ensured that students were well-equipped to apply the app in their clinical assessments, using it to guide them through structured decision-making processes relevant to the course’s topics. The training included instruction on aspects such as creating decision trees for the app from the web server, registering new users, creating forms for data-driven decision-making and receiving and responding to feedback from the support team.

Once training was completed, students began identifying real-life cases within their immediate environment, involving individuals with joint or bone pathologies. The focus was on assessing limitations in activities of daily living. Each student was responsible for obtaining informed consent from their participants, ensuring all evaluations were conducted securely and confidentially. This was achieved by following the app’s internal privacy protocols, and the process was closely monitored by the course instructor to ensure ethical compliance.

Throughout the course, students used the app to evaluate their selected cases, following the structured traumatological pathway to assess any limitation. The teaching staff provided continuous support, acting as facilitators and prompting students with additional evaluation questions when necessary. This mirrored the cooperative, interactive model used in the international project, encouraging students to engage deeply with each case and to refine their clinical reasoning skills.

This process was carried out over the entire semester. At the end of the term, students were asked to complete an evaluation of the ValTO app, focusing on its usability and overall contribution to their learning experience.

Data analysis

Descriptive statistics were calculated to provide an overview of the MARS and SUS measurements. Bivariate analysis was conducted to explore relationships between app usability, app quality and the independent variables. Pearson’s correlation coefficient was used to assess relationships between continuous variables, while the chi-square test was applied to examine associations between categorical variables. Additionally, an analysis of variance was performed to control for the effect of participation in the Mali project on app-related data.

All statistical analyses were conducted using IBM SPSS v.25 and Jamovi v.2.4.7.0 to ensure the accuracy and reliability of the results.

Results

A total of 32 students participated in the study, with varying response rates across the measures. Of the total population, 100% (n = 32) completed the LSS, while 56.25% (n = 18) completed both the MARS and SUS.

The overall mean MARS score from the n = 18 evaluations, measuring the usability of the ValTO app within this project, was 4.343 (SD = 0.390). There were no discernible differences in mean evaluation scores between students with prior experience in the project and those without (Table 1).

Regarding the SUS evaluations, only n = 4 students (22.2%) rated the app lower than the mean percentile score (<68), around a quarter (n = 5; 27.8%) scored it as satisfactory, and half (n = 9; 50%) gave it an excellent rating, with percentiles above 80. Minimal difference was observed between those with prior project experience and those without (Table 1).

A comparison of percentile results underscored a distinct pattern of outcomes that emphasized the relationship between the SUS and MARS scales (Table 2). Higher SUS percentiles were associated with higher scores on the MARS subscales, indicating that increased app usability corresponded with improved ratings across MARS subscales. Pearson correlation analysis showed statistically significant relationships between the MARS Functionality, Aesthetic and Information Quality subscales and SUS scores (p < 0.01), with r = 0.558, r = 0.555 and r = 0.612, respectively.

The chi-square test (χ²) revealed no statistically significant relationship between prior project involvement and various qualitative scores of the SUS (χ² = 0.311; df = 2; and p = 0.856).

In terms of an analysis of variance, no significant differences (p > 0.05) were found in SUS and MARS mean scores based on participation in the international cooperation project or gender. Age was not analyzed deeply because of insufficient observations.

Results from the LSS indicated a notable increase in student satisfaction with the project compared to previous years (Table 3). Enhanced satisfaction was consistently observed across all areas, particularly in the educational program, student guidance, teaching and practical content.

Discussion

The primary aim of this study was to assess the usability of the ValTO app as a valuable and innovative resource for occupational therapy education. The findings demonstrated generally positive student responses to the app’s usability and quality, as indicated by the MARS and SUS scores, which align with previous studies on mobile health (mHealth) applications in educational settings.

The MARS score of 4.343 (SD = 0.390) reflects a high level of usability, functionality and aesthetic appeal. These results are consistent with other studies evaluating health-related mobile apps. For example, Rodrigues Cunha et al. (2021) found similar usability scores when integrating mHealth tools into occupational therapy curricula, highlighting the positive impact these tools can have on student engagement and learning outcomes. Comparable studies evaluating health applications have reported MARS scores ranging from 4.0 to 4.5, especially when focusing on apps used in education or clinical settings. This shows that the ValTO app performs on par with other educational health apps in terms of supporting student learning and user experience (LeBeau et al., 2019).

The results from the SUS scale also align with previous findings in mHealth research. With 50% of students rating the app as “excellent” and 27.8% as “satisfactory,” the app shows strong acceptance among users. Studies like Reynoldson et al. (2014) reported similarly high SUS ratings for health-care apps designed to facilitate clinical learning and patient management. High usability scores are often correlated with the simplicity and intuitive design of these apps, which appear to resonate with students in health-care programs (Zapata et al., 2015). However, the 22.2% of students who rated the app below average raise questions about engagement and relevance. Previous studies, such as one conducted by McCurdie et al. (2012), suggest that engagement is often the weaker link in health applications, and users may not find the app necessary unless it aligns closely with their educational goals.

Interestingly, the data revealed no significant differences in usability scores based on prior participation in international projects or gender. This mirrors findings from studies such as Johnson et al. (2021), where health students’ ratings of app usability were driven more by individual preferences and learning contexts than by demographic factors like gender or previous exposure to similar tools. This indicates that while prior experience with international cooperation might have been expected to influence outcomes, factors such as individual learning styles may play a more important role.

The LSS results indicated an increase in satisfaction across multiple areas compared to previous years, particularly in student guidance, teaching quality and practical content. This is consistent with the findings of other research that emphasizes the benefits of integrating technology into health education programs. For instance, Rodrigues Cunha et al. (2021) observed that mHealth applications tailored to educational settings tend to enhance student satisfaction, especially when the apps are used to supplement hands-on learning experiences. The integration of ValTO, which provides a dynamic learning environment, likely contributed to improved student perceptions of the course content and structure.

It is also worth noting that a small subset of students rated the ValTO app lower than average. This could be because of a variety of factors, such as students perceiving the app as less relevant to their specific educational needs or learning styles. Studies have shown that individual differences in perceived relevance and the actual utility of apps play a significant role in how they are evaluated by users (Lattie et al., 2019; Zapata et al., 2015). Thus, exploring strategies to increase the perceived relevance of the ValTO app and customizing its features to better align with the specific needs of occupational therapy students could improve its overall reception.

Limitations

This study provides valuable insights into the usability and educational impact of the ValTO app, but it is important to acknowledge its limitations.

First, the sample size of 32 students is relatively small, which limits the generalizability of the findings. A larger and more diverse sample would enhance the external validity. The focus on students enrolled in the Geriatrics, Bone and Joint Pathologies course also means that the app’s effectiveness in other courses or settings may vary. Additionally, this study was conducted at the University of Oviedo, which may affect the transferability of the results to other institutions with different contexts and student populations.

The methodology, primarily relying on quantitative self-reported measures, could be improved with a mixed-methods approach. This would provide both quantitative results and qualitative insights to refine the app’s interface and features.

Moreover, this study’s timeframe was limited to a single academic year, which restricts understanding of the app’s long-term impact on learning outcomes and student satisfaction. A longitudinal study over multiple years would provide more comprehensive insights. Additionally, a larger and more diverse sample would allow for more robust analyses, including multivariate techniques like regression analyses.

Despite these limitations, the results highlight the potential of the ValTO app as a tool for enhancing clinical reasoning skills and the overall educational experience in occupational therapy. Further exploration and refinement could unlock even greater benefits.

Conclusions

ValTO app shows considerable promise as an educational tool for occupational therapy students, particularly in increasing overall student satisfaction. The positive correlations between app usability and educational content quality underscore the importance of well-designed learning technologies. However, the findings also point to the need for continuous refinement, ensuring that the app remains adaptable to diverse learning preferences and remains relevant to all students, regardless of prior experience. As mHealth applications continue to proliferate in health sciences education, studies like this one provide valuable insights into optimizing these tools for maximal educational benefit.

Table 1.

Descriptive results for mobile application rating scale and system usability scale

MARS subscales		Mean	SD	Participation in bridging horizons		Mean	SD
Engagement		4.144	0.517	No		4.022	0.474
				Yes		4.267	0.557
Functionality		4.319	0.506	No		4.111	0.588
				Yes		4.528	0.317
Aesthetic		4.426	0.496	No		4.296	0.484
				Yes		4.556	0.500
Information quality		4.481	0.416	No		4.333	0.456
				Yes		4.630	0.331
Quality (global)		4.343	0.390	No		4.191	0.402
				Yes		4.495	0.330
Subjective quality		3.639	0.626	No		3.417	0.573
				Yes		3.861	0.626
SUS categories (percentile)	N (%)	Mean	SD	Participation in bridging horizons	N (%)	Mean	SD
Below the mean	22.2	61.875	2.394	No	2 (11.1)	61.250	1.768
(<68)				Yes	2(11.1)	62.500	3.536
Acceptable	27.8	74.500	2.092	No	2 (11.1)	73.333	1.443
(68–78)				Yes	3 (16.65)	76.250	1.768
Excellent	50.0	88.889	5.743	No	5 (27.8)	88.125	6.884
(>78)				Yes	4 (22.2)	89.500	5.420
Total	18(100)	78.889	12.012

Source: Table created by authors

Table 2.

Relations between participants’ system usability scale and mobile application rating scale scores

	SUS categories
MARS subscales	Below average		Satisfactory		Excellent
	Mean	SD	Mean	SD	Mean	SD
Engagement	3.813	0.688	3.150	0.487	3.833	0.573
Functionality	4.131	0.216	4.138	0.506	4.551	0.282
Aesthetic	4.208	0.370	4.433	0.535	4.630	0.331
Information quality	4.167	0.430	4.267	0.596	4.630	0.423
Quality (global)	4.000	0.456	4.050	0.597	4.611	0.309
Subjective quality	4.150	0.300	3.800	0.548	4.333	0.520

Source: Table created by authors

Table 3.

Progression of satisfaction before and after project implementation

	Previous results	Year 2021
Educational program	7.55	8.8
Student guidance	7.025	8.5
Teaching	8.625	9
Subject	8.45	8.8
Practical content	7.125	8

Source: Table created by authors