Using participant-oriented research in post-secondary mental health program development and evaluation

Abstract

Purpose

With the recent release of the National Standard for Mental Health and Well-Being for Post-Secondary Students, there is increased interest to integrate research and practice for mental health services on post-secondary campuses. Participant-oriented research is a useful framework to bridge this gap. This paper aims to describe the program development and evaluation process and reports challenges and lessons learned to inform future implementation strategies for similar endeavours.

Design/methodology/approach

A participant-oriented research approach was used to revise and evaluate an innovative interdisciplinary resilience program, entitled Roots of Resiliency, for post-secondary students.

Findings

This case analysis used the development and evaluation of Roots of Resiliency to demonstrate some of the strategies and challenges that exist for participant-oriented research related to mental health in the post-secondary context. Collaborative relationships among the various development team members contributed to an overall positive experience. Some challenges that others who work in post-secondary mental health field may consider include the need for content expertise, the ongoing need for communication among team members and the need for an effective system to give voice to all participants.

Originality/value

Any mental health program has a cultural component and is best co-developed by the particular students (e.g. indigenous students) who are to be served by the program. In this regard, the co-design and shared development and evaluation of the current mental health program is an example that can be emulated in other programs within the post-secondary context.

Citation

Thannhauser, J.E., Szeto, A.C.H., Dobson, K.S. and Nordstokke, D. (2022), "Using participant-oriented research in post-secondary mental health program development and evaluation", Journal of Public Mental Health, Vol. 21 No. 4, pp. 323-330. https://doi.org/10.1108/JPMH-05-2022-0047

The mental health landscape at post-secondary institutions is complex. Self-report surveys such as the National College Health Assessment (NCHA) (American College Health Association [ACHA], 2019a) have demonstrated a rise in reported mental illnesses in the past decade (Linden and Stuart, 2020). This trend has resulted from a confluence of several factors (Szeto et al., 2021; Szeto and Lindsay, 2021), including increased severity and complexity of mental health problems and increased availability of mental health resources. Further, increased mental health literacy and reduced stigma have increased the demand for mental health services for campus providers (Duffy et al., 2019). This increased demand has partially driven the need to implement enhanced programs, resources and initiatives on post-secondary campuses to better address student mental health needs.

Post-secondary institutions continually implement new initiatives to better support their students. Recent developments in post-secondary wellness centres include stepped care models (Centre for Innovation in Campus Mental Health, 2022; Cornish et al., 2017) and single-session counselling models (Shefet, 2018). Given the relatively new status of these models and despite promising initial research, more research is needed to understand their effectiveness and how these models fit within different types of post-secondary institutions (Cornish et al., 2017). One noted gap in the post-secondary mental health context is the lack of evaluation of mental health-related programs, resources and initiatives (Best Practices in Canadian Higher Education, 2019; Linden and Stuart, 2020; Szeto and Lindsay, 2021).

The National Standards of Canada for Mental Health and Well-being for Post-Secondary Students (Canadian Standards Association [CSA], 2020) is a recent voluntary standard to help develop and implement a holistic framework to address mental health needs at post-secondary institutions. The Standard enjoins institutions to adopt an upstream and preventative approach, address services and supports for acute mental health needs, and address organizational and structural factors that promote mental health problems or serve as barriers to care. The first stated core value in the Standard is being student-centred:

• Being student-centred means prioritizing the mental health and well-being of students and including them in the process of developing, implementing and evaluating the institution’s framework.

• Students are valuable partners and their opinions are required, valued and respected as vital expertise (p. 16).

Student centredness is woven throughout the Standard. For example, clause 5.4.6.5 states that student mental health supports should be designed and implemented in consultation with students and should take an approach that is “designed from a student-centred focus” (p. 32). Thus, students should be viewed not only as the recipients of services but also should be incorporated in the co-design and development of the programs that are offered.

A patient-oriented research approach lends itself well to the post-secondary mental health context, particularly given the call for increase evaluation and a student-centred focus. The intention of patient-oriented research is to actively engage and empower service users in research that directly impacts them and their communities. This engagement is expected to improve the acceptability, effectiveness, efficiency and delivery of healthcare services (Brett et al., 2014; Pauly et al., 2019). While the term “patient-oriented research” is the predominant term used by Canadian funding agencies (Canadian Institutes of Health Research [CIHR], 2019b), we intentionally chose the term “participant-oriented research” to reflect the post-secondary context of the current research and the active and collaborative relationship “with” students. The current case analysis discusses strategies and challenges to engage in practice-based, participant-oriented research in the post-secondary context. This report is not focused on results of the evaluation of any specific participant-oriented mental health intervention but on strategies and challenges to the evaluation of mental health programs in the post-secondary context.

Redevelopment and evaluation of mental health program

The current report has its basis in a participant-oriented project, designed to integrate literature and student experiences in the revision and evaluation of an existing program, entitled Roots of Resiliency, to strengthen its fit and effectiveness for students at this specific institution. Briefly, this program is focused on the mental health of diverse post-secondary students, with a particular consideration of indigenous students, that emphasizes the development of resilience as a core construct (Thannhauser, 2020). Program curriculum includes an introduction to resilience and opportunity to develop a personal resilience plan, physical literacy, healthy nutrition for a student lifestyle, social connection, stress management, spiritual connectedness and resilient thinking.

Program redevelopment

The program redevelopment phase included two student advisory committees, each composed of six volunteers representing undergraduate and graduate students, that reviewed and provided feedback on existing program content and structure. Each committee met for three 2-h consultation sessions with the program facilitators. One student advisory committee was composed of indigenous students to better align the program with indigenous ways of knowing, being and learning. Program review was completed collaboratively, allowing facilitators to engage in dialogue with and seek clarification from the advisory committee members. Kerrie Moore, a Métis-Cree Indigenous Knowledge Keeper, provided feedback on the content and structure of the curriculum. Indigenous knowledge keepers have been gifted with teachings by other elders or knowledge keepers, typically over years of mentorship and teaching, and play a central role in indigenous communities (Queen’s University, 2022).

The redevelopment phase resulted in a 7-week interactive and experiential program composed of seven 1.5-h sessions. Each session addresses a specific domain of wellness and integrates lifestyle enhancement strategies, social connection opportunities and mental health coping skills. Session facilitators all had a background in the post-secondary context and included a kinesiologist, nutritionist, faith and spirituality leaders, psychologist, social worker and a health promotion coordinator (Thannhauser, 2020).

Program evaluation

Program evaluation using validated measures was integrated with the delivery of Roots of Resiliency. The 9-item Patient Health Questionnaire (PHQ-9) (Kroenke et al., 2001) was used to assess self-reported symptoms of depression. The 7-item Generalized Anxiety Disorder (GAD-7) (Spitzer et al., 2006) scale was used to assess symptoms of anxiety. The Connor-Davidson Resilience Scale (CD-RISC) (Connor and Davidson, 2003) was used as a self-report measure of resilience. Quantitative measures were completed using paper forms at the beginning of the first session and at the end of the last session. Electronic versions of the quantitative measures were collected at 1-month and 3-months following program completion. Participants were also invited to participate in a focus group at the 1-month follow up evaluation period.

Research results

Participants were recruited from a large research university in Western Canada. Undergraduate and graduate students who self-identified as experiencing mild to moderate stress, depression and/or anxiety were eligible to participate in the program. No clinical diagnosis was required, and there was no other program pre-requisite. The study included three offerings of the program between September 2018 and April 2020. A total of 28 participants completed the program, and 15 participants completed data at all time points.

Quantitative data was analyzed using SPSS 26. To determine whether there was a reduction in anxiety and depression scores as well as an increase in resilience, repeated measures analysis of variance was utilized across the four time points for each of the variables of anxiety, depression and resilience. To protect against familywise error for the three repeated measures ANOVAs, a Bonferroni correction was applied (0.05/3 = 0.016). The results demonstrated that reductions in anxiety approached significance, depression did not appear to be reduced, and there were significant increases in resilience.

Qualitative data was analyzed using Braun and Clarke’s thematic analysis framework (Braun and Clarke, 2008; Clarke and Braun, 2017). Two primary themes were identified through the analysis. Participant development encapsulates the personal growth and learning experienced by the program participants. Participant development included three sub-themes: connecting, shifting mindset and enhancing coping. Program development captures participants’ feedback about participant engagement, program structure and program format. A detailed description of quantitative and qualitative evaluation results are published elsewhere (Thannhauser, 2020).

Scientific importance of study

Many post-secondary institutions implement evidence-based and evidence-informed interventions to support student mental health. Various factors encourage such developments, including the effort to focus resources on programs that are effective; the desire to apply for funding from government and donors, marketing to participants, encouraging a culture of evaluation, and more generally supporting the culture of research-intensive institutions. Despite such factors, rigorous evaluation of these programs is often lacking (Best Practices in Canadian Higher Education, 2019; Linden and Stuart, 2020; Szeto and Lindsay, 2021). Program evaluation helps to both understand the program’s efficacy, and to suggest program improvements. Program evaluation also helps post-secondary institutions decide how best to use their limited funding. Evaluations may further support requests for additional program funding from government and private sources.

Program evaluations offer an avenue to better promote a program to students through its effectiveness. Similarly, having program evaluation become a regular practice increases their credibility at post-secondary institutions, particularly those that are research-intensive. For example, professors may be more willing to direct students to wellness resources if they understand that programs are evaluated or evidence-based. Finally, the promotion of evaluated programs helps to induce other program implementers to evaluate their programs and support a culture of evaluation at post-secondary institutions (Szeto and Bruckner, 2019).

Strategies and challenges to research

Engaging post-secondary students in research process

Collaborating with students in the development and evaluation of Roots of Resiliency was critical for the program’s success. Recruiting volunteers for our advisory committees through existing student groups, such as Students’ Union representatives or Mental Health club members, contributed to a broader understanding of the student experience. These students are often familiar with the common issues and experiences faced by the student population because of the advocacy work they do on campus. We also intentionally recruited students from a range of programs and levels (i.e. undergraduate, graduate) to enhance the relevance of the program content and structure for a breadth of student experiences.

Anecdotally, many student counselling centres report difficulty retaining participation in mental health group programs. Participant-oriented research provides the opportunity to collaborate with students in the development of programs by students, for students. Collaboration throughout the planning and development of the program helped students to understand the relevance and value of participation in mental health group programming. These students also became advocates for the program and encouraged their peers to participate. Focus group feedback from program participants highlighted the importance of effective marketing, so students understand the relevance of mental health programs to their needs. Despite this success, retention continued to be a challenge throughout the program. Critical attrition points were following the first session and over the final three sessions, which aligned with the final weeks of classes prior to final exams. Participants provided feedback about the importance of fostering relationships among participants and explicitly connecting the relevance of the program to overall student success from the first session. Participants also highlighted the challenge of balancing their academic responsibilities with voluntary mental health programming. Opportunities to integrate mental health programming directly into students’ academic programs may increase access to mental health information and skills.

The student life-cycle posed both a challenge and an opportunity for the research process. First, our student research assistants (RAs) were senior undergraduate students. The graduation of one RA and hiring of a replacement RA caused some disruption to the program development process. However, engaging multiple RAs in the development and evaluation process also provided diverse student perspectives to inform the research process. Second, recruiting volunteers for the student advisory committees occurred during the summer months, which was challenging. However, targeted recruitment through relevant student groups (e.g. Indigenous Students’ Association) and the ability to offer a modest honorarium supported recruitment. Finally, the student life-cycle affected the collection of longitudinal data. For example, the 1-month follow up focus groups aligned with the start of either a new semester or spring semester when many students graduated or started summer employment. For many participants, this period of transition impeded their involvement in the focus groups. It was difficult to retain engagement with participants for evaluation purposes after program completion. Researchers are advised to consider the timing of follow-up evaluation and strategies to maintain engagement in the evaluation process following participants’ completion of the program.

Multi-disciplinary collaboration for mental health program development

Historically, mental health programs for post-secondary students are developed by mental health professionals and delivered to students. Canada’s Strategy for Patient-Oriented Research (SPOR) emphasizes a collaborative approach to research that includes all major stakeholders, such as patients, researchers, health practitioners and policy makers, in support of improved health care practices and outcomes (CIHR, 2019b). A core guiding principle of capacity development for SPOR is to “support a collaborative, interdisciplinary approach to patient-oriented research by fostering integration, respect and mentorship among patients, researchers, health practitioners, administrators, and policy-makers” (Canadian Institutes of Health Research, 2019a). With this principle in mind, we brought together program facilitators and the student advisory committee members to engage in a collaborative review of the program content and structure. To prioritize the student voice, the student advisory committee members first met with the student research assistant to discuss their understanding of resilience, barriers to resilience and resources to build resilience. The committee members later met with the session facilitators to discuss the structure and content. Facilitators engaged in conversations from a stance of openness and curiosity. Student members were actively engaged throughout and provided detailed and insightful feedback. Several students described the experience as empowering. Following the feedback from the student advisory committees, the program facilitators met several times to collaboratively engage in redevelopment of Roots of Resiliency. Specifically, the facilitators developed a shared understanding of the core tenets, common structure and key outcomes for the program. Critical to this collaboration were trust, respect, open communication, shared power and appreciation for the unique knowledge of each professional.

Building partnership with indigenous community

Information about the mental health experiences of indigenous students on Canadian campuses is unfortunately limited. Despite the limited available data from the 2019 NCHA survey (American College Health Association, 2019b), certain trends were apparent, including most indigenous students reported feeling overwhelmed by all that they had to do (91.5%), mentally exhausted (96%) and with overwhelming anxiety (80.9%). While indigenous students were more likely to access mental health support than the general student population, they were less likely to do so from university health or counselling services. This data spoke to the need for more culturally informed mental health services for indigenous students. One of the goals of this project was to engage in the indigenization of Roots of Resiliency in effort to honour the experience of indigenous people and to promote indigenous ways of knowing, teaching, learning and research.

In Canada, the Tri-Council Policy Statement on Ethical Conduct for Research Involving Humans (TCPS2) includes a chapter specifically dedicated to the issue of involving indigenous peoples in research (CIHR, Natural Sciences and Engineering Research Council of Canada [NSERC], and Social Sciences and Humanities Research Council [SSHRC], 2019). The aims of a participant-oriented research approach and the guidance provided in this article both highlight the importance of respectful relationships and a spirit of collaboration as foundational to ethical research. Noteworthy, the TCPS2 guidelines acknowledge “building reciprocal, trusting relationships will take time” [para. 6]. It became apparent early in the research process that the time required to build relationships with indigenous elders and knowledge keepers within the community does not often align with timelines imposed by funding agencies and the demand for timely innovation in academic environments. We were unable to build all of the relationships desired within the constraints of our project timeline. We have also become aware of the recommendations of the First Nations Information Governance Centre in regard to the ownership, control, access and possession of research data (First Nations Information Governance Centre, 2022), and the duties it proposes. Western models for research continue to impose unnecessary barriers to truly collaborative relationships with indigenous community members and may constrain the ability to learn from traditional knowledge in the co-creation of mental health programming for indigenous students.

To be culturally relevant, people who develop and evaluate programs in the post-secondary environment must understand the histories, experience and traditions of indigenous students. A true intercultural dialogue is essential to develop supportive and relevant programming. In this spirit, the program facilitators met with the indigenous students’ advisory committee to listen, learn and collaborate. The students emphasized the importance of an intercultural lens through which the content for the Roots of Resiliency program was evaluated. The students specifically recommended maintaining the general content and structure of the program, while also integrating indigenous cultural values and practices into the existing curriculum. For example, a discussion about traditional foods for students from diverse cultural backgrounds was added to the nutrition module, and the physical literacy module included examples of traditional games of indigenous peoples. Session facilitators also set up the physical learning spaces in circles, entering into dialogue with program participants as collaborators rather than experts to honour knowledge and experiences held by participants and facilitators. Finally, to support indigenous students with connecting to spirit prior to engaging in the program, they were invited to participate in cultural ceremonies relevant to the students’ traditions. This opportunity was extended to, but not imposed on, all program participants. Students on the indigenous advisory committee reported feeling valued in their roles and empowered to help develop mental health programming on campus. One committee member noted, “as an indigenous student, we have a lot of things that are developed for us, rather than with us. This development process centered our voices and made us feel we were included.”

Conclusions

This case analysis has used the development and evaluation of a specific program, the Roots of Resiliency, to demonstrate some of the strategies and challenges that exist for participant-oriented research related to mental health in the post-secondary context. This report demonstrates some of the best practices that can be garnered from sources such as Canada’s SPOR (CIHR, 2019b) and the TCPS2 (CIHR, NSERC, and SSHRC, 2019), but augments these more formal documents with experiences from an actual program development, implementation and evaluation process. The experiences of the development team were positive, in particular as a result of the collaborative relationships among the various people involved in the program development. This report also highlights some challenges and concerns that others who work in this field may consider. One of the notable aspects of the current program is that it was explicitly developed and intended for indigenous participants, but it can be argued that any program has a cultural component and is best co-developed by the particular set of students who are to be served by the program. Ultimately, the co-design and shared development and evaluation of any mental health program is encouraged within the post-secondary context.

Acknowledgements

The authors are grateful for the contributions of the many students and stakeholders who made the Roots of Resiliency program and this research possible, including: 1) student research assistants Tessa Neilson and Hayley Brillon, 2) participants in the general and indigenous student advisory committees, 3) indigenous knowledge keeper, Kerrie Moore, MSW, 4) Roots of Resiliency program facilitators and 5) the Roots of Resiliency program participants. This study was funded by a CIHR-SPOR grant, in partnership with the Rossy Family Foundation. The funding bodies were not involved in the design of the study; collection, analysis or interpretation of the data; or in writing the manuscript. Ethical approval has been obtained from the University of Calgary’s Conjoint Faculties Research Ethics Board (Certificate number REB14-1358). The authors declare that they have no competing interests.

Funding: Canadian Institutes of Health Research (397532).