Feminist Approaches to Monitoring, Evaluation and Learning

Background and Context

While Skateistan's work in Afghanistan has garnered the most attention from media and supporters within the SDP community, in this chapter, we focus on a 2016 training held at their Johannesburg (South Africa) facility. While Skateistan had been running skateboarding sessions in Johannesburg for several years, it was not until 2016 – and the opening of the newly built Skate School facility – that they could focus on building their educational programs and developing more systematic approaches to MEL. ¹ During the time of our collaboration, S-SA sought assistance on designing and integrating an MEL system that facilitated their reporting to external entities, aligned with their program's sensibility and design, and was transferable to their projects in Afghanistan and Cambodia. As feminist academics, we saw this as a unique opportunity to explore and re-imagine the way that MEL is understood, taught, and represented within SDP contexts. In this chapter, we utilize feminist decolonial theory in combination with an innovative narrative approach to encourage locally specific approaches to MEL that disrupt global SDP/SGD knowledge hierarchies and engage S-SA staff (SSA).

Drawing upon broad feminist principles and a decolonial approach, we encouraged a (MEL) workshop approach that allowed for the co-construction of knowledge about MEL practices and avoided a “top-down” mentality that positioned us (as academics) as sources of knowledge and the SSA as recipients. Such approaches are useful for reversing the paradigm by changing the sequence of typical training methods. This includes involving the intended beneficiaries and NGO staff (Mayoux & Chambers, 2005) in early discussions, decisions which are suggestive of moving toward a more participatory monitoring and evaluation strategy (Holte-McKenzie, Forde, & Theobald, 2006). In the case study presented in this chapter, we document our attempt to apply insights from feminist decolonial theory to both re-imagine the MEL workshop and to represent the findings/knowledge of this endeavor. In so doing, we hope this chapter makes a unique contribution to the growing body of literature on MEL in SDP, particularly in outlining how one might utilize insights from feminist decolonial theory to develop meaningful MEL trainings and practices. This chapter establishes insights into the MEL process that derive from the direct experiences of practitioners and academics and reveals the importance of considering feminist approaches to working in local contexts and with local communities, despite growing pressures from stakeholders and other key groups to “quantify” impact in particular ways.

Feminist Evaluation: Who Counts?

Feminist scholars have long questioned the dominance of particular forms of knowledge and challenged assumptions underpinning what “data” are and what ways of knowing count. They have often noted the exclusion of women's voices and knowledge, and this extends to the realm of evaluation as well. In the introduction of their special volume in New Directions for Evaluation, Sielbeck-Bowen, Brisolara, Seigart, Tischler, & Whitmore (2002) offer a corrective to this issue and outline six key understandings of feminist evaluation:

1. Feminist evaluation has as a central focus the gender inequities that lead to social injustice.

2. Discrimination or inequality based on gender is systemic and structural.

3. Evaluation is a political activity; the contexts in which evaluation operates are politicized; and the personal experiences, perspectives, and characteristics evaluators bring to evaluations (and with which we interact) lead to a particular political stance.

4. Knowledge is a powerful resource that serves an explicit or implicit purpose. Knowledge should be a resource of and for the people who create, hold, and share it. Consequently, the evaluation or research process can lead to significant negative or positive effects on the people involved in the evaluation/research.

5. Knowledge and values are culturally, socially, and temporally contingent. Knowledge is also filtered through the knower.

6. There are multiple ways of knowing; some ways are privileged over others (pp. 3–4).

This concise presentation has allowed for practitioners and scholars to apply feminist principles in their evaluation work although many readily acknowledge the challenges in doing this. For instance, Azevedo et al. (2019), a group of feminists working for Oxfam GB, reflect on their efforts to embed these values into their MEL work. In addition to considering the “successes and challenges” faced while utilizing MEL for gender equality, they also recognized the necessity to explore the “structural changes needed to bring about social changes through MEL and research more broadly” (Azevedo et al., 2019, p. 485).

Acknowledging these concerns, some feminist development scholars are working to advance approaches to MEL that can help highlight gender inequities. Writing in a special issue focused on gender and MEL, Bowman and Sweetman (2014) state:

MEL provides feminists with the means to explore the gendered impact of programmes and projects on the women and girls, and men and boys, whose lives are affected by it …. MEL can render development policymakers, practitioners and researchers accountable to the individuals and groups they aim to support, as well as accountable to the funders and supporters of that work.

(p. 201)

Yet how such data are gathered, what knowledge is prioritized, and what is done with evidence are deeply political issues imbued with gendered power relations. Such inquiries and issues are also highly relevant to MEL in SDP.

Though not adopting a focused gender lens, Nicholls, Giles, and Sethna (2011) raised comparable questions and concerns for the SDP sector, arguing that “the increasing call for positivist scientific evidence limits our ability to incorporate knowledge from grassroots practitioners into the policy development process” (p. 250). Conducting interviews with 17 SDP practitioners, they reveal how policy within SDP, as well as development discourses more broadly, tend to prioritize voices and expertise from the Global North (Nicholls et al., 2011). Moreover, they argue that one of the challenges to incorporating and amplifying voices from the Global South is a discourse of partnership which fails to consider the power dynamic embedded within donor-recipient dyads as well as colonial legacies around knowledge, voice, and expertise (Nicholls et al., 2011). This is one issue that could be mitigated by more attention to feminist principles in MEL, and examples do exist within SDP wherein partnerships appear to have carefully considered the power dynamics within the “donor-recipient-researcher” arrangement.

In their work with Go Sisters, an SDP project in Zambia focused on girls' empowerment, Kay, Mansfield, and Jeanes (2016) sought to create a space for reciprocal learning to proceed within their research project. They worked with the Go Sisters staff to help establish an M&E system that aligned with the requirements of their key donor and helped to develop the research capacity of the organization. Within the collaboration, they sought to imbue knowledge and insights from Go Sisters staff throughout the research and evaluation process (Museke, Namukanga, & Palmer-Felgate, 2016). One critical component of this was the need for qualitative approaches in order to monitor and evaluate the program's success and effectiveness. As Kay et al. (2016) note, critiques of qualitative methods in SDP within M&E “do not always distinguish between ‘stories’ compiled for promotional purposes and bona fide research-based accounts obtained through rigorous qualitative enquiry” (p. 224). They point to two examples wherein qualitative data and implicit challenges to the notion of objective and detached research afforded key learning opportunities for Go Sisters.

Over the past few years, some scholars have turned their attention to SDP practitioners to better understand their experiences and perspectives on M&E (Harris, 2018; Jeanes & Lindsey, 2014; Kaufman, Rosenbauer, & Moore, 2013). Kaufman et al. (2013) sought to understand how SFD practitioners in the Caribbean felt about M&E in order to assist others who seek to create effective M&E systems. Through interviews with five program coordinators, the following themes emerged which provided insight into the challenges faced by those tasked with M&E responsibilities: the need for capacity building; lack of sufficient time; difficult geographies; (the need for) technology and online tools; and passion for results. These concerns likely prove common to those working in the field, but as Harris and Adams (2016) suggest, we may not know them because (with noted exceptions) practitioners' perspectives on M&E are seldom sought. Moreover, Harris and Adams (2016) argue that

Practitioners have a limited voice in the discursive formation of evidence and to date have been unable to challenge dominant approaches to evidence. To what extent practitioners are docile bodies and represent dominant or alternative discourses are open to debate.

(p. 104)

In their estimation, SFD scholarship would be well served to learn from, and understand, practitioners' “relationship and view of evidence” (Harris & Adams, 2016, p. 104). Harris (2018) answers this call and further makes the case that practitioners are best suited to offer valuable insights into the M&E process because they are “entirely contextualized within their projects and are well placed to make sense of and reflect upon the extent to which any change is manifesting itself” (p. 3). To sum, much of this literature presents M&E as a critical element of SDP programming, especially for practitioners to report outcomes and demonstrate the effectiveness of SDP. It further shows how M&E “patterns of work” are increasingly becoming critical to the sustainability of the SDP field (McSweeney et al., forthcoming, p. 12).

In addition to the calls for more engagement with practitioners, scholars in both SDP and international development are considering broader questions of evidence and M&E in increasingly complex ways. For instance, Henne's (2017) recent work on the power of “indicator culture” implores SDP scholars to move away from an exclusive focus on human actors as the sole shapers of M&E to consider the ways that technologies hold sway in the MEL realm. This research encourages us to consider how indicators are agentic and become both “descriptive and instructive” when it comes to SDP projects and outcomes (Henne, 2017). Eyben, Guijt, Roche, and Shutt's (2015) work on The Big Push Forward (conference and initiatives) within the international development sector raised meaningful questions about the “results and evidence agenda.” Commencing in September 2010, the “Big Pushback” initiative consisted of an informal network of development practitioners interested in “constructive ways to advance conceptually and methodologically … beyond the narrow bureaucratic protocols that assume guaranteed predictable outcomes” (The Big Push Forward, n.d., “About,” para. 2). For Eyben, Guijt, Roche, and Shutt (2015), the results agenda is designed “to improve and manage development aid through protocols, procedures, and mechanisms for reporting, tracking, disbursing, appraising, and evaluating its effectiveness and impact” (p. 1). In a similar vein, other development scholars critique indicator culture in development writ large, suggesting that there is an urgent need to question “indicator culture” that is mostly driven by Global North actors in wealthy, industrialized nations, ensuring that “indicators emerge through social processes shaped by power relations, expertise, and techniques of measurement” (Merry, 2016, p. 25). Thus, Eyben, Guijt, Roche, and Shutt (2015) and Merry (2016) seek to move scholars and practitioners beyond a straightforward belief in the value of evidence. They work to identify “collective efforts to make development work for social justice” as opposed to top-down approaches to M&E (The Big Push Forward, n.d., “What We Are,” para. 6). They inspire new lines of questioning about MEL that are significant for work in SDP, such as: Who is asking for evidence? How is power associated with knowledge? Who is seen as a credible source or reporter of evidence? This chapter seeks to explore such questions and bridges the divide between academic and practitioner insights on M&E, in part, by foregrounding how local Skateistan practitioners and IS respond to various calls within the sector for evidence, results, and robust M&E systems.

Toward a Feminist Decolonial Approach to MEL: Framing the Case Study

As feminist researchers, we are generally highly critical of top-down approaches to training and education and thus approached the S-SA training with specific understandings about what this training could and should look like. Mainly, we sought to find ways to collaborate and work with international and local staff to ensure that the methods being developed consider local knowledge and are meaningful to the local program. In so doing, our theoretical orientation and methodological approach were guided by feminist decolonial theory and crystallization, which are complementary but distinct ways of unsettling notions and representations of knowledge and power. Feminist approaches to the decolonization of SDP also underpin our thinking about the politics, power relations, and potential of MEL (see Chapter 2 for more on decolonization in SDP). In an applied sense, we used insights from decolonial thinkers to inform how we constructed and delivered the workshop.

Our desire to engage in a more collaborative process of MEL training aligns with recent calls from academic writing in the SDP field (e.g., Sherry, Schulenkorf, Seal, Nicholson, & Hoye, 2017; Spaaij, Schulenkorf, Jeanes, & Oxford, 2018). While we do not claim that our approach to MEL training was participatory action research (PAR) in its purest sense, we included many of the typical sensibilities of PAR, which includes careful considerations of “participation, power, and reflexivity” (Spaaij et al., 2018, p. 27). As academics traveling to South Africa from the United States and Aotearoa New Zealand, we were especially mindful of embracing a “decolonial research consciousness that [was] … attentive to the productions, circuits, policing(s), and geopolitics of knowledge” throughout the entirety of this project (Murrey, 2017, p. 80). Outlining such sensibilities, Murrey (2017) suggests that this type of approach “refrains from claims to authority and challenges the positivist notions of objective knowledge …. [It] is an ethos that is questioning, humble, and grounded in the respectful turn and return to the voices and stories of people” (p. 80). Murrey's (2017) representation of their research in Cameroon included “de-privileged knowledge expression [by] including poetry, joke-telling, and narrative” to highlight how knowledge is co-created through exchanges as opposed to emanating from one (or more) particular academic experts (p. 80). We attempt to access this sensibility in the narrative “write-up” of this chapter.

Additionally, taking feminist critiques of evaluation and MEL to heart, we did not approach the workshop process as “experts” seeking to “empower” local staff with “our” knowledge of MEL. Instead, we sought to break down power relations and create space for dialogue and mutual learning between the authors and the staff, before, during, and after the workshop. We did this when we sought regular feedback on the training schedule, workbook materials, and the training itself. We intentionally co-designed a program that valued the multiple voices and perspectives of those involved in this process, even (and especially when) those voices diverged, revealed uncomfortable tensions, and offered insights that challenged our perceptions and understandings.

These practical approaches reflect the tenets of decolonial theory and its utility in SDP (Mendoza, 2015; also see Chapter 2). Mendoza (2015) offers a comprehensive overview of decolonial theory and its history. For our purposes, the central tenet of decolonial theory that we sought to utilize was the ways in which knowledge is (re)conceptualized. According to Mendoza (2015), colonialism and coloniality (the present-day manifestation of colonialism) purport that:

European knowledge production was accredited as the only valid knowledge, [and] indigenous epistemologies were relegated to the status of primitive superstition or destroyed. Eurocentrism locked intersubjective relations between the European and the non-European in a temporal frame that always positioned the European as more advanced.

(Decolonial theory, para. 6)

This matters within our project as we recognized the potential for (mis)understandings of who could and would be knowledge producers within this workshop and within this research. That is, as researchers, we traveled from outside South Africa with academic and geopolitical “credentials,” and we were likely perceived as bringing knowledge to the local staff. ² Aware of potential knowledge hierarchies, we tried to complicate such perceptions, as evidenced in both the structure of the workshop and the responses provided in the case study narrative section below.

We also draw insights from decolonial feminist scholar Lugones (2010) whose work affords us the opportunity to consider how “categorical, dichotomous, hierarchical logic” is vital to “modern, colonial, capitalist thinking about race, gender, and sexuality” (p. 741). In this case, we considered how this logic manifests in the establishment of categories, dichotomies, and hierarchies present in most SDP MEL as it relies on logic frameworks (logframes) to set its course. We did not necessarily seek to challenge the notion or viability of MEL through the course of the workshop, though this certainly could be a part of the application of decolonial theory. In this project we sought to build on Lugones' (2010) understanding that “feminism does not just provide an account of the oppression of women … it enable[s] women to understand their situation without succumbing to it” (p. 747). In a similar vein, within the workshop we sought to acknowledge the tensions around MEL that exist for the organization and the sector. Alongside this, we co-created space to identify resources for making the most of the industry expectations around M&E.

MEL in Skateistan-South Africa: A Narrative Approach

The following paragraphs offer important background information about the MEL workshop and the context of the case study that is the focus of the remainder of this chapter. In early 2016, Megan and Holly were involved in a meeting with the international Skateistan team in their headquarters in Berlin. This meeting emerged from Holly's research with Skateistan. The meeting considered the potential for Skateistan to develop MEL approaches that could meet the organization's needs and were consistent with their philosophies, as opposed to being steered by their funders (Win, 2004). Following this meeting, Megan engaged in regular virtual meetings with an IS member involved in program development across the three countries and another IS member who was recently appointed the role of managing M&E across the organization. Based on her experience of consulting with other organizations on these topics, she offered feedback on Skateistan's Performance Management Plan, theory of change, and other key organization documents. This pro bono consulting role included regular Skype meetings, written feedback, and productive questions on the aforementioned documents. These exchanges built the foundation for the eventual training workshop at S-SA.

Through the course of these encounters and the eventual workshop with S-SA, our assistance was sought to help Skateistan move from a focus on M&E to an approach that encouraged monitoring, evaluation, and organizational learning (MEL). The organization sought to make M&E more useful and meaningful for the organization instead of just fulfilling a requirement for donors; they hoped to learn and grow from this process. Skateistan staff wanted to embed organizational learning in the M&E process through collaboration with academics (the AT) as well as through considered involvement of IS and local practitioners. The vision for the workshop at S-SA was that staff would learn about M&E and be encouraged to think through their M&E options. Then, they would be equipped to make informed decisions about their MEL systems and design an approach that worked with and for them. To facilitate this, Megan and Holly were to acquaint SSA with a range of methods available for evaluating their work with youth. Importantly, we were not expected to design or implement M&E systems for the organization. We wanted to introduce a variety of ideas and options and to let those closest to the work decide on a path forward for the organization.

The Workshop

It was from both Megan and Holly's relationships with Skateistan that the opportunity to plan and deliver S-SA M&E training/workshop came about. They worked closely with international Skateistan staff to plan an eight-day workshop that would meet the needs of the organization and local staff. The preparations included the creation of a workbook that was used daily and that could be used for future training sessions at other locations. Closer to the date of the training, Thorpe was unable to travel to Johannesburg due to health reasons, one of her (then) PhD students (Nida Ahmad) stepped in to help facilitate the workshop. Following a series of research team meetings, in July 2016, Chawansky and Ahmad traveled to Johannesburg to facilitate the workshop. Their travel costs, accommodation, and a modest per diem were paid for by S-SA through grant funding that was allocated for M&E development.

The eight-day workshop included the following topics: interviews, observations, visual and creative methods, life history methods, curriculum-based methods, and focus groups. Each day of the workshop differed due to the on-the-ground demands of local staff, but most days included a lecture/discussion on a method, an applied activity that allowed practitioners to “try out” the methods and reflections on the viability of the method for S-SA. The reflections focused on cultural considerations and necessary modifications within the South African context. For most days during the eight-day program (see Table 1), four local staff members attended the training sessions, as did a member of the IS. On occasion, part-time or volunteer staff would participate in portions of the training, and we welcomed a dynamic, open space where people and ideas could flow in and out freely.

Chawansky and Ahmad worked closely throughout the workshop to prepare and then to reflect on the sessions. They submitted daily summaries of learnings and reflections to Thorpe to ensure she could remain engaged in the process and offer feedback. These summaries were also shared via email with IS to keep them informed of the workshop progress. Throughout the entire process, reflections from the authors occurred in person and via email or Skype with IS. Interviews and focus groups involving staff members occurred at the international headquarters of Skateistan as well as the S-SA offices. During the training, we collected data and reflections on the experiences for both the research projects and also used these moments as teaching tools. At the time of our workshop with S-SA, Skateistan was in the early stages of developing organizational MEL approaches and strategies, and the workshop and subsequent MEL efforts were part of this broader work. Since our first training session, the organization has continued to invest considerable time and energies in their MEL approaches, and in March 2017, they joined forces with a consortium of eight international NGOs to form the “Action Impact Network” – a group that is working together to “develop a series of indicators and data collection tools to measure the impact of youth action sport for development programs” (Skatesian, 2017, p. 36). This involvement reflects Skateistan's continued interest and investment in this area.

Crystallization of the Voices of MEL

This section summarizes the key learnings that emerged from our attempt to co-construct an M&E workshop and assist in the development of an organization's MEL practices while engaging with feminist decolonial perspectives. As noted above, our practical application of this theoretical orientation valued collaboration, local and organizational knowledge, and sought to break down expected knowledge hierarchies related to both the workshop and the embedded academic/practitioner dynamics. To organize this section, we offer creative and re-created conversations in response to a series of common questions about MEL within SDP. In our attempt to bring multiple voices into conversation, we offer (partial) answers to these questions by drawing upon interviews, email correspondence, daily summaries of the training sessions, and post-training reflection notes.

The voices of IS, SSA, or the AT are presented in a manner that complicates a singular and linear narrative about MEL. The IS includes those Skateistan staff members who worked across projects/locations on MEL while the SSA staff included the four participants who worked for the organization and attended the 2016 training. As the AT, we seek to avoid speaking over or for our collaborators or to write ourselves in as the experts. The lack of clear attribution to one named individual is intentional. The narratives that follow are intentionally provocative to add complexity and nuance to the discussion rather than seeking to prescribe a correct way to “do” MEL workshops and systems. Decolonization is ongoing, and in this understanding, it requires embracing the incompleteness and messiness of the process, and subsequently, of the learnings that have continued long after the workshop came to a close.

Conclusion

Life without MEL [training] is a bit weird because now we have all the cool and useful information and knowledge, but we don't know where to start. I won't lie, it's a bit nice not learning the whole day for 10 days [sic], but it was really nice having you guys coming from so far to share your expertise with us.

– SSA

We conclude this chapter with an unsolicited message offered by one SSA staff member to the AT. This email excerpt came to the AT after we had returned to our respective homes and captures some of the complex feelings we have about the workshop. The staff member suggests that getting back to daily life after an intense training on MEL can be an adjustment. Throughout the duration of our training we were conscious of the double duty that SSA managed. They worked hard to stay attentive and engaged during the MEL training, and we often adjusted our plans because of their other commitments and daily responsibilities. We changed plans on the fly to integrate unique learning opportunities and learned to sit back and watch when no adjustments were possible. We tried hard to challenge traditional knowledge hierarchies within the workshop through co-creating the workshop plan and the teaching style we imparted, but in the end, we were still lauded for “sharing our expertise” with the local staff. This post-training reflection perhaps indicates some of the limitations of our desired decolonial approach and of the structures we were up against.

Despite this, we remain committed to both further developing a decolonial approach and to assisting the organization in their MEL efforts. We have also maintained contact with Skateistan since completing the MEL workshop with S-SA. Thorpe continued her larger research project with the organization, and in June 2017, Chawansky observed the international Skateistan staff lead a MEL training in Cambodia. She served as a “critical friend” and provided constructive feedback to the staff running the sessions for their first time. The training attended to the local staff needs and differed in significant ways from the S-SA workshop, but stayed true to the spirit of trying to build MEL culture from within the organization. While Skateistan seeks some common tools and indicators across program locations, they are flexible enough to adjust and manage their expectations around MEL as it relates to local staff, language, experience, and cultural context. We believe that we took important steps in the S-SA workshop to help Skateistan in their efforts toward rethinking MEL. ³

Through this example we shared why and how one might decolonize elements of SDP, especially as it relates to transnational partnerships and relationships. We see this chapter as extending Oxford and Spaaij's (2019) use of decolonial feminist analysis insofar as it illustrates our attempt to actively decolonize the process of teaching, learning, and subsequently designing MEL for one organization. Our narrative approach sought to bring to life our quest to co-create knowledge and amplify local voices, especially when it concerns MEL. This topic (MEL) proves especially fruitful for this task as so many of the conversations we had with SSA alluded to the imposition of M&E from outside (funders). Though the voices and perspectives of funders were not included in an explicit sense in this research, their perspectives and voices pervaded the conversations we had before and during the workshop.

As a counter to this, our final section did not privilege our analysis or voices as another external imposition telling readers what or how to interpret the information presented. Taking inspiration from feminist critiques of evaluation and MEL, we attempted to challenge conventions of transnational knowledge production with our representational choice. In this way, we follow Mignolo (2009) who argues that “it is not enough to change the content of the conversation … it is of the essence to change the terms of the conversation” (italics included, p. 162). We believe this is one of the most valuable contributions of this chapter, and we hope it inspires further considerations of the politics of knowledge and representation in MEL and SDP. Using feminist decolonial theory as a guide to inform our workshop allowed us to co-construct knowledge in unique and unanticipated ways. In this instance, we suggest it helped us to recognize the opportunities and challenges that come with continuous reflection on power, privilege, and knowledge within MEL and the SDP sector. Using this approach does not guarantee success or specific outcomes; it is an ongoing project and process.

While decolonization is always an ongoing project, the quest to rethink how, where, and from whom knowledge originates is the cornerstone of this chapter. In so doing, this chapter offers a specific example of some of the key issues raised in Chapter 2. We hope the insights we share via our case study are useful to those working in MEL or researching MEL or SGD. The insights intend to encourage a rethinking of how one can “do” MEL and of what feminist principles can bring to the practice of MEL. The case study also extends recent critiques of how and why we evaluate our programs, what we “do” with data, and how to make it work for organizations. The long-standing feminist concerns around what “counts” as knowledge, whose voices are prioritized, and how such knowledge is represented (in reports or research) are equally valid in SGD. We believe this chapter offers a model and encouragement for those working in the sector to pay attention to the power relations and ethical considerations in such knowledge production processes. We also hope that our efforts to offer an alternative representational style – a narrative style that privileges multiple voices across time and space – encourage others to rethink the ways we write in our academic outputs as well as organizational reports. There is much potential in future feminist-inspired SGD work to explore alternative modes of representation that better reflect the seismic shifts still required to destabilize the patriarchal, neocolonialist, and Global North/South power relations that remain deeply entrenched in the field of SDP scholarship and practice.