INTRODUCTION

The past two decades have witnessed an increased recognition of the positive contribution sport can make in a development context (Collison et al., 2019; Darnell, 2012; Kidd, 2008; Levermore & Beacom, 2009; Morgan et al., 2021; Schulenkorf et al., 2016). Effectively organized sport activities create an environment where, apart from obvious health benefits, children can learn, play, and interact in ways that positively contribute to their physical, emotional, and social development. However, providing evidence for such multidimensional impacts can be challenging. Consequently, the increased recognition of sport for development (SFD) has been accompanied by calls for greater evidence regarding the impact of these interventions (Adams & Harris, 2014; Coalter, 2007; Whitley et al., 2020), which, in turn, has generated an increasing focus on monitoring and evaluation (M&E) methods that may achieve this.

Numerous stakeholders, including researchers, policymakers, government officials, businesses, educational institutions, development agencies, Non-Governmental Organizations (NGOs)/Non-Profit Organizations (NPOs), and civil society have employed M&E to gather evidence on how initiatives function and to better understand their impact (UN, 2021). Within the field of SFD, the United Nations (UN) provides a useful framework for defining the scope, purpose, and usefulness of M&E. In their Sport for Development and Peace Monitoring and Evaluation Toolkit (2021), the UN differentiates monitoring from evaluation. Monitoring refers to “the regular, systematic, collection and analysis of information related to a planned and agreed program or action [to provide] evidence of the extent to which the program is being delivered as intended” (UN, 2021, pg. 7). Evaluation in contrast refers to

a systematic assessment with a high level of impartiality (avoiding conflict of interest and biases) of an activity, project, program, strategy, policy, theme, operational area of institutional performance, amongst others. [To] provide information that is credible, reliable and useful […] to determine the relevance and fulfillment of objectives, development efficiency, effectiveness, [and] impact (UN, 2021, pg. 8).

Accordingly, evaluating empirically the multidimensional impact of a sports-based intervention focusing on vulnerable children in a development cooperation context requires a four-step approach: conceptualizing a context-specific and culture-sensitive data collection procedure; rigorous data collection that captures in multiple ways and across multiple events the program impacts; systematic analyses of the collected data that are suitable for the program, evaluation, and data collected; and a synthesis of the data analysis that reveals the program’s relevance, fulfillment of objectives, efficiency, and effectiveness (UN, 2021).

M&E Approaches to Collect and Analyze SFD Data

Traditional M&E approaches are usually divided into two types: quantitative methods, which typically employ various kinds of surveys and quasi-experiments, and qualitative methods, mainly based on interviews but sometimes combined with non-participant observations. Most M&E approaches, especially those employing qualitative research methods, rely on experiential reports from stakeholders involved directly or indirectly in the SFD program. Stakeholders providing experiential data include program beneficiaries as well as family members or care providers, intervention or program coordinators or administrators, and community members who are in some way involved with or impacted by the program. Methods used to collect experiential (self-)reports include informal conversations, formal one-to-one or group interviews, focus groups, or open-ended written responses, including experiential or reflection diaries or essays (UN, 2021).

Numerous challenges are associated with experiential (self-)reports, specifically if the intervention beneficiaries are children (Caqueo-Urízar et al., 2022; Datta & Pettigrew, 2013; Hunleth et al., 2022; Kooijman et al., 2022; Moore et al., 2015; Nixon et al., 2022). First, experiential or behavioral reports from beneficiaries or significant others tend to be subjective and prone to bias (Caqueo-Urízar et al., 2022; Kooijman et al., 2022). Second, children’s development is influenced by external factors beyond the scope of the program, which complicates assessment of a program’s impact (Moore et al., 2015). Third, the heterogeneity among program participants and the variability in degree and type of participation, especially in a development cooperation context, makes it difficult to define or assess reliably baseline, intervention conditions, and outcome measures (Datta & Pettigrew, 2013; Hunleth et al., 2022). Finally, the power relations between data collectors and intervention beneficiaries, especially between adults and children, introduces data collection and data quality limitations (Hunleth et al., 2022; Nixon et al., 2022). For example, when conducting interviews with children, efforts can be made to make the data collection procedure less intimidating, such as integrating a trusted third party into the interview situation, building rapport with the child, reducing formalisms, integrating props, or adapting body language and language register (Hunleth et al., 2022). However, even if these strategies have created an interview situation where children are able and willing to participate narratively, they may still seek to please the adult by saying what they think the adult wants to hear (Kutroavatz, 2017; Ponizovsky-Bergelson et al., 2019). Finally, it is unlikely that a child would be able to communicate the effectiveness of a program as it relates to relational, institutional, or other such abstract levels. Adult-child power relations may be exacerbated further when the adult is an outgroup member or external to the SFD activity. For instance, what a child shares with someone they are familiar with (a care provider, peer, or program coordinator) may differ significantly from what they might say to someone they are less familiar with – irrespective of the efforts made to build trust and rapport within a short interview period.

Quantitative approaches aim to overcome some of the biases and power dynamics associated with qualitative approaches. They tend to rely on a range of performance measures, standardized assessment tools, and surveys. Interventions that are less focused on the physiological benefits of an intervention frequently employ surveys, which may include quasi-experimental designs or baseline/pre-intervention and post-intervention measures, as well as field experiments (UN, 2021). Often designed to assess concrete development and program targets, these methods are valued for their ability to identify and measure change among SFD beneficiaries. If validated instruments, such as Rosenberg’s (1965) self-esteem scale or Smith et al.’s (2008) survey about resilience are employed, they may provide cost-effective, reliable, and often cross-culturally validated evidence of intervention success from a large number of respondents. While there are clear benefits in using established scales and surveys, they are not free from bias. Established scales, for example, require the focus of an intervention to be on what the scale was designed to measure. In addition, and like interviews, surveys are self-reports and can lead children (or adults) to knowingly or unknowingly misrepresent their knowledge, skills, and abilities (Dang et al., 2020). If an intervention aims at behavior or behavioral change, then most scales and surveys are limited to assessing behavioral reports, which diverge considerably from behaviors in many evaluation contexts. Finally, fundamental assumptions associated with survey research are that the items in the questionnaire pertain to the focus associated with the intervention, and that all respondents understand the questionnaire items, have the information required to respond to the items, are able and willing to respond to them truthfully, and do not differ significantly in how they decode the questions, irrespective of social group, developmental stage, religious affiliation and faith-based practices, etc. (i.e. respondents need to decode the questionnaire items the way the researcher intended them to be decoded, which assures that all respondents receive the same ‘stimuli’). Although efforts can be made to reduce potential biases associated with surveys (e.g., by using established scales or developing a valid instrument, training data collectors in data collection methods, establishing rapport and trust, reiterating the importance of honest feedback, granting anonymity and confidentiality, etc.), no data collection method is free of bias. Moreover, surveys require a certain level of literacy and degree of abstraction to understand and respond to a questionnaire, especially if it is designed to assess the impact of an intervention. Even if Computer Assisted Personal Interviewing (CAPI), Computer Assisted Telephone Interviewing (CATI), or Computer Assisted Web Interviewing (CAWI) is used to assist respondents with lower literacy levels, this approach can be time-consuming and might not be appropriate otherwise, depending on the characteristics of the SFD beneficiaries, the intervention, or the intervention context. Especially when self-reporting the effectiveness of an intervention by a beneficiary, such reports may tap into a subjective assessment or affective reaction, rather than reliable data pertaining to intervention outcomes.

Most of these challenges associated with traditional data collection methods are recognized within the SFD community (Levermore, 2011) and the broader non-profit sector (Innovation Network, 2016). Unsurprisingly, authors like Engelhardt (2019) have called on researchers and practitioners to be more open, and to experiment with different methods and tools (e.g., community mapping, drawings, storytelling, videos) to collect data that may be more effective in observing, evaluating, or monitoring the impact of SFD activities in a systematic and useful manner. We concur with Engelhardt, and a central component of this paper is to expand the repertoire of data collection methods to include observational and visual data for the evaluation of SFD activities for vulnerable children. The aim of our paper is to illustrate to SFD practitioners the potential of using these tools to systematically collect and analyze visual data for assessing the impact of an intervention.

Observational Methods and Visual Data Collection Methods in SFD

Participant or non-participant observational methods, conducted by an intervention coordinator or an external researcher, refer to a group of research methods that are based on data that, first, can be directly observed or experienced by a data collector and, second, that will be systematically analyzed and interpreted in line with what was observed and experienced (Ciesielska et al., 2017). Observational methods are much less frequently employed in M&E (Abbato, 2023; Borg, 2021), yet they are especially attractive because they have the potential to provide a direct way of assessing the impact of a program and its benefits. While relying heavily on the interpretation of the observed behavior, interviews and surveys are equally burdened by interpretation, as such data also has to be analyzed and interpreted. Observational methods may be as time and resource intensive as more conventional data collection methods, such as interviews (Westbrook & Woods, 2009). Resource constraints often limit the time spent in the field, which in turn restricts the scope and reliability of observational data. According to Boyko (2013), this delimits the contribution of observational methods as observational ‘snapshots’ are insufficient to assess the association between SFD program activities and goals. The ability to directly observe SFD activities from various angles before, during, and after an intervention nevertheless provides immense opportunities to capture nuances of SFD programs that are difficult to match through (self-)reports. Given the different advantages and disadvantages of experiential reports and observations, a combination of observational methods and experiential reports when studying the impact of an SFD intervention improves on the limits imposed by any single data collection method (Christensen et al., 2022; Holtrop et al., 2022; Gamarel et al., 2021; Odendaal et al., 2016).

Visual data collection methods, sometimes also referred to as art-based methods (McMahon, 2017), include methods that employ drawings, posters, maps, photos, and videos (Phoenix, 2010). While these methods have remained at the margins of social science for decades, they have become more popular (Jewitt, 2012; Phoenix, 2010) due to the so-called visual turn in the social sciences, the rise of social media, and advances in the analysis of non-numeric and non-textual data. Numerous studies have begun to explore the potential of using these methods in SFD: Kuhn (2003) and Noonan et al. (2016), for example, used drawings to explore children’s perceptions around physical activity, while van Ingen (2016) used paintings accompanied by text to explore the thoughts and experiences of survivors of gender-based violence during a boxing and art-based project. Sobotová et al. (2016) used participatory mapping to explore perceptions of security in relation to spaces where sport and art-based activities take place. Numerous studies integrate photography to capture and encourage individuals to express their thoughts and experiences of participating in sport-based activities (Hayhurst, 2017; McSweeney et al., 2022; Strachan & Davies, 2015), explore the barriers to participating in physical activity (Rivard, 2013), or document changes in children’s nutritional behavior (Bush et al., 2018). Video-based data has been used to collect and showcase stories from participants (Asadullah & Muñiz, 2015). Furthermore, Bean et al. (2018) assessed program quality in youth sport and recommended that future studies might also benefit from the recording of videos to support the observational process. Our work builds on this growing body of practices. We showcase the potential of two data collection methods that have shown significant promise in expanding the scope of M&E to monitor the impact of SFD activities and highlight the benefits for participating children.

Drawing as an M&E Data Collection Technique

Drawings provide children with a medium to express their thoughts and feelings (Kuhn, 2003; Moskal, 2017; Noonan et al., 2016). When evaluating the impact of SFD activities, children’s drawings can offer valuable insights into the reasons why children participate, what they like or dislike, and how they integrate intervention activities into their thought patterns. For instance, at the end of an activity, children can be encouraged to draw what they liked most about it. Subsequent conversations about their drawings, known as episodic interviewing, help clarify aspects of the activity that elicit the children’s interests and why they are interested in specific aspects. Such data can not only be used to study the impact of an activity but also how to improve this activity. Using drawings as a visual method has many advantages, particularly when working with children, as this method helps reduce power imbalances encountered in traditional interviews, especially because the child uses as their reference point the drawing, rather than the question by the interviewer (Søndergaard & Reventlow, 2019). Furthermore, the child at least partially replicates in their narrative associated with the drawing some thoughts and emotions they had while drawing the picture, rather than having to produce thoughts and emotions as an answer to an interview question (Rose, 2022; Water et al., 2018). Using a drawing as the focal point of a discussion is a child-friendly way to reduce potential stressors, helping to minimize the formality of capturing children’s thoughts, emotions, and experiences (Rose, 2022). Allowing children to explain their drawings in their own words can, furthermore, help to avoid misinterpretations (Kuhn, 2003; Noonan et al., 2016) and provides in-depth data for analyzing the impact of SFD activities. Additionally, utilizing children’s drawings safeguards the child’s perspective from being lost or taken out of context during the analysis process (Merriman & Guerin, 2012; Søndergaard & Reventlow, 2019). Apart from drawings as a visual data collection tool, we now turn to the utility of collecting video data for M&E.

Videos as an M&E Data Collection Technique

The widespread proliferation of smartphones and social media across the globe presents a unique opportunity for introducing visual methods of data collection for M&E of SFD programs and activities (Shaw et al., 2021). The ubiquity of smartphones has normalized the practice of taking pictures and making videos in everyday life, thus replacing traditional debates that emphasized the invasiveness and potential bias associated with this kind of data (Asan & Montague, 2014; Nassauer & Legewie, 2019; Robson, 2011). We have observed this in our own M&E activities. Despite initial concerns that video recording with a smartphone during SFD activities would influence the children’s behavior, their attention was quickly recaptured by the SFD activities, even in extreme field contexts such as orphanages and refugee camps. This could be due in part to the continuous presence of smartphones in the lives of children and the ability to collect high-quality video data using appropriately equipped smartphones, rather than employing bulky camera equipment. Rosenstein (2002) also notes that, through her experience of using video, both adults and children respond in a similar way if there is an in-person observation taking place: They are initially self-conscious of the camera or observer, but they soon forget that they are being observed. Video-based observations are even less distracting and intimidating if they are conducted by someone familiar to the child, which highlights the potential for involving SFD practitioners or coordinators in the data collection process (Asan & Montague, 2014). Our own fieldwork confirmed that almost all people involved in SFD activities, such as staff, coordinators, or coaches, use smartphones, which means that SFD staff could support data collection efforts alongside their SFD program roles. While this required training in data collection, it yielded greater buy-in to the M&E process, empowered staff, and improved the scope and quality of the data for the evaluation of the program.

When collecting video data involving children, ethical considerations deserve detailed discussion. It is important to acknowledge, for example, that there are several risks associated with digitalizing observational data (Facca et al., 2020; Rutanen et al., 2018). Video-based observations create a visual record of activities that cannot fully obscure individual identities, locations, and behaviors, even when findings are reported in a way that do not connect the individual to the evaluation and report (Facca et al., 2020; Rutanen et al., 2018). Therefore, obtaining consent is essential for any video data collection, which has its own set of challenges in a development cooperation context and with children (Facca et al., 2020). Additionally, because visual data can be easily copied and shared (Mok et al., 2015), organizations, practitioners, and researchers must take rigorous measures to protect the data, including the use of secure servers (Rutanen et al., 2018). Staff or volunteers supporting the data collection process should be well-trained in handling data to ensure privacy and data protection rights (Rutanen et al., 2018). This becomes even more crucial when working with vulnerable groups or on sensitive topics because disclosing data or revealing identities could jeopardize the participant’s safety and welfare (Rutanen et al., 2018).

In addition to data protection and privacy concerns, it is essential to consider the cultural appropriateness of collecting visual data, such as videos or photos (Schulenkorf et al., 2020). Ethical data collection requires embedding methods within the local context and cultural norms (Facca et al., 2020). This applies to all methods of data collection, but arguably more so when it comes to visual representations of people, behaviors, and personal or community spaces, which usually reveal much more than a simple tick on a questionnaire (Facca et al., 2020). This reveals an advantage of the method because culturally sensitive data collection requires a systematic and in-depth exploration of the ‘inside view’ to ensure interpretations are grounded in local cultures and embedded in participants’ perspectives (Denzin & Lincoln, 2017; Schulenkorf et al., 2020). Collecting visual data, especially using videos, allows us to move beyond isolated snapshot-observations. While not free from a culturally-biased gaze, they nevertheless allow us to more systematically study the cultural and contextual embeddedness of participants.

Another area of caution relates to the analysis process, which might have contributed to the limited use of videos in the context of M&E. Quick and easy video recording on smartphones often leads to the temptation of capturing numerous or lengthy videos to cover everything comprehensively. However, managing large amounts of unfocused data becomes time-consuming, requires substantial storage space, and can be expensive to review and analyze, while coding and analyzing video data can be challenging, necessitating training and expertise (Dash et al., 2019; Fan et al., 2014).

Drawing from our experiences, incorporating ethical data collection approaches, such as using video, audio, drawing, and storytelling with children for M&E of SFD programs, can significantly enhance the observation and evaluation process within the SFD context. Especially when expanding observational methods to include different visual data collection methods in SFD, there is the potential to overcome some of the limitations outlined earlier. Unlike most observational methods during field visits, which are often constrained in time and scope, video data allows for recording entire SFD activities across different sites and observation periods. Instead of reducing observations to field notes or predetermined observation grids, videos capture children’s participation in SFD activities as natural occurrences, providing insight into the child within their context, social interactions between the child and others (including other participating and non-participating children, care providers, intervention coordinators, community members, etc.), and the context within which the activity is taking place. This has multiple benefits, including overcoming the restricted scope of one-off field visits without requiring a substantial increase in time and resource investments. Multi-site, prolonged, and repeated visits could assist in observing changes, establish systematic connections between SFD activities and program goals, and mitigate researcher bias through collaborative reviews of video data by a combination of multiple researchers. Video data can also be reviewed and discussed in a participatory manner with practitioners and participants of the SFD program (Borg, 2021; Rosenstein, 2002). This approach helps to address potential misinterpretation by outsiders, while also fostering ownership, buy-in, making the M&E process more inclusive, and generating ideas for program improvement.

In sum, the need to observe, evaluate, and monitor the impact of SFD activities in a systematic and useful manner is growing. Traditional qualitative and quantitative approaches in M&E, such as interviews and surveys, face numerous challenges when applied in SFD programs involving children, especially in a development cooperation context. While observational methods offer a more direct way to assess the impact of SFD activities on children, they are frequently limited in time, scope, and necessity to interpret observations by outsiders. To address this, and in line with Engelhardt’s appeal to widen the scope of data collection methods in SFD (2019), we propose expanding the repertoire of M&E methods to include a toolkit that integrates visual and observational data collection methods as a complement to traditional evaluation methods, particularly in SFD contexts involving vulnerable children.

Frameworks to Assess the Impact of SFD Activities

Thus far, we have focused on the first component of M&E: the collection and analysis of data to observe, evaluate, and monitor the impact of SFD activities. This section is dedicated to exploring the second component of M&E, which relates to evaluation. As outlined earlier, the goal of evaluation is to employ a framework to assess the impact of SFD activities in relation to program goals in a way that is systematic, credible, reliable, and useful. In most instances, a clear line exists between the intended objectives and goals of a program and its activities, making an evaluation framework a natural outcome of clearly articulating and assessing this connection (Bao et al., 2015; Kusek & Rist, 2004).

Various frameworks have captured the wider benefits of SFD activities. For example, the Human Capital Model (Bailey et al., 2013; Whitehead et al., 2012) provided a valuable overview of physical, emotional, individual, social, financial, and intellectual benefits associated with physical activity, sports, and physical education. The UK based Sport for Development Coalition (2015) presented The Outcomes Model which identified four areas of outcomes associated with SFD activities i.e. social, emotional and cognitive capabilities; individual achievements and behaviors; interpersonal relationships; and benefits to society. Furthermore, the Commonwealth Secretariat (2020) developed a toolkit to measure the benefits of sport by creating direct links to the global Sustainable Development Goals. The above frameworks reinforce the desire and rationale within the SFD community to better capture the wider benefits of sport both at a local and international level.

The remainder of this paper will present an evaluation framework developed by Bergman & Bergman (2020) which complements the above models. This framework was designed using observational and visual methods. The goal was to better understand the expansive benefits children gain from SFD activities, and to support practitioners and researchers with a practical tool to collect and analyze these benefits.

Development and Application of the IRI-Framework

The Scort Foundation is a non-profit organization based in Switzerland that implements SFD programs in crisis and former conflict regions. Scort, together with local and international partners, train young adults (referred to as Young Coaches) in their respective countries who then deliver SFD activities for vulnerable children. Although Scort already assessed how Young Coaches benefit from this training (referred to as the Young Coach Education Program), there was a need to better understand how children benefit from the SFD activities delivered by these trained coaches. Although the positive impact of the SFD activities on children seemed obvious to Scort and other stakeholders, it was difficult to evidence the presumed benefits due to several challenges: First, Scort’s focus lies in training and assessing Young Coaches, with children being the indirect beneficiaries. Second, part of the data collection on the benefits children gain from Young Coaches SFD activities would need to be conducted through partner organizations or Young Coaches. Third, the involvement of various stakeholders globally (through partner organizations and Young Coaches) made standardized data collection impossible, given the sometimes weekly changes of intervention context within and between sites. Fourth, carrying out SFD programs in various crisis and former conflict regions introduces cultural and contextual dynamics that profoundly impact the form and function of each program. While the program’s adaptability to changing dynamics is one of its strengths, conducting a systematic M&E across different settings to understand its impact is challenging. Fifth, the composition of the participants change based on gender, migration background, ability, ambition, disability status, religious and cultural practices, etc. These changes differ not only between sites but also within sites and between sessions. To better understand how children benefit from participating in SFD activities, Scort needed an evaluation framework that could accommodate a wide variety of parameters while concurrently conceptualizing and assessing a broad set of possible benefits.

As a response to this set of requirements, the IRI-Framework was developed between 2018 and 2020 (Bergman & Bergman, 2020), and funded by Fondation Botnar. The development of the IRI-Framework began as a systematic review of the academic literature on intervention studies that use modern, non-conventional, non-survey-based evaluation methods, tools, and instruments when working with children in various development cooperation contexts or a combination thereof. Examples of non-conventional approaches that were reviewed include ethnographic studies, sociological life-world analyses, psychological studies on interactions with and between children, and systematic, direct observations (e.g., Ager et al., 2014; Barrios, 2014; Bell & Bell, 2017; Ben-Arieh, 2006; Johnston, 2008; Kuhn, 2003; Pfadenhauer, 2005; Sleijpen et al., 2016). A specific emphasis was placed on reviewing approaches that employ writing and drawing exercises, narrative and self-report measures, as well as projective techniques as data collection methods to explore the potential of developing episodic interviewing with children (for example, Backet-Milburn & McKie, 1999; Bland, 2018; Chorney et al., 2015; Deighton et al., 2014; Einarsdottir et al., 2009; Evans & Reilly, 1996; Halle & Darling-Churchill, 2016; Noonan et al., 2016). Behavioral observation strategies from the fields of education, child, social, and personality psychology were incorporated, along with measurement methods used in applied developmental psychology, pedagogy, ethnography, and needs and risk assessment approaches in community and refugee studies (see, for example, Ben-Arieh et al., 2001; Campbell et al., 2016; Gifford et al., 2007; Hart, 2009; Robinson et al., 2014; Tol et al., 2011; Volpe et al., 2005). Finally, reviews of studies in the field of sports sciences that use novel approaches to understanding how participation in sports benefits children and adolescents physically, psychologically, and socially (such as Eime et al., 2013; Holt et al., 2011; Whitley et al., 2016) completed this phase of framework development (see Bergman & Bergman, 2020 for a more detailed description of this phase). In a second step, the initial, literature-based framework was complemented with site visits to the Young Coaches’ activities to identify which aspects of child well-being could be observed in situ. The site visits were significant to the development of the evaluation because they allowed the initial IRI-Framework to be tested and refined in a variety of geographic, cultural, and national contexts. In total, local and international collaborators collected data during 27 observation periods in India, Mexico, Rwanda, Tanzania, and Uganda. While the first third of observation periods were used to supplement the evaluation scope of the IRI-Framework in terms of range, appropriateness, and usefulness, subsequent data collection periods, which included observational, visual, and interview data, were used to evaluate the benefits associated with the Young Coaches’ regular activities with children. Ultimately, the benefits captured by the IRI-Framework were consistently observable during SFD activities, irrespective of the local context within which they took place. The following is a summative overview of the IRI-Framework:

Table 1 – IRI-Framework Overview

The IRI-Framework identifies three core levels of benefits for children that relate to individual (e.g., cognitive, skills), relational (e.g., communication between peers, turn-taking, cooperation), and institutional (e.g., social norms, cultural values) benefits. Each of these are formed of at least four sub-categories as illustrated in Table 1. The benefits associated with each level can overlap, as can the benefits between the levels. At times, clear distinctions were made for conceptual, illustrative, and analytic purposes, despite the frequent interrelations between benefits. This also helps ensure that benefits may be explored separately and more systematically from different disciplinary or need-based perspectives in the future.

As the IRI-Framework presents a variety of observable benefits in SFD activities for children, it provides a useful tool that can support researchers and practitioners in systematically building, structuring, and refining their evaluation of SFD organizations. Below, we highlight several strategies of how this can be done.

The IRI-Framework to Guide and Complement Data Collection

Especially in combination with visual data collection, the IRI-Framework can be used to develop a focus for research, evaluation, monitoring, or implementation, and, accordingly, select and structure the data collection and analysis process. This can help reduce the amount of video recordings, given a clear focus between intention, data collection, and data analysis, thus streamlining a specific project. For example, a SFD program aiming to evaluate how a specific activity or coach impacts the children can use the IRI-Framework to select focal topics, such as how socialization and cooperation connect to relational benefits (Table 1, Column 2). Based on this focus, they can collect observational data by making short videos of the coach interacting with children, as well as interactions between peers. The IRI-Framework can then guide the analysis. Specifically, the researcher or practitioner can review the videos using sample questions, detailed in Table 2 of the IRI-Framework below (see Column 5, Rows 5 to 8) to prompt their analysis. Accordingly, questions such as: “How do the children support each other?” and “How do the children interact with the coach and with each other respectfully?” can be used to focus and structure the analysis. Equally, if a researcher or practitioner wanted to study individual benefits to the children (as listed in Column 1 of Table 1), they may decide to focus on cognitive and affective benefits by collecting video data similar to the process outlined above. In this case, their analysis would be guided by questions such as “What focuses or distracts children’s attention during activities?”, “Who or what refocuses children’s attention to the tasks?”, or “How do children internalize acquired knowledge or practices, and how do they disseminate this effectively to their peers?” (as shown in Table 2, Row 1, Column 5).

Table 2 – Selected Examples from the IRI-Framework

The IRI-Framework to Identify New or Unexpected Impact

While defining specific focal points a priori to guide data collection and analysis focuses a conventional evaluation, such a priories also narrow the scope, potentially missing unexpected elements that emerge during SFD activities. And it is not only new discoveries that make this form of assessment interesting. It is also well-known benefits, such as learning turn-taking, that can be studied in situ, for example as the establishment of justice over strength as a norm (institutional benefit) that leads to the practice and monitoring of turn-taking among peers (relational benefit). The IRI-Framework can also be used to explore the overall impact of an SFD program by assisting researchers and practitioners in identifying unanticipated benefits arising from a program’s activities or by adapting the program or practice to enhance certain benefits. During the evaluation of the Scort program, the IRI-Framework revealed several unexpected findings. These included the cascading positive impact on Young Coaches and the larger community, indicating that the benefits experienced by the children extended to bring about advantages for others as well. Established SFD activities within a communal space create an ecosystem that fosters inclusion by temporarily suspending differences in skill levels, ambition, age, gender, disability status, religious and cultural orientation, or social background among participants. This dynamic spilled over into the immediate environment, where older children congregated to watch the activities, parents and other care providers intermingled and coordinated future activities, and, at many SFD activity sites, small venders set up market stalls that attracted many members of the community just before, during, and just after the SFD activity. These examples illustrate how a broader application of the framework can help articulate more abstract and difficult-to-observe impacts, such as those related to institutional benefits, including language, norms, values, and community building.

The IRI-Framework to Enable Internal Learning

The framework can support the internal learning processes of a SFD program by using the video-based observations to understand how improvements could be made in its delivery. Building on the previous example of cognitive benefits relating to attention (Table 2, Row 1), researchers or practitioners could assess how a child’s experience and learning could be enhanced. A review of pertinent video material on attention could improve the design of the program itself, it could be used to assess and improve on how coaches deliver instructions and monitor specific activities, and it could help coaches through self-reflection to improve their own practice on developing attention in a group that diverges considerably in age, psychological development, ambition, ability, etc.

Using the IRI-Framework to Raise Awareness and Disseminate Findings

The IRI-Framework can also assist in selecting video scenes that not only add value to the collection and analysis of data but that can be used to raise awareness about SFD activities. In particular, videos provide a visual experience, bringing activities to life, and can more effectively convince parents, teachers, and funders of the associated benefits. For example, during the COVID-19 pandemic, video-based activities brought SFD activities into the homes of families around the world. One of our partners explained that “parents have been feeding back that they didn’t realize that the activities were so much more than [playing] football”. Furthermore, selected scenes can help disseminate findings about the benefits associated with activities to a wider audience. Written reports that often follow in-person observations, surveys, or interview findings, are not always accessible to audiences with low literacy levels, may not be translated into the local language, or could be inaccessible due to jargon. Thus, the use of videos can be engaging and informative and the IRI-Framework can help select which scenes could best reveal visible benefits, understandable to all.

The IRI-Framework to Make Explicit Connections between Monitoring, Evaluation, and Dissemination

The IRI-Framework is useful at all stages for video-based observations in a SFD context, including the selection of focal points, collection of data or selecting a subset from existing data that relates to the selected focal points, analyzing the data in relation to the focal points, and disseminating findings to different stakeholders, including funders, care providers, participating or not-yet participating children, NGOs/NPOs, or researchers. Furthermore, it can easily integrate visual methods such as children’s drawings. For instance, when analyzing the drawings and corresponding conversations with children, the framework can help associate observational, visual, and verbal data with specific benefits. When asked to draw what they liked most about that day’s activity, children participating in the activities delivered by Young Coaches often drew or talked about specific exercises, the skills they learned associated with the exercises, and how this helped them do well in the practice. Children thus highlight the learning impact of an activity (Table 2, Row 1), and this relates to a number of benefits, including retention, recall, awareness of accomplishment, awareness of the benefits of learning, and pride in accomplishment. When children draw players and explain that they enjoyed being a member of a team, even of a team that did not do well in a set of exercises on that day, the IRI-Framework can help reveal how children benefit from the activity in terms of peer-to-peer cooperation and friendship bonds, as indicated in Table 1, Column 2 and Table 2, Row F. Children often drew their team jersey or badge, and during conversations about their picture reinforced the sense of belonging and team aspiration that was fostered by team activities. This can be associated with a positive group identity and its benefits, as shown in Table 1, Column 3 and Table 2, bottom row.

In the analysis process, the IRI-Framework can help establish themes in drawings and conversations and relate them to associated benefits outlined in the framework. The effort of decoding and describing the impact of the activities can contribute to disseminating the benefits of an SFD program, including any communication to specific stakeholder groups or a wider public. For example, in a report, pairing a drawing with a key quote from the child can help explain the benefits the child enjoyed as part of their participation.

Conclusion and Recommendations

Most people involved in SFD programs are well aware of the immediate and overall benefit of participation. However, it is quite challenging to assess or communicate to relevant stakeholders the multidimensional nature of these benefits, especially with vulnerable children in a development cooperation context. This paper presents some of the limitations associated with traditional forms of data collection employed in M&E of SFD programs, such as surveys and interviews, and explores the potential of including more creative methods, such as visual and video-based observation methods. Specifically, we discuss how visual data collection methods, such as drawings and videos, could complement existing data collection methods in SFD contexts, especially when involving vulnerable children. The widespread proliferation of smartphones and social media presents a unique opportunity to introduce video-based observational methods. Similarly, the ubiquity of children’s enjoyment of drawing presents an excellent opportunity to include visual methods in M&E of SFD programs. As demonstrated in this paper, these methods are highly versatile and adaptable to data collection, analysis, and reporting in complex evaluation contexts. The integration of these approaches has the potential of researching new phenomena or new aspects of established phenomena, reach new audiences, strengthen research, evaluation, monitoring, and practices, and strengthen the understanding of the efficiency and effectiveness of sports for development well beyond the Global South or vulnerable populations. The employment of alternative methodological approaches has also proven to be effective in reducing the power relations between adults and children, which otherwise hinders the collection of reliable data.

Furthermore, the IRI-Framework, which explicitly identifies and helps to provide evidence for the multidimensional benefits of a complex SFD program, serves as a useful tool to achieve this. Beyond its immediate application, the IRI-Framework may serve as a guide for the conceptualization, data collection, and data analysis on how children and significant others may benefit from participating in SFD activities at intrapersonal, relational, and institutional levels. When used alongside visual or more conventional data collection methods, such as one-to-one or group interviews, focus groups, or surveys, the IRI-Framework has illustrated its potential to systematically gather evidence of the wide range of benefits that SFD activities have. Considering the scope of the IRI-Framework, it would be quite easy to adapt, pilot, and apply the framework to other target groups, including different age cohorts, psychological or physical disabilities, religious or cultural groups, etc. Furthermore, the IRI-Framework may also be a useful starting-point to explore programs and interventions beyond sports.

Overall, the IRI-Framework can help identify a study, intervention, evaluation, or monitoring activity, focus the data collection and analysis process based on this focus and activity, loosen conceptual limitations, sharpen observation skills, and structure dissemination and communication elements of a project.

The paper also makes an argument for the utilization of digital tools when presenting and reporting on a project. We are living in a highly visual and visualized world. Accordingly, observational and visual methods can make findings more appealing, accessible, and memorable to a wider audience. Visualizations in the form of exemplars or findings can raise awareness of SFD program successes, advocate for support, and promote coordination and networking between stakeholders and partners. Our fieldwork has also clearly demonstrated that visual evidence is highly effective with SFD participants, program staff, and program coordinators. Collecting, displaying, and illustrating findings visually provided information and feedback to program coordinators, staff, and participants in ways that are very difficult to achieve with tables, figures, and statistical coefficients.

However, we fully acknowledged that there are additional challenges when collecting and analyzing data beyond well-established scales and quasi-experimental designs. Beyond ethical concerns relating to consent, privacy, anonymity, confidentiality, and data protection, the obvious limitations of non-standardized procedures include difficulty of replication and well-trodden paths associated with validity concerns. This is not the place to rehearse yet again the advantages and disadvantages of qualitative versus quantitative research and evaluation designs. Instead, we hope to have made an important contribution to the qualitative evaluation of ambitious and complex SFD programs, and we hope that future M&E projects either explore the extensions proposed here or, going a step further, reflect on the relatively unchartered territory of mixed methods design, specifically to integrate in SFD programs and their evaluation video-based observations and visual methods. It is our hope that this paper will inspire ideas and foster debates about how individuals and organizations, even those with limited M&E expertise, can approach the evaluation of their activities or projects. We strongly advocate for the value of incorporating observational and visual data collection methods. By doing so, we believe that a broader range of stakeholders, especially practitioners, can better assess the impact and benefits of programs and initiatives.

Acknowledgements

It is important to acknowledge the various organizations and individuals in India, Mexico, Rwanda, Tanzania, and Uganda who have supported the data collection process and thus contributed to the development and testing of the framework and visual data collection methods.

Furthermore, the support provided by Fondation Botnar, the University of Basel, and the wider team at the Scort Foundation, which allowed the exploration of new approaches to assessing the benefits that children gain from the SFD activities around the globe is greatly appreciated.