FINDINGS AND DISCUSSION

Terminologies: Imprecisions in understanding, application and interplay

Out of our total sample of 204 publications, 21 specifically related to terminologies used in SFD research. Across these documents, the terms research, evaluation, M&E and more recently also MEL (Monitoring, Evaluation, Learning) or MERL (Monitoring, Evaluation, Research, Learning) are used with a wide range of variations and are often employed interchangeably. In short, there is no clear conceptual demarcation. ‘Conducting research’ is often equated with M&E activities, which inevitably leads to an undifferentiated discourse (Kay, 2012). Here, the rather simplistic merger of the terms monitoring and evaluation into a single entity is problematic, as the two research functions require specific approaches as they serve different aspects: “As they pose and respond to different types of questions, it is evident that monitoring and […] evaluation […] require different tools, different skills, different strategies and ideally different personnel“ (Kaufman et al., 2014, p. 177).

Numerous authors critically note that even well-established SFD scholars tend to equate research with program evaluation and they emphasize the need for a clear differentiation (Jeanes & Lindsey, 2014; Welty Peachey & Cohen, 2015). For Jeanes and Lindsey (2014, p. 199), M&E should be considered a “processes of research”. They see the distinction in the fact that M&E is mainly used in the program context and aims at optimization, while research goes beyond that. However, Whitley et al. (2020, p. 22) note that “evaluation and research are not mutually exclusive, and there are arguments that evaluation is a subset of research (and vice versa)”. Collison et al. (2019a, p. 6) concur that research simply cannot be separated from M&E processes:

We would argue that the assumption that research necessarily informs, guides, influences or even constructs M&E frameworks or evidence is misguided. Progressive research methodologies focused on M&E, for example participatory action research, may well serve the dual purpose of knowledge production while producing and assisting with the formal process of evidencing and reporting, but the relationship between these processes requires ongoing negotiation and reflexivity.

Overall, this review reveals that the current incoherence in the SFD research field stems from the imprecisions in the use of terminology and concepts. The absence of clarity concerning research, evaluation, and M&E terminologies inevitably leads to a variety of debates in SFD research, including the role of researchers (who carries out the research and under which conditions?), methodological procedures (which designs and methods are best suited?) and the overall objective of the research (is it merely about generating new knowledge, about making strategic decisions or about receiving funding?).

A starting point for addressing imprecision in the use of terminology and related discussions would be the establishment of a shared vocabulary with the intention of ‘finding a common language’ (Barisch-Fritz & Volk, 2016). The benefit of such a vocabulary – particularly in the context of interdisciplinary research – is the ability to overcome obstacles by creating a unified language based on mutual understanding and effective dialogue among researchers from a range of disciplines. However, achieving a shared understanding requires acknowledging and including external expertise from a variety of areas. Moreover, given the potpourri of approaches taken by researchers – including fundamental research, evaluation research, or evaluation based on monitored information, and the different objectives and strategies in place – active collaboration and clear communication are required to achieve overall consensus.

Within the described debates and underpinned by the results of our study, there remains a regrettable lack of acknowledgement and inclusion of traditional debates from the social sciences and development studies. In fact, it seems important to consider multi-perspective considerations, such as critical voices of – and relations between – researchers, evaluators, commissioners and donor organizations, to find some common ground. Instead, SFD is as heavily influenced by hegemonic discourses, particularly in relation to the concept of development. In fact, it is widely acknowledged that the conceptualization of development presents one of the foremost challenges in the field of SFD (Hartmann & Kwauk, 2011). Accordingly, the field of SFD has encountered similar debates and policy challenges to those reported in wider development studies (Darnell & Black, 2011). Overall, it is suggested that unless the SFD research community addresses the described ambiguities and establishes crucial distinctions and a common language through active engagement and deliberation, progress towards terminological and conceptual clarity will be difficult to achieve.

Approaches and classifications: Many use what they know, few use what is established

Before the different sub-categories from the ERF are discussed (approaches based on scope, theory-driven and participatory approaches, designs and methods), general theorization processes in SFD research are highlighted first. This is done to provide the wider context and to ‘couch’ the applied findings related to approaches and classifications. Due to their importance in SFD research, the aspects of impacts and sustainability – which from an evaluation-theoretical perspective are subject of ex-post evaluations – are considered separately.

The theorization of SFD

Our scoping study identified 127 publications that engaged with aspects of theorization in SFD. Our review revealed that a lack of a theoretical foundation for research in the field of SFD was an early warning raised by numerous authors and that over the years, the demand for its establishment and the call for connectivity to other disciplines has only intensified (e.g., Coalter, 2013b; Darnell et al., 2019; Lyras & Welty Peachey, 2011; Massey & Whitley, 2019; Siefken, 2022; Welty Peachey & Cohen, 2015; Zanotti & Stephenson, 2019). Despite a stated lack of theoretical grounding and the scholarly verdict that “much work remains to be done” (Zanotti & Stephenson, 2019, p. 172), it should be noted that a number of significant theoretical and conceptual developments have taken place over the past decade (Darnell et al., 2019; Massey & Whitley, 2019; Schulenkorf et al., 2016). Specifically, the contributions in the Routledge Handbook of Sport for Development and Peace (Collison et al., 2019b), as well as Welty Peachey et al.’s (2021) meta-analysis of theoretical advances in SFD illustrate the research community’s willingness and associated attempts to develop, employ and advance theoretical approaches.

Given the need to advance theoretical and conceptual thinking in SFD, a number of studies have employed a distinct sport-focused model of theoretical framing. Here, Welty Peachey et al. (2021) have identified the Sport for Development Theory (SFDT) (Lyras & Welty Peachey, 2011), the S4D Framework (Schulenkorf, 2012), Sugden’s Ripple Effect Model (2014), Coalter’s Program Theory (2013b) and Schulenkorf and Siefken’s (2019) Sport-for-Health Model as relevant examples. However, to date, most of these are hardly used to guide or support other studies which speaks to the relative infancy of SFD theory and the need to do more and better in an attempt to truly legitimize the field (Welty Peachey et al., 2021).

Meanwhile, where SFD studies have been underpinned by a derivate model of theoretical framing (see Chalip, 2006), the concepts of social capital and Positive Youth Development (PYD) theories have most commonly been employed in the context of SFD (Schulenkorf et al., 2016). However, despite their popularity, Darnell, Chawansky and colleagues (2018, pp. 138-139) describe these approaches as “relatively neutral” and “apolitical” and are calling for more critical SFD research that uses politicizing approaches (e.g., postcolonial or feminist), including those “that draw attention to the roots of inequality”.

As part of this development towards critical scholarship, academics from different disciplinary backgrounds have started to compile their varied theoretical approaches and brainstorm possibilities for transdisciplinary research in SFD. Here, a special issue in the journal Social Inclusion has opened transdisciplinary and intersectoral perspectives by providing a selection of articles that bring together various disciplinary streams (Delheye et al., 2020). Moreover, a recent journal article in JSFD has described selected disciplinary trends from the fields of sport sociology, social anthropology, sport management, public health, leisure, sport pedagogy, and sport psychology and provided critical avenues for transdisciplinary engagement (Delheye et al., 2020). Further, Siefken et al. (2022) emphasized the necessity to connect physical activity research with SFD, as highlighted in their recently published edited volume addressing opportunities and challenges in low- and middle-income countries. The call for cross-disciplinary and interdisciplinary work has been made. The integration of these perspectives certainly provides first important steps towards interdisciplinarity modifiable; however, thus far the selection of viewpoints has largely been based on the research backgrounds of the contributing authors of the articles published and a concerted effort to include all disciplines of the SFD field is yet to be realized.

In order to foster true interdisciplinarity, it is essential to ‘identify shared problem perspectives’ (Barisch-Fritz & Volk, 2016). Here, Whitley et al. (2022, p. 9) emphasize that common interests between different disciplines include life skills development and transfer, as well as the parallels between PYD theory and “the anthropological examination of youth in SFD”. Furthermore, the wide-ranging orientation of SFD organizations towards the Sustainable Development Goals and related issues such as social inequality, environment, safeguarding, refugees, and social entrepreneurship (Giulianotti et al., 2019) could form common denominators.

Approaches based on scope

The scope of research studies varies considerably across academic domains and our review revealed that this is no different in the SFD space. Overall, we identified 18 publications related to scope-based approaches. Specifically, after an initial focus on micro-level case studies and first attempts to ‘map the field’ (e.g., Hillyer et al., 2011), SFD researchers have now embarked on the next level of systematic reviews and assessments. In this context, Darnell, Chawansky and colleagues (2018, p. 134) call it “a marker of the field’s maturation” that more and more researchers are conducting and publishing comprehensive systematic reviews and meta-analyses, in this case based on available SFD literature (Holt et al., 2017; Jones et al., 2017; Langer, 2015; Schulenkorf et al., 2016; Svensson & Woods, 2017; Welty Peachey et al., 2021; Whitley et al., 2019a).

Whilst these review studies have contributed significantly towards a broad picture of the SFD overall research landscape, the findings show that there remains a lack of specific meta-evaluations that examine the quality of studies and evaluations, including those that focus on specific themes or domains such as impact and sustainability – issues that are considered critical in SFD work and which are discussed in more detail later in this article. Moreover, our scoping analysis revealed that to date only one systematic review has examined the methodological quality of studies in detail (Darnell et al., 2019) and can therefore be classified as a meta-evaluation in the context of the ERF. To further support SFD’s ‘maturation’ process, it seems essential that further studies with a wider scope and deeper focus – in particularly meta-evaluations that explore and improve the quality of SFD studies – will be undertaken in future research.

Our review further shows that most SFD studies take place during project implementation (“ongoing”) or right at the end of the project (“final”) when research funding is often still available. No studies could be identified that explicitly focus on the planning (“ex-ante”) or post project/program phase (“ex-post”), highlighting the persistent lack of focus on researching long-term impacts and aspects of sustainability. Both these aspects are discussed in more detail at the end of this section.

Theory-driven vs. participatory approaches

Overall, the scoping study identified 47 publications that discussed theory-driven or participatory approaches. Despite their ability to demonstrate the causal relationship between projects inputs, outputs, outcomes and impacts (Kurz & Kubek, 2016), to date, only few SFD organizations can be credited for – or have proven to employ – theory-driven approaches such as logic models to underpin their operations (Whitley et al., 2019b). However, there seems to be a growing interest and increased understanding in this critical space. Most notably, the theory-driven Results Based Management (RBM) approach – which has been widely used in general development cooperation work since the 1990s (Binnendijk, 2000) – is part of the most recently published SFD-guidelines of the Commonwealth Secretariat (2019). However, the findings show that overall attitudes towards theory-driven logic models still differ among SFD researchers: on the one hand, critics describe them as being overly output-oriented, linear and rigid, and largely top-down or donor-imposed (Kay, 2012; Lindsey & Grattan, 2012; Spaaij et al., 2018). On the other hand, according to proponents, they represent flexible frameworks “that are participative, collaborative, iterative, and developmental” (Whitley et al., 2020, p. 23). Preti (2012) points out that criticism in this regard must distinguish between the general approach (i.e., project planning including problem analysis, development of objectives and indicators, identification of risks and assumptions) and the logframe matrices used in programs summarizing the main elements. The latter tend to have numerous shortcomings: They often remain “inflexible blueprints” in that they are established before a project begins, ‘imposed’ on a project, and therefore limited in their ability to make regular adjustments during the course of the project. Hence, Whitley et al. (2020, p. 24) advocate for active support of SFD organizations “in setting up their own […] frameworks […] or adapting/adopting existing frameworks” that hold credibility and provide legitimacy to funders, align with national policy priorities, and enable organizational development and learning.

Meanwhile, our review revealed that numerous researchers are also calling for more participatory, culturally and context-sensitive approaches, designs, and methods to holistically understand complex development dynamics, including M&E processes (Burnett, 2015; Collison & Marchesseault, 2018; Darnell, Chawansky, et al., 2018; Darnell et al., 2016; Kay, 2012; Sherry et al., 2017). In general, participatory approaches such as Participatory Action Research (PAR) have been used on a regular basis when analyzing SFD projects and programs (Burnett, 2015; Sherry et al., 2017). Furthermore, participatory approaches are modifiable into guiding study models or frameworks, for example in the form of the Sport for Development Impact Assessment Tool (SDIAT) (Burnett & Hollander, 2007), Participatory Social Interaction Research (PSIR) (Collison & Marchesseault, 2018), Post-colonial-feminist Participatory Action Research (PFPAR) (Hayhurst et al., 2015) and the qualitative Sport in development settings (SPIDS) framework – where reflection and reflexivity are given a special role in sport and development scholarship (Schulenkorf et al., 2020). The latter comes on the back of research by Spaaij and colleagues (2018) who examined a variety of participatory SFD studies which showed particular deficits in reflection and critical questioning of the researchers’ roles (reflexivity) – an area that has previously been highlighted as a critical yet understudied space in SFD research (Darnell, Giulianotti, et al., 2018).

Designs and methods

In our analysis, we identified 23 publications related to the designs and methods used in the field of SFD. In their integrative review of SFD scholarship, Schulenkorf et al. (2016, p. 35) identified a “potpourri of research approaches and methods” with more qualitative designs being used overall and fewer quantitative or mixed methods designs. For data collection, mainly traditional qualitative methods have been used, including observations, interviews, and document analyses (Schulenkorf et al., 2016). The benefits of innovative, creative, culturally appropriate, and technologically savvy designs and methods are only starting to be realized. Specifically, methods that use innovative media technologies, such as videos, iPads or social media, as well as diverse creative and flexible survey types such as drawings, poems, stories of change or participatory mapping have been suggested as critical elements for methodological advancement (Darnell et al., 2016; Luguetti et al., 2022; Preti, 2012; Schulenkorf et al., 2016; Sherry et al., 2017).

It is logical that a field of research – which is shaped by researchers from various disciplines with different theoretical approaches and educational backgrounds – cannot show unity with regard to research paradigms or methodological procedures. However, the review uncovered that in the specific SFD context, more seems to be at play. At first glance, it may be seen as merely a methodological dispute that takes place in empirical social research, i.e., a split between supporters of the quantitative and qualitative ‘camps’. On closer examination, however, the dispute goes beyond the discussion about the ‘right’ methodological approaches and leads to fundamental debates about power and ownership, including top-down and bottom-up (research) approaches in development work. Here, discussion topics go beyond the value and rigor of scientific traditions (positivist vs. interpretivist, constructivist or critical paradigms) and extend to the research perspective of researchers from the Global North (‘colonizer’) evaluating projects and organizations in the Global South; the conceptualization and definition of ‘development’ in general; and the production of new knowledge and localized, Indigenous voices in particular (Hartmann & Kwauk, 2011; Whitley et al., 2020).

In this context, critics often question positivist approaches and are concerned that associated methods and top-down procedures (external researchers assessing local projects without local contributions) reinforce neo-colonial power structures and hegemonic systems while suppressing local knowledge production (Kay, 2012; Lindsey & Grattan, 2012). Opposing voices portray this as a critique of militant “liberation methodologists” (Coalter, 2013a, p. 38) who defy reality and who seek to put neo-colonial attitudes on par with positivist methodological approaches whilst avoiding the much-needed defining and measuring of outcomes and impacts (Whitley et al., 2020). Overall, the repartee is sometimes more, sometimes less extensive across all kinds of publications (e.g., Coalter, 2013a; Darnell et al., 2016; Kay 2012; Lindsey & Grattan, 2012) which leads Massey and Whitley (2019, p. 177) to suggest that to meaningfully address this issue, it would be better to have a more nuanced debate on methodology: “Rather than lay blanket critiques across different research paradigms and epistemologies, there is a need to discuss higher levels of sophistication in both instrumental/positivist (i.e., quantitative) and descriptive/critical (i.e., qualitative) SDP research”. In building on this recommendation, researchers are now increasingly trying to bridge the gap between the two main streams. In other words, in an attempt to avoid an ‘either-or’ perspective scholars have started to engage in more nuanced discussions for more inclusive theoretical and methodological solutions (Whitley et al., 2020). Here, the previously discussed use of theoretical approaches that focus on systemic interrelationships – such as the concept of policy coherence (Lindsey & Darby, 2019) or actor-network theory (Darnell, Giulianotti, et al., 2018) – could be valid ways forward.

Impacts and sustainability

Overall, we identified 44 publications related to impacts and sustainability aspects. Due to its label as a “low-cost, high-impact tool” for international development (Kay & Dudfield, 2013, p. 13), sport is – perhaps more than any other tool – under pressure to prove what it can contribute to various development outcomes (Levermore, 2011). Hence, impact studies have become a popular approach to justify sport-based development programs. However, critical voices have objected the instrumentalization of impact studies and have accused both implementers and donors to mainly use them as a vehicle that legitimizes their investments and shows alleged “proof” of the effects of their programs (Burnett, 2015; Preti, 2012; Schulenkorf et al., 2016).

The difficulty of tracking, measuring, or even isolating the ‘sport-made’ impacts and thus closing the attribution gap has long been recognized in the SFD domain (Coalter, 2013b, 2013a; Levermore, 2011). In fact, in some cases attempts to measure sport-specific contributions have been declared as impossible (Lindsey & Chapman, 2017). While there is agreement in regards to the criticality of making impacts ’visible’ (Commonwealth Secretariat, 2019), impact attributions are therefore formulated more ’cautiously’ (Kay & Dudfield, 2013; Schulenkorf, 2017) and scholars mainly talk about (often indirect) contributions to specific development objectives or the SDGs more broadly (Lindsey & Chapman, 2017). Perhaps the exaggeration of sport being an “all-purpose social vaccine” (Coalter, 2013a, p. 55) has finally been demystified, or – as Schulenkorf (2017, p. 244) has put it – “it seems that the SFD community has largely cured its own biggest social ill; namely, the simplistic view that sport, and even SFD, automatically leads to positive social, cultural, educational, health-related and/or economic development”.

Popular across the natural sciences, RCTs – also known as the ‘gold standard’ of evaluations – are supposed to close attribution gaps and isolate the effects of an intervention by comparing changes of randomly selected intervention and control groups (Mueller & Albrecht, 2016; Scriven, 2008). In the SFD context, this rigorous yet complex design is rarely feasible. Although there is an increasing demand for robust SFD study designs (Darnell et al., 2019; Kaufman et al., 2014; Lindsey & Darby, 2019; Massey & Whitley, 2019), researchers are weary that in addition to methodological and practical challenges of RCT trials – including limited opportunities for control groups – ethical challenges remain. Specifically, interventions have to be controlled and standardized in order to explain causal mechanisms in a way that may neglect the specific needs and concerns of vulnerable populations, including individual children (Mueller & Albrecht, 2016). This also speaks to a wider issue of quantitative impact studies which tend to neglect the social context of programs. Here, internal and external factors and their influence on an intervention’s effects are often not appropriately considered or underestimated. Associated social, managerial and political factors such as internal organizational structures, staff rotations, project durations as well as political systems and cultural peculiarities do have a great influence on the impacts and outcomes of a project or program, and they tend to be largely ignored in RCT assessments (Lindsey, 2017; Stockmann, 2006).

While the absence of RCT studies can be explained in part by the factors above, the limited use of ex-post evaluations – meaning studies which are set-up after the completion of a project in order to assess long-term effects (impacts) and sustainability – remains a surprise, specifically as there are also readily available theoretical models for impact evaluation in evaluation research (Oberndörfer et al., 2010). Aspects of sustainability (in terms of durability) of donor-dependent SFD projects have previously been questioned and criticized (Lindsey, 2017). Furthermore, (long-term) ecological impacts have been discussed in the context of sport and environmental issues (Darnell, 2019) and ‘thought about’ in the context of the S4D Framework (Schulenkorf, 2012). However, the issue of sustainability per se has received little explicit practical attention in SFD evaluations and research. In fact, with the notable exception of Lindsey’s (2008) conceptual work, suitable theoretical approaches or assessment frameworks have not yet been developed (Sherry & Osborne, 2019). As the findings of the category “approaches based on scope” have already shown, no studies could be identified that explicitly focus on the post project/program phase (“ex-post”), highlighting the persistent lack of focus on researching long-term effects and aspects of sustainability. Again, these omissions provide a worrying status quo as it has long been argued that conducting more ex-post evaluations on completed projects is crucial to assess processes (e.g., the influence of project planning, management, and follow-up on sustainability), sustainability (i.e., the fate of projects after funding), and overall SFD impact (i.e., intended and unintended effects, identify factors of success and failure).