Evaluating Societal Impact

A rough how-to guide

This guide offers more practical information related to societal impact evaluation. It should be seen as an emergent, ever evolving, collaborative document – the intention is to incorporate further evaluation methods and examples. So, if you develop or discover a useful technique for evaluation or have questions about the current content do let us know.

It draws upon a range of sources, including the experience of ESI and literature on societal impact evaluation methods, to provide a brief introduction to a range of evaluation methodologies and examples of how they can be used throughout the life of your project. For each piece of content we indicate the purpose (why), target audience (who), resource type (what) and what point in the research cycle (when).

Why

General tips&tricks/ Grant/ Insight/ Collaboration/ Formal evaluation (e.g. SEP)/ Annual interview

Who

Policy/ Support/ Research/ Education

What

Impact evaluation and activity/ Impact environment and capacity/ Impact strategy and ambition

When

Plan/ Do/ Describe/ Share

 

 

Why

How to get started

Who

Academics

What

Tips

When

Plan

Before setting out and undertaking impact related activities it is important to consider the aims of the project (i.e. what you want to achieve). As you will be unable to tell whether your project was a success unless you have decided what it was supposed to accomplish. Also good to be aware of the difference between outcomes, outputs and activities: outcomes are the changes you hope to bring about as a result of what you’re doing; activities are the things you do in order to make these changes happen and outputs are the things you can measure when it comes to the results of your activities. A typical outcome would be “to increase awareness in Rotterdam of EUR’s work in sustainability”. This would lead to activities with outputs such as “to host a series of public events that effectively increase the awareness of those attending”. Outputs and outcomes should be as specific and detailed as possible, as they will form the basis for any evaluation undertaken. The next things to consider are: what evidence you need to collect to know you have met your outputs (i.e. indicators), the focus of the evaluation (i.e. what and who with) and when the evaluation will take place (i.e. pre and post project, during the life of the project).

To help establish an impact proposition, you can think of possible activities on a canvas such as this:

1. Impact proposition

2. Target groups

3. Key partners

4. Key activities

What can your research add for others?

Who can benefit fom your impact proposition?

Who do you need to reach your target group?

What will you and your key partners do to realise your value proposition?

 

 

 

 

5. Channels

6. Quantified reach

7. Key resources

8. Target group relations

What channels will you use to reach your partners and target group? (e.g. media, personal contacts, etc)

How can you quantify your reach with your target group?

What people or external resources are you dependent on to realise your impact proposition?

How will you keep in touch with your target group?

 

 

 

 

9. Cost stucture

10. Revenue

What costs are associated with your impact proposition?

Is there potential for financial revenue?

 

 

The answers to these questions will shape what method(s) you decide are appropriate for the evaluation of your project. The more you think about the answers to these questions the more you will be drawn towards particular methods of evaluation. Furthermore, the clearer you are in your outcomes and outputs, the more you will be able to use monitoring and evaluation to judge your success.

The table below summarises some of the most common evaluation techniques, so you can think through and choose the most appropriate techniques to evaluate your project. You shouldn’t restrict yourself to this list – if your thinking about your aims has given you ideas for new and better ways to measure success, don’t be afraid to use them!

Whatever evaluation methods you decide to use, it is good practice to tell people what information you are collecting, why you are collecting it, and what you will do with it (i.e. the purpose and uses). The approach to evaluation should be transparent and it is important to make it clear to all parties involved how much (or how little) the information you gather will inform future work.

Why

Ensure you are aware of the diversity of impacts you can plan for.

Who

Policy/Support/Research/Education

What

Impact strategy and ambition

When

Plan

We distinguish 9 types of impact corresponding to the broad scope: commercial impact, policy impact, social impact, health impact, cultural impact, environmental impact, technological impact, legal impact and international impact.

  1. ECONOMIC IMPACT Driving economic growth, generating new products and services and creating jobs.
  2. POLICY IMPACT Informing, influencing and improving decision-making by government and public bodies, NGOs and in the private sector. Increasing the efficiency and/or quality of public services, directing investment to priority areas and raising business productivity.
  3. SOCIAL IMPACT Informing public debate, stimulating public interest, improving welfare, equality and inclusion, and improving quality of life and opportunities.
  4. HEALTH IMPACT Creating new drugs and treatments and developing new therapies. Improving education and training, public awareness, and access to health care provision, as well as policy, legislation, standards or guidelines.
  5. CULTURAL IMPACT Enhancing and preserving our cultural heritage, producing cultural artefacts, creating, inspiring and supporting new forms of expression, and enhancing our understanding of minority groups and communities.
  6. ENVIRONMENTAL IMPACT Delivering energy savings and reduced emissions, improving management and conservation of natural resources, stimulating public awareness and influencing policy, improving business and public service operations, and environmental risk management.
  7. TECHNOLOGICAL IMPACT Developing new and improving existing technologies.
  8. LEGAL IMPACT Improving law enforcement methods, effecting legislative change and improving legal practice and access to justice.
  9. GLOBAL IMPACT Delivering positive impacts from our research overseas including collaborating with partners in other countries.

Why

To develop understanding societal impact

Who

Policy/Support/Research/Education

What

Impact environment and capacity

When

Share

To work with an embedded and positive impact culture as EUR community, it is imperative that we develop a shared understanding of what societal impact entails. As a first step in collectively defining our impact identity, the Evaluating Societal Impact (ESI) team and Strategy Office invite you to join a series of dialogue sessions.

We want you to connect to colleagues and share insights about creating societal impact in your daily work, so our shared understanding of societal impact pays tribute to the diversity within the EUR.  At the end of a dialogue session you... ​

  • Know what we consider good practices that capture the heart of our impact mission.​
  • Are aware how EUR colleagues (like you) work on societal impact, and of how you can help each other.​
  • Worked on finding connections and new possibilities for societal impact. ​
  • Know how to contribute to EUR's societal impact mission (and what you might need to support your work on impact).​

We challenge you to ask yourself: ‘What is societal impact for me?’, ‘How do I work on impact?’ and ‘What skills and people-based factors do I have or would I like to have to achieve impact?’. We encourage asking these questions and keeping the conversation going in your own network or on a theme that connects to your own work. That is why we invite you to organize your own Impact Dialogue. Re-watch the Impact Dialogue Kick-off here.

Need more information and/or download our Impact Dialogue Starter kit? Browse the information page here or contact Lotte Houtepen

What is SIAMPI?

The SIAMPI method differs from other impact assessment methods in its focus on processes, interactions and stakeholders rather than prioritizing outputs of research. A SIAMPI analysis focuses heavily on the research process and the associated interactions that take place within it. The method is aimed at highlighting the mechanisms that might bring about societal impact, with a focus on so-called productive interactions and the role of stakeholders. According to Spaapen and Van Drooge (2011), an interaction is productive when it leads to efforts by stakeholders to apply research results to societal goals, i.e., when it leads to behavioral change. 
Three types of interactions are distinguished:

  1. Direct interactions: Through direct personal contact (ranging from ordinary meetings to complex research collaboration arrangements)
  2. Indirect interactions: Through indirect contact or mediation by specific 'carriers' (expert reports, public goods, products, policy instruments, training and education)
  3. Financial interactions: Through funding or other support mechanisms such as cooperative agreements

Spaapen and Van Drooge (2011) describe these interactions as mechanisms for societal impact. They consider ‘productive interactions’ as a pre-condition for the social impact of research, by arguing that “in order to have impact you’ve got to have contact – direct, indirect, and/or financial. Still, the definition of societal impact given in SIAMPI articles is kept rather open: it concerns the well-being of people (quality of life) and/or the social relationships between people and organizations. SIAMPI does not clarify the boundaries between productive and unproductive interactions, or when a productive interaction does or does not lead to societal impact.

Why should it be used?

Given the many issues with assessing impact, SIAMPI focuses on interactions between diverse actors in the research project that might lead to impact. It provides good insight into the social relationships, and networks that emerge during, and support, the research process. SIAMPI can also contribute to the creation of a theoretical framework within which interactions are made visible and can be assessed.

When should it be used?

Through SIAMPI, light can be shed on what happens in the process of knowledge production, and the role different stakeholders play in it. The focus of a SIAMPI evaluation is on the process of interaction, rather than on a research entity. Thus, SIAMPI is a suitable evaluation method for projects with elements of co-creation, action research, living labs approaches, or other forms of stakeholder engagement.

How can it be used?

SIAMPI as an approach to impact case studies is relatively broad: the main point is to map qualitatively and, if relevant, quantitatively, the different types of interactions between involved actors in a project, unit, or program. This means one can use this approach both during a project and after its conclusion. 
 
During the project (ex durante), SIAMPI can be a good guide for interactive learning between project members. Doing the interviews, as well as discussing (preliminary) results of the assessment within the team, can make all stakeholders more aware and reflective of what interactions are productive and how they might contribute to the societal change they want to achieve. 
 
Afterwards (ex post), SIAMPI can be a useful approach to elicit what interactions ended up being productive i.e., leading to changes in behavior of stakeholders (including the researchers!). Also, it is then possible to gain some insight into the relation between interactions and (intended) impact.

What is obtained?

The outcome of a SIMAPI can take on different formats and there is not one way to do it. Given that SIAMPI is mainly focused on interactions the outcome of such an evaluation should be focused on making these interactions visible. Preliminary visualization of productive interactions can be discussed with stakeholders to check whether it aligns with their experience and to facilitate learning processes (formative evaluation).  
 
Take a look at an example of how the ESI researchers have done it in this case study.

Who is a stakeholder and who is involved in an assessment using this method?

‘Stakeholders’ in this approach comprise all actors involved in achieving social impact: researchers, industry and private parties, public organizations, the government as well as members of the general public. Still, a SIAMPI analysis emphasizes the relationship between researchers and other stakeholders, including academics in neighboring fields. The aim is also to involve all stakeholders in the assessment procedure through e.g., focus groups or making them part of a review committee. 

How to do it?

Data for the impact evaluation could be quantitative and qualitative data for each type of interaction. This data can be e.g., narratives, case studies, documents, etc. 
 
As SIAMPI does not provide a how-to guide for impact assessment some practical issues have to be decided by the analysts themselves, such as e.g., time span of the assessment. Analysts and involved actors should discuss a clear period for the assessment and decide what will happen afterward.

Overall, SIAMPI is lacking more precise guidance. Stakeholders could benefit a lot from concrete tools with which to continue the method themselves after researchers conducting impact evaluation withdraw from the project. However, it also offers opportunities to explore creative ways to identify productive interactions and thus, according to Spaapen and Van Drooge, societal impact.

What is a Theory of Change?

A Theory of Change (ToC) is a formative evaluation method, meaning it is usually done at the beginning or during a project to inform or improve project planning or design (Frey, 2018). Such process-based evaluation can help identify strengths and weaknesses of a project (Schaefer et.al., 2021).

Through a ToC process it can be assessed whether and how outcomes or process changes can be achieved or (if done ex post) were achieved. A ToC can illustrate and describe “how and why a desired change is expected to occur [or occurred] in a specific problem context” (Belcher, Davel and Claus, 2020). This helps to monitor changes in a system to which your intervention may have contributed.

One of the goals for a ToC process can be to determine how a project can achieve the impact it desires to make. Based on the assumption that creating impact is a non-linear process, but instead a web of causes and effects, a ToC process is often build around defining a project’s activities and subsequential outputs, to achieve desired outcomes and ultimately impact. As mentioned, make clear why and how the output from an activity is expected to work in support of the outcome and impact.

A ToC starts by identifying a clear problem you are contributing to and working backwards to establish preconditions for reaching that goal. Key elements in developing a ToC are:

  1. Mission or bigger goal your work is contributing to (description of context, current state and other actors able to influence change)
  2. Main actors either involved in the process or who have a stake in the work (long-term outcomes that you seek to support and for whose ultimate benefit)
  3. Main activities, their resulting outputs, and outcomes (broad sequence of events anticipated (or required) to lead to the desired long-term outcome)

During the ToC process all these elements are unpacked and underlying assumptions about how change happens are made explicit. Often assumptions underlie the logic of going from activities through outputs and outcomes to impact. Making this explicit helps to understand and make clear for yourself why and how you expect the output from an activity to work in support of the outcome and impact. 

Why should it be used?

ToC is a central tenet to creating societal impact. The purpose of the ToC process is to allow people to think about what must change before doing it. It can be seen as a general steppingstone to impact related work with a multitude of potential use cases, such as writing impact narratives, impact sections in grant applications, monitoring progress or evaluation. Although it is good to realise a ToC does not provide a specific implementation plan but rather a direction; think compass, not map.

A major benefit comes from making different views and assumptions about the change process explicit, especially seemingly obvious ones. Within multi-stakeholder projects there may be different perspectives or even different realities regarding what the desired change is, why it is desired and how it could and should happen. A shared ToC process can facilitate bringing these differences to the surface and develop a sense for what drives different stakeholders and their understanding of the problem. This process can be quite confrontational, especially if done in an organisation or team, but can contribute to a more shared understanding of a project’s purpose and strategic choices (Es et.al., 2015).  

When should it be used? 

A ToC can be used throughout the whole span of a project – ex-ante to ex-post. Depending on the context or stage of a project in which the ToC is being used, there are different benefits. Using a ToC as an ex-ante planning tool can facilitate critical reflections on ‘what needs to change’ before doing it and can therefore allow for a project to be planned and designed towards impact. Further, as a monitoring and evaluation tool, it can help to adapt activities where needed to assure they are still aiding to reach the desired outcomes. As an ex-post assessment tool, it allows you to trace back which activities led to which outcomes in the change process. A retrospective ToC becomes more accurate when it can build upon a ToC done prospectively (Belcher & Claus, 2020). 

How can it be used?

Most often a ToC is done in a workshop setting with multiple stakeholders present. Such a setting allows for deliberate conversations about assumptions regarding the change process. A ToC can facilitate these conversations by using it as a framework to guide the conversation and through that shape the project.

Belcher et.al. (2020) describe the chain from activities through outputs and outcomes to impact through different spheres:

  • Sphere of control – direct influence
    What the project does
    - Activities: the insights that are needed to bring about the desired outcomes and how these insights will be gained
    Outputs: the tangible products as a result of the activities. Usually expressed as nouns, tangible and can be counted (15 trainings, 5 market entry activities, 20 technical assistance missions). 
     
  • Sphere of influence – indirect influence
    Who the project works with and through
    Outcomes: relate to changes in behaviour, relationships, actions, and activities of stakeholders resulting from exchange of knowledge and the uptake of research outputs. Identify who has to do what differently to achieve the desired impact. Changes can be:

- Instrumental: plans, decisions, behaviour, practices, actions, policies
- Conceptual: changes to knowledge, awareness, attitudes, emotions that contribute to the understanding of issues and reframing debates
- Capacity building: technical and personal skills and expertise
- Network: number and quality of relationships and trust
- Knowledge culture: attitudes towards knowledge exchange and impact itself

  • Sphere of interest – higher level project aims, outside of project influence
    Improved conditions that the project hopes to see
    - Impact: A deliberate chain of activities and conditions by which the potential impact that the project is aiming for could be realised

Underlying all of this is the belief that engaging with stakeholders throughout the project and the activities increases the chance for productive interactions, leading to impact.

Actively thinking about and defining these aspects for a project helps align expectations and guide the project towards reaching its higher aims and achieving societal impact.
 

What is obtained?

The output of a ToC process is not necessarily a tangible product since the added value comes from the process itself and the resulting conversations. However, a ToC process can result in a graphical depiction and/or impact narrative of the impact pathways identified throughout the process. Such a ToC narrative can be a useful starting point for a SEP case study impact narrative. If there is a specific desired outcome product, the process needs to be designed to facilitate reaching this goal.

Who is a stakeholder and who is involved in an assessment using this method?

Participants in a ToC workshop usually include the research project management, beneficiaries, and ideally any key stakeholders who will be engaged (i.e., consulted, informed, or involved) in the research. It is useful to have a facilitator moderate the discussion (Belcher & Claus, 2020). When ToC is used as an evaluation tool it is particularly relevant to include key stakeholders in the process.

How to do it?

Literature

Belcher et.al., 2020. A refined method for theory-based evaluation of the societal impacts of research. Published by Elsevier B.V.

Belcher B, Claus R 2020. Theory of Change. td-net toolbox profile (5). Swiss Academies of Arts and Sciences: td-net toolbox for co-producing knowledge. www.transdisciplinarity.ch/toolbox. doi.org/10.5281/zenodo.3717451

Es et al., 2015. Hivos ToC Guidelines: Theory of Change thinking in practice. https://hivos.org/document/hivos-theory-of-change/

Frey, B. (2018). The SAGE encyclopedia of educational research, measurement, and evaluation (Vols. 1-4). Thousand Oaks,, CA: SAGE Publications, Inc. doi: 10.4135/9781506326139

Schaefer, T., Kieslinger, B., Brandt, M., van den Bogaert, V. (2021). Evaluation in Citizen Science: The Art of Tracing a Moving Target. In: , et al. The Science of Citizen Science. Springer, Cham. https://doi-org.eur.idm.oclc.org/10.1007/978-3-030-58278-4_25

Compare @count study programme

  • @title

    • Duration: @duration
Compare study programmes