Impact evaluation

Theoretical concepts

The Evaluating Societal Impact team describes the theoretical concepts that are important to keep in mind when we discuss impact evaluation as an instrument for learning and improvement at Erasmus University Rotterdam. 

Evaluation is a systematic way of reflecting on and assessing the value of what is being done (i.e. a project, a programme, an event). Evaluation is commonly interpreted as an end product or an activity taking place at the end of a project. However evaluation can also be considered as a process, taking place across all phases of a project, used to determine what has happened and whether the initial aims of the project have been carried out and achieved. Evaluation can do more than assess and measure; it helps set the stage for a culture of learning, change and improvement.

We put the development and learning process central (formative evaluation), instead of looking back and assessing completed work (summative evaluation). In accordance with the state of the art in the field and the recently updated Strategy Evaluation Protocol (SEP), impact assessment is then part of a strategic approach in which impact ambitions, activities, evaluation and learning are intertwined. This entails a shift from impact as something that just happens – a welcome, but accidental by-product – towards impact as an explicit goal for which specific actions are taken and a stimulating environment is provided.

Guiding principles

Based on our expertise and knowledge, the ESI team has formulated ten guiding principles for the evaluation of societal impact of research:

  1. Societal impact occurs when research helps society deal with the challenges it faces;
  2. An impact-driven university knows about, supports and promotes the societal impact of its research;
  3. Clear strategies enable research to increase its impact;
  4. Impact assessment is part of a learning process that aims to reflect on and improve strategies for achieving impact;
  5. Impact is assessed in light of the impact ambition and strategy of the project or group concerned;
  6. Ambitions for impact specify goals, target groups and interaction mechanisms (theory of change);
  7. Evaluation combines narratives (productive interactions + impact pathways) and context-specific indicators;
  8. Evaluation also maps how the institutional environment supports and facilitates the creation of impact;
  9. Stakeholders from society play an active role in creating ambitions for, and assessing, societal impact;
  10. By 2024, impact ambitions, activities and assessments are embedded in ongoing processes so that evaluating societal impact is part of a learning, responsible and inclusive organization;

Underpinning these guiding principles are core concepts. In the drop down menu below each core concept is described including signposts to relevant practical information in the rough guide section of this website.

Understanding our societal impact is important because EUR is undertaking a transition towards an impact-driven university. But it is also a challenge because it requires deep reflection and continuous dialogue at all levels of the organization, for example about the question what ‘positive’ impact consists of. 

Many different definitions of societal impact circulate in the academic world and beyond. Ultimately, as a university we create societal impact when our core activities contribute to a better understanding of societal challenges and the capacity to deal with them. The ESI project does not aim to implement one authoritative definition for the entire university, as the types of impact differ significantly between and within disciplines and research projects. 

Instead, we provide a list of characteristics of societal impact so that impact ambitions, strategies and activities can be developed appropriate for the various domains and levels within the EUR. 

  • contribute to positive change in societal challenges, either directly or indirectly;  
  • result from productive interactions between academics and their stakeholders, which occur at all stages of the process;   
  • depend upon supportive organizational environments;  

This characterization highlights that the possibility of societal impact often relies on gradually established networks and conditions. But we also describe impact in terms of actual changes in society. Consciously, this surpasses the current Strategic Evaluation Protocol (SEP), which operationalizes impact in terms of research output for and use and recognition by societal stakeholders. We understand these marks of relevance as preconditions for impact. A university that aims to create positive impact should dare to focus on achieved changes, without losing sight of the fact that this is often largely out of our sphere of control or influence. 

Impact ambitions are central to formative evaluation of academic teams and research units. 

Awareness of the current potential impact and a theory of change are key to the formulation of an impact ambition. 

 

A stakeholder consultation can greatly improve the mapping of a group’s potential and can be the start of a process of meaningful interaction with society. A theory of change comprises a deliberate chain of activities and conditions by which this potential impact could be realized. Based upon this, a research group can deliberately choose its impact strategies, the specific means and activities by which impact is generated. Eventually, the evaluation of societal impact will consist of narratives (mirroring the theory of change) and indicators (selection based on own strategies).

 

An impact ambition describes:  

  • The societal challenges that a group will contribute to;  
  • The ultimate beneficiaries, intermediaries and other societal partners;  
  • The interaction mechanisms and impact strategies to reach these actors;  
  • The ultimate change that is aimed for.  

More information 

If you are interested in practical tips & tricks to formulate an impact ambition, you might find the following topics useful in our rough guide section: 

  • Before you start

Within the ESI team, Tung Tung Chan is the go-to person when it comes to questions around impact ambition and strategy. 

How can researchers think about and organize their societal impact? Even though societal impact implies a focus on an end-result, what matters are the processes of interactions between various actors. 

Many roads lead to Rome, and many pathways can lead to impact. Often it differs per societal sector and scientific discipline what kinds of activities and interactions are suitable, accessible and effective (see table below for examples).

 

Table 1: Various activities that researchers can employ as part of their pathway to impact.

Professional training 

Academic consultancy 

Action research 

Commercialization 

Evaluation & monitoring 

Co-creation 

Policy advice/criticism 

Platforming voices 

Design methods 

Academic education

Public engagement 

Engaged scholarship 

Science communication 

Personnel secondment 

Living labs 

Tools and services 

Stakeholder dialogues 

Citizen science 

 

Specific activities can fit into broader strategies that connect activities and outputs to outcomes and impacts. When deciding on the type of impact activity and thus impact pathway, be aware of the following aspects to support their application (Smit and Hessels 2021):  

  • Potential types of impact that this could lead to, e.g. economic, technological, health, social, educational, policy, cultural and public impacts (Grønvad, Hvidtfeldt, and Pedersen 2017)  
  • Aspects of a ‘productive configuration’ or organizational environment that enables researchers to engage in impact-related activities’ (Joly et al. 2015);  
  • Types of ‘productive interactions’ and intermediaries (Spaapen and Van Drooge 2011); 
  • Kinds of envisioned end-users.  

More information

If you are interested in getting started with your own impact evaluation and activities, you might find the following topics useful in our rough guide section: 

  • Before you start
  • Types of impact

Within the ESI team, Lisa Burghardt is the go-to person when it comes to questions around impact evaluation. 

References 

Grønvad, Jonas Følsgaard, Rolf Hvidtfeldt, and David Budtz Pedersen. 2017. “Analysing Co-Creation in Theory and in Practice: A Systemic Review of the SSH Impact Literature.” https://vbn.aau.dk/en/publications/analysing-co-creation-in-theory-and-in-practice-a-systemic-review. 

Joly, Pierre-Benoît, Ariane Gaunand, Laurence Colinet, Philippe Larédo, Stéphane Lemarié, and Mireille Matt. 2015. “ASIRPA: A Comprehensive Theory-Based Approach to Assessing the Societal Impacts of a Research Organization.” Research Evaluation 24 (4): 440–53. https://doi.org/10.1093/reseval/rvv015. 

Smit, Jorrit P., and Laurens K. Hessels. 2021. “The Production of Scientific and Societal Value in Research Evaluation: A Review of Societal Impact Assessment Methods.” Research Evaluation 30. 

Spaapen, Jack, and Leonie Van Drooge. 2011. “Introducing ‘Productive Interactions’ in Social Impact Assessment.” Research Evaluation 20 (3): 211–218. 

The capacity of an organisation to achieve impact involves creating the organisational, material, and personal conditions to work on impact. This requires specific knowledge, infrastructure and support for impact. Ranging from free parking for stakeholders to which roles need to be present. 

Having an impact strategy and plans in place helps, but to create societal impact you also need impact literate staff and a supportive impact literate institution. It starts with fundamental knowledge and understanding about impact that underpin the skills and conditions to apply and modify approaches in changing contexts, before moving towards more advanced oversight and directive action. Bayley & Phipps (2017) defined impact literacy as the ability: “to identify appropriate impact goals and indicators, critically appraise and optimise impact pathways, and reflect on the skills needed to tailor approaches across contexts.”

On an individual level, a researcher can be considered impact literate if they understand the specific benefits being sought, the activities which would achieve these, and how approaches need tailoring in specific contexts:

  1. the identification, assessment, evidencing and articulation of impact endpoints (“what”);
  2. the practices that create impact (“how”);
  3. the successful integration of these by research impact practitioners (“who”).

Although impact is created and captured at the level of the research project, it is delivered within an institutional structure. An individual can build their own competencies, but the extent to which they can implement these depends on their institutional work context. Similarly, institutional drives to improve the non-academic benefit of research are ultimately proportionate to the abilities of those involved to deliver on higher level goals. Therefore, individual capacity building for impact needs to be supported by institutional approaches and institutional capacity building needs to be informed by individual expertise (Bayley & Phipps, 2019).

References  

Bayley J and Phipps D. Extending the concept of research impact literacy: levels of literacy, institutional role and ethical considerations [version 2; peer review: 2 approved]. Emerald Open Res 2019, 1:14 (https://doi.org/10.35241/emeraldopenres.13140.2)

Bayley, J & Phipps, D 2017, 'Building the Concept of Research Impact Literacy' Evidence and Policy, vol 15, no 4, 597–606 https://dx.doi.org/10.1332/174426417X15034894876108

More information 

If you are interested in building an impact environment, you might find the following topics useful in our rough guide section: 

  • Impact dialogue

Within the ESI team, Lotte Houtepen is the go-to person when it comes to questions around impact capacity and building an impact environment.

Compare @count study programme

  • @title

    • Duration: @duration
Compare study programmes