This is an introduction to productive interactions/SIAMPI as a method for impact assessment. At the bottom of the page, you can find a case study (including a downloadable report) describing how ESI has made use of this method to conduct an impact assessment and our reflections.
SIAMPI Method
Written by: Lisa Burghardt
Date: 30.05.2022
What is SIAMPI?
The SIAMPI method differs from other impact assessment methods in its focus on processes, interactions and stakeholders rather than prioritizing outputs of research. A SIAMPI analysis focuses heavily on the research process and the associated interactions that take place within it. The method is aimed at highlighting the mechanisms that might bring about societal impact, with a focus on so-called productive interactions and the role of stakeholders. According to Spaapen and Van Drooge (2011), an interaction is productive when it leads to efforts by stakeholders to apply research results to societal goals, i.e., when it leads to behavioral change.
Three types of interactions are distinguished:
- Direct interactions: Through direct personal contact (ranging from ordinary meetings to complex research collaboration arrangements)
- Indirect interactions: Through indirect contact or mediation by specific 'carriers' (expert reports, public goods, products, policy instruments, training and education)
- Financial interactions: Through funding or other support mechanisms such as cooperative agreements
Spaapen and Van Drooge (2011) describe these interactions as mechanisms for societal impact. They consider ‘productive interactions’ as a pre-condition for the social impact of research, by arguing that “in order to have impact you’ve got to have contact – direct, indirect, and/or financial. Still, the definition of societal impact given in SIAMPI articles is kept rather open: it concerns the well-being of people (quality of life) and/or the social relationships between people and organizations. SIAMPI does not clarify the boundaries between productive and unproductive interactions, or when a productive interaction does or does not lead to societal impact.
Why should it be used (or why not)?
Given the many issues with assessing impact, SIAMPI focuses on interactions between diverse actors in the research project that might lead to impact. It provides good insight into the social relationships, and networks that emerge during, and support, the research process. SIAMPI can also contribute to the creation of a theoretical framework within which interactions are made visible and can be assessed.
While SIAMPI does not specifically mention it, within our case study we linked SIAMPI outcomes to the Theory of Change (Belcher et al, 2020) in order to further define (intended) societal impact
A couple of remarks can be considered when using SIAMPI:
(1) The SIAMPI methodology is not fully developed. SIAMPI does not clearly indicate how the productive interactions are transformed into (intended) social impact; where do the productive interactions 'stop' and societal impact emerges? It is assumed that the productive interaction itself constitutes an impact.
(2) SIAMPI does not mention how a potential productive interaction, or an obstruction of a productive interaction needs to be indicated. It can be interesting to integrate these potential or obstructive mechanisms within the analytical framework to understand why a productive interaction did not (fully) occur.
(3) SIAMPI does not mention a time indication to follow a research project and / or to (re) evaluate in the future. (Will it be measured again x number of years later?) Therefore, clarify a time frame on forehand in which a SIAMPI evaluator defines when his/her involvement starts and ends. This also allows the involved stakeholders to prepare themselves in order to hand over an evaluating role within their research project.
(4) When using in depth-interviews, the topic-list of SIAMPI is broadly formulated. Therefore, the topic-list allows tailoring to a specific context and type of stakeholder (primary/secondary/researcher).
When should it be used?
Through SIAMPI, light can be shed on what happens in the process of knowledge production, and the role different stakeholders play in it. The focus of a SIAMPI evaluation is on the process of interaction, rather than on a research entity. Thus, SIAMPI is a suitable evaluation method for projects with elements of co-creation, action research, living labs approaches, or other forms of stakeholder engagement.
How can it be used?
SIAMPI as an approach to impact case studies is relatively broad: the main point is to map qualitatively and, if relevant, quantitatively, the different types of interactions between involved actors in a project, unit, or program. This means one can use this approach both during a project and after its conclusion.
During the project (ex durante), SIAMPI can be a good guide for interactive learning between project members. Doing the interviews, as well as discussing (preliminary) results of the assessment within the team, can make all stakeholders more aware and reflective of what interactions are productive and how they might contribute to the societal change they want to achieve.
Afterwards (ex post), SIAMPI can be a useful approach to elicit what interactions ended up being productive i.e., leading to changes in behavior of stakeholders (including the researchers!). Also, it is then possible to gain some insight into the relation between interactions and (intended) impact.
What is obtained?
The outcome of a SIMAPI can take on different formats and there is not one way to do it. Given that SIAMPI is mainly focused on interactions the outcome of such an evaluation should be focused on making these interactions visible. Preliminary visualization of productive interactions can be discussed with stakeholders to check whether it aligns with their experience and to facilitate learning processes (formative evaluation).
Take a look at an example of how the ESI researchers have done it in the case study below this text.
Who is a stakeholder and who is involved in an assessment using this method?
‘Stakeholders’ in this approach comprise all actors involved in achieving social impact: researchers, industry and private parties, public organizations, the government as well as members of the general public. Still, a SIAMPI analysis emphasizes the relationship between researchers and other stakeholders, including academics in neighboring fields. The aim is also to involve all stakeholders in the assessment procedure through e.g., focus groups or making them part of a review committee.
How to do it?
Data for the impact evaluation could be quantitative and qualitative data for each type of interaction. This data can be e.g., narratives, case studies, documents, etc.
As SIAMPI does not provide a how-to guide for impact assessment some practical issues have to be decided by the analysts themselves, such as e.g., time span of the assessment. Analysts and involved actors should discuss a clear period for the assessment and decide what will happen afterward.
Overall, SIAMPI is lacking more precise guidance. Stakeholders could benefit a lot from concrete tools with which to continue the method themselves after researchers conducting impact evaluation withdraw from the project. However, it also offers opportunities to explore creative ways to identify productive interactions and thus, according to Spaapen and Van Drooge, societal impact.
Literature
The societal impacts of research. MethodsX, (7). Belcher, B.M., Davel, R. & Claus, R. (2020). A refined method for theory-based evaluation of the societal impacts of research. MethodsX, (7).
Edelenbos, J., & Monnikhof, R. (2001). Lokale interactieve beleidsvorming: Een vergelijkend onderzoek naar de consequenties van interactieve beleidsvorming vergelijkend onderzoek naar de consequenties van interactieve beleidsvorming voor het functioneren van de lokale democratie. Utrecht: Lemma voor het functioneren van de lokale democratie. Utrecht: Lemma
Polonsky, M.J. and Scott, D. (2005), "An empirical examination of the stakeholder strategy matrix", European Journal of Marketing, Vol. 39 No. 9/10, pp. 1199-1215.
Spaapen, J and L van Drooge 2011. Productive interactions as a tool for social impact assessment of research. Research Evaluation, 20(3). ment of research. Research Evaluation, 20(3).
Productive Interactions in Education - SIAMPI as an impact evaluation tool
Written by: Lisa Burghardt
Date: 23.02.2022
Project duration: Project Impact Assessment (PIA) took place between February and June 2021; the Nudging in Education Project runs from 2018 to 2022
Stakeholders: Project ‘Nudging in het Onderwijs’ at Department of Psychology and Pedagogical Sciences (DPECS) at the Faculty of Social Science of Erasmus University Rotterdam (ESSB), Evaluating Societal Impact
ESI Researchers: Margriet Nguyen, Lucy van Eck & Jorrit Smit
In this case study on a 'Nudging in Education’ research project, we have used the SIAMPI (Social Impact Assessment Methods through Productive Interactions) approach as introduced by Spaapen & Van Drooge (2011). Because of the many issues with assessing impact, SIAMPI focuses on interactions between diverse actors in the research project that might lead to impact. It provides good insight into the social relationships and networks that emerge during, and that support, the research process.
SIAMPI can also contribute to the creation of a framework within which interactions are made visible and assessed. Preliminary visualization of productive interactions can be discussed with stakeholders to check whether it aligns with their experience and to facilitate learning processes (formative evaluation). We have tested this method, and reflected on its use, in an evaluation of the societal impact of a research project within the educational sector. Our experience and impressions are summarized below.
An example of SIAMPI ex durante: Nudging in Education
ESI used SIAMPI to assess productive interactions in a collaborative pedagogical research project.
“The research project ‘Nudging in Education’ (Nudging in het Onderwijs) is a collaboration between different partners: Erasmus University Rotterdam, Da Vinci College, Hogeschool Rotterdam, De Rooi Pannen, BUAS, ECBO, and TU Eindhoven. Researchers, teachers, and management of these institutions work together in this project with the common goal of helping students demonstrate autonomous learning behavior at MBO levels. The research project is made possible by the grant for Practical Research from NWO-NRO” (nudginginhetonderwijs.nl, 2021).
‘Nudging in Education’ focuses on professionalizing teachers through nudging theory. Nudges are almost imperceptible changes to our decision environment that are meant to subtly guide behaviour into (more) healthy or otherwise ‘desirable’ directions. As part of this project teachers learn about nudging theory, develop nudges together with researchers and professionals and test these nudges experimentally in their educational practices. Based on these experiments, the researchers expect to generate insights on the applicability of nudging in education (specifically for teacher professionalization). In addition, they will develop practical tools that teachers and teacher educators can use to support students in their learning process (NRO, 2018).
What did the assessment consist of?
Impact assessment with SIAMPI relies first and foremost on qualitative data. In this case, ESI studied available documents and websites of the project and conducted eight interviews with a diverse set of stakeholders. The interviews were coded to distinguish between different productive interactions, as well as related changes in attitude, knowledge, relationships, network, and skills (outcomes).
The ESI team also paid attention to different ideas about the intended societal impact of the project 'Nudging in Education'. It should be noted that the productive interactions and outcomes in themselves can also, to a certain extent, be understood as societal impact. Impact, therefore, does not necessarily have to be related to the intended impact of the researchers themselves. It is precisely by putting the web of interactions (see p. 17 in report) and its outcomes in the spotlight that it becomes clearer what changes in behaviour and relations have already taken place due to the research project, the researchers’ involvement, and the efforts of the stakeholders.
What was the result of the SIAMPI analysis?
This impact assessment, guided by SIAMPI, resulted in a report. It contains a narrative that summarizes in which way the Nudging in Education project aims to achieve societal impact. Furthermore, there is a table that captures the different kinds of productive interactions (direct, indirect, financial) that already do, or in the future might, contribute to this change (see here or report p. 13).
In addition, the ESI team decided it would be worthwhile to visualize the web of productive interactions. Therefore, they devised a stakeholder map of the Nudging in Education project, in close consultation with the actors. In this visualization we decided to include the level of involvement and participation of different actors. The level of involvement was indicated through the participation ladder which is often used in public administration (see Edelenbos and Monninkhof, 2001). This visualization (p. 17 in report) created a good opportunity to discuss and (re)evaluate the involvement of the stakeholders in one of their consortium meetings. This visualization made the role of the partners more transparent and stimulated further discussion on their modes of interaction.
We would like to highlight that such a visualization is not a standard or ‘requirement’ of a SIAMPI evaluation. In this case, the visualization not only improved the understanding of the ESI analysts, but it also provided a basis for deliberate conversations amongst all stakeholders involved in the process.
The ‘Nudging in Education’ project is an example of scientific research with a high degree of practical relevance, where different societal stakeholders are enabled to co-create based on their own expertise. The entire project is a learning environment in which people learn from one another and social contacts are intensified. Crucial is the key role of the PhD-student, who has a strong mediating role between the practice and the scientific environment. The result is a strengthened network, changes in attitude and awareness of teaching skills and the teacher’s own role, as well as an increased degree of knowledge sharing within and outside the consortium. These outcomes contribute to the intended societal impact of teacher professionalization and increased student autonomy.
Please find the report 'Nudging in het hoger onderwijs 2021' below as a downloadable file.