Homework answers / question archive / Measuring the Impact of Zoonotic Disease research in Emerging Livestock Systems

Measuring the Impact of Zoonotic Disease research in Emerging Livestock Systems

Writing

Share With

---

Measuring the Impact of Zoonotic Disease research in Emerging Livestock Systems. A realist evaluation of zoonotic influenza social policy change mechanisms in Bangladesh.

Introduction

The notion of social Impact requires research to bring about Change (2-4), which we can define as a demonstrable difference for a specific outcome upon a specific population e.g. communities, groups or individuals (7), which could be through the economy, society, culture, public policy or services, health, and/or the environment (5).  The role science plays in forming and shaping society and culture has been well studied (1).  The notion of Impact requires research to bring about Change (2-4). This change must have an effect on people and might include villages individuals, households, groups, communities or communities, practitioners or organisations (2), and  can be through the economy, society, culture, public policy or services, health,  and/or the environment (5).  The population can be defined as outcome of the economy, society, culture, public policy or services, health, the environment (6), individuals, households, communities or firms (2), or simply a particular group of people (7).  Many of these definitions go further to discuss the role of the research project in delivering this change in outcome on the defined population (8). A demonstrable difference to be achieved for a specific outcome on a specific population is a requirement for impact. 

 

The global financial crisis of 2007, Covid-19 and the energy crisis have precipitated increased scrutiny over expenditure, with large-scale cuts to public services including health and education.  Research and development money must therefore be justified, and its effectiveness evaluated.  The ‘Impact Agenda’ has evolved gradually through changes in research funding allocation and measurement of academic excellence.  In 2014 the Research Excellence Framework (REF) introduced ‘Pathways to Impact’.  The REF accountability review (23)  estimated the cost of conducting the REF at £246M  (24), highlighting the political establishments commitment to evaluate research impact.  Understanding whether applied research achieves change for those whom it was designed is of central importance, both for intrinsic and extrinsic assessment criteria.   For the purpose of this research, I am considering the role of knowledge generation on social change and international development.   Consequently any evaluation should be framed in the context of the objectives of the aid agenda, and must ‘promote the economic development and welfare of a developing country, or countries, as their primary objective’ and be aligned with the Paris Declaration Principles (10).  

Process-based approaches (8) have been used to monitor and evaluate development, or complex medical interventions, making key assumptions about the way these processes create impact, and allowing the researcher to hypothesise on direct or indirect processes which might lead to change. These are particularly effective in understanding complexity (32, 33).  Process-based approaches  allow attribution to be built into the analysis (4).  Program theory focuses on explaining how the processes occur in contrast to other logic models or pipeline approaches (34). The method requires the development of a causal chain of events leading to the desired ‘big picture’ goals (35), focusing the direction of the change and measuring indicators along the way to creating this change.  These non-counter-factual methods are largely qualitative, although many contain quantitative hypothesis-based indicator measures. Examples of these approaches  include: Process evaluation (36), Realist evaluation (37),  General Elimination Methodology (38),  Process Tracing (39), Qualitative Impact Pathway analysis (QuIP)(40), and Contribution Analysis (41).   The most robust for forming causal arguments is realist evaluation. 

 

2.            Methods

Overarching methods.

Data were gathered from Sept 2016 – May 2019 primarily in Dhaka, Bangladesh and online teleconference.    Data were primarily qualitative including, in-depth interviews  and Group Interviews, containing 53 participants, with 59 audio files and transcripts, participant observations of key dissemination events, and policy documentation.   Quantitative components consisted of a short, structured feedback questionnaire to elicit knowledge of project outputs and perceptions of policy options. Analysis consisted of (i) narrative formulation for each policy component, (ii) meta-analysis of each of the policy formulation processes with thematic evaluation (iii) realist evaluation conducted in a knowledge dissemination and knowledge utilisation phase.

Interviews.

Interviews followed a mixture of open questions, semi-structured interview guide and realist questioning. Individual written consent was obtained, and data were audio recorded and transcribed for analysis.  Participants are detailed in Appendix 1.

Policy Documents.

Policy documents were collected in-country and interviewees were asked if there was additional relevant documentation.  Specific legislation (Animal Slaughter Act and Animal Welfare Act) was only available in Bengali and was translated into English for analysis.

Feedback forms.

Perceptions and knowledge of research outputs for avian influenza research were elicited through the use of feedback forms (n = 40).   These forms were collected in 2017 following the round-table event organised by Chatham House and participants were also asked to complete the form during the interviews.  A similar feedback form was used for interview participants in 2019 and attendants of the one-health conference.  Feedback form results Appendix 2.

 

 

3.            Narrative Analysis

Background and Introduction

In order to critique and evaluate the current effectiveness of knowledge use for policy change, one must first explore the mechanisms through which evidence has been used to create change historically, i.e. the pathways by which this has happened.

There are many different types of narrative analysis appropriate for research questions.  The current guidance for research impact case studies focuses on a narrative approach (30), however there can be lessons learned from the established literature on narrative research (48, 49).  The way that social actors produce, represent and contextualize their experience and personal knowledge can be considered a Narrative, how they make sense of what happened, which is often a reflection of experiences in their own lives.  Stories have been used by many cultures for centuries to provide explanations or lessons which have meaning, so it makes sense for social sciences to analyse these narratives. Some of the characteristics include: how people create meaning in terms of who they are -  Including how organisations make sense of the world (50), e.g. How policy documents make sense of the world.

Norman Denzin (51) considers the narrative as a story or a sequence of events that have significance for narrator and audience.   The narrative has a beginning, middle and end, and a logic.  Narratives are also temporal – they have a causal sequence, one event or experience sets the scene for the next  (52, 53). Lieblich (year) (54) goes beyond this to define the type of narrative as holistic (big picture) or categorical made up of content or form, to allow for more structure in the analysis.   Reissman (year) (55) described a formative narrative analysis , whereby the structure of a story is used to analyse how people choose to tell the story, this means how words are chosen, and what the choice of words (or evidence) used says.  Analysis can also include how narratives are linked in with structures of power.  Lavrov’s educational model uses a different structural form consisting of abstract, orientation, complication, evaluation, results and codes as a basis for analysis.  

Discourses are bound up with institutional practices, ways of organising social life and focusing on the power and the politics behind them. Fouccoudian approaches often take a position in contrast to the overarching oppressive discourse in order to understand power dynamics – the macro level of discourses (56).   Leach, Scones and Stirling build on this argument to understand dominant and alternative narratives to give a voice to repressed narratives. (57)

Critics of narrative research might include arguments such as questioning the motivation for stories, e.g. Illness narratives of coping; which might be an attempt to look competent rather than truly coping. (58)  Another criticism is how competing narratives are managed.  Eg. In organisations competing narratives may reflect different political ideologies.  But sometimes research needs to offer reasons, causes and accounts (50).  This approach will allow development and discovery of the pathways knowledge has been used to create policy change from a historical perspective in Bangladesh.