© The Author (2009). Published by Oxford University Press. All rights reserved.
This article appears in the following Journal of Human Rights Practice issue: Special Issue: Where Is The Evidence? [View the issue table of contents]
Producing the Evidence that Human Rights Advocacy Works: First Steps towards Systematized Evaluation at Human Rights Watch
Human Rights Watch
Program Office
2-12 Pentonville Road
London N1 9HF
UK
gorvini{at}hrw.org www.hrw.org
 |
Abstract |
|---|
 Top Abstract IntroductionThe Context at Human...Seeing the Opportunities,...Turning Concept into PracticeConclusionsNotesReferences |
|---|
In the community of human rights activists and professionals, we share a conviction that we make a difference. But attributing positive change to our own work is often uncertain. At the same time, as our presence in the media and in discussions with policy-makers grows, and is seen to grow, we face a hostile audience as never before. There are many—our direct targets and others—who would like to discredit and dismiss human rights organizations, or are skeptical of the value of condemning human rights abuse in the absence of an appetite among influential governments to apply meaningful leverage. Both the hostility and the skepticism raise the stakes for us to explain our purpose and our tactics, including in terms of how we assess that we are effective.
Yet even major, established human rights organizations are still getting to grips with more systematically evaluating whether and how we achieve the outcomes we seek from our advocacy efforts. How do we locate the reliable evidence that our approaches to human rights problems actually work? And if we do, how do we make that exercise truly worthwhile by establishing an organizational culture of evaluating and learning? This paper aims to present Human Rights Watch's work-in-progress as we think through and pilot a systematized evaluation process.
Keywords: impact, institutional learning, monitoring, nongovernmental organizations, theory of change
|
Introduction |
|---|
|
TopAbstractIntroductionThe Context at Human...Seeing the Opportunities,...Turning Concept into PracticeConclusionsNotesReferences |
|---|
What do we think? What do we know? What can we prove? That mantra, which I first encountered in the movie adaptation of Randy Shilts's ‘And The Band Played On’, about epidemiologists' endeavours to isolate the virus causing AIDS (Spottiswoode, 1993), epitomises the rigor that should be present in any undertaking to produce and present evidence. It applies as much in the human rights activist's world as in the clinician's: research findings on which we base our case have to stand up to intense scrutiny if we are to be credible advocates.
Now, the ‘think-know-prove’ mantra has a newer application as organizations like Human Rights Watch get to grips with more systematically evaluating whether and how we achieve the outcomes we seek from our advocacy efforts. All of us who engage in human rights promotion and protection—volunteer activists, professional nongovernmental organization (NGO) workers, our vast array of partners in other parts of civil society, and the personnel of the international/intergovernmental human rights monitoring system alike—do what we do from a conviction that our work can or should make a positive difference. But expressing why we think or know that, and making a convincing, evidence-based case to others that there is a good foundation for our own conviction, is often challenging. So too is the ability to learn from the evidence we find for impact and how we achieved it, and having that learning influence strategies and planning.
Like many similar organizations, Human Rights Watch finds itself grappling with the truism that achieving meaningful change in human rights work is difficult—particularly in the short term—and subject to a wide range of influences, many of which are beyond our control. Attributing positive change to our own work is often uncertain; even more so is attributing the absence of negative change (i.e. things did not get worse).
How, then, do we locate the reliable evidence that our approaches to human rights problems actually work? And if we do, how do we make that exercise truly worthwhile by establishing an organizational culture of evaluating and learning? This paper aims to present Human Rights Watch's work-in-progress as we think through and pilot a systematized evaluation process.
|
The Context at Human Rights Watch |
|---|
|
TopAbstractIntroductionThe Context at Human...Seeing the Opportunities,...Turning Concept into PracticeConclusionsNotesReferences |
|---|
Human Rights Watch is not alone in being an organization that is established, experienced, and trusted in the field of research and advocacy on human rights, but has no truly systematized process for evaluating the effectiveness of its project work. We took up the question of how we build such a process and the culture that supports it as we embarked on a strategic review that ran through 2008.
Of course, Human Rights Watch, in existence since 1978, would not have pressed on for three decades without evidence to support our conviction that our work does make a difference. We have highlights to point to across the years, from our pursuit of former Peruvian president Alberto Fujimori leading to his arrest in Chile, his extradition, and his conviction in Peru for crimes relating to serious human rights violations (Human Rights Watch, 2009a), to sharing in the 1997 Nobel Peace Prize as a founding partner in the International Campaign to Ban Landmines. More broadly, a quick internal survey in the spring of 2008 confirmed that we were already doing a variety of activities related to evaluating impact: project design documents inevitably identify impacts being sought; uptake of the message and recommendations in reports, news releases, and other initiatives is routinely monitored; and annual retreats of individual departments and annual planning meetings include reflection on work delivered. As an organization supported financially by contributions from private individuals and foundations, our reporting to donors spotlights the positive impact of our work. More rarely there has been an in-depth evaluation of specific bodies of work as a donor requirement: for example, the body of work on Angola from 2002 to 2008 was subject to a review by an external evaluator.
Despite these experiences, the survey highlighted that our exercises in evaluation are non-systematic, and insufficiently resourced.
Addressing the deficits in our evaluation process and culture has acquired an urgency in direct relation to our organization's growth, visibility, and internationalization. From our beginnings as the New York-based Helsinki Watch, Human Rights Watch has grown in size and global footprint, especially in the past five years—the number of people we employ grew by more than 10% annually between 2004 and 2008. Our staff now total over 250, operating out of more than a dozen main offices in major cities in North America, Europe, the Middle East, Africa, and East Asia, and a similar number of field offices in or adjacent to countries that are continuously a focus of our research. In all, Human Rights Watch currently researches and advocates on issues in around 90 countries.
Yet until June 2009 we had no venue for recording institutional memory that was readily accessible to all staff, and to carry practice from one project to the next we were excessively dependent on individual staff members' memory, and on corridor discussions—acceptable perhaps when you are a small staff concentrated in an office in New York City, but problematic when staff are now spread across four continents and 15 time zones.
Add to that the challenges that go with the success of becoming more visible. As our presence in the media and in discussions with policy-makers grows, and is seen to grow, we also face a hostile audience as never before. There are many—our direct targets and others—who would like to discredit and dismiss Human Rights Watch and organizations like us. And we operate in an environment also where commentators increasingly express skepticism about the value of condemning human rights abuse in the absence of an appetite among influential governments to apply meaningful leverage—what the UK journalist Simon Jenkins recently called ‘piccolo diplomacy’ in respect of the present UK government, for example (Jenkins, 2009). Both the hostility and the skepticism raise the stakes for us to explain our purpose and our tactics, including in terms of how we assess that we are effective.
|
Seeing the Opportunities, Getting Past the Obstacles |
|---|
|
TopAbstractIntroductionThe Context at Human...Seeing the Opportunities,...Turning Concept into PracticeConclusionsNotesReferences |
|---|
Within our strategic review, therefore, the starting premise for the staff discussions on monitoring and evaluation was that, by documenting the impact of our work and better understanding how and why we succeed—or not—we can become more effective. There was early consensus that project-based organizations of the size and global footprint of Human Rights Watch should have a monitoring and evaluation function as an essential component of management, coordination, planning, and reporting. With further growth internationalization an essential part of our strategic plan, we also recognized that it is imperative that we manage knowledge and information more effectively and that we develop an in-house capacity to systematically evaluate the impact of our work, to draw lessons learnt, and develop and replicate good practice.
Nevertheless, we also had to recognize anxieties and hesitations among staff about evaluation exercises. Fears about extra workload are clearly present: given the many other demands on staff, we need to build in time to do evaluation effectively. More acute is the anxiety that there will be a blurring of project evaluation for impact and staff evaluation; clearly, evaluation of impact must be seen as distinct from personal performance evaluation if it is to be an honest and unbiased exercise about which staff can feel enthusiastic.
Related to this was the concern that all our programmatic departments should not be held to the same benchmarks for evaluation. It is in the nature of human rights advocacy that change can be slow in coming, and incremental, but we also have experiences of direct and obvious success, and it is these that tend to get remembered and celebrated. Our researchers and advocates were emphatic that our success is not just defined by these ‘champagne moments’. They stressed that for any issue where we get clear forward motion if not out-and-out success, there are dozens where we are confronting the entrenched and apparently intractable — be it the abuses perpetrated by belligerents such as in Somalia or in the Democratic Republic of Congo who show no sign of ending their armed conflict and pay no mind to civilian casualties; the violations of authoritarian regimes such as in Iran, Burma, or Turkmenistan that seem heedless of their detractors; or draconian and abusive aspects of a populist law enforcement policy, such as Jamaica's criminalization of gay sex and the associated impunity for hate crimes against lesbian, gay, bisexual and transgender (LGBT) people, or European Union states' treatment of migrants trying to reach southern Europe. With little expectation of a successful outcome anytime soon, how do we determine that we are having any impact at all? And what of those situations where we confront apparently relentless backsliding, and the best we might hope for is to try to hold the line and stop things getting worse: the shrinking space for free media and civil society in Russia; the threats to political opposition and labour activism in Venezuela; China's internet censorship, for example?
Finally, we anticipated that it will prove difficult to isolate the impact of any individual actor in a shared or common advocacy effort. After all, it is rare indeed that an abusive government will come out and tell us, ‘We saw the error of our ways thanks to you, and we have changed’. Sometimes there is clear cause and effect: for example, when we exposed the scandal of a backlog of untested rape kits in Los Angeles (Human Rights Watch, 2009b), we saw the city and county government move quickly to allocate funds to tackle that backlog.1 But more often it is a convergence of multiple groups doing advocacy, prominent local activism (whether or not facilitated or supported by international groups), press exposure, diplomatic pressure, or just felicitous timing, that sees matters move in the right direction. As an advocacy group, how do you decipher your part in that?
Finding Appropriate Indicators
With these comments and questions in mind, we turned to the difficult issue of how we define impact indicators. First, colleagues warned that the best intentions to find the evidence of our impact could easily be subverted if we created a list of impact indicators that staff find mechanical, stultifying, and bureaucratic. The bigger difficulty, however, was that our past, unstructured thinking about the impact of our work had sometimes attached excessive importance to measuring outputs, and this had produced a degree of cynicism over and above the anxieties staff were having about evaluation. Within the theory of change model underpinning our strategic review, ‘outputs’ are the work we do; ‘outcomes’ are concrete, positive, and conclusive results from the work. This may sound like stating the obvious, but we found substantial confusion about these distinctions when we talked with colleagues about what constitutes a success. In particular, there was a strong attachment to interpreting visibility—especially our presence in major US media outlets—as a measure of success per se, even if that visibility did not produce the real success we were seeking, namely a positive change in respect for human rights within a given context.
When so much of the thinking in this field has derived from evaluations among non-profits in the humanitarian and service provision sectors, it is not surprising that people looking to measure advocacy success try to harness quantitative indicators. There is certainly some merit in counting outputs: numbers of media stories picking up our reporting, numbers of reactions from the advocacy target, numbers of other organizations using or quoting our materials, to name a few. But the evaluation analysis has to go much deeper than the numbers; it has to look at whether any of the outputs are or have the potential to be instrumental in delivering actual outcomes.
We determined therefore that progress indicators in our evaluations need to explicitly reflect the impact being achieved on at least three levels: outputs, intermediate outcomes, and outcomes. Our progress and success indicators also need to be sufficiently wide-ranging and subtle to encompass the projects that deliver the champagne moments, the work where our tenacity is our strength, and every kind of project in between.
Indicators of progress and success criteria should be defined taking into account that the uptake of our recommendations may be very far distant but modest intermediate outcomes reflect positive impact. We heard comments positively evaluating individual projects along the lines of ‘we kept the problem in the media spotlight, and on the diplomatic agenda’, or that we ‘challenged and debunked the alternative, more expedient narrative policy-makers had been inclined to listen to’. At the then UN Commission on Human Rights in 2005, for example, Human Rights Watch advocacy was instrumental in ensuring that Uzbekistan did not succeed in its efforts to have itself relieved of all monitoring procedures. While we did not get the monitoring mechanism we called for (Uzbekistan was retained in the confidential 1503 procedure whereas we had called for examination under agenda item 9 that took up the ‘question of the violation of human rights and fundamental freedoms in any part of the world'), our message at the Commission was a harbinger of the terrible Andijan events barely a month later, and some of those governments who in Geneva had been inclined to take a soft line were compelled by this bitter lesson to give greater credit to our and other NGOs' analysis and recommendations on Uzbekistan. As I write this, Human Rights Watch is tracking the response to our exposé of the corruption, mismanagement and associated rights violations that have accompanied Equatorial Guinea's oil boom (Human Rights Watch, 2009c), and the very obvious impact discernible in the Spanish foreign minister's reported complaint that journalists accompanying him on a visit to Equatorial Guinea are ‘paying more attention to NGO criticism of [President Teodoro] Obiang than to the positive signs the Spanish government sees’ in that country (Fernández, 2009).2
There were also strong feelings among our staff that we should continue to make advocacy recommendations for outcomes that are ambitious to the point of being practically unattainable in prevailing conditions, if to do otherwise sells fundamental human rights short. To put forward solely easy-to-implement recommendations—low-hanging fruit—for the sake of being able to say that advocacy works is to trivialize the enterprise, and risks making the work of taking on the truly difficult human rights issues harder. Instead, we aim to strike the right balance between principle and realism by making the ambitious recommendations but also making recommendations that would deliver intermediate outcomes that move matters in the right direction. This goes to the heart of a discussion on how interim outcome success criteria need to be defined in such a way that they recognize our work in intractable situations.
As our working group deliberated, we also noted two factors in our work that need to be accommodated within any overall evaluation concept and framework: that some research projects are purposely undertaken without real hope of concrete impact, at least in the short term; and that there is an element of moral imperative behind parts of our work—we cannot simply commit to do just what is ‘profitable’ in terms of impact.
|
Turning Concept into Practice |
|---|
|
TopAbstractIntroductionThe Context at Human...Seeing the Opportunities,...Turning Concept into PracticeConclusionsNotesReferences |
|---|
We have framed our overall concept as ‘Monitoring, Learning, and Impact’ (MLI).
As core principles, first, our MLI methodology and processes now in development are based on what is practical and pragmatic, oriented to our capacities and our needs, and not overly complex or burdensome. We do not now, and do not ever envisage trying to evaluate every project and burden our staff. Internal evaluation is our standard position—as most conducive to internal learning in a supportive way—but there should be room for using external evaluators, who are particularly useful for bringing fresh methodologies and perspectives.
Secondly, our MLI process should be part of how we explain ourselves to donors and supporters. The methodology should also be sufficiently sophisticated, comprehensive, and robust that—ideally—donors will defer to our own evaluation model rather than require adherence to their own; it should otherwise be easily adaptable to standard donor monitoring and evaluation frameworks.
Finally, monitoring and evaluation should inform not just knowledge management, but good management per se. MLI should inform strategy making and be integral to programming, and should be viewed not as a ‘management’ function, but something owned by staff throughout the organization, and taking place at all levels. This extends to all staff having a standing invitation to propose evaluation subjects.
Translating that into practice, we determined that we need a varied approach that makes appropriate selections and distinguishes, in terms of scope and frequency, between MLI needs at different levels. We settled on a three-tier approach:
- An evaluation at the project-level (tier 1) by the lead person on the project is being piloted, with a view to it becoming standard for every project (where project is defined as multiple activities with a coherent objective for change, and can stem from any department of the organization). Additionally, we plan to select five to six projects a year for a formal, wider-ranging MLI exercise; these would be a mix of high-profile and other projects. This will involve individual submissions using 360-degree review models, addressing questions about Human Rights Watch's processes (core departmental involvement; the coordination across advocacy, communications, and research, etc.) as well as project ‘post-mortem’ meetings where the full team of advocates, communications staff, researchers and managers would discuss what worked well and what did not, initially right after the project is launched, and again at an appropriate later date (e.g. six months after). A specific protocol for this ‘post-mortem’ will set the parameters of the discussion and facilitate systematic analysis of findings.
- Evaluation of country work (tier 2—no more than one country per year per region) will assess the outcomes of a body of work by HRW over time to determine how much a country's human rights situation has changed, and identify what could be done to improve it further. It will also be useful for identifying best practices we can replicate in other country work. Thematic evaluation (also tier 2—no more than one per year for the whole of HRW) could be conducted across regional divisions to determine the impact of related projects, with a view to refining the methodology for doing specific types of projects.
- ‘Organization-level’ evaluation (tier 3) encompasses aggregation of project-level evaluation findings, organizational impact assessments, and internal operational assessments.
As we move from framing our concept and approach to putting them into practice, we were fortunate to be able to draw on the expertise of outside thinkers and practitioners who are close to Human Rights Watch. They directed us to utilization-focused evaluation, which in essence requires that the evaluation exercise is conceived with two central questions in mind: what is the intended use, and who are the intended users of findings (Duggan, 2007). We are now adjusting our first pilot project-level evaluations, designed in our Asia Division, to a format that reflects this focus on utility and actual use, and rolling that format out to a second regional division for piloting.
Questions We Still Grapple With
The first key unresolved question is how to precisely attribute positive change. In particular, we struggle with how to reach the interlocutors outside our own organization who can help us to analyse the cause and effect in any positive outcome. Particularly in current economic conditions where we cannot boost our staff and operations resources beyond present levels, how do we reach a sufficient number of outside respondents to help us critically evaluate whether our advocacy recommendations are influential? There are practical barriers. It is expensive to send an evaluator into the field to interview target officials, local civil society, and the victims on whose behalf we advocate. Development NGOs have years of experience of doing just that, of course, but it entails years of experience with in-house evaluation units, rosters of consultant evaluators with appropriate field experience and expertise, and donors who expect a significant percentage of a project budget to be earmarked for evaluation. Advocacy NGOs, and most of their funders, have very little such experience. Long-distance communication is a poor substitute for the field research: uptake of requests for telephone interviews may be low, uptake of requests to fill in a questionnaire even lower.
At another practical level, can you ever expect your advocacy target to tell you directly whether you were influential? How do you even have that conversation—when your dialogue has for years entailed your advocating that an interlocutor change behaviour, how do you suddenly adjust the nature of that dialogue to ask that interlocutor ‘how was it for you?’
A second big unresolved question is deciding at what times and frequencies we take an evaluative look at a project. Unlike development projects, a human rights advocacy project is rarely declared ‘completed’, but very often once the major work is done our researchers and advocates move on to new projects. How often, and how far out from a project's inception, should we keep checking back for impact? Is it even practical to do so?
|
Conclusions |
|---|
|
TopAbstractIntroductionThe Context at Human...Seeing the Opportunities,...Turning Concept into PracticeConclusionsNotesReferences |
|---|
A meaningful and usable MLI framework should grow organically from the work of those who are going to use it. It is doubtful that anything in our thinking at Human Rights Watch has been original, but it seems that in the notoriously difficult area of evaluating advocacy impact, with few workable qualitative measures to go by, each advocacy organization needs to think this through for itself. Although in turning to outside expertise, we learned that the body of good practice and methodology guidance that we can all access is growing fast.
Human Rights Watch subscribes to the view put forward by Landman and Abraham, in their 2004 evaluation of nine human rights NGOs commissioned by the Dutch Ministry of Foreign Affairs: ‘It is obvious that having in place the systems and mechanisms that provide regular and frequent feedback on activities and outcomes necessarily entails additional staff, time and financial burdens on the organisation, but in the long run will contribute to its overall accountability and legitimacy. Demonstration of long term achievement of discrete outcomes linked to overall aims and objectives should contribute to the health of the organisation in its ability ... to make a demonstrable difference in the field of human rights’ (Landman and Abraham, 2004).3 In thinking through our requirements and expectations for a system enabling us to evaluate our impact, we were able to reaffirm our initial premise that trying to systematically identify and articulate the evidence that we have been effective will make us more effective: not just able to learn as an institution, but more credible and persuasive towards our audience.
|
Acknowledgements |
|---|
Many colleagues at Human Rights Watch contributed to the discussions that are summarized here, and I am particularly grateful to Iain Levine, program director, for reviewing and offering guidance on this paper. I am indebted, too, to James Ron, associate professor at the Norman Paterson School of International Affairs at Carleton University, Ottawa, and Colleen Duggan, senior program specialist with the Evaluation Unit at the International Development Research Centre (IDRC) in Ottawa, for their strong support and advice to Human Rights Watch, and to Danielle Celermajer, director of the Asia Pacific Masters of Human Rights and Democratisation at the University of Sydney, and Yasmine Ergas, associate director of the Center for the Study of Human Rights at the School of International and Public Affairs, Columbia University, New York, for involving me in their ongoing discussions on assessing impact of human rights advocacy.
|
Notes |
|---|
|
TopAbstractIntroductionThe Context at Human...Seeing the Opportunities,...Turning Concept into PracticeConclusionsNotesReferences |
|---|
1 A rape kit contains DNA taken from a victim's body after she reports the rape to the police or hospital. A rape kit backlog occurs when police make a decision not to send a collected rape kit to a DNA crime laboratory for testing, accumulating untested rape kits in their storage facilities.
2 Fernández (2009) said, ‘Moratinos ... lamentó durante una conversación informal con los periodistas que le acompañan en este viaje que presten más atención a los informes de los organismos internacionales críticos con Obiang que los síntomas positivos que el Ejecutivo español percibe en el país, sobre todo si se le compara con los países de su entorno’.
3 Human Rights Watch was not among the evaluation subjects.
|
References |
|---|
|
TopAbstractIntroductionThe Context at Human...Seeing the Opportunities,...Turning Concept into PracticeConclusionsNotesReferences |
|---|
-
And The Band Played On (1993) New York, NY: HBO. directed by Roger Spottiswoode.
Duggan C. Demystifying Evaluation & Building a Culture of Evaluative Thinking. (2009) Ottawa: International Development Research Centre (IDRC). Powerpoint presentation to Human Rights Watch, 23 April 2009.
Human Rights Watch. Peruvian Court Convicts Fujimori for Human Rights Violations. (2009a) Human Rights Watch news release, 5 May 2009. Available at http://www.hrw.org/en/news/2009/05/05/peruvian-court-convicts-fujimori-human-rights-violations.
Human Rights Watch. Testing Justice: The Rape Kit Backlog in Los Angeles City and County. (2009b) 1-56432-461-3, March 2009. Available at http://www.hrw.org/en/reports/2009/03/31/testing-justice-0.
Human Rights Watch. Well-Oiled: Oil and Human Rights in Equatorial Guinea. (2009c) 1-56432-516-4, July 2009. Available at http://www.hrw.org/en/reports/2009/07/09/well-oiled-0.
Fernández B. La oposición guineana pide a España que la mejora de la relación con Malabo no sea "a expensas" de los DDHH. (2009) Europa Press. 10 July 2009. Available at http://www.europapress.es/internacional/noticia-guinea-ec-oposicion-guineana-pide-espana-mejora-relacion-malabo-no-sea-expensas-ddhh-20090710155054.html (retrieved 14 July 2009).
Jenkins S. David Miliband's Piccolo Diplomacy. In: Guardian.co.uk (2009) 19 May 2009. Available at http://www.guardian.co.uk/commentisfree/2009/may/19/david-miliband-sri-lanka-diplomacy (retrieved 22 May 2009).
Landman T., Abraham M. Evaluation of Nine Non-Governmental Human Rights Organisations. (2004) IOB Working Document, February 2004, reproduced at http://www.minorityrights.org/?lid=6302 (retrieved 14 July 2009).
CiteULike 
Connotea 
Del.icio.us What's this?
| ||||||||||||||||||||||||||||||||||||||||||||||