© The Author (2009). Published by Oxford University Press. All rights reserved.
This article appears in the following Journal of Human Rights Practice issue: Special Issue: Where Is The Evidence? [View the issue table of contents]
Evaluation at Minority Rights Group
Minority Rights Group, 54 Commercial Street, London E1 6LT, UK
claire.thomas{at}mrgmail.org
 |
Abstract |
|---|
 Top Abstract Evaluation at Minority Rights...Evaluation and Programme Design...What Have We Learnt...What Are the Challenges?ConclusionNotesReference |
|---|
This practice note sets out some of the thinking behind evaluation at Minority Rights Group (MRG) and looks at our current practice in this area. It considers how MRG has changed in terms of evaluation in recent years. It discusses what we have learnt about evaluation, and the current major challenges that we face in this aspect of our work.
Keywords: attribution, intervention logic, outcome mapping, theory of change, time frames, triangulation
|
Evaluation at Minority Rights Group |
|---|
|
TopAbstractEvaluation at Minority Rights...Evaluation and Programme Design...What Have We Learnt...What Are the Challenges?ConclusionNotesReference |
|---|
This note sets out Minority Rights Group's (MRG's) approach to evaluation. It discusses how we actually do evaluations and considers some changes in our approach to evaluation over recent years – as well as highlights some of the lessons that we have learnt about this aspect of our work. Finally, the note reflects on the challenges that we still struggle with regarding evaluation.
MRG is a relatively small organization with around 30 staff and a budget of approximately £2.2 million. We are an organization with a very focused mandate: to secure the rights of minority and indigenous communities around the world and to improve cooperation between communities. Essentially, our work concerns empowering minority and indigenous communities to be aware of and to demand fulfilment of their rights to equality in all areas of life. We train minority activists, and we support advocacy campaigns and legal cases to bring about policy changes and implementation. Although primarily a human rights organization, we are to some extent ‘inter-disciplinary’ – also working within the paradigms of development, conflict prevention and anti-racism when appropriate. This means that we can learn from good practice in each of these related professions. This has been important to us as it has given us access to a broader menu of tools, approaches and thinking from which we have been able to pick and choose elements that we find useful.
MRG believes strongly in the value of evaluation. We include external evaluations as a standard part of all our projects. We also carry out organization-wide evaluations or reviews periodically (which are not required by donors), and we hold ‘learning from evaluations’ sessions to pass on the conclusions of particular evaluations to all staff. A comparative evaluation of nine international human rights organizations in 2004 reported: ‘Overall, MRG has a well-defined mandate, which is complemented by a high degree of internal reflection over strategies, projects, and areas of work. ... its evaluation procedures are extremely well established’.1
At MRG, a typical evaluation involves commissioning an external consultant who will review all of the monitoring data for a particular project or programme, and contact all of the partners and some beneficiaries – as well as some advocacy targets or independent informed people. Using all of these data, she/he will then reach an independent view of how the project was run, its results and potential long-term impact. Clearly, exactly who is spoken to and the records reviewed vary according to the nature of the programme. Where we have aimed to build the capacity of individuals, the evaluator will contact people who have attended training (possibly several years after their attendance) and ask whether they still apply any of the skills or knowledge gained. If not, she/he will be interested in why this is the case. Where we have aimed to build the capacity of organizations, the evaluator will independently review the records – assessing the organization's capacity at the beginning and end of the project, verifying any claims and assessing the significance of any changes.
A typical positive conclusion on an evaluation might read something like:
It is clear that partners feel much more confident on issues of human rights and advocacy since the project began in 2001. Links with government and NGOs are strengthening all the time ... . Not all of today's relationships can be ascribed to the MRG project, but the project certainly played a vital part in this outreach process.2Any shortcomings of projects are normally expressed as recommendations. Often, MRG is already taking steps to implement evaluators' recommendations – as we have also reached similar conclusions about an area of work. The evaluation quoted above (concerning MRG's work on the rights of the Batwa pygmies) also noted that:
Future interventions should seek to address the factors that keep women out of the realm of political discussion and awareness. These include women's daily struggle to feed their families, fear of attack when travelling, childcare obligations, and lack of education. One approach would be to find ways of relieving daily burdens or perceived risks so that women become available to attend workshops and information-sharing opportunities. Prior to such events women should be consulted as to what format and timetabling would enable their participation. Since Batwa men tend to be more confident about speaking in public it may be worth considering conducting gender-specific sessions.3In this instance, MRG had anticipated this feedback and was already fundraising for a new programme of work that focused specifically on the needs of Batwa women. Other feedback from evaluators has identified a weakness around disseminating our publications, or continuing dilemmas in terms of our working relationships with partners.
|
Evaluation and Programme Design Evolves over Time |
|---|
|
TopAbstractEvaluation at Minority Rights...Evaluation and Programme Design...What Have We Learnt...What Are the Challenges?ConclusionNotesReference |
|---|
This section will discuss how MRG's evaluations have helped us design better and more effective programmes; how we now integrate all of our activities into concerted campaigns; and how we look beyond completing activities and focus on the outcomes and impacts of our work in the real world. Whenever I am asked to talk or write about evaluation, I always end up first talking about the programme design – as the most important factor in determining whether a programme is successful or not is probably whether the initial design is right.
MRG has been around for 40 years, and over the decades we have achieved many successes (examples might include, at the macro level, the 1982 UN Declaration on Minorities, and at a local level, our mentoring programmes with young Roma leaders run through the mid-1990s – which are still referred to and emulated today). However, 10 or 15 years ago, like many others, MRG was less conscious of achieving specific outcomes and our activities were less targeted. We would identify a problem and we would generally apply the same set of solutions to each and every problem. MRG's projects generally contained training, research/publications and international advocacy, because these were the things we were good at, that we were known for, and that we were comfortable doing. Sometimes these were exactly the type of activities needed and the projects succeeded – but other times, the situation really called for another kind of intervention and our project did not make much difference.
Project evaluations were very important in changing this, as they told us that, in some cases, despite doing high-quality work thoroughly and well, the activities were not bringing about the changes that we hoped. It was essential that these evaluations looked beyond activities and examined the overall impact or effect of the work. Where projects achieved less, this was often because we had not fully analysed the problem in sufficient depth in the early stages. We may have made assumptions about what was needed or what one initial change would lead to. We may not have understood all the factors that influenced or affected the problem we were trying to solve or progress. For example, it was quite common for MRG to assume or to believe that publishing a report about an issue would, in and of itself, somehow bring about a change in policy. It was as if a lack of knowledge or awareness of a problem was the sole factor preventing change – whereas in fact ‘stand alone’ reports only ever result in changes if someone else picks up the conclusions and recommendations and campaigns for them to overcome the status quo, vested interests and inertia that almost always prevent change. Relying on someone else to pick up your research and campaign on the issues is very haphazard. One example of an evaluation finding that captures the issue of needing to think through how to bring about changes more carefully is:
At the overall strategic level MRG is found to have been a pioneer in seeking to enhance minority rights (MR) in development and to establish linkages between the human rights and the development community. The overall relevance and justification of the programme can easily be confirmed. The pioneering work has necessitated an explorative and iterative approach. Yet, a more articulated strategy and change perspective could have enhanced MRG's performance.4As a result, around 2003–04 we started to think through problems much more carefully. We now try to identify a very specific change that we are attempting to bring about and we try to think very carefully how to maximize our chances of making this specific change happen. In essence, we should now have a very well developed intervention logic or theory of change for each project or programme. We now rarely produce ‘stand alone’ publications. Instead, research is now dovetailed with programmes of capacity building, advocacy, and legal work in support of the research findings.
For example, a recent programme in Turkey produced three publications: a report on Turkey overall; a report on minorities and education in Turkey; and a guide for those displaced on the right to return. All three publications are closely linked to our strategy to maximize the use of the ongoing European Community pre-accession process and dialogue to ensure meaningful change in the Government of Turkey's attitudes to and policies on minorities. Thus, we have visited Ankara and Brussels to discuss each report in detail with decision-makers and have run a series of other linked workshops and events to push for the recommendations made in each report to be implemented.
Our ‘inter-disciplinary’ nature has been very helpful here as we have been able to take programme design methods like problem tree analysis, logical frameworks, and baseline assessments, which are more frequently used in development circles than in human rights, and apply them to our work. Most of our borrowings have been from the development sector as they have done a lot of work on design and evaluation. It is not always straightforward to take a methodology and apply it exactly as organizations in another sector do. Another area where development organizations have made great progress is ensuring participation in the design and evaluation of programmes. This is usually more problematic in human rights work, where initiatives are more likely to be national than local and also where governments are more likely to be hostile to initiatives. Often though, there is a kernel or essence that lies behind a method that we can identify and apply in our own ways. MRG does now try to ensure that samples of beneficiaries are involved in many evaluations. In borrowing from other organizations (whether from the development sector or not), we aim to stay a few years behind the cutting edge. In a sense, it is preferable to allow larger and better-funded organizations to be those who experiment with new techniques. When the usefulness (and any limitations) of a new approach have been identified by these more abundantly resourced organizations, we are better able to consider whether it might work for us.
Although we do still evaluate individual activities on occasion – e.g. training events and occasionally publications, we are now much more likely to evaluate whole programmes of work. We are less likely to focus exclusively on whether a piece of work was done well, and more likely to extend our consideration to whether it has had the desired effect. This shift was at the time quite difficult for staff – as it took the focus of the evaluation away from what we as an organization can control (running high quality pieces of work) and into the realm of what we have managed to achieve where we may have very little, if any, power and only limited influence. Our programmes involve influencing complex networks of actors who compete and cooperate, with a multiplicity of different interests and forces having a bearing on what we want to change. This complex pattern also shifts over time, so that the situation as analysed when the programme started may change midway during implementation. In these contexts, programmes are judged not on what they have delivered or produced in the first instance but on how this has been used, responded to, taken up or ignored by the targets of the work. This is a different level of ambition and much more likely to have some degree of ‘failure’ attached – through no fault of the staff working on the programme. Staff have, though, now become used to being evaluated at this level and, where findings are positive, it confirms the extent to which our work ‘has made a difference’ and can therefore be very supportive of staff's motivation and commitment.
In doing evaluations, we try to work out what has worked really well in terms of achieving what we set out to do and why – in order to consider whether the methodology may work elsewhere. We also try to work out what has worked less well and why – the better to avoid making the same mistakes again.
|
What Have We Learnt about Evaluating? |
|---|
|
TopAbstractEvaluation at Minority Rights...Evaluation and Programme Design...What Have We Learnt...What Are the Challenges?ConclusionNotesReference |
|---|
Attribution
It is true that attribution is a difficult issue, but we feel this does not prevent meaningful evaluations. Almost all evaluations will be qualified in terms of acknowledging that other actors played a role in achieving any results. Where organizations work with partner organizations, this will always be the case. Where organizations carry out advocacy, this will also always be the case. Only in service delivery can a partnership claim to have single-handedly brought about results. However, evaluators can and do sometimes say that the contribution made by an advocacy project was a necessary but not sufficient factor in achieving an outcome. In other words, that it was one of several elements provided by different contributors – and that without any one of these elements, an outcome would not have been achieved (i.e. to make a finding along the lines of : ‘If MRG had not run this programme, the outcome xx would not have occurred. However, for xx to occur it was also necessary that several other actors took particular decisions or acted in particular ways’).
In our field, evaluation inevitably involves making judgements and it is never a completely scientific or quantitative process. To decide not to evaluate because you can never be entirely sure to what extent your organization contributed to an outcome when several actors were involved is counter-productive – and perhaps an example of the good and useful being set aside because it is not perfect.
Triangulation
One of the problems with evaluating advocacy is that the targets of the work are generally inherently unlikely to admit (particularly on the record) that they have been influenced by a campaign. This is a problem, but again it is not insurmountable. Generally, you can find independent third parties who have been watching a situation closely or who are experts and who will give their opinion. Third-party views can be particularly useful where changes sought have included changes in attitudes or changes in the tone of media coverage – giving greater visibility or influence to minorities or to a minority issue (normally milestones on the way to a more concrete policy change). They can also be useful where judgements are needed on how significant a particular policy change is – as some of these may be window dressing, while some are meaningful, substantive changes. In our evaluation terms of reference (TORs), we generally suggest that we will supply a certain number of contacts for people we know are aware of the project. But we then also ask the evaluator to independently contact a number of people not on our lists from their networks or research who are knowledgeable about the area in order to get their views, too.
Outcome Mapping/Downstream Impact
Outcome mapping is a very specific evaluation methodology that moves away from predictable simple causes and effects and aims to track the way that inputs flow through complex systems.5 It is generally also much more participatory than many other evaluation methods currently in use. One element of outcome mapping is sometimes termed ‘downstream impact assessement’. The analogy often used is dropping dye into a river and then seeing where it is carried by currents and how it flows through the river system. In human rights programmes, this method involves pinpointing one human rights programme or intervention and trying to track what changes have resulted from this over time. We have not applied outcome mapping fully to our work, although one of our evaluations did experiment in applying some of the principles with interesting results.6 The programme evaluated involved training and supported advocacy by partners. It showed that it is possible to track the downstream impacts of advocacy and training and that in some cases these effects were long lasting, i.e. still noticeable six or more years after the event. It showed that outcomes generally resulted from partners applying new skills learnt consistently and at different fora over a period. There were few if any traceable impacts resulting from the one advocacy intervention supported by the programme. Not surprisingly, where changes had occurred, many different actors or factors had had a bearing.
Internal/External
Generally, MRG does commission entirely independent and external evaluators to review our work. Certainly, the fresh eyes and independent viewpoints are extremely useful whether they identify aspects of programmes that we had not thought of or independently confirm what we had already privately concluded. However, we have also experimented with staff running evaluations – including self-evaluations, where the staff who have been in charge of work in a particular area later evaluated that work. It is often felt that staff would be more positive about the work than an outside party or that they would be blind to any failings. In fact, this was not the case and this particular evaluation7 was more critical of both the work and its impacts than our average evaluation or the external evaluations of parts of the work that were covered in this exercise.
Having an internal evaluator had certain advantages. It is probably the case that other staff were more likely to accept and learn from lessons identified by one of their colleagues. The fact that an internal evaluator will have an in-depth understanding and appreciation of the work, and has confronted the day-to-day problems that arise in the work, can be contrasted with the views of an evaluator who is seen to drop in, review and then move on to other things. It also means that you have very great ownership of lessons learnt – of which the internal evaluator will remind staff when appropriate (otherwise, it usually falls to senior staff to do this). Some evaluation findings do have more credibility when made by an external evaluator, e.g. the finding that we should budget for more staff time on a particular programme. Helpfully, this was made by an external evaluator who, unusually, was identified and commissioned by the major donor to the programme and not by MRG.
Learning
Evaluation is pretty pointless if you do not take the lessons and the recommendations made and then apply them. Every few months at MRG we review recently completed evaluations and extract some findings or lessons that we think are generally relevant across many programmes, and we get together to discuss them. We discuss the feedback and whether we agree with it; how we could change things to address the points made; and how to take these changes forward. All staff are involved in the evaluations of their own programmes. In most cases, they are already aware of some shortcomings and agree with the evaluators' assessments. Where there are differences of opinion or judgement between evaluators and staff, these are often thrashed out during the interviews or meetings that are part of the evaluation process. As we have an established culture of evaluation and clearly value the learning from evaluations, staff are now generally very open to the feedback. Evaluators often find positive as well as constructively critical things to say about all programmes and the fact that learning is phrased in terms of recommendations is helpful. No individual staff member at MRG has sole responsibility for the design of a programme, so where shortcomings are at the level of the logic of the intervention and not its implementation, responsibility is always shared across a wider team.
Monitoring
Monitoring and assessing progress is essential throughout the programme, and it cannot be left to the final months. Otherwise, you will not know whether you are on track to achieve what you set out to do, or whether you need to adapt as circumstances change. Monitoring should also help you assess early on whether your analysis was accurate – and should take place in a timely fashion in order to allow for any necessary changes. If you have not monitored, you will not have material for an evaluator to work with.
Timing
Although it is tempting to wait until all work is finished before evaluating, we now try to commission an evaluation six months before the work ends. If an evaluator will spend, say, 20 working days on an evaluation, this is as likely to be spread over six months as compressed into one month or six weeks. Ideally, the evaluator will attend at least one programme activity, attend a final partners' meeting, as well as review all of the materials and speak with partners. Wherever possible, targets and independent but well-informed or expert commentators should also be contacted.
Use Evaluations to Celebrate Success
I was once told of an organization that only ever commissioned an evaluation when it wanted to close a programme. Evaluations were the death knell for a piece of work and, as such, were dreaded by everyone who worked there. MRG aims to go as far to the opposite end of the spectrum on this as possible. If we have a particularly successful programme, we will always evaluate it. We may well want to try to replicate some of the methodology elsewhere, and if so, we really need to understand what factors made it work so well. Within a programme, we will always highlight the positives in a project as well as any more critical lessons learnt.
|
What Are the Challenges? |
|---|
|
TopAbstractEvaluation at Minority Rights...Evaluation and Programme Design...What Have We Learnt...What Are the Challenges?ConclusionNotesReference |
|---|
It is often somewhat difficult to separate lessons learnt from remaining challenges. In addition to challenges alluded to above, other challenges remain for MRG in its evaluation work.
Time Frames
Compared with the relatively minor issue of the timing of evaluation at the end of a programme, the time frames needed to allow impacts to emerge are a very real challenge in any exercise. MRG often runs programmes which are two–five years long (which is both as far as we can reasonably plan ahead and as long as most institutional donors will grant funding for). However, when we embark on work in a new area, we often know that it is likely to take eight–10 years, or even more, to have a real impact on entrenched discrimination and complex cycles of exclusion that are very hard to break. We deal with this by breaking programmes into phases with different interim outcomes for each stage or phase.
An example I often quote to staff is that of monitoring HIV infection rates where there is a long time delay between infection and any symptoms in many cases. In the early stages of their work, organizations working on HIV/AIDS dealt with this by identifying other sexually transmitted diseases that are spread in the same ways but which produce symptoms much more quickly. They took these diseases to be proxies that could stand in for HIV infection rates. My search for similar shorter term proxies in the area of human rights is still ongoing, but it would be lovely if we could use this sort of thinking to help us out of this problem. It may be tempting to simply monitor for eight or 10 years and only finally evaluate at the end of the whole period of intervention, as prior to this you are unlikely to be identifying very many real sustainable impacts. However, the time lag adds to the attribution problem as it increases the number of other factors influencing the problem or situation. It is also difficult in practice as it is harder and harder to stay in contact with all the major players, and the number of people who have seen the process from beginning to end and can comment on it as a whole is reduced dramatically. In an ideal world, you would evaluate as you went along over, say, 10 years – but your final evaluation would not only evaluate the final phase but also would look back at the whole 10-year intervention and seek to view it as a whole.
Participation/Accountability to Beneficiaries
Although we involve partners in all evaluations, we are not as consistent in involving as many beneficiaries as we should in all cases. Also, our evaluations are not designed to be read by or reported back to beneficiaries. This is something that we need to improve on.
Funding
We include budgets for external evaluations at the end of all projects (and interim ones part way through some projects). Our donors all welcome these, and some require them. Therefore, funding for evaluating a specific project should not be a problem, provided that it is ring-fenced and protected. That said, as it is the last cost on any project it is always vulnerable to being cut if not all the funding is finally received or if there are overspends elsewhere.
However, evaluations that go beyond a specific project are much more challenging to fund. MRG has very limited unrestricted funding and far more demands for it than funding available. Although there is a clear need for evaluation that reviews the work of the organization as a whole, these are very hard to prioritize given other demands. Occasionally, a donor will commission such an evaluation and pay for it – as was the case with the evaluation of nine human rights NGOs quoted above – but this is rare.
|
Conclusion |
|---|
|
TopAbstractEvaluation at Minority Rights...Evaluation and Programme Design...What Have We Learnt...What Are the Challenges?ConclusionNotesReference |
|---|
Evaluation takes up time, money, effort – but it is essential. We are still feeling our way in terms of how to make human rights work translate regularly and reliably into concrete and sustainable differences for marginalized and excluded communities around the world. Evaluation confirms when things work well. It highlights problems and should mean we do not repeat mistakes. Sadly, identifying a problem does not always mean solving it and MRG continues to struggle with many difficult and intractable problems that are identified again and again in evaluations, but which do not have easy answers. An example of this is how to organize our work with partners most effectively. Sometimes different evaluators will identify the same problem, but propose diametrically opposed solutions. One might say that we should have delegated more responsibility to our partners, while another might say that we overestimated the capacity of partners to carry out work. But the insights and feedback from our evaluations has definitely helped us to improve our work and to make it more effective. I consider the time, money and effort all well spent.
|
Acknowledgements |
|---|
MRG is very grateful to all the donors who have supported evaluations of particular projects over many years (too many to mention) as well as the funders who continue to support us with much needed and much too scarce unrestricted funding.
|
Notes |
|---|
|
TopAbstractEvaluation at Minority Rights...Evaluation and Programme Design...What Have We Learnt...What Are the Challenges?ConclusionNotesReference |
|---|
1 Landman and Abrahams (2004). All the evaluations that MRG has completed in recent years are available at http://www.minorityrights.org/6287/evaluations/.
2 Luke Freeman, ‘Promoting the Rights of the Batwa Pygmies: Recognition, Representation and Cooperation’. Available at http://www.minorityrights.org/6287/evaluations/batwa-project-evaluation.html.
4 Hanne Lund Madsen, ‘Evaluation of Minority Rights and Development Programme’. Available at http://www.minorityrights.org/6293/evaluations/evaluation-of-minority-rights-and-development-programme.html.
5 For explanations of this evaluation method, see http://www.idrc.ca/en/ev-26586-201-1-DO_TOPIC.html.
6 See Mike Dottridge, ‘Assessment of the Impact of NGO Activities Supported by the Minority Rights Group (MRG) at the Time of the World Conference against Racism’. Available at http://www.minorityrights.org/6296/evaluations/world-conference-against-racism-wcar-evaluation.html.
7 Anne-Marie Biro et al., ‘Comprehensive Evaluation of Programmes Implemented in Central Eastern and Southeast Europe from 1996–2002’. Available at http://www.minorityrights.org/6298/evaluations/comprehensive-evaluation-of-programmes-implemented-in-central-eastern-and-southeast-europe-from-19962002.html.
|
Reference |
|---|
|
TopAbstractEvaluation at Minority Rights...Evaluation and Programme Design...What Have We Learnt...What Are the Challenges?ConclusionNotesReference |
|---|
-
Landman T., Abrahams M. Evaluation of Nine Non-Governmental Human Rights Organisations. (2004) 86. Available at http://www.minorityrights.org/6302/evaluations/evaluation-of-nine-nongovernmental-human-rights-organizations.html.
CiteULike 
Connotea 
Del.icio.us What's this?
| ||||||||||||||||||||||||||||||||||||||||||||||