ePapers Repository

Quantitative and Qualitative Methods in Impact Evaluation and Measuring Results

Garbarino, Sabine and Holland, Jeremy (2009) Quantitative and Qualitative Methods in Impact Evaluation and Measuring Results. Discussion Paper. University of Birmingham, Birmingham, UK.

[img]
Preview
PDF
Available under License Creative Commons Attribution .

eirs4.pdf
509Kb

URL of Published Version: http://www.gsdrc.org/docs/open/EIRS4.pdf

Abstract

There has been a renewed interest in impact evaluation and measuring results in recent years amongst development agencies and donors. This paper reviews the case for promoting and formalising qualitative and combined methods for impact evaluation and measuring results, as part of a broader strategy amongst donors and country partners for tackling the evaluation gap. The accompanying workshop report provides a summary of the January 2009 workshop “Make an Impact: Tackling the “I” and the “D” of Making It Happen”, which aimed to familiarise DFID staff with the use of qualitative methods in impact evaluation and measuring results.

The case for qualitative and combined methods is strong. Qualitative methods have an equal footing in evaluation of development impacts and can generate sophisticated, robust and timely data and analysis. Combining qualitative research with quantitative instruments that have greater breadth of coverage and generalisability can result in better evaluations that make the most of their respective comparative advantages.

Type of Work:Monograph (Discussion Paper)
School/Faculty:Colleges (2008 onwards) > College of Social Sciences
Number of Pages:51
Department:International Development Department
References:

Alsop R, Bertelsen, M. and J Holland. (2006). Empowerment in practice: From Analysis to implementation. The World Bank: Washington D.C.

Appleton, S. and D. Booth. (2001). ‘Combining participatory and survey-based approaches to poverty monitoring and analysis’. Background Paper for Workshop held in Entebbe, Uganda 30 May-1 June 2001. Overseas Development Institute: London.

Appleton, S. and D. Booth. (2005). ‘Strong fences make good neighbours: survey and participatory appraisal methods in poverty evaluation and poverty reduction strategy monitoring’ in Holland, J. with J. Campbell (eds) (2005). Methods in development research: Combining qualitative and quantitative approaches. ITDG Publications: London.

Barahona, C. and S. Levy. (2002). ’How to generate statistics and influence policy using participatory methods in research’. Statistical Services Centre Working Paper, University of Reading. Barahona, C. and S. Levy. (2005) ‘The best of both worlds: Producing national statistics using participatory methods’. Working Paper No. 11. Q-Squared. Toronto, Canada. Blattman, C. (2008) Impact Evaluation 2.0. Presentation to the Department of International Development, 14 February 2008.

Booth, D., Holland, J., Hentschel, J., Lanjouw, P. and A. Herbert. (1998) ‘Participation and combined methods in African poverty evaluation: Renewing the agenda’. Department for International Development, Issues, February, pp 127, ISBN 86192 027 X. http://www.dfid.gov.uk/pubs/files/sddafpov.pdf.

Brook, S., Thomson, A. and M. Strode. (2006) DFID New Economists Guide: Monitoring Chapter, Oxford Policy Management: Oxford.

Carraro, L., Weinzierl, S and S. Hunt. (2005). ‘A quantitative evaluation of social exclusion in Pakistan’. Oxford Policy Management: Oxford.

Carvalho, S. and H. White. (1997) ‘Combining the quantitative and qualitative approaches to poverty measurement and analysis’, Technical Paper 366. The World Bank: Washington D.C.

Center for Global Development. (2006) ‘When will we ever learn? Improving lives through impact evaluation.’ Report of the Evaluation Gap Working Group. Washington, D.C.

Chambers, R. (1994) ‘The origins and practice of participatory rural appraisal’ in: World Development 22, pp: 959-961.

Chambers, R. (1995) ‘Poverty and livelihoods: Whose reality counts?’, IDS Discussion Paper No. 347, IDS: Brighton.

Chambers, R. (1997) ‘Whose reality counts? Putting the first last’. ITDG: London.

Chambers, R. (2003) ‘The best of both worlds’, in Kanbur, R. (ed) Q-Squared: Qualitative and quantitative methods of poverty appraisal, Permanent Black: Delhi.

Chambers, R. (2007) ‘Who counts? The quiet revolution of participation and numbers.’ Working Paper 296. Institute of Development Studies: Brighton.

Chung, K. (1997) ‘Identifying good practice for the integration of qualitative and quantitative methods for poverty research’, Providence, RI, Brown University, draft.

de Haan, A. (1999) ‘Social exclusion: Towards an holistic understanding of deprivation’. Department for International Development: London.

Department for International Development (DFID). (2005) Poverty and social impact analysis: Principles for good practice. DFID, London.

Department for International Development (DFID). (2007) Results action plan. http://www.dfid.gov.uk/Pubs/files/results-action-plan08.pdf [13 October 2008].

Department for International Development (DFID). (2008) ‘(Draft) Policy statements on impact evaluation—September version.’ Internal Draft.

Department for International Development (DFID). (2008b) ‘(Draft) Gender and Social Policy Exclusion Analysis’. How to note. Internal Draft, October 2008.

Department of International Development (DFID). Project memorandum: DFID support to the international initiative on impact evaluation (3ie).

Earl, S., Carden, F. and T. Smutylo. (2001) Outcome mapping building learning and reflection into development programs. IRDC http://www.idrc.ca/en/ev-28377-201-1-DO_TOPIC.html [4 December 2008].

Gacitua-Mario, E. and Q. Woden. (2001) ‘Measurement and meaning: Combining quantitative and qualitative methods for the analysis of poverty and social exclusion in Latin America’. The World Bank: Washington D.C.

Grootaert, C. and van Bastelaer, T. (2001) Understanding and measuring social capital: A synthesis of findings and recommendations from the social capital initiative. Social Capital Initiative. Working Paper No. 24. The World Bank, Washington D.C.

Gunning, J. W. (2006) ‘Aid evaluation: Pursuing development as if evidence matters’ in Swedish Economic Policy Review 13, pp: 145-163.

Heller, P. (1996) Social capital as a product of class mobilization and state intervention: Industrial workers in Kerala, India, in World Development 24 (6), pp: 1055-1071.

Hentschel, J. (1999) ‘Contextuality and data collection methods: A Framework and application to health service utilisation’ in The Journal of Development Studies 35, pp: 64-94.

Holland, J. (2007) Tools for Institutional, Political and Social Analysis of Policy Reform (TIPA): A Sourcebook for Development Practitioners, The World Bank: Washington D.C.

Holland, J. with J. Campbell (eds). (2005) ‘Methods in development research: Combining qualitative and quantitative approaches. ITDG Publications: London.

Holland, J., Brook, S., Dudwick, N., Bertelsen, M. and G. Yaron. (2007) Monitoring empowerment in policy and programme interventions: Combining Qualitative and Quantitative Approaches’, Q-Squared Working Paper No. 45, University of Toronto.

Independent Evaluation Group. (nd) Impact evaluation—the experience of the Independent Evaluation Group of the World Bank. World Bank, Washington D.C. http://lnweb90.worldbank.org/oed/oeddoclib.nsf/DocUNIDViewForJavaSearch/35BC420995BF58F8852571E00068C6BD/$file/impact_evaluation.pdf [20 October 2008].

Jones, V. N. and M. Woolcock. (2007) ‘Using mixed methods to assess social capital in low income countries: A practical guide’. BWPI Working Paper 12. Brooks World Poverty Institute. Manchester.

Kabeer, N. (2000) ‘Social exclusion, poverty and discrimination: Towards an analytical framework’, IDS Bulletin, 31(4), Institute of Development Studies, Brighton.

Kanbur, R. (ed) (2003) Q-Squared: Qualitative and quantitative methods of poverty appraisal, Permanent Black: Delhi.

Knack, S. and P. Keefer. (1997) ‘Does social capital have an economic payoff? A cross-country investigation’. The Quarterly Journal of Economics, MIT Press, vol. 112(4), pp. 1251-1288.

Kuenzi, T. (2008) ‘Social capital and political trust in West Africa’. Afrobarometer Working Paper No. 96.

McGee, R. (2003) ‘Qualitative and quantitative poverty appraisal workshop: Some reflections and responses’, in Kanbur R (ed) Q-Squared: Qualitative and Quantitative Methods of Poverty Appraisal. Permanent Black: Delhi.

Moser, C. and C. McIlwaine. (2004) ‘Encounters with Violence in Latin America: Urban Poor Perceptions from Colombia and Guatemala.’ Routledge: New York and London.

Narayan, D. and L. Pritchett. (1997) Cents and sociability: Household income and social capital in rural Tanzania. Policy Research Working Paper 1796. The World Bank: Washington D.C.

Network of Networks for Impact Evaluation (NONIE): http://www.worldbank.org/ieg/nonie/index.html

No author (2007) Impact evaluation of new aid instruments and country programs. Discussion paper (second draft, December 12, 2007).

OECD (nd). Evaluation and aid effectiveness: Glossary of key terms in evaluation and results-based management, OECD DAC.

Prowse, M. (2007) Aid effectiveness: the role of qualitative research in impact evaluation. Background note. Overseas Development Institute. London.

Putnam, R. D. (1993) Making democracy work: Civic traditions in modern Italy. Princeton University Press: Princeton, New Jersey.

Q2: http://www.q-squared.ca/papers.html

Ravallion, M. (2008) Evaluation in the practice of development. Policy Research Working Paper 4547, The World Bank: Washington D.C.

Riemenschneider, N. and J. Holland (forthcoming). ‘Combining qualitative and quantitative results: do they tell the same story? Experience from using Community Scorecards and Citizen Report Cards in the Maldives’, OPM Briefing Note, Oxford Policy Management: Oxford.

Schnell, S., Poulsen, P., Condy, A., Tertsunen, M. and J. Holland. (2005) Principles for PSIA process in policy cycles and stakeholder participation: A document produced jointly by GTZ and DFID for sharing with the PSIA network. Department for International Development: London.

Sen, A. (2000) ‘Social exclusion: Concept, application, and scrutiny'. Asian Development Bank, Manila.

Spencer, L., Ritchie, J., Lewis, J. and L. Dillon (2004) ‘Quality in qualitative evaluation: A framework for assessing research evidence’, Cabinet Office, Government Chief Social Researcher’s Office, 2nd Revised Edition.

UNDP (1996) UNDP’s 1996 Report on Human Development in Bangladesh, Volume 3 Poor People’s Perspectives, UNDP: Dhaka.

UNDP (2000) UNDP strategy note on governance for human development. UNDP: New York.

UNDP (nd) Governance indicators: A Users’ Guide, Oslo: UNDP.

Venables, T. and S. Holloway (2008) Using number to improve decision taking. PS/PUSS Malik. DFID internal note.

White, H. (2002) ‘Combining quantitative and qualitative approaches in poverty analysis’, in World Development 30(3)

White, H. (2005) ‘Challenges in evaluating development effectiveness’. Working Paper 242, Institute of Development Studies: Brighton.

White, H. (2006). DFID Economists’ Manual: Impact Evaluation, Independent Evaluation Group, World Bank, Washington D.C.

White, H. (2008) Of probits and participation: The use of mixed methods in quantitative impact evaluation. Working Paper No. 6, Network of Networks on Impact Evaluation (NONIE). http://www.worldbank.org/ieg/nonie/docs/WP7_White.pdf [14 October 2008].

White, H. (nd). Making impact evaluation matter. International Initiative for Impact Evaluation. http://www.ideas-int.org/Documents/Howard_White.doc [accessed: 2 December 2008].

White, H., Sinha, S. and A. Flanagan. (nd) A review of the state of impact evaluation. Independent Evaluation Group, World Bank.

World Bank (2006a) Conducting quality impact evaluations under budget, time and data constraints, Independent Evaluation Group, The World Bank: Washington D.C.

World Bank (2006b) A decade of measuring the quality of governance. Governance matters 2006. Worldwide governance indicators. The World Bank: Washington D.C.

World Bank (2008) Good practice note: Using Poverty and Social Impact Analysis to support development policy operations. The World Bank, Washington D.C.

Yates, J., Cooper, R. and J. Holland (2006) ‘Social Protection and Health: Experiences in Uganda’, in Development Policy Review 24 (3) 339-356.

Projects:Governance and Social Development Resource Centre
Series/Collection Name:GSDRC Emerging Issues Papers
Keywords:monitoring, evaluation, accountability, impact evaluation, qualitative methods, quantitative methods
Subjects:H Social Sciences > HN Social history and conditions. Social problems. Social reform
H Social Sciences > HD Industries. Land use. Labor > HD28 Management. Industrial Management
H Social Sciences > HA Statistics
Related URLs:
URLURL Type
http://www.gsdrc.orgOrganisation
Funders:UK Department for International Development (DFID)
Copyright Status:Copyright 2009 University of Birmingham
Copyright Holders:University of Birmingham
ID Code:646

Export Reference As : ASCII + BibTeX + Dublin Core + EndNote + HTML + METS + MODS + OpenURL Object + Reference Manager + Refer + RefWorks
Share this item :
QR Code for this page

Repository Staff Only: item control page