Evaluation and Performance Measurement

Capra Witness to Canadian Senate Committee on Banking Trade and Commerce ‎‎‎‎[1]‎‎‎

One of Capra International Inc.’s areas of concentration is policy and program evaluation in the public and private sectors of Canada, as well as with international organizations. In that context, Capra has extensive experience with both formative and summative evaluations.

Formative evaluations are also known as implementation evaluations, since they assess, usually at mid-term, the extent to which policies and programs, or initiatives, are being implemented as planned, and which adjustments, if any, may be required. 

Summative evaluations are also known as impact evaluations, since they focus on the extent to which expected results have been achieved during a given cycle of operation.

A special variant of both evaluation types is the horizontal evaluation. This type may be very complex, since it involves as client more than one organization. Significant difficulties may arise when trying to achieve a unified understanding among the partners regarding the substance and approach to the evaluation, as well as data sharing (see presentation notes).

All types of evaluation usually involve both secondary and primary research.  Secondary research usually comes first and consists of a review of existing documents, files and databases. Primary research follows secondary research. It consists of developing new knowledge by means of several lines of evidence (or data sources), such as key informant interviews, focus groups, case studies, surveys and site visits.

The planning for an evaluation normally consists of the development of a detailed Evaluation Framework, comprising a profile of the policy, program or initiative to be evaluated (including a logic model), a description of the objectives and context of the evaluation, the issues, questions and indicators to be addressed, data analysis and reporting the stakeholders involved, the deliverables expected, the corresponding timeline, the human resources involved to execute the evaluation, as well as the overall budget.

To achieve a high degree of quality, various organizations have developed quality control standards, and or norms, for the execution of evaluations. Capra’s standards can be accesses by following this link.

Some evaluation assignments address only certain parts, such as case studies, surveys, etc. Others focus on preparatory activities, such as performance measurement. The latter concerns the definition of performance indicators and the corresponding data collection. This activity is typically performed by management on an ongoing basis. When this task is performed by a third party on behalf of a client, it is often referred to as “monitoring”.

In Canada, evaluations within the federal government must be executed in accordance with the Treasury Board of Canada evaluation policy.

Several Capra evaluators hold membership in the Canadian Evaluation Society (CES) and in the International Development Evaluation Association (IDEAS). Capra received international recognition by the Independent Evaluation Group (IEG) of The World Bank, which recognized Capra International Inc. as an evaluation provider in its newSourcebook of Indicative Principles and Standards for Evaluating Global and Regional Partnership Programs (2007).

Capra routinely offers evaluation and performance measurement services in a multi-language environment. Besides English and French, Capra has offered services in: Spanish, Mandarin, Cantonese, Vietnamese, Tagalog, Dari, Hindi, Punjabi, Gujarati, Somali, Arabic, as well as the Canadian indigenous languages of Ojibwe and Inuktitut. Other languages are also available upon request.

Partial List of Current and Recent Evaluation Projects:
  • DFATD. Formative Mid-Term Evaluation of the Support to Rice Research in Africa Project (Africa Rice). (September 2014 – March 2015). 
  • UNICEF. Evaluation of UNICEF-supported MOET’s initiative of mother-tongue based bilingual education in Viet Nam. (September 2014 – August 2015). 
  • WTO-EIF Board. Comprehensive Evaluation of the Enhanced Integrated Framework [for Trade-related Technical Assistance to the Least Developed Counties]. 50 countries . Published Report: http://enhancedif.org/en/system/files/uploads/evaluation_of_the_enhanced_integrated_framework_final_report_english_20141128.pdf’; http://enhancedif.org/en/system/files/uploads/evaluation_of_the_enhanced_integrated_framework_20141115_final_french_20141128.pdf. (April – November 2014). 
  • ILO. Independent Mid-Term Evaluation of the Programme Support to the Partnership Programme to Prevent and Eliminate Child Labour. (2013-2014). 
  • UNDP. Global and Thematic Evaluation of the MDG Achievement Fund. (March 2013-September 2014). 50 countries. Published report: http://www.mdgfund.org/sites/default/files/UNDP_MDG-F_Evaluation_Final_Report_20140929.pdf
  • CIDA. Mid-Term Formative Evaluation of the Partnership for CARICOM Private Sector Development Project (International Finance Corporation). (2013-2014).
  • CIDA/DFID. Evaluation of the Ministry of Education Bilingual Education Pilot Initiative in Mozambique. (2012-2013). 
  • Norwegian Refugee Council. Costs of Displacement – General Research Overview. Occupied Palestinian Territories (Area C and East Jerusalem). (2012-2013).
  • CIDA. Thematic Evaluation on the Cooperative Model. (2012).
  • CIDA. Colombia. End of Project Summative Evaluation of the Right to Education and Participation for Children and Youth in Nariño (Save-the-Children and Norwegian Refugee Council). (2012).
  • CIDA. Evaluation of the International Youth Internship Program, providing quality assurance (2012). 
  • Justice Canada. Performance measurement strategy for the crimes against humanity and war crimes program (December 2011-March 2012).
  • UNICEF, Nigeria. Impact Evaluation, Conditional Cash Transfer Programme in Bauchi State and Katsina State (2011-2012).
  • Foreign Affairs and International Trade (DFAIT). 2011 Review of the Canadian Commercial Corporation (2012).
  • Canadian International Development Agency. Summative Evaluation of the CIDA – Canadian Catholic Organization for Development and Peace 2006-2011 Program (2010).
  • Finance Canada. 10-yr Evaluation of Canada’s Anti-Money Laundering and Anti-Terrorist Financing Regime (awarded in February 2010).
  • Foreign Affairs and International Trade and Canadian International Development Agency. Canada Investment Fund for Africa Mid-term Review (awarded in October 2009).
  • Human Resources and Skills Development. Summative Evaluation of the International Trade and Labour Program (awarded in July 2009).
  • Industry Canada, Office of Consumer Affairs. Evaluation of the Contribution Program for the Non-Profit Consumer and Voluntary Organizations (awarded in April 2009).
  • Human Resources and Skills Development. Employer Survey for the Sector Council Program (2009).
  • Human Resources and Skills Development. Development of an Evaluation Framework for the Federal Elder Abuse Initiative (awarded in Februry2009).
  • SNV Netherlands Development Organisation. Evaluation of the Honduras Country Program (2008).
  • Suriname Ministry of Trade and Industry (MTI). Evaluation of the Trade Sector Support Program (TSSP) (awarded in October 2008, continuing).
  • Health Canada. Summative Evaluation of the Contribution Program to Improve Access to Health Services for Official Language Minority Communities (awarded in January 2008).
  • Public Safety Canada. Summative Evaluation of the National Security Capacity Initiative (awarded in January 2008).
  • Department of Canadian Heritage. Evaluation of the Trade Routes Program (2007-2008)
  • Service Canada. Evaluation of Multi-language Service Initiative (included client survey in two Canadian Aboriginal and 10 international languages) (2007-2008).
  • Canadian Heritage. Formative Evaluation of the Law Enforcement Aboriginal and Diversity Network (2007).
  • Department of Natural Resources (Canada). Evaluation of the North American Forest Commission (Canada, Mexico, USA) (2006).
  • Department of Natural Resources (Canada). Development of a Geopolitical Analysis Framework for International Relations in Natural Resources and its application in China, India and Brazil (2004).
  • World Trade Organization, The World Bank, International Monetary Fund, International Trade Centre, UNCTAD, UNDP. Evaluation of the Integrated Framework for Trade-Related Technical Assistance to Least Developed Countries (Haiti, Mauritania, Senegal, Guinea, The Gambia, Mali, Uganda, Malawi, Bangladesh, Nepal, Cambodia, Yemen, Tanzania, Lesotho, Burundi, Djibouti, Ethiopia, Eritrea, Madagascar) (2003).