Job no: 556695
Contract type: Consultant
Duty Station: New York
Level: Consultancy
Location: United States
Categories: Research, Planning, Monitoring and Evaluation
For every child, evaluate:
The UNICEF Evaluation Office, located in New York headquarters, provides global leadership and oversight of the evaluation function in the organization. The Evaluation Office is commissioning a formative evaluation of the Risk Communication and Community Engagement (RCCE) Collective Service to assess its preliminary outcomes and inform its strategic vision moving forward.
The RCCE Collective Service is a collaborative partnership between the International Federation of Red Cross and Red Crescent Societies (IFRC), the United Nations Children’s Fund (UNICEF), and the World Health Organization (WHO), with the support of the Global Outbreak Alert and Response Network (GOARN), and key stakeholders from the public health, humanitarian and development sectors. Under the endorsement of the Inter-Agency Standing Committee Principals, the Service was launched in 2020 with support from the Bill and Melinda Gates Foundation.
How can you make a difference?
The purpose of the Service is to ensure that the complementary strengths of all partners are supported and leveraged to deliver the greatest impact, and to bring together a wide range of organizations involved in RCCE policy, practice, and research to provide practical support to those delivering on the ground. Created in response to the COVID-19 pandemic, the overarching goal of the Collective Service is that people-centred and community-led approaches are championed widely to result in increased trust and social cohesion, and ultimately a reduction in the negative impacts of COVID-19.
After more than two and a half years in operation, with the ambition to expand the role of the Collective Service beyond the COVID-19 response to support other public health and humanitarian emergencies, a formative, outcome-oriented evaluation is foreseen to provide credible evidence on the extent to which the Collective Service is yielding results in terms of its key objectives.
Two complementary purposes animate this evaluation. The first is to confirm the theory of change of the Collective Service and its implementation (the practice of change), to assess the Collective Service’s contribution to the overarching goal of the RCCE systems-strengthening in the public health, humanitarian, and development responses to the COVID-19 pandemic. The second purpose is to assess the Collective Service’s readiness and inform the future strategy and vision by providing credible and reliable evidence and identifying concrete good practices and lessons learned.
The specific objectives of the evaluation are the following:
- Quality of the design and approach: Assess the Collective Service design (theory of change, logical framework) and strategy, the level of alignment to international practices in providing coordination and support services, and the comparative advantage of the Service positioning and role in the COVID-19 response.
- Achievement of preliminary outcomes: Determine the level of preliminary achievement of the intended overall outcomes (i.e., strengthened collaborative RCCE approaches to increase quality, harmonisation, optimisation, and integration of RCCE; availability of evidence to inform policy and programming and improve effectiveness and efficiency systematically; improved quality and consistency of risk communication and community engagement approaches; reinforced national capacity for improved local solutions).
- Wide coordination and collaboration: Assess the quality of coordination and cooperation at the global level primarily and then at the regional levels comparing regions where interagency teams have been put in place (West and Central Africa and Southern and Eastern Africa) versus areas where interagency teams have not been put in place yet; in selecting and managing partnerships to advance the overarching goal and objectives of the Service in RCCE, during the COVID-19 public health emergency.
- Management, governance, and resourcing arrangements: Examine the quality and efficiency of the Collective Service management and control in attaining the expected outcomes based on similar comparative experiences.
- Adequacy of Data for Action approach: Investigate the quality and overall coherence of the Data for Action approach used by the Collective Service and the use of evidence and knowledge management because of this approach to inform RCCE.
- Adequacy of internal M&E and knowledge management systems: Assess the quality and overall coherence of internal M&E and knowledge management systems utilised by the Collective Service.
The evaluation will be articulated along two main and interlinked areas of investigation, i.e., the analysis of the RCCE Collective Service organizational design and approach and the assessment of the achievement of preliminary outcomes. In investigating such components, the evaluation will look at the arrangements that have been put in place to achieve the overarching goal and objectives (coordination, collaboration, resourcing, etc.) and the extent to which the Service used data and evidence.
This evaluation is not intended to examine the impact of higher-level results since two and a half years of implementation is not considered sufficient. However, the assessment will try to identify and note preliminary, emerging outcomes and unintended positive and negative effects of the Collective Service.
To ground the analysis, deep-dive case studies will be examined to provide an in-depth analysis of specific areas/pilots/themes to inform the Collective Service’s work further. The exact number will be determined in the inception phase of the evaluation. Tentatively, these will include COVID-19 responses in West and Central Africa and East and Southern Africa, as well as other regions where interagency response teams were not formally established; information management and communication in Malawi; the Ebola response in Guinea; the COVID-19 database and dashboard.
The evaluation will focus on the 2020-2022 period and beyond to inform the way forward for the Collective Service. Any experience of coordinated information services before 2022 may be used as a comparator where relevant to examine the design and approach of the RCCE Collective Service.
The evaluation is expected to answer a set of questions to meet its purpose and objectives. The proposed evaluation questions are tentative and expected to be refined during the inception phase of the evaluation, based on initial exploratory findings and careful consideration of which questions appear to be most helpful.
The suggested set of questions are the following:
Q1. To what extent is the Collective Service design and approach to address community engagement coordination and support, both during COVID-19 and moving forward, clear, relevant, coherent, and appropriate, as well as equity, gender, and disability-inclusive?
- To what extent is there clarity and a coherent understanding of RCCE approaches for systems strengthening among Collective Service partners?
- To what extent was it realistic to achieve the goals and objectives of the Service given the context in which the operations have started (resource constraints, COVID-19, existing types of partnerships, etc.)?
- To what extent is the Collective Service design and approach relevant and appropriate to achieve the results stated in the theory of change and the logical framework considering the financial constraints faced by the Collective Service?
- What relative strengths have IFRC, UNICEF, WHO, and GOARN brought in incorporating the best approaches for RCCE systems strengthening in the Collective Service? To what extent have these strengths been utilised and leveraged to enhance RCCE systems?
- To what extent has the RCCE Collective Service design and approach paid attention to aspects of equity, gender inclusion and diversity? How have communities and stakeholders been involved in the planning and implementation of interventions?
- What lessons and actions could be taken to improve the relevance, clarity, coherence, appropriateness, and inclusiveness of the RCCE Collective Service’s approach and design in the transitioning phase from COVID-19?
Q2. To what extent and how have the preliminary outcomes of the Collective Service been achieved?
- What is the evidence, if any, that the Collective Service outcomes contribute to strengthening RCCE systems in response to COVID-19 and other crises?
- How has the RCCE Collective Service supported the achievement of outcomes? How have the Collective Service activities contributed to achieving the four outcomes and strengthening RCCE systems?
- How has the planning and implementation of the Collective Service maximised positive unintended outcomes or mitigated unintended negative consequences, if any?
- What is the likelihood that preliminary positive outcomes may be sustained in the short- and medium-term?
- How and to what extent has the Service contributed to strengthening national and inter-agency mechanisms?
Q3. To what extent has the Collective Service-wide coordination and collaboration been managed efficiently, effectively, and sustainably to achieve the overall goal and objectives of the Service?
- How well has the Service effectively coordinated and collaborated with partners to advance its objectives? Moreover, what efficiencies, if any, were realised through coordinated strategies and collaboration?
- How and to what extent has the existing management structure of the Collective Service impacted the implementation and the realisation of the goal and objectives of the Service? How can the current management structure be enhanced to sustain the completion of the goals and objectives of the Service?
- What RCCE capacities has the Service strengthened through its engagement with partners? What trade-offs have been made to ensure that partnership arrangements work as intended?
- How and to what extent have partners contributed to realising goals and preliminary outcomes?
- To what extent has the Service embedded international good practices and benchmarks in providing coordination and support services in response to crises?
Q4. To what extent are the Collective Service management and governance systems, as well as human and financial resources and commitments relevant, efficient, effective, sustainable, and equity-inclusive in attaining the expected outcomes?
- To what extent do the Collective Service management, governance and resourcing capacities match the Service’s overarching goal, objectives, and accountabilities? Have they been revised over time? What are the needs in terms of management, governance, and resourcing, including financial commitments, budgeting, and human resources, to achieve the outlined goals of the Collective Service?
- What adjustments should be made, if any, to the Service to scale up to go beyond COVID-19? What could be done to improve sustainability in the new Service strategy?
- How and to what extent have the Service’s stakeholders, namely coordinators, steering committee, and technical group, contributed to relevant, effective, efficient, sustainable, and equity-inclusive management and governance systems?
- What are lessons learned from the management and governance systems of the Service that could be used for further duplication and scaling up?
- What are the differences in regional management and governance systems leading to success stories and challenges in attaining the expected outcomes?
- What are the commitments needed by partners to ensure equitable contribution to the work of Collective Service? What are the best ways to implement and sustain these commitments?
Q5. To what extent and how has the Data for Action approach been effective and efficient in fostering accountability and learning?
- To what extent have previous studies, research, and other evaluations informed the Collective Service approach? To what extent have available resources and shared data contributed to the course taken?
- To what extent has the Data for Action approach contributed to realising goals and objectives? What is the value-add of the process?
- How can this approach be better utilised to foster accountability and learning, enhance implementation, and achieve the goals and objectives?
Q6. To what extent are internal data, M&E, and knowledge management systems in place to foster accountability and learning?
- To what extent do the existing internal Collective Service monitoring and data collection systems allow for and provide clear guidance for the collection of SMART data (Specific, Measurable, Achievable, Relevant and Time-bound) for monitoring indicators and reporting? How realistic are the targets set in the logical framework?
- To what extent have previous studies, research, and other evaluation informed the Collective Service’s work?
- How can M&E and knowledge management systems be further improved to foster accountability and learning, enhance the implementation, and achieve the goals and objectives?
The evaluation is expected to use a mixed methods approach, so to collect and analyse both quantitative and qualitative data, to gain convergence and increase validity through triangulation, using the strengths of each technique to overcome the weaknesses of the other and, in the end, obtain a fuller picture.
The following data gathering methods and approaches are suggested:
- Desk review, including documents on the Collective Service as well as information on good practices, definitions, and theoretical frameworks in international cooperation, public health, humanitarian and development assistance, risk communication and community engagement.
- Online survey(s) to obtain data from a range of stakeholders involved in the Collective Service, such as management staff, steering committee, technical group, and broader implementing partners.
- Key informant interviews and focus group discussions with a selected sample of key informants, including internal stakeholders of all organizations involved and key external partners contributing to the Collective Service
- Field missions (to be confirmed) to gather targeted comprehensive data for deep-dive case studies
- Readiness analysis, grounded in techniques measuring readiness, and assessing the status of the Collective Service and its preparedness for scaling up to other crises beyond COVID-19
- Sentiment analysis / natural language processing (NLP), to detect positive or negative sentiments in text data, or other innovative machine-learning-based methods.
Furthermore, evaluation consultants applying for this assignment are encouraged to suggest specific methodological approaches that they consider suitable and helpful, such as contribution analysis and/or process tracing, outcome mapping, outcome harvesting, or realist evaluation. Likewise, applicants may propose further strategies, methods, or ideas to achieve the evaluation’s purpose and objectives.
The evaluation process is expected to be in line with UNICEF standards on evaluation ethics and quality, UNEG Ethical Guidelines, as well as the UNEG Guidance on Human Rights and Gender Equality in Evaluation. The selected consultant must sign a non-disclosure agreement, abide by UNICEF’s security protocols, and ensure that sensitive data is protected.
The UNICEF Evaluation Office is commissioning the evaluation and will establish an evaluation team. An Evaluation Specialist in the UNICEF Evaluation Office (Institutional Effectiveness portfolio) will act as the Evaluation Manager and supervise the exercise.
The team conducting this evaluation is expected to consist of up to three team members, one team leader (Senior Evaluation Expert) and one or two team member/s (Evaluation Expert). This vacancy is for the team leader of this exercise, a separate vacancy for team members is advertised separately. Coordinated applications as a team are welcomed and encouraged.
The role and responsibilities of the Senior Evaluation Expert will include the following:
- Lead and direct all parts of the evaluation, including work planning.
- Coordinate and supervise the evaluation team’s work in their contributing roles.
- Act as the contact person for the Evaluation Manager and work closely with the Evaluation Manager throughout the process, ensuring regular updates and consultation.
- Manage and ensure the quality of the evaluation process and methodological approach.
- Ensure the comprehensiveness and quality of all data collected and analyzed, and of key deliverables before submission to the UNICEF Evaluation Office, by reviewing and harmonizing their format and content in line with the requirements specified in the TOR
- Take direct responsibility for all deliverables being delivered timely and of satisfactory quality.
- Participate, design, and facilitate the validation workshop in coordination with the Evaluation Manager.
A Management Team will support the exercise, consisting of the Evaluation Manager and representatives from the organizations overseeing the Collective Service. Furthermore, a Reference Group will support the evaluation team and the management team in an advisory capacity. The Reference Group will consist of members of the Steering Committee and external experts in RCCE, public health and evaluation.
A timeline of around seven months is envisaged for the evaluation, from December 2022 to June 2023. The proposed organization of the evaluation phases is as follows:
Inception phase (six weeks): During this phase, the evaluation team is expected to gain a deep understanding of the proposed documentation, assess possible information gaps, refining the scope, methods, and critical stakeholders. The main deliverable for this phase will be the inception report, presenting a detailed description of the final scope; revised methodological approach, including any data collection and analytical instruments (also piloted during this phase); preliminary evidence from the initial desk review and critical informant consultations; as well as the structure of the final report and an updated timeline for deliverables.
Data collection and analysis phase (twelve weeks): Additional primary and secondary data is to be collected, using instruments developed and piloted during the inception phase. All data gathered shall be duly analysed, stored in a secure repository, cleaned, and processed to ensure the anonymity of key informants. To conclude this phase, a presentation with preliminary findings and conclusions is foreseen.
Report drafting and dissemination phase (seven weeks): The main deliverables for this phase are the preliminary draft of the final report and the final agreed report. In addition, it is expected that the main findings, conclusions, recommendations, and lessons learned will be presented to the Reference Group and other relevant stakeholders, with a stand-alone Evaluation Brief and PowerPoint to be delivered.
In line with the process described above, the expected key deliverables for this evaluation will be:
- a draft inception report; approx. by mid-January 2023
- a final inception report; approx. by early February 2023
- a presentation with preliminary findings and conclusions; approx. by mid-April 2023
- a draft evaluation report; approx. by mid-May 2023
- a final evaluation report; approx. by mid-June 2023
- a final presentation and an evaluation brief; approx. by end-June 2023.
Draft deliverables are first to be shared with the Evaluation Office for quality assurance and once approved, will be more widely disseminated to key stakeholders for comments. Comments received are expected to be transparently addressed by the evaluation team when providing the revised version of the reports (for example, by providing an additional track-change performance or an audit trail).
To qualify as an advocate for every child you will have…
- At least ten years of professional experience in leading evaluations, research, or other formative and summative exercises in international cooperation, public health, humanitarian and development assistance, risk communication and community engagement, and organisational/institutional effectiveness issues.
- Minimum an advanced degree [Master’s Level] in a relevant field across the social sciences, with an advantage given to degrees or major emphases in evaluation or related evidence fields.
- Extensive experience working with multilateral or intergovernmental organisations.
- A good understanding of the public health and humanitarian emergency response mechanisms and inter-agency coordination infrastructure.
- Proven experience in leading exercises similar in scope to the present task.
- Demonstrated ability to supervise an evaluation team at the talent and experience level required.
- Excellent written and oral communication skills in English, and ideally in at least one other UN langu
Please note that an estimated level of 90 working days is foreseen for this role, spread over the entire timeline of the evaluation process, starting in December 2022.
How to Apply:
For your application to this role, please include an updated CV and a cover letter.
Ideally, your cover letter will include the following:
- An outline of why you are interested in this evaluation
- Specific expertise and experience that qualifies you for this assignment
- References to evaluation reports you worked on (through links provided or attached documents)
- An indication of your daily fee
If you intend to apply as a team, please also indicate this in your cover letter and ensure that your suggested fellow team members also submit an application for their suggested role.
For every Child, you demonstrate…
UNICEF’s values of Care, Respect, Integrity, Trust, and Accountability and core competencies in Communication, Working with People and Drive for Results.
UNICEF has a zero-tolerance policy on conduct that is incompatible with the aims and objectives of the United Nations and UNICEF, including sexual exploitation and abuse, sexual harassment, abuse of authority and discrimination. UNICEF also adheres to strict child safeguarding principles. All selected candidates will be expected to adhere to these standards and principles and will therefore undergo rigorous reference and background checks. Background checks will include the verification of academic credential(s) and employment history. Selected candidates may be required to provide additional information to conduct a background check.
Remarks:
Only shortlisted candidates will be contacted and advance to the next stage of the selection process.
Advertised: 31 Oct 2022 Eastern Daylight Time
Deadline: 14 Nov 2022 Eastern Standard Time
