ECW framework (ENG)

Click to explore!

Strengthening Holistic Learning Outcome Measurement Systems

Project Overview

Why focus on education assessment, data, and evidence systems? Watch to learn more!

The Pilot Results

The HOLAS Framework

With the support of Education Cannot Wait, NYU Global TIES for Children and the Universidad de los Andes have collaborated to develop and pilot a suite of resources that supports stakeholders' ability to understand, map, and assess holistic learning measurement systems with a focus on equity, inclusion, and well-being. Click to your right to explore our project overview, the Holistic Learning Assessment Systems (HOLAS) framework, and our results and recommendations piloted in Colombia and Peru!

nteract with the framework!

ow did you develop the HOLAS framework?

hy develop the HOLAS framework?

hat is the purpose of the HOLAS framework?

FRAME WORK

PREVIOUS PAGE

HOME

We designed the framework to provide detailed information that enables the development and prioritization of concrete strategies for strengthening alignment in holistic M&E systems.

We piloted a methodology for education systems analysis which facilitates replication and transparency through the use of open-source quantitative and qualitative tools.

We sought to clearly identify and describe elements of holistic learning monitoring and evaluation (M&E) systems using user-friendly language and incorporating multiple stakeholder voices.

FRAMEWORK FAQS

Click each number to read more!

To prioritize strategies for strengthening alignment within systems to support equitable holistic learning.

To build consensus on areas of (mis)alignment within holistic learning M&E systems.

To facilitate clear and inclusive communication about the elements of holistic learning M&E systems.

In order to strengthen coherence and equity in the Colombian and Peruvian holistic learning outcome measurement systems, we needed a framework that would allow us to bound, map, and communicate about such systems. The HOLAS framework was developed for three purposes:

PREVIOUS PAGE

HOME

NEXT PAGE

hat is the purpose of the HOLAS framework?

SUPPORT

Click to learn more about the elements and dimensions of the framework!

Each systemic element consists of four dimensions.

Three main elements - information, goals, and support - define holistic learning outcome measurement systems.

Ensure access to high-quality professional development opportunities around data, evidence, and measurement

Provide high- quality holistic learning opportunities to support equity and well-being among authorities and front-line providers

Establish strong organizational structures for effective monitoring and evaluation

Provide high-quality assessment guidance materials

Promote agency among frontline providers and authoriities in monitoring and evaluation

Take evidence-based decisions

Align assessments with key elements of education systems

Establish clear goals for holistic learning

Enable access to information

Ensure quality of information

Use information for supporting equitable holistic learning

Diversify, understand, and align the purposes of assessment

GOALS

INFORMATION

PREVIOUS PAGE

HOME

SUPPORT

Ensure access to high-quality professional development opportunities around data, evidence, and measurement

Provide high- quality holistic learning opportunities to support equity and well-being among authorities and front-line providers

Establish strong organizational structures for effective monitoring and evaluation

Provide high-quality assessment guidance materials

Promote agency among frontline providers and authoriities in monitoring and evaluation

Take evidence-based decisions

Align assessments with key elements of education systems

Establish clear goals for holistic learning

Enable access to information

Ensure quality of information

Use information for supporting equitable holistic learning

Diversify, understand, and align the purposes of assessment

GOALS

INFORMATION

PREVIOUS PAGE

HOME

BACK TO FRAME WORK

The extent to which assessments of similar skills or outcomes that were designed for different purposes provide information that is aligned or coherent

Alignment between assessments for different purposes

This dimension relates to the variety of and alignment between assessments in the educational system and the extent to which stakeholders understand their "fit for purpose." We specifically consider three criteria within this dimension.

Variety of assessment purposes and types

Click each flag to read pilot findings from each country on this dimension.

Colombia

Peru

Click on underlined text to see definitions of terms.

The extent to which different stakeholders can clearly and accurately identify the purposes for which educational assessments were originally designed.

Stakeholders' understanding of assessment purposes

The extent to which a variety of types of assessments are available to assess different holistic outcomes and processes.

Diversify, understand, and align the purposes of assessments

INFORMATION

NEXT SUB- ELEMENT

NAVIGATE THIS PAGE

HOME

This dimension relates to the extent to which information from a variety of types of assessments - as well as from monitoring and evaluation systems - is used responsibly by stakeholders to make holistic learning-oriented decisions. We specifically consider three criteria within this dimension.

The extent to which information is used in a way that is fair and equitable. This includes the extent to which information does not explicitly or implicitly sitgmatize childhoods in contexts of marginalization, and actively supports equity in the allocation of resources and opportunities.

The extent to which information informs decision-making by authorities, and is also shared back with and used by schools, teachers, or community stakeholders.

The extent to which information is used (or not) for decision making that support holistic learning outcomes.

Type of decisions

BACK TO FRAME WORK

Click each flag to read pilot findings from each country on this dimension.

Colombia

Peru

Click on underlined text to see definitions of terms.

Fair use

Eco-system information flows for decision-making

Use information for supporting equitable holistic learning

INFORMATION

NEXT SUB- ELEMENT

NAVIGATE THIS PAGE

HOME

The extent to which there are mechanisms and practices in place to ensure inclusion and meaningful representation of childhoods in contexts of marginalization at all stages of the assessment process, from the design to data collection to analysis and dissemination.

The extent to which reports of the psychometric properties/quality of assessments are publicly available.

The extent to which practices and mechanisms are in place and utilized to ensure data from assessments or tools meets quality standards commensurate with the intended use of the information.

The extent to which there is sufficient evidence that information provided by the assessments or tools is valid, reliable, and/or fair. Sufficiency is determined based on the purpose of the assessment.

This dimension relates to the quality of education measures, assessments, evaluation tools, monitoring and evaluation systems and/or the data obtained from them. We specifically consider four criteria within this dimension.

Fairness

Sufficiency of evidence of information quality

BACK TO FRAME WORK

Click each flag to read pilot findings from each country on this dimension.

Colombia

Peru

Click on underlined text to see definitions of terms.

Availability of evidence of information quality

Mechanisms and practices to ethnically ensure information quality

Ensure quality of information

INFORMATION

NEXT SUB- ELEMENT

NAVIGATE THIS PAGE

HOME

Principled availability of/access to guidance documents, data, or results

The extent to which M&E objects from various stages of the monitoring and evaluation process can be obtained. This includes whether there are clear pathways for accessing or retrieving objects and whether objects are readily provided upon request.

The extent to which there are mechanisms, processes, and expectations in place for stakeholders to share and disseminate M&E objects to diverse stakeholders.

This dimension relates to the degree (or not) of physical or digital access to various objects used in or resulting from the monitoring and evaluation process, including results, data, measures, materials, and analytic code (hereafter, monitoring and evaluation or M&E objects). It also relates to stakeholders' practices of sharing such M&E objects. We specifically consider two criteria within this dimension.

BACK TO FRAME WORK

Click each flag to read pilot findings from each country on this dimension.

Colombia

Peru

Click on underlined text to see definitions of terms.

Sharing of evaluations, data, results and materials

Enable access to information based on open-science principles

INFORMATION

NEXT SUB- ELEMENT

NAVIGATE THIS PAGE

HOME

BACK TO FRAME WORK

Click each flag to read pilot findings from each country on this dimension.

Colombia

Peru

Click on underlined text to see definitions of terms.

GOALS

The extent to which there are official documents that outline clear and explicit objectives to strengthen, promote, and enhance the holistic learning outcomes of groups in contexts of marginalization.

The extent to which academic and social and emotional learning objectives are outlined in official documents in a coherent, consistent and explicit manner.

This dimension relates to whether clear learning objectives for both academic and social and emotional learning are outlined in official documents for all children within the education system. We specifically consider two criteria within this dimension.

Clarity on the objectives of holistic learning

Clarity of objectives to support holistic learning for refugees and groups in contexts of marginalization

Establish clear goals for holistic learning

NAVIGATE THIS PAGE

NEXT SUB- ELEMENT

HOME

INFORMATION

BACK TO FRAME WORK

Click each flag to read pilot findings from each country on this dimension.

Colombia

Peru

Click on underlined text to see definitions of terms.

GOALS

The extent to which there are clear and specified linkages between an educational assessment or tool and corresponding national or sub-national standards or regulations.

This dimension relates to the extent to which education measures, assessments, evaluation tools, and/or monitoring and evaluation systems are aligned with curricula, standards, and professional development trainings within the education system. We specifically consider three criteria within this dimension.

Alignment of educational assessments with corresponding standards or regulations

The extent to which the design and results of educational assessments and tools are aligned with other elements of the education system, such as non-formal education.

Alignment of asessments with other elements of the system

NAVIGATE THIS PAGE

The extent to which the design and results of educational assessments and tools are aligned with professional development opportunities for frontline service providers.

Alignment of educational assessments with frontline provider training opportunities

Align assessments with key elements of education systems

NEXT SUB- ELEMENT

HOME

INFORMATION

Evidence of factors that enhance student holistic learning

BACK TO FRAME WORK

Click each flag to read pilot findings from each country on this dimension.

Colombia

Peru

Click on underlined text to see definitions of terms.

GOALS

The extent to which educational authorities make decisions based on evidence of drivers of holistic learning outcomes, including access, quality, and continuity of education.

This dimension relates to the extent to which decisions are made based on a cumulative body of knowledge about the drivers of holistic learning outcomes. We specifically consider one criteria within this dimension.

NAVIGATE THIS PAGE

Take evidence-based decisions

NEXT SUB- ELEMENT

INFORMATION

HOME

BACK TO FRAME WORK

Click each flag to read pilot findings from each country on this dimension.

Colombia

Peru

Click on underlined text to see definitions of terms.

GOALS

Stakeholder support for school staff design and use of assessments

The extent to which diverse stakeholders perceive that frontline providers are capable of designing and implementing their own assessments or tools, as well as using the resulting data.

The extent to which diverse stakeholders perceive that sub-national educational authorities are capable of designing and implementing their own educational assessments or tools, as well as using the resulting data

This dimension refers to the extent to which frontline providers such as teachers and principals and sub-national educational authorities are perceived as capable of undertaking monitoring and evaluation activities. We specifically consider two criteria within this dimension.

Stakeholder support for sub-national staff design and use of assessments

NAVIGATE THIS PAGE

Promote agency among front-line providers and authorities in monitoring and evaluation

NEXT SUB- ELEMENT

HOME

INFORMATION

BACK TO FRAME WORK

SUPPORT

Click each flag to read pilot findings from each country on this dimension.

Colombia

Peru

Click on underlined text to see definitions of terms.

Materials for monitoring and evaluation assessment design, understanding, and data utilization

The quality of guiding materials that foster knowledge and skills related to the design, comprehension, and the effective utilization of education monitoring and evaluation data. Such resources may include manuals, frameworks, item banks, infographcs, and more.

The extent to which materials to support classroom assessment design and use - such as dashboards, rubrics, item banks, and workbooks with assessment examples - are perceived as practical and high quality.

This dimension relates to the existence and quality of guidance materials that allow sub-national authorities and front-line service providers to design, administer, and effectively use assessments. We specifically consider two criteria within this dimension.

Materials for classroom assessments design and use

Provide high-quality assessment guidance materials

NAVIGATE THIS PAGE

NEXT SUB- ELEMENT

HOME

INFORMATION

This dimension relates to the extent to which there are strong national and sub-national organizational structures to support monitoring and evaluation (M&E) efforts. We specifically consider five criteria within this dimension.

The extent to which M&E organizational structures maintain (or not) their long-term vision and operational capacity despite the changes that occur within educational systems, particularly during times of political and economic crisis.

The extent to which national and sub-national M&E offices are capable of establishing external partnerships that facilitate and enhance M&E processes at all stages.

The extent to which adequate funding is secured on an annual basis for national and sub-national monitoring and evaluation offices.

The extent to which there are qualified, permanent staff who are part of the offices leading the monitoring and evaluation processes.

The extent to which national or sub-national M&E offices exist and have clear and established mandates to lead monitoring and evaluation processes; and the degree to which these offices coordinate with each other when required.

Continuity of M&E efforts in challenging political contexts

Research, practice, policy, and partnerships

Existence and leadership

BACK TO FRAME WORK

SUPPORT

Click each flag to read pilot findings from each country on this dimension.

Colombia

Peru

Click on underlined text to see definitions of terms.

Funding

Dedicated and trained staff

Establish strong organizational structures for effective monitoring and evaluation

INFORMATION

NEXT SUB- ELEMENT

NAVIGATE THIS PAGE

HOME

Provide high-quality holistic learning opportunities to support equity and well-being among educational authorities and frontline providers

This dimension relates to the extent to which there are system-level mechanisms in place to strengthen educational authorities and front-line service providers' capacities for equitable and inclusive practices. We specifically consider three criteria within this dimension.

The extent to which front-line providers have access to regular peer and professional supports to implement specific strategies to promote holistic well-being, equity, and inclusion.

The extent to which educational authorities and front-line service providers have access to pre-service or in-service initiatives to strengthen capacities for addressing the needs and ensuring equitable and inclusive participation of childhoods in contexts of marginalization in curriculum and assessment.

The extent to which there are evidence-informed pre-service and in-service professional development opportunities that acknowledge and support the well-being and social and emotional skills of front-line service providers.

Opportunities to foster the social and emotional skills of frontline service providers

BACK TO FRAME WORK

SUPPORT

Click each flag to read pilot findings from each country on this dimension.

Colombia

Peru

Click on underlined text to see definitions of terms.

Availability of peer support for front-line providers to improve equity and inclusion

Training opportunities for education authorities and frontline providers to improve equity and inclusion

INFORMATION

NEXT SUB- ELEMENT

NAVIGATE THIS PAGE

HOME

This dimension relates to the extent to which there are system-level mechanisms in place to strengthen educational authorities' and front-line service providers' capacities for generating and using assessments, data and evidence. We specifically consider three criteria within this dimension.

Ensure access to high-quality professional development opportunities around data, evidence, and measurement

The extent to which a strong peer network is available at the school or program level to provide regular feedback and improve front-line providers' use of assessments, data, and evidence.

The extent to which professional development opportunities for educational authorities and front-line service providers around assessment, data, and evidence allow for hands-on experience, knowledge application, and certification as opposed to "chalk and talk" methods.

The extent to which professional development opportunities for educational authorities and front-line service providers includes content related to assessment, data, and evidence.

Content of professional development opportunities around assessments, data, and evidence

BACK TO FRAME WORK

SUPPORT

Click each flag to read pilot findings from each country on this dimension.

Colombia

Peru

Click on underlined text to see definitions of terms.

Availability of peer support and supervision in data, testing and evaluation for front-line providers

Quality of professional development opportunities around data, evidence and assesments

INFORMATION

NEXT SUB- ELEMENT

NAVIGATE THIS PAGE

HOME

We piloted the HOLAS Framework in the current initiative in Colombia and Peru. In doing so, we aimed to provide a proof of concept of the utility of such an approach for strengthening holistic monitoring and evaluation (M&E) systems in contexts of emergency and protracted crisis, beginning in two countries with strong governmental educational M&E systems. You can review our diagnostic reports, which summarize the methodology, the results, and the recommendations of the systems analysis below.

PREVIOUS PAGE

HOME

he Pilot Results

Neither RISE nor SABER explicitly attend to power dynamics and equity among childhoods in contexts of marginalization. Systems frameworks can provide a powerful tool for interrogating power and relational dynamics.

Both RISE and SABER focus only on the assessment of academic learning outcomes. However, education systems globally are being called upon to foster holistic learning - both academic knowledge and social and emotional skills and well-being - that can support children to navigate uncertain futures.

Neither RISE nor SABER specifically focus on education systems within contexts of emergency and protracted crisis. Yet, as climate change, conflict, and human rights violations have displaced over 100 million people there is a need for greater coordination and collaboration between actors working towards national and sub-national education development and actors supporting education as part of humanitarian response to crisis.

The need for a systemic focus on equity.

FRAMEWORK FAQS

PREVIOUS PAGE

HOME

The need for a systemic focus on holistic learning outcomes.

NEXT PAGE

The need for a systemic focus on the development–humanitarian nexus.

As a starting point, we identified two existing approaches to education systems analysis: The RISE diagnostic framework and the SABER tool. However, we recognized that each framework had complementary strengths and shortcomings for our purposes, leading to our decision to adapt, integrate, and extend them to create HOLAS. Specifically, we noted three, interrelated limitations that required the extension of the RISE and SABER frameworks:

hy develop the HOLAS framework?

After collecting a first tranche of data, we held a workshop with our Steering Committees to review preliminary results and provide evidence on the validity of the HOLAS framework 2.0. Based on the results of these validation activities, we revised the HOLAS framework with a focus on (1) Improving the naming conventions of the sub-elements within the HOLAS framework; (2) Clearly identifying the actors and levels of the education system analyzed within each sub-element; and (3) Strengthening the focus within sub-elements on inclusion.

Given our team’s expertise, assets, and relationships, we made the decision to center on the relationship between educational authorities and organizations and frontline service providers, and on the information, support, and goals elements that define interactions within this relationship.

In doing so, we added new sub-elements or dimensions to the RISE framework to highlight key M&E processes. We also revised the RISE elements to more systematically consider the role of different types of assessments. This resulted in version 1.0 of the HOLAS framework. We then revised version 1.0 of the HOLAS framework with an eye towards three criteria: extension, usability, and replicability.

We conducted a mixed-methods study with a wide range of stakeholders in Colombia and Peru to triangulate information about the elements and sub-elements within the HOLAS framework.

FRAMEWORK FAQS

PREVIOUS PAGE

HOME

We mapped indicators from the SABER Student Assessment national large-scale and classroom assessment instruments to the different elements and sub-elements within RISE.

NEXT PAGE

First, we reviewed the RISE and SABER frameworks and recognized that the resources and timeline available for this project did not permit us to fully adapt and extend the frameworks.

ow did you develop the HOLAS framework?

The goals element focuses on the definition and clarity of holistic learning objectives within the system, the alignment of information from education monitoring and evaluation systems with these objectives and other crucial education system components, and the established norms governing the use of this information in decision-making, including the level of autonomy stakeholders have in doing so. We specifically consider four dimensions within this element.

Goals

GOALS

The information element focuses on how and with what quality the information produced by education monitoring and evaluation systems is generated, accessed, used, and shared by education authorities, organizations, frontline providers, and researchers for a variety of purposes. We specifically consider four dimensions within this element.

Information

The Project

The HOLAS framework was developed by NYU Global TIES for Children (NYU-TIES) and Universidad de los Andes (Uniandes) researchers in the scope of the "Strengthening Holistic Learning Measurement Systems in Education in Emergencies and Protracted Crises in Colombia and Peru" project, sponsored by Education Cannot Wait (ECW). Visit our website to learn more.

Support

The support element includes the mechanisms that are in place to ensure comprehensive, evidence-informed holistic learning at different levels of the education system, including the availability and quality of resources, professional development opportunities, and organizational structures. We specifically consider four dimensions within this element.

SUPPORT