Copy - AEA NAF PAR Demonstration Session

A Mission-Aligned Process to Evaluate and Design to more EquitableCollege and Career Readiness Programs

Youth Participatory Action Research (YPAR)

DEE CHAMBLISS & Julie Poncelet

November 10, 2022

Demonstration SessionAmerican Evaluation AssociationEvaluation 2022

What Brings You to this Session?

• Maybe we do some quick polling to learn:
• Who is in the room?
• What brings them to our session?
• Experiences they have with ensuring college access and future-readiness programs engage in equitable and mission-aligned evaluation practices

Click and jointhe poll

NAF's Developing Approach to Future Ready Competencies

ASPIRATIONS

SKILL SETS

CONNECTIONS

A S P I R A T I O N S Students will participate in activities that help them explore careers of interest, set goals for themselves and make action plans to accomplish milestones along the way. As they navigate their unique journeys to discover what they do or do not want to pursue along their skills-driven path, they will continue to learn, connect, and reflect on their growth towards college and career readiness.

C O N N E C T I O N S Throughout their NAF academy experience, students will learn how to develop, sustain, and leverage network connections to help them pursue their college and career aspirations and build their social capital. This network will layer in an ecosystem of support on top of a student’s hard work and capabilities. Looking back, many of us can identify close professional or personal relationships that helped point us in the right direction or bolster our confidence in our abilities to pursue an opportunity. While degrees and credentials are certainly important, a young person can greatly benefit from the connections they are able to make to provide them with opportunities to demonstrate their abilities. As part of this new approach, NAF will provide content, services, and engagements to help students intentionally build these professional relationships, so this important component is not left to chance.

SKILL SETS

COLLABORATION

P R O B L E MS O L V I N G

COMMUNICATION

P L A N N I N G F O RS U C C E S S

S O C I A LA W A R E N E S S

I N I T I A T I V E & S E L F -D I R E C T I O N

C O L L A B O R A T I O N + Encourages the ideas, opinions, and contributions of others, leveraging individual strengths + Provides feedback in a manner that is sensitive to others’ situation and feelings + Clarifies areas of disagreement/conflict that need to be addressed to achieve a common goal + Seeks to obtain resolution of disagreements/conflicts to achieve a common goal Also known as: accountability, cooperation, helpfulness, leadership, participation, conflict resolution, teamwork

C O M M U N I C A T I O N + Organizes information that serves the purpose of the message, context, and audience + Uses and adjusts communication strategies as needed based on the purpose of the message, context, and audience + Signals listening according to the rules/norms of the context and audience + Seeks input to gauge others’ understanding of the message + Asks questions to deepen and/or clarify one’s understanding when listening to others Also known as: listening, negotiation, persuasion, presentation, verbal communication, written communication

P R O B L E M S O L V I N G + Defines problems by considering all potential parts and related causes + Gathers and organizes relevant information about a problem from multiple sources + Generates potential solutions to a problem, seeking and leveraging diverse perspectives + Identifies alternative ideas/processes that are more effective than the ones previously used/suggested + Evaluates the advantages and disadvantages associated with each potential solution identified for a problem + Selects and implements the best solution based on evaluation of advantages and disadvantages of each potential solution Also known as: creative, critical thinking, decision making, innovative, process-oriented, strategic

P L A N N I N G F O RS U C C E S S + Sets and prioritizes goals that reflect a self-awareness of one’s capabilities, interests, emotions, and/or needs + Breaks goals into actionable steps + Accurately estimates level of effort and establishes realistic timelines + Manages time to complete tasks on schedule + Applies existing/newly acquired knowledge, skills, and/or strategies that one determines to be useful for achieving goals + Monitors progress and own performance, adjusting approach as necessary + Demonstrates a belief that one’s own actions are associated with goal attainment Also known as: ambition, assertive, goal-setting, impulse control, mastery-focuses goals, metacognitive strategies, motivation, multi-task, plan, punctual, preparedness, self-regulation, self-reliance, time management

S O C I A L A W A R N E S S + Demonstrates integrity and situational ethics + Recognizes the consequences of one’s actions + Balances own needs with the needs of others + Takes into consideration others’ situation/feelings + Develops and implements strategies for navigating in different cultures/contexts Also known as: appreciation, code-switching, cultural competence, diversity, empathy, emotional intelligence, ethics, fit, global awareness, perceptive, responsibility, social intelligence, social skills

I N I T I A T I V E &S E L F - D I R E C T I O N + Demonstrates awareness of own abilities and performance + Seeks knowledge and information proactively to build understanding and anticipate what is needed + Needs minimal supervision to complete tasks + Attempts to complete tasks independently before asking for help + Follows rules/directions as required by the task/situation + Maintains focus on tasks despite internal and/or external distractions + Strives to overcome barriers or set-backs, seeking assistance when needed + Adapts approach in response to new conditions or others’ actions Also known as: adaptable, agile, dedicated, focus, grit, motivated, optimism, perseverance, persistent, project management, self-disciple, work ethic

Youth Participatory Action Research

PAR affects youth, their learning journeys, and learning environments in numerous ways. Youth-level outcomes are relevant career and college readiness program missions to ensure that students are college, career, and future-ready. PAR has been shown to: + improve academic outcomes + strengthen communication (especially with adults), problem-solving, and critical-thinking skills + improve information gathering and analysis skills + increase awareness of and exposure to higher education + foster civic engagement, civic identities, and political awareness + promote intergenerational relationships + strengthen youth's realization that their voice matters PAR can also improve formal or informal learning environments: “it is an opportunity to consider issues from the point of view of youth, which can be beneficial because it reveal points of school life that can go unnoticed by adults but are significant to the students. Therefore youth can be a source of creative ideas for improving schools, improving student interest and participation.”

What to consider when doing YPAR? + Are adult allies trusted by youth? Will they support and empower youth as agents for change? Are adult allies committed to participatory action-oriented, and shared ownership processes? Will they be receptive to youth-led recommendations for change? + Do pre-existing power structures within the learning environment allow for power-sharing? Will youth be permitted to lead the selection and diagnosis of the issue without authorities rejecting chosen themes? Will adultism hinder youth-centered PAR? + Will youth be compensated for their efforts, financially, via credits, or other support (e.g., food during meetings, supplies to aid the process)? Learning environments should be designed to accommodate other aspects of the lives of young people and competing interests (e.g., caregiving, test preparation). + Will administrators, principals, and other gatekeepers gladly accept youth-centered PAR? Are these stakeholders ready to leverage their power and influence to make PAR happen at their sites and connect youth to adult allies with resources? + Will PAR fit into existing schedules? Can learning environments provide the time and flexibility needed for PAR to succeed? Will sites provide youth and adult allies with the space and materials to support participation and follow-through in the processes?

PAR advocates that those most affected by an issue, such as youth, be involved throughout the process. In order words, youth are not just passive participants, but active in making decisions through the process.

Action is achieved through a reflective cycle, whereby youth and their adult allies collect and analyze data on an issue, then determine what action should follow. The purpose of an action is to change or improve an issue being researched. You can think of Action-oriented as providing students with real-life work experiences organized around an issue where they can apply a range of skills from leadership, communication, collaboration, and critical thinking, to develop a deeper understanding through active exploration and take ownership of their learning and their future employability.

PAR is driven by those with a stake in an issue. Outside stakeholders (such as researchers) can support youth and their adult allies, but youth and allies drive the process.

“Youth-centered PAR is an “innovative approach to positive youth and community development based in social justice principles in which young people are trained to conduct systematic research to improve their lives, their communities, and the institutions intended to serve them.”

Issue Identification is the youth-led selection and diagnosis of a salient issue at school or in the community

Action Planning what actions can be taken to address an issue. Examples: + Preparing for information-gathering efforts such as data collection. + Providing decision-makers with a debrief on what youth have learned up to this point. + Planning an event.

Action Taking is implementing planned action(s). Examples: + Administering a survey with fellow students to learn about their future readiness needs + Debriefing principals or program managers with what students learned from their Issue Identification process + Piloting of a new internship program or career readiness curriculum + Hosting event (e.g., a career fair or a community workshop)

Evaluation & Reflection is taking stock of the effects of action-taking by gathering and analysis information/feedback from those who participated in the action. It involves youth and adult allies reflecting on what they have learned and the youth’s experiences with the process to determine the next steps, which could include another PAR cycle (e.g., starting with planning). And sharing what was done and learned with decision-makers.

NAF Youth PAR Resources

NAF PAR Toolkit:Youth Engagement + PAR 101

NAF PAR Toolkit: What does it mean to be an Adult Ally in Youth PAR?

NAF PAR Toolkit: How can we successfully implement PAR with Young People?

NAF PAR Toolkit: Are you READY for Youth PAR?

Introductory Videos:

Youth PAR Literature Review

We Know Us: How to Engage Young People

YPAR Toolkit: With activities and lesson plans

How to Use the NAF Youth PAR Toolkit for More Equitable, Mission-Aligned Evaluation Practices and College & Career Readiness Programs

YPARScenario #1

How to Building Collaborative Practices

YPAR Scenario #2

Developing a Culture Amplifying Youth Voice and Youth-informed Action

YPAR Scenario #3

EncouragingYouth-led Action for Change

Scenario #1 - How to Building Collaborative Practices?

Using the toolkit’s participatory activities to supplement existing data collection activities, support dialogue, increase youth voice in program development or evaluation, or have more engaged learning in classrooms or college & career readiness programs.

Example #1

Supplementing Data Collection

Example #2

Exploring with youth a data trend or issue that arises from collected data

Supplementing Data Collection Your college and career readiness program administers with youth participants a survey at the end of the program cycle/year. You would like to supplement that data collection to capture more nuanced, qualitative feedback from youth. You would also like this data collection opportunity to (1) align with your organization’s college and career readiness mission and (2) be an opportunity for youth participants to showcase the skills they have developed in your program. You could use the Temperature Gauge activity to have participants map out the high points and low points of their experiences in your program. Click here to access NAF's Temperature Gauge Facilitator Agenda. Temperature Gauge

Exploring with youth a data trend or issue that arises from collected data A unique issue arises from the administration of your college and career readiness program’s end-of-year data collection activities with youth participants. You’re not sure why this issue occurred or what it means for youth participants and your program. You want to organize a focus group or a sensemaking meeting to review the issue or data trend with youth participants and explore the subject in greater depth. You could use the Force Field Analysis or Fish and Boulders activity. If you need a higher degree of anonymity, a Graffiti Wall might be a perfect fit! These activities allow youth to explore the issue by identifying supportive forces, obstacles, and barriers. Click on the activities to be linked to NAF's Facilitator Agendas. Force Field Analysis

College and career readiness program managers can use specific guides NAF’s PAR toolkit to structure conversations or learning experiences without having to do a complete PAR project. NAF's PAR toolkit includes guides to support program managers to implement powerful participatory activities with students. These activities can provide students with a space to share their ideas, engage in collaborative experiences, consider data and different perspectives, and strengthen skills crucial to improving conditions and identifying solutions.

Scenario #2 - Developing a culture amplifying youth voice and youth informed action

Let's explore how youth participants and educators could structure a "mini-PAR process" to tackle a specific issue in a meaningful and rapid way. Click the box for the scenario background, followed by each element in the PAR process for links to resources.

Issue Identification Youth should lead the exploration of why the communication issue exists by using the NAF PAR Toolkit’s 5 Whys activity (root cause analysis). A root cause analysis aims to engage youth in diagnosing the selected (YPAR) issue by determining what is causing or driving it based on lived experiences. NAF provides alternative issue identification activities in the 5 Whys guide.

Planning With adult support, youth can explore what information they need based on their Root Cause Analysis outcomes to plan specific actions or the next steps. Check out the 6-Ws activity in the NAF PAR Toolkit. Once youth have identified what information they need, use NAF's "Deciding & Planning on Actions to Take" to facilitate the conversation and shared decision-making on the next step - taking action! Given the nature of a "mini-PAR" project, the planned action may be limited to whatever information can be gathered with your existing group of YPAR participants (i.e., research) or preparing to debrief program or school decision-makers on what the process has revealed.

Taking Action The NAF PAR toolkit includes several "action-taking" resources to aid youth and adults in gathering information, speaking with influential people, and planning an event. They could help develop recommendations based on the collected data and present these to the program’s designers or decision-makers. Refer to the table on page 15 of the NAF PAR toolkit for action-specific resources (e.g., research, debriefing, events).

Evaluation After youth have planned and taken action is an excellent opportunity to learn from their experiences with the "mini-PAR" process and how it affected them. Depending on the project timeline and what actions were taken, youth could also assess how their actions may have affected those who took part and whether the actions met their objectives. Also, consider supplementing your evaluation activities by linking the "mini-PAR" process to your college and career readiness outcomes to understand better it can support the kinds of future-ready skills we hope to see develop or strengthen in youth participants. The NAF PAR Toolkit includes:

• Fun, interactive activities youth could implement to evaluate or assess how their actions affected other people and met their objections. Check-out page 16 "Other Resources" in the Toolkit.
• A sense-making guide to support youth and adults to interpret the information they have collected on implemed actions. We also include a number of other evaluation resources on pages 16/17 in the Toolkit.
• A post-PAR experience guide to facilitate reflections on youth's experiences with the mini-PAR process.

Learning Staff (with support from youth participants) can summarize what they learned from the mini-process and decide how to improve the program. Refer to the Evaluation resources.

Let’s imagine your end-of-year data collection activities uncovered that youth participants are unaware of work-based learning opportunities. Consequently, your college and career readiness staff want to organize a weekend or week-long process to identify ways to improve and standardize communication about work-based learning (WBL) opportunities at the local and regional levels. A "mini PAR process" would allow youth participants (the “users” of WBL opportunities) to explore how best to adapt communication and outreach strategies to encourage youths’ greater participation in these experiences. Or perhaps the process might uncover that the issue is not "communication" but the relevance of the WBL experiences being offered to youth.

Scenario #3 - Encouraging Youth-led Action for Change

College and Career Readiness Programs could use PAR to structure the development of youth advisory groups, culminating graduation projects, or youth-led solutions to a specific issue. Click the box for the scenario background and each element in the PAR process for links to resources.

Issue Identification Youth should lead the identification of an issue relevant to their career and college readiness experiences and needs (or context). They should also lead the exploration of that issue and why it exists based on their lived experiences. The NAF PAR Toolkit includes excellent background information and resources for starting a complete YPAR project. We also include guides to help with the facilitation of Issue Identification and a Root Cause Analysis (5 Whys). All of the guides include alternative or supplemental activities.

Planning Once youth have identified and explored an issue, they should identify what information they need relative to that issue. Listing out what we know and what we don't is crucial to planning any project's next steps. The NAF PAR Toolkit includes step-by-step guides for activities like the 6 Ws - Asking Questions. Once youth understand better what they know and need to know about an issue, it's time for them to plan to take action. We recommend youth consider one or a combination of the following types of action - do some research, debrief stakeholders, or plan an event. Use this guide, Deciding & Planning on What Actions, to help youth and adults plan how best to address the college and career readiness (or other) they identified.

Taking Action The NAF PAR toolkit includes several "action-taking" resources to aid youth and adults in gathering information, speaking with influential people, and planning events. Refer to the table on page 15 of the NAF PAR toolkit for numerous action-specific resources developed by NAF and other organizations specializing in YPAR. For example, are guides on:

• Speaking to powerful people
• Engaging your school principal
• Developing an Elevator pitch
• presentation skills
• public speaking skills
• communicating with difficult people
• Event planning

You'll also find guidance on how youth can collect data by developing surveys or conducting focus groups or interviews.

Evaluation After the youth have planned and taken action is an excellent opportunity (1) assess how their actions may have affected those who took part and whether those actions met their objectives; and (2) learn from youth's experiences with the process and how it affected them. The NAF PAR Toolkit includes the following:

• Fun, interactive activities youth could implement to evaluate or assess how their actions affected other people and met their objections. Check-out page 16 "Other Resources" in the Toolkit.
• A sense-making guide to support youth and adults to interpret the information they have collected on implemed actions. We also include a number of other evaluation resources on pages 16/17 in the Toolkit.
• A post-PAR experience guide to facilitate reflections on youth's experiences with the mini-PAR process.

Learning Information and rich insights have been gathered, and youth have had future-readiness-building experiences. Reflect on what was learned and work with youth to identify program, project, and process-related recommendations. The "Evaluation Resources" include strategies for collaboratively developing with youth recommendations.

For this scenario, College and Career Readiness Programs could use PAR to develop culturally responsive practices that reflect youth participants' lived experiences and not adults' assumptions of those experiences. In this example, PAR would be a longer-term process, similar to NAF’s PAR pilot. At the start of the school year, your program could develop a youth advisory group to inform program design and evaluation or a culminating graduation project. Or perhaps, this is an excellent opportunity to revamp in collaboration with youth a particular aspect of your program (i.e., communication & outreach).

NAF Youth PAR Pilot Project

NAF, a national college and career future readiness organization supporting small learning communities in high schools.

NAF is seeking to augment youth engagement and youth voice in local decision-making and planning to advance more equitable learning experiences and environments.

Pilot the toolkit with one school in Birmingham, Al. The student-identified issue was not being future-ready; unaware of what to expect & a lack of accessible and relevant college and career resources.

Toolkit relies on Participatory Action Research (PAR) and is flexible to meet diverse youth engagement, learning, and experience goals.

Planned as short-term in-person processes across multiple sites before the pandemic and implemented virtually and longer-term with one school during the pandemic.

Youth received a stipend

NAF Youth PAR Pilot Project Process & Timeline

ADULT-ALLY READINESS& PRE-PLANNING

PLANNING

ACTION

EVALUATION &REFLECTION

AUGUST 2020

NOVEMBER 2020

FEBRUARY 2021

April 2021 & MAY 2022

ADULT-ALLY READINESS & PRE-PLANNING Introduction to PAR as a tool for Youth Engagement & Evaluation of Learning Experiences Adult Ally Readiness Assessment Project Logistics Planning

PLANNING PAR Orientation Issue Selection Assessment of Knowledge & Gaps Define PAR Questions Data Collection 101 Meeting with Adult Allies

ACTION Develop 2 Data Collection Tools (Peers, Educators) Collect Data from Peers & Educators Quantitative & Qualitative Data Analysis Identifying Findings & Recommendations Presentation of Findings & Recommendations to Adult Allies

EVALUATION & REFLECTION Post PAR Journey Youth Researcher Survey Making Meaning of Youth Researcher Survey Findings Season Facilitator Discussion with Adult Allies One-Year Reunion & Reflection Workshop with Youth Researchers

Youth Reflections & Evaluation of the Pilot |

REFLECTIONS ON A NAF PAR JOURNEY BY WENONAH HIGH SCHOOL YOUTH RESEARCHERS - AN IMPACTFUL FUTURE-READINESS EXPERIENCE!

STAGES OF THEJOURNEY

ACTIVITIES

FEELINGS

Adult FacilitatedYouth Led Actions

NOVEMBER 2020

FEBRUARY 2021

APRIL 2021 & MAY 2022

PLANNING

PAR OrientationIssue SelectionAssessment of Knowledge & GapsDefine PAR QuestionsData Collection 101Meeting with Adult Allies

Develop 2 Data Collection Tools(Peers, Educators)Collect Data from Peers & EducatorsQuantitative & Qualitative Data AnalysisIdentifying Findings & RecommendationsPresentation of Findings & Recommendations to Adult Allies

Post PAR Journey Youth Researcher SurveyMaking Meaning of Youth Researcher Survey Findings SeasonFacilitator Discussion with Adult AlliesOne-Year Reunion & Reflection Workshop with Youth Researchers

ACTION

EVALUATION &REFLECTION

We were uncertain about PAR as adults don't engage students in decision-making.

We were disappointed that we could not recruit more students to participate in the PAR project.

We struggled to build trust as a group of peers as we were not all friends or in the same grade, and with the adult allies. We could not trust that the adults genuinely cared and had to trust the process.

We enjoyed the brainstorming and other PAR planning activities; they were engaging & an excellent opportunity to reflect critically on what matters to students.

We, the youth researchers, selected a relevant school-based issue for the PAR Project, not the adults. The most salient topic? Future readiness!

Engaging in activities and having to build consensus has improved our abilities to address problems and challenging situations.

We appreciated having adult allies support us on this journey and participate.

But we felt very vulnerable or dismissed when trying to speak our truth or about our experiences, as some adult allies reacted negatively without letting us finish.

We don't appreciate it when adults speak over us or try to rephrase what we are trying to say.

Over 200 students and nearly all of our teachers participated in our PAR project by completing our surveys.

We were very disappointed that more seniors did not complete the survey to share their experiences, which were relevant and timely.

We learned how to use Excel or Google Sheets to analyze the quantitative & qualitative data from our peers and teachers.

We were thankful to have a team of caring peers who could pick up the slack when life happened.

We were compensated for our work on the PAR project; our time and expertise were valuable!

The PAR project was an excellent future readiness opportunity! We got to experience what felt like a real first job and have something different to add to our resumes.

Some of us had awful internet connections, making it challenging to join meetings, engage in conversation, or complete the work.

We feel stronger speaking up as a result of the PAR Project! The experience also helped improve our communication skills with adults.

The PAR project overall - We were committed and had an impact! We were proud of our small and loyal group of students dedicated to sparking change. We learned that a small number of people could do a lot!

Nonetheless, there were real struggles! As a result of the pandemic, we struggled with mental health issues; it was hard for us to communicate these struggles with our peers or adult allies.

NAF Youth PAR Session Q&A

YPAR is a robust process to ensure college access and future-readiness programs engage in equitable and mission-aligned evaluation practices to better meet the evolving needs of young people while simultaneously strengthening their readiness competencies.Questions?

DEE - Anything beyond this points is left over materials from the webinar

NAF Youth PAR Pilot Project Context - Wenonah High School Academy of Hospitality & Tourism

NOVEMBER 2020

FEBRUARY 2021

MAY 2022

PLANNING

• Issue selection process
• Root cause analysis using “5 Whys”
• Assessing knowledge gaps using “6 Ws”
• Developing PAR questions process; supportive activities like “Identifying Informants”Data collection 101 orientation, supportive activities like “Good Survey, Bad Survey”

ACTION

• Developing data collection tool step-by-step process
• Google Forms to collect survey data
• Taught thematic analysis and coding using excel; youth analyzed the data
• Data interpretation, judgment, and recommendation process
• Youth presented findings to the principal, educators, district leader

EVALUATION & REFLECTION

• Post-experience survey implemented with youth and focus group with youth using “Body Mapping,” “Temperature Gauge
• Adult conducted a preliminary analysis of the survey and focus group data
• Data confirmation and interpretation process with youth
• One year post-ex in-person reflection session with youth (and adult allies) using “What, So, What, What Now?” “Temperature Gauge” and “Dot Democracy”